Two swarm-wide facts were being read off this machine's deploy set, so the
answer differed between two hosts of one swarm:
- `swarm.authelia.oidc.hiveIdentities` defaulted to `deploy.nats.enable`,
so whether a hive gets an identity at all depended on whether the IdP
host happened to also run the queue. It is on by default now: a swarm's
hives have identities, and the clients are inert until used.
- `swarm.statusPublish.tokenEndpoint` defaulted through `queueLocal`
(`deploy.nats.enable && deploy.authelia.enable`), so a hive that was not
the swarm host had no token endpoint even when the swarm's IdP was
reachable and named. It follows `swarm.authelia.url` now — the same
derivation `swarm-controller.nix`'s own `queue.tokenEndpoint` already
uses, which is correct for a remote provider.
`deploy.nix:1-30` is what makes this a rule rather than a preference:
`swarm.*` is "identical on every host, byte for byte" and `deploy.*` is
"necessarily different on every host". A swarm value derived from a deploy
value cannot satisfy both.
The all-or-nothing status-publish assertion follows: the token endpoint is
no longer one of the coordinates that says this hive publishes — every hive
in a swarm with an IdP has one — so the two per-host coordinates are what
must agree, and they now require the endpoint rather than being counted
beside it.
`queueLocal` itself stays for the three remaining host-local addresses
(`natsUrl`, `clientSecretFile`, `agentNatsUrl`): each of those is a
`deploy.*` value that genuinely differs per host.
Closes#4048
`services.hyperhive.gateway.enable`, `gateway.dns.enable` and
`network.enable` replace the `hyperhive.enable` gate on all three. Each
defaults to false; the modules that need one assert it with `mkDefault
true` from inside the guard their own deployment already carries, and
`swarm-required-services.nix` — the module that owns what the
swarm-services toggle implies — asserts all three explicitly.
hive-c0re asserts all three unconditionally, so an ordinary hive keeps
getting them with no opt-in: it is the host's only knowledge that agent
containers exist.
The resolver moves to its own `hive-gateway/dns.nix` so it can be gated
without reindenting the nginx half of the module.
Reinstates `network.enable`, dropping its `mkRemovedOptionModule` shim.
A config still carrying `network.enable = false` from before the removal
now switches the bridge off instead of failing eval.
Also deletes a duplicate `centralToggleOff` fixture in nix/module-eval.nix.
Two sibling slices added it independently (c5f60fd5, ce3b3d94); the merge
was textually clean and left `main` failing to evaluate at all, so this
file could not be gated without removing one.
The rebase onto main resolved an additive conflict in
module-eval.nix by keeping both fixture blocks, leaving
centralToggleOff defined twice and failing nix flake check with
'attribute centralToggleOff already defined'. Drop the later
duplicate; the earlier definition already covers every downstream
case that references it.
`systemd.paths.forge-avatar-sync` was gated on `agent.icon != null` alone,
while the `systemd.services.forge-avatar-sync` it triggers is gated on
`agent.icon != null && agent.forge.url != null`. An agent with an icon and no
forge URL therefore rendered a `.path` unit, pulled into multi-user.target,
watching for a forge-token whose arrival would activate a unit that does not
exist.
The module already documents the fixed behaviour: `forge.url`'s own option
description says the tea-login and avatar-sync units are "not generated at all"
when it is null --- an absent integration, never a misdirected one. That
sentence was true of the oneshot and false of its watcher.
Latent, not live: hive-c0re renders `forge.url` into every agent's config from
the host's `HIVE_FORGE_URL`, so on a real hive it is always set and the
asymmetric arm is unreachable. It is reachable wherever the agent modules are
evaluated outside a hive.
A module-eval case pins both halves absent for an agent with an icon and no
forge; it fails on the parent commit, where the path unit renders.
The receiver was defined for `scrapeTargets != {} || publishedScrapeTargets
!= {}`, but the swarm-tier pipeline named it only for `scrapeTargets != {}`.
A collector with published targets and no loopback ones therefore rendered
scrape configs that belonged to no pipeline: requested, parsed, delivered
nowhere, and valid enough to deploy.
Latent, not live: the module seeds `scrapeTargets.collector` under its own
`enable`, so the loopback set is never empty in a real deploy and the
disagreeing arm is unreachable today. Nothing would have noticed if that
seeding became conditional, which is why the module-eval case pins the
pipeline rather than trusting the seed to keep masking it.
Refs #4511
The base set (skill-creator + base@hyperhive) was declared via the
option's `default`, so a per-agent definition of claudePlugins
replaced it wholesale. Move the base set to a plain `config`
definition instead: a listOf option merges multiple plain definitions
by concatenation, so an agent's own list now adds to the base set
rather than replacing it, while lib.mkForce / lib.mkOverride on the
agent side still replace the whole merged list deliberately (mkDefault
was ruled out explicitly).
Also de-dup at the JSON-render site with lib.unique, so an agent that
names a base-set entry itself doesn't get it installed twice, and
reword the option doc, which still claimed the old REPLACES semantics.
Four module-eval cases cover the unset / agent-adds / mkForce-replaces
/ duplicate-entry shapes.
Refs #4467
`services.hyperhive.enable` is going away (Refs #4500). These four blocks
ANDed it with a second condition that was already the load-bearing one:
the bridge-firewall hole asks whether an operator named any
`network.exposeHostPorts`, and the three swarm-level services ask their
own `deploy.*.enable`, each of which defaults false (swarm-ui derives
from the controller's toggle, a sibling deployment decision) and none of
which is derived from the hive toggle. So dropping the conjunct turns
nothing on by itself — the remaining condition still decides.
The way that claim fails is by something becoming unconditional, so
module-eval gets absence arms for all four, each asserted on a host with
the hive ON and on one with it OFF, plus the two controls that make the
absences mean something.
The swarm-controller arm is probed by the credential oneshot and by the
daemon's `ExecStart` rather than by the unit name: hive-tls defines an
environment key on that name, which leaves an inert fragment behind on
any hive with a CA whether or not the controller runs there.
hive-forge's publicUrl and behindGateway, and swarm-controller's
forgeTokenFile, no longer gate their default on
config.services.hyperhive.enable — none of the three has a reader that
depends on hyperhive being enabled to make sense of the value, so the
extra condition only added a hidden coupling. module-eval.nix gains a
centralToggleOff fixture plus four cases asserting each affected
default now resolves identically whether the toggle is on or off.
Refs #4500
`swarm-controller` mints an agent's mTLS leaf at creation and publishes it
at `swarm/agents/<agent>/bao-mtls`. Nothing read it back. This adds the
hop that carries it the rest of the way, and the in-container consumer
that proves the hop works.
Host side, `lifecycle::agent_identity` reads the row under *this hive's*
own certificate — the hive is a principal the store already knows — and
stages the leaf and its key `0600` under a new `agent-identity/<name>`
state dir, deliberately outside every bind-mounted tree. Both files go in
as systemd credentials rather than binds, the same answer and the same
mode reason as the queue secret beside it: the staged key is unreadable
to the unprivileged agent user, and the container manager reads a
`--load-credential` source as root before re-exposing it under the
consuming unit's own `User=`. The agent is never asked to authenticate in
order to obtain the thing it authenticates with.
Container side, `hive-agent-bao-identity.service` logs in with that
certificate and reads the agent's own path back, failing the unit when
either step does not succeed. It fails loudly where the hive-side readers
degrade quietly, because a refused certificate means an agent that
believes it reaches the store and never does — a cause only the login
itself can name.
The address is the whole switch, no separate `enable`, matching how
`queue.nix` and `logs.nix` already gate themselves. A hive with a store
forwards `HIVE_AGENT_BAO_ADDR` and every agent on it gets the check; a
hive without one forwards nothing and no agent does. That is what keeps
the delivery from landing in a container with nothing to read it.
The hive can now reach an agent's identity, so hive privilege covers
agent privilege. Accepted, not mitigated: the alternative is an agent
fetching its own credential with a credential it does not yet have.
Refs #4137
`services.hyperhive.agent.matrix.enable` was a second source of truth for
a fact the account set already carried: after ①-③ the hive-internal
`main` account is an ordinary `matrixAccounts` entry, so "does this agent
have matrix" and "does this agent have an account" were the same question
asked twice, with the boolean able to disagree.
The option is gone and a non-empty `matrixAccounts` now gates the daemon
unit, its token path-watcher and the injected `extraMcpServers.matrix`
entry.
That is only a real condition because `matrixAccounts.main` is itself
gated: it is declared when `matrix.url != null`, never unconditionally. A
`main` with no homeserver is an account the daemon can never log in as,
so declaring one always would have made the signal trivially true and
turned matrix on for every agent in every hive. With the URL gate, the
empty set is reachable exactly for an agent the hive gave no homeserver
and whose operator declared no account of its own — the state the old
`enable = false` expressed.
Assertions: "extras require enable" is deleted, having become the
definition of the thing it checked (an external-only account with its own
homeserver is now rendered rather than rejected). `main.tokenFile` stays
pinned, re-guarded on `? main` instead of on the flag, since `main` is
absent whenever the URL is null and an unguarded index would throw there.
Both spellings of the option get `mkRemovedOptionModule`, following
../host-modules/deploy.nix's registrationTokenFile pair rather than a
silent delete: the definition whose meaning changes is `false`, and left
undeclared it would be ignored and hand the agent the tools its operator
turned off. Failing the eval with the replacement spelling is the only
outcome that cannot.
module-eval gains the three arms — URL, nothing, external-only — with the
middle one carrying why it exists: it is the only thing in the suite that
would notice `main` becoming unconditional again.
Refs #4475
Same slice, same fact: socketPath lost its pre-rename path, so only three
old paths are defined in the fixture now, not four. Missed this sentence
while fixing the count three lines below it.
readOnly means socketPath can only ever hold the default, so the fixture's
old custom value can no longer reach the unit and the check.module-eval
assertion for it was failing. Assert the default instead of deleting the
check -- it still catches the env var vanishing or being misspelled, which
deleting it would not.
Also corrects the block's stale 'FOUR movers' count: socketPath lost its
pre-rename shim in the same slice, so only three of the asserted values
are still rename-shim movers.
readOnly makes a bad socketPath inexpressible rather than diagnosed
after the fact. Since readOnly rejects any definition including one
arriving through a rename shim, drop the deploy.nix rename entry for
it and the fixture line exercising it, and update the controllerOldPath
comment's shim count from seven to six.
Refs #4208
`glue-matrix-bao-token.nix` has read
`secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but
nothing ever wrote that path. The store was empty in every deployment, so
every read degraded to "keep what activation minted" and each hive stayed
the origin of a value the swarm has to agree on — two hives never
converged.
`swarm-secret-publish` is now the producer. It already holds a store
identity, already writes under the hive prefix, and already runs per
hive in the roster, so the mint is a third loop beside the two OIDC
copies rather than a second shape of this unit.
Idempotence comes from a record of its own, not from the store: this
principal is granted `create`/`update` with no `read`, so it cannot ask
whether a hive already has a token. It keeps what it minted under
`StateDirectory=` (0700 dir, 0600 file) and mints only when that file is
missing or empty; the `put` runs every time, because re-putting the same
bytes changes nothing for a reader while a mint whose publish failed must
not be left as a token this host holds and no hive can reach.
The token never becomes a nix literal and never reaches argv: the mint
redirects into a file, and the publish hands bao `value=@<path>` so bao
opens it itself — the same handling the OIDC loops use.
`hive-matrix.nix`'s activation mint stays as the genuine first-boot
fallback. It already fires only when the token file is absent, so it
cannot clobber a value the store delivered; `hs_token` has no swarm half
and is still minted there for real.
Refs #4402
Nothing reads it any more: hive-c0re creates accounts as the hive's
appservice, so the mint, the host file, the bind mount, the
`LoadCredential` entry and tuwunel's `registration_token_file` all go.
⚠️ `allow_registration` has to go to `false` in the same change, and not
as hardening. tuwunel refuses to START when registration is allowed with
no token configured — it demands
`yes_i_am_very_very_sure_…_open_registration_…` instead — so dropping the
token and leaving the flag true is not a lax homeserver, it is one that
does not boot. The flag is checked only for requests arriving without an
appservice token, so hive-c0re provisions exactly as before and everyone
else is refused outright.
The swarm secret store keeps its role, repointed at the credential that
replaced the token (`swarm/hives/<hive>/matrix/appservice-token`). Its
unit now also re-runs hive-matrix's own registration renderer after
writing the file: the token is half an agreement, and a registration
still naming the previous value authenticates nobody. The renderer is
shared through an internal option rather than copied, so the
registration's shape has one home.
Both spellings of `registrationTokenFile` become
`mkRemovedOptionModule` with a message naming what replaced them. A hive
that never set the option — the default — is unaffected; one that pinned
it fails to evaluate with instructions instead of a silent no-op.
An upgraded hive needs no intervention: the activation script has both
halves in place before the homeserver restarts, existing agents keep the
tokens their devices already hold, and the old token file is left on
disk read by nothing. docs/integrations/matrix.md spells the path out.
Refs #4402
The swarm can already tell whether an agent is alive — the `agent-status`
KV bucket republishes once a minute — but not what it is doing right now.
A header bar wants the second thing, and a minute-old answer to "is this
agent thinking" is the wrong answer most of the time it is read.
`hive-agent` now publishes a turn-state header to
`$SWARM.agent-state.<hive>.<agent>`, a core subject beside the terminal
rows it already sends. It goes out **on transition, not on a timer**: the
publisher watches the event bus, rebuilds the header, and sends only when
the serialised result differs from the last one it sent — so a second
periodic writer, which is the problem this exists to fix, is not what
replaces the bucket.
The payload is the published contract a swarm-level renderer is written
against, so the test asserts on the serialised JSON keys rather than on
Rust field names. Two fields deliberately depart from the per-agent web
UI's `StateSnapshot`: `turn_state_since` is an ISO 8601 UTC string rather
than unix seconds, matching the sibling `$SWARM.term` subject's stamp, and
`agent_state` carries the swarm's own `AgentState` vocabulary rather than
a `paused` boolean, so a reader can compare actual against wanted without
translating. `turn_state` and `agent_state` stay two separate fields:
neither vocabulary contains the other's values.
Swarm-side, `GET /api/agents/{name}/state/stream` relays the subject as
SSE, resolving the agent's hive at request time exactly as the terminal
stream does and passing the bytes through without parsing them.
The broker grant is a second `--agent-publish-subject` rather than a
widening of the existing one, so the terminal family and the header family
stay independently revocable, and a `module-eval` arm pins the rendered
flag and its argument together — the doubled dollar included, since a
single one expands to nothing in `ExecStart` and yields a grant that
matches nothing.
Refs #3802
otelNoIdentity points swarm.authelia.url at a remote authelia while
deploy.authelia.enable stays false, so ssoLocal is false and nothing
mkDefaults the forge's deploy.forgejo.sso.clientSecretFile. The forge
module's config is gated only on services.hyperhive.enable (there is
no deploy.forgejo.enable to opt out of), so its SSO assertion fired
against every fixture that lacks the secret — entirely orthogonal to
what this fixture tests. Supply the by-hand path, mirroring
otelRemoteAuthelia's own hand-delivered secret.
The swarm.otel.journaldUnits = [ "nginx" ] line added in the previous
commit targeted an assertion that was never firing: journaldUnits
already defaults non-empty, so the effective list just contained
"nginx" twice. Removed as redundant.
Refs #4374
Refs #4374
- otelNoIdentity: name journaldUnits so the fixture trips the store-
identity path it's testing instead of swarm-otel's unrelated
journald-safety assertion (an empty list with log collection on is
refused as "collect everything", not "collect nothing").
- otelNoStores: give it a bao client identity. The secret gate moved
from deployCfg.authelia.enable to a real client cert/key pair, so a
fixture meaning "no telemetry stores" now needs its own secret
identity to keep exercising the exporter/authenticator wiring it
was written for.
- docs/swarm/secrets.md: two vale fixes — a contraction, and drop a
condescending "simply".
The swarm collector's OIDC client secret only existed where authelia
did: `swarm-otel-oidc-secret.service` copied the minted plaintext out
of authelia's container tree, reachable only because the two share a
host's network namespace. A swarm that placed authelia elsewhere
delivered nothing, and the option's own description said so —
"a deployment that places authelia elsewhere points this at a file it
delivers itself." Same gap as #3853 and #4234, and this is the
swarm-otel twin of #4234's fix for Grafana.
Mirrors PR #4361 (Grafana) almost exactly:
- `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, in every
deployment, replacing the co-located copy unit outright — one
delivery route, not two, per the ruling that landed under #4234.
- Client registration moved out of `swarm-otel.nix`'s own `config`
block (gated on this host running the collector) into
`glue-swarm-otel-oidc-client.nix` (gated on this host running
authelia), the same split `glue-grafana-oidc-client.nix` made. It
was broken the same way: a split deployment registered the client
nowhere at all, so authelia never minted a secret for the publisher
to send on.
- The publisher's `services` prefix (write grant in `swarm-bao.nix`,
hive read grant in `policy::render`) already covers any service's
path — nothing to add there. `swarm-secret-publisher.nix` only grew
`serviceClientIds` by one entry.
One judgement call, stated rather than buried: the store-reading unit
renders only where this host holds a client identity
(`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting
it the way `swarm-grafana.nix` does. Grafana's local login form is
disabled unconditionally, so a Grafana with no OIDC secret has no way
in at all — that earns a hard refusal. This collector without a
credential still receives every hive's telemetry; only its own pushes
to the stores go out unauthenticated and get refused there, an
already-supported degrade the module's own `haveCollectorSecret` flag
named before this change. So the reading unit follows the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use
for their own optional readers: no unit when the identity is absent,
not a build refusal.
Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia`
are the positive pair (co-located and split, both reading through the
store), `otelNoIdentity` is the negative — no reading unit, no
assertion firing, `clientSecretFile` left null.
Refs #4258
An agent container cannot dial the store's loopback listener: the bridge
to-loopback DROP rule is there precisely to stop that, and the store
authenticates every reader by client certificate, so the usual answer —
a gateway vhost — is the one shape that cannot work. A terminating proxy
strips the certificate and bao sees nginx as the client for every hive.
nginx's stream module does not terminate. `ssl_preread` reads the SNI off
the ClientHello and splices the rest of the connection through byte for
byte, so bao completes the handshake itself and authenticates the client
it actually has. That is the no-vhost rule kept, not bent.
The listener binds the bridge IP rather than every address, because bao
already holds `127.0.0.1:<port>` in the same netns and a wildcard bind
there is EADDRINUSE — nginx would fail to start, taking the gateway with
it. Nothing moves as a result: the name already resolves two ways, so a
host-side reader still goes straight to loopback and an agent goes
through the passthrough, both on one `BAO_ADDR`.
Renders only inside the store's own `deploy.bao.enable` region; a host
that runs no store grows no listener and opens no port.
Per-agent certificates and per-agent policy are separate work.
Refs #4386
The previous commit left two delivery paths and a three-way gate:
`swarm-grafana-oidc-secret.service` copied authelia's minted plaintext
out of its host tree wherever the two were co-located,
`swarm-bao-grafana-oidc.service` read the same value from the swarm
secret store wherever they were not, and `ssoConfigured && (ssoLocal ||
haveClientIdentity)` decided whether Grafana got an OIDC block at all.
Delete the co-located path. The store reader is now THE delivery unit,
in every deployment — the publisher on authelia's host writes
`swarm/services/<id>/oidc/client` whether the reader is a network away
or in the container next door. The ruling behind it: the store exists so
a host holds ONE out-of-band secret, its client certificate, and reads
everything else with it. Skipping the store when the producer happens to
be local saves a round trip and costs a second delivery unit, a second
way for the file to be wrong, and a gate to choose between them.
The gate goes too, and both of its questions become assertions, scoped
to hosts that run Grafana:
- `swarm.authelia.url` must be set. `auth.disable_login_form` is
unconditional — Grafana ships an admin/admin account on a public
vhost — so dropping the OIDC block when the swarm names no IdP
produced a container with no SSO and no password box, silently. An
eval-time refusal naming the option is the only report that reaches
anyone, the shape swarm-nats.nix already uses for the same option.
- `deploy.bao.clientCertFile` / `clientKeyFile` must be set. This
replaces a warning that nothing reads back, and its message names both
options and where the leaf comes from.
Fixtures follow. `grafanaWithAuthelia` gains the cert pair, because a
co-located host is a store reader like any other. The old
`grafanaRemoteAutheliaNoIdentity` is kept rather than deleted, renamed
`grafanaNoIdentity`: the shape is still reachable, only its deliverable
changed from silence to a refusal, and an arm now reads that refusal
back. Its mirror `grafanaNoSso` covers the other assertion, each fixture
wrong in exactly one way so an arm can name which refusal fired. Every
positive keeps an explicit negative — the one-delivery-unit arm asserts
the deleted unit is absent in both topologies rather than merely that
the store reader is present.
Refs #4234
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grafana's OIDC client secret only existed where authelia did. One
`ssoLocal` gate — `grafana.enable && authelia.enable` — decided the
client registration, the minted secret's delivery and the whole
`auth.generic_oauth` block, so a swarm whose authelia runs on another
host got Grafana with no SSO wiring at all. The local login form is
disabled unconditionally, so that is no way in.
Split the one gate into the two questions it was conflating:
- `ssoConfigured` — does this SWARM have an identity provider
(`swarm.authelia.url`, which is swarm-wide and whose own description
makes null mean "no SSO configured"). With a delivery route present
this is what emits Grafana's OIDC block.
- `ssoLocal` — is authelia on THIS host, now spelled as the forge and
matrix modules spell it. It decides only which unit delivers the
secret.
Where authelia is elsewhere, `swarm-bao-grafana-oidc.service` reads the
secret from the swarm secret store, shaped after
glue-queue-agent-credential.nix: cert login fails loudly because a retry
fixes every state it fails on, the read degrades quietly because no
retry turns "no value there" into a value, and nothing writes a
stand-in. The producer is the publisher that already runs on authelia's
host, which gains the swarm's service clients beside the per-hive ones
at `swarm/services/<id>/oidc/client` — with the write grant in
swarm-bao.nix and the hive read grant in `policy::render` to match.
Registration moved to glue-grafana-oidc-client.nix. It has to be
declared where authelia's config is rendered, and swarm-grafana.nix's
config block hangs off this host running Grafana.
Two judgement calls stated rather than buried: a hive's read policy now
grants the whole `services` prefix, because a service's path names the
service and nothing swarm-wide records which hive runs it (cost recorded
in docs/trust-boundary/security.md); and the client is registered on any
authelia host, because no swarm-wide "this swarm has a Grafana" fact
exists to gate it on.
Refs #4234
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three cases beside the existing ingest ones, since the read route has the
same trap and one property of its own.
- the route exists and carries `auth_request`;
- it has no login fallback — with the browser location as the positive
control, so a pass means the two routes differ rather than that
`error_page` and the login host are absent from the whole vhost;
- it injects no filter, on `extraConfig` and on `proxyPass` both, because
VictoriaLogs takes its filters as request parameters and those ride an
upstream URI as easily as a directive. A filter arriving later is then a
visible diff here rather than a quiet change of rule.
Each absence arm leads with a clause that proves the location resolved and
that `hasInfix` finds what is really in that string, so "not there" cannot
be read off an unreadable path.
Refs #3870
systemd's default OOMPolicy=stop tears the whole unit down the moment the
kernel kills any process in its cgroup, so a single over-large subagent
takes the daemon and every sibling session with it — measured on a live
agent, both units show OOMPolicy=stop today, which is exactly the "every
live subagent session was cut mid-turn" symptom.
That blast radius is also what would make the preceding commit a bad
trade: deliberately putting this unit first in the OOM queue is only an
improvement if losing one subagent isn't losing all of them. OOMPolicy=
continue scopes the loss to the process the kernel actually chose, and
leaves the daemon alive to report the kill instead of vanishing and being
restarted with an empty session map.
Refs #4316
Both units ran at OOMScoreAdjust=0, so under container memory pressure the
kernel picked purely on footprint — and the agent's own claude is often
the fattest process in the container, which means the session supervising
the work died before the work did.
The sign is the load-bearing part and is easy to invert: a HIGHER
OOMScoreAdjust means MORE likely to be killed, because the kernel adds it
to the badness score it derives from the process's memory footprint and
then kills the highest scorer. So hive-subagent-daemon gets +500 (first in
line) and hive-agent gets -500 (last in line). Written backwards this
makes the reported bug worse rather than better, so module-eval pins the
order as an inequality.
Both values are inherited by the nested claude each unit spawns as a
child, so ordering the units orders the sessions underneath them. -500
rather than -1000 on the harness: fully exempting it would leave the
kernel nothing to kill in a container whose only large process is the
harness.
Refs #4316
The daemon spawns every nested claude as a plain child, so its cgroup is
already the "all subagents" cgroup — but it ran with MemoryHigh=infinity
and MemoryMax=infinity, so nothing slowed a subagent down before the
kernel's OOM killer stopped the unit and cut every live session with it.
MemoryHigh= and not MemoryMax=: a soft ceiling reclaims and stalls the
cgroup past two thirds of the container's cap, which turns a silent kill
into a visible throttle, while still letting a single subagent exceed its
share when the container has memory free. A hard per-agent cap would make
overprovisioning impossible, which is not wanted — most of the time
nothing in these sessions is compiling.
The fraction is taken from hyperhive.claudeMemoryMaxBytes, the container's
own effective MemoryMax= that meta.rs already bakes in per agent. When
that is null (an `infinity` or percentage cap) the unit renders no ceiling
rather than a fabricated constant, and module-eval pins both arms.
Refs #4316
The responder has had an agent arm since the principal was minted, but
no deployment ever passed `--agent-publish-subject`, and an empty list
is a refusal by design: `Policy::permissions` returns `None` rather than
a grant that can do nothing, so every agent was turned away at CONNECT.
The subject itself is the one this thread settled on, `$SWARM.term`
namespaced per hive.
The value has to reach the responder with a literal dollar. systemd
substitutes `$NAME` in `ExecStart` whether or not the word is quoted,
so a single dollar expands `SWARM` — unset, therefore empty — and the
responder is handed `.term.{hive}.>`. That grant validates (it carries
the `{hive}` placeholder), is accepted, and matches nothing any agent
publishes to, so the failure surfaces as an authorization violation far
from its cause. `$$` in the unit text is the escape for one dollar.
The module-eval case reads the rendered unit rather than the module
source, because the single-dollar version renders perfectly well; the
doubled dollar is the only thing that distinguishes them before deploy.
The grant is per-hive, not per-agent: an agent's identity names its
hive, so any agent in a hive can publish as another. That is the
tradeoff ruled acceptable for now, tracked separately for tightening.
Refs #3805
The swarm always has exactly one queue; a hive can only lack its
address. Reworded every prose site this PR added that stated or
implied the opposite, to name what is actually absent (coordinates,
credential, or address) instead of the queue itself.
Refs #3805
The host end: `HIVE_C0RE_AGENT_QUEUE_CREDENTIAL_DIR` tells the daemon
where the reader unit put the files, and a new
`deploy.hive-controller.queue.agentNatsUrl` says where the queue is as an
agent *container* reaches it. That address defaults to the bridge one and
never to loopback — `statusPublish.natsUrl` beside it is loopback and
correct, because hive-c0re shares the host netns and an agent does not.
Paired with the swarm's token endpoint, gated together, and forwarded by
`hive_c0re::meta` as both an env var and an agent option: the harness
reads the variable at runtime, its unit is built from the option.
The agent end: `nix/agent-modules/queue.nix` declares that option pair
and, when set, has the harness unit inherit the two credentials by name.
Bare-id `LoadCredential=` is the terse form documented for inheriting
what the service manager received, and is non-fatal when the credential
is absent — which a hive whose publisher has not run yet needs.
No `HIVE_AGENT_OIDC_CA_FILE`: the meta flake already embeds the hive CA
and the swarm root into each container's trust store at build time, and
reqwest's rustls backend verifies against it.
Refs #3805
Ruled: swarm-bao-queue-agent.service must run before hive-c0re.service
and be wanted (not required) by it, so no agent container renders
ahead of the reader's attempt at its credential. An unreachable store
delays hive-c0re's start by the reader's own start-limit window rather
than failing it outright.
Refs #4314
The swarm queue's rendered settings never named max_payload, so it ran
on nats-server's upstream default of 1 MiB. That default is about to
be too small: the broker is going to carry agent terminal rows
(whole TermMsg bodies) published as complete messages rather than
split, and a publish over the limit does not truncate — the server
answers -ERR 'Maximum Payload Violation' and closes the connection,
dropping the row.
Set max_payload = 8388608 explicitly in the settings merge, with a
comment on what it bounds and what bounds it (max_pending, which
nats-server refuses to start past). Add a module-eval case that reads
the rendered container config so a future edit that drops or shadows
the key fails eval instead of surfacing as a dropped row in
production.
Refs #3805
The publisher on the authelia host has been writing
`secret/swarm/hives/<hive>/queue/agent` — the OIDC client secret agent
containers present to the swarm queue, plus the client id it belongs to —
and nothing read it. This is the reader: a oneshot `swarm-bao-queue-agent`
that logs in with the host's certificate and lands the two fields as two
files under `deploy.hive-controller.queue.agentCredentialDir`, the secret
`0600` and the client id `0644`.
Two files rather than one because that is the consumer's shape:
`swarm_queue_client::QueueConfig::from_env` takes the secret as a path and
the client id as a value, so the split here is what keeps the next slice
from parsing anything.
Same shape as the store's first reader, `glue-matrix-bao-token.nix` — a
cert login that fails loudly under `Restart=on-failure` because every state
it fails on is one a retry fixes, then reads that degrade quietly because no
retry turns "no value there" into a value. Unlike the matrix token there is
no local fallback and none is possible, so absent files mean this hive's
agents do not connect, which is the ordinary state of a swarm before the
publisher has run.
Nothing consumes the files yet and this unit is ordered `Before=` nothing.
The next slice bind-mounts them into agent containers through hive-c0re and
adds the ordering edge along with them.
Refs #3805
`swarm-bao-matrix-token` reads `secret/swarm/matrix/registration-token`
and is refused with `Code: 403 — permission denied`, measured on this
host at 16:17:49Z after a successful cert login.
path.rs makes every swarm path `swarm/<kind>/<name>/…` where Kind is a
closed set of four: agents, hives, services, controller. `matrix` sits
where a kind belongs, so policy.rs's read document — which emits exactly
`swarm/agents/*` and `swarm/hives/<hive>/*` — cannot cover it. The
module's own doc predicted this: "a misspelled kind is a 403 at provision
time rather than anything a compiler sees".
Moves the token to `swarm/hives/<hive>/matrix/registration-token`, built
through principal_prefix(Kind::Hive, …) like its per-hive sibling
queue::agent_client_path. The policy is untouched: render already grants
that prefix. mara chose this over widening the namespace.
The nix reader interpolates hyperhiveCfg.hiveName, with the no-fallback
reasoning glue-bao-tls.nix already gives at its own use of it.
path.rs's MOUNT doc justified itself by citing the old literal, which
this commit deletes; rewritten to cite the nix reader instead.
Scope: this makes the read reachable, not the value present. Nothing
writes that path yet, and a 403 says nothing about presence — the two
are separate findings and only the first is fixed here. No migration:
nothing ever wrote the old path and no read ever succeeded.
Gate: cargo fmt 0, cargo test -p swarm-secret-client 0 — 32 passed
against a 29-passed baseline with the change stashed, so the three new
tests are accounted for rather than assumed.
Refs #4308
Every agent container reports the hostname `nixos`, so every log line it
ships carries that as its `_HOSTNAME`. Measured: host `muede-lpt2`,
`hive-matrix` (declared as `containers.hive-matrix`) `hive-matrix`, and
`h-atlas` `nixos`.
nixpkgs sets the hostname in the merge function of the
`containers.<name>.config` option (nixos-containers.nix:524), so it reaches
a guest evaluated through that option and nothing else. Agent containers are
`nixos-container create --flake meta#<name>` — an independent `nixosSystem`
off the meta flake, which never evaluates that submodule.
nspawn also names a container's hostname after the machine by default; that
is ruled out as the source here, because `h-atlas`'s machine name is
`h-atlas` and it reports `nixos`.
The machine name rather than the logical one: `stats/otel_metrics.rs:345`
already labels metrics `container.name = "h-<name>"`, so the logical name
would make logs say `atlas` while metrics say `h-atlas` — a prefix transform
on every join between the two signals. Declarative containers and nspawn
both use the machine name too, so this is one rule with no exception for
agents.
The emission sits next to `hyperhive.user.name = name;`, which already
derives the container's unix user from the agent name; the hostname was the
one identity attr nobody wired.
Also drops the prose in swarm-otel.nix and module-eval.nix that explained
`_MACHINE_ID` by "every container is `nixos`" — that motivating example is
what this commit removes, and the argument for `_MACHINE_ID` never depended
on it.
Checked before editing: nothing in the tree assumes the hostname is
"nixos" (0 hits across *.rs and *.nix), and nothing reads the hostname at
runtime.
Gate: cargo fmt 0, clippy -D warnings 0, cargo test -p hive-c0re meta::
0 (23 run, 22 passed, 1 ignored), nix fmt 0 changed.
Refs #4304
The unit treats every failure as permanent: it prints why and `exit 0`s,
with no `Restart=`, so one bad moment costs the whole boot. Two of its
failure modes deserve that and one does not.
`bao login` fails when the store is unreachable, sealed, or has not been
given this host's cert-auth role yet. All three are transient. Measured
on this morning's rebuild:
11:24:05 Started Container 'swarm-bao'
11:24:06 could not log in to swarm-bao with this host's certificate
11:24:07 Success! Data written to: auth/cert/certs/swarm-secret-publisher
It lost by one second, and stayed degraded for the boot. The publisher
next to it hit the same race and recovered on its first retry, because it
has `Restart=on-failure`.
`bao kv get` returning nothing is the opposite: the store answered, and
holds no token at that path. A retry cannot improve it, so that branch
keeps `exit 0` and the local token.
The bound is sized for this race, not for an unseal.
`swarm-bao-controller-policy` waits 2880 x 30s because a shamir unseal is
a human action and that unit blocks nothing. This one is `Before=` the
homeserver's container, so every retry is time the homeserver may spend
waiting -- 4 x 15s covers a container-start race with margin, and a store
still sealed after it degrades exactly as it does today.
`StartLimit*` are `[Unit]` settings and are ignored under `[Service]`, so
they are top-level attrs here. The module-eval case asserts the window
outlasts `RestartSec x burst`, since a burst that cannot be reached is a
unit that looks like it retries and does not.
Refs #4303
`_stream_fields=_HOSTNAME,_SYSTEMD_UNIT` was chosen to give "one stream
per unit per machine", which is what the comment above it claims. It does
not: no container sets `networking.hostName`, so `_HOSTNAME` is the NixOS
default `nixos` in every one of them, and every container's stream for a
given unit name merges into a single series.
Measured from inside an agent container:
journalctl -n1 -o json -> "_MACHINE_ID":"5d1427ea…", "_HOSTNAME":"nixos"
cat /etc/machine-id -> 5d1427ea…
git grep -i hostname -- nix/ -> 20 hits, all public vhost names,
zero `networking.hostName =`
`_MACHINE_ID` is written by journald per machine and is already on every
entry, so adding it to the stream key makes the partition what its own
comment says it is. That merge is also why the host collector and the
containerised one were one series: both log to
`opentelemetry-collector.service`, and the field meant to separate them
was a constant.
The second comment touched here asserted a reader "can still tell the
origins apart" from `_HOSTNAME`/`_SYSTEMD_UNIT`/`_MACHINE_ID`. True only
of the last one; it now says so.
Not a full fix for attribution: `_MACHINE_ID` is opaque hex and nothing
maps it to a container name. Naming is the other half and is a separate
change.
Refs #4304
`BAO_CLIENT_CERT` decides which certificate the TLS handshake presents. It is
not an identity: cert auth is a login, and a `bao kv` call without a token asks
its token helper for one instead — a `sh` neither unit carries on `path`.
Measured on this host, from `swarm-bao-matrix-token.service`:
swarm-bao did not return secret/swarm/matrix/registration-token
failed to get token helper: error expanding config path "":
exec: "sh": executable file not found in $PATH
So the first credential meant to travel through the store never has, and the
publisher added last week would not have either.
`-token-only` rather than a plain login: storing is the default, and it stores
through that same helper, so the obvious form reproduces the failure one line
further down. It is `-field=token -no-store`, which keeps the token on stdout
and out of the filesystem.
The two units degrade differently on purpose, and that is preserved. The matrix
fetch is `Wants=`-only and must not hold up the homeserver, so a refused login
reports why and keeps the token already in place. The publisher is
`Restart=on-failure`, where a store that cannot authenticate this host is worth
retrying and "published 0" would read as an ordinary quiet day.
`swarm-bao.nix` is untouched: it authenticates with the bootstrap token from a
file, which is a real identity and not a cert exchange. Its shape is where the
export idiom here comes from.
Five cases: the leaf and the pairing that points at it, that the push
hands bao a path rather than the secret, a path per hive in the roster,
that a publisher holding an identity renders on a host with no store,
and the control that renders none without an identity.
They fail on this branch, and that is the finding. The suite was green
at 100 cases with the publisher already committed, because no fixture
enabled both authelia and a store identity — so the module's `config`
never activated and its script was never evaluated. An imported module
whose config never fires is as unmeasured as an unimported one.
What they surface: `swarm.authelia.agentClientSuffix` does not exist on
main. The agent principal these secrets belong to is minted by the PR
for the agent queue principal, whose branch adds both the option and the
`agentClients` list authelia mints from. Delivery of a secret nothing
mints cannot evaluate, let alone run — so that PR lands first and this
one rebases onto it.
The argv case strips comments before matching, which it earned: a
`script` renders its own comments into the text, and this unit's
comments name the hazard verbatim so the next editor does not
reintroduce it. Matching the raw text read that warning and failed —
a check the artifact defeats by describing the thing it is checked for.
Refs #3853
A sibling unit rather than more script in swarm-bao-controller-policy, because
that unit's name is an operator-facing string: docs/getting-started/setup.md
tells a reader to run `systemctl status swarm-bao-controller-policy`. Widening
it to two principals makes the name wrong; renaming it makes the instruction
wrong.
`after` and not `requires`. The controller's unit creates the KV and cert-auth
mounts this one writes into, so the ordering is real — but a failed oneshot
still counts as finished, so `requires` would neither wait for its success nor
re-run this unit when the sibling's own retry eventually lands. Ordering plus
this unit's `Restart=on-failure` is what converges.
Four module-eval cases, because the unit arrived with every claim about it in
prose and the suite still reporting the same count: the grant is write-only and
reaches the hive prefix alone (pinned as the whole capability list, since an
added capability is what a presence check misses, with negative arms for the
agent prefix, the bare swarm prefix and the policy path); it is ordered after
the unit that creates the mounts; it renders on the host; and the control, that
it does not render inside the store's container.
Refs #3853
The unit that will copy authelia's minted OIDC client secrets into the store
needs an identity of its own. Not the controller's: that grant includes
rewriting every hive's policy and login role, which a unit whose whole job is
copying one file has no business holding.
The subject joins `certAuthCns`, so it is unrepresentable as a hive name for
the same reason the controller's is — cert auth trusts the CA, and a hive's
own leaf carries its name as the CN.
The module-eval case collides with the SECOND list element and leaves the
controller's subject at its default. A list with one consulted element and one
dead one is indistinguishable from the first element's case, so without this
the addition could be inert and nothing would say so.
Refs #3853
A collector's `refused` / `failed` / queue-depth counters are the only
signal that says telemetry is being dropped, and nothing read them at
any tier — so a collector losing records looked exactly like a quiet
system.
The hive tier could not be scraped without first naming its port. 8888
is the collector's built-in default and appeared in no config, which is
also why nothing comparing configured ports could see it clash with a
co-located collector — swarm-otel.nix sidesteps 8888 by hand for that
reason, and says so. Declaring the port and binding it explicitly makes
the value comparable; wiring the scrape is then one entry per tier.
Extending the port-collision assertion to cover it is deliberately left
out: that belongs with the other port work, and coupling a collision
fix to a scraping fix makes both harder to review.
Gate: 101 module properties hold, was 95. The six cases pin the rendered
scrape job rather than the option; the metrics pipeline naming the
prometheus receiver, a path never emitted on any hive before this since
the hive tier's scrapeTargets was empty everywhere; the `readers`
spelling, with a control so a missing telemetry block cannot pass the
port check vacuously; the swarm tier's own entry; the two tiers not
claiming the same port; and the absence arm, a hive with no collector
declaring no target.
scrapeTargets' description said "Empty by default, and that is the
shipped case". This makes that false, so the paragraph moves with it.
The reader had one failure branch, and it could not fail. It named three
states — the store holds nothing, the store is sealed, the store is
unreachable — treated all three as success, and sent bao's stderr to
/dev/null, which is the only thing that tells them apart.
The degraded mode is right and is unchanged: a missing registration token
means new agent accounts cannot be provisioned and nothing else breaks,
so this still exits 0 and leaves the local token alone. What changes is
that the journal now carries the store's own message instead of a
sentence of ours asserting all three at once.
Deliberately not branching on that message. Distinguishing "no value
found" from "permission denied" programmatically would mean matching
prose from an external tool that I cannot reach a live instance of to
confirm, which is inventing a discriminator rather than reporting one.
Nothing asserted this script before — eleven module-eval cases cover the
unit's existence and its ordering, none its contents, which is how the
branch stayed. The case added here covers the property that was missing.
Gated: `96 module properties hold`, exit 0 (95 on the base commit, +1 =
the case added). `nix fmt` reported 0 changed over 723 files emitted in a
fresh worktree. Tracker-tag, comment-block and doc-pointer lints all exit
0 after staging.
The two hive-name guards lived in swarm-otel.nix, inside its
`config = lib.mkIf (… && deployCfg.swarm-otel.enable)`. A swarm running the
secret store and the controller but no collector therefore had no hive-name
check at all, while the names were still composed into OIDC client ids, bao
policies and cert-auth roles exactly the same way. They move to swarm.nix,
which declares `swarm.hives` and is unconditional. swarm-otel keeps the
assertion that its own entry is still in the shared list — that one is about
this module's stake in a file it no longer controls.
The equality guard also takes the store's cert-auth subjects now. Cert auth
trusts the CA, so `allowed_common_names` is the whole of what narrows a role
to one identity, and the same CA signs every hive's leaf with the hive's name
as its CN. A hive named after a role's subject presents a certificate that
role accepts, which for the controller is write access to every hive's
credentials and policies.
A list rather than the one string, because the next role added beside it
widens what a hive name must not collide with, and because the subject is an
option an operator sets — a literal deny entry covers the default and nothing
else.
Four module-eval cases, two of them controls. The fixture overrides the
subject to `ctl` on purpose: the default contains `swarm`, which the substring
guard catches whatever the new arm does, so a fixture using it could not tell
the two apart. The controls are that a legal roster trips neither guard, and
that all three fixtures really do have the collector disabled — without the
second, every case would pass while testing the arrangement they exist to
rule out.
The crate had a single path convention and it was per-agent:
`swarm/agents/<agent>/matrix/<account>`. The secrets still to move into the
store do not fit it — one belongs to a hive, one to a swarm service, one to
the controller itself — so each would have picked its own shape, and each
would have been a separate grant to get wrong.
mara ruled the scheme on the epic: `swarm/<kind>/<name>/<secret>`, over
`agents`, `hives`, `services` and `controller`. This lands it.
`Kind` is an enum rather than free strings for one reason: the store's grant
is written in nix and cannot be reached from Rust, so a misspelled kind is a
403 at provision time and not a compile error. `Kind::ALL` lets a test
enumerate the set instead of restating it, which is what makes adding a kind
a deliberate edit rather than an accidental grant.
Note `Kind` sits beside `checked_segment`'s existing `kind` argument, which
means something else entirely — the label of the name being validated. They
are not the same concept and should not be merged.
Nothing about the rendered policy changes. `policy::render` still grants read
on the agent kind alone; the other kinds are absent on purpose, because what a
hive may read of its own kind is a boundary question and not a consequence of
the namespace growing. The controller's write grant likewise stays scoped to
`agents/` — it widens when a path outside it gains a writer, not when the
kinds are declared.
Verified: `cargo test -p swarm-secret-client` 23 passed, 0 failed. The two
tests pinning the rendered strings (`the_document_grants_read_over_the_whole_agent_prefix`
and matrix's path assertion) still assert the same literals they did before,
which is what shows this is a faithful port rather than a reshape. `nix fmt`
710 emitted, 10 formatted, 0 changed; the three scripts/check-*.sh lints pass
with the change staged. No reference to the removed `path::AGENT_PREFIX`
survives in the crate or in nix — checked with a scoped pattern, because the
unqualified name also belongs to hive-host-sock's container prefix and greps
for it are answering a different question.
`auth.disable_login_form` was gated on `ssoLocal` — `grafana.enable &&
authelia.enable`, i.e. "both of them run on THIS host". With authelia
elsewhere in the swarm that is false, so a deployment that is very much
using SSO still rendered grafana with its local login form enabled, on a
vhost the gateway publishes, for a product that ships an `admin`/`admin`
account.
The reason that matters was already in the module, three lines up
("Grafana ships an `admin`/`admin` account, and this vhost is on the
public gateway") — it was just attached to a conditional. Whether a
password box sits on a public login page is not a per-host question.
Per mara on the docs PR for this: "grafana requires sso - no local
login". The OIDC block below stays gated on locality; making that follow
the same swarm-wide question is a larger change with its own tracking.
The module-eval suite already had the fixture this needed: `grafanaOldPath`
enables grafana and not authelia, which is exactly the shape the login
form stayed enabled in, so the regression case needs no new hive. 90 -> 91
properties.
Closes#4218.
`nix/host-modules/swarm-peers-removed.nix` exists to turn an "option does
not exist" error into a warning that says where the entries went. That
warning is its whole deliverable, and no fixture set `swarm.peers` — the
string appeared 0 times in module-eval.nix, so the shim was never
evaluated by anything.
It differs from its ten siblings in what a broken shim looks like. The
others re-route a value, so a failure shows up as a wrong rendered
config. This one renders prose nothing reads back, so a `mkIf` that
stops matching or a rename of the `swarm.hives` it points at fails
silently, and lands on the one operator who needed it.
The check already carries an old-path fixture for ten migrated
namespaces (wireguard, forge, matrix, nats, authelia, controller, ui,
stores, grafana, statusPublish). `swarm.peers` was the eleventh and the
only uncovered one.
Two peers, only one carrying `caCert`, because the module emits a second
warning filtered on exactly that attribute — with a single peer the
filtered list and `attrNames` are the same list, so a `withCaCert` that
had collapsed into `attrNames` would still read green. The third case is
the control: a hive that never set `peers` must get neither warning,
without which the other two pass on any config whose warning list
happens to carry the string.
Closes#4188.
`deploy.allSwarmServices` derives nine service enables and appeared
nowhere in the check, so the tier a service sits on was prose only. The
tenth, the swarm controller, already had this exact pair of cases — it
rides `singleHostSwarm` instead, and swarm-ui follows the controller.
Four cases: the switch turns its nine on, a hive that does not host them
runs none, an operator placing one elsewhere still wins over the
`mkDefault`, and hosting the shared services does not make a hive the
swarm's control plane.
The roster is counted before it is read: `lib.all` over an empty set
holds vacuously, so a roster that lost a member would otherwise turn the
case green by measuring nothing.
Closes#4186
Nothing in this suite evaluated nix/agent-modules/ at all: every fixture
was a host, so a typo in a rendered container config surfaced on a real
deploy and nowhere else. This adds an `agent` constructor beside `hive`,
off the same `nixosModules.agent-base` the meta flake hands a container.
It also adds the suite's first two-hive fixture. Every existing one
declares `swarm.hives.h1` alone, so a per-hive arm written against those
passes on a hardcoded literal — which is exactly what the new per-hive
logs pipeline needed covering.
Eight cases, each paired with the control that makes it mean something:
the absence arms with a presence half, the per-hive arm with a roster
length check, because `lib.all` over an empty roster holds vacuously.
Each was then shown to fail. Eight mutations across the three files —
`directory` back to the runtime default, a pipeline naming no receiver,
an exporter aimed at a loopback literal, an exporter name that stops
reading `protocol`, a missing hive-tier logs pipeline, the metrics-only
processor inside it, a per-hive pipeline hardcoded to one hive, and logs
pipelines exporting to the metrics store — all caught, none survived,
none skipped, each run's baseline green.
Part of #3940.
The bootstrap unit writes a policy granting `secret/data/swarm/agents/*` and
nothing creates that engine. A fresh OpenBao has no `secret/` — only a dev-mode
one does — so `swarm-controller`'s first credential write answers `no handler
for route "secret/data/swarm/agents/<agent>/matrix/<name>". route entry not
found.` Measured on the live host at 21:27:27Z; #4171.
`git grep` for `secrets enable`, `kv-v2`, `kv_v2` and `sys/mounts` returned zero
across the whole tree. Control, so the zero means something: `auth enable` in
this same file returns 2 — the same defect was already found and fixed once, for
the cert auth mount, with a comment that states the principle. This is the other
half of it.
The mount name is now bound once and interpolated into both the policy text and
the new step, because a grant and a mount that disagree is exactly the failure
being fixed.
Placed outside the client-CA block: the controller writes *through* this mount
regardless of whether anything can log in by certificate. `module-eval` asserts
that, since one indentation level decides it.
Grants, measured against a real openbao 2.6.2 rather than derived:
`-output-policy` asks for `sys/mounts/secret` create+update, and a token holding
exactly `sys/mounts` read + `sys/mounts/<path>` create/update enabled the engine
— **no `sudo`**, unlike `sys/auth/cert`. Negative control: the same token on an
ungranted path got 403, so the grant is what made it work. `setup.md`'s
documented policy gains those two.
Also from that session, each deciding how this is written: re-enabling an
existing path errors (exit 2), so this asks first like the auth mount does;
`secrets list -format=json` keys look like `"secret/"`, so the `case` idiom
ports over; and `kv put -mount=<p>` reports `<p>/data/...`, confirming v2 — the
prefix the policy grants and the client writes.
setup.md also drops a check that cannot work: it told the operator to confirm
with `bao read auth/cert/…`, which 403s because the host wrapper carries no
token. `systemctl status swarm-bao-controller-policy` needs no credential and
names the three success lines. The first-attempt-after-rebuild race is now
written down too — the store is still coming up, and the 30s retry is what
lands.
Refs #4171.
`swarm-bao-controller-policy` creates the `swarm-controller` policy and
cert-auth role — the credential every hive logs in with. It has never
succeeded on any deployment, and the reason is where it ran.
Inside the container it had neither of the two things the store demands.
Its `BAO_ADDR` was the public DNS name, which from that netns resolves to
the hive bridge: `dial tcp 10.42.0.1:8200: connect: connection refused`.
And every API listener carries `tls_require_and_verify_client_cert`, while
`tlsDir` holds the server's leaf and the CA that signs clients — no client
identity at all. Fixing only the address moves the failure one hop.
The comment above the unit asserted the opposite — that in there the store
is "reachable without a client certificate at all, which is the point". The
listener config decides that, and says otherwise. That belief is what put
the unit in the container, so it goes with it.
On the host all four coordinates already exist: `baoCli` carries the
address, the CA, the certificate and the key, so the unit needs no
`environment` block at all. `bootstrapTokenFile` was always a host path —
the container only saw it through a bind mount. Nothing new crosses the
boundary; the mount gets no wider.
The retry bound is resized with it. 10 attempts at 30s is five minutes,
and under `seal = "shamir"` an operator unseals by hand, so it would give
up before a human arrived — permanently, because `start-limit-hit` does
not self-heal. That is the same silent no-bootstrap this issue is about.
2880 × 30s covers a day, inside a 25h window.
module-eval follows the unit to the host and gains an arm asserting it is
NOT rendered inside the container: the move is the fix, so the side it
landed on is worth pinning.
