Four units read one path each out of the store, and all four logged in
holding `deploy.bao.clientCertFile` — the hive's own leaf. Bao identifies
a principal by the subject of the certificate it presents, so four
readers behind one certificate were ONE principal, and the only grant
expressible was the union of what the four need: read on
`swarm/agents/*`, `swarm/hives/<hive>/*` and `swarm/services/*`. The unit
fetching Grafana's OIDC client secret could fetch every agent credential
in the swarm; the one fetching this hive's matrix token could fetch
Grafana's. Least privilege was not misconfigured here, it was
unrepresentable.
Each now holds a leaf, a cert-auth role and a policy of its own, and each
policy is the single `secret/data/…` path that unit's own script names —
spelled to the leaf, not to a prefix, the way matrix-ctl's already is.
Following the four exemplars in-tree rather than building a mechanism:
`signLeaf` mints the leaves, `swarm-bao.nix` writes the roles from the
bootstrap token, the consumers name their own pair.
Two of the four are written PER HIVE and two are not, which is the shape
of the paths rather than a preference. A matrix appservice token and a
queue credential live under `swarm/hives/<name>/` and every hive runs a
reader for its own, so one role for all of them would have to be granted
`hives/*` — letting one hive read another's, a reach no hive has today.
An OIDC client secret lives under `swarm/services/<client-id>/` and a
swarm registers each exactly once, so one role each is enough. The
per-hive subjects are `<prefix>-<hive>` and swarm.nix reserves every
composed spelling as a hive name, so a hive cannot be named into another
hive's role.
The shared leaf stays: hive-c0re still passes it into its container, the
`bao` CLI wrapper still defaults to it, and the three
`glue-*-bao-identity.nix` files derive the PKI directory from it.
module-eval-bao-grants gains a negative arm per principal — each pins the
three stanzas the hive's leaf carried and the two wildcards a later
widening would reach for, so a policy that grows fails here rather than
in a store. Plus the consuming side: repointing a unit back at the hive's
leaf would evaluate, deploy and log in, and silently restore the union.
A hive that reads a store on another machine now places one leaf per
principal instead of one shared by four. That cost is the point, and
docs/swarm/secrets.md lists the pairs.
`services.opentelemetry-collector` already defines
`serviceConfig.Restart = "always"` at the normal priority. The forwarder
added in #4537 defined `"on-failure"` beside it, and two definitions at
one priority are a conflict the module system refuses to resolve — so
`containers.swarm-bao` stopped evaluating at all (#4615).
Drop our definition rather than force a value over it, which is what the
sibling collector in swarm-otel.nix already does. The resolved value is
`"always"`, which is the one we want here: a forwarder that exits for any
reason, clean or not, has stopped shipping the store's journal.
`RestartSec` stays — it conflicts with nothing and is what keeps the
restarts during first-boot secret delivery under the default start-rate
limit.
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.
What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.
Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.
The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.
Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.
The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
Both earlier versions asked the wrong host. `hyperhive.otel.enable` asked
whether this host runs a HIVE collector; `deploy.swarm-otel.enable` asked
whether this host runs the SWARM one. Neither answers the question the
forwarder actually has — "is there a collector to forward to" — and that
question cannot be false: a swarm always runs at least one instance of
every swarm-level service. So the forwarder renders under the condition
already enclosing it, that the store is deployed here, and nothing else.
`scrapeHere` deliberately keeps its `deploy.*` gate one line up. It is a
loopback metrics listener, which genuinely only works where the scraper
is — the two are different tiers, and the name says so.
The module-eval case that pins it is the split topology: the swarm
collector on another host, nothing local naming it, and the forwarder
still enabled and still addressed at `swarm.otel.domain`'s route. Both
removed gates render nothing in that fixture, which the co-located ones
they shipped with could not show.
The forwarder pointed at the hive bridge address and was gated on the
hive's `otel.enable`, so it existed only where a hive collector stood
beside it. It now exports to `swarm.otel.domain` — the gateway-served
name that resolves locally when co-located and over the network
otherwise — on the swarm tier's own producer route, and is gated on
`deploy.swarm-otel.enable` like its sibling `scrapeHere`.
Refs #4526
Every container is supposed to run a collector that passes its logs and
metrics to the next hop. swarm-bao did not: its journal reached the store
only because `--link-journal=host` puts it in the host tree, where the
swarm collector — a different container — reads it through a unit
allowlist. That is the topology being retired, and in this deployment it
delivers nothing: no `_SYSTEMD_UNIT` value in the seven-day store mentions
openbao at all.
So the store's container now runs its own journal forwarder, copied from
an agent container's (nix/agent-modules/otel.nix): the whole journal, no
unit allowlist, pushed to the same first hop every agent on the host
already exports to. A local collector reads the local journal, so there is
nothing for a list of unit names to disagree with.
The `swarm.otel.journaldUnits` entry and `--link-journal=host` both stay.
Every sibling swarm container still rides the shared collector, and they
come out once each of them has a forwarder of its own.
Closes#4526
A swarm runs one homeserver and every hive on it logged in as the same
`@hive:` localpart, holding the same access token out of one swarm-wide
store path. That is one matrix identity for N hives: the homeserver
cannot attribute an action to the hive that took it, and revoking one
hive's standing revokes every hive's.
Three changes, and the third is the one that makes the other two real:
- **The localpart carries the hive's name** (`hive-<hive>`), derived in
one place, `swarm_secret_client::matrix::hive_localpart`.
`hive-matrix.nix` renders the same string as the appservice
registration's `sender_localpart`, so the shared account stops being
created rather than merely stops being used.
- **The store path is templated by hive**, not a constant. The
"a swarm runs one homeserver, so this is a constant rather than a
parameter" rationale went with it; it stopped holding the moment two
hives shared the homeserver it describes.
- **The path moved out from under the grant every hive has.** It sat at
`swarm/services/matrix/sender-token`, inside the
`secret/data/swarm/services/*` read stanza `policy::render` gives every
hive. It now sits under that hive's own stanza,
`secret/data/swarm/hives/<hive>/*`, which interpolates the reader's
name — so a hive reads its own token and is refused another's. The
policy renderer itself is unchanged: narrowing the `services/*` grant
would break the OIDC-secret read it exists for, and moving the
credential is what this needed instead. A policy test walks the
rendered stanzas and asserts none of hive alpha's covers hive beta's
sender token, so a later stanza that widened it fails here.
`swarm-matrix-ctl` takes a new required `MATRIX_MINT_HIVE` and writes
that hive's path; its store grant in `swarm-bao.nix` follows, scoped to
one hive's leaf via the new `deploy.bao.matrixCtlHiveName` (defaulting to
this host's `hiveName`) rather than a `hives/*` wildcard, which would
hand the matrix container every hive's token back.
Migration: no outage at deploy. `ensure_hive_user` short-circuits on the
local token file, so a hive keeps running on what it has; with no such
file it reads the new per-hive path, finds nothing, and falls through to
the existing register-or-appservice-login ladder against its own
localpart — which needs only the per-hive `as_token` on local disk. The
old shared object is read by nothing afterwards. Rooms do not follow the
identity, and that is the one operator step; both ways out are written
into `docs/integrations/matrix.md`.
No admin standing is granted to the per-hive accounts: `admin_execute`
stays empty and the assertion pinning it is untouched.
Renames `swarm-matrix-minter` and reshapes it around subcommands. Minting
is now `swarm-matrix-ctl mint`.
Running rust inside `containers.hive-matrix` is not free: it needs its own
store identity, its own cert role and its own bind mounts, and every one of
those is per-*container*, not per-task. A second single-purpose crate would
have had to duplicate that plumbing to add one action, so the next thing
that has to run in there should be a verb here rather than a new crate.
The old name guaranteed the opposite.
`main.rs` is clap dispatch; the minting logic moves to `mint.rs` unchanged.
A bare invocation is refused: `mint` writes a credential, so "no verb"
defaulting to it would make a typo in the unit mint rather than fail.
The environment prefix moves with it, `MATRIX_MINTER_*` → `MATRIX_MINT_*`.
Scoped to the verb and not to the binary, because a binary-scoped prefix is
one the next verb has to share or widen, and a widened one never narrows
again. A test asserts every variable carries the verb's prefix.
The principal renames too. The cert role, bao policy, granting unit, leaf
filename and `certAuthCns` entry all have to spell one string the same way,
so leaving them as `swarm-matrix-minter` would have rebuilt the naming
split this branch exists to remove. Renaming the nix options alongside is
free here: every one of them is introduced by this PR and has never been
released, so no operator config names them yet.
`ExecStart` now names the verb, which is a contract between a nix string
and a clap enum that fails at deploy time with no local signal. Both ends
assert it: `mint_is_spelled_the_way_the_unit_invokes_it` in the crate, and
a new module-eval arm reading the rendered `ExecStart`.
docs/getting-started/setup.md drops the sender token from its "live on the
host" list: setup does not touch this credential, so a setup guide has no
reason to name it.
The store path and every identifier around it called this an admin
token. It is not one: of ~15 hive-c0re call sites only two need
homeserver admin, and the homeserver no longer promotes the account at
boot, so the name overstated both what the credential is and what it may
do.
Renaming it to the account was not enough either. "The `@hive:` token"
reads as the token of a hive user, and no such user is provisioned —
`@hive:<server_name>` is the appservice registration's own
`sender_localpart`, an account the homeserver creates for itself when it
loads the registration.
So it is the **sender token**: the matrix appservice sender account's
access token, at `swarm/services/matrix/sender-token`. The name says
what it authenticates as rather than what it may do, which is the part
that was wrong.
The path has one constructor, and the bao grant, the grant assertion and
three unit tests pin its literal independently — so a half-finished
rename fails a check rather than leaving the minter and its readers
disagreeing at runtime. `tracing` messages are renamed with the code, so
the journal reads the way the source does.
The host-side file keeps its name (`matrix/access-token`): it carried no
admin framing, and renaming it would orphan the file on every deployed
hive for nothing.
`docs/tools/hivectl-cli.md` is regenerated from the clap tree.
A swarm runs one homeserver and a homeserver has one appservice sender
account, so "mint it once" is a property of the thing being minted
rather than something a lock has to enforce. That is what makes this
account the one to move first: no trigger route, no controller change
and no agent list — a boot-time oneshot beside tuwunel is the whole
mechanism.
`swarm-matrix-minter` runs inside `containers.hive-matrix`, which
already holds the appservice token: the rendered registration is bound
in read-only because that is how tuwunel is handed it. What the
container lacked was an identity of its own, so this adds one — a leaf
from the store's CA with a grant of exactly one path, not the hive's
leaf, which reads every secret in the store.
Both ends of the credential ship here. The minter reads the path it
publishes to before it touches the homeserver, and returning on a
non-empty read IS the "only once"; `hive-c0re`'s `ensure_hive_user`
reads the same path, authenticating with the hive name already in
`HYPERHIVE_HIVE_NAME`. The existing mint-then-`M_USER_IN_USE`-login
ladder stays as the fallback for a store that is empty, unconfigured or
unreachable, which is every swarm deployed before this — so nothing
needs backfilling and nothing breaks if the rest of the sequence never
lands.
The credential is not an admin credential, and is not named like one.
It is the access token of the appservice registration's own
`sender_localpart` — `@hive:<server_name>`, an account the homeserver
creates for itself when it loads the registration. The store path is
`swarm/services/matrix/sender-token`, the host path is
`matrix/access-token`, and the homeserver no longer runs an
`admin_execute` promotion for that account at boot. Everything the hive
provisions with it — the Space, the chat room, their hierarchy and join
rules, the invites — rides on being the creator of those rooms at power
level 100, not on homeserver admin; there is no Synapse admin API here
to need, tuwunel has none.
Two operations do need an admin *sender* and therefore stop working:
`hivectl matrix promote-user` and `hivectl matrix reset-password`, both
`!admin …` messages into `#admins:<server>`, plus the password-reset
recovery path that an agent with a lost password file falls back to.
They are swarm-level operations and are left failing loudly rather than
served by an over-privileged token every other call site would also
carry. The sweep's own admin-rights check and self-repair go with them:
an account that is deliberately not an admin has nothing to check.
`ephemeral = false` stays, and hive root can still read the container's
filesystem. Accepted: what this buys is identity separation — no hive
*process* holds or reads the appservice token — not physical isolation.
Refs #4345
`services.hyperhive.gateway.enable`, `gateway.dns.enable` and
`network.enable` replace the `hyperhive.enable` gate on all three. Each
defaults to false; the modules that need one assert it with `mkDefault
true` from inside the guard their own deployment already carries, and
`swarm-required-services.nix` — the module that owns what the
swarm-services toggle implies — asserts all three explicitly.
hive-c0re asserts all three unconditionally, so an ordinary hive keeps
getting them with no opt-in: it is the host's only knowledge that agent
containers exist.
The resolver moves to its own `hive-gateway/dns.nix` so it can be gated
without reindenting the nginx half of the module.
Reinstates `network.enable`, dropping its `mkRemovedOptionModule` shim.
A config still carrying `network.enable = false` from before the removal
now switches the bridge off instead of failing eval.
Also deletes a duplicate `centralToggleOff` fixture in nix/module-eval.nix.
Two sibling slices added it independently (c5f60fd5, ce3b3d94); the merge
was textually clean and left `main` failing to evaluate at all, so this
file could not be gated without removing one.
First slice of the move off the hand-rolled swarm services sub-CA: the
store gains the three objects the eventual minter needs, and nothing
else.
- a `pki` secrets mount, enabled ask-first off the same `bao secrets
list` snapshot the kv-v2 mount beside it uses, so a rebuild that finds
it mounted does nothing;
- a `swarm-services` role on it, `allowed_domains` read straight out of
`swarm.serviceDomains` — the same swarm-tier list swarm-ca.nix
name-constrains its sub-CA to and hive-tls.nix carries as the leaf's
SANs — with subdomains, localhost, globs and IP SANs all off;
- a `swarm-services-issuer` policy granting `update` on
`pki/issue/swarm-services`.
Inert: the mount has no issuer generated into it, the role therefore
cannot issue, and no login role attaches the policy, so no token in the
swarm carries it. No consumer changes; swarm-ca.nix still mints the
services sub-CA exactly as before. Reversible with
`bao secrets disable pki`.
The cert-auth role that attaches the policy waits for the leaf carrying
its CN, which glue-bao-tls.nix mints — a later step.
Refs #4256
Adds approle to the auth methods enabled during swarm-bao bootstrap,
using the same idempotent check pattern as cert. No role or policy
is configured yet — those are handled in later slices.
Refs #4386
An agent container cannot dial the store's loopback listener: the bridge
to-loopback DROP rule is there precisely to stop that, and the store
authenticates every reader by client certificate, so the usual answer —
a gateway vhost — is the one shape that cannot work. A terminating proxy
strips the certificate and bao sees nginx as the client for every hive.
nginx's stream module does not terminate. `ssl_preread` reads the SNI off
the ClientHello and splices the rest of the connection through byte for
byte, so bao completes the handshake itself and authenticates the client
it actually has. That is the no-vhost rule kept, not bent.
The listener binds the bridge IP rather than every address, because bao
already holds `127.0.0.1:<port>` in the same netns and a wildcard bind
there is EADDRINUSE — nginx would fail to start, taking the gateway with
it. Nothing moves as a result: the name already resolves two ways, so a
host-side reader still goes straight to loopback and an agent goes
through the passthrough, both on one `BAO_ADDR`.
Renders only inside the store's own `deploy.bao.enable` region; a host
that runs no store grows no listener and opens no port.
Per-agent certificates and per-agent policy are separate work.
Refs #4386
Grafana's OIDC client secret only existed where authelia did. One
`ssoLocal` gate — `grafana.enable && authelia.enable` — decided the
client registration, the minted secret's delivery and the whole
`auth.generic_oauth` block, so a swarm whose authelia runs on another
host got Grafana with no SSO wiring at all. The local login form is
disabled unconditionally, so that is no way in.
Split the one gate into the two questions it was conflating:
- `ssoConfigured` — does this SWARM have an identity provider
(`swarm.authelia.url`, which is swarm-wide and whose own description
makes null mean "no SSO configured"). With a delivery route present
this is what emits Grafana's OIDC block.
- `ssoLocal` — is authelia on THIS host, now spelled as the forge and
matrix modules spell it. It decides only which unit delivers the
secret.
Where authelia is elsewhere, `swarm-bao-grafana-oidc.service` reads the
secret from the swarm secret store, shaped after
glue-queue-agent-credential.nix: cert login fails loudly because a retry
fixes every state it fails on, the read degrades quietly because no
retry turns "no value there" into a value, and nothing writes a
stand-in. The producer is the publisher that already runs on authelia's
host, which gains the swarm's service clients beside the per-hive ones
at `swarm/services/<id>/oidc/client` — with the write grant in
swarm-bao.nix and the hive read grant in `policy::render` to match.
Registration moved to glue-grafana-oidc-client.nix. It has to be
declared where authelia's config is rendered, and swarm-grafana.nix's
config block hangs off this host running Grafana.
Two judgement calls stated rather than buried: a hive's read policy now
grants the whole `services` prefix, because a service's path names the
service and nothing swarm-wide records which hive runs it (cost recorded
in docs/trust-boundary/security.md); and the client is registered on any
authelia host, because no swarm-wide "this swarm has a Grafana" fact
exists to gate it on.
Refs #4234
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A sibling unit rather than more script in swarm-bao-controller-policy, because
that unit's name is an operator-facing string: docs/getting-started/setup.md
tells a reader to run `systemctl status swarm-bao-controller-policy`. Widening
it to two principals makes the name wrong; renaming it makes the instruction
wrong.
`after` and not `requires`. The controller's unit creates the KV and cert-auth
mounts this one writes into, so the ordering is real — but a failed oneshot
still counts as finished, so `requires` would neither wait for its success nor
re-run this unit when the sibling's own retry eventually lands. Ordering plus
this unit's `Restart=on-failure` is what converges.
Four module-eval cases, because the unit arrived with every claim about it in
prose and the suite still reporting the same count: the grant is write-only and
reaches the hive prefix alone (pinned as the whole capability list, since an
added capability is what a presence check misses, with negative arms for the
agent prefix, the bare swarm prefix and the policy path); it is ordered after
the unit that creates the mounts; it renders on the host; and the control, that
it does not render inside the store's container.
Refs #3853
Write-only `create`/`update` on `secret/data/swarm/hives/*`, and nothing else.
It copies secrets in and never reads one back; a read capability would let a
file-copier recover every hive's credentials rather than merely replace them.
`hives/` and not `swarm/*` because this principal has no business with an
agent's or a service's credentials, and the hive prefix is the only one whose
paths it produces. `secret/data/` is KV v2's ACL prefix, inserted by the engine
rather than written by the caller — the same trap the controller's grant
documents one binding up.
Named outside `hive-*`: the controller may create policies under that prefix,
and a policy it can rewrite is not a constraint on it.
The unit that writes this lands next. Refs #3853
The unit that will copy authelia's minted OIDC client secrets into the store
needs an identity of its own. Not the controller's: that grant includes
rewriting every hive's policy and login role, which a unit whose whole job is
copying one file has no business holding.
The subject joins `certAuthCns`, so it is unrepresentable as a hive name for
the same reason the controller's is — cert auth trusts the CA, and a hive's
own leaf carries its name as the CN.
The module-eval case collides with the SECOND list element and leaves the
controller's subject at its default. A list with one consulted element and one
dead one is indistinguishable from the first element's case, so without this
the addition could be inert and nothing would say so.
Refs #3853
Both places that warned about a hive named after the controller's cert-auth
subject still told the reader it was unmitigated, and the option's description
recommended a migration — "change this to something outside the hive-name
grammar, at the cost of a role rename in any store that has already run the
granting unit" — that is no longer the answer and is not cheap.
swarm.nix now feeds the subject into the guard on `swarm.hives`, so a
colliding roster fails evaluation. Both paragraphs get SHORTER saying so: a
guarded hazard needs the consequence ("reserved as a hive name") rather than
the threat model, and the write-site keeps only the sentence the next person
needs — a role added beside this one must join that list.
Found by sweeping for the claim rather than for the symbol: the change that
made these stale touched swarm.nix, swarm-otel.nix and module-eval.nix, so no
diff-context or doc-comment rule covers a paragraph two files away. Grepping
the tree for "would satisfy" and for prose about a hive named after a subject
turned up exactly these two and nothing in docs/.
The bootstrap unit writes a policy granting `secret/data/swarm/agents/*` and
nothing creates that engine. A fresh OpenBao has no `secret/` — only a dev-mode
one does — so `swarm-controller`'s first credential write answers `no handler
for route "secret/data/swarm/agents/<agent>/matrix/<name>". route entry not
found.` Measured on the live host at 21:27:27Z; #4171.
`git grep` for `secrets enable`, `kv-v2`, `kv_v2` and `sys/mounts` returned zero
across the whole tree. Control, so the zero means something: `auth enable` in
this same file returns 2 — the same defect was already found and fixed once, for
the cert auth mount, with a comment that states the principle. This is the other
half of it.
The mount name is now bound once and interpolated into both the policy text and
the new step, because a grant and a mount that disagree is exactly the failure
being fixed.
Placed outside the client-CA block: the controller writes *through* this mount
regardless of whether anything can log in by certificate. `module-eval` asserts
that, since one indentation level decides it.
Grants, measured against a real openbao 2.6.2 rather than derived:
`-output-policy` asks for `sys/mounts/secret` create+update, and a token holding
exactly `sys/mounts` read + `sys/mounts/<path>` create/update enabled the engine
— **no `sudo`**, unlike `sys/auth/cert`. Negative control: the same token on an
ungranted path got 403, so the grant is what made it work. `setup.md`'s
documented policy gains those two.
Also from that session, each deciding how this is written: re-enabling an
existing path errors (exit 2), so this asks first like the auth mount does;
`secrets list -format=json` keys look like `"secret/"`, so the `case` idiom
ports over; and `kv put -mount=<p>` reports `<p>/data/...`, confirming v2 — the
prefix the policy grants and the client writes.
setup.md also drops a check that cannot work: it told the operator to confirm
with `bao read auth/cert/…`, which 403s because the host wrapper carries no
token. `systemctl status swarm-bao-controller-policy` needs no credential and
names the three success lines. The first-attempt-after-rebuild race is now
written down too — the store is still coming up, and the 30s retry is what
lands.
Refs #4171.
`swarm-bao-controller-policy` creates the `swarm-controller` policy and
cert-auth role — the credential every hive logs in with. It has never
succeeded on any deployment, and the reason is where it ran.
Inside the container it had neither of the two things the store demands.
Its `BAO_ADDR` was the public DNS name, which from that netns resolves to
the hive bridge: `dial tcp 10.42.0.1:8200: connect: connection refused`.
And every API listener carries `tls_require_and_verify_client_cert`, while
`tlsDir` holds the server's leaf and the CA that signs clients — no client
identity at all. Fixing only the address moves the failure one hop.
The comment above the unit asserted the opposite — that in there the store
is "reachable without a client certificate at all, which is the point". The
listener config decides that, and says otherwise. That belief is what put
the unit in the container, so it goes with it.
On the host all four coordinates already exist: `baoCli` carries the
address, the CA, the certificate and the key, so the unit needs no
`environment` block at all. `bootstrapTokenFile` was always a host path —
the container only saw it through a bind mount. Nothing new crosses the
boundary; the mount gets no wider.
The retry bound is resized with it. 10 attempts at 30s is five minutes,
and under `seal = "shamir"` an operator unseals by hand, so it would give
up before a human arrived — permanently, because `start-limit-hit` does
not self-heal. That is the same silent no-bootstrap this issue is about.
2880 × 30s covers a day, inside a 25h window.
module-eval follows the unit to the host and gains an arm asserting it is
NOT rendered inside the container: the move is the fix, so the side it
landed on is worth pinning.
`StartLimitBurst` sat in `serviceConfig`, so it rendered into `[Service]`,
where systemd silently ignores it — the unit retried every 30s forever.
Measured on a live store: the journal reports `restart counter is at 18`
against a burst of 10.
This repo already states the rule and pins it with a test:
`hive-priv/src/main.rs` renders its drop-in with `StartLimit*` under
`[Unit]` and says why — "systemd silently ignores them under `[Service]`,
so a bound that moved sections would look configured and do nothing".
That is exactly what happened here, in another module.
Moving the burst alone would not have fixed it. systemd's default window
is 10s while `RestartSec = 30`, so at most one restart falls inside it and
a burst of 10 is unreachable; the interval has to exceed `RestartSec` times
the burst. 600 matches the value hive-priv already uses.
Uses the NixOS service-level options rather than a hand-written
`unitConfig`: nixpkgs renders `startLimitBurst` / `startLimitIntervalSec`
into `unitConfig` itself (`nixos/lib/systemd-lib.nix`), and `hive-ci.nix`
already sets `startLimitIntervalSec` that way.
The module-eval case asserts placement where nixpkgs puts it, and that
`serviceConfig` does not carry it — so moving it back fails the build.
Reading a role out of the store took four round-trips of environment
guessing: the certificate has no IP SAN so loopback cannot verify, the
DNS name resolves to the bridge from inside the container, and the CA
the client needs lives on the host, which had no `bao` at all.
The wrapper carries the address, the server CA and — where the PKI glue
minted one — the reader's own leaf, so cert-auth login needs nothing
typed. `--set-default` throughout: an operator pointing BAO_ADDR
elsewhere still wins.
Only the wrapper reaches PATH. `wrapProgram` renames the real binary,
so there is no unwrapped `bao` to reach by accident, and the module-eval
case asserting the package's absence is what keeps a later "install it
too" from undoing that.
Closes#4124.
The controller's policy granted only the bootstrap paths -- hive cert-auth
roles and hive ACLs. #4113 then made it a secret WRITER, and nothing related
the grants to the paths the code writes, so every matrix token provision
answered 403. The two halves landed on different issues and neither looked
wrong on its own.
`secret/data/` is KV v2's ACL prefix and is absent from the path the code
passes, so matching `swarm-secret-client`'s spelling literally would have
granted nothing. Write-only: the controller mints these and never reads one
back, and a read capability would let it recover every agent's credentials
rather than only replace them.
The gate is the point. Two module-eval arms -- the grant exists and is not
a broader wildcard, and its capability list is pinned whole, because an
ADDED capability is what a presence check misses -- plus a test in path.rs
pinning MOUNT/AGENT_PREFIX and naming the nix file, since renaming either
constant is a silent 403 rather than a compile error.
setup.md carried two warnings this makes false: that nothing in the tree had
ever authenticated to the store, and that no deployment shape mints a leaf
whose CN reads swarm-controller. glue-bao-tls.nix has minted one since #3726
item 1.
Two halves of one interface that currently exists in one file only.
The CN was a `let` binding in swarm-bao.nix. Whatever mints the
controller's leaf has to spell it identically, and that lives in another
file, so a literal in one place is an interface with no name. It becomes
`deploy.bao.controllerCommonName` -- under bao because it is a property
of the role this module writes, not a credential belonging to the
controller.
The certificate gets `deploy.swarm-controller.baoClientCertFile` /
`baoClientKeyFile` rather than reusing `deploy.bao.clientCertFile`. That
one means "this host as a reader" and carries the hive's name, while the
controller's policy lets it create roles for every hive; one certificate
serving both would hand that power to whatever else reads the store
here.
Both default to null. The glue that mints the leaf sets them with
`mkDefault`, the same way glue-bao-tls.nix already supplies the reader's
paths -- so the controller module names no path of bao's, and deleting
the glue leaves a controller that takes operator-provided ones.
Nothing reads the new options yet; the minting half is the next commit.
`module-eval`'s derivation is unchanged, which is the honest result for
a change that adds options without altering any asserted value -- it
proves the tree still evaluates, not that anything behaves differently.
The policy the granting unit already writes grants paths under
`auth/cert/certs/*`, and nothing in the tree creates that mount. Every
certificate login therefore fails against a path that is not there --
the controller's own, and the per-hive ones it is meant to issue
against the same mount.
Same unit, same bootstrap token: check whether cert auth is mounted,
enable it if not, then write a role binding CN `swarm-controller` to
the `swarm-controller` policy.
Idempotency is a read rather than a tolerated error. `auth enable`
fails on an existing mount, and recognising that would tie a rebuild to
an error string no run of this store has ever produced, so the unit
asks `bao auth list` and mounts only on absence. That read is why the
token policy in setup.md gains `sys/auth`.
Each grant came from `bao <cmd> -output-policy`, which prints what a
command requires without running it -- the same way controllerPolicyText
was derived. Enabling an auth method needs `sudo` on `sys/auth/cert`,
which the documented token did not have.
Gated on `clientCaFile`, not on the token alone: `swarm-bao-certs`
installs `client-ca.pem` only under that condition, and a role's
`certificate=` has to name a real CA. The policy write, which needs no
CA, is unchanged in that case.
Nothing can present a certificate for this role yet -- the only client
leaf the tree mints carries CN = the hive's name -- and none of this has
been run against a live store. Both are stated in setup.md.
Cert auth answers a role, so nothing in the swarm can authenticate to
the store until some role exists. Creating the first one therefore
cannot itself use a certificate — the credential has to come from
outside that cycle, and an operator places it.
`deploy.bao.bootstrapTokenFile` names that token. A one-shot unit inside
the store's container reads it and writes the `swarm-controller` policy;
`local-defaults.nix` supplies the path (never the file) on an all-local
deploy, since co-location makes only the location derivable. `bao
operator init` stays an operator action in every shape.
The unit runs in the container rather than on the host because writing
the first grant is a store-side operation: it reaches the store locally
and needs no client certificate. Gating it on the store being here is
therefore not the co-location assumption glue-matrix-bao-token.nix
warns about — a reader has to work from anywhere, the first write never
does.
Policy only, deliberately: a cert-auth role binds a certificate and the
controller has no bao identity yet — it holds no leaf and contains no
bao code at all. The two certificates that do exist are both wrong to
bind. `clientCertFile` is the store host's own reader leaf rather than
the controller's, and the CA that signed it also signs every other
reader's, so binding that would let any reader authenticate as the
controller. Whoever gives the controller an identity writes the role.
The grants are scoped to `hive-*`. Without the prefix the controller
could rewrite the policy that constrains it.
Two things the module-eval arms pin: the unit renders inside the
container with the token path in both its script and its
ConditionPathExists, and a host that names a token while running no
store gets neither the unit nor the host-side directory.
The capabilities were derived with `bao write -output-policy` rather
than written from memory, and the setup.md commands were run against a
real binary for shape before being written down.
Refs #3726.
The block header says `deploy.bao.*` is what the host RUNNING the store
decides and that none of it means anything on a host that only reads
secrets. Three options in that block are exactly what a read-only host must
set, and their own descriptions say so a screen further down — so reading
the options never corrects the header.
It reads as though an off-host reader were inexpressible. It is already
supported: glue-matrix-bao-token gates on the identity rather than on
`enable`, and module-eval's baoRemoteReader is that deployment.
Also drops a sentence describing what used to be undeclarable.
The swarm collector reads /var/log/journal and has never seen a single
container unit. mara's count-by-unit against VictoriaLogs returns four
units, all host-tier; openbao -- which runs inside the swarm-bao
container -- is absent.
Cause: nixos-containers.nix hardcodes `--link-journal=try-guest` for
every non-ephemeral container. With `guest`, the host's
/var/log/journal/<machine-id> is a SYMLINK into the container's transient
root; a reader in the host namespace cannot follow it, and it dangles as
soon as the container stops. `ls -la /var/log/journal/` on the host shows
one real directory and a pile of `-> /tmp/nspawn-root-*` links dating
back to May.
swarm-otel.nix asserted the opposite, and that assertion is why the
receiver's path was considered sufficient: it said the files "live here"
and are "bind-mounted into the guest rather than the other way round".
That describes `--link-journal=host`. The same sentence names the flag we
actually use. The flag was right and the behaviour it described was not,
so grepping for the flag confirmed the comment and taught nothing.
`containers.<name>.extraFlags` feeds EXTRA_NSPAWN_FLAGS, which the
invocation expands after the hardcoded flag, so `--link-journal=host`
wins. The comment now describes what the code does instead of the other
way round.
Two payoffs, and the smaller one is the one the issue is about:
container logs become collectable, and -- independently -- they become
durable at all, rather than dying with the container.
Ten identical edits because ten host-modules hand-roll their own
container block; that duplication is #3773, not something to invent an
abstraction for here.
NOT VERIFIED: that systemd-nspawn honours the last `--link-journal` of
two. Everything else here is read out of nixpkgs; that step is a claim
about its argument parsing which cannot be exercised without starting a
container. It is settled by deploying one and re-running the `ls`: the
machine-id entry becomes a real directory instead of a symlink.
Refs #3849
The store's scrape target rendered as `host:port` alone, so the collector
requested `/metrics` — a path openbao does not serve on any listener. Every
scrape since the target landed has 404'd, which reads as a dead exporter
rather than a wrong address, and no bao sample has ever reached the store.
`scrapeTargets` values may now carry a path and query. Both are optional and
omitted when absent, so every existing target renders the config it rendered
before. The query cannot ride in `metrics_path`: prometheus percent-encodes
the `?`, so it has to become `params`.
Measured against openbao 2.6.2 and otelcol-contrib 0.151.0, the pinned
versions: `/metrics` 404s ("this listener only serves the metrics endpoint"),
`/v1/sys/metrics` answers JSON, and only `?format=prometheus` — or an Accept
header the collector happens to send today — returns exposition text. The
query param is the half that does not depend on content negotiation.
`bao operator init` fails at the seal with CKR_MECHANISM_INVALID. The
pkcs11 seal asks for AES-GCM by default and the TPM does not offer it: a
TPM 2.0's symmetric modes are CBC/CFB/CTR/OFB/ECB, and openbao accepts
only AEAD mechanisms — AES-GCM or RSA-OAEP — so RSA-OAEP is the single
mechanism both sides implement.
Measured on the deployed token (`pkcs11-tool --list-mechanisms`, #3860):
no AES-GCM, and `RSA-PKCS-OAEP, keySize={1024,2048}, hw, encrypt,
decrypt` present. CBC is not a fallback — openbao's
`MechanismFromString` rejects `CKM_AES_CBC_PAD` as deprecated and its
encrypt path implements exactly the two AEAD mechanisms.
The key gets a new label so a store provisioned by the earlier module
keeps its unusable AES key without the two resolving to one label, and
the addkey step is now keyed on the label rather than on the store not
existing — otherwise an existing deployment never gains the RSA key.
#3880 bound /dev/tpmrm0 into the store's container, which was necessary and
not sufficient: a bind mount preserves the host's ownership, openbao runs as a
DynamicUser with an empty CapabilityBoundingSet (so no CAP_DAC_OVERRIDE), and
the host applies no tpm udev rule at all — measured, 0 hits across all 41 host
closures against 62 rule files as a control. The node therefore keeps the
kernel default and the seal cannot open it.
A name cannot fix this. NixOS allocates system groups at activation, per
machine, so `tss` — or any group declared on both sides — gets two different
ids, and the device node carries the number. mara picked pinning a gid with an
overridable default (deploy.bao.tpmGid).
The default sits above the range NixOS auto-assigns system groups from
(400-999, measured in update-users-groups.pl) and above the normal-user range,
and below systemd's DynamicUser range (61184-65519), so it collides with
nothing any of those allocate.
The module-eval case compares the two sides rather than checking each against
a literal: the property is that they AGREE, not what they agree on. Its
absence arm is a shamir store, which never opens a TPM and must not claim a
device node's group — without it, pinning unconditionally would look identical.
openbao runs as a DynamicUser, so the uid that has to open the token
store is allocated by the container's PID 1 and cannot be named by a
host unit — the store was created root-owned 0700 and the seal could
never read it. The provisioning unit moves inside the container and
hands the sqlite store over by group; the host keeps only the mkdir the
bind mount needs, create-only so a reboot does not re-impose 0700.
Two further layers blocked the same start, both measured while fixing
this one:
- DynamicUser implies ProtectSystem=strict (systemd.exec(5)), so the
bind mount was read-only to openbao however it was owned, and the
pkcs11 library opens its store read-write. ReadWritePaths= is
required and was absent.
- allowedDevices renders DeviceAllow= and nothing else, and nspawn
builds its own /dev as a fixed tmpfs and cannot create device nodes
— verified against a live container, whose /dev holds no host
devices at all. /dev/tpmrm0 was therefore absent inside swarm-bao,
not merely unpermitted. It is now bound in.
Whether openbao's dynamic uid may *open* that node is a third question:
the tss gid is dynamically allocated, so no name or number means the
same thing on both sides of the boundary. Filed separately rather than
guessed at here.
The two module-eval cases that asserted the unit on the host now assert
it in the container and absent from the host, and two new cases pin the
device bind and the write access — each was individually valid and
collectively required, which is the state no assertion catches. Both
new cases select with `or [ ]`: mutation-testing them showed that a bare
select aborts the run with a nix trace instead of failing the case by
name, which also hid the second failure behind the first.
The store declared no journald units and served no metrics: nothing in
`swarm-bao.nix` mentioned either, while every sibling swarm service
declares both.
Metrics get their own loopback listener rather than a flag on the API
one, and that follows from what a scraper can express rather than from
taste: `swarm.otel.scrapeTargets` is `host:port`, plaintext and with no
credential, while the API listener is TLS and demands a client
certificate once a client CA is set. `metrics_only` narrows the new
listener to the metrics path; `prometheus_retention_time` is what serves
the endpoint at all.
Measured against openbao 2.6.2 before writing any of it: the metrics
path answers 200 on such a listener *while the node is sealed and
uninitialised*, 503 on the API listener, and 404 for a non-metrics path
on the metrics listener.
The listener exists only where a collector does — it is unauthenticated
by design for now, and an endpoint with no reader would be exposure
bought for nothing.
The port cannot be the API port + 1: openbao derives every listener's
cluster address as its own port plus one, so that number is already
taken. An assertion says so, since the failure is otherwise a race with
no log line.
Journald units are declared by the module that defines each unit, not
gathered here, matching the option's own rule — a name nothing defines
is silently ignored, so a central list would read as coverage on hives
that have neither glue module.
Refs #3849
openbao has never started on a hive that deploys it. With raft storage
and no `cluster_addr`, `bao server` exits 1 before serving anything:
cluster address must be set when using raft storage
Reproduced against openbao 2.6.2 with the module's own config shape:
the same JSON minus `cluster_addr` exits 1 with that line, and with it
the server comes up and configures every listener.
Both addresses are built from `swarm.bao.domain` rather than a bind
address — that is the URL a reader already dials (`BAO_ADDR` in
glue-matrix-bao-token) and the name the server certificate carries.
Cluster traffic sits one port up, upstream's convention.
The `unknown or unsupported field loopback` warning in the same startup
log is unrelated and cosmetic: openbao's unknown-field check does not
know about named listener blocks, but the parser honours `type` and
configures each one (measured, two named listeners, both served).
Noted in the module so the next reader does not chase it; the JSON
array form that avoids the warning does not typecheck against nixpkgs'
`settings.listener` (`attrsOf`).
module-eval gains the regression gate plus a control that the settings
it reads vary per deployment.
Refs #3860
Nothing resolved `bao.<swarm.domain>`. It is absent from
`swarm.serviceDomains` and, unlike every other swarm service, contributed
no `gateway.localNames` entry — so the name a reader dials answered only
if an operator had published an external record, which nothing asks them
to do.
Cross-hive traffic goes via the domain either way; only what it resolves
to varies, and for a multi-host swarm that is the operator's upstream DNS.
This covers the deployment with no upstream record to configure.
DNS only, and that asymmetry is deliberate: bao stays out of
`serviceDomains` and gets no vhost. Its leaf comes from its own CA, so a
services-CA entry would mint one nothing uses, and nginx terminating TLS
would strip the client certificate the store authenticates every hive
with. `swarm-nats.nix` documents the same exclusion for itself.
module-eval gains the presence and absence arms: claiming a name this
host does not serve would point every local reader at the wrong machine.
Refs #3862
`swarm-bao.nix` declared the store's half of the mTLS pair as options —
`serverCertFile`, `serverKeyFile`, `clientCaFile` — and left the reader's
half as a literal inside `glue-bao-tls.nix`, which only runs where
`deploy.bao.enable` is set. A hive that did not host the store therefore
could not read from it and could not be pointed at a certificate even
when one had been placed by hand.
Adds `clientCertFile`, `clientKeyFile` and `serverCaFile` beside their
three server siblings, `mkDefault`ed by the glue to the leaf it already
mints, and moves `glue-matrix-bao-token.nix` onto them. Its gate becomes
"this host holds an identity" rather than "the store is a neighbour",
and the unit ordering that names store-local units is now conditional --
`Requires=` on an absent unit fails the job.
`serverCaFile` is separate from `clientCaFile` on purpose: one is the
store choosing which readers to trust, the other a reader choosing which
store to trust. Self-signing collapses them to one file, which is a
property of that deployment and not of the pairing.
Closes#3855.
The state directory was bind-mounted from the host so a nixos-container
destroy could not take the swarm's secrets with it. No sibling service does
that -- swarm-grafana keeps its sqlite database inside the container on
ephemeral = false -- and the bind is what broke the store: upstream pairs
StateDirectory= with DynamicUser=, systemd relocates the state to
/var/lib/private/openbao, and that rename fails EBUSY on an active mount
point, so the unit died at STATE_DIRECTORY before bao ever ran.
The TLS material still has to cross the boundary, because a host unit writes
it and the container reads it, so it moves to its own small bind at
/var/lib/swarm-bao-tls rather than riding along in the state directory. That
directory is 0755 and read-only inside: the certificate and client CA are
public and are read straight off the mount.
The private key is not. install -m 0600 leaves it root-owned and the service
runs as a DynamicUser, so the bind-mounted file is unreadable to it -- which
the old layout hid, because StateDirectory chowned the whole tree on the way
past. LoadCredential is systemd's mechanism for precisely this: PID 1 opens
the source as root and re-exposes it inside the unit owned by the service's
own account.
`services.hyperhive.tls.{stateDir,caValidityDays,leafValidityDays}` sat at
the top of `services.hyperhive`, which is meant to be everything about
hyperhive rather than the settings of one hive. Where the hive CA lives,
how long it lasts and how long the leaves it signs last are decisions of
the host holding the key — `deploy.*`, by the same rule as the switches
that moved before them.
`hive-controller` is hive-c0re's new name (mara on the issue), so the
knobs hang off the daemon that owns the CA rather than off a bare `tls`
at the root. mkRenamedOptionModule entries carry existing configs.
⚠️ Unlike the two switch renames, these names are NOT unique, so this was
swept by ALIAS BINDING rather than by identifier: hive-tls.nix alone holds
two options spelled `stateDir` — its own `cfg.stateDir` and the swarm CA's
`swarmCaCfg.stateDir`, four sites that must not move. Nine files bind an
alias to this config; the rename followed those bindings.
Two sites were invisible to the obvious check, and an unanchored sweep for
`hyperhive\.tls\b` is what found them: the option declaration (`= {` after
the path, so no trailing `.` or `;`) and the alias convention documented in
a comment in lib/hive-ca-trust.nix.
Also renamed the `<tls.stateDir>` shorthand in four docs and two Rust doc
comments, anchored on its delimiters — the new path contains the old one
as a substring, so an unanchored replace would have doubled the prefix.
Three places said nothing supplies them — the module header, the
serverCertFile and clientCaFile descriptions, and the assertion message
an operator actually hits. All three were true when written and
falsified by glue-bao-tls.nix landing in the same PR, which mkDefaults
all three paths.
The load-bearing half is unchanged: the store never reaches for an
authority, because it must not take its identity from one it will itself
distribute. What was wrong is the claim that therefore nothing fills them
in. The service module declares no default and does not know where the
value comes from; the glue supplies one where the store is deployed; an
operator's own path beats it.
The assertion also says what reaching it now means — the glue is absent,
or something set the paths back to null.
OpenBao in a swarm-bao nixos-container, unsealed by the host TPM through
the built-in pkcs11 seal, with shamir as the documented opt-out for
hardware that has no TPM. The seal guard asks the package whether it was
built with an HSM rather than reading its version, and falls back to the
version cliff only when a package advertises no tags -- it fails closed.
Every listener serves TLS: loopback unconditionally, because the host
running the store is always one of its readers, plus whatever
extraListenAddresses names, because which network the other hives share
is a deployment fact. Client-certificate verification turns on only when
clientCaFile says what to trust.
The store's identity is an input, and nothing in THIS module fills it
in. Service-to-store mTLS is a separate trust domain from the gateway's
HTTPS certificates and from both CAs in this tree: a store must not take
its identity from an authority it will itself distribute, or it cannot
come up before the thing that issues it. The module asserts when the
paths are unset rather than reaching for whichever CA happens to be
wired; what supplies them on a self-contained deploy is the next commit.
The leaf is copied to the container's state dir by a host unit rather
than bind-mounted: nixos-container refuses to start when a bind source
is missing, and a directory holding a leaf usually holds the CA's key
beside it.
