vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
hyperhive/docs/swarm/secrets.md
atlas f4df4fc4a9 nix: issue the swarm-services leaf from bao's pki mount 
The `pki` mount had no issuer and no principal could log in to it, so the
swarm's service certificates were still minted by two openssl hops from a
root key on disk. Close both halves and retire the openssl path with them.

The mount now generates its own root, once. The granting unit asks bao
whether an issuer already exists (`bao list pki/issuers`) before calling
`pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role
and the grant without touching the anchor — a root that changed per boot
would invalidate every certificate issued under it and every browser
taught to trust it. The guard asks the store rather than looking for a
marker file on this host's disk: a file is a claim about a mount that may
have been restored from a snapshot or disabled and re-enabled underneath
it.

`swarm-services-issuer` stops being an inert policy. A fourth cert-auth
role attaches it, following the shape the controller, the publisher and
matrix-ctl already use, and glue-bao-tls.nix signs the leaf carrying its
CN — that credential is what opens the mount, so it cannot come out of it.

`swarm-services-cert.service` logs in with that leaf, calls
`pki/issue/swarm-services`, and writes the result to the path
hive-tls.nix already wrote and the gateway already copies from. The
sub-CA layer does not move; it stops existing. The role's
`allowed_domains`, read from the same `swarm.serviceDomains` the SANs
come from, enforces at issue time what the sub-CA encoded in x509
`nameConstraints`, and with the root inside the mount there is nothing
left for an intermediate to be an intermediate of.

Not a flag day: the issuing root is published beside the leaf as
`swarm-services-root.pem` (0644) and joins `trust-bundle.pem`, where the
swarm root still sits. A leaf chaining to the old sub-CA and one issued
by the store both verify against the same bundle, so hives can be
rebuilt in any order. The same file is what an operator hands a browser
— readable without a store login, which matters because every listener
demands a client certificate.

The eval-time warning about uncovered service names is gone rather than
reworded. It fired on "this host does not hold the swarm root key", which
was the reason a hive could end up serving its own leaf on a
swarm-service name. Every hive now asks the store with its own identity,
so that stopped being the thing that decides.

Closes #4586
2026-09-23 21:00:02 +02:00

438 lines
39 KiB
Markdown
Raw Blame History

# Swarm secrets: what exists, and where each one lives
A swarm generates its credentials in three different places and reads them
in a fourth, so "where does this file go" has a different answer per deployment.
This page is that answer, one row per secret.
Two rules run through all of it.
**Private key material and access tokens are paths, never values.** Every option
carrying one takes a file path (`*File`), because a literal written into a nix
expression ends up in the nix store — world-readable and permanent. No
option in this tree accepts one inline, and adding one would be a leak rather
than a convenience.
The rule is about what must stay secret, not about credentials generally.
**Public material is a value**: a certificate, or a public nkey like
`deploy.nats.calloutUserPublicKey`, reaches every client that connects,
so the store is a perfectly good place for it.
**The generator and the reader typically live in different containers.** They share
the host's network namespace, which makes them feel co-located, but their
filesystem roots are separate. That's why delivery is a **host-side copy rather
than a bind mount**: `nixos-container` refuses to start when a bind source is
missing, and a secret minted on another container's first boot doesn't exist
yet. Binding it would make one container wait on a file that waits on a
container that starts after it.
## Topologies, by who places secrets
Read every row below against one of these. This is a different cut
from the [deployment shapes](services.md#deployment-shapes) --- those
say _where services run_, these say _who is responsible for a secret
file being there_ --- the two lists don't line up one-to-one, and
neither is a renaming of the other.
| topology | what it means | who places secrets |
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when the operator sets `autoConfigure` | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** is published at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when the operator sets `deploy.nats.autoGenerateCallout` | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
Authelia mints the three keys for itself, in-container, precisely
because nothing outside that container ever reads them. **That's the test worth
applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
Three telemetry collectors exist, and they don't land on the same side of
that test.
The **hive's** collector needs no delivery step. It authenticates to the swarm's
collector as its own hive, and it's a host unit rather than a container, so on
an all-local swarm it reads authelia's file where it lies and nothing makes a
second copy. On any other topology it's an ordinary "operator provides the file"
case — see `services.hyperhive.otel.clientSecretFile`.
The **swarm's** collector does need one, and its delivery route is the fourth
store reader — the same shape as Grafana's, described in full below. In short:
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
out of the store on every host that runs the collector and holds a store
identity, whether authelia is a network away or in the container next door.
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
the client it reads for unconditionally: authelia refuses a
bearer-authz client with no audience, and the push audiences supply one even
for a collector with nothing published to scrape.
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
where an absent OIDC block is a container with no way in at all, a collector
with nothing to authenticate with still receives every hive's telemetry —
only its own pushes to the stores go out unauthenticated and get refused
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under this host's certificate and
lands it in the container's tree, where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it, and only a first boot ever waits.
### Minting the queue's callout nkeys
`deploy.nats.autoGenerateCallout` mints both keypairs on the host before the queue
starts. It's on by default only under `singleHostSwarm` — the one
topology where the queue, its responder and the operator are the same person. On
every other topology, mint them yourself:
```
nk -gen user > callout-user.seed # the responder's own identity
nk -gen account > issuer.seed # signs the user JWTs it hands out
nk -inkey callout-user.seed -pubout # → calloutUserPublicKey
nk -inkey issuer.seed -pubout # → calloutIssuerPublicKey
```
Keep both seeds at `0600` and name them in `deploy.nats.calloutUserSeedFile` /
`deploy.nats.calloutIssuerSeedFile`. Possession of the **issuer** seed is the authority to
admit anyone to the queue, so it belongs wherever the responder runs and nowhere
else.
A hive that sets neither the public keys nor `autoGenerateCallout` fails at
eval, naming the option it wants. That's deliberate: a queue that started
without them would accept `CONNECT {"user":"auth"}` from anyone sharing the
host's network namespace, and nothing would look wrong until somebody connected.
**All four or none** — you must supply the seed paths too, not just the public
keys. They're two halves of the same pair: the server verifies with the public
half, the responder signs with the private one. Supplying only the public keys
used to pass eval and leave the queue with an auth-callout nobody answers, which
refuses every client rather than degrading — and a refusal reaches the client as
a timeout, so the symptom is every consumer hanging with nothing logged.
One consequence of the generated path worth knowing before you debug it: with
`autoGenerateCallout` set, the queue assembles its config at boot rather than at
build time, so a malformed one surfaces when the container starts instead of
when the system builds. The server names the offending file and refuses to run.
## Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
Both delivery units wait for authelia's first boot to mint the secret — a
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
produces a service whose login button always fails, which is a symptom many
layers from its cause.
The store's **first reader** is the matrix appservice token, and it's worth
saying why that one: it's an opaque 32-byte value with no second file and no
format. Authelia's OIDC secret needs a `.secret` _and_ a matching `.digest`, so
starting there would have meant debugging "can a reader authenticate and get
bytes back" and "is authelia's file format right" at once, with an
SSO outage as the failure mode.
`glue-matrix-bao-token.nix` fetches it, writes the file `hive-matrix.nix`
already reads, and then runs that module's own renderer to re-stamp the
appservice registration naming the token — so the homeserver never learns the
store exists. The re-render isn't housekeeping: the token is half an
agreement, and a registration carrying the previous value authenticates
nobody. Every failure path — no such key, sealed store, unreachable store,
empty value — leaves the locally minted token in place, so a hive with no store
behaves exactly as it did before.
The store path is `swarm/hives/<hive>/matrix/appservice-token`, and the
**producer is `swarm-secret-publish`** — the same unit that copies authelia's
OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s
it there, so the store is the source of truth and every hive converges on the
value it holds. The hive's own activation mint is still there, but it's a
first-boot fallback now: it fires only when the token file is absent, and the
reader overwrites whatever it produced.
The mint is **idempotent by keeping its own record**, not by asking the store.
The publisher's grant is `create`/`update` under `swarm/hives/*` with no
`read` — write-only on purpose, so a file-copier can't recover every hive's
credentials — which means it can't check whether a hive already has a token.
Instead it keeps the value it minted under its `StateDirectory`,
`/var/lib/swarm-secret-publish/matrix-appservice-token/<hive>` (`0700` dir,
`0600` file), and mints only when that file is missing or empty. The `put`
itself runs every time: re-putting the same bytes changes nothing for any
reader, while a mint whose publish failed must not leave behind a token
this host holds and no hive can reach.
Lose the state directory and the next run mints once more and republishes.
That rotates the token, which readers pick up on their next start — nothing
that already registered breaks, because the token authenticates the
appservice rather than living inside any account it created.
The path was `…/matrix/registration-token` while the homeserver still took a
shared registration secret; nothing reads a value left at the old path.
The **second reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is
possible, so absent files mean this hive's agents don't connect, which is what
a swarm looks like before the publisher on the authelia host has run. The
reader runs before `hive-c0re.service`, which wants it (doesn't require it), so
an agent container never renders ahead of the credential; an unreachable store
delays the daemon's start rather than failing it.
That credential still has one hop left, because the reader of it lives inside
an agent container. **It crosses as a systemd credential, not as a bind
mount**, and the mode decides that: the secret is `root:0600` and a harness
runs as its own unprivileged agent user, so a bind would deliver a file that
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
it needs no read access to either — and hands them to
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
Inside, `hive-agent.service` names the same two ids in `LoadCredential=` and
reads them out of its own `$CREDENTIALS_DIRECTORY`, owned by the agent user
and by nobody else. `hive-c0re` never reads the bytes at any point: it runs as
`hive-core`, which is the reason a copy wasn't an option either. When the
files aren't there the daemon forwards nothing and says so in its journal, and
the harness logs that it has no queue coordinates — the same absent-and-legal state, twice,
rather than a container that refuses to start.
The **third reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the publisher on
authelia's host writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext
straight out of authelia's tree when the two happen to share a host is one
round trip cheaper and costs a second delivery unit, a second way for the file
to be wrong, and a gate deciding between them — and the gate is the expensive
part, because whatever it's wrong about is an outage nobody can read. The store
exists so the only secrets a host holds out of band are **certificates**, and
a host reads everything else with them.
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
running Grafana requires `swarm.authelia.url` (this module disables its local
login form unconditionally, so SSO isn't a feature of some topologies — it's the
only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
a null URL dropped the OIDC block, a missing leaf produced a warning, and
either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana.
The **fourth reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
differs is what an absent credential means: the collector still receives
telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them, the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
their own optional reads.
The **fifth reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
fourth in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is this host's static `swarm-bao-pki` leaf, not anything the store
mints.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs
a given service — placement is a `deploy.*` fact, per-host by definition — so
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
every swarm service's client secret, and that's stated in
`swarm-secret-client`'s `policy` module beside the grant itself.
⚠️ **The readers no longer present it.** Four units used to log in with
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
principal by the subject of the certificate it presents — so four readers behind
one leaf were one principal holding the union of their needs. Each now holds a
leaf of its own, and a policy naming only the path that unit reads. See
[per-principal identities](#per-principal-identities) below.
Absence behaves
as it does for the other readers: a store that says "not here" leaves the
file alone and says so, since there is no such thing as a locally valid OIDC
client secret to stand in.
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
credential you must already hold to authenticate from the thing it
authenticates you to, so the store's identity can't come from an authority the store
distributes — which excludes the hive CA and the swarm CA both, and has nothing
to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
mints a CA that signs a fixed, short list of leaves — the store's server
certificate, the hive's own reader certificate, and one per service principal —
and distributes nothing. A deployment with a real
internal CA deletes that file and names its own paths in
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
A hive that reads from a store on **another** machine names the reader's half —
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
plus one leaf per principal it runs (the table below names the options). Those are the
credentials that can't come out of the store, being what opens it; everything
else a hive needs does.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. These four units read one path
each and each holds a leaf, a role and a policy of its own:
| unit | option pair under `deploy.bao.` | reads |
| ------------------------ | -------------------------------------------------------- | -------------------------------------------------- |
| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives/<hive>/matrix/appservice-token` |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
The first two exist **per hive**, because the path they read carries a hive
name and every hive runs its own reader. Their subjects are
`<deploy.bao.matrixTokenCommonNamePrefix>-<hive>` and
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves both
composed spellings as hive names, so nobody can name a hive into another hive's
role. The other two read a path that names a swarm service rather than a hive, so
one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults
all eight options, and there is nothing to do. Elsewhere you issue each leaf from
that CA out of band and name it here — one file per principal rather than one file
shared by four, which is the whole of what this buys.
Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four
readers used to render only where their leaf existed, so a hand-configured
remote-store hive that named the hive's own `clientCertFile` and missed a
principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing
the build. All four now refuse it, each naming its own option pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named seven of the eight. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no homeserver, no collector, no Grafana — has no
unit to skip, so the second clause is false. Only the queue-credential reader
has no toggle of its own to check: every hive runs one for its own agents, so
reading the store at all is what asks for it.
The collector keeps a separate degrade underneath this, unchanged: a host with no
store identity runs a collector with no OIDC client secret and still receives
telemetry. The assertion doesn't touch that shape, because that host has no
`clientCertFile` either.
## How a reader reaches the store
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
the host that runs the store that name resolves twice. `/etc/hosts` answers a
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
and the network namespace that asked decides which half of it answers.
The container's half goes through nginx and **still doesn't terminate TLS**. It
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
ClientHello and splices the rest of the connection through byte for byte, so
openbao completes the handshake itself. A vhost would decrypt here instead, and
the client certificate — the store's whole authentication — would stop at nginx,
leaving openbao seeing one client for every hive in the swarm. That's why the
store has no vhost and gets a passthrough: not an exception to the no-vhost
rule, the one shape that keeps it.
The stream listener binds the bridge IP rather than every address, because
openbao already holds loopback on that port in the same network namespace and a
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
on the bridge firewall and nowhere else.
Reaching the port grants nothing by itself: openbao answers nothing without a
client certificate signed by `deploy.bao.clientCaFile`. The passthrough carries
whichever certificate the reader presents, unchanged.
## The constraint that decides where the root lives
A hive CA carries `nameConstraints=permitted;DNS:<hive domain>`, and **a swarm
service name is a sibling of the hive domain rather than a child** — `forge.<swarm>`
next to `<hive>.<swarm>`. A hive CA can't issue a certificate for a swarm
service. Not by policy: by construction, and openssl enforces it.
Whatever holds the swarm root is therefore what makes swarm-service certificates
possible at all. Two things follow:
- **The root's private key is a runtime file and must never enter the nix
store**, so nothing build-time can name it — the system reads
`security.pki.certificateFiles`' value at build time, which makes it the
wrong tool here. Trust reaches containers through a bind-mounted bundle
assembled at boot instead.
- **On any topology other than all-local, placing that key is an operations
decision**, not something this module tree makes for you. A hive that hosts no
swarm services needs only the root's _cert_, to trust what others issue.
## Adding a secret
State three things, in the row you add above: **who mints it**, **which
container reads it**, and **what happens when they differ**. If they differ, it
needs a delivery unit, and the unit copies — it doesn't bind.
Reference in a new issue View git blame Copy permalink