22a87f7268
Active-voice / contraction rewrites only, no meaning changes; the allowed_domains/SANs sentence and the published-cert table cell were checked against the parked services-issuer/role-type/vhost-scope questions on #4622 and don't touch any of them.
438 lines
39 KiB
Markdown
438 lines
39 KiB
Markdown
# Swarm secrets: what exists, and where each one lives
|
|
|
|
A swarm generates its credentials in three different places and reads them
|
|
in a fourth, so "where does this file go" has a different answer per deployment.
|
|
This page is that answer, one row per secret.
|
|
|
|
Two rules run through all of it.
|
|
|
|
**Private key material and access tokens are paths, never values.** Every option
|
|
carrying one takes a file path (`*File`), because a literal written into a nix
|
|
expression ends up in the nix store — world-readable and permanent. No
|
|
option in this tree accepts one inline, and adding one would be a leak rather
|
|
than a convenience.
|
|
|
|
The rule is about what must stay secret, not about credentials generally.
|
|
**Public material is a value**: a certificate, or a public nkey like
|
|
`deploy.nats.calloutUserPublicKey`, reaches every client that connects,
|
|
so the store is a perfectly good place for it.
|
|
|
|
**The generator and the reader typically live in different containers.** They share
|
|
the host's network namespace, which makes them feel co-located, but their
|
|
filesystem roots are separate. That's why delivery is a **host-side copy rather
|
|
than a bind mount**: `nixos-container` refuses to start when a bind source is
|
|
missing, and a secret minted on another container's first boot doesn't exist
|
|
yet. Binding it would make one container wait on a file that waits on a
|
|
container that starts after it.
|
|
|
|
## Topologies, by who places secrets
|
|
|
|
Read every row below against one of these. This is a different cut
|
|
from the [deployment shapes](services.md#deployment-shapes) --- those
|
|
say _where services run_, these say _who is responsible for a secret
|
|
file being there_ --- the two lists don't line up one-to-one, and
|
|
neither is a renaming of the other.
|
|
|
|
| topology | what it means | who places secrets |
|
|
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
|
|
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
|
|
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
|
|
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
|
|
|
|
## Swarm-level — one of each per swarm
|
|
|
|
| secret | generated by | lives at | hive elsewhere |
|
|
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| swarm root CA cert | `swarm-ca.nix` first-boot unit, when the operator sets `autoConfigure` | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public |
|
|
| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below |
|
|
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for |
|
|
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
|
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
|
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
|
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
|
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
|
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
|
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
|
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
|
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
|
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when the operator sets `deploy.nats.autoGenerateCallout` | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
|
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
|
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
|
|
|
Authelia mints the three keys for itself, in-container, precisely
|
|
because nothing outside that container ever reads them. **That's the test worth
|
|
applying to any secret added here** — and the client secret's plaintext half is
|
|
the one row that fails it, which is the entire reason a delivery step exists.
|
|
|
|
Three telemetry collectors exist, and they don't land on the same side of
|
|
that test.
|
|
|
|
The **hive's** collector needs no delivery step. It authenticates to the swarm's
|
|
collector as its own hive, and it's a host unit rather than a container, so on
|
|
an all-local swarm it reads authelia's file where it lies and nothing makes a
|
|
second copy. On any other topology it's an ordinary "operator provides the file"
|
|
case — see `services.hyperhive.otel.clientSecretFile`.
|
|
|
|
The **swarm's** collector does need one, and its delivery route is the fourth
|
|
store reader — the same shape as Grafana's, described in full below. In short:
|
|
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
|
|
out of the store on every host that runs the collector and holds a store
|
|
identity, whether authelia is a network away or in the container next door.
|
|
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
|
|
the client it reads for unconditionally: authelia refuses a
|
|
bearer-authz client with no audience, and the push audiences supply one even
|
|
for a collector with nothing published to scrape.
|
|
|
|
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
|
|
where an absent OIDC block is a container with no way in at all, a collector
|
|
with nothing to authenticate with still receives every hive's telemetry —
|
|
only its own pushes to the stores go out unauthenticated and get refused
|
|
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
|
|
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
|
|
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
|
|
uses for its non-optional one. The operator can still deliver the secret by
|
|
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
|
|
collector this unit never reaches.
|
|
|
|
The **secret store's own** collector — the forwarder inside the `swarm-bao`
|
|
container — needs a delivery step too, and it takes the same route with one
|
|
principal of its own: `swarm-bao-forwarder-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` under this host's certificate and
|
|
lands it in the container's tree, where `LoadCredential` hands it to the
|
|
collector. The client id is its own
|
|
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
|
|
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
|
|
collector admits it on a receiver of its own — one identity per principal, so
|
|
this forwarder borrows neither the swarm collector's client nor
|
|
`swarm-controller`'s.
|
|
|
|
⚠️ **This one has no degraded mode, and the difference is structural.** The
|
|
route it exports over accepts an authenticated request and nothing else, so a
|
|
forwarder without the secret has nothing to fall back to: the unit fails and
|
|
retries, and the collector refuses to start until the file arrives rather than
|
|
exporting into a 401 while reporting healthy. Its ordering differs from the
|
|
unit above it for the same structural reason — it reads the store inside the
|
|
container it delivers into, so it runs **after** that container rather than
|
|
before it, and only a first boot ever waits.
|
|
|
|
### Minting the queue's callout nkeys
|
|
|
|
`deploy.nats.autoGenerateCallout` mints both keypairs on the host before the queue
|
|
starts. It's on by default only under `singleHostSwarm` — the one
|
|
topology where the queue, its responder and the operator are the same person. On
|
|
every other topology, mint them yourself:
|
|
|
|
```
|
|
nk -gen user > callout-user.seed # the responder's own identity
|
|
nk -gen account > issuer.seed # signs the user JWTs it hands out
|
|
nk -inkey callout-user.seed -pubout # → calloutUserPublicKey
|
|
nk -inkey issuer.seed -pubout # → calloutIssuerPublicKey
|
|
```
|
|
|
|
Keep both seeds at `0600` and name them in `deploy.nats.calloutUserSeedFile` /
|
|
`deploy.nats.calloutIssuerSeedFile`. Possession of the **issuer** seed is the authority to
|
|
admit anyone to the queue, so it belongs wherever the responder runs and nowhere
|
|
else.
|
|
|
|
A hive that sets neither the public keys nor `autoGenerateCallout` fails at
|
|
eval, naming the option it wants. That's deliberate: a queue that started
|
|
without them would accept `CONNECT {"user":"auth"}` from anyone sharing the
|
|
host's network namespace, and nothing would look wrong until somebody connected.
|
|
|
|
**All four or none** — you must supply the seed paths too, not just the public
|
|
keys. They're two halves of the same pair: the server verifies with the public
|
|
half, the responder signs with the private one. Supplying only the public keys
|
|
used to pass eval and leave the queue with an auth-callout nobody answers, which
|
|
refuses every client rather than degrading — and a refusal reaches the client as
|
|
a timeout, so the symptom is every consumer hanging with nothing logged.
|
|
|
|
One consequence of the generated path worth knowing before you debug it: with
|
|
`autoGenerateCallout` set, the queue assembles its config at boot rather than at
|
|
build time, so a malformed one surfaces when the container starts instead of
|
|
when the system builds. The server names the offending file and refuses to run.
|
|
|
|
## Hive-level — one of each per hive
|
|
|
|
| secret | generated by | lives at |
|
|
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
|
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
|
|
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
|
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
|
|
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
|
|
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
|
|
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
|
|
|
|
Both delivery units wait for authelia's first boot to mint the secret — a
|
|
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
|
|
produces a service whose login button always fails, which is a symptom many
|
|
layers from its cause.
|
|
|
|
The store's **first reader** is the matrix appservice token, and it's worth
|
|
saying why that one: it's an opaque 32-byte value with no second file and no
|
|
format. Authelia's OIDC secret needs a `.secret` _and_ a matching `.digest`, so
|
|
starting there would have meant debugging "can a reader authenticate and get
|
|
bytes back" and "is authelia's file format right" at once, with an
|
|
SSO outage as the failure mode.
|
|
|
|
`glue-matrix-bao-token.nix` fetches it, writes the file `hive-matrix.nix`
|
|
already reads, and then runs that module's own renderer to re-stamp the
|
|
appservice registration naming the token — so the homeserver never learns the
|
|
store exists. The re-render isn't housekeeping: the token is half an
|
|
agreement, and a registration carrying the previous value authenticates
|
|
nobody. Every failure path — no such key, sealed store, unreachable store,
|
|
empty value — leaves the locally minted token in place, so a hive with no store
|
|
behaves exactly as it did before.
|
|
|
|
The store path is `swarm/hives/<hive>/matrix/appservice-token`, and the
|
|
**producer is `swarm-secret-publish`** — the same unit that copies authelia's
|
|
OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s
|
|
it there, so the store is the source of truth and every hive converges on the
|
|
value it holds. The hive's own activation mint is still there, but it's a
|
|
first-boot fallback now: it fires only when the token file is absent, and the
|
|
reader overwrites whatever it produced.
|
|
|
|
The mint is **idempotent by keeping its own record**, not by asking the store.
|
|
The publisher's grant is `create`/`update` under `swarm/hives/*` with no
|
|
`read` — write-only on purpose, so a file-copier can't recover every hive's
|
|
credentials — which means it can't check whether a hive already has a token.
|
|
Instead it keeps the value it minted under its `StateDirectory`,
|
|
`/var/lib/swarm-secret-publish/matrix-appservice-token/<hive>` (`0700` dir,
|
|
`0600` file), and mints only when that file is missing or empty. The `put`
|
|
itself runs every time: re-putting the same bytes changes nothing for any
|
|
reader, while a mint whose publish failed must not leave behind a token
|
|
this host holds and no hive can reach.
|
|
|
|
Lose the state directory and the next run mints once more and republishes.
|
|
That rotates the token, which readers pick up on their next start — nothing
|
|
that already registered breaks, because the token authenticates the
|
|
appservice rather than living inside any account it created.
|
|
|
|
The path was `…/matrix/registration-token` while the homeserver still took a
|
|
shared registration secret; nothing reads a value left at the old path.
|
|
|
|
The **second reader** is the agent containers' queue credential:
|
|
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
|
|
the client id it authenticates, because that's how a queue client reads them —
|
|
the secret by path, the id by value. No local fallback exists, and none is
|
|
possible, so absent files mean this hive's agents don't connect, which is what
|
|
a swarm looks like before the publisher on the authelia host has run. The
|
|
reader runs before `hive-c0re.service`, which wants it (doesn't require it), so
|
|
an agent container never renders ahead of the credential; an unreachable store
|
|
delays the daemon's start rather than failing it.
|
|
|
|
That credential still has one hop left, because the reader of it lives inside
|
|
an agent container. **It crosses as a systemd credential, not as a bind
|
|
mount**, and the mode decides that: the secret is `root:0600` and a harness
|
|
runs as its own unprivileged agent user, so a bind would deliver a file that
|
|
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
|
|
it needs no read access to either — and hands them to
|
|
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
|
|
Inside, `hive-agent.service` names the same two ids in `LoadCredential=` and
|
|
reads them out of its own `$CREDENTIALS_DIRECTORY`, owned by the agent user
|
|
and by nobody else. `hive-c0re` never reads the bytes at any point: it runs as
|
|
`hive-core`, which is the reason a copy wasn't an option either. When the
|
|
files aren't there the daemon forwards nothing and says so in its journal, and
|
|
the harness logs that it has no queue coordinates — the same absent-and-legal state, twice,
|
|
rather than a container that refuses to start.
|
|
|
|
The **third reader** is Grafana's OIDC client secret, and it's the first one
|
|
that's a _swarm service's_ own credential rather than a hive's. It has exactly
|
|
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
|
that runs Grafana. Where authelia is doesn't enter into it — the publisher on
|
|
authelia's host writes that path whether the reader is a network away or in the
|
|
container next door.
|
|
|
|
That's a deliberate refusal of the shortcut. Copying the plaintext
|
|
straight out of authelia's tree when the two happen to share a host is one
|
|
round trip cheaper and costs a second delivery unit, a second way for the file
|
|
to be wrong, and a gate deciding between them — and the gate is the expensive
|
|
part, because whatever it's wrong about is an outage nobody can read. The store
|
|
exists so the only secrets a host holds out of band are **certificates**, and
|
|
a host reads everything else with them.
|
|
|
|
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
|
|
running Grafana requires `swarm.authelia.url` (this module disables its local
|
|
login form unconditionally, so SSO isn't a feature of some topologies — it's the
|
|
only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
|
|
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
|
|
a null URL dropped the OIDC block, a missing leaf produced a warning, and
|
|
either one left a Grafana with no SSO and no password box — nothing failed, so
|
|
nothing said anything. Registration is separate and stays where authelia is
|
|
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
|
|
so it's declared on the host running authelia, which isn't necessarily the host
|
|
running Grafana.
|
|
|
|
The **fourth reader** is the swarm collector's own OIDC client secret, the
|
|
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
|
runs the collector, and registration lives separately in
|
|
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
|
|
differs is what an absent credential means: the collector still receives
|
|
telemetry with none, so `swarm-otel.nix` doesn't assert
|
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
|
|
does — it
|
|
doesn't render the reading unit without them, the shape
|
|
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
|
|
their own optional reads.
|
|
|
|
The **fifth reader** is the secret store forwarder's OIDC client secret:
|
|
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
|
|
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
|
|
fourth in both directions. It renders unconditionally, because the export it
|
|
authenticates has no unauthenticated mode to degrade into — and it orders
|
|
itself **after** `container@swarm-bao`, because the store it reads runs in the
|
|
container it delivers into. Nothing circular sits behind that: the identity it
|
|
logs in with is this host's static `swarm-bao-pki` leaf, not anything the store
|
|
mints.
|
|
|
|
A service's secret is one value for the whole swarm rather than one per hive, so
|
|
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
|
whole. That's because nothing in a swarm's configuration records which hive runs
|
|
a given service — placement is a `deploy.*` fact, per-host by definition — so
|
|
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
|
|
every swarm service's client secret, and that's stated in
|
|
`swarm-secret-client`'s `policy` module beside the grant itself.
|
|
|
|
⚠️ **The readers no longer present it.** Four units used to log in with
|
|
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
|
|
principal by the subject of the certificate it presents — so four readers behind
|
|
one leaf were one principal holding the union of their needs. Each now holds a
|
|
leaf of its own, and a policy naming only the path that unit reads. See
|
|
[per-principal identities](#per-principal-identities) below.
|
|
|
|
Absence behaves
|
|
as it does for the other readers: a store that says "not here" leaves the
|
|
file alone and says so, since there is no such thing as a locally valid OIDC
|
|
client secret to stand in.
|
|
|
|
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
|
|
credential you must already hold to authenticate from the thing it
|
|
authenticates you to, so the store's identity can't come from an authority the store
|
|
distributes — which excludes the hive CA and the swarm CA both, and has nothing
|
|
to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
|
|
mints a CA that signs a fixed, short list of leaves — the store's server
|
|
certificate, the hive's own reader certificate, and one per service principal —
|
|
and distributes nothing. A deployment with a real
|
|
internal CA deletes that file and names its own paths in
|
|
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
|
|
A hive that reads from a store on **another** machine names the reader's half —
|
|
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
|
|
plus one leaf per principal it runs (the table below names the options). Those are the
|
|
credentials that can't come out of the store, being what opens it; everything
|
|
else a hive needs does.
|
|
|
|
### Per-principal identities
|
|
|
|
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
|
identity and sharing one merges the identities. These four units read one path
|
|
each and each holds a leaf, a role and a policy of its own:
|
|
|
|
| unit | option pair under `deploy.bao.` | reads |
|
|
| ------------------------ | -------------------------------------------------------- | -------------------------------------------------- |
|
|
| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives/<hive>/matrix/appservice-token` |
|
|
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
|
|
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
|
|
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
|
|
|
The first two exist **per hive**, because the path they read carries a hive
|
|
name and every hive runs its own reader. Their subjects are
|
|
`<deploy.bao.matrixTokenCommonNamePrefix>-<hive>` and
|
|
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves both
|
|
composed spellings as hive names, so nobody can name a hive into another hive's
|
|
role. The other two read a path that names a swarm service rather than a hive, so
|
|
one role each is enough and their subjects are the flat
|
|
`deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`.
|
|
|
|
On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults
|
|
all eight options, and there is nothing to do. Elsewhere you issue each leaf from
|
|
that CA out of band and name it here — one file per principal rather than one file
|
|
shared by four, which is the whole of what this buys.
|
|
|
|
Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four
|
|
readers used to render only where their leaf existed, so a hand-configured
|
|
remote-store hive that named the hive's own `clientCertFile` and missed a
|
|
principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing
|
|
the build. All four now refuse it, each naming its own option pair — see
|
|
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
|
|
|
### What a missing leaf costs
|
|
|
|
Each of the four modules asserts its own pair, on the same condition: **this host
|
|
already reads the store** — it holds `deploy.bao.clientCertFile` and
|
|
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
|
|
The refusal names the two options, so an operator acts on the message without
|
|
opening the nix.
|
|
|
|
That condition fires on exactly one shape, the hand-configured remote-store hive
|
|
that named seven of the eight. It stays silent everywhere else, and each half of
|
|
that matters on its own:
|
|
|
|
- **A host with no store identity at all** holds no `clientCertFile` either, so
|
|
the first clause is false. That's the supported no-store deployment, and also
|
|
the one an operator passes through while bringing a hive up — neither should
|
|
fail to evaluate.
|
|
- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as
|
|
defaults, so the third clause is false. Nothing to set, nothing to refuse.
|
|
- **A consumer that's off** — no homeserver, no collector, no Grafana — has no
|
|
unit to skip, so the second clause is false. Only the queue-credential reader
|
|
has no toggle of its own to check: every hive runs one for its own agents, so
|
|
reading the store at all is what asks for it.
|
|
|
|
The collector keeps a separate degrade underneath this, unchanged: a host with no
|
|
store identity runs a collector with no OIDC client secret and still receives
|
|
telemetry. The assertion doesn't touch that shape, because that host has no
|
|
`clientCertFile` either.
|
|
|
|
## How a reader reaches the store
|
|
|
|
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
|
|
the host that runs the store that name resolves twice. `/etc/hosts` answers a
|
|
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
|
|
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
|
|
and the network namespace that asked decides which half of it answers.
|
|
|
|
The container's half goes through nginx and **still doesn't terminate TLS**. It
|
|
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
|
|
ClientHello and splices the rest of the connection through byte for byte, so
|
|
openbao completes the handshake itself. A vhost would decrypt here instead, and
|
|
the client certificate — the store's whole authentication — would stop at nginx,
|
|
leaving openbao seeing one client for every hive in the swarm. That's why the
|
|
store has no vhost and gets a passthrough: not an exception to the no-vhost
|
|
rule, the one shape that keeps it.
|
|
|
|
The stream listener binds the bridge IP rather than every address, because
|
|
openbao already holds loopback on that port in the same network namespace and a
|
|
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
|
|
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
|
|
on the bridge firewall and nowhere else.
|
|
|
|
Reaching the port grants nothing by itself: openbao answers nothing without a
|
|
client certificate signed by `deploy.bao.clientCaFile`. The passthrough carries
|
|
whichever certificate the reader presents, unchanged.
|
|
|
|
## The constraint that decides where the root lives
|
|
|
|
A hive CA carries `nameConstraints=permitted;DNS:<hive domain>`, and **a swarm
|
|
service name is a sibling of the hive domain rather than a child** — `forge.<swarm>`
|
|
next to `<hive>.<swarm>`. A hive CA can't issue a certificate for a swarm
|
|
service. Not by policy: by construction, and openssl enforces it.
|
|
|
|
Whatever holds the swarm root is therefore what makes swarm-service certificates
|
|
possible at all. Two things follow:
|
|
|
|
- **The root's private key is a runtime file and must never enter the nix
|
|
store**, so nothing build-time can name it — the system reads
|
|
`security.pki.certificateFiles`' value at build time, which makes it the
|
|
wrong tool here. Trust reaches containers through a bind-mounted bundle
|
|
assembled at boot instead.
|
|
- **On any topology other than all-local, placing that key is an operations
|
|
decision**, not something this module tree makes for you. A hive that hosts no
|
|
swarm services needs only the root's _cert_, to trust what others issue.
|
|
|
|
## Adding a secret
|
|
|
|
State three things, in the row you add above: **who mints it**, **which
|
|
container reads it**, and **what happens when they differ**. If they differ, it
|
|
needs a delivery unit, and the unit copies — it doesn't bind.
|