cf7d357f17
The docs, option descriptions and comments that named the swarm root, the swarm CA or the hive CA as an existing trust anchor now name the secret store's pki root or the trust bundle that holds it: - docs/swarm/README.md, secrets.md, bao.md - deploy.bao.clientCaFile's and gateway.verifiedProxyTo's descriptions - comments in hive-c0re/src/meta.rs, hive-agent/src/swarm_queue.rs, swarm-logs/src/auth.rs, swarm-controller/src/vcs_metrics.rs, hive-forge, swarm-authelia, swarm-ci-runner, glue-bao-tls, swarm-bao, swarm-grafana, swarm-otel, otel, hive-c0re, lib/hive-ca-trust.nix, hive-forge/Cargo.toml and the flake-update workflow Nothing behaves differently: every edit is a nix, Rust or TOML comment, an option description, or a test assertion message. The Rust edits change the workspace source, which this PR already rebuilds for hivectl. Refs #4588
457 lines
41 KiB
Markdown
457 lines
41 KiB
Markdown
# Swarm secrets: what exists, and where each one lives
|
|
|
|
A swarm generates its credentials in three different places and reads them
|
|
in a fourth, so "where does this file go" has a different answer per deployment.
|
|
This page is that answer, one row per secret.
|
|
|
|
Two rules run through all of it.
|
|
|
|
**Private key material and access tokens are paths, never values.** Every option
|
|
carrying one takes a file path (`*File`), because a literal written into a nix
|
|
expression ends up in the nix store — world-readable and permanent. No
|
|
option in this tree accepts one inline, and adding one would be a leak rather
|
|
than a convenience.
|
|
|
|
The rule is about what must stay secret, not about credentials generally.
|
|
**Public material is a value**: a certificate, or a public nkey like
|
|
`deploy.nats.calloutUserPublicKey`, reaches every client that connects,
|
|
so the store is a perfectly good place for it.
|
|
|
|
**The generator and the reader typically live in different containers.** They share
|
|
the host's network namespace, which makes them feel co-located, but their
|
|
filesystem roots are separate. That's why delivery is a **host-side copy rather
|
|
than a bind mount**: `nixos-container` refuses to start when a bind source is
|
|
missing, and a secret minted on another container's first boot doesn't exist
|
|
yet. Binding it would make one container wait on a file that waits on a
|
|
container that starts after it.
|
|
|
|
## Topologies, by who places secrets
|
|
|
|
Read every row below against one of these. This is a different cut
|
|
from the [deployment shapes](services.md#deployment-shapes) --- those
|
|
say _where services run_, these say _who is responsible for a secret
|
|
file being there_ --- the two lists don't line up one-to-one, and
|
|
neither is a renaming of the other.
|
|
|
|
| topology | what it means | who places secrets |
|
|
| ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------- |
|
|
| **all-local** | one host runs the swarm's shared services and its own hive | nobody — the reader itself generates each secret, or a host unit copies it |
|
|
| **swarm-managed** | the swarm's services run on a host with `swarmctl` | `swarmctl` writes what it owns; the rest is still generated in place |
|
|
| **hive elsewhere** | a hive that federates with a swarm it doesn't host | the operator provides the file and names it in config |
|
|
|
|
## Swarm-level — one of each per swarm
|
|
|
|
| secret | generated by | lives at | hive elsewhere |
|
|
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
|
|
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
|
|
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
|
|
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
|
|
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service, `otel.clientSecretFile` for the hive's telemetry collector |
|
|
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
|
|
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
|
|
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
|
|
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
|
|
| the queue responder's copy of its OIDC secret | `swarm-bao-nats-auth-oidc.service` reads it out of the swarm secret store, on the host that runs the queue, under the responder's own leaf | `/var/lib/swarm-nats-auth/oidc-client.secret` inside the `swarm-nats` container, handed to the responder through `LoadCredential` | same unit, same path, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
|
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
|
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
|
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when the operator sets `deploy.nats.autoGenerateCallout` | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
|
|
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
|
|
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
|
|
|
|
Authelia mints the three keys for itself, in-container, precisely
|
|
because nothing outside that container ever reads them. **That's the test worth
|
|
applying to any secret added here** — and the client secret's plaintext half is
|
|
the one row that fails it, which is the entire reason a delivery step exists.
|
|
|
|
Three telemetry collectors exist, and they don't land on the same side of
|
|
that test.
|
|
|
|
The **hive's** collector needs no delivery step. It authenticates to the swarm's
|
|
collector as its own hive, and it's a host unit rather than a container, so on
|
|
an all-local swarm it reads authelia's file where it lies and nothing makes a
|
|
second copy. On any other topology it's an ordinary "operator provides the file"
|
|
case — see `services.hyperhive.otel.clientSecretFile`.
|
|
|
|
The **swarm's** collector does need one, and its delivery route is the fourth
|
|
store reader — the same shape as Grafana's, described in full below. In short:
|
|
`swarm-bao-otel-oidc.service` reads `swarm/services/<client-id>/oidc/client`
|
|
out of the store on every host that runs the collector and holds a store
|
|
identity, whether authelia is a network away or in the container next door.
|
|
The same call `glue-grafana-oidc-client.nix` makes for Grafana's registers
|
|
the client it reads for unconditionally: authelia refuses a
|
|
bearer-authz client with no audience, and the push audiences supply one even
|
|
for a collector with nothing published to scrape.
|
|
|
|
⚠️ **A collector with no store identity isn't refused.** Unlike Grafana,
|
|
where an absent OIDC block is a container with no way in at all, a collector
|
|
with nothing to authenticate with still receives every hive's telemetry —
|
|
only its own pushes to the stores go out unauthenticated and get refused
|
|
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
|
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
|
|
`glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix`
|
|
uses for its non-optional one. The operator can still deliver the secret by
|
|
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
|
|
collector this unit never reaches.
|
|
|
|
The **secret store's own** collector — the forwarder inside the `swarm-bao`
|
|
container — needs a delivery step too, and it takes the same route with one
|
|
principal of its own: `swarm-bao-forwarder-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` under a leaf of its own and
|
|
lands it in the container's tree, where `LoadCredential` hands it to the
|
|
collector. The client id is its own
|
|
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
|
|
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
|
|
collector admits it on a receiver of its own — one identity per principal, so
|
|
this forwarder borrows neither the swarm collector's client nor
|
|
`swarm-controller`'s.
|
|
|
|
⚠️ **This one has no degraded mode, and the difference is structural.** The
|
|
route it exports over accepts an authenticated request and nothing else, so a
|
|
forwarder without the secret has nothing to fall back to: the unit fails and
|
|
retries, and the collector refuses to start until the file arrives rather than
|
|
exporting into a 401 while reporting healthy. Its ordering differs from the
|
|
unit above it for the same structural reason — it reads the store inside the
|
|
container it delivers into, so it runs **after** that container rather than
|
|
before it, and only a first boot ever waits.
|
|
|
|
### Minting the queue's callout nkeys
|
|
|
|
`deploy.nats.autoGenerateCallout` mints both keypairs on the host before the queue
|
|
starts. It's on by default only under `singleHostSwarm` — the one
|
|
topology where the queue, its responder and the operator are the same person. On
|
|
every other topology, mint them yourself:
|
|
|
|
```
|
|
nk -gen user > callout-user.seed # the responder's own identity
|
|
nk -gen account > issuer.seed # signs the user JWTs it hands out
|
|
nk -inkey callout-user.seed -pubout # → calloutUserPublicKey
|
|
nk -inkey issuer.seed -pubout # → calloutIssuerPublicKey
|
|
```
|
|
|
|
Keep both seeds at `0600` and name them in `deploy.nats.calloutUserSeedFile` /
|
|
`deploy.nats.calloutIssuerSeedFile`. Possession of the **issuer** seed is the authority to
|
|
admit anyone to the queue, so it belongs wherever the responder runs and nowhere
|
|
else.
|
|
|
|
A hive that sets neither the public keys nor `autoGenerateCallout` fails at
|
|
eval, naming the option it wants. That's deliberate: a queue that started
|
|
without them would accept `CONNECT {"user":"auth"}` from anyone sharing the
|
|
host's network namespace, and nothing would look wrong until somebody connected.
|
|
|
|
**All four or none** — you must supply the seed paths too, not just the public
|
|
keys. They're two halves of the same pair: the server verifies with the public
|
|
half, the responder signs with the private one. Supplying only the public keys
|
|
used to pass eval and leave the queue with an auth-callout nobody answers, which
|
|
refuses every client rather than degrading — and a refusal reaches the client as
|
|
a timeout, so the symptom is every consumer hanging with nothing logged.
|
|
|
|
One consequence of the generated path worth knowing before you debug it: with
|
|
`autoGenerateCallout` set, the queue assembles its config at boot rather than at
|
|
build time, so a malformed one surfaces when the container starts instead of
|
|
when the system builds. The server names the offending file and refuses to run.
|
|
|
|
## Hive-level — one of each per hive
|
|
|
|
| secret | generated by | lives at |
|
|
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
|
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
|
|
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
|
|
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
|
|
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
|
|
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
|
|
|
|
The forge's delivery unit waits for authelia's first boot to mint the secret — a
|
|
bounded wait, 120s — and then **fails loudly** rather than skipping. A silent skip
|
|
produces a service whose login button always fails, which is a symptom many
|
|
layers from its cause. The homeserver's reads the store under matrix-ctl's leaf,
|
|
the way Grafana's does below.
|
|
|
|
The store's **first reader** is the agent containers' queue credential:
|
|
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
|
|
the client id it authenticates, because that's how a queue client reads them —
|
|
the secret by path, the id by value. No local fallback exists, and none is
|
|
possible, so absent files mean this hive's agents have no `swarm-logs`, which
|
|
is what a swarm looks like before the publisher on the authelia host has run.
|
|
The reader runs before `hive-c0re.service`, which wants it (doesn't require
|
|
it), so an agent container never renders ahead of the credential; an
|
|
unreachable store delays the daemon's start rather than failing it.
|
|
|
|
That credential still has one hop left, because the reader of it lives inside
|
|
an agent container. **It crosses as a systemd credential, not as a bind
|
|
mount**, and the mode decides that: the secret is `root:0600` and a harness
|
|
runs as its own unprivileged agent user, so a bind would deliver a file that
|
|
user can't open. `hive-c0re` stats the two files — the directory is `0755`, so
|
|
it needs no read access to either — and hands them to
|
|
`systemd-nspawn --load-credential` through `hive-priv`, which runs as root.
|
|
Inside, `hive-agent.service` names the same two ids in `LoadCredential=`, which
|
|
lands them in its own `$CREDENTIALS_DIRECTORY`, owned by the agent user and by
|
|
nobody else, where the agent's `swarm-logs` reads them. The harness itself
|
|
doesn't: it connects to the queue with the agent's own credential. `hive-c0re`
|
|
never reads the bytes at any point: it runs as `hive-core`, which is the reason
|
|
a copy wasn't an option either. When the files aren't there the daemon forwards
|
|
nothing and says so in its journal, and the container starts without them.
|
|
|
|
The **second reader** is Grafana's OIDC client secret, and it's the first one
|
|
that's a _swarm service's_ own credential rather than a hive's. It has exactly
|
|
one delivery route: `swarm-bao-grafana-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
|
|
that runs Grafana. Where authelia is doesn't enter into it — the publisher on
|
|
authelia's host writes that path whether the reader is a network away or in the
|
|
container next door.
|
|
|
|
That's a deliberate refusal of the shortcut. Copying the plaintext
|
|
straight out of authelia's tree when the two happen to share a host is one
|
|
round trip cheaper and costs a second delivery unit, a second way for the file
|
|
to be wrong, and a gate deciding between them — and the gate is the expensive
|
|
part, because whatever it's wrong about is an outage nobody can read. The store
|
|
exists so the only secrets a host holds out of band are **certificates**, and
|
|
a host reads everything else with them.
|
|
|
|
Two things follow, and `swarm-grafana.nix` asserts both rather than degrading:
|
|
running Grafana requires `swarm.authelia.url` (this module disables its local
|
|
login form unconditionally, so SSO isn't a feature of some topologies — it's the
|
|
only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile`
|
|
/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent:
|
|
a null URL dropped the OIDC block, a missing leaf produced a warning, and
|
|
either one left a Grafana with no SSO and no password box — nothing failed, so
|
|
nothing said anything. Registration is separate and stays where authelia is
|
|
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
|
|
so it's declared on the host running authelia, which isn't necessarily the host
|
|
running Grafana. The swarm controller, the queue and the homeserver register the
|
|
same way (`glue-controller-oidc-client.nix`, `glue-nats-oidc-client.nix`,
|
|
`glue-matrix-oidc-client.nix`), and the queue's responder and the homeserver read
|
|
their secrets from the store too: `swarm-bao-nats-auth-oidc.service` under the
|
|
responder's own leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's. Each
|
|
renders wherever its consumer runs and fails, naming the option pair, when that
|
|
leaf is unset — neither consumer starts without its secret.
|
|
|
|
The **third reader** is the swarm collector's own OIDC client secret, the
|
|
same shape one level down: `swarm-bao-otel-oidc.service` reads
|
|
`swarm/services/<client-id>/oidc/client` out of the store on every host that
|
|
runs the collector, and registration lives separately in
|
|
`glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it
|
|
differs is what an absent credential means: the collector still receives
|
|
telemetry with none, so `swarm-otel.nix` doesn't assert
|
|
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
|
|
does — it
|
|
doesn't render the reading unit without them, the shape
|
|
`glue-queue-agent-credential.nix` uses for its own read.
|
|
|
|
The **fourth reader** is the secret store forwarder's OIDC client secret:
|
|
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
|
|
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
|
|
third in both directions. It renders unconditionally, because the export it
|
|
authenticates has no unauthenticated mode to degrade into — and it orders
|
|
itself **after** `container@swarm-bao`, because the store it reads runs in the
|
|
container it delivers into. Nothing circular sits behind that: the identity it
|
|
logs in with is the static `forwarder-oidc.pem` leaf `swarm-bao-pki` signs, not
|
|
anything the store mints. With no leaf it has no fallback, so `swarm-bao.nix`
|
|
refuses the build and names `deploy.bao.forwarderOidcClientCertFile` /
|
|
`forwarderOidcClientKeyFile`.
|
|
|
|
A service's secret is one value for the whole swarm rather than one per hive, so
|
|
it lives under the `services` prefix, and a hive's read policy grants that prefix
|
|
whole. That's because nothing in a swarm's configuration records which hive runs
|
|
a given service — placement is a `deploy.*` fact, per-host by definition — so
|
|
there is nothing to scope the grant to. A hive's **own** leaf can therefore read
|
|
every swarm service's client secret, and that's stated in
|
|
`swarm-secret-client`'s `policy` module beside the grant itself.
|
|
|
|
⚠️ **The readers no longer present it.** Five units used to log in with
|
|
`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a
|
|
principal by the subject of the certificate it presents — so five readers behind
|
|
one leaf were one principal holding the union of their needs. Each now holds a
|
|
leaf of its own, and a policy naming only the path that unit reads. See
|
|
[per-principal identities](#per-principal-identities) below.
|
|
|
|
Absence behaves
|
|
as it does for the other readers: a store that says "not here" leaves the
|
|
file alone and says so, since there is no such thing as a locally valid OIDC
|
|
client secret to stand in.
|
|
|
|
⚠️ **Service↔store mTLS is its own trust domain.** You can't fetch a
|
|
credential you must already hold to authenticate from the thing it
|
|
authenticates you to, so the store's identity can't come from an authority the store
|
|
distributes — which excludes the store's `pki` root, and has nothing to do
|
|
with the gateway's HTTPS certificates either way. `glue-bao-tls.nix`
|
|
mints a CA that signs a fixed, short list of leaves — the store's server
|
|
certificate, the hive's own reader certificate, and one per service principal —
|
|
and distributes nothing. A deployment with a real
|
|
internal CA deletes that file and names its own paths in
|
|
`deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion.
|
|
A hive that reads from a store on **another** machine names the reader's half —
|
|
`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand,
|
|
plus one leaf per principal it runs (the table below names the options). Those are the
|
|
credentials that can't come out of the store, being what opens it; everything
|
|
else a hive needs does.
|
|
|
|
Agents are the one principal whose leaf the store issues. Its `pki-agents`
|
|
mount (`deploy.bao.agentPkiMountPath`) holds an agent CA generated inside the
|
|
store, and `swarm-controller`, already logged in under its own host leaf, asks
|
|
that mount's one role for a `hive-agent-<agent>` client certificate at agent
|
|
creation. Host roles never pin that CA and agent roles pin only it, so an
|
|
agent's certificate opens that agent's own `swarm/agents/<agent>/*` and
|
|
nothing else: `read` on the values there and `list` on their names.
|
|
`swarm-controller` writes that policy at agent creation and rewrites every
|
|
agent's at its own start, so a change to it reaches existing agents with the
|
|
next controller restart.
|
|
|
|
### Per-principal identities
|
|
|
|
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
|
identity and sharing one merges the identities. These five units read one path
|
|
each and each holds a leaf, a role and a policy of its own:
|
|
|
|
| unit | option pair under `deploy.bao.` | reads |
|
|
| -------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- |
|
|
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
|
|
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
|
|
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
|
|
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
|
|
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` |
|
|
|
|
The first exists **per hive**, because the path it reads carries a hive name
|
|
and every hive runs its own reader. Its subject is
|
|
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
|
composed spelling as a hive name, so nobody can name a hive into another hive's
|
|
role. The other four read a path that names a swarm service rather than a hive,
|
|
so one role each is enough and their subjects are the flat
|
|
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
|
`deploy.bao.forwarderOidcCommonName` and `deploy.bao.ciRunnerCommonName`.
|
|
|
|
On a host that mints its own PKI, `glue-bao-tls.nix` signs all five and defaults
|
|
all ten options, and there is nothing to do. Elsewhere you issue each leaf from
|
|
that CA out of band and name it here — one file per principal rather than one file
|
|
shared by five, which is the whole of what this buys. `swarm-ci-runner.nix`
|
|
refuses the build wherever the runner runs without its pair, as
|
|
`swarm-grafana.nix` does for Grafana.
|
|
|
|
On a host without the store, where the forwarder never runs, forgetting one of
|
|
the other six isn't a quiet degrade. A hand-configured remote-store hive that
|
|
names the hive's own `clientCertFile` and misses a principal's pair has the
|
|
build refused, by the module of the reader it missed, naming that pair — see
|
|
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
|
|
|
### What a missing leaf costs
|
|
|
|
Each of the three modules asserts its own pair, on the same condition: **this host
|
|
already reads the store** — it holds `deploy.bao.clientCertFile` and
|
|
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
|
|
The refusal names the two options, so an operator acts on the message without
|
|
opening the nix.
|
|
|
|
That condition fires on exactly one shape, the hand-configured remote-store hive
|
|
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
|
|
that matters on its own:
|
|
|
|
- **A host with no store identity at all** holds no `clientCertFile` either, so
|
|
the first clause is false. That's the supported no-store deployment, and also
|
|
the one an operator passes through while bringing a hive up — neither should
|
|
fail to evaluate.
|
|
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
|
|
defaults, so the third clause is false. Nothing to set, nothing to refuse.
|
|
- **A consumer that's off** — no collector, no Grafana — has no
|
|
unit to skip, so the second clause is false. Only the queue-credential reader
|
|
has no toggle of its own to check: every hive runs one for its own agents, so
|
|
reading the store at all is what asks for it.
|
|
|
|
The collector keeps a separate degrade underneath this, unchanged: a host with no
|
|
store identity runs a collector with no OIDC client secret and still receives
|
|
telemetry. The assertion doesn't touch that shape, because that host has no
|
|
`clientCertFile` either.
|
|
|
|
## How a reader reaches the store
|
|
|
|
Every reader dials the same URL — `https://bao.<swarm domain>:<port>` — and on
|
|
the host that runs the store that name resolves twice. `/etc/hosts` answers a
|
|
host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a
|
|
container with the bridge IP, where nginx holds the same port. One `BAO_ADDR`,
|
|
and the network namespace that asked decides which half of it answers.
|
|
|
|
The container's half goes through nginx and **still doesn't terminate TLS**. It
|
|
is an nginx `stream` server with `ssl_preread on`: it reads the SNI off the
|
|
ClientHello and splices the rest of the connection through byte for byte, so
|
|
openbao completes the handshake itself. A vhost would decrypt here instead, and
|
|
the client certificate — the store's whole authentication — would stop at nginx,
|
|
leaving openbao seeing one client for every hive in the swarm. That's why the
|
|
store has no vhost and gets a passthrough: not an exception to the no-vhost
|
|
rule, the one shape that keeps it.
|
|
|
|
The stream listener binds the bridge IP rather than every address, because
|
|
openbao already holds loopback on that port in the same network namespace and a
|
|
wildcard bind would fail with `EADDRINUSE` — taking nginx, and with it every
|
|
hive domain behind the gateway, down. `network.exposeHostPorts` opens the port
|
|
on the bridge firewall and nowhere else.
|
|
|
|
Reaching the port grants nothing by itself: openbao answers nothing without a
|
|
client certificate signed by `deploy.bao.clientCaFile` or by the store's own
|
|
agent CA. The listener reads both from `listener-client-ca.pem`, which no
|
|
cert-auth role pins. The passthrough carries
|
|
whichever certificate the reader presents, unchanged.
|
|
|
|
## The browser UI
|
|
|
|
OpenBao's built-in web UI is at `https://bao-ui.<swarm domain>/`
|
|
(`swarm.bao.ui.domain`), for members of authelia's `admins` group only. Log in
|
|
with the **OIDC** tab, or with a bao token under **Other**. The gateway vhost
|
|
checks the session with authelia and proxies to an nginx inside the store's
|
|
container on
|
|
`127.0.0.1:<deploy.bao.uiProxyPort>`. That nginx forwards `/ui/` and `/v1/` to
|
|
a second openbao listener on `127.0.0.1:<deploy.bao.uiPort>`, answers 403 on
|
|
the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on
|
|
everything else. ⚠️ That listener asks for **no client certificate**, because
|
|
a browser has none to present, so on this one door a bao token is the whole
|
|
credential. Anything on the store's host that can dial loopback, and any
|
|
`admins` session through the vhost, needs only a token to use the API. Unseal
|
|
from the host's `bao` CLI; the UI's unseal form gets a 403. On a self-signed
|
|
gateway where the UI is the only swarm name its host fronts, that host's nginx
|
|
doesn't wait for the store: it serves the hive leaf, or on a host that runs no
|
|
hive a throwaway self-signed certificate, on the UI's name (a browser warning)
|
|
until the unsealed store issues the services leaf, so the
|
|
stream passthrough readers use on that host comes up with the store sealed.
|
|
|
|
### OIDC login
|
|
|
|
The OIDC tab logs in through authelia as the client `swarm-bao-ui`
|
|
(`swarm.bao.ui.oidc.clientId`). An `admins` member gets a token under
|
|
`swarm-operator-viewer`: `list` and `read` on `secret/metadata/*`. That shows
|
|
the key tree and each key's versions and timestamps, and no value: the store
|
|
refuses every `secret/data/` read. Anything more needs a token.
|
|
|
|
| piece | written by |
|
|
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
|
| authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs |
|
|
| its secret at `swarm/services/swarm-bao-ui/oidc/client` | `swarm-secret-publish`, like every other service's |
|
|
| the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token |
|
|
| `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config |
|
|
|
|
The granter holds no `sys/auth`, so enabling the mount stays a bootstrap-token
|
|
step. A store set up before the mount existed needs the
|
|
[one-time granter step](../getting-started/setup.md) again: it re-writes
|
|
`bao-bootstrap`, which covers `sys/auth/oidc`, and the granter's own
|
|
policy, which covers `auth/oidc/`. Until then
|
|
`swarm-bao-operator-viewer-policy` writes the viewer policy, logs the step, and
|
|
exits 0, and the UI offers token login only. `swarm-bao-granter-role` re-runs
|
|
it each time the step succeeds, so OIDC login needs no further restart.
|
|
|
|
## The constraint that decides where the root lives
|
|
|
|
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
|
|
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
|
|
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
|
|
That role can't issue a certificate for a swarm service; the store refuses it.
|
|
|
|
The store's `swarm-services` role issues them instead, from the root inside its
|
|
`pki` mount. That root's private key never leaves the mount, and its
|
|
certificate is the only anchor the swarm has. Each hive fetches the certificate
|
|
from the store at boot, so nothing build-time can name it — the system reads
|
|
`security.pki.certificateFiles`' value at build time, which makes it the wrong
|
|
tool here. Trust reaches containers through a bind-mounted bundle assembled at
|
|
boot instead.
|
|
|
|
## Adding a secret
|
|
|
|
State three things, in the row you add above: **who mints it**, **which
|
|
container reads it**, and **what happens when they differ**. If they differ, it
|
|
needs a delivery unit, and the unit copies — it doesn't bind.
|