b5053d9a80
`swarm-bao-pki` was a `RemainAfterExit` oneshot. Once active, nothing re-ran it: a switch restarts it only when its definition changes, and a `Requires=`/`Wants=` on an active unit is satisfied without running it. On muede-lpt2 the 10:57Z switch that added the builder line did change the unit, so switch-to-configuration stopped it in its stop phase. A start of `container@swarm-bao` 4 s later (before the daemon-reload at 10:57:35Z) pulled it back in through `swarm-bao-certs` with the OLD definition, which had no builder line: 48 ms, nothing missing. The start phase then found it active and did nothing. The 20:14Z switch that enabled the builder left the unit unchanged, so it was never re-run, and `container@swarm-builder` crash-looped on the absent `builder.pem`. The unit now goes inactive after each run. Every switch starts `multi-user.target`, which runs it again post-reload with the current definition, and every unit that wants or requires it runs it on its own start. Each line is idempotent on presence, so an existing leaf is never re-signed. The three consumers started from `machines.target` rather than `multi-user.target` also want it, since `After=` alone does not order against a job in another transaction: the builder's and the review classifier's containers (nspawn `--load-credential`) and `swarm-nats-auth-secrets` (pulled in by the queue's container). Fixes #5284
139 lines
7.8 KiB
Markdown
139 lines
7.8 KiB
Markdown
# The swarm secret store (OpenBao)
|
|
|
|
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
|
|
credential an agent or service needs comes from it under that
|
|
principal's own certificate identity. What lives in it and who reads what:
|
|
[`secrets.md`](secrets.md). This page covers the store itself — how it
|
|
comes up, who may write its grants, and how it's sealed and reached.
|
|
|
|
The operator steps (init, the one-time granter bootstrap) are in
|
|
[`../getting-started/setup.md`](../getting-started/setup.md#1--secret-store);
|
|
this page is the why behind them.
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
## The granter
|
|
|
|
Cert auth answers a _role_, so nothing can authenticate until some role
|
|
exists. A short-lived bootstrap token breaks that cycle once, and it's the
|
|
only step that needs the root token.
|
|
|
|
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
|
|
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
|
|
covers the auth mounts and the granter's own policy and role, and nothing
|
|
else. CI fails when the unit using the token needs a path it lacks. The token
|
|
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
|
|
names for you; on a store host that isn't all-local, set it and rebuild first.
|
|
|
|
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
|
|
auth methods (and disables `approle` if an older store still has it mounted),
|
|
writes the `bao-granter` policy, and creates the `bao-granter` role, which
|
|
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
|
|
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
|
|
mounts the KV and pki engines and writes the `swarm-controller` role, and each
|
|
sibling unit writes its own principal's policy and role. Every one runs on the
|
|
host, because every API listener but the loopback UI one demands a client
|
|
certificate and the host is the side that has one.
|
|
|
|
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
|
|
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
|
|
the token file is absent, which is the steady state afterwards; the token's
|
|
TTL means a forgotten file expires rather than lingering.
|
|
|
|
After that, a new or changed `swarm-*` grant needs no operator step: the unit
|
|
that writes it changes, and the deploy restarts it. A root step comes back only
|
|
when the granter itself needs a path it lacks, such as a new mount.
|
|
|
|
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
|
|
them, not on every deploy. When a grant drifts in the store and its unit stays
|
|
the same, the next boot re-asserts it, not the next switch.
|
|
|
|
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
|
|
wrapper carries an address, a CA and a client certificate but deliberately
|
|
**no token**, so that read answers `403` whether or not the role exists. Read
|
|
the unit's journal instead.
|
|
|
|
### Residual risk
|
|
|
|
The granter is root-equivalent. It may write any `swarm-*` policy with any
|
|
content, and attach it to a role that accepts any certificate; no bao ACL can
|
|
constrain what a policy says. What bounds it:
|
|
|
|
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
|
|
module-eval pins each principal's grants. **Merging a change to that policy
|
|
text is granting it**: it takes effect on the next deploy with no bao step,
|
|
so code review is the only gate.
|
|
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
|
|
root in a `0700` directory, readable only by root units on the store's host.
|
|
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
|
|
and `controller-key.pem`, whose policy is already root-equivalent. Root on
|
|
that host gains nothing new.
|
|
- **Never copy `granter-key.pem` off the host** the way operators copy the
|
|
other leaves in that directory. That hands out root-equivalence.
|
|
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
|
|
subject. Rotate the store's CA, or have root point the `bao-granter` role at
|
|
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
|
|
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
|
|
one.
|
|
|
|
## Sealing
|
|
|
|
`services.hyperhive.deploy.bao.seal`:
|
|
|
|
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
|
|
unseals itself on every restart. `init` is the only manual step.
|
|
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
|
|
restart, with the keys `init` printed.
|
|
|
|
⚠️ **A sealed store still answers.** The container is up and the port
|
|
responds while every read times out — the failure looks like a hang, not like
|
|
a store that was never initialised.
|
|
|
|
## TLS
|
|
|
|
The store serves TLS, and on a host that deploys it you need do nothing:
|
|
`swarm-bao-pki` mints a CA of the store's own plus the leaves it signs — the
|
|
store's server certificate, this host's client certificate, and one per service
|
|
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
|
|
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
|
|
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
|
|
It runs at every boot and every deploy, and before each start of a container
|
|
that loads one of its leaves, and signs only the files that are missing, so a
|
|
leaf a deploy adds, or a service a deploy enables, has its file on that deploy.
|
|
|
|
Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
|
certificates come from a real internal CA; the store has no opinion about
|
|
which. A host that does **not** deploy the store names the reader's three
|
|
itself, plus a pair for every principal it runs — see
|
|
[per-principal identities](secrets.md#per-principal-identities) for the list.
|
|
They're the credentials the store can't hand you, being what opens it, so the
|
|
store's host signs them and the operator copies them to the host that needs
|
|
them and names them in config. The service principals' leaves are at the top of
|
|
`/var/lib/swarm-bao-pki/`, named after the principal.
|
|
|
|
For every hive in `swarm.hives` other than its own, the store's host signs
|
|
that hive's three leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
|
|
|
|
| file | subject | name it in |
|
|
| ---------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------- |
|
|
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
|
|
| `queue-agent.pem`, `queue-agent-key.pem` | `<deploy.bao.queueAgentCommonNamePrefix>-<hive>` | `deploy.bao.queueAgentClientCertFile`, `.queueAgentClientKeyFile` |
|
|
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
|
|
|
|
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
|
|
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on
|
|
the store host's next deploy; a leaf already there is never re-signed.
|
|
|
|
⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
|
|
between services and the store**, a separate trust domain, because a store
|
|
that took its identity from an authority it itself distributes could never
|
|
come up before that authority.
|
|
|
|
Inside the store's container (`nixos-container root-login swarm-bao`) the
|
|
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
|
|
and its DNS name resolves to the bridge from in there: export
|
|
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
|
|
domain>`. The host's `bao` wrapper already carries the address, CA and client
|
|
certificate, so the host is the shorter path.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|