9a65643769
swarm-bao-granter-role exits 4 when the store is unsealed but refuses the bootstrap token, and Restart=on-failure retried that every 30s for as long as the token file existed. RestartPreventExitStatus=4 makes that permanent failure leave the unit failed once. Exit 1 (store sealed or unreachable) keeps retrying. On deploy: the unit goes red once instead of looping every 30s. The stale token on muede-lpt2 still needs the operator step (remove it or mint a fresh one); this does not clear it.
148 lines
8.4 KiB
Markdown
148 lines
8.4 KiB
Markdown
# The swarm secret store (OpenBao)
|
|
|
|
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
|
|
credential an agent or service needs comes from it under that
|
|
principal's own certificate identity. What lives in it and who reads what:
|
|
[`secrets.md`](secrets.md). This page covers the store itself — how it
|
|
comes up, who may write its grants, and how it's sealed and reached.
|
|
|
|
The operator steps (init, the one-time granter bootstrap) are in
|
|
[`../getting-started/setup.md`](../getting-started/setup.md#1--secret-store);
|
|
this page is the why behind them.
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
## The granter
|
|
|
|
Cert auth answers a _role_, so nothing can authenticate until some role
|
|
exists. A short-lived bootstrap token breaks that cycle once, and it's the
|
|
only step that needs the root token.
|
|
|
|
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
|
|
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
|
|
covers the auth mounts and the granter's own policy and role, and nothing
|
|
else. CI fails when the unit using the token needs a path it lacks. The token
|
|
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
|
|
names for you; on a store host that isn't all-local, set it and rebuild first.
|
|
|
|
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
|
|
auth methods (and disables `approle` if an older store still has it mounted),
|
|
writes the `bao-granter` policy, and creates the `bao-granter` role, which
|
|
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
|
|
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
|
|
mounts the KV and pki engines and writes the `swarm-controller` role, and each
|
|
sibling unit writes its own principal's policy and role. Every one runs on the
|
|
host, because every API listener but the loopback UI one demands a client
|
|
certificate and the host is the side that has one.
|
|
|
|
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
|
|
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
|
|
the token file is absent, which is the steady state afterwards; the token's
|
|
TTL means a forgotten file expires rather than lingering.
|
|
|
|
A sealed or unreachable store makes the unit retry every 30 s. A store that
|
|
refuses the token (expired, revoked, or missing the `bao-bootstrap` policy)
|
|
makes it exit 4 and stay `failed` with the one-time step in its log; mint a
|
|
fresh token or remove the file, then restart it.
|
|
|
|
After that, a new or changed `swarm-*` grant needs no operator step: the unit
|
|
that writes it changes, and the deploy restarts it. A root step comes back only
|
|
when the granter itself needs a path it lacks, such as a new mount.
|
|
|
|
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
|
|
them, not on every deploy. When a grant drifts in the store and its unit stays
|
|
the same, the next boot re-asserts it, not the next switch.
|
|
|
|
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
|
|
wrapper carries an address, a CA and a client certificate but deliberately
|
|
**no token**, so that read answers `403` whether or not the role exists. Read
|
|
the unit's journal instead.
|
|
|
|
### Residual risk
|
|
|
|
The granter is root-equivalent. It may write any `swarm-*` policy with any
|
|
content, and attach it to a role that accepts any certificate; no bao ACL can
|
|
constrain what a policy says. What bounds it:
|
|
|
|
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
|
|
module-eval pins each principal's grants. **Merging a change to that policy
|
|
text is granting it**: it takes effect on the next deploy with no bao step,
|
|
so code review is the only gate.
|
|
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
|
|
root in a `0700` directory, readable only by root units on the store's host.
|
|
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
|
|
and `controller-key.pem`, whose policy is already root-equivalent. Root on
|
|
that host gains nothing new.
|
|
- **Never copy `granter-key.pem` off the host** the way operators copy the
|
|
other leaves in that directory. That hands out root-equivalence.
|
|
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
|
|
subject. Rotate the store's CA, or have root point the `bao-granter` role at
|
|
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
|
|
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
|
|
one.
|
|
|
|
## Sealing
|
|
|
|
`services.hyperhive.deploy.bao.seal`:
|
|
|
|
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
|
|
unseals itself on every restart. `init` is the only manual step.
|
|
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
|
|
restart, with the keys `init` printed.
|
|
|
|
⚠️ **A sealed store still answers.** The container is up and the port
|
|
responds while every read times out — the failure looks like a hang, not like
|
|
a store that was never initialised.
|
|
|
|
## TLS
|
|
|
|
The store serves TLS, and on a host that deploys it you need do nothing:
|
|
`swarm-bao-pki` mints a CA of the store's own plus the leaves it signs — the
|
|
store's server certificate, this host's client certificate, and one per service
|
|
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
|
|
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
|
|
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
|
|
It runs at every boot and every deploy, and before each start of a container
|
|
that loads one of its leaves, and signs only the files that are missing, so a
|
|
leaf a deploy adds, or a service a deploy enables, has its file on that deploy.
|
|
The store's readers and granting units on this host order after it and want it,
|
|
but none requires it: when a run fails, each of them still starts, and one
|
|
whose leaf is missing fails its login into its own retry. It has no start
|
|
limit, since a mass container start pulls it in once for each unit it starts.
|
|
|
|
Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
|
certificates come from a real internal CA; the store has no opinion about
|
|
which. A host that does **not** deploy the store names the reader's three
|
|
itself, plus a pair for every principal it runs — see
|
|
[per-principal identities](secrets.md#per-principal-identities) for the list.
|
|
They're the credentials the store can't hand you, being what opens it, so the
|
|
store's host signs them and the operator copies them to the host that needs
|
|
them and names them in config. The service principals' leaves are at the top of
|
|
`/var/lib/swarm-bao-pki/`, named after the principal.
|
|
|
|
For every hive in `swarm.hives` other than its own, the store's host signs
|
|
that hive's three leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
|
|
|
|
| file | subject | name it in |
|
|
| ---------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------- |
|
|
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
|
|
| `queue-agent.pem`, `queue-agent-key.pem` | `<deploy.bao.queueAgentCommonNamePrefix>-<hive>` | `deploy.bao.queueAgentClientCertFile`, `.queueAgentClientKeyFile` |
|
|
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
|
|
|
|
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
|
|
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on
|
|
the store host's next deploy; a leaf already there is never re-signed.
|
|
|
|
⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
|
|
between services and the store**, a separate trust domain, because a store
|
|
that took its identity from an authority it itself distributes could never
|
|
come up before that authority.
|
|
|
|
Inside the store's container (`nixos-container root-login swarm-bao`) the
|
|
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
|
|
and its DNS name resolves to the bridge from in there: export
|
|
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
|
|
domain>`. The host's `bao` wrapper already carries the address, CA and client
|
|
certificate, so the host is the shorter path.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|