cf7d357f17
The docs, option descriptions and comments that named the swarm root, the swarm CA or the hive CA as an existing trust anchor now name the secret store's pki root or the trust bundle that holds it: - docs/swarm/README.md, secrets.md, bao.md - deploy.bao.clientCaFile's and gateway.verifiedProxyTo's descriptions - comments in hive-c0re/src/meta.rs, hive-agent/src/swarm_queue.rs, swarm-logs/src/auth.rs, swarm-controller/src/vcs_metrics.rs, hive-forge, swarm-authelia, swarm-ci-runner, glue-bao-tls, swarm-bao, swarm-grafana, swarm-otel, otel, hive-c0re, lib/hive-ca-trust.nix, hive-forge/Cargo.toml and the flake-update workflow Nothing behaves differently: every edit is a nix, Rust or TOML comment, an option description, or a test assertion message. The Rust edits change the workspace source, which this PR already rebuilds for hivectl. Refs #4588
121 lines
6.2 KiB
Markdown
121 lines
6.2 KiB
Markdown
# The swarm secret store (OpenBao)
|
|
|
|
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
|
|
credential an agent or service needs comes from it under that
|
|
principal's own certificate identity. What lives in it and who reads what:
|
|
[`secrets.md`](secrets.md). This page covers the store itself — how it
|
|
comes up, who may write its grants, and how it's sealed and reached.
|
|
|
|
The operator steps (init, the one-time granter bootstrap) are in
|
|
[`../getting-started/setup.md`](../getting-started/setup.md#1--secret-store);
|
|
this page is the why behind them.
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
## The granter
|
|
|
|
Cert auth answers a _role_, so nothing can authenticate until some role
|
|
exists. A short-lived bootstrap token breaks that cycle once, and it's the
|
|
only step that needs the root token.
|
|
|
|
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
|
|
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
|
|
covers the auth mounts and the granter's own policy and role, and nothing
|
|
else. CI fails when the unit using the token needs a path it lacks. The token
|
|
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
|
|
names for you; on a store host that isn't all-local, set it and rebuild first.
|
|
|
|
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
|
|
auth methods (and disables `approle` if an older store still has it mounted),
|
|
writes the `bao-granter` policy, and creates the `bao-granter` role, which
|
|
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
|
|
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
|
|
mounts the KV and pki engines and writes the `swarm-controller` role, and each
|
|
sibling unit writes its own principal's policy and role. Every one runs on the
|
|
host, because every API listener but the loopback UI one demands a client
|
|
certificate and the host is the side that has one.
|
|
|
|
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
|
|
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
|
|
the token file is absent, which is the steady state afterwards; the token's
|
|
TTL means a forgotten file expires rather than lingering.
|
|
|
|
After that, a new or changed `swarm-*` grant needs no operator step: the unit
|
|
that writes it changes, and the deploy restarts it. A root step comes back only
|
|
when the granter itself needs a path it lacks, such as a new mount.
|
|
|
|
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
|
|
them, not on every deploy. When a grant drifts in the store and its unit stays
|
|
the same, the next boot re-asserts it, not the next switch.
|
|
|
|
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
|
|
wrapper carries an address, a CA and a client certificate but deliberately
|
|
**no token**, so that read answers `403` whether or not the role exists. Read
|
|
the unit's journal instead.
|
|
|
|
### Residual risk
|
|
|
|
The granter is root-equivalent. It may write any `swarm-*` policy with any
|
|
content, and attach it to a role that accepts any certificate; no bao ACL can
|
|
constrain what a policy says. What bounds it:
|
|
|
|
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
|
|
module-eval pins each principal's grants. **Merging a change to that policy
|
|
text is granting it**: it takes effect on the next deploy with no bao step,
|
|
so code review is the only gate.
|
|
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
|
|
root in a `0700` directory, readable only by root units on the store's host.
|
|
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
|
|
and `controller-key.pem`, whose policy is already root-equivalent. Root on
|
|
that host gains nothing new.
|
|
- **Never copy `granter-key.pem` off the host** the way operators copy the
|
|
other leaves in that directory. That hands out root-equivalence.
|
|
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
|
|
subject. Rotate the store's CA, or have root point the `bao-granter` role at
|
|
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
|
|
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
|
|
one.
|
|
|
|
## Sealing
|
|
|
|
`services.hyperhive.deploy.bao.seal`:
|
|
|
|
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
|
|
unseals itself on every restart. `init` is the only manual step.
|
|
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
|
|
restart, with the keys `init` printed.
|
|
|
|
⚠️ **A sealed store still answers.** The container is up and the port
|
|
responds while every read times out — the failure looks like a hang, not like
|
|
a store that was never initialised.
|
|
|
|
## TLS
|
|
|
|
The store serves TLS, and on a host that deploys it you need do nothing: a
|
|
first-boot unit mints a CA of the store's own plus the leaves it signs — the
|
|
store's server certificate, this host's client certificate, and one per service
|
|
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
|
|
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
|
|
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
|
|
|
|
Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
|
certificates come from a real internal CA; the store has no opinion about
|
|
which. A host that does **not** deploy the store names the reader's three
|
|
itself, plus a pair for every principal it runs — see
|
|
[per-principal identities](secrets.md#per-principal-identities) for the list.
|
|
The operator issues those leaves out of band; they're the credentials the
|
|
store can't hand you, being what opens it.
|
|
|
|
⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
|
|
between services and the store**, a separate trust domain, because a store
|
|
that took its identity from an authority it itself distributes could never
|
|
come up before that authority.
|
|
|
|
Inside the store's container (`nixos-container root-login swarm-bao`) the
|
|
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
|
|
and its DNS name resolves to the bridge from in there: export
|
|
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
|
|
domain>`. The host's `bao` wrapper already carries the address, CA and client
|
|
certificate, so the host is the shorter path.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|