docs: setup.md as a short all-local checklist; bao internals move to swarm/bao.md
This commit is contained in:
müde
parent
3a0f74c1e7
commit
a7991c9242
3 changed files with 224 additions and 289 deletions
134
docs/swarm/bao.md
Normal file
134
docs/swarm/bao.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# The swarm secret store (OpenBao)
|
||||
|
||||
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
|
||||
credential an agent or service needs is fetched from it under that
|
||||
principal's own certificate identity. What lives in it and who reads what:
|
||||
[`secrets.md`](secrets.md). This page covers the store itself — how it
|
||||
comes up, who may write its grants, and how it's sealed and reached.
|
||||
|
||||
The operator steps (init, the one-time granter bootstrap) are in
|
||||
[`../getting-started/setup.md`](../getting-started/setup.md#1--secret-store);
|
||||
this page is the why behind them.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
## The granter
|
||||
|
||||
Cert auth answers a _role_, so nothing can authenticate until some role
|
||||
exists. A short-lived bootstrap token breaks that cycle once, and it's the
|
||||
only step that needs the root token.
|
||||
|
||||
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
|
||||
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
|
||||
covers the auth mounts and the granter's own policy and role, and nothing
|
||||
else. CI fails when the unit using the token needs a path it lacks. The token
|
||||
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
|
||||
names for you; on a store host that isn't all-local, set it and rebuild first.
|
||||
|
||||
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
|
||||
auth methods (and disables `approle` if an older store still has it mounted),
|
||||
writes the `bao-granter` policy, and creates the `bao-granter` role, which
|
||||
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
|
||||
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
|
||||
mounts the KV and pki engines and writes the `swarm-controller` role, and each
|
||||
sibling unit writes its own principal's policy and role. Every one runs on the
|
||||
host, because every API listener but the loopback UI one demands a client
|
||||
certificate and the host is the side that has one.
|
||||
|
||||
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
|
||||
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
|
||||
the token file is absent, which is the steady state afterwards; the token's
|
||||
TTL means a forgotten file expires rather than lingering.
|
||||
|
||||
After that, a new or changed `swarm-*` grant needs no operator step: the unit
|
||||
that writes it changes, and the deploy restarts it. A root step comes back only
|
||||
when the granter itself needs a path it lacks, such as a new mount.
|
||||
|
||||
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
|
||||
them, not on every deploy. When a grant drifts in the store and its unit stays
|
||||
the same, the next boot re-asserts it, not the next switch.
|
||||
|
||||
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
|
||||
wrapper carries an address, a CA and a client certificate but deliberately
|
||||
**no token**, so that read answers `403` whether or not the role exists. Read
|
||||
the unit's journal instead.
|
||||
|
||||
<details><summary>Upgrading a swarm set up with the older swarm-bootstrap policy</summary>
|
||||
|
||||
A store set up before the granter existed has every grant, but no `bao-granter`
|
||||
policy or role. After the deploy that introduces it, each `swarm-bao-*-policy`
|
||||
unit fails and logs the one-time step. Run the setup step as it stands. The
|
||||
old policy can go, with the root token again:
|
||||
|
||||
```bash
|
||||
bao policy delete swarm-bootstrap
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Residual risk
|
||||
|
||||
The granter is root-equivalent. It may write any `swarm-*` policy with any
|
||||
content, and attach it to a role that accepts any certificate; no bao ACL can
|
||||
constrain what a policy says. What bounds it:
|
||||
|
||||
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
|
||||
module-eval pins each principal's grants. **Merging a change to that policy
|
||||
text is granting it**: it takes effect on the next deploy with no bao step,
|
||||
so code review is the only gate.
|
||||
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
|
||||
root in a `0700` directory, readable only by root units on the store's host.
|
||||
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
|
||||
and `controller-key.pem`, whose policy is already root-equivalent. Root on
|
||||
that host gains nothing new.
|
||||
- **Never copy `granter-key.pem` off the host** the way operators copy the
|
||||
other leaves in that directory. That hands out root-equivalence.
|
||||
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
|
||||
subject. Rotate the store's CA, or have root point the `bao-granter` role at
|
||||
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
|
||||
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
|
||||
one.
|
||||
|
||||
## Sealing
|
||||
|
||||
`services.hyperhive.deploy.bao.seal`:
|
||||
|
||||
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
|
||||
unseals itself on every restart. `init` is the only manual step.
|
||||
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
|
||||
restart, with the keys `init` printed.
|
||||
|
||||
⚠️ **A sealed store still answers.** The container is up and the port
|
||||
responds while every read times out — the failure looks like a hang, not like
|
||||
a store that was never initialised.
|
||||
|
||||
## TLS
|
||||
|
||||
The store serves TLS, and on a host that deploys it you need do nothing: a
|
||||
first-boot unit mints a CA of the store's own plus the leaves it signs — the
|
||||
store's server certificate, this host's client certificate, and one per service
|
||||
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
|
||||
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
|
||||
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
|
||||
|
||||
Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
||||
certificates come from a real internal CA; the store has no opinion about
|
||||
which. A host that does **not** deploy the store names the reader's three
|
||||
itself, plus a pair for every principal it runs — see
|
||||
[per-principal identities](secrets.md#per-principal-identities) for the list.
|
||||
The operator issues those leaves out of band; they're the credentials the
|
||||
store can't hand you, being what opens it.
|
||||
|
||||
⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is **mTLS
|
||||
between services and the store**, a separate trust domain, because a store
|
||||
that took its identity from an authority it itself distributes could never
|
||||
come up before that authority.
|
||||
|
||||
Inside the store's container (`nixos-container root-login swarm-bao`) the
|
||||
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
|
||||
and its DNS name resolves to the bridge from in there: export
|
||||
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
|
||||
domain>`. The host's `bao` wrapper already carries the address, CA and client
|
||||
certificate, so the host is the shorter path.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
Loading…
Reference in a new issue