docs: sort the top-level directories into five sections by what the reader wants
Every docs directory moves under 1-getting-started, 2-handbook, 4-implementation or 5-project; the options reference (section 3) comes from the website and has no directory here. trust-boundary stays whole under the handbook for now. web-ui, swarm and tools split between handbook and implementation, and the command-line references move to 2-handbook/reference. Links, path references in code and nix, and the docs index follow the moves; no page content changes.
This commit is contained in:
iris
mara
parent
53ccbcb4b3
commit
6739bb903d
1 changed files with 736 additions and 701 deletions
152
docs/4-implementation/swarm/bao.md
Normal file
152
docs/4-implementation/swarm/bao.md
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
# The swarm secret store (OpenBao)
|
||||
|
||||
The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
|
||||
credential an agent or service needs comes from it under that
|
||||
principal's own certificate identity. What lives in it and who reads what:
|
||||
[`credentials-reference.md`](../../2-handbook/swarm/credentials-reference.md). This page covers the store itself — how it
|
||||
comes up, who may write its grants, and how it's sealed and reached.
|
||||
|
||||
The operator steps (init, the one-time granter bootstrap) are in
|
||||
[`../getting-started/setup.md`](../../1-getting-started/setup.md#1--secret-store);
|
||||
this page is the why behind them.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
## The granter
|
||||
|
||||
Cert auth answers a _role_, so nothing can authenticate until some role
|
||||
exists. A short-lived bootstrap token breaks that cycle once, and it's the
|
||||
only step that needs the root token.
|
||||
|
||||
The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
|
||||
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
|
||||
covers the auth mounts and the granter's own policy and role, and nothing
|
||||
else. CI fails when the unit using the token needs a path it lacks. The token
|
||||
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
|
||||
names for you; on a store host that isn't all-local, set it and rebuild first.
|
||||
|
||||
`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
|
||||
auth methods (and disables `approle` if an older store still has it mounted),
|
||||
writes the `bao-granter` policy, and creates the `bao-granter` role, which
|
||||
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
|
||||
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
|
||||
mounts the KV and pki engines and writes the `swarm-controller` role, and each
|
||||
sibling unit writes its own principal's policy and role. Every one runs on the
|
||||
host, because every API listener but the loopback UI one demands a client
|
||||
certificate and the host is the side that has one.
|
||||
|
||||
Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
|
||||
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
|
||||
the token file is absent, which is the steady state afterwards; the token's
|
||||
TTL means a forgotten file expires rather than lingering.
|
||||
|
||||
A sealed or unreachable store, or one whose listing fails with something other
|
||||
than a refusal (a node still becoming active), makes the unit retry every
|
||||
30 s with the store's error in the log. A store that refuses the token (expired,
|
||||
revoked, or missing the `bao-bootstrap` policy)
|
||||
makes it exit 4 and stay `failed` with the one-time step in its log; mint a
|
||||
fresh token or remove the file, then restart it. If the granter's own
|
||||
certificate login already works, the unit logs that the bootstrap token isn't
|
||||
needed, deletes the dead token file and exits 0, so a stale file is harmless
|
||||
and the unit runs once. A successful run deletes the token file too.
|
||||
|
||||
After that, a new or changed `swarm-*` grant needs no operator step: the unit
|
||||
that writes it changes, and the deploy restarts it. A root step comes back only
|
||||
when the granter itself needs a path it lacks, such as a new mount.
|
||||
|
||||
⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
|
||||
them, not on every deploy. When a grant drifts in the store and its unit stays
|
||||
the same, the next boot re-asserts it, not the next switch.
|
||||
|
||||
⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
|
||||
wrapper carries an address, a CA and a client certificate but deliberately
|
||||
**no token**, so that read answers `403` whether or not the role exists. Read
|
||||
the unit's journal instead.
|
||||
|
||||
### Residual risk
|
||||
|
||||
The granter is root-equivalent. It may write any `swarm-*` policy with any
|
||||
content, and attach it to a role that accepts any certificate; no bao ACL can
|
||||
constrain what a policy says. What bounds it:
|
||||
|
||||
- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
|
||||
module-eval pins each principal's grants. **Merging a change to that policy
|
||||
text is granting it**: it takes effect on the next deploy with no bao step,
|
||||
so code review is the only gate.
|
||||
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
|
||||
root in a `0700` directory, readable only by root units on the store's host.
|
||||
That host already holds `ca-key.pem`, which can mint a leaf with any subject,
|
||||
and `controller-key.pem`, whose policy is already root-equivalent. Root on
|
||||
that host gains nothing new.
|
||||
- **Never copy `granter-key.pem` off the host** the way operators copy the
|
||||
other leaves in that directory. That hands out root-equivalence.
|
||||
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
|
||||
subject. Rotate the store's CA, or have root point the `bao-granter` role at
|
||||
a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
|
||||
restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
|
||||
one.
|
||||
|
||||
## Sealing
|
||||
|
||||
`services.hyperhive.deploy.bao.seal`:
|
||||
|
||||
- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
|
||||
unseals itself on every restart. `init` is the only manual step.
|
||||
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
|
||||
restart, with the keys `init` printed.
|
||||
|
||||
⚠️ **A sealed store still answers.** The container is up and the port
|
||||
responds while every read times out — the failure looks like a stall, not like
|
||||
a store that was never initialised.
|
||||
|
||||
## TLS
|
||||
|
||||
The store serves TLS, and on a host that deploys it you need do nothing:
|
||||
`swarm-bao-pki` mints a CA of the store's own plus the leaves it signs — the
|
||||
store's server certificate, this host's client certificate, and one per service
|
||||
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
|
||||
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
|
||||
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.
|
||||
It runs at every boot and every deploy, and before each start of a container
|
||||
that loads one of its leaves, and signs only the files that are missing, so a
|
||||
leaf a deploy adds, or a service a deploy enables, has its file on that deploy.
|
||||
The store's readers and granting units on this host order after it and want it,
|
||||
but none requires it: when a run fails, each of them still starts, and one
|
||||
whose leaf is missing fails its login into its own retry. It has no start
|
||||
limit, since a mass container start pulls it in once for each unit it starts.
|
||||
|
||||
Those are `mkDefault`s, so naming your own paths wins. Do that when your
|
||||
certificates come from a real internal CA; the store has no opinion about
|
||||
which. A host that does **not** deploy the store names the reader's three
|
||||
itself, plus a pair for every principal it runs — see
|
||||
[per-principal identities](../../2-handbook/swarm/credentials-reference.md#store-identities-one-per-principal) for the list.
|
||||
They're the credentials the store can't hand you, being what opens it, so the
|
||||
store's host signs them and the operator copies them to the host that needs
|
||||
them and names them in config. The service principals' leaves are at the top of
|
||||
`/var/lib/swarm-bao-pki/`, named after the principal.
|
||||
|
||||
For every hive in `swarm.hives` other than its own, the store's host signs
|
||||
that hive's two leaves into `/var/lib/swarm-bao-pki/hives/<hive>/`:
|
||||
|
||||
| file | subject | name it in |
|
||||
| ------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------- |
|
||||
| `client.pem`, `client-key.pem` | `<hive>` | `deploy.bao.clientCertFile`, `.clientKeyFile` |
|
||||
| `hive-leaf.pem`, `hive-leaf-key.pem` | `<deploy.bao.hiveLeafCommonNamePrefix>-<hive>` | `deploy.hive-controller.tls.baoHiveIdentityCertFile`, `.baoHiveIdentityKeyFile` |
|
||||
|
||||
Copy `/var/lib/swarm-bao-pki/ca.pem` with them and name it in
|
||||
`deploy.bao.serverCaFile`. A hive added to `swarm.hives` gets its directory on
|
||||
the store host's next deploy; a leaf already there is never re-signed.
|
||||
|
||||
⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
|
||||
between services and the store**, a separate trust domain, because a store
|
||||
that took its identity from an authority it itself distributes could never
|
||||
come up before that authority.
|
||||
|
||||
Inside the store's container (`nixos-container root-login swarm-bao`) the
|
||||
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
|
||||
and its DNS name resolves to the bridge from in there: export
|
||||
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
|
||||
domain>`. The host's `bao` wrapper already carries the address, CA and client
|
||||
certificate, so the host is the shorter path.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
Loading…
Reference in a new issue