hyperhive/docs/swarm/bao.md

# The swarm secret store (OpenBao)

The swarm runs one OpenBao store, `swarm-bao`. Every swarm-level
credential an agent or service needs comes from it under that
principal's own certificate identity. What lives in it and who reads what:
[`secrets.md`](secrets.md). This page covers the store itself — how it
comes up, who may write its grants, and how it's sealed and reached.

The operator steps (init, the one-time granter bootstrap) are in
[`../getting-started/setup.md`](../getting-started/setup.md#1--secret-store);
this page is the why behind them.

<!-- vale write-good.Passive = NO -->

## The granter

Cert auth answers a _role_, so nothing can authenticate until some role
exists. A short-lived bootstrap token breaks that cycle once, and it's the
only step that needs the root token.

The policy that token carries is `nix/host-modules/bao-bootstrap-policy.hcl`,
shipped on the store's host at `/etc/hyperhive/bao-bootstrap-policy.hcl`. It
covers the auth mounts and the granter's own policy and role, and nothing
else. CI fails when the unit using the token needs a path it lacks. The token
file is `services.hyperhive.deploy.bao.bootstrapTokenFile`, which all-local
names for you; on a store host that isn't all-local, set it and rebuild first.

`swarm-bao-granter-role` runs **on the host**. It enables the cert and oidc
auth methods (and disables `approle` if an older store still has it mounted),
writes the `bao-granter` policy, and creates the `bao-granter` role, which
accepts the leaf `/var/lib/swarm-bao-pki/granter.pem`. Every
`swarm-bao-*-policy` unit then logs in with that leaf. The controller's unit
mounts the KV and pki engines and writes the `swarm-controller` role, and each
sibling unit writes its own principal's policy and role. Every one runs on the
host, because every API listener but the loopback UI one demands a client
certificate and the host is the side that has one.

Until the granter exists, each `swarm-bao-*-policy` unit **fails** and logs
the bootstrap commands. It never skips. `swarm-bao-granter-role` skips while
the token file is absent, which is the steady state afterwards; the token's
TTL means a forgotten file expires rather than lingering.

After that, a new or changed `swarm-*` grant needs no operator step: the unit
that writes it changes, and the deploy restarts it. A root step comes back only
when the granter itself needs a path it lacks, such as a new mount.

⚠️ The granting units re-run on **boot** and whenever a deploy **changes**
them, not on every deploy. When a grant drifts in the store and its unit stays
the same, the next boot re-asserts it, not the next switch.

⚠️ Don't reach for `bao read auth/cert/…` to check a role. The host's `bao`
wrapper carries an address, a CA and a client certificate but deliberately
**no token**, so that read answers `403` whether or not the role exists. Read
the unit's journal instead.

### Residual risk

The granter is root-equivalent. It may write any `swarm-*` policy, and bao's
`default` policy, with any content, and attach a policy to a role that accepts
any certificate; no bao ACL can constrain what a policy says. What bounds it:

- `nix/host-modules/swarm-bao.nix` renders every policy it writes, and
  module-eval pins each principal's grants. **Merging a change to that policy
  text is granting it**: it takes effect on the next deploy with no bao step,
  so code review is the only gate.
- Its key sits permanently at `/var/lib/swarm-bao-pki/granter-key.pem`, `0600`
  root in a `0700` directory, readable only by root units on the store's host.
  That host already holds `ca-key.pem`, which can mint a leaf with any subject,
  and `controller-key.pem`, whose policy is already root-equivalent. Root on
  that host gains nothing new.
- **Never copy `granter-key.pem` off the host** the way operators copy the
  other leaves in that directory. That hands out root-equivalence.
- Nothing revokes a stolen leaf on its own: the role trusts the CA plus the
  subject. Rotate the store's CA, or have root point the `bao-granter` role at
  a new `deploy.bao.granterCommonName`. Deleting `granter{,-key}.pem` and
  restarting `swarm-bao-pki` mints a new leaf, but doesn't invalidate the old
  one.

### What every token may read

Anything under `secret/swarm/public/` is readable by every token the store
issues. `swarm-bao-public-read-policy` puts that one read into bao's `default`
policy, which every token carries, between two marker lines. It reads the
policy, replaces only its own block, and writes the result back when it
differs, so the store's own stanzas stay as the store wrote them. It never
writes a policy it read back empty. The callout xkey's public half lives
there ([`secrets.md`](secrets.md#the-queues-callout-xkey)).

Writing `default` is a grant on the granter's own policy, so on a store
whose granter predates it, the unit says so and exits 0 until the bootstrap
step re-runs. Until then only principals whose policy names a `swarm/public/`
path read it; the queue's responder is one, so the queue doesn't wait on this.

## Sealing

`services.hyperhive.deploy.bao.seal`:

- **`pkcs11`** (the default) — binds the key to the host's TPM, and the store
  unseals itself on every restart. `init` is the only manual step.
- **`shamir`** — no TPM, so run `bao operator unseal` again after every
  restart, with the keys `init` printed.

⚠️ **A sealed store still answers.** The container is up and the port
responds while every read times out — the failure looks like a hang, not like
a store that was never initialised.

## TLS

The store serves TLS, and on a host that deploys it you need do nothing: a
first-boot unit mints a CA of the store's own plus the leaves it signs — the
store's server certificate, this host's client certificate, and one per service
principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and
`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and
`.serverCaFile` at the reader's, and each principal's own pair at its own leaf.

Those are `mkDefault`s, so naming your own paths wins. Do that when your
certificates come from a real internal CA; the store has no opinion about
which. A host that does **not** deploy the store names the reader's three
itself, plus a pair for every principal it runs — see
[per-principal identities](secrets.md#per-principal-identities) for the list.
The operator issues those leaves out of band; they're the credentials the
store can't hand you, being what opens it.

⚠️ Not the gateway's HTTPS certificates and not the store's `pki` root — this is **mTLS
between services and the store**, a separate trust domain, because a store
that took its identity from an authority it itself distributes could never
come up before that authority.

Inside the store's container (`nixos-container root-login swarm-bao`) the
`bao` CLI needs two extra pieces, because the certificate carries no IP SAN
and its DNS name resolves to the bridge from in there: export
`BAO_ADDR=https://127.0.0.1:8200` alongside `BAO_TLS_SERVER_NAME=bao.<swarm
domain>`. The host's `bao` wrapper already carries the address, CA and client
certificate, so the host is the shorter path.

<!-- vale write-good.Passive = YES -->