hyperhive/docs/swarm/ca.md

# Swarm CA

A hive's gateway serves a leaf issued by the secret store's `pki` mount,
through a role that belongs to that hive. A peer that trusts the mount's
root once validates every hive's gateway and every swarm service name,
present and future, instead of pinning to each one by hand.

That's the point of one issuer — it turns per-peer trust from O(n²)
hand-pinning into one anchor per swarm.

## The gateway leaf comes from the secret store

`hive-tls-resign.service` logs in to the store with this hive's own
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
the files the gateway copies from. The leaf's subject is the hive domain,
and its SANs are the hive domain and `*.<hive domain>`.

**Each hive's role covers that hive alone.** Its `allowed_domains` is the
hive's domain and `*.<hive domain>`, read from
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
IP SANs and no `localhost`. A request for any other name is a refusal from
the store naming that name. The hive's policy grants `update` on that one
issue path and nothing else. `swarm-bao-hive-leaf-policy.service`, on the
store's host, writes the role, the policy and the cert-auth role for every
hive in `swarm.hives`.

**Every hive needs its own store identity.** It's a client certificate
from the store's own client CA, with the subject
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
`baoHiveIdentityKeyFile`:

- On the host running the store, `swarm-bao-pki.service` mints it at
  `/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
- On every other host, copy a leaf with that subject from the store's host
  and set both options, before the host's next deploy.

Without it, `hive-tls-resign.service` fails with a message naming both
options, and so does `swarm-services-cert.service` on a hive fronting no
swarm name. The unit checks the identity before the leaf, so it fails even while a
valid leaf keeps the gateway up. The identity is the hive's alone:
it issues this hive's leaf and fetches the store's root, and it's never the
identity a swarm service uses.

**When it issues.** The unit runs at boot, on every deploy, and from
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
the current one is missing, past half the role's 720-hour lifetime, missing
a configured name, doesn't match the key beside it, or doesn't chain to the
store's root (`swarm-services-root.pem`). The first run that reaches the
store replaces a leaf that doesn't chain to that root, whoever signed it.

**The gateway doesn't wait for the store.** Nothing orders the gateway after
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
current leaf in place, says so in the journal, and retries every 30 seconds
for a day. Only a hive with no leaf yet needs the store reachable and
unsealed: its gateway's certificate import fails until the first leaf
lands, and then the unit starts nginx.

nginx serves the leaf alone, without its issuer: the issuer is the
self-signed root a consumer already holds in its trust bundle.

**A gateway host that runs no hive** (`deploy.hive-controller.enable` off,
such as a dedicated swarm-services or secret-store host) has no hive leaf, no
`hive-tls-resign.service`, no hive identity and no hive-domain vhost. Its
gateway serves the swarm-services leaf as its default certificate too. Where it
fronts other swarm names it waits for `swarm-services-cert.service` at start.
Where the store's UI is its only swarm name it starts on a throwaway
self-signed certificate and switches to the services leaf once the store
issues it.

## The trust bundle

`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
the file every consumer anchors on. It holds one certificate, a copy of
`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and
the swarm-services leaf both chain to. It's the only root the swarm has. Until
the first fetch of that root lands, the bundle is empty.

⚠️ **A consumer reads its trust when it starts.** An agent or service
container that last started before the store's root reached the bundle
fails TLS to a gateway serving a store-issued leaf, until it restarts.

## The swarm's service names come from the secret store

The swarm's service names (`swarm.forge.domain`,
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
siblings of the hive domain — `forge.<swarm.domain>`, not
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
**single-label** wildcard over it, and its role allows nothing outside the hive's
domain, so it can't cover them.

The secret store's `pki` mount issues them a leaf of their own, through its
`swarm-services` role. `swarm-services-cert.service` logs in with
this host's services-issuer certificate, calls
`pki/issue/swarm-services`, and writes the result to
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
copies from. Every hive does this with its own identity, so the store's
role, not a key on the hive, decides whether a hive can serve its swarm's
names.

`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
same list on every host, populates the role's
`allowed_domains`, so asking for a name nobody configured is a refusal
from the store naming that name — not a certificate quietly issued for
it. A host's own SANs are narrower still: it asks only for the names it
fronts a vhost for (`swarm.localServiceDomains`), so the key on one
gateway can't serve a swarm service that runs behind another.

**The root's public certificate is a file, on every hive:**
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
swarm name asks for no services leaf and reads the root alone, from the store's
`pki/ca/pem`, presenting its own store identity
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
`baoHiveIdentityKeyFile`) for the handshake. Every hive must hold one:
without it, or with either file absent,
`swarm-services-cert.service` fails with a message naming both options.
On the host running
the store it's also at
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
hand a browser, and reading it needs no store login — which matters,
because every store listener but the loopback UI one demands a client
certificate, and authelia gates that one to the `admins` group, not
open to an anonymous browser fetching a trust anchor.

**The granting unit generates the root once, and never again.** It asks
the mount whether it already has an issuer (`bao list pki/issuers`)
before calling `pki/root/generate/internal`, so a rebuild or a reboot
re-asserts the role and the grant without touching the anchor. A root
that changed per boot would invalidate every certificate issued under it
and every browser taught to trust it.

**A daily timer renews the leaf.** The store's role issues it for 720
hours. `swarm-services-cert-renew.timer` runs the same script as
`swarm-services-cert.service` once a day, and that script asks the store
for a new leaf once the current one is past half that window, is
missing a configured name, or doesn't chain to the store's root
(`swarm-services-root.pem`). A rotation re-imports the leaf into the
gateway and reloads nginx. A store that can't answer — still sealed, or
out of reach — fails the unit, which leaves the current leaf in place,
and the next day's run tries again: about fifteen tries before the leaf
lapses. `systemctl
list-timers swarm-services-cert-renew` shows the next run.

## Constraints on the material

No CA private key exists on any hive. The `pki` root's key stays inside the
store's mount (`root/generate/internal`), and each hive's narrowing is its
role's `allowed_domains`, enforced by the store at issue time: a hive's
identity that leaks can get certificates for that hive's names and no
others.

No host holds a CA or its key, except the store's mTLS CA
(`glue-bao-tls.nix`), which signs only the store's server certificate and the
certificates that log in to it. At boot and on every deploy, every host
deletes `/var/lib/swarm-ca` and the `ca.pem`, `ca-previous.pem`,
`ca-key.pem`, `ca.srl` and `.swarm-ca-adopted` in
`<deploy.hive-controller.tls.stateDir>`. The swarm distributes only
certificates.

⚠️ **The deletion is irreversible.** Rolling back to a generation that still
has `swarm-ca.service` finds no root under `/var/lib/swarm-ca` and mints a
new one, unrelated to the deleted root. That's harmless: nothing issues
under it.

A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or
`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails
evaluation. Those options name the store's mTLS CA,
`/var/lib/swarm-bao-pki/ca.pem` on the host that runs the store.

## What to hand a peer

`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
block a peer operator pastes into their own config. When this hive's
gateway serves a self-signed leaf (detected by the presence of
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints the
secret store's `pki` root itself, read from
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
gateway leaf and the swarm's service names under that root, so it's the one
anchor a peer host needs.

The operator installs the root **once per swarm host**, not once per peer,
so a hive joining later needs no edit on the hives already running. A hive
whose cert already chains to a public CA has nothing to install —
`peer-config` says so instead.

## Distributing the root

The root **certificate** is a runtime file: each hive fetches it from the
store at boot, so nothing can name it at build time. That has a cost worth
naming, because it's subtle and it bites at a distance:

> **Nothing whose trust store nix assembles at build time can reference
> the store's root.** The derivation reads `security.pki.certificateFiles`
> inside itself; the root doesn't exist there.

Two consumers, and only one of them is fine:

- **`hive-tls.nix` covers agents.** It folds the root into this hive's
  `trust-bundle.pem`, hive-c0re receives that path as
  `HIVE_TLS_CA_PATH`, and the meta-flake renderer embeds that one file
  next to every agent's flake. The bundle is the runtime-to-build-time
  bridge.
- **The Matrix container crosses the same bridge**, via the shared
  `lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
  use: the bundle is bind-mounted read-only into the container, and a
  oneshot concatenates it with the system CAs before tuwunel starts.

  The consumption differs per runtime and is the part worth knowing.
  tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
  — it's not: its rustls-based TLS stack still resolves trust through the
  platform certificate store on Linux, and that store honors
  `SSL_CERT_FILE`, so the env var takes effect the same way it would for
  an OpenSSL-linked binary.

  > ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
  > `openssl-probe` uses it _instead of_ the default store, so naming
  > just the bundle would drop every public CA and break federation with
  > the wider matrix network — a much bigger outage than the one being
  > fixed. The same caveat applies to `hive-forge` (Go) for the same
  > reason.

A hive in the swarm fetches the root itself. A host outside it gets the
certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)).
One anchor per host, not one per peer: a hive joining later needs no edit
on the hives already running.