hyperhive/docs/swarm/ca.md

Swarm CA

A hive's gateway serves a leaf issued by the secret store's pki mount, through a role that belongs to that hive. A peer that trusts the mount's root once validates every hive's gateway and every swarm service name, present and future, instead of pinning to each one by hand.

That's the point of one issuer — it turns per-peer trust from O(n²) hand-pinning into one anchor per swarm.

The gateway leaf comes from the secret store

hive-tls-resign.service logs in to the store with this hive's own identity, calls pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName> (swarm-hive-leaf-<hiveName> by default), and writes the result to <deploy.hive-controller.tls.stateDir>/gateway.pem and gateway-key.pem, the files the gateway copies from. The leaf's subject is the hive domain, and its SANs are the hive domain and *.<hive domain>.

Each hive's role covers that hive alone. Its allowed_domains is the hive's domain and *.<hive domain>, read from services.hyperhive.swarm.hives.<name>.domain on the store's host, with no IP SANs and no localhost. A request for any other name is a refusal from the store naming that name. The hive's policy grants update on that one issue path and nothing else. swarm-bao-hive-leaf-policy.service, on the store's host, writes the role, the policy and the cert-auth role for every hive in swarm.hives.

Every hive needs its own store identity. It's a client certificate from the store's own client CA, with the subject <deploy.bao.hiveLeafCommonNamePrefix>-<hiveName> (swarm-bao-hive-leaf-<hiveName> by default), and it's named by services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile and baoHiveIdentityKeyFile:

• On the host running the store, swarm-bao-pki.service mints it at /var/lib/swarm-bao-pki/hive-leaf.pem and the two options default to it.
• On every other host, copy a leaf with that subject from the store's host and set both options, before the host's next deploy.

Without it, hive-tls-resign.service fails with a message naming both options, and so does swarm-services-cert.service on a hive fronting no swarm name. The unit checks the identity before the leaf, so it fails even while a valid leaf keeps the gateway up. The identity is the hive's alone: it issues this hive's leaf and fetches the store's root, and it's never the identity a swarm service uses.

When it issues. The unit runs at boot, on every deploy, and from hive-tls-resign.timer once a day. It asks the store for a new leaf when the current one is missing, past half the role's 720-hour lifetime, missing a configured name, or doesn't chain to the store's root (swarm-services-root.pem). That last check is what replaces a leaf the retired hive CA signed, on the first run that reaches the store.

The gateway doesn't wait for the store. Nothing orders the gateway after hive-tls-resign.service, so a hive holding a valid leaf starts its gateway while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the current leaf in place, says so in the journal, and retries every 30 seconds for a day. Only a hive with no leaf yet needs the store reachable and unsealed: its gateway's certificate import fails until the first leaf lands, and then the unit starts nginx.

nginx serves the leaf alone, without its issuer: the issuer is the self-signed root a consumer already holds in its trust bundle.

The trust bundle

hive-tls-ca.service writes <deploy.hive-controller.tls.stateDir>/trust-bundle.pem, the file every consumer anchors on. It holds whichever of these exist:

• swarm-services-root.pem, the store's pki root, which the gateway leaf and the swarm-services leaf both chain to.
• ca.pem and ca-previous.pem, the retired hive CA's public certificates, so a leaf it signed keeps verifying until it's replaced. Removing them is a deliberate later step.
• the swarm root, <swarm.ca.stateDir>/root.pem, on the host that generates it.

No hive keeps a CA key. hive-tls-ca.service deletes <deploy.hive-controller.tls.stateDir>/ca-key.pem where an earlier generation left one, and generates no CA.

⚠️ A consumer reads its trust when it starts. An agent or service container that last started before the store's root reached the bundle fails TLS to a gateway serving a store-issued leaf, until it restarts.

The swarm's service names come from the secret store

The swarm's service names (swarm.forge.domain, swarm.matrix.gatewayHost, swarm.authelia.domain) default to siblings of the hive domain — forge.<swarm.domain>, not forge.<hive domain>. The gateway leaf carries the hive's domain and a single-label wildcard over it, and its role allows nothing outside the hive's domain, so it can't cover them.

The secret store's pki mount issues them a leaf of their own, through its swarm-services role. swarm-services-cert.service logs in with this host's services-issuer certificate, calls pki/issue/swarm-services, and writes the result to /var/lib/hive-tls/swarm-services.pem — the path the gateway already copies from. Every hive does this with its own identity, so holding the swarm root's private key stopped being what decides whether a hive can serve its swarm's names.

services.hyperhive.swarm.serviceDomains, every swarm service name and the same list on every host, populates the role's allowed_domains, so asking for a name nobody configured is a refusal from the store naming that name — not a certificate quietly issued for it. A host's own SANs are narrower still: it asks only for the names it fronts a vhost for (swarm.localServiceDomains), so the key on one gateway can't serve a swarm service that runs behind another.

The root's public certificate is a file, on every hive: /var/lib/hive-tls/swarm-services-root.pem (0644), written beside the leaf it signed and folded into trust-bundle.pem. A hive fronting no swarm name asks for no services leaf and reads the root alone, from the store's pki/ca/pem, presenting its own store identity (deploy.hive-controller.tls.baoHiveIdentityCertFile / baoHiveIdentityKeyFile) for the handshake. Every hive must hold one: without it, or with either file absent, swarm-services-cert.service fails with a message naming both options. On the host running the store it's also at /var/lib/swarm-bao-services-pki/services-root.pem. That's the file to hand a browser, and reading it needs no store login — which matters, because every store listener but the loopback UI one demands a client certificate, and authelia gates that one to the admins group, not open to an anonymous browser fetching a trust anchor.

The granting unit generates the root once, and never again. It asks the mount whether it already has an issuer (bao list pki/issuers) before calling pki/root/generate/internal, so a rebuild or a reboot re-asserts the role and the grant without touching the anchor. A root that changed per boot would invalidate every certificate issued under it and every browser taught to trust it.

A daily timer renews the leaf. The store's role issues it for 720 hours. swarm-services-cert-renew.timer runs the same script as swarm-services-cert.service once a day, and that script asks the store for a new leaf once the current one is past half that window or is missing a configured name. A rotation re-imports the leaf into the gateway and reloads nginx. A store that can't answer — still sealed, or out of reach — fails the unit, which leaves the current leaf in place, and the next day's run tries again: about fifteen tries before the leaf lapses. systemctl list-timers swarm-services-cert-renew shows the next run.

Constraints on the material

No CA private key exists on any hive. The pki root's key stays inside the store's mount (root/generate/internal), and each hive's narrowing is its role's allowed_domains, enforced by the store at issue time: a hive's identity that leaks can get certificates for that hive's names and no others.

The swarm root's private key never reaches the nix store: the store is world-readable and content-addressed, so a key committed to a flake is a key published to everyone who builds it. The swarm distributes only certificates.

What to hand a peer

hivectl peer-config prints the services.hyperhive.swarm.hives."<name>" block a peer operator pastes into their own config. When this hive's gateway serves a self-signed leaf (detected by the presence of <deploy.hive-controller.tls.stateDir>/trust-bundle.pem), it also prints a one-time scp line installing the swarm root — <swarm.ca.stateDir>/root.pem, not this hive's own CA — on the peer's host:

scp /var/lib/swarm-ca/root.pem <peer-host>:/var/lib/swarm-ca/root.pem

It then prints the secret store's pki root itself, read from /var/lib/hive-tls/swarm-services-root.pem. The store issues every hive's gateway leaf and the swarm's service names under that root, so a peer host trusting only the swarm root can't verify them.

The operator installs the roots once per swarm host, not once per peer, so a hive joining later needs no edit on the hives already running. A hive whose cert already chains to a public CA has nothing to install — peer-config says so instead.

Distributing the root

The root key is a runtime file for the reason in Constraints on the material.

The root certificate is a runtime file as a consequence — it lives beside the key under swarm.ca.stateDir — and that has a cost worth naming, because it's subtle and it bites at a distance:

Nothing whose trust store nix assembles at build time can reference the swarm root. The derivation reads security.pki.certificateFiles inside itself; the root doesn't exist there.

Two consumers, and only one of them is fine:

• hive-tls.nix covers agents. It folds the root into this hive's trust-bundle.pem, hive-c0re receives that path as HIVE_TLS_CA_PATH, and the meta-flake renderer embeds that one file next to every agent's flake. The bundle is the runtime-to-build-time bridge.

• The Matrix container crosses the same bridge, via the shared lib/hive-ca-trust.nix helper that hive-ci and hive-forge already use: the bundle is bind-mounted read-only into the container, and a oneshot concatenates it with the system CAs before tuwunel starts.

  The consumption differs per runtime and is the part worth knowing. tuwunel links no openssl, which makes SSL_CERT_FILE look inapplicable — it's not: its rustls-based TLS stack still resolves trust through the platform certificate store on Linux, and that store honors SSL_CERT_FILE, so the env var takes effect the same way it would for an OpenSSL-linked binary.

  ⚠️ Concatenate; never point SSL_CERT_FILE at the anchor alone. openssl-probe uses it instead of the default store, so naming just the bundle would drop every public CA and break federation with the wider matrix network — a much bigger outage than the one being fixed. The same caveat applies to hive-forge (Go) for the same reason.

To put the root on another host, copy the certificate to the same path there (scp <stateDir>/root.pem <host>:<stateDir>/root.pem). One anchor per host, not one per peer: a hive joining later needs no edit on the hives already running, which is the whole point of the hierarchy.