# Swarm CA A hive's gateway serves a leaf issued by the secret store's `pki` mount, through a role that belongs to that hive. A peer that trusts the mount's root once validates every hive's gateway and every swarm service name, present and future, instead of pinning to each one by hand. That's the point of one issuer — it turns per-peer trust from O(n²) hand-pinning into one anchor per swarm. ## The gateway leaf comes from the secret store `hive-tls-resign.service` logs in to the store with this hive's own identity, calls `pki/issue/-` (`swarm-hive-leaf-` by default), and writes the result to `/gateway.pem` and `gateway-key.pem`, the files the gateway copies from. The leaf's subject is the hive domain, and its SANs are the hive domain and `*.`. **Each hive's role covers that hive alone.** Its `allowed_domains` is the hive's domain and `*.`, read from `services.hyperhive.swarm.hives..domain` on the store's host, with no IP SANs and no `localhost`. A request for any other name is a refusal from the store naming that name. The hive's policy grants `update` on that one issue path and nothing else. `swarm-bao-hive-leaf-policy.service`, on the store's host, writes the role, the policy and the cert-auth role for every hive in `swarm.hives`. **Every hive needs its own store identity.** It's a client certificate from the store's own client CA, with the subject `-` (`swarm-bao-hive-leaf-` by default), and it's named by `services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and `baoHiveIdentityKeyFile`: - On the host running the store, `swarm-bao-pki.service` mints it at `/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it. - On every other host, copy `hive-leaf.pem` and `hive-leaf-key.pem` from `/var/lib/swarm-bao-pki/hives//` on the store's host, which `swarm-bao-pki.service` mints for every other hive in `swarm.hives`, and set both options, before the host's next deploy. Without it, `hive-tls-resign.service` fails with a message naming both options, and so does `swarm-services-cert.service` on a hive fronting no swarm name. The unit checks the identity before the leaf, so it fails even while a valid leaf keeps the gateway up. The identity is the hive's alone: it issues this hive's leaf and fetches the store's root, and it's never the identity a swarm service uses. **When it issues.** The unit runs at boot, on every deploy, and from `hive-tls-resign.timer` once a day. It asks the store for a new leaf when the current one is missing, past half the role's 720-hour lifetime, missing a configured name, doesn't match the key beside it, or doesn't chain to the store's root (`swarm-services-root.pem`). The first run that reaches the store replaces a leaf that doesn't chain to that root, whoever signed it. **The gateway doesn't wait for the store.** Nothing orders the gateway after `hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the current leaf in place, says so in the journal, and retries every 30 seconds for a day. Only a hive with no leaf yet needs the store reachable and unsealed: its gateway's certificate import fails until the first leaf lands, and then the unit starts nginx. nginx serves the leaf alone, without its issuer: the issuer is the self-signed root a consumer already holds in its trust bundle. **A gateway host that runs no hive** (`deploy.hive-controller.enable` off, such as a dedicated swarm-services or secret-store host) has no hive leaf, no `hive-tls-resign.service`, no hive identity and no hive-domain vhost. Its gateway serves the swarm-services leaf as its default certificate too. Where it fronts other swarm names it waits for `swarm-services-cert.service` at start. Where the store's UI is its only swarm name it starts on a throwaway self-signed certificate and switches to the services leaf once the store issues it. ## The trust bundle `hive-tls-ca.service` writes `/trust-bundle.pem`, the file every consumer anchors on. It holds one certificate, a copy of `swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and the swarm-services leaf both chain to. It's the only root the swarm has. **Without the store's root, the bundle is empty and its readers go red until it lands.** While `/swarm-services-root.pem` is missing or empty, as on a first boot before the store answers, `hive-tls-ca.service` writes an empty bundle and logs that file's path. Every `-ca-bundle` unit then fails with ` contributed no certificate to the bundle`, and the consumers that `Requires=` it don't start. `swarm-services-cert.service` reloads `hive-tls-ca.service` when it writes a new root. That rewrites the bundle in place and restarts no container, the store's included. **Each `-ca-bundle` unit converges on its own.** Beside it, a `-ca-bundle-refresh.path` unit watches the file the bundle derives from: inside a container the bound `/run/hive-ca/trust-bundle.pem`, on the host `/trust-bundle.pem`. A write to it starts `-ca-bundle-refresh.service`. That service does nothing while the file holds no certificate, or while it matches the copy the last assembly took (`/run/-ca/anchor.pem`). Otherwise it restarts the bundle unit and starts its consumers without waiting. The restart reaches the running consumers through their `Requires=`; the start picks up those a first boot left failed. The same service runs once at boot, after the bundle unit, which covers a root that landed before the watch began. A first boot needs no second apply, and a root rotation restarts the consumers once and the container never. The watch fires for the host's in-place rewrite because inotify keys on the inode and a bind mount shares it, so `hive-tls-ca` must keep rewriting the file in place and never rename over it. A deploy that changes `hive-tls-ca.service` reloads it too, with the same effect. ⚠️ **A consumer outside `trustBundle` reads its trust when it starts.** A container whose nix predates the refresh units keeps the bundle it started with until it restarts, and fails TLS to a gateway serving a store-issued leaf in the meantime. ## The swarm's service names come from the secret store The swarm's service names (`swarm.forge.domain`, `swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to siblings of the hive domain — `forge.`, not `forge.`. The gateway leaf carries the hive's domain and a **single-label** wildcard over it, and its role allows nothing outside the hive's domain, so it can't cover them. The secret store's `pki` mount issues them a leaf of their own, through its `swarm-services` role. `swarm-services-cert.service` logs in with the services-issuer identity, calls `pki/issue/swarm-services`, and writes the result to `/var/lib/hive-tls/swarm-services.pem` — the path the gateway already copies from. The store's role, not a key on the host, decides which names a gateway can serve. **Every self-signed gateway that fronts a swarm name needs the services-issuer identity**, whether or not a hive runs on its host. A hive fronting none needs no copy of it: it fetches the store's root with its own store identity, as below. It's a client certificate from the store's own client CA, with the subject `deploy.bao.servicesIssuerCommonName` (`swarm-services-issuer` by default), and it's named by `services.hyperhive.deploy.bao.servicesIssuerClientCertFile` and `servicesIssuerClientKeyFile`. Its policy may issue from `pki/issue/swarm-services` and nothing else, and it's the same certificate on every gateway host: - On the host running the store, `swarm-bao-pki.service` mints it at `/var/lib/swarm-bao-pki/services-issuer.pem` and the two options default to it. - On every other gateway host fronting a swarm name, copy `services-issuer.pem` and `services-issuer-key.pem` from the store's host and set both options. A host fronting a swarm name without both options set still evaluates and deploys, and `swarm-services-cert.service` fails, naming them. One that names files it can't read fails the same unit, naming the paths. The old spellings, `deploy.hive-controller.tls.baoClientCertFile` and `baoClientKeyFile`, are renames of these two: a config that still sets them evaluates with a warning naming both paths. `services.hyperhive.swarm.serviceDomains`, every swarm service name and the same list on every host, populates the role's `allowed_domains`, so asking for a name nobody configured is a refusal from the store naming that name — not a certificate quietly issued for it. A host's own SANs are narrower still: it asks only for the names it fronts a vhost for (`deploy.hive-controller.tls.localServiceDomains`), so the key on one gateway can't serve a swarm service that runs behind another. **The root's public certificate is a file, on every self-signed gateway host:** `/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the leaf it signed and folded into `trust-bundle.pem`. A hive fronting no swarm name asks for no services leaf and reads the root alone, from the store's `pki/ca/pem`, presenting its own store identity (`deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile`) for the handshake without logging in with it. Every hive must hold one: without it, or with either file absent, `swarm-services-cert.service` fails with a message naming both options. On the host running the store it's also at `/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to hand a browser, and reading it needs no store login — which matters, because every store listener but the loopback UI one demands a client certificate, and authelia gates that one to the `admins` group, not open to an anonymous browser fetching a trust anchor. **The granting unit generates the root once, and never again.** It asks the mount whether it already has an issuer (`bao list pki/issuers`) before calling `pki/root/generate/internal`, so a rebuild or a reboot re-asserts the role and the grant without touching the anchor. A root that changed per boot would invalidate every certificate issued under it and every browser taught to trust it. **A daily timer renews the leaf.** The store's role issues it for 720 hours. `swarm-services-cert-renew.timer` runs the same script as `swarm-services-cert.service` once a day, and that script asks the store for a new leaf once the current one is past half that window, is missing a configured name, or doesn't chain to the store's root (`swarm-services-root.pem`). A rotation re-imports the leaf into the gateway and reloads nginx. A store that can't answer — still sealed, or out of reach — fails the unit, which leaves the current leaf in place, and the next day's run tries again: about fifteen tries before the leaf lapses. `systemctl list-timers swarm-services-cert-renew` shows the next run. ## Constraints on the material No CA private key exists on any hive. The `pki` root's key stays inside the store's mount (`root/generate/internal`), and each hive's narrowing is its role's `allowed_domains`, enforced by the store at issue time: a hive's identity that leaks can get certificates for that hive's names and no others. No host holds a CA or its key, except the store's mTLS CA (`glue-bao-tls.nix`), which signs only the store's server certificate and the certificates that log in to it. Every host deletes `/var/lib/swarm-ca` and the `ca.pem`, `ca-previous.pem`, `ca-key.pem`, `ca.srl` and `.swarm-ca-adopted` in ``: `systemd-tmpfiles` removes them at boot, and on a deploy that changes the host's `tmpfiles.d` rules. The swarm distributes only certificates. ⚠️ **The deletion is irreversible.** Rolling back to a generation that still has `swarm-ca.service` finds no root under `/var/lib/swarm-ca` and mints a new one, unrelated to the deleted root. That's harmless: nothing issues under it. A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or `deploy.swarm-controller.hiveClientCaFile` names one of those paths fails evaluation, however it's spelled with `//`, `.`, `..` or a trailing `/`. A symlink to one isn't resolved, so it gets past the check. Those options name the store's mTLS CA, `/var/lib/swarm-bao-pki/ca.pem` on the host that runs the store. ## What to hand a peer `hivectl peer-config` prints the `services.hyperhive.swarm.hives.""` block a peer operator pastes into their own config. When this hive's gateway serves a self-signed leaf (detected by the presence of `/trust-bundle.pem`), it also prints the secret store's `pki` root itself, read from `/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's gateway leaf and the swarm's service names under that root, so it's the one anchor a peer host needs. The operator installs the root **once per swarm host**, not once per peer, so a hive joining later needs no edit on the hives already running. A hive whose cert already chains to a public CA has nothing to install — `peer-config` says so instead. ## Distributing the root The root **certificate** is a runtime file: each hive fetches it from the store at boot, so nothing can name it at build time. That has a cost worth naming, because it's subtle and it bites at a distance: > **Nothing whose trust store nix assembles at build time can reference > the store's root.** The derivation reads `security.pki.certificateFiles` > inside itself; the root doesn't exist there. Two consumers, both reading the root at runtime: - **Agents fetch it themselves.** `hive-agent-trust-bundle.service` (`nix/agent-modules/hive-ca.nix`) reads `/ca/pem` from the store with the agent's own store certificate, then writes `/run/hive-agent-trust-bundle/bundle.pem` (system CAs plus the root) and `root.pem` (the root alone). The agent's units point `SSL_CERT_FILE` at the first (Rust, Go and OpenSSL clients) and `NODE_EXTRA_CA_CERTS` at the second (Node and Claude Code), and order after the unit with `Requires=`. The meta flake names no CA file, and a timer re-reads the root hourly. A consumer reads it at start, so a rotated root reaches an agent unit at its next start. - **The Matrix container gets a bind mount**, via the shared `lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner` use: the bundle is bind-mounted read-only into the container, and a oneshot concatenates it with the system CAs before tuwunel starts. The consumption differs per runtime and is the part worth knowing. tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable — it's not: its rustls-based TLS stack still resolves trust through the platform certificate store on Linux, and that store honors `SSL_CERT_FILE`, so the env var takes effect the same way it would for an OpenSSL-linked binary. > ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.** > `openssl-probe` uses it _instead of_ the default store, so naming > just the bundle would drop every public CA and break federation with > the wider matrix network — a much bigger outage than the one being > fixed. The same caveat applies to `hive-forge` (Go) for the same > reason. A hive in the swarm fetches the root itself. A host outside it gets the certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)). One anchor per host, not one per peer: a hive joining later needs no edit on the hives already running.