272 lines
14 KiB
Markdown
272 lines
14 KiB
Markdown
# Swarm CA
|
|
|
|
A hive's gateway serves a leaf issued by the secret store's `pki` mount,
|
|
through a role that belongs to that hive. A peer that trusts the mount's
|
|
root once validates every hive's gateway and every swarm service name,
|
|
present and future, instead of pinning to each one by hand.
|
|
|
|
That's the point of one issuer — it turns per-peer trust from O(n²)
|
|
hand-pinning into one anchor per swarm.
|
|
|
|
## The gateway leaf comes from the secret store
|
|
|
|
`hive-tls-resign.service` logs in to the store with this hive's own
|
|
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
|
|
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
|
|
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
|
|
the files the gateway copies from. The leaf's subject is the hive domain,
|
|
and its SANs are the hive domain and `*.<hive domain>`.
|
|
|
|
**Each hive's role covers that hive alone.** Its `allowed_domains` is the
|
|
hive's domain and `*.<hive domain>`, read from
|
|
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
|
|
IP SANs and no `localhost`. A request for any other name is a refusal from
|
|
the store naming that name. The hive's policy grants `update` on that one
|
|
issue path and nothing else. `swarm-bao-hive-leaf-policy.service`, on the
|
|
store's host, writes the role, the policy and the cert-auth role for every
|
|
hive in `swarm.hives`.
|
|
|
|
**Every hive needs its own store identity.** It's a client certificate
|
|
from the store's own client CA, with the subject
|
|
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
|
|
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
|
|
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
|
|
`baoHiveIdentityKeyFile`:
|
|
|
|
- On the host running the store, `swarm-bao-pki.service` mints it at
|
|
`/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
|
|
- On every other host, copy `hive-leaf.pem` and `hive-leaf-key.pem` from
|
|
`/var/lib/swarm-bao-pki/hives/<hiveName>/` on the store's host, which
|
|
`swarm-bao-pki.service` mints for every other hive in `swarm.hives`, and
|
|
set both options, before the host's next deploy.
|
|
|
|
Without it, `hive-tls-resign.service` fails with a message naming both
|
|
options, and so does `swarm-services-cert.service` on a hive fronting no
|
|
swarm name. The unit checks the identity before the leaf, so it fails even while a
|
|
valid leaf keeps the gateway up. The identity is the hive's alone:
|
|
it issues this hive's leaf and fetches the store's root, and it's never the
|
|
identity a swarm service uses.
|
|
|
|
**When it issues.** The unit runs at boot, on every deploy, and from
|
|
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
|
|
the current one is missing, past half the role's 720-hour lifetime, missing
|
|
a configured name, doesn't match the key beside it, or doesn't chain to the
|
|
store's root (`swarm-services-root.pem`). The first run that reaches the
|
|
store replaces a leaf that doesn't chain to that root, whoever signed it.
|
|
|
|
**The gateway doesn't wait for the store.** Nothing orders the gateway after
|
|
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
|
|
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
|
|
current leaf in place, says so in the journal, and retries every 30 seconds
|
|
for a day. Only a hive with no leaf yet needs the store reachable and
|
|
unsealed: its gateway's certificate import fails until the first leaf
|
|
lands, and then the unit starts nginx.
|
|
|
|
nginx serves the leaf alone, without its issuer: the issuer is the
|
|
self-signed root a consumer already holds in its trust bundle.
|
|
|
|
**A gateway host that runs no hive** (`deploy.hive-controller.enable` off,
|
|
such as a dedicated swarm-services or secret-store host) has no hive leaf, no
|
|
`hive-tls-resign.service`, no hive identity and no hive-domain vhost. Its
|
|
gateway serves the swarm-services leaf as its default certificate too. Where it
|
|
fronts other swarm names it waits for `swarm-services-cert.service` at start.
|
|
Where the store's UI is its only swarm name it starts on a throwaway
|
|
self-signed certificate and switches to the services leaf once the store
|
|
issues it.
|
|
|
|
## The trust bundle
|
|
|
|
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
|
|
the file every consumer anchors on. It holds one certificate, a copy of
|
|
`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and
|
|
the swarm-services leaf both chain to. It's the only root the swarm has.
|
|
|
|
**Without the store's root, the bundle is empty and its readers go red.**
|
|
While `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem` is
|
|
missing or empty, as on a first boot before the store answers,
|
|
`hive-tls-ca.service` writes an empty bundle and logs that file's path. Every
|
|
`<name>-ca-bundle` unit then fails with `<bundle> contributed no certificate
|
|
to the bundle`, and the consumers that `Requires=` it don't start.
|
|
`swarm-services-cert.service` restarts `hive-tls-ca.service` when it writes a
|
|
new root.
|
|
|
|
A deploy that changes `hive-tls-ca.service` reloads it: the same script
|
|
rewrites the bundle in place, and nothing that `Requires=` the unit restarts,
|
|
including the store's container. A `<name>-ca-bundle` unit keeps the copy it
|
|
made when it started.
|
|
|
|
⚠️ **A consumer reads its trust when it starts.** An agent or service
|
|
container that last started before the store's root reached the bundle
|
|
fails TLS to a gateway serving a store-issued leaf, until it restarts.
|
|
|
|
## The swarm's service names come from the secret store
|
|
|
|
The swarm's service names (`swarm.forge.domain`,
|
|
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
|
|
siblings of the hive domain — `forge.<swarm.domain>`, not
|
|
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
|
|
**single-label** wildcard over it, and its role allows nothing outside the hive's
|
|
domain, so it can't cover them.
|
|
|
|
The secret store's `pki` mount issues them a leaf of their own, through its
|
|
`swarm-services` role. `swarm-services-cert.service` logs in with
|
|
the services-issuer identity, calls
|
|
`pki/issue/swarm-services`, and writes the result to
|
|
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
|
|
copies from. The store's role, not a key on the host, decides which names a
|
|
gateway can serve.
|
|
|
|
**Every self-signed gateway that fronts a swarm name needs the
|
|
services-issuer identity**, whether or not a hive runs on its host. A hive
|
|
fronting none needs no copy of it: it fetches the store's root with its own
|
|
store identity, as below. It's a client certificate
|
|
from the store's own client CA, with the subject
|
|
`deploy.bao.servicesIssuerCommonName` (`swarm-services-issuer` by default),
|
|
and it's named by `services.hyperhive.deploy.bao.servicesIssuerClientCertFile`
|
|
and `servicesIssuerClientKeyFile`. Its policy may issue from
|
|
`pki/issue/swarm-services` and nothing else, and it's the same certificate on
|
|
every gateway host:
|
|
|
|
- On the host running the store, `swarm-bao-pki.service` mints it at
|
|
`/var/lib/swarm-bao-pki/services-issuer.pem` and the two options default to
|
|
it.
|
|
- On every other gateway host fronting a swarm name, copy
|
|
`services-issuer.pem` and `services-issuer-key.pem` from the store's host
|
|
and set both options.
|
|
|
|
A host fronting a swarm name without both options set still evaluates and deploys, and `swarm-services-cert.service` fails, naming
|
|
them. One that names files it can't read fails the same unit, naming the
|
|
paths. The old spellings,
|
|
`deploy.hive-controller.tls.baoClientCertFile` and `baoClientKeyFile`, are
|
|
renames of these two: a config that still sets them evaluates with a warning
|
|
naming both paths.
|
|
|
|
`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
|
|
same list on every host, populates the role's
|
|
`allowed_domains`, so asking for a name nobody configured is a refusal
|
|
from the store naming that name — not a certificate quietly issued for
|
|
it. A host's own SANs are narrower still: it asks only for the names it
|
|
fronts a vhost for (`swarm.localServiceDomains`), so the key on one
|
|
gateway can't serve a swarm service that runs behind another.
|
|
|
|
**The root's public certificate is a file, on every self-signed gateway host:**
|
|
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
|
|
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
|
|
swarm name asks for no services leaf and reads the root alone, from the store's
|
|
`pki/ca/pem`, presenting its own store identity
|
|
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
|
|
`baoHiveIdentityKeyFile`) for the handshake without logging in with it.
|
|
Every hive must hold one: without it, or with either file absent,
|
|
`swarm-services-cert.service` fails with a message naming both options.
|
|
On the host running
|
|
the store it's also at
|
|
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
|
|
hand a browser, and reading it needs no store login — which matters,
|
|
because every store listener but the loopback UI one demands a client
|
|
certificate, and authelia gates that one to the `admins` group, not
|
|
open to an anonymous browser fetching a trust anchor.
|
|
|
|
**The granting unit generates the root once, and never again.** It asks
|
|
the mount whether it already has an issuer (`bao list pki/issuers`)
|
|
before calling `pki/root/generate/internal`, so a rebuild or a reboot
|
|
re-asserts the role and the grant without touching the anchor. A root
|
|
that changed per boot would invalidate every certificate issued under it
|
|
and every browser taught to trust it.
|
|
|
|
**A daily timer renews the leaf.** The store's role issues it for 720
|
|
hours. `swarm-services-cert-renew.timer` runs the same script as
|
|
`swarm-services-cert.service` once a day, and that script asks the store
|
|
for a new leaf once the current one is past half that window, is
|
|
missing a configured name, or doesn't chain to the store's root
|
|
(`swarm-services-root.pem`). A rotation re-imports the leaf into the
|
|
gateway and reloads nginx. A store that can't answer — still sealed, or
|
|
out of reach — fails the unit, which leaves the current leaf in place,
|
|
and the next day's run tries again: about fifteen tries before the leaf
|
|
lapses. `systemctl
|
|
list-timers swarm-services-cert-renew` shows the next run.
|
|
|
|
## Constraints on the material
|
|
|
|
No CA private key exists on any hive. The `pki` root's key stays inside the
|
|
store's mount (`root/generate/internal`), and each hive's narrowing is its
|
|
role's `allowed_domains`, enforced by the store at issue time: a hive's
|
|
identity that leaks can get certificates for that hive's names and no
|
|
others.
|
|
|
|
No host holds a CA or its key, except the store's mTLS CA
|
|
(`glue-bao-tls.nix`), which signs only the store's server certificate and the
|
|
certificates that log in to it. Every host deletes `/var/lib/swarm-ca` and
|
|
the `ca.pem`, `ca-previous.pem`, `ca-key.pem`, `ca.srl` and
|
|
`.swarm-ca-adopted` in
|
|
`<deploy.hive-controller.tls.stateDir>`: `systemd-tmpfiles` removes them at
|
|
boot, and on a deploy that changes the host's `tmpfiles.d` rules. The swarm
|
|
distributes only certificates.
|
|
|
|
⚠️ **The deletion is irreversible.** Rolling back to a generation that still
|
|
has `swarm-ca.service` finds no root under `/var/lib/swarm-ca` and mints a
|
|
new one, unrelated to the deleted root. That's harmless: nothing issues
|
|
under it.
|
|
|
|
A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or
|
|
`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails
|
|
evaluation, however it's spelled with `//`, `.`, `..` or a trailing `/`. A
|
|
symlink to one isn't resolved, so it gets past the check. Those options name
|
|
the store's mTLS CA, `/var/lib/swarm-bao-pki/ca.pem` on the host that runs
|
|
the store.
|
|
|
|
## What to hand a peer
|
|
|
|
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
|
|
block a peer operator pastes into their own config. When this hive's
|
|
gateway serves a self-signed leaf (detected by the presence of
|
|
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints the
|
|
secret store's `pki` root itself, read from
|
|
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
|
|
gateway leaf and the swarm's service names under that root, so it's the one
|
|
anchor a peer host needs.
|
|
|
|
The operator installs the root **once per swarm host**, not once per peer,
|
|
so a hive joining later needs no edit on the hives already running. A hive
|
|
whose cert already chains to a public CA has nothing to install —
|
|
`peer-config` says so instead.
|
|
|
|
## Distributing the root
|
|
|
|
The root **certificate** is a runtime file: each hive fetches it from the
|
|
store at boot, so nothing can name it at build time. That has a cost worth
|
|
naming, because it's subtle and it bites at a distance:
|
|
|
|
> **Nothing whose trust store nix assembles at build time can reference
|
|
> the store's root.** The derivation reads `security.pki.certificateFiles`
|
|
> inside itself; the root doesn't exist there.
|
|
|
|
Two consumers, and only one of them is fine:
|
|
|
|
- **`hive-tls.nix` covers agents.** It folds the root into this hive's
|
|
`trust-bundle.pem`, `glue-hive-tls-controllers.nix` hands hive-c0re that
|
|
path as `HIVE_TLS_CA_PATH`, and the meta-flake renderer embeds that one
|
|
file next to every agent's flake. The bundle is the runtime-to-build-time
|
|
bridge.
|
|
- **The Matrix container crosses the same bridge**, via the shared
|
|
`lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
|
|
use: the bundle is bind-mounted read-only into the container, and a
|
|
oneshot concatenates it with the system CAs before tuwunel starts.
|
|
|
|
The consumption differs per runtime and is the part worth knowing.
|
|
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
|
|
— it's not: its rustls-based TLS stack still resolves trust through the
|
|
platform certificate store on Linux, and that store honors
|
|
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
|
|
an OpenSSL-linked binary.
|
|
|
|
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
|
|
> `openssl-probe` uses it _instead of_ the default store, so naming
|
|
> just the bundle would drop every public CA and break federation with
|
|
> the wider matrix network — a much bigger outage than the one being
|
|
> fixed. The same caveat applies to `hive-forge` (Go) for the same
|
|
> reason.
|
|
|
|
A hive in the swarm fetches the root itself. A host outside it gets the
|
|
certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)).
|
|
One anchor per host, not one per peer: a hive joining later needs no edit
|
|
on the hives already running.
|