fc33a62bf8
hive-tls-ca no longer deletes a ca-key.pem an earlier generation left: nothing reads it, and leaving it means a rollback finds the old key instead of minting a new anchor. Deleting it moves to PR 3, beside dropping ca.pem from the trust bundle. hive-tls-resign now also re-issues when the leaf's public key differs from the key file's. The leaf and key are moved into place by two renames, and an interrupted pair left a certificate beside the wrong key that the validity check accepted. Refs #4588
228 lines
11 KiB
Markdown
228 lines
11 KiB
Markdown
# Swarm CA
|
|
|
|
A hive's gateway serves a leaf issued by the secret store's `pki` mount,
|
|
through a role that belongs to that hive. A peer that trusts the mount's
|
|
root once validates every hive's gateway and every swarm service name,
|
|
present and future, instead of pinning to each one by hand.
|
|
|
|
That's the point of one issuer — it turns per-peer trust from O(n²)
|
|
hand-pinning into one anchor per swarm.
|
|
|
|
## The gateway leaf comes from the secret store
|
|
|
|
`hive-tls-resign.service` logs in to the store with this hive's own
|
|
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
|
|
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
|
|
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
|
|
the files the gateway copies from. The leaf's subject is the hive domain,
|
|
and its SANs are the hive domain and `*.<hive domain>`.
|
|
|
|
**Each hive's role covers that hive alone.** Its `allowed_domains` is the
|
|
hive's domain and `*.<hive domain>`, read from
|
|
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
|
|
IP SANs and no `localhost`. A request for any other name is a refusal from
|
|
the store naming that name. The hive's policy grants `update` on that one
|
|
issue path and nothing else. `swarm-bao-hive-leaf-policy.service`, on the
|
|
store's host, writes the role, the policy and the cert-auth role for every
|
|
hive in `swarm.hives`.
|
|
|
|
**Every hive needs its own store identity.** It's a client certificate
|
|
from the store's own client CA, with the subject
|
|
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
|
|
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
|
|
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
|
|
`baoHiveIdentityKeyFile`:
|
|
|
|
- On the host running the store, `swarm-bao-pki.service` mints it at
|
|
`/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
|
|
- On every other host, copy a leaf with that subject from the store's host
|
|
and set both options, before the host's next deploy.
|
|
|
|
Without it, `hive-tls-resign.service` fails with a message naming both
|
|
options, and so does `swarm-services-cert.service` on a hive fronting no
|
|
swarm name. The unit checks the identity before the leaf, so it fails even while a
|
|
valid leaf keeps the gateway up. The identity is the hive's alone:
|
|
it issues this hive's leaf and fetches the store's root, and it's never the
|
|
identity a swarm service uses.
|
|
|
|
**When it issues.** The unit runs at boot, on every deploy, and from
|
|
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
|
|
the current one is missing, past half the role's 720-hour lifetime, missing
|
|
a configured name, doesn't match the key beside it, or doesn't chain to the
|
|
store's root (`swarm-services-root.pem`). That last check is what replaces a
|
|
leaf the retired hive CA signed, on the first run that reaches the store.
|
|
|
|
**The gateway doesn't wait for the store.** Nothing orders the gateway after
|
|
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
|
|
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
|
|
current leaf in place, says so in the journal, and retries every 30 seconds
|
|
for a day. Only a hive with no leaf yet needs the store reachable and
|
|
unsealed: its gateway's certificate import fails until the first leaf
|
|
lands, and then the unit starts nginx.
|
|
|
|
nginx serves the leaf alone, without its issuer: the issuer is the
|
|
self-signed root a consumer already holds in its trust bundle.
|
|
|
|
## The trust bundle
|
|
|
|
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
|
|
the file every consumer anchors on. It holds whichever of these exist:
|
|
|
|
- `swarm-services-root.pem`, the store's `pki` root, which the gateway leaf
|
|
and the swarm-services leaf both chain to.
|
|
- `ca.pem` and `ca-previous.pem`, the retired hive CA's public certificates,
|
|
so a leaf it signed keeps verifying until it's replaced. Removing them is a
|
|
deliberate later step.
|
|
- the swarm root, `<swarm.ca.stateDir>/root.pem`, on the host that generates
|
|
it.
|
|
|
|
No hive generates a CA or signs with a CA key. A
|
|
`<deploy.hive-controller.tls.stateDir>/ca-key.pem` an earlier generation left
|
|
stays on disk, unread: removing it comes with dropping `ca.pem` from the
|
|
bundle.
|
|
|
|
⚠️ **A consumer reads its trust when it starts.** An agent or service
|
|
container that last started before the store's root reached the bundle
|
|
fails TLS to a gateway serving a store-issued leaf, until it restarts.
|
|
|
|
## The swarm's service names come from the secret store
|
|
|
|
The swarm's service names (`swarm.forge.domain`,
|
|
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
|
|
siblings of the hive domain — `forge.<swarm.domain>`, not
|
|
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
|
|
**single-label** wildcard over it, and its role allows nothing outside the hive's
|
|
domain, so it can't cover them.
|
|
|
|
The secret store's `pki` mount issues them a leaf of their own, through its
|
|
`swarm-services` role. `swarm-services-cert.service` logs in with
|
|
this host's services-issuer certificate, calls
|
|
`pki/issue/swarm-services`, and writes the result to
|
|
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
|
|
copies from. Every hive does this with its own identity, so holding the
|
|
swarm root's private key stopped being what decides whether a hive can
|
|
serve its swarm's names.
|
|
|
|
`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
|
|
same list on every host, populates the role's
|
|
`allowed_domains`, so asking for a name nobody configured is a refusal
|
|
from the store naming that name — not a certificate quietly issued for
|
|
it. A host's own SANs are narrower still: it asks only for the names it
|
|
fronts a vhost for (`swarm.localServiceDomains`), so the key on one
|
|
gateway can't serve a swarm service that runs behind another.
|
|
|
|
**The root's public certificate is a file, on every hive:**
|
|
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
|
|
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
|
|
swarm name asks for no services leaf and reads the root alone, from the store's
|
|
`pki/ca/pem`, presenting its own store identity
|
|
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
|
|
`baoHiveIdentityKeyFile`) for the handshake. Every hive must hold one:
|
|
without it, or with either file absent,
|
|
`swarm-services-cert.service` fails with a message naming both options.
|
|
On the host running
|
|
the store it's also at
|
|
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
|
|
hand a browser, and reading it needs no store login — which matters,
|
|
because every store listener but the loopback UI one demands a client
|
|
certificate, and authelia gates that one to the `admins` group, not
|
|
open to an anonymous browser fetching a trust anchor.
|
|
|
|
**The granting unit generates the root once, and never again.** It asks
|
|
the mount whether it already has an issuer (`bao list pki/issuers`)
|
|
before calling `pki/root/generate/internal`, so a rebuild or a reboot
|
|
re-asserts the role and the grant without touching the anchor. A root
|
|
that changed per boot would invalidate every certificate issued under it
|
|
and every browser taught to trust it.
|
|
|
|
**A daily timer renews the leaf.** The store's role issues it for 720
|
|
hours. `swarm-services-cert-renew.timer` runs the same script as
|
|
`swarm-services-cert.service` once a day, and that script asks the store
|
|
for a new leaf once the current one is past half that window or is
|
|
missing a configured name. A rotation re-imports the leaf into the
|
|
gateway and reloads nginx. A store that can't answer — still sealed, or
|
|
out of reach — fails the unit, which leaves the current leaf in place,
|
|
and the next day's run tries again: about fifteen tries before the leaf
|
|
lapses. `systemctl
|
|
list-timers swarm-services-cert-renew` shows the next run.
|
|
|
|
## Constraints on the material
|
|
|
|
No CA private key exists on any hive. The `pki` root's key stays inside the
|
|
store's mount (`root/generate/internal`), and each hive's narrowing is its
|
|
role's `allowed_domains`, enforced by the store at issue time: a hive's
|
|
identity that leaks can get certificates for that hive's names and no
|
|
others.
|
|
|
|
The swarm root's private key never reaches the nix store: the store is
|
|
world-readable and content-addressed, so a key committed to a flake is a
|
|
key published to everyone who builds it. The swarm distributes only
|
|
certificates.
|
|
|
|
## What to hand a peer
|
|
|
|
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
|
|
block a peer operator pastes into their own config. When this hive's
|
|
gateway serves a self-signed leaf (detected by the presence of
|
|
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints a
|
|
one-time `scp` line installing the **swarm root** —
|
|
`<swarm.ca.stateDir>/root.pem`, not this hive's own CA — on the peer's
|
|
host:
|
|
|
|
```
|
|
scp /var/lib/swarm-ca/root.pem <peer-host>:/var/lib/swarm-ca/root.pem
|
|
```
|
|
|
|
It then prints the secret store's `pki` root itself, read from
|
|
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
|
|
gateway leaf and the swarm's service names under that root, so a peer host
|
|
trusting only the swarm root can't verify them.
|
|
|
|
The operator installs the roots **once per swarm host**, not once per peer,
|
|
so a hive joining later needs no edit on the hives already running. A hive
|
|
whose cert already chains to a public CA has nothing to install —
|
|
`peer-config` says so instead.
|
|
|
|
## Distributing the root
|
|
|
|
The root **key** is a runtime file for the reason in
|
|
[Constraints on the material](#constraints-on-the-material).
|
|
|
|
The root **certificate** is a runtime file as a _consequence_ — it lives
|
|
beside the key under `swarm.ca.stateDir` — and that has a cost worth
|
|
naming, because it's subtle and it bites at a distance:
|
|
|
|
> **Nothing whose trust store nix assembles at build time can reference
|
|
> the swarm root.** The derivation reads `security.pki.certificateFiles`
|
|
> inside itself; the root doesn't exist there.
|
|
|
|
Two consumers, and only one of them is fine:
|
|
|
|
- **`hive-tls.nix` covers agents.** It folds the root into this hive's
|
|
`trust-bundle.pem`, hive-c0re receives that path as
|
|
`HIVE_TLS_CA_PATH`, and the meta-flake renderer embeds that one file
|
|
next to every agent's flake. The bundle is the runtime-to-build-time
|
|
bridge.
|
|
- **The Matrix container crosses the same bridge**, via the shared
|
|
`lib/hive-ca-trust.nix` helper that `hive-ci` and `hive-forge` already
|
|
use: the bundle is bind-mounted read-only into the container, and a
|
|
oneshot concatenates it with the system CAs before tuwunel starts.
|
|
|
|
The consumption differs per runtime and is the part worth knowing.
|
|
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
|
|
— it's not: its rustls-based TLS stack still resolves trust through the
|
|
platform certificate store on Linux, and that store honors
|
|
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
|
|
an OpenSSL-linked binary.
|
|
|
|
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
|
|
> `openssl-probe` uses it _instead of_ the default store, so naming
|
|
> just the bundle would drop every public CA and break federation with
|
|
> the wider matrix network — a much bigger outage than the one being
|
|
> fixed. The same caveat applies to `hive-forge` (Go) for the same
|
|
> reason.
|
|
|
|
To put the root on another host, copy the certificate to the same path
|
|
there (`scp <stateDir>/root.pem <host>:<stateDir>/root.pem`). One anchor
|
|
per host, not one per peer: a hive joining later needs no edit on the
|
|
hives already running, which is the whole point of the hierarchy.
|