docs: sort the top-level directories into five sections by what the reader wants
Every docs directory moves under 1-getting-started, 2-handbook, 4-implementation or 5-project; the options reference (section 3) comes from the website and has no directory here. trust-boundary stays whole under the handbook for now. web-ui, swarm and tools split between handbook and implementation, and the command-line references move to 2-handbook/reference. Links, path references in code and nix, and the docs index follow the moves; no page content changes.
This commit is contained in:
iris
mara
parent
53ccbcb4b3
commit
6739bb903d
1 changed files with 736 additions and 701 deletions
289
docs/4-implementation/swarm/ca.md
Normal file
289
docs/4-implementation/swarm/ca.md
Normal file
|
|
@ -0,0 +1,289 @@
|
|||
# Swarm CA
|
||||
|
||||
A hive's gateway serves a leaf issued by the secret store's `pki` mount,
|
||||
through a role that belongs to that hive. A peer that trusts the mount's
|
||||
root once validates every hive's gateway and every swarm service name,
|
||||
present and future, instead of pinning to each one by hand.
|
||||
|
||||
That's the point of one issuer — it turns per-peer trust from O(n²)
|
||||
hand-pinning into one anchor per swarm.
|
||||
|
||||
## The gateway leaf comes from the secret store
|
||||
|
||||
`hive-tls-resign.service` logs in to the store with this hive's own
|
||||
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
|
||||
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
|
||||
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
|
||||
the files the gateway copies from. The leaf's subject is the hive domain,
|
||||
and its SANs are the hive domain and `*.<hive domain>`.
|
||||
|
||||
**Each hive's role covers that hive alone.** Its `allowed_domains` is the
|
||||
hive's domain and `*.<hive domain>`, read from
|
||||
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
|
||||
IP SANs and no `localhost`. A request for any other name is a refusal from
|
||||
the store naming that name. The hive's policy grants `update` on that one
|
||||
issue path and `read` on its own `attic/public-key`, `attic/pull-token` and
|
||||
`squid-selfcheck`
|
||||
(see [the cache facts for hive-priv](../../2-handbook/swarm/credentials-reference.md) and the
|
||||
squid self-check password), and nothing
|
||||
else. `swarm-bao-hive-leaf-policy.service`, on the
|
||||
store's host, writes the role, the policy and the cert-auth role for every
|
||||
hive in `swarm.hives`.
|
||||
|
||||
**Every hive needs its own store identity.** It's a client certificate
|
||||
from the store's own client CA, with the subject
|
||||
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
|
||||
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
|
||||
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
|
||||
`baoHiveIdentityKeyFile`:
|
||||
|
||||
- On the host running the store, `swarm-bao-pki.service` mints it at
|
||||
`/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
|
||||
- On every other host, copy `hive-leaf.pem` and `hive-leaf-key.pem` from
|
||||
`/var/lib/swarm-bao-pki/hives/<hiveName>/` on the store's host, which
|
||||
`swarm-bao-pki.service` mints for every other hive in `swarm.hives`, and
|
||||
set both options, before the host's next deploy.
|
||||
|
||||
Without it, `hive-tls-resign.service` fails with a message naming both
|
||||
options, and so does `swarm-services-cert.service` on a hive fronting no
|
||||
swarm name. The unit checks the identity before the leaf, so it fails even while a
|
||||
valid leaf keeps the gateway up. The identity is the hive's alone:
|
||||
it issues this hive's leaf and fetches the store's root, and it's never the
|
||||
identity a swarm service uses.
|
||||
|
||||
**When it issues.** The unit runs at boot, on every deploy, and from
|
||||
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
|
||||
the current one is missing, past half the role's 720-hour lifetime, missing
|
||||
a configured name, doesn't match the key beside it, or doesn't chain to the
|
||||
store's root (`swarm-services-root.pem`). The first run that reaches the
|
||||
store replaces a leaf that doesn't chain to that root, whoever signed it.
|
||||
|
||||
**The gateway doesn't wait for the store.** Nothing orders the gateway after
|
||||
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
|
||||
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
|
||||
current leaf in place, says so in the journal, and retries every 30 seconds
|
||||
for a day. Only a hive with no leaf yet needs the store reachable and
|
||||
unsealed: its gateway's certificate import fails until the first leaf
|
||||
lands, and then the unit starts nginx.
|
||||
|
||||
nginx serves the leaf alone, without its issuer: the issuer is the
|
||||
self-signed root a consumer already holds in its trust bundle.
|
||||
|
||||
**A gateway host that runs no hive** (`deploy.hive-controller.enable` off,
|
||||
such as a dedicated swarm-services or secret-store host) has no hive leaf, no
|
||||
`hive-tls-resign.service`, no hive identity and no hive-domain vhost. Its
|
||||
gateway serves the swarm-services leaf as its default certificate too. Where it
|
||||
fronts other swarm names it waits for `swarm-services-cert.service` at start.
|
||||
Where the store's UI is its only swarm name it starts on a throwaway
|
||||
self-signed certificate and switches to the services leaf once the store
|
||||
issues it.
|
||||
|
||||
## The trust bundle
|
||||
|
||||
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
|
||||
the file every consumer anchors on. It holds one certificate, a copy of
|
||||
`swarm-services-root.pem`: the store's `pki` root, which the gateway leaf and
|
||||
the swarm-services leaf both chain to. It's the only root the swarm has.
|
||||
|
||||
**Without the store's root, the bundle is empty and its readers go red until
|
||||
it lands.**
|
||||
While `<deploy.hive-controller.tls.stateDir>/swarm-services-root.pem` is
|
||||
missing or empty, as on a first boot before the store answers,
|
||||
`hive-tls-ca.service` writes an empty bundle and logs that file's path. Every
|
||||
`<name>-ca-bundle` unit then fails with `<bundle> contributed no certificate
|
||||
to the bundle`, and the consumers that `Requires=` it don't start.
|
||||
`swarm-services-cert.service` reloads `hive-tls-ca.service` when it writes a
|
||||
new root. That rewrites the bundle in place and restarts no container, the
|
||||
store's included.
|
||||
|
||||
**Each `<name>-ca-bundle` unit converges on its own.** Beside it, a
|
||||
`<name>-ca-bundle-refresh.path` unit watches the file the bundle derives
|
||||
from: inside a container the bound `/run/hive-ca/trust-bundle.pem`, on the host
|
||||
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`. A write to it starts
|
||||
`<name>-ca-bundle-refresh.service`. That service does nothing while the file
|
||||
holds no certificate, or while it matches the copy the last assembly took
|
||||
(`/run/<name>-ca/anchor.pem`). Otherwise it restarts the bundle unit and starts
|
||||
its consumers without waiting. The restart reaches the running consumers
|
||||
through their `Requires=`; the start picks up those a first boot left failed.
|
||||
The same service runs once at boot, after the bundle unit, which covers a root
|
||||
that landed before the watch began. A first boot needs no second apply, and a
|
||||
root rotation restarts the consumers once and the container never. The watch
|
||||
fires for the host's in-place rewrite because inotify keys on the inode and a
|
||||
bind mount shares it, so `hive-tls-ca` must keep rewriting the file in place
|
||||
and never rename over it.
|
||||
|
||||
A deploy that changes `hive-tls-ca.service` reloads it too, with the same
|
||||
effect.
|
||||
|
||||
⚠️ **A consumer outside `trustBundle` reads its trust when it starts.** A
|
||||
container whose nix predates the refresh units keeps the bundle it started
|
||||
with until it restarts, and fails TLS to a gateway serving a store-issued leaf
|
||||
in the meantime.
|
||||
|
||||
## The swarm's service names come from the secret store
|
||||
|
||||
The swarm's service names (`swarm.forge.domain`,
|
||||
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
|
||||
siblings of the hive domain — `forge.<swarm.domain>`, not
|
||||
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
|
||||
**single-label** wildcard over it, and its role allows nothing outside the hive's
|
||||
domain, so it can't cover them.
|
||||
|
||||
The secret store's `pki` mount issues them a leaf of their own, through its
|
||||
`swarm-services` role. `swarm-services-cert.service` logs in with
|
||||
the services-issuer identity, calls
|
||||
`pki/issue/swarm-services`, and writes the result to
|
||||
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
|
||||
copies from. The store's role, not a key on the host, decides which names a
|
||||
gateway can serve.
|
||||
|
||||
**Every self-signed gateway that fronts a swarm name needs the
|
||||
services-issuer identity**, whether or not a hive runs on its host. A hive
|
||||
fronting none needs no copy of it: it fetches the store's root with its own
|
||||
store identity, as below. It's a client certificate
|
||||
from the store's own client CA, with the subject
|
||||
`deploy.bao.servicesIssuerCommonName` (`swarm-services-issuer` by default),
|
||||
and it's named by `services.hyperhive.deploy.bao.servicesIssuerClientCertFile`
|
||||
and `servicesIssuerClientKeyFile`. Its policy may issue from
|
||||
`pki/issue/swarm-services` and nothing else, and it's the same certificate on
|
||||
every gateway host:
|
||||
|
||||
- On the host running the store, `swarm-bao-pki.service` mints it at
|
||||
`/var/lib/swarm-bao-pki/services-issuer.pem` and the two options default to
|
||||
it.
|
||||
- On every other gateway host fronting a swarm name, copy
|
||||
`services-issuer.pem` and `services-issuer-key.pem` from the store's host
|
||||
and set both options.
|
||||
|
||||
A host fronting a swarm name without both options set still evaluates and deploys, and `swarm-services-cert.service` fails, naming
|
||||
them. One that names files it can't read fails the same unit, naming the
|
||||
paths. The old spellings,
|
||||
`deploy.hive-controller.tls.baoClientCertFile` and `baoClientKeyFile`, are
|
||||
renames of these two: a config that still sets them evaluates with a warning
|
||||
naming both paths.
|
||||
|
||||
`services.hyperhive.swarm.serviceDomains`, every swarm service name and the
|
||||
same list on every host, populates the role's
|
||||
`allowed_domains`, so asking for a name nobody configured is a refusal
|
||||
from the store naming that name — not a certificate quietly issued for
|
||||
it. A host's own SANs are narrower still: it asks only for the names it
|
||||
fronts a vhost for (`deploy.hive-controller.tls.localServiceDomains`), so the key on one
|
||||
gateway can't serve a swarm service that runs behind another.
|
||||
|
||||
**The root's public certificate is a file, on every self-signed gateway host:**
|
||||
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
|
||||
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
|
||||
swarm name asks for no services leaf and reads the root alone, from the store's
|
||||
`pki/ca/pem`, presenting its own store identity
|
||||
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
|
||||
`baoHiveIdentityKeyFile`) for the handshake without logging in with it.
|
||||
Every hive must hold one: without it, or with either file absent,
|
||||
`swarm-services-cert.service` fails with a message naming both options.
|
||||
On the host running
|
||||
the store it's also at
|
||||
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
|
||||
hand a browser, and reading it needs no store login — which matters,
|
||||
because every store listener but the loopback UI one demands a client
|
||||
certificate, and authelia gates that one to the `admins` group, not
|
||||
open to an anonymous browser fetching a trust anchor.
|
||||
|
||||
**The granting unit generates the root once, and never again.** It asks
|
||||
the mount whether it already has an issuer (`bao list pki/issuers`)
|
||||
before calling `pki/root/generate/internal`, so a rebuild or a reboot
|
||||
re-asserts the role and the grant without touching the anchor. A root
|
||||
that changed per boot would invalidate every certificate issued under it
|
||||
and every browser taught to trust it.
|
||||
|
||||
**A daily timer renews the leaf.** The store's role issues it for 720
|
||||
hours. `swarm-services-cert-renew.timer` runs the same script as
|
||||
`swarm-services-cert.service` once a day, and that script asks the store
|
||||
for a new leaf once the current one is past half that window, is
|
||||
missing a configured name, or doesn't chain to the store's root
|
||||
(`swarm-services-root.pem`). A rotation re-imports the leaf into the
|
||||
gateway and reloads nginx. A store that can't answer — still sealed, or
|
||||
out of reach — fails the unit, which leaves the current leaf in place,
|
||||
and the next day's run tries again: about fifteen tries before the leaf
|
||||
lapses. `systemctl
|
||||
list-timers swarm-services-cert-renew` shows the next run.
|
||||
|
||||
## Constraints on the material
|
||||
|
||||
No CA private key exists on any hive. The `pki` root's key stays inside the
|
||||
store's mount (`root/generate/internal`), and each hive's narrowing is its
|
||||
role's `allowed_domains`, enforced by the store at issue time: a hive's
|
||||
identity that leaks can get certificates for that hive's names and no
|
||||
others.
|
||||
|
||||
No host holds a CA or its key, except the store's mTLS CA
|
||||
(`glue-bao-tls.nix`), which signs only the store's server certificate and the
|
||||
certificates that log in to it. Every host deletes `/var/lib/swarm-ca` and
|
||||
the `ca.pem`, `ca-previous.pem`, `ca-key.pem`, `ca.srl` and
|
||||
`.swarm-ca-adopted` in
|
||||
`<deploy.hive-controller.tls.stateDir>`: `systemd-tmpfiles` removes them at
|
||||
boot, and on a deploy that changes the host's `tmpfiles.d` rules. The swarm
|
||||
distributes only certificates.
|
||||
|
||||
⚠️ **The deletion is irreversible.** Rolling back to a generation that still
|
||||
has `swarm-ca.service` finds no root under `/var/lib/swarm-ca` and mints a
|
||||
new one, unrelated to the deleted root. That's harmless: nothing issues
|
||||
under it.
|
||||
|
||||
A host whose `deploy.bao.clientCaFile`, `deploy.bao.serverCaFile` or
|
||||
`deploy.swarm-controller.hiveClientCaFile` names one of those paths fails
|
||||
evaluation, however it's spelled with `//`, `.`, `..` or a trailing `/`. A
|
||||
symlink to one isn't resolved, so it gets past the check. Those options name
|
||||
the store's mTLS CA, `/var/lib/swarm-bao-pki/ca.pem` on the host that runs
|
||||
the store.
|
||||
|
||||
## What to hand a peer
|
||||
|
||||
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
|
||||
block a peer operator pastes into their own config. When this hive's
|
||||
gateway serves a self-signed leaf (detected by the presence of
|
||||
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints the
|
||||
secret store's `pki` root itself, read from
|
||||
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
|
||||
gateway leaf and the swarm's service names under that root, so it's the one
|
||||
anchor a peer host needs.
|
||||
|
||||
The operator installs the root **once per swarm host**, not once per peer,
|
||||
so a hive joining later needs no edit on the hives already running. A hive
|
||||
whose cert already chains to a public CA has nothing to install —
|
||||
`peer-config` says so instead.
|
||||
|
||||
## Distributing the root
|
||||
|
||||
The root **certificate** is a runtime file: each hive fetches it from the
|
||||
store at boot, so nothing can name it at build time. That has a cost worth
|
||||
naming, because it's subtle and it bites at a distance:
|
||||
|
||||
> **Nothing whose trust store nix assembles at build time can reference
|
||||
> the store's root.** The derivation reads `security.pki.certificateFiles`
|
||||
> inside itself; the root doesn't exist there.
|
||||
|
||||
Two consumers, both reading the root at runtime:
|
||||
|
||||
- **Agents fetch it themselves.** `hive-agent-trust-bundle.service` (`nix/agent-modules/hive-ca.nix`) reads `<mount>/ca/pem` from the store with the agent's own store certificate, then writes `/run/hive-agent-trust-bundle/bundle.pem` (system CAs plus the root) and `root.pem` (the root alone). The agent's units point `SSL_CERT_FILE` at the first (Rust, Go and OpenSSL clients) and `NODE_EXTRA_CA_CERTS` at the second (Node and Claude Code), and order after the unit with `Requires=`. The fetch retries inside its start job for up to 14 minutes, with a wait that doubles from 2 to 60 seconds, so a store name that doesn't resolve yet delays those units instead of failing them; after that the job fails and they fail with it. The meta flake names no CA file, and a timer re-reads the root hourly. A consumer reads it at start, so a rotated root reaches an agent unit at its next start.
|
||||
- **The Matrix container gets a bind mount**, via the shared
|
||||
`lib/hive-ca-trust.nix` helper that `hive-forge` and `swarm-ci-runner`
|
||||
use: the bundle is bind-mounted read-only into the container, and a
|
||||
oneshot concatenates it with the system CAs before tuwunel starts.
|
||||
|
||||
The consumption differs per runtime and is the part worth knowing.
|
||||
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
|
||||
— it's not: its rustls-based TLS stack still resolves trust through the
|
||||
platform certificate store on Linux, and that store honors
|
||||
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
|
||||
an OpenSSL-linked binary.
|
||||
|
||||
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
|
||||
> `openssl-probe` uses it _instead of_ the default store, so naming
|
||||
> just the bundle would drop every public CA and break federation with
|
||||
> the wider matrix network — a much bigger outage than the one being
|
||||
> fixed. The same caveat applies to `hive-forge` (Go) for the same
|
||||
> reason.
|
||||
|
||||
A hive in the swarm fetches the root itself. A host outside it gets the
|
||||
certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)).
|
||||
One anchor per host, not one per peer: a hive joining later needs no edit
|
||||
on the hives already running.
|
||||
Loading…
Reference in a new issue