0649673ebf
The store's `swarm-services` role issues the services leaf for 720h, and `swarm-services-cert` only ever ran at boot or rebuild: it is a `RemainAfterExit` oneshot wanted by `multi-user.target` and no timer targeted it. A hive not rebuilt within 30 days served an expired leaf. `swarm-services-cert-renew` runs the same script from a daily timer. It is a unit of its own because a timer starting the `RemainAfterExit` unit is a no-op, and restarting that unit instead would propagate through `hive-gateway-self-signed-cert`'s `Requires=` to nginx, so a sealed store would take the gateway down over a still-valid leaf. Nothing requires or orders against the new unit; it has no `Restart=`, so a failure stays in `systemctl --failed` until the next tick, and the script only moves files into place after the store has answered. The re-issue threshold was `checkend 2592000`, the whole 30-day lifetime, so every run re-issued. It is now half the role's lifetime, read from a new internal option `deploy.bao.servicesPkiLeafTtlHours` that the role's `ttl`/`max_ttl` also read. Boot and timer share the script and so the threshold. The services-root re-check reads the same option, at the store's own replacement threshold (hours × 3600), so the hive asks for a new leaf when the store replaces its root. A `flock` keeps the two runs from interleaving one issuance's key with another's leaf. `checks.module-eval-hive-tls` pins the timer, that the unit it starts re-runs the issuance without `RemainAfterExit`, that nothing depends on it, and that both the leaf and root thresholds move with the option. Closes #4587
236 lines
11 KiB
Markdown
236 lines
11 KiB
Markdown
# Swarm CA
|
|
|
|
A hive's internal TLS chains to a **swarm root CA**: the root signs each
|
|
hive's own CA, and that hive CA signs the gateway leaf. A peer that
|
|
trusts the root once validates every hive in the swarm, present and
|
|
future, instead of pinning to each one by hand.
|
|
|
|
That's the whole point of the hierarchy — it turns per-peer trust from
|
|
O(n²) hand-pinning into one anchor per swarm.
|
|
|
|
## Two provisioning modes, one structure
|
|
|
|
What differs is who puts the artifacts on disk:
|
|
|
|
| | swarm root | this hive's CA |
|
|
| ---------------------------------------- | --------------------------------------------- | ---------------------------------------------- |
|
|
| default | operator-provided | operator-provided, else self-signed as before |
|
|
| `autoConfigure = true` (all on one host) | generated by `swarm-ca.service` on first boot | issued by `hive-tls-ca.service` under the root |
|
|
|
|
`services.hyperhive.swarm.ca.autoConfigure` selects between them, and is
|
|
**off by default**: a swarm's services and its hives can live on
|
|
different hosts, and a host can't tell whether it's the one holding
|
|
the root, so setting the swarm CA up is an operator action rather than
|
|
something a host assumes. Turn it on for an all-on-one-host deployment
|
|
and the hierarchy costs no configuration.
|
|
|
|
It defaults from `services.hyperhive.deploy.singleHostSwarm`, the single
|
|
switch that says "this box is the whole deployment."
|
|
|
|
**A hive given neither artifact keeps the self-signed CA it has always
|
|
had.** It serves TLS exactly as before and isn't part of a
|
|
swarm's trust hierarchy — the right outcome for a hive nobody has
|
|
federated yet. Only `autoConfigure` issues a hive sub-CA, because only
|
|
that case can: signing one needs the root's private key.
|
|
|
|
Moving the swarm CA onto its own host is then a matter of moving
|
|
`services.hyperhive.swarm.ca.stateDir` and leaving `autoConfigure` off —
|
|
there is no second code path to switch to.
|
|
|
|
### The swarm's service names come from the secret store
|
|
|
|
The swarm's service names (`swarm.forge.domain`,
|
|
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
|
|
siblings of the hive domain — `forge.<swarm.domain>`, not
|
|
`forge.<hive domain>`. The hive CA's leaf is a **single-label** wildcard
|
|
over the hive's own domain, so it can't cover them.
|
|
|
|
The secret store's `pki` mount issues that leaf instead, from a
|
|
root generated inside it. `swarm-services-cert.service` logs in with
|
|
this host's services-issuer certificate, calls
|
|
`pki/issue/swarm-services`, and writes the result to
|
|
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
|
|
copies from. Every hive does this with its own identity, so holding the
|
|
swarm root's private key stopped being what decides whether a hive can
|
|
serve its swarm's names.
|
|
|
|
The same `services.hyperhive.swarm.serviceDomains` that builds the SANs
|
|
also populates the role's `allowed_domains`, so asking for a name nobody
|
|
configured is a refusal from the store naming that name — not a
|
|
certificate quietly issued for it.
|
|
|
|
**The root's public certificate is a file, on every hive:**
|
|
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
|
|
leaf it signed and folded into `trust-bundle.pem`. On the host running
|
|
the store it's also at
|
|
`/var/lib/swarm-bao-services-pki/services-root.pem`. That's the file to
|
|
hand a browser, and reading it needs no store login — which matters,
|
|
because every store listener demands a client certificate.
|
|
|
|
**The granting unit generates the root once, and never again.** It asks
|
|
the mount whether it already has an issuer (`bao list pki/issuers`)
|
|
before calling `pki/root/generate/internal`, so a rebuild or a reboot
|
|
re-asserts the role and the grant without touching the anchor. A root
|
|
that changed per boot would invalidate every certificate issued under it
|
|
and every browser taught to trust it.
|
|
|
|
**A daily timer renews the leaf.** The store's role issues it for 720
|
|
hours. `swarm-services-cert-renew.timer` runs the same script as
|
|
`swarm-services-cert.service` once a day, and that script asks the store
|
|
for a new leaf once the current one is past half that window or is
|
|
missing a configured name. A rotation re-imports the leaf into the
|
|
gateway and reloads nginx. A store that can't answer — still sealed, or
|
|
out of reach — fails the unit, which leaves the current leaf in place,
|
|
and the next day's run tries again: about fifteen tries before the leaf
|
|
lapses. `systemctl
|
|
list-timers swarm-services-cert-renew` shows the next run.
|
|
|
|
## Constraints on the material
|
|
|
|
The root's private key never reaches the nix store: the store is
|
|
world-readable and content-addressed, so a key committed to a flake is a
|
|
key published to everyone who builds it. The swarm distributes only
|
|
certificates.
|
|
|
|
Each hive CA is **name-constrained** (X.509 `nameConstraints`) to that
|
|
hive's own domain, so a hive CA that leaks can only mint names inside
|
|
its own subdomain — enforced by every verifier rather than by
|
|
convention. The constraint excludes both IP families as well, since a
|
|
permitted-DNS-only constraint says nothing about IP SANs.
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
The root is issued with `pathlen:1`: it may sign hive CAs, and those may
|
|
sign leaves, and the chain stops there.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
## What to hand a peer
|
|
|
|
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
|
|
block a peer operator pastes into their own config. When this hive's
|
|
gateway serves a self-signed leaf under the hierarchy (detected by the
|
|
presence of `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it
|
|
also prints a one-time
|
|
`scp` line installing the **swarm root** —
|
|
`<swarm.ca.stateDir>/root.pem`, not this hive's own CA — on the peer's
|
|
host:
|
|
|
|
```
|
|
scp /var/lib/swarm-ca/root.pem <peer-host>:/var/lib/swarm-ca/root.pem
|
|
```
|
|
|
|
That's the point of the hierarchy: the operator installs the root
|
|
**once per swarm host**, not once per peer, so a hive joining later
|
|
needs no edit on the hives already running. A hive whose cert already chains to a public
|
|
CA has nothing to install — `peer-config` says so instead.
|
|
|
|
Handing a peer this hive's own `ca.pem` would not work even as a
|
|
one-off: once a hive CA is an intermediate under the swarm root, it's
|
|
no longer something a verifier can build a chain _to_ — OpenSSL won't
|
|
terminate a chain at a trusted non-self-signed certificate without
|
|
`-partial_chain`. That's why the tool distributes the root, not a
|
|
per-hive file.
|
|
|
|
## Adopting the hierarchy on an existing hive
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
A hive that predates the swarm root carries a self-signed `ca.pem`, and
|
|
adopting the hierarchy means replacing it. That invalidates an anchor
|
|
consumers already trust, and they refresh on their own schedule — agents
|
|
only pick up new trust when their container restarts, peers only on
|
|
their own rebuild. **Who is allowed to decide that's what splits the
|
|
two cases.**
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
### Where this host owns the root (`autoConfigure`)
|
|
|
|
Adoption happens by itself, once. `hive-tls-ca.service` notices that
|
|
`ca.pem` doesn't chain to the root, keeps the old certificate as
|
|
`ca-previous.pem`, and re-issues under the root; the new CA signs the
|
|
next leaf.
|
|
|
|
It's safe to automate here precisely because this is the
|
|
all-on-one-host shape: every consumer is on this box, so "when will they
|
|
have refreshed" is knowable rather than guessed.
|
|
|
|
The old CA stays in `trust-bundle.pem` afterwards, so adoption is
|
|
**additive to the anchor set before it's subtractive** — a container
|
|
that hasn't restarted yet still validates. Removing `ca-previous.pem`
|
|
is a deliberate later step: how long is long enough is a property of the
|
|
deployment, not something the unit can know.
|
|
|
|
A marker file (`.swarm-ca-adopted`) records that this ran. Its _absence_
|
|
is the trigger, so adoption fires once per hive rather than being
|
|
re-decided on every activation.
|
|
|
|
### Everywhere else
|
|
|
|
No automatic adoption. `hive-tls-ca.service` fails, loudly, naming both
|
|
certificates and giving the two-command recipe:
|
|
|
|
```
|
|
rm <deploy.hive-controller.tls.stateDir>/ca.pem <deploy.hive-controller.tls.stateDir>/ca-key.pem
|
|
systemctl restart hive-tls-ca.service
|
|
```
|
|
|
|
<!-- vale write-good.Passive = NO -->
|
|
|
|
Failing rather than warning is deliberate: a hive whose CA doesn't
|
|
chain to the root it received is misconfigured, and a warning in a
|
|
build log isn't something anyone reads twice.
|
|
|
|
<!-- vale write-good.Passive = YES -->
|
|
|
|
To keep the current CA on purpose — a hive that deliberately stays
|
|
outside the hierarchy, or one mid-migration — `touch` the marker file
|
|
named in the message. That's a decision, and it's recorded as one.
|
|
|
|
A hive with **no** root configured at all isn't affected by any of
|
|
this: it self-signs exactly as it always has.
|
|
|
|
## Distributing the root
|
|
|
|
The root **key** is a runtime file for the reason in
|
|
[Constraints on the material](#constraints-on-the-material).
|
|
|
|
The root **certificate** is a runtime file as a _consequence_ — it lives
|
|
beside the key under `swarm.ca.stateDir` — and that has a cost worth
|
|
naming, because it's subtle and it bites at a distance:
|
|
|
|
> **Nothing whose trust store nix assembles at build time can reference
|
|
> the swarm root.** The derivation reads `security.pki.certificateFiles`
|
|
> inside itself; the root doesn't exist there.
|
|
|
|
Two consumers, and only one of them is fine:
|
|
|
|
- **`hive-tls.nix` covers agents.** It folds the root into this hive's
|
|
`trust-bundle.pem`, hive-c0re receives that path as
|
|
`HIVE_TLS_CA_PATH`, and the meta-flake renderer embeds that one file
|
|
next to every agent's flake. The bundle is the runtime-to-build-time
|
|
bridge.
|
|
- **The Matrix container crosses the same bridge**, via the shared
|
|
`lib/hive-ca-trust.nix` helper that `hive-ci` and `hive-forge` already
|
|
use: the bundle is bind-mounted read-only into the container, and a
|
|
oneshot concatenates it with the system CAs before tuwunel starts.
|
|
|
|
The consumption differs per runtime and is the part worth knowing.
|
|
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
|
|
— it's not: its rustls-based TLS stack still resolves trust through the
|
|
platform certificate store on Linux, and that store honors
|
|
`SSL_CERT_FILE`, so the env var takes effect the same way it would for
|
|
an OpenSSL-linked binary.
|
|
|
|
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
|
|
> `openssl-probe` uses it _instead of_ the default store, so naming
|
|
> just the bundle would drop every public CA and break federation with
|
|
> the wider matrix network — a much bigger outage than the one being
|
|
> fixed. The same caveat applies to `hive-forge` (Go) for the same
|
|
> reason.
|
|
|
|
To put the root on another host, copy the certificate to the same path
|
|
there (`scp <stateDir>/root.pem <host>:<stateDir>/root.pem`). One anchor
|
|
per host, not one per peer: a hive joining later needs no edit on the
|
|
hives already running, which is the whole point of the hierarchy.
|