vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-tls: issue the gateway leaf from bao with a per-hive identity

Browse source 
Every hive gets its own store identity (CN
<deploy.bao.hiveLeafCommonNamePrefix>-<hive>) and its own role on the
existing pki mount (<deploy.bao.hiveLeafPkiRolePrefix>-<hive>), whose
allowed_domains is the hive's domain and *.<domain>. The policy grants
update on that one issue path. swarm-bao-hive-leaf-policy writes role,
policy and cert-auth role for every hive in swarm.hives; the store's host
mints its own hive's identity in glue-bao-tls.nix and a new glue file
points deploy.hive-controller.tls.baoHiveIdentity{Cert,Key}File at it.
Hives on other hosts place it by hand. swarm.nix reserves the composed CN.

hive-tls-ca no longer generates a hive CA or signs a leaf: caGenScript,
signLeafScript/signHiveLeaf and the adoption block are gone, ca-key.pem
is deleted where it exists, and the unit only assembles trust-bundle.pem.
The retired ca.pem/ca-previous.pem stay in the bundle so leaves they
signed keep verifying.

hive-tls-resign becomes the bao issuance: at boot, on deploy and from a
daily timer it re-issues when the leaf is missing, past half the role's
lifetime, missing a name, or not chained to the store's pki root. Nothing
orders the gateway after it, so a valid leaf on disk never waits on the
store; a failed renewal leaves the leaf in place. A missing identity
fails the unit naming both options, as does the root fetch on a hive
fronting no swarm name, which now presents this identity instead of the
reader cert.

caValidityDays and leafValidityDays are removed (both spellings), since
nothing reads them.

Refs #4588
This commit is contained in:
atlas 2026-10-04 15:11:48 +02:00 • committed by mara
commit 5526daab97
16 changed files with 859 additions and 628 deletions
Download patch file Download diff file Expand all files Collapse all files

14
docs/networking/gateway.md
View file

@ -46,7 +46,7 @@ The gateway always terminates TLS. Which certificate it serves depends on what y
| mode | config | cert source | `.well-known` scheme | | mode | config | cert source | `.well-known` scheme |
|---|---|---|---| |---|---|---|---|
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | the host's hive CA signs a gateway leaf (RSA-4096) | `https` | | self-signed (default) | neither `tls.certDir` nor `tls.acme` set | the secret store issues a gateway leaf through the hive's own role (RSA-4096) | `https` |
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` | | ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
| operator cert | `tls.certDir` set | read from the operator's dir | `https` | | operator cert | `tls.certDir` set | read from the operator's dir | `https` |
@ -74,19 +74,19 @@ nginx obtains and renews certs via the HTTP-01 challenge on `port` (default 80);
On by default, listening on `httpsPort` (default 443) beside the plain-http `port` (default 80). On by default, listening on `httpsPort` (default 443) beside the plain-http `port` (default 80).
The issuer is a **host-held hive CA**, not a bare self-signed leaf. `hive-tls-ca.service` (from `hive-tls.nix`) generates a long-lived CA (`services.hyperhive.deploy.hive-controller.tls.caValidityDays`, default 7300 days) under `…tls.stateDir` (default `/var/lib/hive-tls`) and signs a gateway **leaf** with it (`leafValidityDays`, default 30). `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. `hive-tls-resign.service` (from `hive-tls.nix`) asks the store for a gateway **leaf** through this hive's own role, presenting the hive's own store identity (`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile`), and writes it under `…tls.stateDir` (default `/var/lib/hive-tls`). `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). [`swarm/ca.md`](../swarm/ca.md) has the role, the identity and what a hive without one does.
⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the CA's `0600 root:root` key), and it makes sure **every cert path the config names exists** — when the swarm-services leaf is missing it installs the hive leaf in its place. nginx refuses a config naming a missing cert file, so without that fallback one missing leaf takes down every vhost, not just one. ⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the source's `0600 root:root` key), and it makes sure **every cert path the config names exists** — when the swarm-services leaf is missing it installs the hive leaf in its place. nginx refuses a config naming a missing cert file, so without that fallback one missing leaf takes down every vhost, not just one.
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the stable CA once; leaf rotation never breaks them. **Why an issuer, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the store's root once; leaf rotation never breaks them.
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is an intermediate under the swarm root ([`swarm/ca.md`](../swarm/ca.md) has the hierarchy), and a verifier can't stop at an intermediate — so the bundle carries the hive CA plus its root. nginx serves the leaf with the hive CA appended for the same reason. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle. **What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root, plus the retired hive CA's public `ca.pem` and the swarm root where they exist ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle.
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https://<host>/.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap. **Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https://<host>/.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap.
**Cert shape**: the leaf's CN is the bare hive domain; its SANs are `<hive>` and `*.<hive>`. The hive CA is name-constrained to `<hive>`, so it can't sign a swarm-service name outside it — a violating SAN would invalidate the whole leaf. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)). **Cert shape**: the leaf's CN is the bare hive domain; its SANs are `<hive>` and `*.<hive>`. The hive's role allows nothing outside `<hive>`, so a swarm-service name can't go on this leaf. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
**Rotation**: `hive-tls-ca.service` re-signs the leaf when it's missing, within 30 days of expiry, or no longer covers the configured names, always under the same CA. It regenerates the CA only if missing or expired. To force a leaf rotation, delete `gateway.pem` under the state dir, restart the unit, then reload `nginx`. **Rotation**: `hive-tls-resign.service` runs at boot, on every deploy and once a day, and asks the store for a new leaf when the current one is missing, past half its 720-hour lifetime, no longer covers the configured names, or doesn't chain to the store's root. A leaf already on disk lets the gateway start while the store can't answer; a failed renewal leaves it in place. To force a rotation, delete `gateway.pem` under the state dir and restart the unit; it re-imports the leaf and reloads `nginx` itself.
**Cert prompts**: browsers warn once per host until you add the hive's `trust-bundle.pem` (the anchor, not the leaf) to the browser or OS trust store. A container that needs to trust it for its own outbound TLS gets the bundle a different way, bind-mounted read-only at `/run/hive-ca/trust-bundle.pem` (`nix/host-modules/lib/hive-ca-trust.nix`). **Cert prompts**: browsers warn once per host until you add the hive's `trust-bundle.pem` (the anchor, not the leaf) to the browser or OS trust store. A container that needs to trust it for its own outbound TLS gets the bundle a different way, bind-mounted read-only at `/run/hive-ca/trust-bundle.pem` (`nix/host-modules/lib/hive-ca-trust.nix`).

2
docs/swarm/README.md
View file

@ -33,7 +33,7 @@ grafana, the controller, authelia, …) →
lives → [`secrets.md`](secrets.md) · the per-secret minter/reader/renewal lives → [`secrets.md`](secrets.md) · the per-secret minter/reader/renewal
contract → [`credentials.md`](credentials.md) · how the store comes up, contract → [`credentials.md`](credentials.md) · how the store comes up,
who writes its grants and how it unseals → [`bao.md`](bao.md) who writes its grants and how it unseals → [`bao.md`](bao.md)
- **swarm CA** — the root every hive's internal TLS chains to, and what to - **swarm CA** — the store root every hive's gateway leaf chains to, and what to
hand a peer (`trust-bundle.pem`, never `ca.pem`). → [`ca.md`](ca.md) hand a peer (`trust-bundle.pem`, never `ca.pem`). → [`ca.md`](ca.md)
- **snapshot store** — the swarm's one `btrfs receive` endpoint, - **snapshot store** — the swarm's one `btrfs receive` endpoint,
`swarm.snapshotStore.{address,port}`. → `swarm.snapshotStore.{address,port}`. →

231
docs/swarm/ca.md
View file

@ -1,52 +1,100 @@
# Swarm CA # Swarm CA
A hive's internal TLS chains to a **swarm root CA**: the root signs each A hive's gateway serves a leaf issued by the secret store's `pki` mount,
hive's own CA, and that hive CA signs the gateway leaf. A peer that through a role that belongs to that hive. A peer that trusts the mount's
trusts the root once validates every hive in the swarm, present and root once validates every hive's gateway and every swarm service name,
future, instead of pinning to each one by hand. present and future, instead of pinning to each one by hand.
That's the whole point of the hierarchy — it turns per-peer trust from That's the point of one issuer — it turns per-peer trust from O(n²)
O(n²) hand-pinning into one anchor per swarm. hand-pinning into one anchor per swarm.
## Two provisioning modes, one structure ## The gateway leaf comes from the secret store
What differs is who puts the artifacts on disk: `hive-tls-resign.service` logs in to the store with this hive's own
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
the files the gateway copies from. The leaf's subject is the hive domain,
and its SANs are the hive domain and `*.<hive domain>`.
| | swarm root | this hive's CA | **Each hive's role covers that hive alone.** Its `allowed_domains` is the
| ---------------------------------------- | --------------------------------------------- | ---------------------------------------------- | hive's domain and `*.<hive domain>`, read from
| default | operator-provided | operator-provided, else self-signed as before | `services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
| `autoConfigure = true` (all on one host) | generated by `swarm-ca.service` on first boot | issued by `hive-tls-ca.service` under the root | IP SANs and no `localhost`. A request for any other name is a refusal from
the store naming that name. The hive's policy grants `update` on that one
issue path and nothing else. `swarm-bao-hive-leaf-policy.service`, on the
store's host, writes the role, the policy and the cert-auth role for every
hive in `swarm.hives`.
`services.hyperhive.swarm.ca.autoConfigure` selects between them, and is **Every hive needs its own store identity.** It's a client certificate
**off by default**: a swarm's services and its hives can live on from the store's own client CA, with the subject
different hosts, and a host can't tell whether it's the one holding `<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
the root, so setting the swarm CA up is an operator action rather than (`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
something a host assumes. Turn it on for an all-on-one-host deployment `services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
and the hierarchy costs no configuration. `baoHiveIdentityKeyFile`:
It defaults from `services.hyperhive.deploy.singleHostSwarm`, the single - On the host running the store, `swarm-bao-pki.service` mints it at
switch that says "this box is the whole deployment." `/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
- On every other host, copy a leaf with that subject from the store's host
and set both options, before the host's next deploy.
**A hive given neither artifact keeps the self-signed CA it has always Without it, `hive-tls-resign.service` fails with a message naming both
had.** It serves TLS exactly as before and isn't part of a options, and so does `swarm-services-cert.service` on a hive fronting no
swarm's trust hierarchy — the right outcome for a hive nobody has swarm name. The unit checks the identity before the leaf, so it fails even while a
federated yet. Only `autoConfigure` issues a hive sub-CA, because only valid leaf keeps the gateway up. The identity is the hive's alone:
that case can: signing one needs the root's private key. it issues this hive's leaf and fetches the store's root, and it's never the
identity a swarm service uses.
Moving the swarm CA onto its own host is then a matter of moving **When it issues.** The unit runs at boot, on every deploy, and from
`services.hyperhive.swarm.ca.stateDir` and leaving `autoConfigure` off — `hive-tls-resign.timer` once a day. It asks the store for a new leaf when
there is no second code path to switch to. the current one is missing, past half the role's 720-hour lifetime, missing
a configured name, or doesn't chain to the store's root
(`swarm-services-root.pem`). That last check is what replaces a leaf the
retired hive CA signed, on the first run that reaches the store.
### The swarm's service names come from the secret store **The gateway doesn't wait for the store.** Nothing orders the gateway after
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
current leaf in place, says so in the journal, and retries every 30 seconds
for a day. Only a hive with no leaf yet needs the store reachable and
unsealed: its gateway's certificate import fails until the first leaf
lands, and then the unit starts nginx.
nginx serves the leaf alone, without its issuer: the issuer is the
self-signed root a consumer already holds in its trust bundle.
## The trust bundle
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
the file every consumer anchors on. It holds whichever of these exist:
- `swarm-services-root.pem`, the store's `pki` root, which the gateway leaf
and the swarm-services leaf both chain to.
- `ca.pem` and `ca-previous.pem`, the retired hive CA's public certificates,
so a leaf it signed keeps verifying until it's replaced. Removing them is a
deliberate later step.
- the swarm root, `<swarm.ca.stateDir>/root.pem`, on the host that generates
it.
No hive keeps a CA key. `hive-tls-ca.service` deletes
`<deploy.hive-controller.tls.stateDir>/ca-key.pem` where an earlier
generation left one, and generates no CA.
⚠️ **A consumer reads its trust when it starts.** An agent or service
container that last started before the store's root reached the bundle
fails TLS to a gateway serving a store-issued leaf, until it restarts.
## The swarm's service names come from the secret store
The swarm's service names (`swarm.forge.domain`, The swarm's service names (`swarm.forge.domain`,
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to `swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
siblings of the hive domain — `forge.<swarm.domain>`, not siblings of the hive domain — `forge.<swarm.domain>`, not
`forge.<hive domain>`. The hive CA's leaf is a **single-label** wildcard `forge.<hive domain>`. The gateway leaf carries the hive's domain and a
over the hive's own domain, so it can't cover them. **single-label** wildcard over it, and its role allows nothing outside the hive's
domain, so it can't cover them.
The secret store's `pki` mount issues that leaf instead, from a The secret store's `pki` mount issues them a leaf of their own, through its
root generated inside it. `swarm-services-cert.service` logs in with `swarm-services` role. `swarm-services-cert.service` logs in with
this host's services-issuer certificate, calls this host's services-issuer certificate, calls
`pki/issue/swarm-services`, and writes the result to `pki/issue/swarm-services`, and writes the result to
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already `/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
@ -65,10 +113,11 @@ gateway can't serve a swarm service that runs behind another.
**The root's public certificate is a file, on every hive:** **The root's public certificate is a file, on every hive:**
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the `/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
swarm name asks for no leaf and reads the root alone, from the store's swarm name asks for no services leaf and reads the root alone, from the store's
`pki/ca/pem`, presenting its store identity `pki/ca/pem`, presenting its own store identity
(`deploy.bao.clientCertFile` / `clientKeyFile`) for the handshake. Every hive (`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
must hold one: without it, or with either file absent, `baoHiveIdentityKeyFile`) for the handshake. Every hive must hold one:
without it, or with either file absent,
`swarm-services-cert.service` fails with a message naming both options. `swarm-services-cert.service` fails with a message naming both options.
On the host running On the host running
the store it's also at the store it's also at
@ -98,32 +147,24 @@ list-timers swarm-services-cert-renew` shows the next run.
## Constraints on the material ## Constraints on the material
The root's private key never reaches the nix store: the store is No CA private key exists on any hive. The `pki` root's key stays inside the
store's mount (`root/generate/internal`), and each hive's narrowing is its
role's `allowed_domains`, enforced by the store at issue time: a hive's
identity that leaks can get certificates for that hive's names and no
others.
The swarm root's private key never reaches the nix store: the store is
world-readable and content-addressed, so a key committed to a flake is a world-readable and content-addressed, so a key committed to a flake is a
key published to everyone who builds it. The swarm distributes only key published to everyone who builds it. The swarm distributes only
certificates. certificates.
Each hive CA is **name-constrained** (X.509 `nameConstraints`) to that
hive's own domain, so a hive CA that leaks can only mint names inside
its own subdomain — enforced by every verifier rather than by
convention. The constraint excludes both IP families as well, since a
permitted-DNS-only constraint says nothing about IP SANs.
<!-- vale write-good.Passive = NO -->
The root is issued with `pathlen:1`: it may sign hive CAs, and those may
sign leaves, and the chain stops there.
<!-- vale write-good.Passive = YES -->
## What to hand a peer ## What to hand a peer
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"` `hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
block a peer operator pastes into their own config. When this hive's block a peer operator pastes into their own config. When this hive's
gateway serves a self-signed leaf under the hierarchy (detected by the gateway serves a self-signed leaf (detected by the presence of
presence of `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints a
also prints a one-time one-time `scp` line installing the **swarm root** —
`scp` line installing the **swarm root** —
`<swarm.ca.stateDir>/root.pem`, not this hive's own CA — on the peer's `<swarm.ca.stateDir>/root.pem`, not this hive's own CA — on the peer's
host: host:
@ -132,80 +173,14 @@ scp /var/lib/swarm-ca/root.pem <peer-host>:/var/lib/swarm-ca/root.pem
``` ```
It then prints the secret store's `pki` root itself, read from It then prints the secret store's `pki` root itself, read from
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues the swarm's `/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
service names under that root, not the swarm root, so a peer host trusting only gateway leaf and the swarm's service names under that root, so a peer host
the swarm root can't verify them. trusting only the swarm root can't verify them.
That's the point of the hierarchy: the operator installs the root The operator installs the roots **once per swarm host**, not once per peer,
**once per swarm host**, not once per peer, so a hive joining later so a hive joining later needs no edit on the hives already running. A hive
needs no edit on the hives already running. A hive whose cert already chains to a public whose cert already chains to a public CA has nothing to install —
CA has nothing to install — `peer-config` says so instead. `peer-config` says so instead.
Handing a peer this hive's own `ca.pem` would not work even as a
one-off: once a hive CA is an intermediate under the swarm root, it's
no longer something a verifier can build a chain _to_ — OpenSSL won't
terminate a chain at a trusted non-self-signed certificate without
`-partial_chain`. That's why the tool distributes the root, not a
per-hive file.
## Adopting the hierarchy on an existing hive
<!-- vale write-good.Passive = NO -->
A hive that predates the swarm root carries a self-signed `ca.pem`, and
adopting the hierarchy means replacing it. That invalidates an anchor
consumers already trust, and they refresh on their own schedule — agents
only pick up new trust when their container restarts, peers only on
their own rebuild. **Who is allowed to decide that's what splits the
two cases.**
<!-- vale write-good.Passive = YES -->
### Where this host owns the root (`autoConfigure`)
Adoption happens by itself, once. `hive-tls-ca.service` notices that
`ca.pem` doesn't chain to the root, keeps the old certificate as
`ca-previous.pem`, and re-issues under the root; the new CA signs the
next leaf.
It's safe to automate here precisely because this is the
all-on-one-host shape: every consumer is on this box, so "when will they
have refreshed" is knowable rather than guessed.
The old CA stays in `trust-bundle.pem` afterwards, so adoption is
**additive to the anchor set before it's subtractive** — a container
that hasn't restarted yet still validates. Removing `ca-previous.pem`
is a deliberate later step: how long is long enough is a property of the
deployment, not something the unit can know.
A marker file (`.swarm-ca-adopted`) records that this ran. Its _absence_
is the trigger, so adoption fires once per hive rather than being
re-decided on every activation.
### Everywhere else
No automatic adoption. `hive-tls-ca.service` fails, loudly, naming both
certificates and giving the two-command recipe:
```
rm <deploy.hive-controller.tls.stateDir>/ca.pem <deploy.hive-controller.tls.stateDir>/ca-key.pem
systemctl restart hive-tls-ca.service
```
<!-- vale write-good.Passive = NO -->
Failing rather than warning is deliberate: a hive whose CA doesn't
chain to the root it received is misconfigured, and a warning in a
build log isn't something anyone reads twice.
<!-- vale write-good.Passive = YES -->
To keep the current CA on purpose — a hive that deliberately stays
outside the hierarchy, or one mid-migration — `touch` the marker file
named in the message. That's a decision, and it's recorded as one.
A hive with **no** root configured at all isn't affected by any of
this: it self-signs exactly as it always has.
## Distributing the root ## Distributing the root

29
docs/swarm/secrets.md
View file

@ -155,14 +155,14 @@ when the system builds. The server names the offending file and refuses to run.
## Hive-level — one of each per hive ## Hive-level — one of each per hive
| secret | generated by | lives at | | secret | generated by | lives at |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) | | hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) | | matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container | | the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) | | the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
Both delivery units wait for authelia's first boot to mint the secret — a Both delivery units wait for authelia's first boot to mint the secret — a
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
@ -477,13 +477,14 @@ it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives ## The constraint that decides where the root lives
A hive CA carries `nameConstraints=permitted;DNS:<hive domain>`, and **a swarm The store issues a hive's gateway leaf through a role whose `allowed_domains` is
service name is a sibling of the hive domain rather than a child** — `forge.<swarm>` that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
next to `<hive>.<swarm>`. A hive CA can't issue a certificate for a swarm the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
service. Not by policy: by construction, and openssl enforces it. That role can't issue a certificate for a swarm service; the store refuses it.
Whatever holds the swarm root is therefore what makes swarm-service certificates The store's `swarm-services` role issues them instead, from the root inside its
possible at all. Two things follow: `pki` mount. The swarm root (`swarm.ca.*`) is still generated on one host, and
two things follow for it:
- **The root's private key is a runtime file and must never enter the nix - **The root's private key is a runtime file and must never enter the nix
store**, so nothing build-time can name it — the system reads store**, so nothing build-time can name it — the system reads

1
nix/host-modules/default.nix
View file

@ -32,6 +32,7 @@
./glue-controller-bao-identity.nix ./glue-controller-bao-identity.nix
./glue-forge-oidc-client.nix ./glue-forge-oidc-client.nix
./glue-grafana-oidc-client.nix ./glue-grafana-oidc-client.nix
./glue-hive-tls-bao-identity.nix
./glue-matrix-bao-token.nix ./glue-matrix-bao-token.nix
./glue-matrix-ctl-bao-identity.nix ./glue-matrix-ctl-bao-identity.nix
./glue-nats-auth-bao-identity.nix ./glue-nats-auth-bao-identity.nix

42
nix/host-modules/deploy.nix
View file

@ -325,23 +325,45 @@ in
[ "services" "hyperhive" "deploy" "singleHostSwarm" ] [ "services" "hyperhive" "deploy" "singleHostSwarm" ]
) )
# The hive CA's own knobs. They sat at the TOP of `services.hyperhive`, # The hive TLS state dir sat at the TOP of `services.hyperhive`, which is
# which is meant to be everything about hyperhive rather than the settings # meant to be everything about hyperhive rather than the settings of one
# of one hive — and where the CA lives, how long it lasts and how long its # hive. `hive-controller` is hive-c0re's new name (mara, on the issue).
# leaves last are decisions of the host that holds the key. `hive-controller`
# is hive-c0re's new name (mara, on the issue), so the daemon that owns the
# CA is what they hang off.
(lib.mkRenamedOptionModule (lib.mkRenamedOptionModule
[ "services" "hyperhive" "tls" "stateDir" ] [ "services" "hyperhive" "tls" "stateDir" ]
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "stateDir" ] [ "services" "hyperhive" "deploy" "hive-controller" "tls" "stateDir" ]
) )
(lib.mkRenamedOptionModule # The hive CA's lifetime and its leaves', both spellings, removed rather
[ "services" "hyperhive" "tls" "caValidityDays" ] # than renamed: no hive CA is generated, and the store's per-hive role
# sets the gateway leaf's lifetime.
(lib.mkRemovedOptionModule [ "services" "hyperhive" "tls" "caValidityDays" ] ''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
'')
(lib.mkRemovedOptionModule [ "services" "hyperhive" "tls" "leafValidityDays" ] ''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
'')
(lib.mkRemovedOptionModule
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "caValidityDays" ] [ "services" "hyperhive" "deploy" "hive-controller" "tls" "caValidityDays" ]
''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
''
) )
(lib.mkRenamedOptionModule (lib.mkRemovedOptionModule
[ "services" "hyperhive" "tls" "leafValidityDays" ]
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "leafValidityDays" ] [ "services" "hyperhive" "deploy" "hive-controller" "tls" "leafValidityDays" ]
''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
''
) )
# The WireGuard mesh, whole. Unlike every rename above this one moves a # The WireGuard mesh, whole. Unlike every rename above this one moves a

8
nix/host-modules/glue-bao-tls.nix
View file

@ -261,6 +261,14 @@ in
# is a swarm singleton, so elsewhere this is the file an operator copies. # is a swarm singleton, so elsewhere this is the file an operator copies.
[ -s ${pkiDir}/nats-auth.pem ] || ${signLeaf} ${pkiDir} nats-auth \ [ -s ${pkiDir}/nats-auth.pem ] || ${signLeaf} ${pkiDir} nats-auth \
${lib.escapeShellArg deployCfg.bao.natsAuthCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.natsAuthCommonName} "" clientAuth
# This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
# subject carries this hive's name, like the matrix-token and
# queue-agent leaves above, because ./swarm-bao.nix writes one role per
# hive. Hives on other hosts are issued theirs out of band.
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth
''; '';
}; };
}; };

34
nix/host-modules/glue-hive-tls-bao-identity.nix Normal file
View file

@ -0,0 +1,34 @@
# Glue: point the gateway-leaf issuance at the hive's own bao identity minted
# for it.
#
# ONE PAIRING PER FILE — the hive's gateway-leaf principal ← bao, and nothing
# else. Deleting this leaves a hive that takes an operator-provided path to
# that credential, which is what every hive on a host without the store does.
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, for the reason ./glue-services-issuer-bao-identity.nix gives.
#
# Gated on the minted PKI existing, as that file is.
#
# Everything is `mkDefault`. An operator naming their own paths wins.
{
lib,
config,
...
}:
let
baoDeploy = config.services.hyperhive.deploy.bao;
# Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own
# path as ./glue-services-issuer-bao-identity.nix derives it.
haveMintedPki = baoDeploy.clientCertFile != null;
pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null;
in
{
config = lib.mkIf haveMintedPki {
services.hyperhive.deploy.hive-controller.tls = {
baoHiveIdentityCertFile = lib.mkDefault "${pkiDir}/hive-leaf.pem";
baoHiveIdentityKeyFile = lib.mkDefault "${pkiDir}/hive-leaf-key.pem";
};
};
}

2
nix/host-modules/hive-gateway/default.nix
View file

@ -286,7 +286,7 @@ in
chmod 0755 ${tlsDir} chmod 0755 ${tlsDir}
# Copy the leaf in. `install` writes atomically with the # Copy the leaf in. `install` writes atomically with the
# target mode; runs as root so the 0600 root:root key written # target mode; runs as root so the 0600 root:root key written
# by hive-tls-ca is readable. Key ends up root:nginx 0640 so # by hive-tls-resign is readable. Key ends up root:nginx 0640 so
# nginx-pre-start # nginx-pre-start
# (which runs `nginx -t` as the nginx user, not root) can # (which runs `nginx -t` as the nginx user, not root) can
# read it — a 0600 root:root key passes the master load but # read it — a 0600 root:root key passes the master load but

705
nix/host-modules/hive-tls.nix
View file

@ -49,167 +49,68 @@ let
gatewayWaitsForServicesLeaf = gatewayWaitsForServicesLeaf =
localServiceDomains != [ ] && localServiceDomains != [ baoCfg.ui.domain ]; localServiceDomains != [ ] && localServiceDomains != [ baoCfg.ui.domain ];
# The host-managed hive CA is the trust anchor for self-signed mode. # The gateway's own leaf is only issued when the gateway actually serves a
# It is only stood up when the gateway actually serves a self-signed # self-signed cert: the gateway must run here and be in self-signed mode.
# cert: the gateway must run here and be in self-signed mode. The # The self-signed condition is the gateway module's single source of truth
# self-signed condition is the gateway module's single source of truth
# (`gateway.useSelfSigned`): true when neither an operator cert # (`gateway.useSelfSigned`): true when neither an operator cert
# (`tls.certDir`) nor ACME is set. # (`tls.certDir`) nor ACME is set.
active = gatewayCfg.enable && gatewayCfg.useSelfSigned; active = gatewayCfg.enable && gatewayCfg.useSelfSigned;
# How this hive's CA comes into existence when it is missing — and it # The role this hive's gateway leaf is issued through. ./swarm-bao.nix
# is one of exactly two things, chosen by config rather than by what # writes one per hive in the swarm directory, under the same prefix, and
# happens to be on disk. # narrows it to that hive's domain and `*.` that domain. So the leaf carries
# # those two names and no others: a configured service name goes on the
# Issuing a sub-CA requires the swarm root's PRIVATE key, so it can # swarm-services leaf below, and asking this role for one is a refusal.
# only happen where that key legitimately lives: the single-host hiveLeafRole =
# deployment that `swarm.ca.autoConfigure` describes. A host cannot if hyperhiveCfg.hiveName == null then
# infer that it is that host — a swarm's services and its hives can null
# sit anywhere — so with the flag off this hive self-signs exactly as else
# it always has, and an operator who wants it in the hierarchy "${baoDeploy.hiveLeafPkiRolePrefix}-${hyperhiveCfg.hiveName}";
# installs the CA themselves. Falling back to self-signed rather than
# failing keeps a plain hive working out of the box; what it loses is # The hive's own store identity: its subject is
# membership of a swarm's trust, which is the correct thing to lose # `<hiveLeafCommonNamePrefix>-<hiveName>`, and its policy may issue from
# for a hive nobody has federated. # `hiveLeafRole` alone. It also carries the root fetch's handshake.
caGenScript = haveHiveIdentity = cfg.baoHiveIdentityCertFile != null && cfg.baoHiveIdentityKeyFile != null;
if swarmCaCfg.autoConfigure then showPath = p: if p == null then "unset" else p;
hiveIdentityOptions = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile (${showPath cfg.baoHiveIdentityCertFile}) and services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile (${showPath cfg.baoHiveIdentityKeyFile})";
# Fails the unit, naming both options, when the identity is unset or a file
# is unreadable. Every hive must hold one.
requireHiveIdentity =
purpose:
if haveHiveIdentity then
'' ''
# The root should exist — `swarm-ca.service` runs before this and missing=""
# is required by it. If it doesn't, something upstream failed and for f in ${
# signing with a half-provisioned root would be worse than stopping. lib.escapeShellArgs [
if [ ! -s "$root" ] || [ ! -s "$rootk" ]; then cfg.baoHiveIdentityCertFile
echo "swarm.ca.autoConfigure is set but there is no root CA key at $rootk" >&2 cfg.baoHiveIdentityKeyFile
echo "(swarm-ca.service should have generated it) — refusing to issue a hive CA." >&2 ]
}; do
[ -r "$f" ] || missing="$missing $f"
done
if [ -n "$missing" ]; then
echo "this hive has no usable store identity, so it cannot ${purpose}: cannot read$missing." >&2
echo "place the files named by ${hiveIdentityOptions}." >&2
exit 1 exit 1
fi fi
echo "issuing fresh hive CA at $ca under the swarm root"
cacsr="$(mktemp "$d/ca.csr.XXXXXX")"
caext="$(mktemp "$d/ca.ext.XXXXXX")"
trap 'rm -f "$cacsr" "$caext"' EXIT
openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout "$cak" -out "$cacsr" \
-subj "/CN=hive-ca ${domain}"
# printf (not a heredoc) so the ext-file lines carry no leading
# whitespace once nix has stripped the indented-string indent.
{
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\n'
printf 'keyUsage=critical,keyCertSign,cRLSign\n'
printf 'subjectKeyIdentifier=hash\n'
printf 'authorityKeyIdentifier=keyid:always\n'
# The constraint is the point of the hierarchy, not a
# flourish: without it a leaked hive CA mints any name in
# the swarm, and it is verifiers that enforce this, not our
# good behaviour. The IP exclusions are not redundant — a
# DNS constraint says nothing about an iPAddress SAN, and a
# name type nobody constrained is a name type this CA is
# unconstrained for.
printf 'nameConstraints=critical,permitted;DNS:%s,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0\n' \
${lib.escapeShellArg domain}
} > "$caext"
openssl x509 -req -in "$cacsr" -CA "$root" -CAkey "$rootk" \
-CAcreateserial -days ${toString cfg.caValidityDays} -sha256 \
-extfile "$caext" -out "$ca"
rm -f "$cacsr" "$caext"
trap - EXIT
'' ''
else else
'' ''
echo "generating fresh self-signed hive CA at $ca" echo "this hive has no store identity, so it cannot ${purpose}." >&2
openssl req -x509 -newkey rsa:4096 -nodes -sha256 \ echo "set ${hiveIdentityOptions} and place both files." >&2
-days ${toString cfg.caValidityDays} \ exit 1
-keyout "$cak" -out "$ca" \
-subj "/CN=hive-ca ${domain}" \
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
-addext "keyUsage=critical,keyCertSign,cRLSign"
''; '';
# Sign one leaf. Parameterised rather than hardcoded to `gateway.*`
# because there are now two: the hive's own leaf, issued by the hive
# CA, and the swarm-services leaf, issued by the services sub-CA that
# ./swarm-ca.nix maintains. Same ceremony, different issuer and names
# — and one script means the two cannot drift in how they are built.
#
# $1 stateDir $2 basename $3 CN $4 SAN list $5 issuer cert $6 issuer key
signLeafScript = pkgs.writeShellScript "hive-tls-sign-leaf" ''
set -euo pipefail
d="$1"
base="$2"
cn="$3"
sans="$4"
ca="$5"
cak="$6"
leaf="$d/$base.pem"
leafk="$d/$base-key.pem"
csr="$(mktemp "$d/$base.csr.XXXXXX")"
ext="$(mktemp "$d/$base.ext.XXXXXX")"
only="$(mktemp "$d/$base.leaf.XXXXXX")"
trap 'rm -f "$csr" "$ext" "$only"' EXIT
openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout "$leafk" -out "$csr" \
-subj "/CN=$cn"
# printf (not a heredoc) so the ext-file lines carry no leading
# whitespace once nix has stripped the indented-string indent.
{
printf 'subjectAltName=%s\n' "$sans"
printf 'basicConstraints=critical,CA:FALSE\n'
printf 'keyUsage=critical,digitalSignature,keyEncipherment\n'
printf 'extendedKeyUsage=serverAuth\n'
} > "$ext"
openssl x509 -req -in "$csr" -CA "$ca" -CAkey "$cak" \
-CAcreateserial -days ${toString cfg.leafValidityDays} -sha256 \
-extfile "$ext" -out "$only"
# nginx serves this file verbatim, so it must carry the leaf AND its
# issuer: the hive CA is an intermediate under the swarm root, and a
# client that anchors on the root cannot build the middle of the
# chain by itself. Agents anchor on the hive CA directly and
# validated either way — the appended cert is what makes a swarm
# peer, or anything else holding only the root, work.
# Leaf first: both `ssl_certificate` and the `openssl x509 -in`
# expiry checks read the first cert in the file.
cat "$only" "$ca" > "$leaf"
chmod 0600 "$leafk"
chmod 0644 "$leaf"
'';
# The hive's own leaf: signed by the hive CA, covering the hive domain
# and its sub-domains.
#
# `forge.` and `matrix.` were listed here explicitly and both were
# redundant with the wildcard beside them — a wildcard covers exactly
# one label, and those are one label. Naming them read as policy
# ("these two services live under this leaf"), which is the reason they
# outlived it: a configured service name is no longer necessarily under
# this domain, and when it isn't, this is the one list it cannot join.
# The hive CA is `nameConstraints=permitted;DNS:<hive domain>`, and a
# violation invalidates the CERTIFICATE rather than the offending name
# — measured with openssl, not reasoned about — so one sibling name
# added here takes the leaf down for the dashboard and every other
# vhost sharing it. Configured service names get the swarm-services
# leaf below, whose sub-CA is constrained to exactly those names.
signHiveLeaf = ''
${signLeafScript} "$d" gateway ${lib.escapeShellArg domain} \
${lib.escapeShellArg "DNS:${domain},DNS:*.${domain}"} \
"$d/ca.pem" "$d/ca-key.pem"
'';
# Shared by every unit that decides whether to re-issue a leaf. It # Shared by every unit that decides whether to re-issue a leaf. It
# lives here rather than in one of them because the guards have to # lives here rather than in one of them because the guards have to
# agree: they answer the same question at different times # agree: they answer the same question for different leaves
# (`hive-tls-ca` at service activation, i.e. on the deploy; # (`hive-tls-resign` for the hive leaf, at boot, on the deploy and from
# `hive-tls-resign` from a weekly timer for a host that stays up long # its timer; `swarm-services-cert` for the services leaf). A rule
# enough to drift; `swarm-services-cert` for the leaf the store # implemented in one and not the others is worse than one implemented
# issues). A rule implemented in one and not the others is worse than # in none — it looks fixed and only fires on whichever path you did not
# one implemented in none — it looks fixed and only fires on whichever # take, which is exactly how a corrected `serviceDomains` still served a
# path you did not take, which is exactly how a corrected # stale leaf.
# `serviceDomains` still served a stale leaf.
# #
# Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and # Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and
# `covers`. # `covers`.
@ -254,18 +155,13 @@ let
[ "$_missing" = 0 ] [ "$_missing" = 0 ]
} }
# True when the leaf this host SIGNS is present and carries its # True when the hive leaf is present and carries its configured
# configured names. Expiry is the callers' own business — they use # names. Expiry is the callers' own business — they use different
# different windows. # windows.
# #
# ⚠️ The services leaf is deliberately NOT part of this any more, and # The services leaf is not part of this: `swarm-services-cert` holds
# the omission is load-bearing rather than a simplification. Its two # the same rule for it, using the `covers` helper above so the two
# callers sign under the hive CA's key; the services leaf comes out # cannot drift.
# of the secret store, which neither of them can reach. A guard that
# went on reporting it stale would make `hive-tls-ca` re-sign the
# hive leaf forever over a file it has no way to fix. The
# `swarm-services-cert` unit below holds the same rule for that leaf,
# using the `covers` helper above so the two cannot drift.
leavesCoverNames() { leavesCoverNames() {
covers "$leaf" "$hiveNames" covers "$leaf" "$hiveNames"
} }
@ -306,9 +202,9 @@ let
// lib.optionalAttrs (baoDeploy.serverCaFile != null) { // lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile; BAO_CACERT = baoDeploy.serverCaFile;
}; };
haveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# The root-only fetch. Same store and server CA as `servicesCertEnvironment`, # The root-only fetch. Same store and server CA as `servicesCertEnvironment`,
# but the hive's reader identity: a host fronting no swarm name holds no # but the hive's own identity: a host fronting no swarm name holds no
# services-issuer pair. # services-issuer pair.
storeRootCurlTls = lib.escapeShellArgs ( storeRootCurlTls = lib.escapeShellArgs (
lib.optionals (baoDeploy.serverCaFile != null) [ lib.optionals (baoDeploy.serverCaFile != null) [
@ -317,59 +213,53 @@ let
] ]
++ [ ++ [
"--cert" "--cert"
baoDeploy.clientCertFile cfg.baoHiveIdentityCertFile
"--key" "--key"
baoDeploy.clientKeyFile cfg.baoHiveIdentityKeyFile
] ]
); );
# What `hive-tls-resign` issues the hive leaf with: the services leaf's
# store and server CA, presenting the hive's own identity.
hiveLeafEnvironment = {
inherit (servicesCertEnvironment) BAO_ADDR;
}
// lib.optionalAttrs (cfg.baoHiveIdentityCertFile != null) {
BAO_CLIENT_CERT = cfg.baoHiveIdentityCertFile;
}
// lib.optionalAttrs (cfg.baoHiveIdentityKeyFile != null) {
BAO_CLIENT_KEY = cfg.baoHiveIdentityKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
storeRetry = import ./lib/store-retry.nix { }; storeRetry = import ./lib/store-retry.nix { };
in in
{ {
# Host-side TLS trust root for the self-signed gateway mode. # Host-side TLS for the self-signed gateway mode.
# #
# A bare self-signed leaf would be its own trust anchor, so every # The gateway serves a short-lived **leaf** issued by the secret store's
# regeneration would be a new anchor and every consumer (agents, # `pki` mount through this hive's own role, and consumers (agents,
# federation peers) would have to re-trust on each rotation — and a # federation peers) anchor on that mount's root, which ships in
# runtime-generated, in-container cert can't be wired into an agent's # `trust-bundle.pem`. Leaf rotation never re-breaks them, and no key that
# build-time trust store at all. # signs a certificate exists on this host. See
# `docs/networking/gateway.md` ("Self-signed TLS") and docs/swarm/ca.md.
# #
# So the issuer is a long-lived **hive CA** held on the host. The # The trust bundle still carries the retired hive CA's public `ca.pem`
# gateway serves a **leaf** signed by that CA (via the `tls.certDir` # where one exists, so a leaf it signed keeps verifying until it is
# bind-mount path); agents and federation peers trust the *CA* once, # replaced.
# and leaf rotation never re-breaks them. See `docs/networking/gateway.md`
# ("Self-signed TLS").
#
# That CA is self-signed by default. Under
# `swarm.ca.autoConfigure` — the all-on-one-host case — it is instead
# an intermediate issued under the swarm root (./swarm-ca.nix owns the
# why) and name-constrained to this hive's domain, so it stays the
# anchor agents pin while a peer holding only the root can validate
# everything this hive serves. Either way an existing CA is left
# alone; see the issuance comment below.
options.services.hyperhive.deploy.hive-controller.tls = { options.services.hyperhive.deploy.hive-controller.tls = {
stateDir = lib.mkOption { stateDir = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "/var/lib/hive-tls"; default = "/var/lib/hive-tls";
description = '' description = ''
Host directory holding the hive CA + gateway leaf cert for the Host directory holding the gateway leaf and trust bundle for the
self-signed gateway mode. `ca.pem` (the anchor agents and self-signed gateway mode: `gateway.pem` / `gateway-key.pem` (the
federation peers trust), `ca-key.pem` (0600, never leaves the leaf the store issues and nginx serves) and `trust-bundle.pem` (the
host), `gateway.pem` / `gateway-key.pem` (the leaf the gateway anchors agents and federation peers trust). `ca.pem` and
container bind-mounts and nginx serves). Persistent so the CA `ca-previous.pem`, where present, are the retired hive CA's public
survives reboots — re-deriving it would re-break every consumer. certificates and stay in the bundle; no CA key is kept here.
'';
};
caValidityDays = lib.mkOption {
type = lib.types.int;
default = 7300;
description = ''
Validity window of the hive CA in days (default ~20y). Kept long
and well beyond `leafValidityDays` so the CA outlives many leaf
rotations — the whole point of the CA is to be a stable anchor
that consumers trust once. The CA is regenerated only if missing
or already expired.
''; '';
}; };
@ -389,9 +279,9 @@ in
A **separate** credential from A **separate** credential from
{option}`services.hyperhive.deploy.bao.clientCertFile`, which is this {option}`services.hyperhive.deploy.bao.clientCertFile`, which is this
hive's own reader identity and whose policy reads every secret the hive's own reader identity and whose policy reads every secret the
hive is entitled to. `swarm-services-cert` never logs in with that one: hive is entitled to, and from
on a hive fronting no swarm name it presents it only for the mTLS {option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile`,
handshake of the root fetch, since `pki/ca/pem` takes no token. which the root fetch on a hive fronting no swarm name presents.
On a hive that mints its own store PKI a glue module supplies the leaf On a hive that mints its own store PKI a glue module supplies the leaf
as a `mkDefault`; everywhere else it is the credential an operator as a `mkDefault`; everywhere else it is the credential an operator
@ -412,21 +302,39 @@ in
''; '';
}; };
leafValidityDays = lib.mkOption { baoHiveIdentityCertFile = lib.mkOption {
type = lib.types.int; type = lib.types.nullOr lib.types.str;
default = 30; default = null;
example = "/var/lib/swarm-bao-pki/hive-leaf.pem";
description = '' description = ''
Validity window of the gateway leaf cert in days (default 30). This hive's own store identity. `hive-tls-resign` presents it to ask
Short-lived by design — ahead of the CA/Browser-Forum's move the secret store for the gateway leaf, and `swarm-services-cert` on a
toward ~47-day max lifetimes — which bounds the blast radius of a hive fronting no swarm name presents it for the handshake of the root
leaf-key compromise. The leaf is re-signed by the (stable) CA fetch.
when it is missing or near expiry; because it shares the CA
anchor, a rotation does not disturb consumer trust. Agents and Its subject has to be
federation peers validate against the CA, not browser CA/B-forum `<services.hyperhive.deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`;
limits. The weekly `hive-tls-resign` timer re-signs the leaf once that cert-auth role's policy may issue from this hive's gateway-leaf
it is within half its validity of expiry and propagates the new role and nothing else.
leaf into the running gateway, so a long-uptime host renews
automatically without a reboot. Every hive with a self-signed gateway needs it: without it both units
fail, naming this option and
{option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile`.
On the host that runs the store a glue module supplies the leaf as a
`mkDefault`; on every other host an operator places it.
A path, never a value.
'';
};
baoHiveIdentityKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/hive-leaf-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile`.
Both or neither — a certificate with no key authenticates nothing.
''; '';
}; };
}; };
@ -444,24 +352,18 @@ in
# rebuild of every hive that is not the CA host, about a fallback # rebuild of every hive that is not the CA host, about a fallback
# that no longer happens. # that no longer happens.
# Generate (and rotate) the hive CA + gateway leaf before anything # The state dir and the trust bundle. Never waits on the store: the
# serves it. Idempotent: the CA is created once and reused; the leaf # store's container is ordered after this unit, which writes the
# is re-signed on expiry under the same CA so the anchor is stable. # bundle it bind-mounts, so the leaf is `hive-tls-resign`'s.
systemd.services.hive-tls-ca = { systemd.services.hive-tls-ca = {
description = "Generate hive CA + gateway leaf TLS cert (self-signed mode)"; description = "Assemble the hive TLS trust bundle (self-signed mode)";
wantedBy = [ "multi-user.target" ]; wantedBy = [ "multi-user.target" ];
# The consumer is `hive-gateway-self-signed-cert`, which copies the # The consumer is `hive-gateway-self-signed-cert`, which copies the
# leaf into the gateway's state dir at the mode nginx can read, and # leaf out of the state dir this unit creates, and which nginx in
# which nginx in turn `Requires=`. So this must run first or that # turn `Requires=`. Order against the unit that reads the dir.
# copy fails under `set -eu` and takes nginx down with it — order
# against the unit that reads the file.
before = [ "hive-gateway-self-signed-cert.service" ]; before = [ "hive-gateway-self-signed-cert.service" ];
requiredBy = [ "hive-gateway-self-signed-cert.service" ]; requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The issuance below needs the swarm root key on disk. When this # The bundle carries the swarm root where this host generates it.
# host generates it (single-host swarm) that unit must have run
# first; when the operator provides the material there is no unit
# to wait for, so the dependency is conditional rather than a unit
# that exists and does nothing.
# #
# `swarm-services-cert` is NOT ordered against this unit in either # `swarm-services-cert` is NOT ordered against this unit in either
# direction, and the absence is load-bearing — see the cycle spelled # direction, and the absence is load-bearing — see the cycle spelled
@ -477,8 +379,6 @@ in
# and `swarm-services-cert` restarts this unit once the root it wrote # and `swarm-services-cert` restarts this unit once the root it wrote
# is new — so the bundle converges instead of being a boot behind. # is new — so the bundle converges instead of being a boot behind.
after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ]; after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
requires = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
path = [ pkgs.openssl ];
serviceConfig = { serviceConfig = {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
@ -492,164 +392,103 @@ in
install -d -m 0755 "$d" install -d -m 0755 "$d"
ca="$d/ca.pem" ca="$d/ca.pem"
cak="$d/ca-key.pem"
leaf="$d/gateway.pem"
leafk="$d/gateway-key.pem"
root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"}
rootk=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root-key.pem"}
prev="$d/ca-previous.pem" prev="$d/ca-previous.pem"
marker="$d/.swarm-ca-adopted" root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"}
svcroot="$d/swarm-services-root.pem"
# --- Adoption: a hive whose CA predates the swarm root. # The gateway leaf comes from the store, so no key that signs a
# # certificate is kept here.
# Runs ONLY where this host also owns the root (`autoConfigure`), if [ -e "$d/ca-key.pem" ]; then
# because adoption invalidates an anchor that consumers already echo "removing the retired hive CA key $d/ca-key.pem"
# trust and they refresh on their own schedule. On one box that rm -f "$d/ca-key.pem"
# schedule is knowable; across hosts it is not, so there the
# operator does it and this unit only says so, loudly.
#
# Guarded by a marker rather than by the state of the world: the
# marker's ABSENCE is the trigger, so this fires once per hive
# instead of re-deciding every activation. That is the difference
# between a migration and a kicking machine.
if [ -s "$root" ] && [ -s "$ca" ] && [ ! -e "$marker" ] \
&& ! openssl verify -CAfile "$root" "$ca" >/dev/null 2>&1; then
${
if swarmCaCfg.autoConfigure then
''
echo "adopting the swarm CA: $ca does not chain to $root" >&2
# Keep the old CA as an anchor across the overlap. Consumers
# read the bundle, so adoption is additive before it is
# subtractive — agents pick up new trust only when their
# container restarts, which is a window even on one host.
cp "$ca" "$prev"
chmod 0644 "$prev"
rm -f "$ca" "$cak"
touch "$marker"''
else
''
echo "hive-tls: this hive's CA does not chain to the swarm root." >&2
echo " hive CA: $ca" >&2
echo " swarm root: $root" >&2
echo "Adopting the hierarchy is not automatic here: it invalidates an" >&2
echo "anchor that peers and agents on OTHER hosts still trust, and they" >&2
echo "refresh on their own schedule — only you know when that is safe." >&2
echo "To adopt: rm $ca $cak && systemctl restart hive-tls-ca.service" >&2
echo "To keep the current CA deliberately: touch $marker" >&2
exit 1''
}
fi fi
# --- CA: generated once and reused across leaf rotations, in # --- Trust bundle: what a consumer must TRUST. Why it is a set of
# whichever of the two shapes `caGenScript` selected. # anchors rather than one file is in docs/swarm/ca.md. Three
# Regenerated only if missing or already expired (checkend 0). # constraints the code cannot state:
# A new CA means every consumer must re-trust, so the leaf is
# dropped to force a re-sign under the fresh CA.
#
# An existing CA is never re-rooted here. A hive predating the
# swarm root carries a self-signed `ca.pem`, and swapping it for
# a swarm-issued one would break every consumer already trusting
# it — including agents, whose trust is refreshed only when their
# container restarts. Adopting the hierarchy on such a hive is
# therefore an operator step (drop the CA, let this unit
# re-issue), which is also what keeps this change non-disruptive
# to hives that never adopt it.
if [ ! -s "$ca" ] || [ ! -s "$cak" ] \
|| ! openssl x509 -in "$ca" -noout -checkend 0 >/dev/null 2>&1; then
${caGenScript}
chmod 0600 "$cak"
chmod 0644 "$ca"
rm -f "$leaf" "$leafk"
fi
# --- Leaf: (re)sign when missing, within 30 days of expiry, or no
# longer covering the configured names, always under the current
# (stable) CA.
${leafCoverage}
# ⚠️ This is the guard that runs ON THE DEPLOY — `hive-tls-resign`
# only fires from a weekly timer, so a rule enforced only there is
# up to a week late and does nothing for the rebuild that changed
# the names in the first place.
#
resign=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || resign=1
openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1 || resign=1
leavesCoverNames || resign=1
if [ "$resign" = 1 ]; then
echo "signing gateway leaf at $leaf (missing, near expiry, or missing a configured name)"
${signHiveLeaf}
fi
# --- Trust bundle: what a consumer must TRUST, as opposed to
# `ca.pem`, which is what this host SIGNS with. Why they stopped
# being the same file, and why `ca-previous.pem` stays in the set
# after an adoption, are in docs/swarm/ca.md. Three constraints
# the code cannot state:
# #
# Written IN PLACE, never renamed into position — containers # Written IN PLACE, never renamed into position — containers
# bind-mount this file and a bind mount follows the inode, so a # bind-mount this file and a bind mount follows the inode, so a
# rename leaves every consumer holding the old one. # rename leaves every consumer holding the old one.
# #
# Dropping `ca-previous.pem` is deliberately NOT done here: # Dropping `ca.pem` and `ca-previous.pem` is deliberately NOT done
# "long enough" is a deployment fact, not a unit's call. # here: "long enough" is a deployment fact, not a unit's call.
# #
# Built as an array with `if` guards, not # Built as an array with `if` guards, not
# `[ -s f ] && anchors+=(f)`: under `set -e` an && list whose test # `[ -s f ] && anchors+=(f)`: under `set -e` an && list whose test
# fails IS a failing command and kills the unit — on exactly the # fails IS a failing command and kills the unit — on exactly the
# hive where the optional file is legitimately absent. The # hive where the optional file is legitimately absent.
# reflexive `|| true` is worse; it also swallows a real failure to
# read the hive CA.
# #
# `swarm-services-root.pem` joins the set for the reason the swarm # `swarm-services-root.pem` is the store's `pki` root, which both
# root is in it: it is an ANCHOR a consumer terminates at, not an # the gateway leaf and the swarm-services leaf chain to. `ca.pem`
# intermediate. It is also what keeps this change from being a # and `ca-previous.pem` are the retired hive CA's public
# flag day — the swarm root stays in the bundle beside it, so a # certificates, kept so a leaf they signed verifies until replaced.
# leaf still chaining to the old services sub-CA and one issued by
# the store both verify against the same file, and a hive can be
# rebuilt before or after its peers.
bundle="$d/trust-bundle.pem" bundle="$d/trust-bundle.pem"
anchors=("$ca") anchors=()
if [ -s "$ca" ]; then anchors+=("$ca"); fi
if [ -s "$prev" ]; then anchors+=("$prev"); fi if [ -s "$prev" ]; then anchors+=("$prev"); fi
if [ -s "$root" ]; then anchors+=("$root"); fi if [ -s "$root" ]; then anchors+=("$root"); fi
if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
cat "''${anchors[@]}" > "$bundle" # `cat` with no arguments reads stdin, so no anchor at all is an
# empty file until `swarm-services-cert` writes the root.
if [ "''${#anchors[@]}" = 0 ]; then
echo "no trust anchor on disk yet — $bundle is empty until the store's root is fetched" >&2
: > "$bundle"
else
cat "''${anchors[@]}" > "$bundle"
fi
chmod 0644 "$bundle" chmod 0644 "$bundle"
''; '';
}; };
# Weekly re-sign of the gateway leaf so short-lived leaves renew # The gateway leaf: issued by the secret store's `pki` mount through
# without depending on a reboot. # this hive's own role, at boot, on every deploy and from the daily
# timer below, whenever it is missing, past half its lifetime, missing
# a configured name, or not chained to the store's root.
# #
# `hive-tls-ca` only re-signs at service activation (boot/rebuild); a # ⚠️ Nothing orders the gateway against this unit, which is what lets a
# long-uptime host would otherwise let a 30-day leaf lapse silently. # hive holding a valid leaf start its gateway with the store sealed or
# This service re-signs the leaf directly (not by bouncing hive-tls-ca) # unreachable. Only a hive with no leaf yet has a gateway that waits
# and propagates the new leaf to nginx when the file actually changed. # for the store: its cert import fails until this unit lands a leaf
# and starts nginx.
#
# Failure leaves the current leaf alone: every file the script writes
# is a `.new` until the store has answered with both fields.
# #
# Propagation mechanism: nginx serves a *copy* of the leaf, written by # Propagation mechanism: nginx serves a *copy* of the leaf, written by
# `hive-gateway-self-signed-cert` at the mode nginx can read. Re-signing # `hive-gateway-self-signed-cert` at the mode nginx can read. Issuing
# the source therefore changes nothing on its own — the copy has to be # the source therefore changes nothing on its own — the copy has to be
# remade and nginx reloaded, which is what the two calls below do. # remade and nginx reloaded, which is what the calls at the end do.
# # Neither of the running-nginx calls is `|| true`: a swallowed
# ⚠️ Neither call is `|| true`: a swallowed propagation failure means # propagation failure means the leaf rotates on disk while nginx keeps
# the leaf rotates on disk while nginx keeps serving the old copy # serving the old copy until it expires, with this unit reporting
# until it expires, with this unit reporting success the whole time. # success the whole time.
# A failure here must fail the timer.
systemd.services.hive-tls-resign = { systemd.services.hive-tls-resign = {
description = "Re-sign the gateway TLS leaf and reload nginx"; description = "Issue or renew the gateway TLS leaf from the secret store's PKI";
# hive-tls-ca must have run first so the CA key exists before we try wantedBy = [ "multi-user.target" ];
# to re-sign under it. On first boot `Persistent=true` on the weekly # The store's container and its granting unit where they run here;
# timer fires immediately; without this ordering the resign could race # ignored elsewhere, as for `swarm-services-cert`, which writes the
# the CA initialisation and fail with "no such file" on the CA key. # store's root this unit checks the leaf against.
after = [ "hive-tls-ca.service" ]; after = [
"hive-tls-ca.service"
"container@${baoCfg.machine}.service"
"swarm-bao-pki.service"
"swarm-bao-hive-leaf-policy.service"
"swarm-services-cert.service"
];
wants = [ "container@${baoCfg.machine}.service" ];
path = [ path = [
baoDeploy.package
pkgs.jq
pkgs.openssl pkgs.openssl
pkgs.coreutils pkgs.coreutils
pkgs.systemd pkgs.systemd
]; ];
serviceConfig = { environment = hiveLeafEnvironment;
# ./lib/store-retry.nix: the login below fails for as long as the
# store is sealed or down.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
UMask = "0077"; UMask = "0077";
SyslogIdentifier = "hive-tls-resign"; SyslogIdentifier = "hive-tls-resign";
@ -657,46 +496,93 @@ in
script = '' script = ''
set -euo pipefail set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir} d=${lib.escapeShellArg cfg.stateDir}
install -d -m 0755 "$d"
leaf="$d/gateway.pem" leaf="$d/gateway.pem"
leafk="$d/gateway-key.pem"
# ⚠️ EVERY leaf this host SIGNS must be covered here. A leaf that fail() {
# first-boot issuance creates and this unit does not know about echo "$1" >&2
# looks perfect for its entire validity and then expires with no if [ -s "$leaf" ]; then
# warning — the failure is invisible until it is total. The name echo "the current gateway leaf at $leaf stays in place" >&2
# checks come from the shared snippet, so this unit and fi
# `hive-tls-ca` cannot disagree about what a good leaf is. exit 1
#
# The services leaf is not one of them any longer: it is issued
# by the secret store, whose key this unit does not hold and
# cannot re-sign under. `swarm-services-cert-renew` renews it,
# on its own daily timer.
${leafCoverage}
# Re-sign only when a leaf is within half its validity of expiry.
# The weekly cadence catches this window well before one lapses.
halflife=$(( ${toString cfg.leafValidityDays} * 86400 / 2 ))
fresh() { # a leaf is fresh if it exists and is not near expiry
[ -s "$1" ] && openssl x509 -in "$1" -noout -checkend "$halflife" >/dev/null 2>&1
} }
if fresh "$leaf" && leavesCoverNames; then # Checked before the leaf is: a hive without its identity is red
echo "leaves valid, and covering the configured names — no resign needed" # even while its current leaf is still valid.
${
if hiveLeafRole == null then
''fail "services.hyperhive.hiveName is unset, so this hive has no gateway-leaf role to issue from."''
else
requireHiveIdentity "ask the store for its gateway leaf"
}
${leafCoverage}
# Half the lifetime the role grants, as for the services leaf, so
# the daily timer has half the lifetime to retry through a sealed
# store.
renewat=$(( ${toString baoDeploy.servicesPkiLeafTtlHours} * 3600 / 2 ))
reissue=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || reissue=1
openssl x509 -in "$leaf" -noout -checkend "$renewat" >/dev/null 2>&1 || reissue=1
leavesCoverNames || reissue=1
# A leaf the retired hive CA signed, or one under a store root since
# replaced, fails this and is re-issued.
{ [ -s "$svcroot" ] && openssl verify -CAfile "$svcroot" "$leaf" >/dev/null 2>&1; } || reissue=1
if [ "$reissue" = 0 ]; then
echo "gateway leaf valid, chained to the store's root and covering the configured names — leaving it alone"
exit 0 exit 0
fi fi
echo "a leaf is missing, near expiry, or missing a configured name — re-signing" err="$(mktemp)"
before="$(sha256sum "$leaf" 2>/dev/null || true)" resp="$(mktemp "$d/gateway.json.XXXXXX")"
trap 'rm -f "$err" "$resp" "$leaf.new" "$leafk.new"' EXIT
${signHiveLeaf} # Cert auth is a login, not a transport setting; `-token-only`
# skips the token helper, for the reason `swarm-services-cert` gives.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
fail "could not log in to the swarm secret store with this hive's identity: $(cat "$err")"
fi
export BAO_TOKEN
after="$(sha256sum "$leaf" 2>/dev/null || true)" echo "requesting the gateway leaf from the store"
if [ "$before" != "$after" ]; then if ! bao write -format=json \
${lib.escapeShellArg "${baoServicesPkiMount}/issue/${toString hiveLeafRole}"} \
common_name=${lib.escapeShellArg domain} \
alt_names=${lib.escapeShellArg "*.${domain}"} \
> "$resp" 2>"$err"; then
fail "the store refused to issue the gateway leaf: $(cat "$err")"
fi
# One issuance, split, for the reasons `swarm-services-cert` gives:
# the key never enters an argv, and `// empty` keeps a missing field
# from becoming the four characters `null`. The leaf alone, without
# its issuer: the issuer is the self-signed root in the bundle.
umask 077
jq -r '.data.private_key // empty' < "$resp" > "$leafk.new"
jq -r '.data.certificate // empty' < "$resp" > "$leaf.new"
for f in "$leafk.new" "$leaf.new"; do
[ -s "$f" ] || fail "the store's response was missing a field: $f is empty"
done
mv -f "$leaf.new" "$leaf"
mv -f "$leafk.new" "$leafk"
chmod 0644 "$leaf"
chmod 0600 "$leafk"
# A gateway already running gets the copy remade and a reload. One
# that is not running — a hive that had no leaf — gets started,
# `--no-block` so a start job still queued absorbs the request.
if systemctl is-active --quiet nginx.service; then
echo "gateway leaf rotated — re-importing and reloading nginx" echo "gateway leaf rotated — re-importing and reloading nginx"
systemctl restart hive-gateway-self-signed-cert.service systemctl restart hive-gateway-self-signed-cert.service
systemctl reload nginx.service systemctl reload nginx.service
else elif [ "$(systemctl show -P LoadState nginx.service)" = loaded ]; then
echo "gateway leaf unchanged (already up to date)" echo "gateway leaf landed with nginx not running — starting it"
systemctl reset-failed nginx.service
systemctl start --no-block nginx.service
fi fi
''; '';
}; };
@ -796,33 +682,18 @@ in
# bundle: every other hive's swarm-service names chain to it. # bundle: every other hive's swarm-service names chain to it.
# `ca/pem` takes no token; the TLS client certificate is the # `ca/pem` takes no token; the TLS client certificate is the
# store's listener's demand, not bao's, so this presents the # store's listener's demand, not bao's, so this presents the
# hive's reader identity and never logs in with it. # hive's own identity and never logs in with it.
# A hive without a readable reader identity fails here, naming the # A hive without a readable identity fails here, naming the
# options: every hive must hold one. # options: every hive must hold one.
if [ "$want_svc" = 0 ]; then if [ "$want_svc" = 0 ]; then
${ ${
if haveReaderIdentity then if haveHiveIdentity then
'' requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root"
missing="" + ''
for f in ${
lib.escapeShellArgs [
baoDeploy.clientCertFile
baoDeploy.clientKeyFile
]
}; do
[ -r "$f" ] || missing="$missing $f"
done
if [ -n "$missing" ]; then
echo "this hive has no usable store identity: cannot read$missing." >&2
echo "place the files named by services.hyperhive.deploy.bao.clientCertFile (${baoDeploy.clientCertFile}) and services.hyperhive.deploy.bao.clientKeyFile (${baoDeploy.clientKeyFile})." >&2
exit 1
fi
echo "no swarm service domains configured — fetching only the store's pki root" echo "no swarm service domains configured — fetching only the store's pki root"
err="$(mktemp)" err="$(mktemp)"
raw="$(mktemp "$d/swarm-services-root.XXXXXX")" raw="$(mktemp "$d/swarm-services-root.XXXXXX")"
trap 'rm -f "$err" "$raw"' EXIT trap 'rm -f "$err" "$raw"' EXIT
# Interim: the reader cert is the handshake identity until each hive
# has its own narrow store identity for issuing its leaf.
if ! curl --fail --silent --show-error \ if ! curl --fail --silent --show-error \
${storeRootCurlTls} --output "$raw" \ ${storeRootCurlTls} --output "$raw" \
"$BAO_ADDR/v1/${baoServicesPkiMount}/ca/pem" 2>"$err"; then "$BAO_ADDR/v1/${baoServicesPkiMount}/ca/pem" 2>"$err"; then
@ -851,14 +722,7 @@ in
fi fi
exit 0'' exit 0''
else else
'' requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root"
echo "this hive has no store identity, so it cannot fetch the store's ${baoServicesPkiMount} root." >&2
echo "set services.hyperhive.deploy.bao.clientCertFile (${
if baoDeploy.clientCertFile == null then "unset" else baoDeploy.clientCertFile
}) and services.hyperhive.deploy.bao.clientKeyFile (${
if baoDeploy.clientKeyFile == null then "unset" else baoDeploy.clientKeyFile
}) and place both files." >&2
exit 1''
} }
fi fi
@ -1075,19 +939,18 @@ in
}; };
}; };
# Daily, for the reason `swarm-services-cert-renew` below is: the leaf
# needs the store, which can be sealed or unreachable.
systemd.timers.hive-tls-resign = { systemd.timers.hive-tls-resign = {
description = "Weekly gateway-leaf re-sign and propagation"; description = "Daily renewal check for the gateway TLS leaf";
wantedBy = [ "timers.target" ]; wantedBy = [ "timers.target" ];
timerConfig = { timerConfig = {
# Run weekly; Persistent=true fires a missed run on next boot if OnCalendar = "daily";
# the timer was not active (e.g. the host was off on the scheduled
# day), preventing a dormant timer from letting the leaf lapse.
OnCalendar = "weekly";
Persistent = true; Persistent = true;
}; };
}; };
# Daily rather than `hive-tls-resign`'s weekly, for the reason # Daily rather than weekly, for the reason
# ./swarm-nats.nix's `swarm-bao-nats-tls` timer is: this leaf needs the # ./swarm-nats.nix's `swarm-bao-nats-tls` timer is: this leaf needs the
# store, which can be sealed or unreachable, so there is a retry per day # store, which can be sealed or unreachable, so there is a retry per day
# from half-life to expiry rather than two. # from half-life to expiry rather than two.
@ -1100,7 +963,7 @@ in
}; };
}; };
# Signal the hive-c0re lifecycle that a hive CA exists: it bind-mounts # Signal the hive-c0re lifecycle that a trust bundle exists: it bind-mounts
# this file (read-only, public certs ONLY — never a key) into each # this file (read-only, public certs ONLY — never a key) into each
# agent container so agents + their tools can trust the gateway's # agent container so agents + their tools can trust the gateway's
# self-signed leaf, and the meta flake wires the per-agent trust # self-signed leaf, and the meta flake wires the per-agent trust
@ -1130,7 +993,7 @@ in
# variable name. The anchor was never missing; nothing pointed the queue # variable name. The anchor was never missing; nothing pointed the queue
# client at it. # client at it.
# #
# Where there is no hive CA this module contributes nothing at all, and # Where the gateway is not self-signed this module contributes nothing, and
# the clients then fall back to the platform roots — which is correct for # the clients then fall back to the platform roots — which is correct for
# a swarm fronted by a public certificate. # a swarm fronted by a public certificate.
systemd.services.hive-c0re.environment.HIVE_C0RE_OIDC_CA_FILE = systemd.services.hive-c0re.environment.HIVE_C0RE_OIDC_CA_FILE =

123
nix/host-modules/swarm-bao.nix
View file

@ -621,10 +621,9 @@ let
# also what an untuned mount clamps the ROOT to, so the two collided at # also what an untuned mount clamps the ROOT to, so the two collided at
# exactly the length that makes issuance impossible; and the guard below # exactly the length that makes issuance impossible; and the guard below
# compares the issuer's remaining life against this number, which it can # compares the issuer's remaining life against this number, which it can
# only do if this number exists. 720h matches # only do if this number exists. Every per-hive gateway-leaf role carries it
# `deploy.hive-controller.tls.leafValidityDays`, the 30 days ./hive-tls.nix # too. An option, read here and by ./hive-tls.nix's renewal thresholds, so
# documents for every other leaf it holds. An option, read here and by # the two cannot disagree.
# ./hive-tls.nix's renewal threshold, so the two cannot disagree.
servicesPkiLeafTtlHours = baoDeploy.servicesPkiLeafTtlHours; servicesPkiLeafTtlHours = baoDeploy.servicesPkiLeafTtlHours;
servicesPkiLeafTtl = "${toString servicesPkiLeafTtlHours}h"; servicesPkiLeafTtl = "${toString servicesPkiLeafTtlHours}h";
servicesPkiLeafTtlSeconds = servicesPkiLeafTtlHours * 3600; servicesPkiLeafTtlSeconds = servicesPkiLeafTtlHours * 3600;
@ -676,6 +675,15 @@ let
} }
''; '';
# One gateway-leaf issuer per hive in the swarm directory, for the reason
# `perHiveReaders` below is per hive. One name for the pki role, the policy
# and the cert-auth role, as for the readers.
hiveLeafIssuers = lib.mapAttrsToList (hiveName: hive: {
name = "${baoDeploy.hiveLeafPkiRolePrefix}-${hiveName}";
cn = "${baoDeploy.hiveLeafCommonNamePrefix}-${hiveName}";
inherit (hive) domain;
}) hyperhiveCfg.swarm.hives;
# The PKI mount agent client certificates are issued from. Its root is # The PKI mount agent client certificates are issued from. Its root is
# generated inside the store, so the agent CA's key never exists outside it. # generated inside the store, so the agent CA's key never exists outside it.
# A mount of its own because the services mount holds exactly one issuer; a # A mount of its own because the services mount holds exactly one issuer; a
@ -1660,6 +1668,35 @@ in
''; '';
}; };
hiveLeafCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-hive-leaf";
example = "swarm-bao-hive-leaf.svc";
description = ''
Prefix of the subject the store's per-hive gateway-leaf roles accept.
The role for hive `h` accepts `<prefix>-h` and may only issue from
`<servicesPkiMountPath>/issue/<hiveLeafPkiRolePrefix>-h`, which issues
for `h`'s domain and `*.` that domain and nothing else.
Per hive for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is:
one subject for every hive could ask for any hive's certificate.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
'';
};
hiveLeafPkiRolePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-hive-leaf";
description = ''
Prefix of the per-hive roles on
{option}`services.hyperhive.deploy.bao.servicesPkiMountPath` that each
hive's gateway leaf is issued through: `<prefix>-<hive>`. The same name
is that hive's policy and cert-auth role. Must start with `swarm-`.
'';
};
grafanaOidcCommonName = lib.mkOption { grafanaOidcCommonName = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "swarm-bao-grafana-oidc"; default = "swarm-bao-grafana-oidc";
@ -2067,14 +2104,18 @@ in
lib.hasPrefix "swarm-" servicesPkiRoleName lib.hasPrefix "swarm-" servicesPkiRoleName
&& lib.hasPrefix "swarm-" natsPkiRoleName && lib.hasPrefix "swarm-" natsPkiRoleName
&& lib.hasPrefix "swarm-" agentPkiRoleName && lib.hasPrefix "swarm-" agentPkiRoleName
&& lib.hasPrefix "swarm-" baoDeploy.hiveLeafPkiRolePrefix
); );
message = '' message = ''
services.hyperhive.deploy.bao.servicesPkiRoleName services.hyperhive.deploy.bao.servicesPkiRoleName
(${servicesPkiRoleName}), (${servicesPkiRoleName}),
services.hyperhive.deploy.bao.natsPkiRoleName (${natsPkiRoleName}) services.hyperhive.deploy.bao.natsPkiRoleName (${natsPkiRoleName}),
and services.hyperhive.deploy.bao.agentPkiRoleName services.hyperhive.deploy.bao.agentPkiRoleName
(${agentPkiRoleName}) must all start with `swarm-`: the bao granter (${agentPkiRoleName}) and
that writes them may write pki roles under that prefix only. services.hyperhive.deploy.bao.hiveLeafPkiRolePrefix
(${baoDeploy.hiveLeafPkiRolePrefix}) must all start with `swarm-`:
the bao granter that writes them may write pki roles under that
prefix only.
''; '';
} }
]; ];
@ -3014,6 +3055,71 @@ in
''; '';
}; };
# A SIXTH sibling, the nats unit's shape once per hive: each hive's
# gateway-leaf role, its policy and its login role. The role allows the
# hive's domain and `*.` that domain, and no IP SANs or localhost.
systemd.services.swarm-bao-hive-leaf-policy = lib.mkIf haveGranter {
description = "write the per-hive gateway-leaf pki roles, bao policies and cert-auth roles";
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
]
++ granterAfter;
requires = [ "swarm-bao-pki.service" ];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
];
environment = granterEnv;
# Same unseal wait as its siblings above.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
${granterLogin}
''
+ lib.concatMapStrings (obj: ''
bao write ${lib.escapeShellArg "${servicesPkiMountPath}/roles/${obj.name}"} \
allowed_domains=${lib.escapeShellArg "${obj.domain},*.${obj.domain}"} \
allow_bare_domains=true \
allow_subdomains=false \
allow_glob_domains=true \
allow_wildcard_certificates=true \
allow_localhost=false \
allow_any_name=false \
allow_ip_sans=false \
enforce_hostnames=true \
server_flag=true \
client_flag=false \
key_type=rsa \
key_bits=4096 \
ttl=${servicesPkiLeafTtl} \
max_ttl=${servicesPkiLeafTtl}
printf '%s' ${lib.escapeShellArg ''
path "${servicesPkiMountPath}/issue/${obj.name}" {
capabilities = ["update"]
}
''} |
bao policy write ${lib.escapeShellArg obj.name} -
bao write auth/cert/certs/${lib.escapeShellArg obj.name} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg obj.cn} \
token_policies=${lib.escapeShellArg obj.name} \
display_name=${lib.escapeShellArg obj.cn}
'') hiveLeafIssuers;
};
# The agent PKI mount: its root, its one role, the host's copy of the # The agent PKI mount: its root, its one role, the host's copy of the
# root's certificate, and the listener's trust in it. # root's certificate, and the listener's trust in it.
# #
@ -3041,6 +3147,7 @@ in
"swarm-bao-forwarder-oidc-policy.service" "swarm-bao-forwarder-oidc-policy.service"
"swarm-bao-services-issuer-policy.service" "swarm-bao-services-issuer-policy.service"
"swarm-bao-nats-tls-policy.service" "swarm-bao-nats-tls-policy.service"
"swarm-bao-hive-leaf-policy.service"
] ]
++ granterAfter; ++ granterAfter;
requires = [ "swarm-bao-pki.service" ]; requires = [ "swarm-bao-pki.service" ];

3
nix/host-modules/swarm-ca.nix
View file

@ -95,8 +95,7 @@ in
default = 10950; default = 10950;
description = '' description = ''
Validity window of the swarm root CA in days (default ~30y). Validity window of the swarm root CA in days (default ~30y).
Deliberately longer than `services.hyperhive.deploy.hive-controller.tls.caValidityDays`: Deliberately long: the root must outlive the hive CAs it issued, or those chains
the root must outlive the hive CAs it issues, or those chains
expire out from under hives that are still perfectly happy with expire out from under hives that are still perfectly happy with
their own intermediate. Rotating a root is the one operation in their own intermediate. Rotating a root is the one operation in
this system with no partial-failure mode — it invalidates every this system with no partial-failure mode — it invalidates every

7
nix/host-modules/swarm.nix
View file

@ -55,16 +55,17 @@ let
deployCfg.bao.granterCommonName deployCfg.bao.granterCommonName
deployCfg.bao.natsAuthCommonName deployCfg.bao.natsAuthCommonName
] ]
# The two per-hive readers' subjects, spelled out per hive rather than as the # The per-hive subjects, spelled out per hive rather than as the prefix.
# prefix. The prefix alone would reserve the wrong string: the role for hive # The prefix alone would reserve the wrong string: the role for hive
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed # `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
# spelling, and a hive named it would present a leaf that role accepts. # spelling, and a hive named it would present a leaf that role accepts.
# #
# Both spellings are reserved for every declared hive, including the one the # Every spelling is reserved for every declared hive, including the one the
# CN belongs to — a hive cannot be named after its own reader either. # CN belongs to — a hive cannot be named after its own reader either.
++ lib.concatMap (hive: [ ++ lib.concatMap (hive: [
"${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}" "${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}"
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}" "${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
"${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}"
]) hiveNames; ]) hiveNames;
# Public hostnames of the swarm's own services, in declaration order. # Public hostnames of the swarm's own services, in declaration order.

104
nix/module-eval/bao-grants.nix
View file

@ -63,6 +63,21 @@ let
deploy.bao.agentPkiRoleName = "agent"; deploy.bao.agentPkiRoleName = "agent";
}; };
# A gateway-leaf role prefix the granter's `roles/swarm-*` does not reach.
baoGranterOddHiveLeafRole = hive {
deploy.bao.enable = true;
deploy.bao.hiveLeafPkiRolePrefix = "hive-leaf";
};
# The store with a second hive in its directory, at a domain that is not
# `<name>.<swarm.domain>`, so each hive's gateway-leaf grant can be told
# apart from the other's.
baoGrantTwoHives = hive {
deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
swarm.hives.h2.domain = "h2.example.org";
};
# The store and the token, with no CA to trust. `mkForce` because the PKI # The store and the token, with no CA to trust. `mkForce` because the PKI
# glue supplies one by default here — this is the deployment that brings its # glue supplies one by default here — this is the deployment that brings its
# own certificates and has not named the authority yet, in which nothing can # own certificates and has not named the authority yet, in which nothing can
@ -151,7 +166,7 @@ let
_: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile _: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile
) baoGrantWithConsumers.systemd.services; ) baoGrantWithConsumers.systemd.services;
# The thirteen units that write a `swarm-*` grant, by name, for the discovery # The fourteen units that write a `swarm-*` grant, by name, for the discovery
# control below. # control below.
grantingUnitNames = [ grantingUnitNames = [
"swarm-bao-controller-policy" "swarm-bao-controller-policy"
@ -167,6 +182,7 @@ let
"swarm-bao-agent-pki" "swarm-bao-agent-pki"
"swarm-bao-nats-auth-policy" "swarm-bao-nats-auth-policy"
"swarm-bao-operator-viewer-policy" "swarm-bao-operator-viewer-policy"
"swarm-bao-hive-leaf-policy"
]; ];
# Comment lines dropped first: both the HCL and the scripts explain # Comment lines dropped first: both the HCL and the scripts explain
@ -784,24 +800,24 @@ let
} }
{ {
# A store host without the granter's pair writes its grants some other # A store host without the granter's pair writes its grants some other
# way, so none of the thirteen units may exist. Without this arm # way, so none of the fourteen units may exist. Without this arm
# `lib.mkIf haveGranter` could be dropped from any of them and every other # `lib.mkIf haveGranter` could be dropped from any of them and every other
# case here would still pass. # case here would still pass.
name = "without the granter's pair none of the thirteen granting units render"; name = "without the granter's pair none of the fourteen granting units render";
ok = ok =
let let
s = baoGranterOptOut.systemd.services; s = baoGranterOptOut.systemd.services;
in in
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ]) lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
# The control: the same store with the pair renders all thirteen. # The control: the same store with the pair renders all fourteen.
&& lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames; && lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames;
} }
{ {
# 🩸 What replaced the silent skip. With no bootstrap token the thirteen still # 🩸 What replaced the silent skip. With no bootstrap token the fourteen still
# render, and a refused granter fails them with the step that fixes it. # render, and a refused granter fails them with the step that fixes it.
# A store host that never named a token is told to name one, since the # A store host that never named a token is told to name one, since the
# unit that sets the granter up renders only where it has. # unit that sets the granter up renders only where it has.
name = "a store host without a bootstrap token renders the thirteen, each failing loudly with the one-time step"; name = "a store host without a bootstrap token renders the fourteen, each failing loudly with the one-time step";
ok = ok =
let let
s = baoGranterNoToken.systemd.services; s = baoGranterNoToken.systemd.services;
@ -1091,6 +1107,78 @@ let
in in
lib.hasInfix "/granter.pem ]" s && lib.hasInfix "bao-granter \"\" clientAuth" s; lib.hasInfix "/granter.pem ]" s && lib.hasInfix "bao-granter \"\" clientAuth" s;
} }
{
# One role, policy and login role per hive in the directory, each
# naming only its own hive: the policy text in the unit is exactly the
# two `update` grants, and each hive's block names neither the other's
# role nor its subject.
name = "each hive's gateway-leaf identity may issue from its own role alone, for its own domain";
ok =
let
s = baoGrantTwoHives.systemd.services.swarm-bao-hive-leaf-policy.script;
blockOf =
h: next:
lib.head (
lib.splitString "pki/roles/swarm-hive-leaf-${next}" (
lib.last (lib.splitString "pki/roles/swarm-hive-leaf-${h}" s)
)
);
h1 = blockOf "h1" "h2";
h2 = blockOf "h2" "h1";
issues =
h: dom: blk:
lib.hasInfix "allowed_domains='${dom},*.${dom}'" blk
&& lib.hasInfix "allow_ip_sans=false" blk
&& lib.hasInfix "allow_localhost=false" blk
&& lib.hasInfix "allow_any_name=false" blk
&& lib.hasInfix "max_ttl=720h" blk
&& lib.hasInfix "bao policy write swarm-hive-leaf-${h} -" blk
&& lib.hasInfix "auth/cert/certs/swarm-hive-leaf-${h}" blk
&& lib.hasInfix "allowed_common_names=swarm-bao-hive-leaf-${h}" blk
&& lib.hasInfix "token_policies=swarm-hive-leaf-${h}" blk;
in
grantsIn s == [
{
path = "pki/issue/swarm-hive-leaf-h1";
caps = [ "update" ];
}
{
path = "pki/issue/swarm-hive-leaf-h2";
caps = [ "update" ];
}
]
&& issues "h1" "h1.t.local" h1
&& issues "h2" "h2.example.org" h2
&& !(lib.hasInfix "h2" h1)
&& !(lib.hasInfix "h1" h2);
}
{
# The minting side, and the hive's own path to it: the store host signs
# its hive's leaf under the composed subject and points the gateway-leaf
# unit at it.
name = "the PKI unit signs this hive's gateway-leaf identity, and the hive presents it";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
t = baoGrantHere.services.hyperhive.deploy.hive-controller.tls;
in
lib.hasInfix "/hive-leaf.pem ]" s
&& lib.hasInfix "swarm-bao-hive-leaf-h1 \"\" clientAuth" s
&& t.baoHiveIdentityCertFile == "/var/lib/swarm-bao-pki/hive-leaf.pem"
&& t.baoHiveIdentityKeyFile == "/var/lib/swarm-bao-pki/hive-leaf-key.pem"
# Its own leaf, not the reader's or the services issuer's.
&& t.baoHiveIdentityCertFile != baoGrantHere.services.hyperhive.deploy.bao.clientCertFile
&& t.baoHiveIdentityCertFile != t.baoClientCertFile;
}
{
name = "a gateway-leaf role prefix outside swarm-* is refused, naming the option";
ok =
let
names = a: lib.hasInfix "services.hyperhive.deploy.bao.hiveLeafPkiRolePrefix" a.message;
in
lib.any (a: !a.assertion && names a) baoGranterOddHiveLeafRole.assertions
&& !(lib.any (a: !a.assertion && names a) baoGrantHere.assertions);
}
{ {
# The granter writes pki roles through `roles/swarm-*` only, so a role # The granter writes pki roles through `roles/swarm-*` only, so a role
# named otherwise is refused at eval rather than 403'd at deploy. # named otherwise is refused at eval rather than 403'd at deploy.
@ -1304,8 +1392,8 @@ let
} }
{ {
# What makes the case above mean something: discovery by the granter's # What makes the case above mean something: discovery by the granter's
# certificate reaches all thirteen units, and each yields calls. # certificate reaches all fourteen units, and each yields calls.
name = "the granter-policy check sees all thirteen granting units, and parses calls from each"; name = "the granter-policy check sees all fourteen granting units, and parses calls from each";
ok = ok =
lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits) && lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits)

145
nix/module-eval/hive-tls.nix
View file

@ -3,7 +3,8 @@
# #
# The swarm-services leaf's renewal: a timer that really re-runs the # The swarm-services leaf's renewal: a timer that really re-runs the
# issuance, at a threshold read off the store role's lifetime. And the # issuance, at a threshold read off the store role's lifetime. And the
# store's root on a hive that asks it for no leaf. # store's root on a hive that asks it for no leaf. And the gateway leaf,
# issued by the store through the hive's own role, with no hive CA key.
{ {
pkgs, pkgs,
lib, lib,
@ -38,15 +39,17 @@ let
}; };
# Fronting none of the swarm's names, reading a store it does not run, # Fronting none of the swarm's names, reading a store it does not run,
# with the reader identity placed by hand. # with the reader identity and the hive's own identity placed by hand.
noService = hive { noService = hive {
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem"; deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem";
deploy.hive-controller.tls.baoHiveIdentityCertFile = "/etc/pki/bao-hive.pem";
deploy.hive-controller.tls.baoHiveIdentityKeyFile = "/etc/pki/bao-hive-key.pem";
}; };
noSvcFetch = noService.systemd.services.swarm-services-cert.script; noSvcFetch = noService.systemd.services.swarm-services-cert.script;
# The same hive with no reader identity at all. # The same hive with no store identity at all.
noIdentity = hive { }; noIdentity = hive { };
noIdFetch = noIdentity.systemd.services.swarm-services-cert.script; noIdFetch = noIdentity.systemd.services.swarm-services-cert.script;
@ -60,6 +63,8 @@ let
units = allLocal.systemd.services; units = allLocal.systemd.services;
boot = units.swarm-services-cert; boot = units.swarm-services-cert;
caUnit = units.hive-tls-ca;
leafUnit = units.hive-tls-resign;
timer = allLocal.systemd.timers.swarm-services-cert-renew; timer = allLocal.systemd.timers.swarm-services-cert-renew;
# What the timer starts, read off the timer rather than assumed from its # What the timer starts, read off the timer rather than assumed from its
@ -151,13 +156,16 @@ let
&& lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' noService.systemd.services.hive-tls-ca.script; && lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' noService.systemd.services.hive-tls-ca.script;
} }
{ {
name = "the root fetch reaches the store with the hive's own reader identity and server CA"; # The hive's own identity, not its reader identity: the reader's policy
# reads every secret the hive is entitled to.
name = "the root fetch reaches the store with the hive's own identity and server CA";
ok = ok =
lib.hasInfix "--cacert ${lib.escapeShellArg "/etc/pki/bao-ca.pem"}" noSvcFetch lib.hasInfix "--cacert ${lib.escapeShellArg "/etc/pki/bao-ca.pem"}" noSvcFetch
&& lib.hasInfix ( && lib.hasInfix (
"--cert ${lib.escapeShellArg "/etc/pki/bao-client.pem"}" "--cert ${lib.escapeShellArg "/etc/pki/bao-hive.pem"}"
+ " --key ${lib.escapeShellArg "/etc/pki/bao-client-key.pem"}" + " --key ${lib.escapeShellArg "/etc/pki/bao-hive-key.pem"}"
) noSvcFetch; ) noSvcFetch
&& !(lib.hasInfix "/etc/pki/bao-client.pem" (noSvcBranch noSvcFetch));
} }
{ {
# No identity to present: the unit fails, naming both options, rather # No identity to present: the unit fails, naming both options, rather
@ -166,11 +174,11 @@ let
name = "a hive fronting no swarm name with no store identity fails the unit, naming both options"; name = "a hive fronting no swarm name with no store identity fails the unit, naming both options";
ok = ok =
noIdentity.services.hyperhive.swarm.localServiceDomains == [ ] noIdentity.services.hyperhive.swarm.localServiceDomains == [ ]
&& noIdentity.services.hyperhive.deploy.bao.clientCertFile == null && noIdentity.services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile == null
&& !(lib.hasInfix "/v1/pki/ca/pem" noIdFetch) && !(lib.hasInfix "/v1/pki/ca/pem" noIdFetch)
&& lib.hasInfix "services.hyperhive.deploy.bao.clientCertFile" (noSvcBranch noIdFetch) && lib.hasInfix "baoHiveIdentityCertFile (unset)" (noSvcBranch noIdFetch)
&& lib.hasInfix "services.hyperhive.deploy.bao.clientKeyFile" (noSvcBranch noIdFetch) && lib.hasInfix "baoHiveIdentityKeyFile (unset)" (noSvcBranch noIdFetch)
&& lib.hasSuffix "exit 1" (noSvcBranch noIdFetch) && lib.hasSuffix "exit 1" (lib.removeSuffix "\n" (noSvcBranch noIdFetch))
&& !(lib.hasInfix "exit 0" (noSvcBranch noIdFetch)) && !(lib.hasInfix "exit 0" (noSvcBranch noIdFetch))
&& !(lib.hasInfix "not staged" noIdFetch) && !(lib.hasInfix "not staged" noIdFetch)
&& lib.hasInfix "/v1/pki/ca/pem" (noSvcBranch noSvcFetch); && lib.hasInfix "/v1/pki/ca/pem" (noSvcBranch noSvcFetch);
@ -179,13 +187,11 @@ let
# An identity named but not on disk fails before curl, naming the paths. # An identity named but not on disk fails before curl, naming the paths.
name = "a hive fronting no swarm name whose identity files are absent fails the unit, naming the paths"; name = "a hive fronting no swarm name whose identity files are absent fails the unit, naming the paths";
ok = ok =
lib.hasInfix "for f in /etc/pki/bao-client.pem /etc/pki/bao-client-key.pem; do" ( lib.hasInfix "for f in /etc/pki/bao-hive.pem /etc/pki/bao-hive-key.pem; do" (noSvcBranch noSvcFetch)
noSvcBranch noSvcFetch
)
&& lib.hasInfix ''[ -r "$f" ] || missing="$missing $f"'' (noSvcBranch noSvcFetch) && lib.hasInfix ''[ -r "$f" ] || missing="$missing $f"'' (noSvcBranch noSvcFetch)
&& lib.hasInfix "cannot read$missing" (noSvcBranch noSvcFetch) && lib.hasInfix "cannot read$missing" (noSvcBranch noSvcFetch)
&& lib.hasInfix "clientCertFile (/etc/pki/bao-client.pem)" (noSvcBranch noSvcFetch) && lib.hasInfix "baoHiveIdentityCertFile (/etc/pki/bao-hive.pem)" (noSvcBranch noSvcFetch)
&& lib.hasInfix "clientKeyFile (/etc/pki/bao-client-key.pem)" (noSvcBranch noSvcFetch); && lib.hasInfix "baoHiveIdentityKeyFile (/etc/pki/bao-hive-key.pem)" (noSvcBranch noSvcFetch);
} }
{ {
# The fetch retries against a sealed store; a gateway `Requires=` on it # The fetch retries against a sealed store; a gateway `Requires=` on it
@ -201,6 +207,113 @@ let
in in
!(holds noService) && holds allLocal; !(holds noService) && holds allLocal;
} }
{
# The controls are the store's PKI unit, which still mints its own CA
# with `openssl req`, and the swarm root's unit, which still reads
# `root-key.pem`: absence here is not a matcher that cannot match.
name = "no hive TLS unit generates a CA, signs with a CA key or reads the swarm root key";
ok =
let
hasCaWork =
s:
lib.any (p: lib.hasInfix p s) [
"openssl req"
"-CAkey"
"root-key.pem"
];
pkiScript = units.swarm-bao-pki.script;
rootScript = units.swarm-ca.script;
in
!(hasCaWork caUnit.script)
&& !(hasCaWork leafUnit.script)
&& !(hasCaWork noService.systemd.services.hive-tls-ca.script)
&& lib.hasInfix "openssl req" pkiScript
&& lib.hasInfix "root-key.pem" rootScript;
}
{
name = "a retired hive CA key is deleted, and its public certificate stays in the trust bundle";
ok =
lib.hasInfix ''rm -f "$d/ca-key.pem"'' caUnit.script
&& lib.hasInfix ''ca="$d/ca.pem"'' caUnit.script
&& lib.hasInfix ''if [ -s "$ca" ]; then anchors+=("$ca"); fi'' caUnit.script
&& lib.hasInfix ''if [ -s "$prev" ]; then anchors+=("$prev"); fi'' caUnit.script
&& lib.hasInfix ''cat "''${anchors[@]}" > "$bundle"'' caUnit.script;
}
{
# The store host's hive presents the leaf ./glue-bao-tls.nix mints for
# it; not the reader's, not the services issuer's.
name = "the gateway leaf is issued through this hive's own role, presenting its own identity";
ok =
let
t = allLocal.services.hyperhive.deploy.hive-controller.tls;
in
lib.hasInfix "pki/issue/swarm-hive-leaf-h1" leafUnit.script
&& lib.hasInfix "common_name=h1.t.local" leafUnit.script
&& lib.hasInfix "alt_names='*.h1.t.local'" leafUnit.script
&& lib.hasInfix "bao login -method=cert -token-only" leafUnit.script
&& leafUnit.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/hive-leaf.pem"
&& leafUnit.environment.BAO_CLIENT_KEY == "/var/lib/swarm-bao-pki/hive-leaf-key.pem"
&& t.baoHiveIdentityCertFile == leafUnit.environment.BAO_CLIENT_CERT
&& leafUnit.environment.BAO_CLIENT_CERT != t.baoClientCertFile
&& leafUnit.environment.BAO_CLIENT_CERT != allLocal.services.hyperhive.deploy.bao.clientCertFile;
}
{
# Checked before the leaf, so the unit is red even while a valid leaf
# keeps the gateway up. The control is a hive with the identity named,
# whose script checks the files instead.
name = "a hive with no store identity fails the gateway-leaf unit, naming both options";
ok =
let
s = noIdentity.systemd.services.hive-tls-resign.script;
before = lib.head (lib.splitString "reissue=0" s);
ok' = noService.systemd.services.hive-tls-resign.script;
in
lib.hasInfix "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile (unset)" before
&& lib.hasInfix "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile (unset)" before
&& lib.hasInfix "exit 1" before
&& lib.hasInfix "for f in /etc/pki/bao-hive.pem /etc/pki/bao-hive-key.pem; do" (
lib.head (lib.splitString "reissue=0" ok')
)
&& !(lib.hasInfix "(unset)" ok');
}
{
# A leaf the retired hive CA signed is re-issued from the store on the
# first run that reaches it; one that renews in time never is.
name = "the gateway leaf is re-issued when not chained to the store's root, and at half its role lifetime";
ok =
lib.hasInfix ''openssl verify -CAfile "$svcroot" "$leaf"'' leafUnit.script
&& lib.hasInfix (renewAt 720) leafUnit.script
&& lib.hasInfix (renewAt 48) shortTtl.systemd.services.hive-tls-resign.script
&& lib.hasInfix ''-in "$leaf" -noout -checkend "$renewat"'' leafUnit.script;
}
{
# A valid leaf on disk starts the gateway with the store sealed: nothing
# requires or waits on the issuing unit. The control is that unit
# itself, which does wait on the store's container.
name = "the gateway does not wait on the gateway-leaf unit, which waits on the store";
ok =
(leafUnit.requiredBy or [ ]) == [ ]
&& (leafUnit.before or [ ]) == [ ]
&& !(lib.any (u: lib.elem "hive-tls-resign.service" ((u.requires or [ ]) ++ (u.after or [ ]))) (
lib.attrValues (removeAttrs units [ "hive-tls-resign" ])
))
&& !(lib.elem "container@swarm-bao.service" ((caUnit.requires or [ ]) ++ (caUnit.after or [ ])))
&& lib.elem "container@swarm-bao.service" leafUnit.after
&& lib.elem "multi-user.target" leafUnit.wantedBy;
}
{
# Not `RemainAfterExit`, or the timer would start an active unit and
# renew nothing.
name = "a daily timer re-runs the gateway-leaf unit";
ok =
let
t = allLocal.systemd.timers.hive-tls-resign;
in
lib.elem "timers.target" t.wantedBy
&& t.timerConfig.OnCalendar == "daily"
&& (t.timerConfig.Unit or "hive-tls-resign.service") == "hive-tls-resign.service"
&& !(remainsAfterExit leafUnit);
}
]; ];
in in
runGroup "hive-tls" cases runGroup "hive-tls" cases

19
nix/module-eval/name-guards.nix
View file

@ -106,6 +106,14 @@ let
swarm.hives.qr-other.domain = "q.t.local"; swarm.hives.qr-other.domain = "q.t.local";
}; };
# The per-hive gateway-leaf subject, composed the same way.
hiveNamedAfterHiveLeafSubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao.hiveLeafCommonNamePrefix = "hl";
swarm.hives.other.domain = "o.t.local";
swarm.hives.hl-other.domain = "hl.t.local";
};
hiveNameWithComposedWord = hive { hiveNameWithComposedWord = hive {
deploy.swarm-otel.enable = false; deploy.swarm-otel.enable = false;
swarm.hives."h1-agent".domain = "a.t.local"; swarm.hives."h1-agent".domain = "a.t.local";
@ -203,6 +211,17 @@ let
a: !a.assertion && lib.hasInfix "'qr-other'" a.message a: !a.assertion && lib.hasInfix "'qr-other'" a.message
) hiveNamedAfterPerHiveReaderSubject.assertions; ) hiveNamedAfterPerHiveReaderSubject.assertions;
} }
{
# A hive taking this spelling would present a leaf hive `other`'s
# gateway-leaf role accepts, and could ask the store for that hive's
# certificate. The control is the legal roster below.
name = "a hive named after another hive's gateway-leaf subject is refused";
ok =
equalityGuardFired hiveNamedAfterHiveLeafSubject
&& lib.any (
a: !a.assertion && lib.hasInfix "'hl-other'" a.message
) hiveNamedAfterHiveLeafSubject.assertions;
}
{ {
# Without this the case above proves nothing: an arm that fires for every # Without this the case above proves nothing: an arm that fires for every
# roster is not a guard, and `hives` is non-empty in both fixtures. # roster is not a guard, and `hives` is non-empty in both fixtures.