vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-tls: issue the gateway leaf from bao with a per-hive identity

Browse source 
Every hive gets its own store identity (CN
<deploy.bao.hiveLeafCommonNamePrefix>-<hive>) and its own role on the
existing pki mount (<deploy.bao.hiveLeafPkiRolePrefix>-<hive>), whose
allowed_domains is the hive's domain and *.<domain>. The policy grants
update on that one issue path. swarm-bao-hive-leaf-policy writes role,
policy and cert-auth role for every hive in swarm.hives; the store's host
mints its own hive's identity in glue-bao-tls.nix and a new glue file
points deploy.hive-controller.tls.baoHiveIdentity{Cert,Key}File at it.
Hives on other hosts place it by hand. swarm.nix reserves the composed CN.

hive-tls-ca no longer generates a hive CA or signs a leaf: caGenScript,
signLeafScript/signHiveLeaf and the adoption block are gone, ca-key.pem
is deleted where it exists, and the unit only assembles trust-bundle.pem.
The retired ca.pem/ca-previous.pem stay in the bundle so leaves they
signed keep verifying.

hive-tls-resign becomes the bao issuance: at boot, on deploy and from a
daily timer it re-issues when the leaf is missing, past half the role's
lifetime, missing a name, or not chained to the store's pki root. Nothing
orders the gateway after it, so a valid leaf on disk never waits on the
store; a failed renewal leaves the leaf in place. A missing identity
fails the unit naming both options, as does the root fetch on a hive
fronting no swarm name, which now presents this identity instead of the
reader cert.

caValidityDays and leafValidityDays are removed (both spellings), since
nothing reads them.

Refs #4588
This commit is contained in:
atlas 2026-10-04 15:11:48 +02:00 • committed by mara
commit 5526daab97
16 changed files with 859 additions and 628 deletions
Download patch file Download diff file Expand all files Collapse all files

14
docs/networking/gateway.md
View file

@ -46,7 +46,7 @@ The gateway always terminates TLS. Which certificate it serves depends on what y
| mode | config | cert source | `.well-known` scheme |
|---|---|---|---|
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | the host's hive CA signs a gateway leaf (RSA-4096) | `https` |
| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | the secret store issues a gateway leaf through the hive's own role (RSA-4096) | `https` |
| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
@ -74,19 +74,19 @@ nginx obtains and renews certs via the HTTP-01 challenge on `port` (default 80);
On by default, listening on `httpsPort` (default 443) beside the plain-http `port` (default 80).
The issuer is a **host-held hive CA**, not a bare self-signed leaf. `hive-tls-ca.service` (from `hive-tls.nix`) generates a long-lived CA (`services.hyperhive.deploy.hive-controller.tls.caValidityDays`, default 7300 days) under `…tls.stateDir` (default `/var/lib/hive-tls`) and signs a gateway **leaf** with it (`leafValidityDays`, default 30). `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. `hive-tls-resign.service` (from `hive-tls.nix`) asks the store for a gateway **leaf** through this hive's own role, presenting the hive's own store identity (`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile`), and writes it under `…tls.stateDir` (default `/var/lib/hive-tls`). `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). [`swarm/ca.md`](../swarm/ca.md) has the role, the identity and what a hive without one does.
⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the CA's `0600 root:root` key), and it makes sure **every cert path the config names exists** — when the swarm-services leaf is missing it installs the hive leaf in its place. nginx refuses a config naming a missing cert file, so without that fallback one missing leaf takes down every vhost, not just one.
⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the source's `0600 root:root` key), and it makes sure **every cert path the config names exists** — when the swarm-services leaf is missing it installs the hive leaf in its place. nginx refuses a config naming a missing cert file, so without that fallback one missing leaf takes down every vhost, not just one.
**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the stable CA once; leaf rotation never breaks them.
**Why an issuer, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the store's root once; leaf rotation never breaks them.
**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is an intermediate under the swarm root ([`swarm/ca.md`](../swarm/ca.md) has the hierarchy), and a verifier can't stop at an intermediate — so the bundle carries the hive CA plus its root. nginx serves the leaf with the hive CA appended for the same reason. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle.
**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root, plus the retired hive CA's public `ca.pem` and the swarm root where they exist ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle.
**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https://<host>/.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap.
**Cert shape**: the leaf's CN is the bare hive domain; its SANs are `<hive>` and `*.<hive>`. The hive CA is name-constrained to `<hive>`, so it can't sign a swarm-service name outside it — a violating SAN would invalidate the whole leaf. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
**Cert shape**: the leaf's CN is the bare hive domain; its SANs are `<hive>` and `*.<hive>`. The hive's role allows nothing outside `<hive>`, so a swarm-service name can't go on this leaf. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
**Rotation**: `hive-tls-ca.service` re-signs the leaf when it's missing, within 30 days of expiry, or no longer covers the configured names, always under the same CA. It regenerates the CA only if missing or expired. To force a leaf rotation, delete `gateway.pem` under the state dir, restart the unit, then reload `nginx`.
**Rotation**: `hive-tls-resign.service` runs at boot, on every deploy and once a day, and asks the store for a new leaf when the current one is missing, past half its 720-hour lifetime, no longer covers the configured names, or doesn't chain to the store's root. A leaf already on disk lets the gateway start while the store can't answer; a failed renewal leaves it in place. To force a rotation, delete `gateway.pem` under the state dir and restart the unit; it re-imports the leaf and reloads `nginx` itself.
**Cert prompts**: browsers warn once per host until you add the hive's `trust-bundle.pem` (the anchor, not the leaf) to the browser or OS trust store. A container that needs to trust it for its own outbound TLS gets the bundle a different way, bind-mounted read-only at `/run/hive-ca/trust-bundle.pem` (`nix/host-modules/lib/hive-ca-trust.nix`).

2
docs/swarm/README.md
View file

@ -33,7 +33,7 @@ grafana, the controller, authelia, …) →
lives → [`secrets.md`](secrets.md) · the per-secret minter/reader/renewal
contract → [`credentials.md`](credentials.md) · how the store comes up,
who writes its grants and how it unseals → [`bao.md`](bao.md)
- **swarm CA** — the root every hive's internal TLS chains to, and what to
- **swarm CA** — the store root every hive's gateway leaf chains to, and what to
hand a peer (`trust-bundle.pem`, never `ca.pem`). → [`ca.md`](ca.md)
- **snapshot store** — the swarm's one `btrfs receive` endpoint,
`swarm.snapshotStore.{address,port}`. →

231
docs/swarm/ca.md
View file

@ -1,52 +1,100 @@
# Swarm CA
A hive's internal TLS chains to a **swarm root CA**: the root signs each
hive's own CA, and that hive CA signs the gateway leaf. A peer that
trusts the root once validates every hive in the swarm, present and
future, instead of pinning to each one by hand.
A hive's gateway serves a leaf issued by the secret store's `pki` mount,
through a role that belongs to that hive. A peer that trusts the mount's
root once validates every hive's gateway and every swarm service name,
present and future, instead of pinning to each one by hand.
That's the whole point of the hierarchy — it turns per-peer trust from
O(n²) hand-pinning into one anchor per swarm.
That's the point of one issuer — it turns per-peer trust from O(n²)
hand-pinning into one anchor per swarm.
## Two provisioning modes, one structure
## The gateway leaf comes from the secret store
What differs is who puts the artifacts on disk:
`hive-tls-resign.service` logs in to the store with this hive's own
identity, calls `pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName>`
(`swarm-hive-leaf-<hiveName>` by default), and writes the result to
`<deploy.hive-controller.tls.stateDir>/gateway.pem` and `gateway-key.pem`,
the files the gateway copies from. The leaf's subject is the hive domain,
and its SANs are the hive domain and `*.<hive domain>`.
| | swarm root | this hive's CA |
| ---------------------------------------- | --------------------------------------------- | ---------------------------------------------- |
| default | operator-provided | operator-provided, else self-signed as before |
| `autoConfigure = true` (all on one host) | generated by `swarm-ca.service` on first boot | issued by `hive-tls-ca.service` under the root |
**Each hive's role covers that hive alone.** Its `allowed_domains` is the
hive's domain and `*.<hive domain>`, read from
`services.hyperhive.swarm.hives.<name>.domain` on the store's host, with no
IP SANs and no `localhost`. A request for any other name is a refusal from
the store naming that name. The hive's policy grants `update` on that one
issue path and nothing else. `swarm-bao-hive-leaf-policy.service`, on the
store's host, writes the role, the policy and the cert-auth role for every
hive in `swarm.hives`.
`services.hyperhive.swarm.ca.autoConfigure` selects between them, and is
**off by default**: a swarm's services and its hives can live on
different hosts, and a host can't tell whether it's the one holding
the root, so setting the swarm CA up is an operator action rather than
something a host assumes. Turn it on for an all-on-one-host deployment
and the hierarchy costs no configuration.
**Every hive needs its own store identity.** It's a client certificate
from the store's own client CA, with the subject
`<deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`
(`swarm-bao-hive-leaf-<hiveName>` by default), and it's named by
`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` and
`baoHiveIdentityKeyFile`:
It defaults from `services.hyperhive.deploy.singleHostSwarm`, the single
switch that says "this box is the whole deployment."
- On the host running the store, `swarm-bao-pki.service` mints it at
`/var/lib/swarm-bao-pki/hive-leaf.pem` and the two options default to it.
- On every other host, copy a leaf with that subject from the store's host
and set both options, before the host's next deploy.
**A hive given neither artifact keeps the self-signed CA it has always
had.** It serves TLS exactly as before and isn't part of a
swarm's trust hierarchy — the right outcome for a hive nobody has
federated yet. Only `autoConfigure` issues a hive sub-CA, because only
that case can: signing one needs the root's private key.
Without it, `hive-tls-resign.service` fails with a message naming both
options, and so does `swarm-services-cert.service` on a hive fronting no
swarm name. The unit checks the identity before the leaf, so it fails even while a
valid leaf keeps the gateway up. The identity is the hive's alone:
it issues this hive's leaf and fetches the store's root, and it's never the
identity a swarm service uses.
Moving the swarm CA onto its own host is then a matter of moving
`services.hyperhive.swarm.ca.stateDir` and leaving `autoConfigure` off —
there is no second code path to switch to.
**When it issues.** The unit runs at boot, on every deploy, and from
`hive-tls-resign.timer` once a day. It asks the store for a new leaf when
the current one is missing, past half the role's 720-hour lifetime, missing
a configured name, or doesn't chain to the store's root
(`swarm-services-root.pem`). That last check is what replaces a leaf the
retired hive CA signed, on the first run that reaches the store.
### The swarm's service names come from the secret store
**The gateway doesn't wait for the store.** Nothing orders the gateway after
`hive-tls-resign.service`, so a hive holding a valid leaf starts its gateway
while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the
current leaf in place, says so in the journal, and retries every 30 seconds
for a day. Only a hive with no leaf yet needs the store reachable and
unsealed: its gateway's certificate import fails until the first leaf
lands, and then the unit starts nginx.
nginx serves the leaf alone, without its issuer: the issuer is the
self-signed root a consumer already holds in its trust bundle.
## The trust bundle
`hive-tls-ca.service` writes `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`,
the file every consumer anchors on. It holds whichever of these exist:
- `swarm-services-root.pem`, the store's `pki` root, which the gateway leaf
and the swarm-services leaf both chain to.
- `ca.pem` and `ca-previous.pem`, the retired hive CA's public certificates,
so a leaf it signed keeps verifying until it's replaced. Removing them is a
deliberate later step.
- the swarm root, `<swarm.ca.stateDir>/root.pem`, on the host that generates
it.
No hive keeps a CA key. `hive-tls-ca.service` deletes
`<deploy.hive-controller.tls.stateDir>/ca-key.pem` where an earlier
generation left one, and generates no CA.
⚠️ **A consumer reads its trust when it starts.** An agent or service
container that last started before the store's root reached the bundle
fails TLS to a gateway serving a store-issued leaf, until it restarts.
## The swarm's service names come from the secret store
The swarm's service names (`swarm.forge.domain`,
`swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to
siblings of the hive domain — `forge.<swarm.domain>`, not
`forge.<hive domain>`. The hive CA's leaf is a **single-label** wildcard
over the hive's own domain, so it can't cover them.
`forge.<hive domain>`. The gateway leaf carries the hive's domain and a
**single-label** wildcard over it, and its role allows nothing outside the hive's
domain, so it can't cover them.
The secret store's `pki` mount issues that leaf instead, from a
root generated inside it. `swarm-services-cert.service` logs in with
The secret store's `pki` mount issues them a leaf of their own, through its
`swarm-services` role. `swarm-services-cert.service` logs in with
this host's services-issuer certificate, calls
`pki/issue/swarm-services`, and writes the result to
`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already
@ -65,10 +113,11 @@ gateway can't serve a swarm service that runs behind another.
**The root's public certificate is a file, on every hive:**
`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the
leaf it signed and folded into `trust-bundle.pem`. A hive fronting no
swarm name asks for no leaf and reads the root alone, from the store's
`pki/ca/pem`, presenting its store identity
(`deploy.bao.clientCertFile` / `clientKeyFile`) for the handshake. Every hive
must hold one: without it, or with either file absent,
swarm name asks for no services leaf and reads the root alone, from the store's
`pki/ca/pem`, presenting its own store identity
(`deploy.hive-controller.tls.baoHiveIdentityCertFile` /
`baoHiveIdentityKeyFile`) for the handshake. Every hive must hold one:
without it, or with either file absent,
`swarm-services-cert.service` fails with a message naming both options.
On the host running
the store it's also at
@ -98,32 +147,24 @@ list-timers swarm-services-cert-renew` shows the next run.
## Constraints on the material
The root's private key never reaches the nix store: the store is
No CA private key exists on any hive. The `pki` root's key stays inside the
store's mount (`root/generate/internal`), and each hive's narrowing is its
role's `allowed_domains`, enforced by the store at issue time: a hive's
identity that leaks can get certificates for that hive's names and no
others.
The swarm root's private key never reaches the nix store: the store is
world-readable and content-addressed, so a key committed to a flake is a
key published to everyone who builds it. The swarm distributes only
certificates.
Each hive CA is **name-constrained** (X.509 `nameConstraints`) to that
hive's own domain, so a hive CA that leaks can only mint names inside
its own subdomain — enforced by every verifier rather than by
convention. The constraint excludes both IP families as well, since a
permitted-DNS-only constraint says nothing about IP SANs.
<!-- vale write-good.Passive = NO -->
The root is issued with `pathlen:1`: it may sign hive CAs, and those may
sign leaves, and the chain stops there.
<!-- vale write-good.Passive = YES -->
## What to hand a peer
`hivectl peer-config` prints the `services.hyperhive.swarm.hives."<name>"`
block a peer operator pastes into their own config. When this hive's
gateway serves a self-signed leaf under the hierarchy (detected by the
presence of `<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it
also prints a one-time
`scp` line installing the **swarm root** —
gateway serves a self-signed leaf (detected by the presence of
`<deploy.hive-controller.tls.stateDir>/trust-bundle.pem`), it also prints a
one-time `scp` line installing the **swarm root** —
`<swarm.ca.stateDir>/root.pem`, not this hive's own CA — on the peer's
host:
@ -132,80 +173,14 @@ scp /var/lib/swarm-ca/root.pem <peer-host>:/var/lib/swarm-ca/root.pem
```
It then prints the secret store's `pki` root itself, read from
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues the swarm's
service names under that root, not the swarm root, so a peer host trusting only
the swarm root can't verify them.
`/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's
gateway leaf and the swarm's service names under that root, so a peer host
trusting only the swarm root can't verify them.
That's the point of the hierarchy: the operator installs the root
**once per swarm host**, not once per peer, so a hive joining later
needs no edit on the hives already running. A hive whose cert already chains to a public
CA has nothing to install — `peer-config` says so instead.
Handing a peer this hive's own `ca.pem` would not work even as a
one-off: once a hive CA is an intermediate under the swarm root, it's
no longer something a verifier can build a chain _to_ — OpenSSL won't
terminate a chain at a trusted non-self-signed certificate without
`-partial_chain`. That's why the tool distributes the root, not a
per-hive file.
## Adopting the hierarchy on an existing hive
<!-- vale write-good.Passive = NO -->
A hive that predates the swarm root carries a self-signed `ca.pem`, and
adopting the hierarchy means replacing it. That invalidates an anchor
consumers already trust, and they refresh on their own schedule — agents
only pick up new trust when their container restarts, peers only on
their own rebuild. **Who is allowed to decide that's what splits the
two cases.**
<!-- vale write-good.Passive = YES -->
### Where this host owns the root (`autoConfigure`)
Adoption happens by itself, once. `hive-tls-ca.service` notices that
`ca.pem` doesn't chain to the root, keeps the old certificate as
`ca-previous.pem`, and re-issues under the root; the new CA signs the
next leaf.
It's safe to automate here precisely because this is the
all-on-one-host shape: every consumer is on this box, so "when will they
have refreshed" is knowable rather than guessed.
The old CA stays in `trust-bundle.pem` afterwards, so adoption is
**additive to the anchor set before it's subtractive** — a container
that hasn't restarted yet still validates. Removing `ca-previous.pem`
is a deliberate later step: how long is long enough is a property of the
deployment, not something the unit can know.
A marker file (`.swarm-ca-adopted`) records that this ran. Its _absence_
is the trigger, so adoption fires once per hive rather than being
re-decided on every activation.
### Everywhere else
No automatic adoption. `hive-tls-ca.service` fails, loudly, naming both
certificates and giving the two-command recipe:
```
rm <deploy.hive-controller.tls.stateDir>/ca.pem <deploy.hive-controller.tls.stateDir>/ca-key.pem
systemctl restart hive-tls-ca.service
```
<!-- vale write-good.Passive = NO -->
Failing rather than warning is deliberate: a hive whose CA doesn't
chain to the root it received is misconfigured, and a warning in a
build log isn't something anyone reads twice.
<!-- vale write-good.Passive = YES -->
To keep the current CA on purpose — a hive that deliberately stays
outside the hierarchy, or one mid-migration — `touch` the marker file
named in the message. That's a decision, and it's recorded as one.
A hive with **no** root configured at all isn't affected by any of
this: it self-signs exactly as it always has.
The operator installs the roots **once per swarm host**, not once per peer,
so a hive joining later needs no edit on the hives already running. A hive
whose cert already chains to a public CA has nothing to install —
`peer-config` says so instead.
## Distributing the root

29
docs/swarm/secrets.md
View file

@ -155,14 +155,14 @@ when the system builds. The server names the offending file and refuses to run.
## Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
| secret | generated by | lives at |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
Both delivery units wait for authelia's first boot to mint the secret — a
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
@ -477,13 +477,14 @@ it each time the step succeeds, so OIDC login needs no further restart.
## The constraint that decides where the root lives
A hive CA carries `nameConstraints=permitted;DNS:<hive domain>`, and **a swarm
service name is a sibling of the hive domain rather than a child** — `forge.<swarm>`
next to `<hive>.<swarm>`. A hive CA can't issue a certificate for a swarm
service. Not by policy: by construction, and openssl enforces it.
The store issues a hive's gateway leaf through a role whose `allowed_domains` is
that hive's domain and `*.` that domain, and **a swarm service name is a sibling of
the hive domain rather than a child** — `forge.<swarm>` next to `<hive>.<swarm>`.
That role can't issue a certificate for a swarm service; the store refuses it.
Whatever holds the swarm root is therefore what makes swarm-service certificates
possible at all. Two things follow:
The store's `swarm-services` role issues them instead, from the root inside its
`pki` mount. The swarm root (`swarm.ca.*`) is still generated on one host, and
two things follow for it:
- **The root's private key is a runtime file and must never enter the nix
store**, so nothing build-time can name it — the system reads

1
nix/host-modules/default.nix
View file

@ -32,6 +32,7 @@
./glue-controller-bao-identity.nix
./glue-forge-oidc-client.nix
./glue-grafana-oidc-client.nix
./glue-hive-tls-bao-identity.nix
./glue-matrix-bao-token.nix
./glue-matrix-ctl-bao-identity.nix
./glue-nats-auth-bao-identity.nix

42
nix/host-modules/deploy.nix
View file

@ -325,23 +325,45 @@ in
[ "services" "hyperhive" "deploy" "singleHostSwarm" ]
)
# The hive CA's own knobs. They sat at the TOP of `services.hyperhive`,
# which is meant to be everything about hyperhive rather than the settings
# of one hive — and where the CA lives, how long it lasts and how long its
# leaves last are decisions of the host that holds the key. `hive-controller`
# is hive-c0re's new name (mara, on the issue), so the daemon that owns the
# CA is what they hang off.
# The hive TLS state dir sat at the TOP of `services.hyperhive`, which is
# meant to be everything about hyperhive rather than the settings of one
# hive. `hive-controller` is hive-c0re's new name (mara, on the issue).
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "tls" "stateDir" ]
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "stateDir" ]
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "tls" "caValidityDays" ]
# The hive CA's lifetime and its leaves', both spellings, removed rather
# than renamed: no hive CA is generated, and the store's per-hive role
# sets the gateway leaf's lifetime.
(lib.mkRemovedOptionModule [ "services" "hyperhive" "tls" "caValidityDays" ] ''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
'')
(lib.mkRemovedOptionModule [ "services" "hyperhive" "tls" "leafValidityDays" ] ''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
'')
(lib.mkRemovedOptionModule
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "caValidityDays" ]
''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
''
)
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "tls" "leafValidityDays" ]
(lib.mkRemovedOptionModule
[ "services" "hyperhive" "deploy" "hive-controller" "tls" "leafValidityDays" ]
''
No hive CA is generated any more: the gateway leaf is issued by the
secret store's pki mount, whose per-hive role sets its lifetime
(services.hyperhive.deploy.bao.servicesPkiLeafTtlHours). Remove this
definition. See docs/swarm/ca.md.
''
)
# The WireGuard mesh, whole. Unlike every rename above this one moves a

8
nix/host-modules/glue-bao-tls.nix
View file

@ -261,6 +261,14 @@ in
# is a swarm singleton, so elsewhere this is the file an operator copies.
[ -s ${pkiDir}/nats-auth.pem ] || ${signLeaf} ${pkiDir} nats-auth \
${lib.escapeShellArg deployCfg.bao.natsAuthCommonName} "" clientAuth
# This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
# subject carries this hive's name, like the matrix-token and
# queue-agent leaves above, because ./swarm-bao.nix writes one role per
# hive. Hives on other hosts are issued theirs out of band.
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \
${lib.escapeShellArg "${deployCfg.bao.hiveLeafCommonNamePrefix}-${clientCn}"} "" clientAuth
'';
};
};

34
nix/host-modules/glue-hive-tls-bao-identity.nix Normal file
View file

@ -0,0 +1,34 @@
# Glue: point the gateway-leaf issuance at the hive's own bao identity minted
# for it.
#
# ONE PAIRING PER FILE — the hive's gateway-leaf principal ← bao, and nothing
# else. Deleting this leaves a hive that takes an operator-provided path to
# that credential, which is what every hive on a host without the store does.
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, for the reason ./glue-services-issuer-bao-identity.nix gives.
#
# Gated on the minted PKI existing, as that file is.
#
# Everything is `mkDefault`. An operator naming their own paths wins.
{
lib,
config,
...
}:
let
baoDeploy = config.services.hyperhive.deploy.bao;
# Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own
# path as ./glue-services-issuer-bao-identity.nix derives it.
haveMintedPki = baoDeploy.clientCertFile != null;
pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null;
in
{
config = lib.mkIf haveMintedPki {
services.hyperhive.deploy.hive-controller.tls = {
baoHiveIdentityCertFile = lib.mkDefault "${pkiDir}/hive-leaf.pem";
baoHiveIdentityKeyFile = lib.mkDefault "${pkiDir}/hive-leaf-key.pem";
};
};
}

2
nix/host-modules/hive-gateway/default.nix
View file

@ -286,7 +286,7 @@ in
chmod 0755 ${tlsDir}
# Copy the leaf in. `install` writes atomically with the
# target mode; runs as root so the 0600 root:root key written
# by hive-tls-ca is readable. Key ends up root:nginx 0640 so
# by hive-tls-resign is readable. Key ends up root:nginx 0640 so
# nginx-pre-start
# (which runs `nginx -t` as the nginx user, not root) can
# read it — a 0600 root:root key passes the master load but

705
nix/host-modules/hive-tls.nix
View file

@ -49,167 +49,68 @@ let
gatewayWaitsForServicesLeaf =
localServiceDomains != [ ] && localServiceDomains != [ baoCfg.ui.domain ];
# The host-managed hive CA is the trust anchor for self-signed mode.
# It is only stood up when the gateway actually serves a self-signed
# cert: the gateway must run here and be in self-signed mode. The
# self-signed condition is the gateway module's single source of truth
# The gateway's own leaf is only issued when the gateway actually serves a
# self-signed cert: the gateway must run here and be in self-signed mode.
# The self-signed condition is the gateway module's single source of truth
# (`gateway.useSelfSigned`): true when neither an operator cert
# (`tls.certDir`) nor ACME is set.
active = gatewayCfg.enable && gatewayCfg.useSelfSigned;
# How this hive's CA comes into existence when it is missing — and it
# is one of exactly two things, chosen by config rather than by what
# happens to be on disk.
#
# Issuing a sub-CA requires the swarm root's PRIVATE key, so it can
# only happen where that key legitimately lives: the single-host
# deployment that `swarm.ca.autoConfigure` describes. A host cannot
# infer that it is that host — a swarm's services and its hives can
# sit anywhere — so with the flag off this hive self-signs exactly as
# it always has, and an operator who wants it in the hierarchy
# installs the CA themselves. Falling back to self-signed rather than
# failing keeps a plain hive working out of the box; what it loses is
# membership of a swarm's trust, which is the correct thing to lose
# for a hive nobody has federated.
caGenScript =
if swarmCaCfg.autoConfigure then
# The role this hive's gateway leaf is issued through. ./swarm-bao.nix
# writes one per hive in the swarm directory, under the same prefix, and
# narrows it to that hive's domain and `*.` that domain. So the leaf carries
# those two names and no others: a configured service name goes on the
# swarm-services leaf below, and asking this role for one is a refusal.
hiveLeafRole =
if hyperhiveCfg.hiveName == null then
null
else
"${baoDeploy.hiveLeafPkiRolePrefix}-${hyperhiveCfg.hiveName}";
# The hive's own store identity: its subject is
# `<hiveLeafCommonNamePrefix>-<hiveName>`, and its policy may issue from
# `hiveLeafRole` alone. It also carries the root fetch's handshake.
haveHiveIdentity = cfg.baoHiveIdentityCertFile != null && cfg.baoHiveIdentityKeyFile != null;
showPath = p: if p == null then "unset" else p;
hiveIdentityOptions = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile (${showPath cfg.baoHiveIdentityCertFile}) and services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile (${showPath cfg.baoHiveIdentityKeyFile})";
# Fails the unit, naming both options, when the identity is unset or a file
# is unreadable. Every hive must hold one.
requireHiveIdentity =
purpose:
if haveHiveIdentity then
''
# The root should exist — `swarm-ca.service` runs before this and
# is required by it. If it doesn't, something upstream failed and
# signing with a half-provisioned root would be worse than stopping.
if [ ! -s "$root" ] || [ ! -s "$rootk" ]; then
echo "swarm.ca.autoConfigure is set but there is no root CA key at $rootk" >&2
echo "(swarm-ca.service should have generated it) — refusing to issue a hive CA." >&2
missing=""
for f in ${
lib.escapeShellArgs [
cfg.baoHiveIdentityCertFile
cfg.baoHiveIdentityKeyFile
]
}; do
[ -r "$f" ] || missing="$missing $f"
done
if [ -n "$missing" ]; then
echo "this hive has no usable store identity, so it cannot ${purpose}: cannot read$missing." >&2
echo "place the files named by ${hiveIdentityOptions}." >&2
exit 1
fi
echo "issuing fresh hive CA at $ca under the swarm root"
cacsr="$(mktemp "$d/ca.csr.XXXXXX")"
caext="$(mktemp "$d/ca.ext.XXXXXX")"
trap 'rm -f "$cacsr" "$caext"' EXIT
openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout "$cak" -out "$cacsr" \
-subj "/CN=hive-ca ${domain}"
# printf (not a heredoc) so the ext-file lines carry no leading
# whitespace once nix has stripped the indented-string indent.
{
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\n'
printf 'keyUsage=critical,keyCertSign,cRLSign\n'
printf 'subjectKeyIdentifier=hash\n'
printf 'authorityKeyIdentifier=keyid:always\n'
# The constraint is the point of the hierarchy, not a
# flourish: without it a leaked hive CA mints any name in
# the swarm, and it is verifiers that enforce this, not our
# good behaviour. The IP exclusions are not redundant — a
# DNS constraint says nothing about an iPAddress SAN, and a
# name type nobody constrained is a name type this CA is
# unconstrained for.
printf 'nameConstraints=critical,permitted;DNS:%s,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0\n' \
${lib.escapeShellArg domain}
} > "$caext"
openssl x509 -req -in "$cacsr" -CA "$root" -CAkey "$rootk" \
-CAcreateserial -days ${toString cfg.caValidityDays} -sha256 \
-extfile "$caext" -out "$ca"
rm -f "$cacsr" "$caext"
trap - EXIT
''
else
''
echo "generating fresh self-signed hive CA at $ca"
openssl req -x509 -newkey rsa:4096 -nodes -sha256 \
-days ${toString cfg.caValidityDays} \
-keyout "$cak" -out "$ca" \
-subj "/CN=hive-ca ${domain}" \
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
-addext "keyUsage=critical,keyCertSign,cRLSign"
echo "this hive has no store identity, so it cannot ${purpose}." >&2
echo "set ${hiveIdentityOptions} and place both files." >&2
exit 1
'';
# Sign one leaf. Parameterised rather than hardcoded to `gateway.*`
# because there are now two: the hive's own leaf, issued by the hive
# CA, and the swarm-services leaf, issued by the services sub-CA that
# ./swarm-ca.nix maintains. Same ceremony, different issuer and names
# — and one script means the two cannot drift in how they are built.
#
# $1 stateDir $2 basename $3 CN $4 SAN list $5 issuer cert $6 issuer key
signLeafScript = pkgs.writeShellScript "hive-tls-sign-leaf" ''
set -euo pipefail
d="$1"
base="$2"
cn="$3"
sans="$4"
ca="$5"
cak="$6"
leaf="$d/$base.pem"
leafk="$d/$base-key.pem"
csr="$(mktemp "$d/$base.csr.XXXXXX")"
ext="$(mktemp "$d/$base.ext.XXXXXX")"
only="$(mktemp "$d/$base.leaf.XXXXXX")"
trap 'rm -f "$csr" "$ext" "$only"' EXIT
openssl req -newkey rsa:4096 -nodes -sha256 \
-keyout "$leafk" -out "$csr" \
-subj "/CN=$cn"
# printf (not a heredoc) so the ext-file lines carry no leading
# whitespace once nix has stripped the indented-string indent.
{
printf 'subjectAltName=%s\n' "$sans"
printf 'basicConstraints=critical,CA:FALSE\n'
printf 'keyUsage=critical,digitalSignature,keyEncipherment\n'
printf 'extendedKeyUsage=serverAuth\n'
} > "$ext"
openssl x509 -req -in "$csr" -CA "$ca" -CAkey "$cak" \
-CAcreateserial -days ${toString cfg.leafValidityDays} -sha256 \
-extfile "$ext" -out "$only"
# nginx serves this file verbatim, so it must carry the leaf AND its
# issuer: the hive CA is an intermediate under the swarm root, and a
# client that anchors on the root cannot build the middle of the
# chain by itself. Agents anchor on the hive CA directly and
# validated either way — the appended cert is what makes a swarm
# peer, or anything else holding only the root, work.
# Leaf first: both `ssl_certificate` and the `openssl x509 -in`
# expiry checks read the first cert in the file.
cat "$only" "$ca" > "$leaf"
chmod 0600 "$leafk"
chmod 0644 "$leaf"
'';
# The hive's own leaf: signed by the hive CA, covering the hive domain
# and its sub-domains.
#
# `forge.` and `matrix.` were listed here explicitly and both were
# redundant with the wildcard beside them — a wildcard covers exactly
# one label, and those are one label. Naming them read as policy
# ("these two services live under this leaf"), which is the reason they
# outlived it: a configured service name is no longer necessarily under
# this domain, and when it isn't, this is the one list it cannot join.
# The hive CA is `nameConstraints=permitted;DNS:<hive domain>`, and a
# violation invalidates the CERTIFICATE rather than the offending name
# — measured with openssl, not reasoned about — so one sibling name
# added here takes the leaf down for the dashboard and every other
# vhost sharing it. Configured service names get the swarm-services
# leaf below, whose sub-CA is constrained to exactly those names.
signHiveLeaf = ''
${signLeafScript} "$d" gateway ${lib.escapeShellArg domain} \
${lib.escapeShellArg "DNS:${domain},DNS:*.${domain}"} \
"$d/ca.pem" "$d/ca-key.pem"
'';
# Shared by every unit that decides whether to re-issue a leaf. It
# lives here rather than in one of them because the guards have to
# agree: they answer the same question at different times
# (`hive-tls-ca` at service activation, i.e. on the deploy;
# `hive-tls-resign` from a weekly timer for a host that stays up long
# enough to drift; `swarm-services-cert` for the leaf the store
# issues). A rule implemented in one and not the others is worse than
# one implemented in none — it looks fixed and only fires on whichever
# path you did not take, which is exactly how a corrected
# `serviceDomains` still served a stale leaf.
# agree: they answer the same question for different leaves
# (`hive-tls-resign` for the hive leaf, at boot, on the deploy and from
# its timer; `swarm-services-cert` for the services leaf). A rule
# implemented in one and not the others is worse than one implemented
# in none — it looks fixed and only fires on whichever path you did not
# take, which is exactly how a corrected `serviceDomains` still served a
# stale leaf.
#
# Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and
# `covers`.
@ -254,18 +155,13 @@ let
[ "$_missing" = 0 ]
}
# True when the leaf this host SIGNS is present and carries its
# configured names. Expiry is the callers' own business — they use
# different windows.
# True when the hive leaf is present and carries its configured
# names. Expiry is the callers' own business — they use different
# windows.
#
# ⚠️ The services leaf is deliberately NOT part of this any more, and
# the omission is load-bearing rather than a simplification. Its two
# callers sign under the hive CA's key; the services leaf comes out
# of the secret store, which neither of them can reach. A guard that
# went on reporting it stale would make `hive-tls-ca` re-sign the
# hive leaf forever over a file it has no way to fix. The
# `swarm-services-cert` unit below holds the same rule for that leaf,
# using the `covers` helper above so the two cannot drift.
# The services leaf is not part of this: `swarm-services-cert` holds
# the same rule for it, using the `covers` helper above so the two
# cannot drift.
leavesCoverNames() {
covers "$leaf" "$hiveNames"
}
@ -306,9 +202,9 @@ let
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
haveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# The root-only fetch. Same store and server CA as `servicesCertEnvironment`,
# but the hive's reader identity: a host fronting no swarm name holds no
# but the hive's own identity: a host fronting no swarm name holds no
# services-issuer pair.
storeRootCurlTls = lib.escapeShellArgs (
lib.optionals (baoDeploy.serverCaFile != null) [
@ -317,59 +213,53 @@ let
]
++ [
"--cert"
baoDeploy.clientCertFile
cfg.baoHiveIdentityCertFile
"--key"
baoDeploy.clientKeyFile
cfg.baoHiveIdentityKeyFile
]
);
# What `hive-tls-resign` issues the hive leaf with: the services leaf's
# store and server CA, presenting the hive's own identity.
hiveLeafEnvironment = {
inherit (servicesCertEnvironment) BAO_ADDR;
}
// lib.optionalAttrs (cfg.baoHiveIdentityCertFile != null) {
BAO_CLIENT_CERT = cfg.baoHiveIdentityCertFile;
}
// lib.optionalAttrs (cfg.baoHiveIdentityKeyFile != null) {
BAO_CLIENT_KEY = cfg.baoHiveIdentityKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
storeRetry = import ./lib/store-retry.nix { };
in
{
# Host-side TLS trust root for the self-signed gateway mode.
# Host-side TLS for the self-signed gateway mode.
#
# A bare self-signed leaf would be its own trust anchor, so every
# regeneration would be a new anchor and every consumer (agents,
# federation peers) would have to re-trust on each rotation — and a
# runtime-generated, in-container cert can't be wired into an agent's
# build-time trust store at all.
# The gateway serves a short-lived **leaf** issued by the secret store's
# `pki` mount through this hive's own role, and consumers (agents,
# federation peers) anchor on that mount's root, which ships in
# `trust-bundle.pem`. Leaf rotation never re-breaks them, and no key that
# signs a certificate exists on this host. See
# `docs/networking/gateway.md` ("Self-signed TLS") and docs/swarm/ca.md.
#
# So the issuer is a long-lived **hive CA** held on the host. The
# gateway serves a **leaf** signed by that CA (via the `tls.certDir`
# bind-mount path); agents and federation peers trust the *CA* once,
# and leaf rotation never re-breaks them. See `docs/networking/gateway.md`
# ("Self-signed TLS").
#
# That CA is self-signed by default. Under
# `swarm.ca.autoConfigure` — the all-on-one-host case — it is instead
# an intermediate issued under the swarm root (./swarm-ca.nix owns the
# why) and name-constrained to this hive's domain, so it stays the
# anchor agents pin while a peer holding only the root can validate
# everything this hive serves. Either way an existing CA is left
# alone; see the issuance comment below.
# The trust bundle still carries the retired hive CA's public `ca.pem`
# where one exists, so a leaf it signed keeps verifying until it is
# replaced.
options.services.hyperhive.deploy.hive-controller.tls = {
stateDir = lib.mkOption {
type = lib.types.str;
default = "/var/lib/hive-tls";
description = ''
Host directory holding the hive CA + gateway leaf cert for the
self-signed gateway mode. `ca.pem` (the anchor agents and
federation peers trust), `ca-key.pem` (0600, never leaves the
host), `gateway.pem` / `gateway-key.pem` (the leaf the gateway
container bind-mounts and nginx serves). Persistent so the CA
survives reboots — re-deriving it would re-break every consumer.
'';
};
caValidityDays = lib.mkOption {
type = lib.types.int;
default = 7300;
description = ''
Validity window of the hive CA in days (default ~20y). Kept long
and well beyond `leafValidityDays` so the CA outlives many leaf
rotations — the whole point of the CA is to be a stable anchor
that consumers trust once. The CA is regenerated only if missing
or already expired.
Host directory holding the gateway leaf and trust bundle for the
self-signed gateway mode: `gateway.pem` / `gateway-key.pem` (the
leaf the store issues and nginx serves) and `trust-bundle.pem` (the
anchors agents and federation peers trust). `ca.pem` and
`ca-previous.pem`, where present, are the retired hive CA's public
certificates and stay in the bundle; no CA key is kept here.
'';
};
@ -389,9 +279,9 @@ in
A **separate** credential from
{option}`services.hyperhive.deploy.bao.clientCertFile`, which is this
hive's own reader identity and whose policy reads every secret the
hive is entitled to. `swarm-services-cert` never logs in with that one:
on a hive fronting no swarm name it presents it only for the mTLS
handshake of the root fetch, since `pki/ca/pem` takes no token.
hive is entitled to, and from
{option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile`,
which the root fetch on a hive fronting no swarm name presents.
On a hive that mints its own store PKI a glue module supplies the leaf
as a `mkDefault`; everywhere else it is the credential an operator
@ -412,21 +302,39 @@ in
'';
};
leafValidityDays = lib.mkOption {
type = lib.types.int;
default = 30;
baoHiveIdentityCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/hive-leaf.pem";
description = ''
Validity window of the gateway leaf cert in days (default 30).
Short-lived by design — ahead of the CA/Browser-Forum's move
toward ~47-day max lifetimes — which bounds the blast radius of a
leaf-key compromise. The leaf is re-signed by the (stable) CA
when it is missing or near expiry; because it shares the CA
anchor, a rotation does not disturb consumer trust. Agents and
federation peers validate against the CA, not browser CA/B-forum
limits. The weekly `hive-tls-resign` timer re-signs the leaf once
it is within half its validity of expiry and propagates the new
leaf into the running gateway, so a long-uptime host renews
automatically without a reboot.
This hive's own store identity. `hive-tls-resign` presents it to ask
the secret store for the gateway leaf, and `swarm-services-cert` on a
hive fronting no swarm name presents it for the handshake of the root
fetch.
Its subject has to be
`<services.hyperhive.deploy.bao.hiveLeafCommonNamePrefix>-<hiveName>`;
that cert-auth role's policy may issue from this hive's gateway-leaf
role and nothing else.
Every hive with a self-signed gateway needs it: without it both units
fail, naming this option and
{option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile`.
On the host that runs the store a glue module supplies the leaf as a
`mkDefault`; on every other host an operator places it.
A path, never a value.
'';
};
baoHiveIdentityKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/hive-leaf-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile`.
Both or neither — a certificate with no key authenticates nothing.
'';
};
};
@ -444,24 +352,18 @@ in
# rebuild of every hive that is not the CA host, about a fallback
# that no longer happens.
# Generate (and rotate) the hive CA + gateway leaf before anything
# serves it. Idempotent: the CA is created once and reused; the leaf
# is re-signed on expiry under the same CA so the anchor is stable.
# The state dir and the trust bundle. Never waits on the store: the
# store's container is ordered after this unit, which writes the
# bundle it bind-mounts, so the leaf is `hive-tls-resign`'s.
systemd.services.hive-tls-ca = {
description = "Generate hive CA + gateway leaf TLS cert (self-signed mode)";
description = "Assemble the hive TLS trust bundle (self-signed mode)";
wantedBy = [ "multi-user.target" ];
# The consumer is `hive-gateway-self-signed-cert`, which copies the
# leaf into the gateway's state dir at the mode nginx can read, and
# which nginx in turn `Requires=`. So this must run first or that
# copy fails under `set -eu` and takes nginx down with it — order
# against the unit that reads the file.
# leaf out of the state dir this unit creates, and which nginx in
# turn `Requires=`. Order against the unit that reads the dir.
before = [ "hive-gateway-self-signed-cert.service" ];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The issuance below needs the swarm root key on disk. When this
# host generates it (single-host swarm) that unit must have run
# first; when the operator provides the material there is no unit
# to wait for, so the dependency is conditional rather than a unit
# that exists and does nothing.
# The bundle carries the swarm root where this host generates it.
#
# `swarm-services-cert` is NOT ordered against this unit in either
# direction, and the absence is load-bearing — see the cycle spelled
@ -477,8 +379,6 @@ in
# and `swarm-services-cert` restarts this unit once the root it wrote
# is new — so the bundle converges instead of being a boot behind.
after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
requires = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
path = [ pkgs.openssl ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
@ -492,164 +392,103 @@ in
install -d -m 0755 "$d"
ca="$d/ca.pem"
cak="$d/ca-key.pem"
leaf="$d/gateway.pem"
leafk="$d/gateway-key.pem"
root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"}
rootk=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root-key.pem"}
prev="$d/ca-previous.pem"
marker="$d/.swarm-ca-adopted"
root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"}
svcroot="$d/swarm-services-root.pem"
# --- Adoption: a hive whose CA predates the swarm root.
#
# Runs ONLY where this host also owns the root (`autoConfigure`),
# because adoption invalidates an anchor that consumers already
# trust and they refresh on their own schedule. On one box that
# schedule is knowable; across hosts it is not, so there the
# operator does it and this unit only says so, loudly.
#
# Guarded by a marker rather than by the state of the world: the
# marker's ABSENCE is the trigger, so this fires once per hive
# instead of re-deciding every activation. That is the difference
# between a migration and a kicking machine.
if [ -s "$root" ] && [ -s "$ca" ] && [ ! -e "$marker" ] \
&& ! openssl verify -CAfile "$root" "$ca" >/dev/null 2>&1; then
${
if swarmCaCfg.autoConfigure then
''
echo "adopting the swarm CA: $ca does not chain to $root" >&2
# Keep the old CA as an anchor across the overlap. Consumers
# read the bundle, so adoption is additive before it is
# subtractive — agents pick up new trust only when their
# container restarts, which is a window even on one host.
cp "$ca" "$prev"
chmod 0644 "$prev"
rm -f "$ca" "$cak"
touch "$marker"''
else
''
echo "hive-tls: this hive's CA does not chain to the swarm root." >&2
echo " hive CA: $ca" >&2
echo " swarm root: $root" >&2
echo "Adopting the hierarchy is not automatic here: it invalidates an" >&2
echo "anchor that peers and agents on OTHER hosts still trust, and they" >&2
echo "refresh on their own schedule — only you know when that is safe." >&2
echo "To adopt: rm $ca $cak && systemctl restart hive-tls-ca.service" >&2
echo "To keep the current CA deliberately: touch $marker" >&2
exit 1''
}
# The gateway leaf comes from the store, so no key that signs a
# certificate is kept here.
if [ -e "$d/ca-key.pem" ]; then
echo "removing the retired hive CA key $d/ca-key.pem"
rm -f "$d/ca-key.pem"
fi
# --- CA: generated once and reused across leaf rotations, in
# whichever of the two shapes `caGenScript` selected.
# Regenerated only if missing or already expired (checkend 0).
# A new CA means every consumer must re-trust, so the leaf is
# dropped to force a re-sign under the fresh CA.
#
# An existing CA is never re-rooted here. A hive predating the
# swarm root carries a self-signed `ca.pem`, and swapping it for
# a swarm-issued one would break every consumer already trusting
# it — including agents, whose trust is refreshed only when their
# container restarts. Adopting the hierarchy on such a hive is
# therefore an operator step (drop the CA, let this unit
# re-issue), which is also what keeps this change non-disruptive
# to hives that never adopt it.
if [ ! -s "$ca" ] || [ ! -s "$cak" ] \
|| ! openssl x509 -in "$ca" -noout -checkend 0 >/dev/null 2>&1; then
${caGenScript}
chmod 0600 "$cak"
chmod 0644 "$ca"
rm -f "$leaf" "$leafk"
fi
# --- Leaf: (re)sign when missing, within 30 days of expiry, or no
# longer covering the configured names, always under the current
# (stable) CA.
${leafCoverage}
# ⚠️ This is the guard that runs ON THE DEPLOY — `hive-tls-resign`
# only fires from a weekly timer, so a rule enforced only there is
# up to a week late and does nothing for the rebuild that changed
# the names in the first place.
#
resign=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || resign=1
openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1 || resign=1
leavesCoverNames || resign=1
if [ "$resign" = 1 ]; then
echo "signing gateway leaf at $leaf (missing, near expiry, or missing a configured name)"
${signHiveLeaf}
fi
# --- Trust bundle: what a consumer must TRUST, as opposed to
# `ca.pem`, which is what this host SIGNS with. Why they stopped
# being the same file, and why `ca-previous.pem` stays in the set
# after an adoption, are in docs/swarm/ca.md. Three constraints
# the code cannot state:
# --- Trust bundle: what a consumer must TRUST. Why it is a set of
# anchors rather than one file is in docs/swarm/ca.md. Three
# constraints the code cannot state:
#
# Written IN PLACE, never renamed into position — containers
# bind-mount this file and a bind mount follows the inode, so a
# rename leaves every consumer holding the old one.
#
# Dropping `ca-previous.pem` is deliberately NOT done here:
# "long enough" is a deployment fact, not a unit's call.
# Dropping `ca.pem` and `ca-previous.pem` is deliberately NOT done
# here: "long enough" is a deployment fact, not a unit's call.
#
# Built as an array with `if` guards, not
# `[ -s f ] && anchors+=(f)`: under `set -e` an && list whose test
# fails IS a failing command and kills the unit — on exactly the
# hive where the optional file is legitimately absent. The
# reflexive `|| true` is worse; it also swallows a real failure to
# read the hive CA.
# hive where the optional file is legitimately absent.
#
# `swarm-services-root.pem` joins the set for the reason the swarm
# root is in it: it is an ANCHOR a consumer terminates at, not an
# intermediate. It is also what keeps this change from being a
# flag day — the swarm root stays in the bundle beside it, so a
# leaf still chaining to the old services sub-CA and one issued by
# the store both verify against the same file, and a hive can be
# rebuilt before or after its peers.
# `swarm-services-root.pem` is the store's `pki` root, which both
# the gateway leaf and the swarm-services leaf chain to. `ca.pem`
# and `ca-previous.pem` are the retired hive CA's public
# certificates, kept so a leaf they signed verifies until replaced.
bundle="$d/trust-bundle.pem"
anchors=("$ca")
anchors=()
if [ -s "$ca" ]; then anchors+=("$ca"); fi
if [ -s "$prev" ]; then anchors+=("$prev"); fi
if [ -s "$root" ]; then anchors+=("$root"); fi
if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
cat "''${anchors[@]}" > "$bundle"
# `cat` with no arguments reads stdin, so no anchor at all is an
# empty file until `swarm-services-cert` writes the root.
if [ "''${#anchors[@]}" = 0 ]; then
echo "no trust anchor on disk yet — $bundle is empty until the store's root is fetched" >&2
: > "$bundle"
else
cat "''${anchors[@]}" > "$bundle"
fi
chmod 0644 "$bundle"
'';
};
# Weekly re-sign of the gateway leaf so short-lived leaves renew
# without depending on a reboot.
# The gateway leaf: issued by the secret store's `pki` mount through
# this hive's own role, at boot, on every deploy and from the daily
# timer below, whenever it is missing, past half its lifetime, missing
# a configured name, or not chained to the store's root.
#
# `hive-tls-ca` only re-signs at service activation (boot/rebuild); a
# long-uptime host would otherwise let a 30-day leaf lapse silently.
# This service re-signs the leaf directly (not by bouncing hive-tls-ca)
# and propagates the new leaf to nginx when the file actually changed.
# ⚠️ Nothing orders the gateway against this unit, which is what lets a
# hive holding a valid leaf start its gateway with the store sealed or
# unreachable. Only a hive with no leaf yet has a gateway that waits
# for the store: its cert import fails until this unit lands a leaf
# and starts nginx.
#
# Failure leaves the current leaf alone: every file the script writes
# is a `.new` until the store has answered with both fields.
#
# Propagation mechanism: nginx serves a *copy* of the leaf, written by
# `hive-gateway-self-signed-cert` at the mode nginx can read. Re-signing
# `hive-gateway-self-signed-cert` at the mode nginx can read. Issuing
# the source therefore changes nothing on its own — the copy has to be
# remade and nginx reloaded, which is what the two calls below do.
#
# ⚠️ Neither call is `|| true`: a swallowed propagation failure means
# the leaf rotates on disk while nginx keeps serving the old copy
# until it expires, with this unit reporting success the whole time.
# A failure here must fail the timer.
# remade and nginx reloaded, which is what the calls at the end do.
# Neither of the running-nginx calls is `|| true`: a swallowed
# propagation failure means the leaf rotates on disk while nginx keeps
# serving the old copy until it expires, with this unit reporting
# success the whole time.
systemd.services.hive-tls-resign = {
description = "Re-sign the gateway TLS leaf and reload nginx";
# hive-tls-ca must have run first so the CA key exists before we try
# to re-sign under it. On first boot `Persistent=true` on the weekly
# timer fires immediately; without this ordering the resign could race
# the CA initialisation and fail with "no such file" on the CA key.
after = [ "hive-tls-ca.service" ];
description = "Issue or renew the gateway TLS leaf from the secret store's PKI";
wantedBy = [ "multi-user.target" ];
# The store's container and its granting unit where they run here;
# ignored elsewhere, as for `swarm-services-cert`, which writes the
# store's root this unit checks the leaf against.
after = [
"hive-tls-ca.service"
"container@${baoCfg.machine}.service"
"swarm-bao-pki.service"
"swarm-bao-hive-leaf-policy.service"
"swarm-services-cert.service"
];
wants = [ "container@${baoCfg.machine}.service" ];
path = [
baoDeploy.package
pkgs.jq
pkgs.openssl
pkgs.coreutils
pkgs.systemd
];
serviceConfig = {
environment = hiveLeafEnvironment;
# ./lib/store-retry.nix: the login below fails for as long as the
# store is sealed or down.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
UMask = "0077";
SyslogIdentifier = "hive-tls-resign";
@ -657,46 +496,93 @@ in
script = ''
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
install -d -m 0755 "$d"
leaf="$d/gateway.pem"
leafk="$d/gateway-key.pem"
# ⚠️ EVERY leaf this host SIGNS must be covered here. A leaf that
# first-boot issuance creates and this unit does not know about
# looks perfect for its entire validity and then expires with no
# warning — the failure is invisible until it is total. The name
# checks come from the shared snippet, so this unit and
# `hive-tls-ca` cannot disagree about what a good leaf is.
#
# The services leaf is not one of them any longer: it is issued
# by the secret store, whose key this unit does not hold and
# cannot re-sign under. `swarm-services-cert-renew` renews it,
# on its own daily timer.
${leafCoverage}
# Re-sign only when a leaf is within half its validity of expiry.
# The weekly cadence catches this window well before one lapses.
halflife=$(( ${toString cfg.leafValidityDays} * 86400 / 2 ))
fresh() { # a leaf is fresh if it exists and is not near expiry
[ -s "$1" ] && openssl x509 -in "$1" -noout -checkend "$halflife" >/dev/null 2>&1
fail() {
echo "$1" >&2
if [ -s "$leaf" ]; then
echo "the current gateway leaf at $leaf stays in place" >&2
fi
exit 1
}
if fresh "$leaf" && leavesCoverNames; then
echo "leaves valid, and covering the configured names — no resign needed"
# Checked before the leaf is: a hive without its identity is red
# even while its current leaf is still valid.
${
if hiveLeafRole == null then
''fail "services.hyperhive.hiveName is unset, so this hive has no gateway-leaf role to issue from."''
else
requireHiveIdentity "ask the store for its gateway leaf"
}
${leafCoverage}
# Half the lifetime the role grants, as for the services leaf, so
# the daily timer has half the lifetime to retry through a sealed
# store.
renewat=$(( ${toString baoDeploy.servicesPkiLeafTtlHours} * 3600 / 2 ))
reissue=0
{ [ -s "$leaf" ] && [ -s "$leafk" ]; } || reissue=1
openssl x509 -in "$leaf" -noout -checkend "$renewat" >/dev/null 2>&1 || reissue=1
leavesCoverNames || reissue=1
# A leaf the retired hive CA signed, or one under a store root since
# replaced, fails this and is re-issued.
{ [ -s "$svcroot" ] && openssl verify -CAfile "$svcroot" "$leaf" >/dev/null 2>&1; } || reissue=1
if [ "$reissue" = 0 ]; then
echo "gateway leaf valid, chained to the store's root and covering the configured names — leaving it alone"
exit 0
fi
echo "a leaf is missing, near expiry, or missing a configured name — re-signing"
before="$(sha256sum "$leaf" 2>/dev/null || true)"
err="$(mktemp)"
resp="$(mktemp "$d/gateway.json.XXXXXX")"
trap 'rm -f "$err" "$resp" "$leaf.new" "$leafk.new"' EXIT
${signHiveLeaf}
# Cert auth is a login, not a transport setting; `-token-only`
# skips the token helper, for the reason `swarm-services-cert` gives.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
fail "could not log in to the swarm secret store with this hive's identity: $(cat "$err")"
fi
export BAO_TOKEN
after="$(sha256sum "$leaf" 2>/dev/null || true)"
if [ "$before" != "$after" ]; then
echo "requesting the gateway leaf from the store"
if ! bao write -format=json \
${lib.escapeShellArg "${baoServicesPkiMount}/issue/${toString hiveLeafRole}"} \
common_name=${lib.escapeShellArg domain} \
alt_names=${lib.escapeShellArg "*.${domain}"} \
> "$resp" 2>"$err"; then
fail "the store refused to issue the gateway leaf: $(cat "$err")"
fi
# One issuance, split, for the reasons `swarm-services-cert` gives:
# the key never enters an argv, and `// empty` keeps a missing field
# from becoming the four characters `null`. The leaf alone, without
# its issuer: the issuer is the self-signed root in the bundle.
umask 077
jq -r '.data.private_key // empty' < "$resp" > "$leafk.new"
jq -r '.data.certificate // empty' < "$resp" > "$leaf.new"
for f in "$leafk.new" "$leaf.new"; do
[ -s "$f" ] || fail "the store's response was missing a field: $f is empty"
done
mv -f "$leaf.new" "$leaf"
mv -f "$leafk.new" "$leafk"
chmod 0644 "$leaf"
chmod 0600 "$leafk"
# A gateway already running gets the copy remade and a reload. One
# that is not running — a hive that had no leaf — gets started,
# `--no-block` so a start job still queued absorbs the request.
if systemctl is-active --quiet nginx.service; then
echo "gateway leaf rotated — re-importing and reloading nginx"
systemctl restart hive-gateway-self-signed-cert.service
systemctl reload nginx.service
else
echo "gateway leaf unchanged (already up to date)"
elif [ "$(systemctl show -P LoadState nginx.service)" = loaded ]; then
echo "gateway leaf landed with nginx not running — starting it"
systemctl reset-failed nginx.service
systemctl start --no-block nginx.service
fi
'';
};
@ -796,33 +682,18 @@ in
# bundle: every other hive's swarm-service names chain to it.
# `ca/pem` takes no token; the TLS client certificate is the
# store's listener's demand, not bao's, so this presents the
# hive's reader identity and never logs in with it.
# A hive without a readable reader identity fails here, naming the
# hive's own identity and never logs in with it.
# A hive without a readable identity fails here, naming the
# options: every hive must hold one.
if [ "$want_svc" = 0 ]; then
${
if haveReaderIdentity then
''
missing=""
for f in ${
lib.escapeShellArgs [
baoDeploy.clientCertFile
baoDeploy.clientKeyFile
]
}; do
[ -r "$f" ] || missing="$missing $f"
done
if [ -n "$missing" ]; then
echo "this hive has no usable store identity: cannot read$missing." >&2
echo "place the files named by services.hyperhive.deploy.bao.clientCertFile (${baoDeploy.clientCertFile}) and services.hyperhive.deploy.bao.clientKeyFile (${baoDeploy.clientKeyFile})." >&2
exit 1
fi
if haveHiveIdentity then
requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root"
+ ''
echo "no swarm service domains configured — fetching only the store's pki root"
err="$(mktemp)"
raw="$(mktemp "$d/swarm-services-root.XXXXXX")"
trap 'rm -f "$err" "$raw"' EXIT
# Interim: the reader cert is the handshake identity until each hive
# has its own narrow store identity for issuing its leaf.
if ! curl --fail --silent --show-error \
${storeRootCurlTls} --output "$raw" \
"$BAO_ADDR/v1/${baoServicesPkiMount}/ca/pem" 2>"$err"; then
@ -851,14 +722,7 @@ in
fi
exit 0''
else
''
echo "this hive has no store identity, so it cannot fetch the store's ${baoServicesPkiMount} root." >&2
echo "set services.hyperhive.deploy.bao.clientCertFile (${
if baoDeploy.clientCertFile == null then "unset" else baoDeploy.clientCertFile
}) and services.hyperhive.deploy.bao.clientKeyFile (${
if baoDeploy.clientKeyFile == null then "unset" else baoDeploy.clientKeyFile
}) and place both files." >&2
exit 1''
requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root"
}
fi
@ -1075,19 +939,18 @@ in
};
};
# Daily, for the reason `swarm-services-cert-renew` below is: the leaf
# needs the store, which can be sealed or unreachable.
systemd.timers.hive-tls-resign = {
description = "Weekly gateway-leaf re-sign and propagation";
description = "Daily renewal check for the gateway TLS leaf";
wantedBy = [ "timers.target" ];
timerConfig = {
# Run weekly; Persistent=true fires a missed run on next boot if
# the timer was not active (e.g. the host was off on the scheduled
# day), preventing a dormant timer from letting the leaf lapse.
OnCalendar = "weekly";
OnCalendar = "daily";
Persistent = true;
};
};
# Daily rather than `hive-tls-resign`'s weekly, for the reason
# Daily rather than weekly, for the reason
# ./swarm-nats.nix's `swarm-bao-nats-tls` timer is: this leaf needs the
# store, which can be sealed or unreachable, so there is a retry per day
# from half-life to expiry rather than two.
@ -1100,7 +963,7 @@ in
};
};
# Signal the hive-c0re lifecycle that a hive CA exists: it bind-mounts
# Signal the hive-c0re lifecycle that a trust bundle exists: it bind-mounts
# this file (read-only, public certs ONLY — never a key) into each
# agent container so agents + their tools can trust the gateway's
# self-signed leaf, and the meta flake wires the per-agent trust
@ -1130,7 +993,7 @@ in
# variable name. The anchor was never missing; nothing pointed the queue
# client at it.
#
# Where there is no hive CA this module contributes nothing at all, and
# Where the gateway is not self-signed this module contributes nothing, and
# the clients then fall back to the platform roots — which is correct for
# a swarm fronted by a public certificate.
systemd.services.hive-c0re.environment.HIVE_C0RE_OIDC_CA_FILE =

123
nix/host-modules/swarm-bao.nix
View file

@ -621,10 +621,9 @@ let
# also what an untuned mount clamps the ROOT to, so the two collided at
# exactly the length that makes issuance impossible; and the guard below
# compares the issuer's remaining life against this number, which it can
# only do if this number exists. 720h matches
# `deploy.hive-controller.tls.leafValidityDays`, the 30 days ./hive-tls.nix
# documents for every other leaf it holds. An option, read here and by
# ./hive-tls.nix's renewal threshold, so the two cannot disagree.
# only do if this number exists. Every per-hive gateway-leaf role carries it
# too. An option, read here and by ./hive-tls.nix's renewal thresholds, so
# the two cannot disagree.
servicesPkiLeafTtlHours = baoDeploy.servicesPkiLeafTtlHours;
servicesPkiLeafTtl = "${toString servicesPkiLeafTtlHours}h";
servicesPkiLeafTtlSeconds = servicesPkiLeafTtlHours * 3600;
@ -676,6 +675,15 @@ let
}
'';
# One gateway-leaf issuer per hive in the swarm directory, for the reason
# `perHiveReaders` below is per hive. One name for the pki role, the policy
# and the cert-auth role, as for the readers.
hiveLeafIssuers = lib.mapAttrsToList (hiveName: hive: {
name = "${baoDeploy.hiveLeafPkiRolePrefix}-${hiveName}";
cn = "${baoDeploy.hiveLeafCommonNamePrefix}-${hiveName}";
inherit (hive) domain;
}) hyperhiveCfg.swarm.hives;
# The PKI mount agent client certificates are issued from. Its root is
# generated inside the store, so the agent CA's key never exists outside it.
# A mount of its own because the services mount holds exactly one issuer; a
@ -1660,6 +1668,35 @@ in
'';
};
hiveLeafCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-hive-leaf";
example = "swarm-bao-hive-leaf.svc";
description = ''
Prefix of the subject the store's per-hive gateway-leaf roles accept.
The role for hive `h` accepts `<prefix>-h` and may only issue from
`<servicesPkiMountPath>/issue/<hiveLeafPkiRolePrefix>-h`, which issues
for `h`'s domain and `*.` that domain and nothing else.
Per hive for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is:
one subject for every hive could ask for any hive's certificate.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
'';
};
hiveLeafPkiRolePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-hive-leaf";
description = ''
Prefix of the per-hive roles on
{option}`services.hyperhive.deploy.bao.servicesPkiMountPath` that each
hive's gateway leaf is issued through: `<prefix>-<hive>`. The same name
is that hive's policy and cert-auth role. Must start with `swarm-`.
'';
};
grafanaOidcCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-grafana-oidc";
@ -2067,14 +2104,18 @@ in
lib.hasPrefix "swarm-" servicesPkiRoleName
&& lib.hasPrefix "swarm-" natsPkiRoleName
&& lib.hasPrefix "swarm-" agentPkiRoleName
&& lib.hasPrefix "swarm-" baoDeploy.hiveLeafPkiRolePrefix
);
message = ''
services.hyperhive.deploy.bao.servicesPkiRoleName
(${servicesPkiRoleName}),
services.hyperhive.deploy.bao.natsPkiRoleName (${natsPkiRoleName})
and services.hyperhive.deploy.bao.agentPkiRoleName
(${agentPkiRoleName}) must all start with `swarm-`: the bao granter
that writes them may write pki roles under that prefix only.
services.hyperhive.deploy.bao.natsPkiRoleName (${natsPkiRoleName}),
services.hyperhive.deploy.bao.agentPkiRoleName
(${agentPkiRoleName}) and
services.hyperhive.deploy.bao.hiveLeafPkiRolePrefix
(${baoDeploy.hiveLeafPkiRolePrefix}) must all start with `swarm-`:
the bao granter that writes them may write pki roles under that
prefix only.
'';
}
];
@ -3014,6 +3055,71 @@ in
'';
};
# A SIXTH sibling, the nats unit's shape once per hive: each hive's
# gateway-leaf role, its policy and its login role. The role allows the
# hive's domain and `*.` that domain, and no IP SANs or localhost.
systemd.services.swarm-bao-hive-leaf-policy = lib.mkIf haveGranter {
description = "write the per-hive gateway-leaf pki roles, bao policies and cert-auth roles";
after = [
"container@${cfg.machine}.service"
"swarm-bao-controller-policy.service"
]
++ granterAfter;
requires = [ "swarm-bao-pki.service" ];
wantedBy = [ "multi-user.target" ];
path = [
baoCli
pkgs.coreutils
];
environment = granterEnv;
# Same unseal wait as its siblings above.
startLimitBurst = 2880;
startLimitIntervalSec = 90000;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Restart = "on-failure";
RestartSec = 30;
};
script = ''
set -euo pipefail
${granterLogin}
''
+ lib.concatMapStrings (obj: ''
bao write ${lib.escapeShellArg "${servicesPkiMountPath}/roles/${obj.name}"} \
allowed_domains=${lib.escapeShellArg "${obj.domain},*.${obj.domain}"} \
allow_bare_domains=true \
allow_subdomains=false \
allow_glob_domains=true \
allow_wildcard_certificates=true \
allow_localhost=false \
allow_any_name=false \
allow_ip_sans=false \
enforce_hostnames=true \
server_flag=true \
client_flag=false \
key_type=rsa \
key_bits=4096 \
ttl=${servicesPkiLeafTtl} \
max_ttl=${servicesPkiLeafTtl}
printf '%s' ${lib.escapeShellArg ''
path "${servicesPkiMountPath}/issue/${obj.name}" {
capabilities = ["update"]
}
''} |
bao policy write ${lib.escapeShellArg obj.name} -
bao write auth/cert/certs/${lib.escapeShellArg obj.name} \
certificate=@${tlsDir}/client-ca.pem \
allowed_common_names=${lib.escapeShellArg obj.cn} \
token_policies=${lib.escapeShellArg obj.name} \
display_name=${lib.escapeShellArg obj.cn}
'') hiveLeafIssuers;
};
# The agent PKI mount: its root, its one role, the host's copy of the
# root's certificate, and the listener's trust in it.
#
@ -3041,6 +3147,7 @@ in
"swarm-bao-forwarder-oidc-policy.service"
"swarm-bao-services-issuer-policy.service"
"swarm-bao-nats-tls-policy.service"
"swarm-bao-hive-leaf-policy.service"
]
++ granterAfter;
requires = [ "swarm-bao-pki.service" ];

3
nix/host-modules/swarm-ca.nix
View file

@ -95,8 +95,7 @@ in
default = 10950;
description = ''
Validity window of the swarm root CA in days (default ~30y).
Deliberately longer than `services.hyperhive.deploy.hive-controller.tls.caValidityDays`:
the root must outlive the hive CAs it issues, or those chains
Deliberately long: the root must outlive the hive CAs it issued, or those chains
expire out from under hives that are still perfectly happy with
their own intermediate. Rotating a root is the one operation in
this system with no partial-failure mode — it invalidates every

7
nix/host-modules/swarm.nix
View file

@ -55,16 +55,17 @@ let
deployCfg.bao.granterCommonName
deployCfg.bao.natsAuthCommonName
]
# The two per-hive readers' subjects, spelled out per hive rather than as the
# prefix. The prefix alone would reserve the wrong string: the role for hive
# The per-hive subjects, spelled out per hive rather than as the prefix.
# The prefix alone would reserve the wrong string: the role for hive
# `h` accepts `<prefix>-h`, so the name a hive must not BE is that composed
# spelling, and a hive named it would present a leaf that role accepts.
#
# Both spellings are reserved for every declared hive, including the one the
# Every spelling is reserved for every declared hive, including the one the
# CN belongs to — a hive cannot be named after its own reader either.
++ lib.concatMap (hive: [
"${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}"
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
"${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}"
]) hiveNames;
# Public hostnames of the swarm's own services, in declaration order.

104
nix/module-eval/bao-grants.nix
View file

@ -63,6 +63,21 @@ let
deploy.bao.agentPkiRoleName = "agent";
};
# A gateway-leaf role prefix the granter's `roles/swarm-*` does not reach.
baoGranterOddHiveLeafRole = hive {
deploy.bao.enable = true;
deploy.bao.hiveLeafPkiRolePrefix = "hive-leaf";
};
# The store with a second hive in its directory, at a domain that is not
# `<name>.<swarm.domain>`, so each hive's gateway-leaf grant can be told
# apart from the other's.
baoGrantTwoHives = hive {
deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
swarm.hives.h2.domain = "h2.example.org";
};
# The store and the token, with no CA to trust. `mkForce` because the PKI
# glue supplies one by default here — this is the deployment that brings its
# own certificates and has not named the authority yet, in which nothing can
@ -151,7 +166,7 @@ let
_: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile
) baoGrantWithConsumers.systemd.services;
# The thirteen units that write a `swarm-*` grant, by name, for the discovery
# The fourteen units that write a `swarm-*` grant, by name, for the discovery
# control below.
grantingUnitNames = [
"swarm-bao-controller-policy"
@ -167,6 +182,7 @@ let
"swarm-bao-agent-pki"
"swarm-bao-nats-auth-policy"
"swarm-bao-operator-viewer-policy"
"swarm-bao-hive-leaf-policy"
];
# Comment lines dropped first: both the HCL and the scripts explain
@ -784,24 +800,24 @@ let
}
{
# A store host without the granter's pair writes its grants some other
# way, so none of the thirteen units may exist. Without this arm
# way, so none of the fourteen units may exist. Without this arm
# `lib.mkIf haveGranter` could be dropped from any of them and every other
# case here would still pass.
name = "without the granter's pair none of the thirteen granting units render";
name = "without the granter's pair none of the fourteen granting units render";
ok =
let
s = baoGranterOptOut.systemd.services;
in
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
# The control: the same store with the pair renders all thirteen.
# The control: the same store with the pair renders all fourteen.
&& lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames;
}
{
# 🩸 What replaced the silent skip. With no bootstrap token the thirteen still
# 🩸 What replaced the silent skip. With no bootstrap token the fourteen still
# render, and a refused granter fails them with the step that fixes it.
# A store host that never named a token is told to name one, since the
# unit that sets the granter up renders only where it has.
name = "a store host without a bootstrap token renders the thirteen, each failing loudly with the one-time step";
name = "a store host without a bootstrap token renders the fourteen, each failing loudly with the one-time step";
ok =
let
s = baoGranterNoToken.systemd.services;
@ -1091,6 +1107,78 @@ let
in
lib.hasInfix "/granter.pem ]" s && lib.hasInfix "bao-granter \"\" clientAuth" s;
}
{
# One role, policy and login role per hive in the directory, each
# naming only its own hive: the policy text in the unit is exactly the
# two `update` grants, and each hive's block names neither the other's
# role nor its subject.
name = "each hive's gateway-leaf identity may issue from its own role alone, for its own domain";
ok =
let
s = baoGrantTwoHives.systemd.services.swarm-bao-hive-leaf-policy.script;
blockOf =
h: next:
lib.head (
lib.splitString "pki/roles/swarm-hive-leaf-${next}" (
lib.last (lib.splitString "pki/roles/swarm-hive-leaf-${h}" s)
)
);
h1 = blockOf "h1" "h2";
h2 = blockOf "h2" "h1";
issues =
h: dom: blk:
lib.hasInfix "allowed_domains='${dom},*.${dom}'" blk
&& lib.hasInfix "allow_ip_sans=false" blk
&& lib.hasInfix "allow_localhost=false" blk
&& lib.hasInfix "allow_any_name=false" blk
&& lib.hasInfix "max_ttl=720h" blk
&& lib.hasInfix "bao policy write swarm-hive-leaf-${h} -" blk
&& lib.hasInfix "auth/cert/certs/swarm-hive-leaf-${h}" blk
&& lib.hasInfix "allowed_common_names=swarm-bao-hive-leaf-${h}" blk
&& lib.hasInfix "token_policies=swarm-hive-leaf-${h}" blk;
in
grantsIn s == [
{
path = "pki/issue/swarm-hive-leaf-h1";
caps = [ "update" ];
}
{
path = "pki/issue/swarm-hive-leaf-h2";
caps = [ "update" ];
}
]
&& issues "h1" "h1.t.local" h1
&& issues "h2" "h2.example.org" h2
&& !(lib.hasInfix "h2" h1)
&& !(lib.hasInfix "h1" h2);
}
{
# The minting side, and the hive's own path to it: the store host signs
# its hive's leaf under the composed subject and points the gateway-leaf
# unit at it.
name = "the PKI unit signs this hive's gateway-leaf identity, and the hive presents it";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
t = baoGrantHere.services.hyperhive.deploy.hive-controller.tls;
in
lib.hasInfix "/hive-leaf.pem ]" s
&& lib.hasInfix "swarm-bao-hive-leaf-h1 \"\" clientAuth" s
&& t.baoHiveIdentityCertFile == "/var/lib/swarm-bao-pki/hive-leaf.pem"
&& t.baoHiveIdentityKeyFile == "/var/lib/swarm-bao-pki/hive-leaf-key.pem"
# Its own leaf, not the reader's or the services issuer's.
&& t.baoHiveIdentityCertFile != baoGrantHere.services.hyperhive.deploy.bao.clientCertFile
&& t.baoHiveIdentityCertFile != t.baoClientCertFile;
}
{
name = "a gateway-leaf role prefix outside swarm-* is refused, naming the option";
ok =
let
names = a: lib.hasInfix "services.hyperhive.deploy.bao.hiveLeafPkiRolePrefix" a.message;
in
lib.any (a: !a.assertion && names a) baoGranterOddHiveLeafRole.assertions
&& !(lib.any (a: !a.assertion && names a) baoGrantHere.assertions);
}
{
# The granter writes pki roles through `roles/swarm-*` only, so a role
# named otherwise is refused at eval rather than 403'd at deploy.
@ -1304,8 +1392,8 @@ let
}
{
# What makes the case above mean something: discovery by the granter's
# certificate reaches all thirteen units, and each yields calls.
name = "the granter-policy check sees all thirteen granting units, and parses calls from each";
# certificate reaches all fourteen units, and each yields calls.
name = "the granter-policy check sees all fourteen granting units, and parses calls from each";
ok =
lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits)

145
nix/module-eval/hive-tls.nix
View file

@ -3,7 +3,8 @@
#
# The swarm-services leaf's renewal: a timer that really re-runs the
# issuance, at a threshold read off the store role's lifetime. And the
# store's root on a hive that asks it for no leaf.
# store's root on a hive that asks it for no leaf. And the gateway leaf,
# issued by the store through the hive's own role, with no hive CA key.
{
pkgs,
lib,
@ -38,15 +39,17 @@ let
};
# Fronting none of the swarm's names, reading a store it does not run,
# with the reader identity placed by hand.
# with the reader identity and the hive's own identity placed by hand.
noService = hive {
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem";
deploy.hive-controller.tls.baoHiveIdentityCertFile = "/etc/pki/bao-hive.pem";
deploy.hive-controller.tls.baoHiveIdentityKeyFile = "/etc/pki/bao-hive-key.pem";
};
noSvcFetch = noService.systemd.services.swarm-services-cert.script;
# The same hive with no reader identity at all.
# The same hive with no store identity at all.
noIdentity = hive { };
noIdFetch = noIdentity.systemd.services.swarm-services-cert.script;
@ -60,6 +63,8 @@ let
units = allLocal.systemd.services;
boot = units.swarm-services-cert;
caUnit = units.hive-tls-ca;
leafUnit = units.hive-tls-resign;
timer = allLocal.systemd.timers.swarm-services-cert-renew;
# What the timer starts, read off the timer rather than assumed from its
@ -151,13 +156,16 @@ let
&& lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' noService.systemd.services.hive-tls-ca.script;
}
{
name = "the root fetch reaches the store with the hive's own reader identity and server CA";
# The hive's own identity, not its reader identity: the reader's policy
# reads every secret the hive is entitled to.
name = "the root fetch reaches the store with the hive's own identity and server CA";
ok =
lib.hasInfix "--cacert ${lib.escapeShellArg "/etc/pki/bao-ca.pem"}" noSvcFetch
&& lib.hasInfix (
"--cert ${lib.escapeShellArg "/etc/pki/bao-client.pem"}"
+ " --key ${lib.escapeShellArg "/etc/pki/bao-client-key.pem"}"
) noSvcFetch;
"--cert ${lib.escapeShellArg "/etc/pki/bao-hive.pem"}"
+ " --key ${lib.escapeShellArg "/etc/pki/bao-hive-key.pem"}"
) noSvcFetch
&& !(lib.hasInfix "/etc/pki/bao-client.pem" (noSvcBranch noSvcFetch));
}
{
# No identity to present: the unit fails, naming both options, rather
@ -166,11 +174,11 @@ let
name = "a hive fronting no swarm name with no store identity fails the unit, naming both options";
ok =
noIdentity.services.hyperhive.swarm.localServiceDomains == [ ]
&& noIdentity.services.hyperhive.deploy.bao.clientCertFile == null
&& noIdentity.services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile == null
&& !(lib.hasInfix "/v1/pki/ca/pem" noIdFetch)
&& lib.hasInfix "services.hyperhive.deploy.bao.clientCertFile" (noSvcBranch noIdFetch)
&& lib.hasInfix "services.hyperhive.deploy.bao.clientKeyFile" (noSvcBranch noIdFetch)
&& lib.hasSuffix "exit 1" (noSvcBranch noIdFetch)
&& lib.hasInfix "baoHiveIdentityCertFile (unset)" (noSvcBranch noIdFetch)
&& lib.hasInfix "baoHiveIdentityKeyFile (unset)" (noSvcBranch noIdFetch)
&& lib.hasSuffix "exit 1" (lib.removeSuffix "\n" (noSvcBranch noIdFetch))
&& !(lib.hasInfix "exit 0" (noSvcBranch noIdFetch))
&& !(lib.hasInfix "not staged" noIdFetch)
&& lib.hasInfix "/v1/pki/ca/pem" (noSvcBranch noSvcFetch);
@ -179,13 +187,11 @@ let
# An identity named but not on disk fails before curl, naming the paths.
name = "a hive fronting no swarm name whose identity files are absent fails the unit, naming the paths";
ok =
lib.hasInfix "for f in /etc/pki/bao-client.pem /etc/pki/bao-client-key.pem; do" (
noSvcBranch noSvcFetch
)
lib.hasInfix "for f in /etc/pki/bao-hive.pem /etc/pki/bao-hive-key.pem; do" (noSvcBranch noSvcFetch)
&& lib.hasInfix ''[ -r "$f" ] || missing="$missing $f"'' (noSvcBranch noSvcFetch)
&& lib.hasInfix "cannot read$missing" (noSvcBranch noSvcFetch)
&& lib.hasInfix "clientCertFile (/etc/pki/bao-client.pem)" (noSvcBranch noSvcFetch)
&& lib.hasInfix "clientKeyFile (/etc/pki/bao-client-key.pem)" (noSvcBranch noSvcFetch);
&& lib.hasInfix "baoHiveIdentityCertFile (/etc/pki/bao-hive.pem)" (noSvcBranch noSvcFetch)
&& lib.hasInfix "baoHiveIdentityKeyFile (/etc/pki/bao-hive-key.pem)" (noSvcBranch noSvcFetch);
}
{
# The fetch retries against a sealed store; a gateway `Requires=` on it
@ -201,6 +207,113 @@ let
in
!(holds noService) && holds allLocal;
}
{
# The controls are the store's PKI unit, which still mints its own CA
# with `openssl req`, and the swarm root's unit, which still reads
# `root-key.pem`: absence here is not a matcher that cannot match.
name = "no hive TLS unit generates a CA, signs with a CA key or reads the swarm root key";
ok =
let
hasCaWork =
s:
lib.any (p: lib.hasInfix p s) [
"openssl req"
"-CAkey"
"root-key.pem"
];
pkiScript = units.swarm-bao-pki.script;
rootScript = units.swarm-ca.script;
in
!(hasCaWork caUnit.script)
&& !(hasCaWork leafUnit.script)
&& !(hasCaWork noService.systemd.services.hive-tls-ca.script)
&& lib.hasInfix "openssl req" pkiScript
&& lib.hasInfix "root-key.pem" rootScript;
}
{
name = "a retired hive CA key is deleted, and its public certificate stays in the trust bundle";
ok =
lib.hasInfix ''rm -f "$d/ca-key.pem"'' caUnit.script
&& lib.hasInfix ''ca="$d/ca.pem"'' caUnit.script
&& lib.hasInfix ''if [ -s "$ca" ]; then anchors+=("$ca"); fi'' caUnit.script
&& lib.hasInfix ''if [ -s "$prev" ]; then anchors+=("$prev"); fi'' caUnit.script
&& lib.hasInfix ''cat "''${anchors[@]}" > "$bundle"'' caUnit.script;
}
{
# The store host's hive presents the leaf ./glue-bao-tls.nix mints for
# it; not the reader's, not the services issuer's.
name = "the gateway leaf is issued through this hive's own role, presenting its own identity";
ok =
let
t = allLocal.services.hyperhive.deploy.hive-controller.tls;
in
lib.hasInfix "pki/issue/swarm-hive-leaf-h1" leafUnit.script
&& lib.hasInfix "common_name=h1.t.local" leafUnit.script
&& lib.hasInfix "alt_names='*.h1.t.local'" leafUnit.script
&& lib.hasInfix "bao login -method=cert -token-only" leafUnit.script
&& leafUnit.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/hive-leaf.pem"
&& leafUnit.environment.BAO_CLIENT_KEY == "/var/lib/swarm-bao-pki/hive-leaf-key.pem"
&& t.baoHiveIdentityCertFile == leafUnit.environment.BAO_CLIENT_CERT
&& leafUnit.environment.BAO_CLIENT_CERT != t.baoClientCertFile
&& leafUnit.environment.BAO_CLIENT_CERT != allLocal.services.hyperhive.deploy.bao.clientCertFile;
}
{
# Checked before the leaf, so the unit is red even while a valid leaf
# keeps the gateway up. The control is a hive with the identity named,
# whose script checks the files instead.
name = "a hive with no store identity fails the gateway-leaf unit, naming both options";
ok =
let
s = noIdentity.systemd.services.hive-tls-resign.script;
before = lib.head (lib.splitString "reissue=0" s);
ok' = noService.systemd.services.hive-tls-resign.script;
in
lib.hasInfix "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile (unset)" before
&& lib.hasInfix "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile (unset)" before
&& lib.hasInfix "exit 1" before
&& lib.hasInfix "for f in /etc/pki/bao-hive.pem /etc/pki/bao-hive-key.pem; do" (
lib.head (lib.splitString "reissue=0" ok')
)
&& !(lib.hasInfix "(unset)" ok');
}
{
# A leaf the retired hive CA signed is re-issued from the store on the
# first run that reaches it; one that renews in time never is.
name = "the gateway leaf is re-issued when not chained to the store's root, and at half its role lifetime";
ok =
lib.hasInfix ''openssl verify -CAfile "$svcroot" "$leaf"'' leafUnit.script
&& lib.hasInfix (renewAt 720) leafUnit.script
&& lib.hasInfix (renewAt 48) shortTtl.systemd.services.hive-tls-resign.script
&& lib.hasInfix ''-in "$leaf" -noout -checkend "$renewat"'' leafUnit.script;
}
{
# A valid leaf on disk starts the gateway with the store sealed: nothing
# requires or waits on the issuing unit. The control is that unit
# itself, which does wait on the store's container.
name = "the gateway does not wait on the gateway-leaf unit, which waits on the store";
ok =
(leafUnit.requiredBy or [ ]) == [ ]
&& (leafUnit.before or [ ]) == [ ]
&& !(lib.any (u: lib.elem "hive-tls-resign.service" ((u.requires or [ ]) ++ (u.after or [ ]))) (
lib.attrValues (removeAttrs units [ "hive-tls-resign" ])
))
&& !(lib.elem "container@swarm-bao.service" ((caUnit.requires or [ ]) ++ (caUnit.after or [ ])))
&& lib.elem "container@swarm-bao.service" leafUnit.after
&& lib.elem "multi-user.target" leafUnit.wantedBy;
}
{
# Not `RemainAfterExit`, or the timer would start an active unit and
# renew nothing.
name = "a daily timer re-runs the gateway-leaf unit";
ok =
let
t = allLocal.systemd.timers.hive-tls-resign;
in
lib.elem "timers.target" t.wantedBy
&& t.timerConfig.OnCalendar == "daily"
&& (t.timerConfig.Unit or "hive-tls-resign.service") == "hive-tls-resign.service"
&& !(remainsAfterExit leafUnit);
}
];
in
runGroup "hive-tls" cases

19
nix/module-eval/name-guards.nix
View file

@ -106,6 +106,14 @@ let
swarm.hives.qr-other.domain = "q.t.local";
};
# The per-hive gateway-leaf subject, composed the same way.
hiveNamedAfterHiveLeafSubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao.hiveLeafCommonNamePrefix = "hl";
swarm.hives.other.domain = "o.t.local";
swarm.hives.hl-other.domain = "hl.t.local";
};
hiveNameWithComposedWord = hive {
deploy.swarm-otel.enable = false;
swarm.hives."h1-agent".domain = "a.t.local";
@ -203,6 +211,17 @@ let
a: !a.assertion && lib.hasInfix "'qr-other'" a.message
) hiveNamedAfterPerHiveReaderSubject.assertions;
}
{
# A hive taking this spelling would present a leaf hive `other`'s
# gateway-leaf role accepts, and could ask the store for that hive's
# certificate. The control is the legal roster below.
name = "a hive named after another hive's gateway-leaf subject is refused";
ok =
equalityGuardFired hiveNamedAfterHiveLeafSubject
&& lib.any (
a: !a.assertion && lib.hasInfix "'hl-other'" a.message
) hiveNamedAfterHiveLeafSubject.assertions;
}
{
# Without this the case above proves nothing: an arm that fires for every
# roster is not a guard, and `hives` is non-empty in both fixtures.