vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm CA: retire the host-held root CA, the store holds the only anchor

Browse source 
The secret store's pki root issues every hive's gateway leaf and the
swarm's service names, so the root swarm-ca.service generated on the
host anchors nothing and its key is a secret outside the store.

- swarm-ca.nix is gone: swarm-ca.service, root.pem and root-key.pem,
  and the swarm.ca.{autoConfigure,stateDir,validityDays} options. A
  definition of any of them fails evaluation with a removal message
  (swarm-ca-removed.nix), and a tmpfiles rule deletes /var/lib/swarm-ca
  on every host. singleHostSwarm no longer sets a swarm.* option.
- hive-tls-ca drops the $root slot: trust-bundle.pem holds the store's
  pki root and a hive CA's public ca.pem/ca-previous.pem only.
- hivectl peer-config no longer prints the scp line for the swarm root;
  the store's pki root is the one step.
- The gateway's cert import no longer stands a hive leaf or a throwaway
  in for a missing swarm-services leaf where it Requires= the issuing
  unit: it fails, naming swarm-services-cert.service. The stand-in stays
  where the gateway does not wait (no swarm name, or the store's UI
  only, whose passthrough must start with the store sealed).

Comments and docs that named swarm-ca.nix, swarm.ca.* or the host root
now name the store's pki root.

Refs #4588
This commit is contained in:
atlas 2026-10-05 19:37:35 +02:00 • committed by mara
commit 93f1a76ea7
22 changed files with 305 additions and 395 deletions
Download patch file Download diff file Expand all files Collapse all files

2
nix/host-modules/default.nix
View file

@ -47,7 +47,6 @@
./swarm-authelia.nix
./swarm-bao-service.nix
./swarm-bao.nix
./swarm-ca.nix
./swarm-ci-runner.nix
./swarm-secret-publisher.nix
./swarm-nats-service.nix
@ -65,6 +64,7 @@
./swarm-victoriametrics.nix
./swarm-wireguard.nix
./swarm.nix
./swarm-ca-removed.nix
./swarm-peers-removed.nix
./swarm-required-services.nix
];

2
nix/host-modules/deploy.nix
View file

@ -4,7 +4,7 @@
# different kinds of fact and only one of them varies per machine:
#
# swarm.* — swarm-wide truth. The swarm's name, domain, hives, peers,
# CA, and where each service lives. **Identical on every
# and where each service lives. **Identical on every
# host**, byte for byte; a hive needs all of it to be a
# *client* of the swarm.
# deploy.* — this machine's deployment decisions. Necessarily different

8
nix/host-modules/hive-c0re/default.nix
View file

@ -128,10 +128,10 @@ in
./theme.nix
# Hive-CA trust for this daemon's outbound TLS. The matrix provisioning
# URL (`swarm.matrix.apiUrl`) defaults to https on the homeserver's
# gateway vhost, which serves a leaf chaining to the store's services
# root or the swarm root — CAs generated at runtime, which no
# build-time trust store can contain. The queue's TLS is verified
# against `HIVE_C0RE_OIDC_CA_FILE` (../hive-tls.nix), not this bundle.
# gateway vhost, which serves a leaf chaining to the store's pki root —
# a CA generated at runtime, which no build-time trust store can
# contain. The queue's TLS is verified against `HIVE_C0RE_OIDC_CA_FILE`
# (../hive-tls.nix), not this bundle.
#
# ⚠️ `SSL_CERT_FILE` REPLACES the trust store, and unlike the other
# consumers of this helper, hive-c0re already makes a *public*-TLS call:

4
nix/host-modules/hive-forge/service.nix
View file

@ -100,8 +100,8 @@ in
⚠️ To use the per-hive name `forge.''${services.hyperhive.domain}`
instead, pin it here. Certificates follow either way: the
swarm-services sub-CA is name-constrained to the configured
names (see `./swarm-ca.nix`), not to a fixed tree.
store's `swarm-services` role issues for the configured names,
not for a fixed tree.
Set to a full hostname (`git.example.com`,
`forge.internal.lan`, etc.) for a bespoke vhost shape — the

81
nix/host-modules/hive-gateway/default.nix
View file

@ -29,6 +29,16 @@ let
# The hive leaf exists only where a hive runs; ../hive-tls.nix issues it
# under that same condition.
runsHive = config.services.hyperhive.deploy.hive-controller.enable;
# Whether the cert import `Requires=` the services leaf's issuing unit
# (../hive-tls.nix decides). Where it does, a missing leaf fails the
# import rather than anything standing in for it.
svcLeafRequired = lib.elem "hive-gateway-self-signed-cert.service" (
config.systemd.services.swarm-services-cert.requiredBy or [ ]
);
svcLeafMissing = ''
echo "no swarm-services leaf at ${tlsStateDir}/swarm-services.pem, and this gateway serves swarm names only with it." >&2
echo "swarm-services-cert.service issues it from the secret store's pki mount; its journal says why it has not." >&2
exit 1'';
# Every vhost claiming `default_server`, ours and the operator's
# alike. Computed once so the assertion below and the message it
@ -252,10 +262,9 @@ in
# 🚨 DO NOT "simplify" this into pointing nginx at the CA dir. It does
# TWO jobs: it re-modes the leaf to the 0640 root:nginx copy below (a
# 0600 key fails nginx's pre-start config test and blocks the unit),
# and it guarantees every cert path the config names exists — which is
# what the swarm-services fallback at the bottom of the script is for.
# A leaf that never issues therefore takes the WHOLE gateway down, not
# one vhost. Both mechanisms, and what it cost the time this was
# and it puts a file at every cert path the config names or fails: one
# nginx cannot find takes the WHOLE gateway down, not one vhost. Both
# mechanisms, and what it cost the time this was
# collapsed: `docs/networking/gateway.md::Self-signed TLS (default)`.
#
# ALWAYS runs (no ConditionPathExists) and is idempotent, because it
@ -307,49 +316,51 @@ in
# because the hive leaf's role allows only its own hive's
# domain, and the service names are siblings of it.
#
# Absent is a normal state, not a failure: a host fronting
# no swarm name has none, and issuance can also fail on a
# host that wants one.
#
# ⚠️ When it is absent the HIVE leaf goes to this path
# anyway, and that fallback is load-bearing rather than
# tidy. nginx refuses to load a config naming a cert file
# that does not exist — `cannot load certificate … no such
# file` fails the pre-start test, so the vhost does not
# degrade, the ENTIRE gateway dies and takes the forge, the
# dashboard and matrix with it. Serving the hive leaf on a
# swarm-service name is a name mismatch: browsers warn,
# strict clients refuse, everything else keeps working, and
# the operator gets a bad cert instead of no hive.
#
# Measured, not theorised: this exact path took pr1ma's
# gateway down when the services sub-CA failed to issue.
# Where the gateway waits for it, a missing leaf fails this
# unit and nginx with it. Elsewhere — no swarm name here, or
# only the store's UI, whose host must start its passthrough
# with the store sealed — the hive leaf stands in, because
# nginx refuses a config naming a missing cert file, until
# `swarm-services-cert` issues one and re-runs this unit.
if [ -s ${tlsStateDir}/swarm-services.pem ]; then
install -m 0644 ${tlsStateDir}/swarm-services.pem ${svcCert}
install -m 0640 -g nginx ${tlsStateDir}/swarm-services-key.pem ${svcKey}
else
echo "no swarm-services leaf — serving the hive leaf on those names (mismatch, not an outage)" >&2
install -m 0644 ${tlsStateDir}/gateway.pem ${svcCert}
install -m 0640 -g nginx ${tlsStateDir}/gateway-key.pem ${svcKey}
${
if svcLeafRequired then
svcLeafMissing
else
''
echo "no swarm-services leaf yet — serving the hive leaf on those names until swarm-services-cert.service issues one" >&2
install -m 0644 ${tlsStateDir}/gateway.pem ${svcCert}
install -m 0640 -g nginx ${tlsStateDir}/gateway-key.pem ${svcKey}''
}
fi''
else
''
# No hive here, so no hive leaf: every vhost, the catch-all
# included, serves the swarm-services leaf. Until it lands a
# throwaway self-signed pair stands in, because nginx refuses
# a config naming a missing cert file. A store-only host does
# not wait for the leaf, so its stream passthrough comes up with
# the store sealed. `swarm-services-cert` re-runs this unit
# once the leaf lands.
# included, serves the swarm-services leaf. Where the gateway
# waits for it, a missing leaf fails this unit. A store-only
# host does not wait, so its stream passthrough comes up with
# the store sealed: a throwaway self-signed pair stands in,
# because nginx refuses a config naming a missing cert file,
# until `swarm-services-cert` issues the leaf and re-runs this
# unit.
if [ -s ${tlsStateDir}/swarm-services.pem ] && [ -s ${tlsStateDir}/swarm-services-key.pem ]; then
cert=${tlsStateDir}/swarm-services.pem
key=${tlsStateDir}/swarm-services-key.pem
else
echo "no swarm-services leaf yet — serving a throwaway self-signed certificate until it lands" >&2
cert=${tlsDir}/fallback.pem
key=${tlsDir}/fallback-key.pem
(umask 077; openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days 30 \
-subj "/CN=hyperhive-gateway-fallback" -keyout "$key" -out "$cert")
${
if svcLeafRequired then
svcLeafMissing
else
''
echo "no swarm-services leaf yet — serving a throwaway self-signed certificate until it lands" >&2
cert=${tlsDir}/fallback.pem
key=${tlsDir}/fallback-key.pem
(umask 077; openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days 30 \
-subj "/CN=hyperhive-gateway-fallback" -keyout "$key" -out "$cert")''
}
fi
install -m 0644 "$cert" ${tlsCert}
install -m 0640 -g nginx "$key" ${tlsKey}

9
nix/host-modules/hive-matrix.nix
View file

@ -12,7 +12,7 @@ let
hyperhiveDomain = config.services.hyperhive.domain;
# Same runtime→build-time bridge hive-forge and swarm-ci-runner cross:
# binds the hive trust bundle (which folds in the swarm root) into the
# binds the hive trust bundle (which folds in the store's root) into the
# container and orders the container after `hive-tls-ca.service`. The
# assembled bundle itself comes from `caTrust.trustBundle`, imported in
# the container config below.
@ -783,10 +783,9 @@ in
# Swarm-internal trust reaches tuwunel at RUNTIME, not via
# `security.pki.certificateFiles`. That option is read when the
# system is BUILT, and the swarm root is deliberately a runtime
# file (`swarm.ca.stateDir`) because its key must never enter the
# store — so there is nothing build-time to name. The bind-mount
# above plus the bundle service below are what replaced it.
# system is BUILT, and the store's root is a runtime file fetched
# from the store at boot — so there is nothing build-time to name.
# The bind-mount above plus the bundle service below carry it.
# tuwunel hard-fails to boot if `/etc/resolv.conf` has no
# `nameserver` line (`Failed to configure DNS resolver ... no

20
nix/host-modules/hive-tls.nix
View file

@ -8,7 +8,6 @@ let
cfg = config.services.hyperhive.deploy.hive-controller.tls;
hyperhiveCfg = config.services.hyperhive;
gatewayCfg = config.services.hyperhive.gateway;
swarmCaCfg = config.services.hyperhive.swarm.ca;
domain = hyperhiveCfg.domain;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
@ -346,15 +345,6 @@ in
};
config = lib.mkIf active {
# 🚫 There is deliberately NO warning that these service names are
# outside this hive's domain and this host cannot sign for them. The
# services leaf comes from the secret store's `pki` mount, which every
# hive reaches over the network with its own identity, so holding the
# swarm root key does not decide whether a hive can serve a
# swarm-service name. Such a warning would fire on every rebuild of
# every hive that is not the CA host, about a fallback that does not
# happen.
# The state dir and the trust bundle. Never waits on the store: the
# store's container is ordered after this unit, which writes the
# bundle it bind-mounts, so the leaf is `hive-tls-resign`'s.
@ -366,8 +356,6 @@ in
# turn `Requires=`. Order against the unit that reads the dir.
before = [ "hive-gateway-self-signed-cert.service" ];
requiredBy = [ "hive-gateway-self-signed-cert.service" ];
# The bundle carries the swarm root where this host generates it.
#
# `swarm-services-cert` is NOT ordered against this unit in either
# direction, and the absence is load-bearing — see the cycle spelled
# out at that unit. It runs after the store's container, the container
@ -381,7 +369,6 @@ in
# "$svcroot" ]` guard is what makes that state legal rather than fatal,
# and `swarm-services-cert` restarts this unit once the root it wrote
# is new — so the bundle converges instead of being a boot behind.
after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
@ -396,7 +383,6 @@ in
ca="$d/ca.pem"
prev="$d/ca-previous.pem"
root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"}
svcroot="$d/swarm-services-root.pem"
# --- Trust bundle: what a consumer must TRUST. Why it is a set of
@ -423,7 +409,6 @@ in
anchors=()
if [ -s "$ca" ]; then anchors+=("$ca"); fi
if [ -s "$prev" ]; then anchors+=("$prev"); fi
if [ -s "$root" ]; then anchors+=("$root"); fi
if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi
# `cat` with no arguments reads stdin, so no anchor at all is an
# empty file until `swarm-services-cert` writes the root.
@ -834,10 +819,7 @@ in
# flattening the chain: the issuer here is a self-signed root, so
# appending it would ship an anchor a client either already holds
# — in which case it is redundant — or does not, in which case
# shipping it proves nothing. The old file appended the services
# sub-CA because that WAS an intermediate, and a client anchored
# on the swarm root could not build the middle of the chain by
# itself.
# shipping it proves nothing.
mv -f "$d/swarm-services-leaf.new" "$svcleaf"
mv -f "$d/swarm-services-key.pem.new" "$d/swarm-services-key.pem"
chmod 0644 "$svcleaf"

27
nix/host-modules/local-defaults.nix
View file

@ -32,7 +32,6 @@ in
toggles that an all-on-one-box deployment implies: the swarm's
shared services
(`services.hyperhive.deploy.allSwarmServices`), the swarm
CA (`services.hyperhive.swarm.ca.autoConfigure`), the swarm
controller (`services.hyperhive.deploy.swarm-controller.enable`), and the
host's `/etc/hosts` entries for the names this hive serves
(`services.hyperhive.gateway.localHostsEntry`) — with no real DNS
@ -71,11 +70,8 @@ in
# break every agent's access to the forge.
config.services.hyperhive.gateway.localHostsEntry = lib.mkDefault cfg.deploy.singleHostSwarm;
# Out of the `swarm` attrset below, because it is a `deploy.*` option now
# (./deploy.nix): "does THIS host run the swarm's services" is a per-host
# decision. Written as a path rather than folded into a second
# `config.services.hyperhive.deploy = { … }` attrset, for the same reason
# the ⚠️ below gives about `swarm`.
# A `deploy.*` option (./deploy.nix): "does THIS host run the swarm's
# services" is a per-host decision.
config.services.hyperhive.deploy.allSwarmServices = lib.mkDefault cfg.deploy.singleHostSwarm;
# The queue's auth-callout nkeys. Generating them is safe exactly
@ -86,15 +82,6 @@ in
config.services.hyperhive.deploy.nats.autoGenerateCallout =
lib.mkDefault cfg.deploy.singleHostSwarm;
# ⚠️ Every `swarm.*` default this mode sets lives INSIDE this attrset, not
# as a second `config.services.hyperhive.swarm.…` path beside it — written
# that way the two definitions of `swarm` collide and the nested one is
# silently lost. The gate caught exactly that on the controller's queue URL,
# when this mode still set it: mode on, `natsUrl` still "".
config.services.hyperhive.swarm = {
ca.autoConfigure = lib.mkDefault cfg.deploy.singleHostSwarm;
};
# The controller is asserted by the MODE and by nothing else. Its own
# option stays `default = false` precisely because running it is a
# statement about swarm topology — but "this box is the whole
@ -103,11 +90,8 @@ in
# instead would be wrong: a hive in a larger swarm can legitimately
# want the shared services without being the host that controls them.
#
# Sits outside the `swarm` attrset above because it is a `deploy.*`
# option (./deploy.nix): "does THIS host run the controller" is exactly
# the per-host fact `swarm.*` may not carry. The ⚠️ collision note above
# does not apply here — that one is about two definitions of `swarm`
# itself, and this is a different top-level path.
# A `deploy.*` option (./deploy.nix): "does THIS host run the controller"
# is exactly the per-host fact `swarm.*` may not carry.
config.services.hyperhive.deploy.swarm-controller.enable = lib.mkDefault cfg.deploy.singleHostSwarm;
# Where the operator drops the token that writes the swarm's first bao
@ -122,8 +106,7 @@ in
# substitute for the other: one unseals the store, this one authorises the
# first write.
#
# Outside the `swarm` attrset above for the same reason the controller's
# options are — a path on this host is a `deploy.*` fact.
# A path on this host is a `deploy.*` fact, like the controller's options.
config.services.hyperhive.deploy.bao.bootstrapTokenFile = lib.mkIf cfg.deploy.singleHostSwarm (
lib.mkDefault "/var/lib/swarm-bao-bootstrap/grant.token"
);

47
nix/host-modules/swarm-bao.nix
View file

@ -597,17 +597,15 @@ let
#
# The mount's own root is generated into it by the bootstrap unit below and
# its private key never leaves — `root/generate/internal` keeps it inside
# the store, which is the whole reason the mount exists rather than another
# key beside ./swarm-ca.nix's.
# the store.
#
# An option rather than a literal because ./hive-tls.nix issues through it
# and has to spell it the same way.
servicesPkiMountPath = baoDeploy.servicesPkiMountPath;
# Subject of the root generated into that mount. A label for a human reading
# a chain, not an identity anything authenticates against — same fall-through
# ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set
# neither a swarm name nor a domain must still evaluate.
# a chain, not an identity anything authenticates against. Falls through so
# a hive that has set neither a swarm name nor a domain still evaluates.
servicesPkiRootLabel =
if hyperhiveCfg.swarm.name != null then
hyperhiveCfg.swarm.name
@ -624,11 +622,9 @@ let
servicesPkiRootDir = "/var/lib/swarm-bao-services-pki";
servicesPkiRootFile = "${servicesPkiRootDir}/services-root.pem";
# ~30y, matching ./swarm-ca.nix's root rather than its services sub-CA. This
# is an ANCHOR: an operator adds it to a browser's trust store by hand, and
# every rotation is a manual re-trust on every machine that holds it. The
# sub-CA's short window bought cheap rotation because the root above it
# absorbed the churn; nothing sits above this one.
# ~30y. This is an ANCHOR: an operator adds it to a browser's trust store by
# hand, and every rotation is a manual re-trust on every machine that holds
# it; nothing sits above this one to absorb the churn.
servicesPkiRootTtl = "262800h";
# The leaf's window, pinned on the role rather than left to the mount's
@ -645,16 +641,16 @@ let
# The role every swarm-service certificate will be issued through, and the
# whole narrowing of this mount. A role rather than a name-constrained
# sub-CA: the engine enforces the same "these names and no others" at issue
# time that ./swarm-ca.nix builds into x509 `nameConstraints` today, so a
# second CA in the chain would re-implement a check that already happens.
# sub-CA: the engine enforces "these names and no others" at issue time, so
# x509 `nameConstraints` on a second CA in the chain would re-implement a
# check that already happens.
servicesPkiRoleName = baoDeploy.servicesPkiRoleName;
# The names that role may issue for. Read out of the swarm-tier option
# rather than assembled here, exactly as ./swarm-ca.nix:45 and
# ./hive-tls.nix:16 read it — the thing that narrows what may be issued and
# the leaf that carries those names as SANs must agree, and a second module
# composing its own list is how they stop agreeing.
# rather than assembled here, exactly as ./hive-tls.nix reads it — the thing
# that narrows what may be issued and the leaf that carries those names as
# SANs must agree, and a second module composing its own list is how they
# stop agreeing.
swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains;
# The identity that asks this mount for a certificate. A FOURTH principal
@ -1374,7 +1370,7 @@ in
clientCaFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-ca/root.pem";
example = "/var/lib/swarm-bao-pki/ca.pem";
description = ''
Authority the store validates hive **client** certificates against.
@ -1385,7 +1381,7 @@ in
On a hive that runs the store, a glue module supplies the CA it
minted for exactly this, as a `mkDefault`. Point this at something
else — the swarm root, an operator's own CA — and yours wins.
else — an operator's own CA — and yours wins.
`null` leaves client-certificate verification off, which is only
appropriate where something else authenticates the connection.
@ -2806,21 +2802,18 @@ in
#
# Every narrowing is load-bearing, and three of them turn OFF a
# default that is on:
# * `allow_subdomains` — the x509 constraint ./swarm-ca.nix writes
# permits a whole subtree per name; a swarm service is an exact
# hostname, so the role is the narrower of the two.
# * `allow_subdomains` — a swarm service is an exact hostname, not
# a subtree.
# * `allow_localhost` — bao's own default is `true`, which would
# make this role issue for a name nobody configured.
# * `allow_ip_sans` — the sub-CA excludes both IP subtrees
# explicitly (./swarm-ca.nix:260-276); an unconstrained name type
# is a name type this issuer is unconstrained for.
# * `allow_ip_sans` — an unconstrained name type is a name type
# this issuer is unconstrained for.
# `server_flag` alone because that is what today's leaf carries
# (`extendedKeyUsage=serverAuth`, ./hive-tls.nix:144), and rsa:4096
# because that is the key the current implementation generates.
#
# An empty `swarm.serviceDomains` renders an empty `allowed_domains`,
# which with `allow_any_name=false` is a role that issues nothing —
# the same outcome as ./swarm-ca.nix's "nothing to issue for" exit,
# which with `allow_any_name=false` is a role that issues nothing,
# reached without a branch.
bao write ${lib.escapeShellArg "${servicesPkiMountPath}/roles/${servicesPkiRoleName}"} \
allowed_domains=${lib.escapeShellArg (lib.concatStringsSep "," swarmServiceDomains)} \

33
nix/host-modules/swarm-ca-removed.nix Normal file
View file

@ -0,0 +1,33 @@
# Removal shim for `services.hyperhive.swarm.ca.*` and the host-held swarm
# root it generated.
#
# The secret store's `pki` root is the only trust anchor: it issues every
# hive's gateway leaf and the swarm's service names, and its key never leaves
# the store. Nothing generates, reads or bundles a root under
# `/var/lib/swarm-ca`, so a definition of these options fails evaluation with
# a message, and the directory is deleted on every host.
#
# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to
# remove when the deprecation window closes.
{ lib, ... }:
let
removed =
name:
lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] ''
The host-held swarm root CA is gone: the secret store's pki root is the
only trust anchor, and nothing generates or reads a root under
/var/lib/swarm-ca. Remove this definition. See docs/swarm/ca.md.
'';
in
{
imports = map removed [
"autoConfigure"
"stateDir"
"validityDays"
];
# The root's private key is a secret outside the store, and the
# certificate beside it is an anchor nothing issues under. The option's
# default path only: a relocated `stateDir` is the operator's to delete.
systemd.tmpfiles.rules = [ "R /var/lib/swarm-ca" ];
}

161
nix/host-modules/swarm-ca.nix
View file

@ -1,161 +0,0 @@
# The swarm root CA: the anchor a whole swarm shares, and the issuer of
# each hive's own CA (which is where it gets used — see ./hive-tls.nix).
#
# One root rather than per-peer pinning, because cross-hive trust is then
# O(1): trust the root once and every present *and future* peer validates,
# instead of every hive having to name every other one.
#
# Two provisioning modes share ONE structure — what differs is who puts
# the artifacts on disk, never what the artifacts are. Which mode does
# what, and what moving between them costs:
# `docs/swarm/ca.md::Two provisioning modes, one structure`.
#
# ⚠️ Root key AND root cert are runtime files, never nix options. The key
# for the obvious reason; the cert as a consequence, and that one costs
# something real — nothing whose trust store is built at build time can
# name it: `docs/swarm/ca.md::Distributing the root`.
#
# ⚠️ This root does not issue the swarm's SERVICE certificates; there is no
# `swarm-services-ca` sub-CA under it. Those come out of the secret store's
# own `pki` mount, anchored on a root generated inside it (./swarm-bao.nix),
# and ./hive-tls.nix asks for them over the network instead of signing them
# here. This root has one job: the anchor each hive's own CA is issued under.
{
lib,
config,
pkgs,
...
}:
let
cfg = config.services.hyperhive.swarm.ca;
hyperhiveCfg = config.services.hyperhive;
# The subject CN is a label for a human reading a chain, not an
# identity anything authenticates against. Fall through swarm name →
# hive domain → a constant so a hive that has set neither still
# evaluates; a missing `domain` is reported by its own assertion in
# hive-network.nix, and shouldn't also surface here as a null.
swarmLabel =
if hyperhiveCfg.swarm.name != null then
hyperhiveCfg.swarm.name
else if hyperhiveCfg.domain != null then
hyperhiveCfg.domain
else
"hyperhive";
in
{
options.services.hyperhive.swarm.ca = {
autoConfigure = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Run the whole swarm CA on this one host: generate the swarm
root when it is missing, and issue this hive's CA under it.
`services.hyperhive.deploy.singleHostSwarm` turns this on as
part of the all-on-one-box mode. Set it here directly to run the
CA on a host that is not otherwise all-local.
**Off by default, deliberately.** A swarm's services and its
hives can live on different hosts, and this host has no way to
tell whether it is the one holding the root — so the swarm CA
is something an operator sets up, not something a host decides
it is. Turn this on for an all-on-one-host deployment (dev
boxes, single-hive swarms) and get the hierarchy for free.
With it off, both artifacts are operator-provided: the root
under `stateDir`, and this hive's CA under
`services.hyperhive.deploy.hive-controller.tls.stateDir`. A hive
given neither keeps the self-signed CA it has always had — it
simply isn't part of a swarm's trust hierarchy, which is the
correct outcome for a hive nobody has federated yet.
'';
};
stateDir = lib.mkOption {
type = lib.types.str;
default = "/var/lib/swarm-ca";
description = ''
Host directory holding the swarm root CA: `root.pem` (the
anchor, safe to distribute — copy it to this same path on every
other host in the swarm) and `root-key.pem`
(0600, the one file that must never reach the nix store or
another host). The directory itself is 0700: nothing reads
out of it but the hive CA issuance in `hive-tls.nix`.
Moving the swarm CA to its own host is a matter of moving this
directory and setting `autoConfigure = false` here.
'';
};
validityDays = lib.mkOption {
type = lib.types.int;
default = 10950;
description = ''
Validity window of the swarm root CA in days (default ~30y).
Deliberately long: the root must outlive the hive CAs it issued, or those chains
expire out from under hives that are still perfectly happy with
their own intermediate. Rotating a root is the one operation in
this system with no partial-failure mode — it invalidates every
peer at once, paced by the slowest peer's rebuild — so it is
never automatic and this window is meant to be uneventful.
'';
};
};
config = lib.mkIf cfg.autoConfigure {
systemd.services.swarm-ca = {
description = "Generate the swarm root CA when absent";
wantedBy = [ "multi-user.target" ];
path = [ pkgs.openssl ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
UMask = "0077";
# Pin the journal identity (else it's the `script` store-path wrapper).
SyslogIdentifier = "swarm-ca";
};
script = ''
set -euo pipefail
d=${lib.escapeShellArg cfg.stateDir}
install -d -m 0700 "$d"
root="$d/root.pem"
rootk="$d/root-key.pem"
# Note the asymmetry with the hive CA in hive-tls.nix, which
# regenerates itself once expired: a root is never replaced
# automatically, not even an expired one. Consumers hold this
# cert, so replacing it is a swarm-wide flag day that wants an
# operator running it deliberately, with both roots trusted
# across the overlap.
if [ -s "$root" ] && [ -s "$rootk" ]; then
echo "swarm root CA already present at $root — leaving it alone"
exit 0
fi
# Half a root is not a root. Generating a fresh key beside an
# already-distributed cert (or the reverse) leaves every
# consumer trusting an anchor that no longer signs anything —
# and it would look like it worked.
if [ -e "$root" ] || [ -e "$rootk" ]; then
echo "swarm root CA half-provisioned ($root / $rootk) — refusing to generate over it" >&2
exit 1
fi
echo "generating swarm root CA at $root"
# pathlen:1 — the root signs hive CAs, which sign leaves. One
# intermediate below the root and no deeper.
openssl req -x509 -newkey rsa:4096 -nodes -sha256 \
-days ${toString cfg.validityDays} \
-keyout "$rootk" -out "$root" \
-subj "/CN=swarm-ca ${swarmLabel}" \
-addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
-addext "keyUsage=critical,keyCertSign,cRLSign"
chmod 0600 "$rootk"
chmod 0644 "$root"
'';
};
};
}

2
nix/host-modules/swarm-controller.nix
View file

@ -669,7 +669,7 @@ in
hiveClientCaFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-ca/root.pem";
example = "/var/lib/swarm-bao-pki/ca.pem";
description = ''
Authority whose leaves are hive client certificates, so that this
daemon can create each hive's cert-auth role. The store matches a

9
nix/host-modules/swarm-peers-removed.nix
View file

@ -57,11 +57,10 @@ in
services.hyperhive.swarm.peers.<hive>.caCert is removed and
ignored, and has no replacement in services.hyperhive.swarm.hives.
Trust inside a swarm now derives from the swarm root CA
(services.hyperhive.swarm.ca — see docs/swarm/ca.md): every hive
under it chains to it, so a per-hive CA is dead weight. What this
genuinely drops is trusting a hive whose root this swarm does NOT
own — another swarm's, or one keeping its own CA.
Trust inside a swarm derives from the secret store's pki root (see
docs/swarm/ca.md): every hive chains to it, so a per-hive CA is dead
weight. What this genuinely drops is trusting a hive whose root this
swarm does NOT own — another swarm's, or one keeping its own CA.
Still set on: ${lib.concatStringsSep ", " withCaCert}
'';

26
nix/host-modules/swarm.nix
View file

@ -72,13 +72,13 @@ let
# `serviceDomains` below is this set sorted + deduplicated.
#
# ⚠️ These are NOT required to be under `swarm.domain`. An earlier
# revision asserted that, reasoning that the services sub-CA is
# constrained to the swarm's tree — but the sub-CA is constrained to
# the **configured names** (./swarm-ca.nix) and the swarm root carries
# no name constraints at all, so any configured name is issuable. The
# assertion encoded an intended shape, not a property of the code, and
# it rejected the supported migration path: a hive pinning its old
# `forge.<hive domain>` while joining a swarm at a different apex.
# revision asserted that, reasoning that the services issuer is
# constrained to the swarm's tree — but the store's `swarm-services`
# role admits exactly the **configured names** (./swarm-bao.nix), so any
# configured name is issuable. The assertion encoded an intended shape,
# not a property of the code, and it rejected the supported migration
# path: a hive pinning its old `forge.<hive domain>` while joining a swarm
# at a different apex.
#
# Every name is listed whether or not this host runs the service: the
# store's issuing role is built from this list on the store's host and
@ -226,9 +226,9 @@ in
visible = false;
internal = true;
description = ''
Removed. Trust inside a swarm comes from the swarm root CA
(`services.hyperhive.swarm.ca`, docs/swarm/ca.md), which
replaces per-hive leaf pinning entirely.
Removed. Trust inside a swarm comes from the secret store's
pki root (docs/swarm/ca.md), which replaces per-hive leaf
pinning entirely.
'';
};
};
@ -369,9 +369,9 @@ in
existed to serve — nothing else ever consumed it.
There is no replacement, and none is needed for a hive inside
this swarm: trust comes from the swarm root CA
(services.hyperhive.swarm.ca — see docs/swarm/ca.md), which
every hive chains to, so one anchor replaces per-hive pinning.
this swarm: trust comes from the secret store's pki root (see
docs/swarm/ca.md), which every hive chains to, so one anchor
replaces per-hive pinning.
What that genuinely drops is trusting a hive whose root this
swarm does NOT own — another swarm's, or one keeping its own
CA. That is a cross-swarm problem and wants a mechanism

109
nix/module-eval/hive-tls.nix
View file

@ -4,7 +4,8 @@
# The swarm-services leaf's renewal: a timer that really re-runs the
# issuance, at a threshold read off the store role's lifetime. And the
# store's root on a hive that asks it for no leaf. And the gateway leaf,
# issued by the store through the hive's own role, with no hive CA.
# issued by the store through the hive's own role, with no hive CA and no
# host-held swarm root.
{
pkgs,
lib,
@ -21,6 +22,7 @@ let
nixosSystem
;
})
hive
hiveWithController
runGroup
;
@ -208,10 +210,9 @@ let
!(holds noService) && holds allLocal;
}
{
# The controls are the store's PKI unit, which still mints its own CA
# with `openssl req`, and the swarm root's unit, which still reads
# `root-key.pem`: absence here is not a matcher that cannot match.
name = "no hive TLS unit generates a CA, signs with a CA key or reads the swarm root key";
# The control is the store's PKI unit, which still mints its own CA
# with `openssl req`: absence here is not a matcher that cannot match.
name = "no hive TLS unit generates a CA or signs with a CA key";
ok =
let
hasCaWork =
@ -219,16 +220,106 @@ let
lib.any (p: lib.hasInfix p s) [
"openssl req"
"-CAkey"
"root-key.pem"
];
pkiScript = units.swarm-bao-pki.script;
rootScript = units.swarm-ca.script;
in
!(hasCaWork caUnit.script)
&& !(hasCaWork leafUnit.script)
&& !(hasCaWork noService.systemd.services.hive-tls-ca.script)
&& lib.hasInfix "openssl req" pkiScript
&& lib.hasInfix "root-key.pem" rootScript;
&& lib.hasInfix "openssl req" pkiScript;
}
{
# Every unit's script, environment and ordering, on the all-local host
# and on a hive that runs no store. The control is the same scan finding
# the store root's file, so an empty result is not a scan that reads
# nothing.
name = "no unit generates, reads or waits on a host-held swarm root, and every host deletes its directory";
ok =
let
texts =
m:
lib.concatMap (u: [
(u.script or "")
(builtins.toJSON (u.environment or { }))
(toString ((u.after or [ ]) ++ (u.requires or [ ]) ++ (u.wants or [ ])))
]) (lib.attrValues m.systemd.services);
mentions = m: needle: lib.any (lib.hasInfix needle) (texts m);
gone =
m:
!(m.systemd.services ? swarm-ca)
&& !(mentions m "swarm-ca.service")
&& !(mentions m "/var/lib/swarm-ca")
&& !(mentions m "root-key.pem")
&& lib.elem "R /var/lib/swarm-ca" m.systemd.tmpfiles.rules;
in
gone allLocal
&& gone noService
&& mentions allLocal "swarm-services-root.pem"
&& mentions noService "swarm-services-root.pem";
}
{
# Three slots and no fourth: a root slot beside them is an anchor
# nothing issues under.
name = "the trust bundle anchors on the store's root and a hive CA's public certificates only";
ok =
let
slots = lib.filter (l: lib.hasPrefix "if [ -s " l && lib.hasInfix "anchors+=(" l) (
lib.splitString "\n" caUnit.script
);
in
lib.length slots == 3
&& lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' caUnit.script
&& !(lib.hasInfix ''anchors+=("$root")'' caUnit.script);
}
{
# The control is the all-local host, which defines none of them.
name = "a definition of a removed swarm.ca option fails evaluation, naming the option and the store's root";
ok =
let
refused =
m: opt:
lib.any (
a:
!a.assertion
&& lib.hasInfix "services.hyperhive.swarm.ca.${opt}" a.message
&& lib.hasInfix "pki root is the" a.message
) m.assertions;
setting = {
autoConfigure = hive { swarm.ca.autoConfigure = true; };
stateDir = hive { swarm.ca.stateDir = "/var/lib/swarm-ca"; };
validityDays = hive { swarm.ca.validityDays = 10950; };
};
in
lib.all (opt: refused setting.${opt} opt) (lib.attrNames setting)
&& !(lib.any (opt: refused allLocal opt) (lib.attrNames setting));
}
{
# Where the import `Requires=` the services leaf's issuing unit, a
# missing leaf fails it; elsewhere the hive leaf stands in. The control
# is a hive fronting no swarm name, which does not wait.
name = "a hive gateway that waits for the services leaf fails its cert import without it, naming the issuing unit";
ok =
let
imp = m: m.systemd.services.hive-gateway-self-signed-cert.script;
waits =
m:
lib.elem "hive-gateway-self-signed-cert.service" m.systemd.services.swarm-services-cert.requiredBy;
standIn = "install -m 0644 /var/lib/hive-tls/gateway.pem /var/lib/hive-gateway/tls/swarm-services.pem";
refusal = "swarm-services-cert.service issues it";
# The `else` branch alone, up to the `fi` that closes it.
missingBranch =
m:
builtins.head (
lib.splitString "\nfi" (lib.last (lib.splitString "swarm-services.pem ]; then" (imp m)))
);
in
waits allLocal
&& lib.hasInfix refusal (missingBranch allLocal)
&& lib.hasSuffix "exit 1" (lib.removeSuffix "\n" (lib.trim (missingBranch allLocal)))
&& !(lib.hasInfix standIn (imp allLocal))
&& !(waits noService)
&& lib.hasInfix standIn (missingBranch noService)
&& !(lib.hasInfix refusal (imp noService));
}
{
# Nothing reads or removes a `ca-key.pem`, so a rollback finds the

38
nix/module-eval/swarm-services-switch.nix
View file

@ -287,30 +287,50 @@ let
lib.elem "hive-gateway-self-signed-cert.service" s.swarm-services-cert.requiredBy
&& lib.elem "hive-gateway-self-signed-cert.service" s.swarm-services-cert.before;
# The cert it installs on the default and swarm-services paths:
# the services leaf when present, a throwaway pair otherwise.
importsServicesLeafOrFallback =
# the services leaf when present.
importsServicesLeaf =
m:
let
imp = m.systemd.services.hive-gateway-self-signed-cert;
in
lib.hasInfix "cert=/var/lib/hive-tls/swarm-services.pem" imp.script
&& lib.hasInfix "openssl req -x509" imp.script
&& lib.hasInfix "cert=/var/lib/hive-gateway/tls/fallback.pem" imp.script
&& lib.hasInfix ''install -m 0644 "$cert" /var/lib/hive-gateway/tls/cert.pem'' imp.script
&& lib.hasInfix ''install -m 0644 "$cert" /var/lib/hive-gateway/tls/swarm-services.pem'' imp.script
&& lib.hasInfix ''install -m 0644 "$cert" /var/lib/hive-gateway/tls/swarm-services.pem'' imp.script;
# Without it: a throwaway pair where the gateway does not wait for
# the leaf, a failure naming the issuing unit where it does.
standsInThrowaway =
m:
let
imp = m.systemd.services.hive-gateway-self-signed-cert;
in
lib.hasInfix "openssl req -x509" imp.script
&& lib.hasInfix "cert=/var/lib/hive-gateway/tls/fallback.pem" imp.script
&& lib.any (p: lib.hasPrefix "openssl" (p.pname or (p.name or ""))) imp.path;
failsWithoutLeaf =
m:
let
s = m.systemd.services.hive-gateway-self-signed-cert.script;
in
lib.hasInfix "swarm-services-cert.service issues it" s
&& lib.hasInfix "exit 1" s
&& !(lib.hasInfix "fallback.pem" s);
store = storeOnlyNoHive;
hs = swarmServicesHere.systemd.services;
in
noHive swarmServicesOnly
&& noHive swarmServicesOnlyNoStore
&& noHive store
&& importsServicesLeafOrFallback swarmServicesOnly
&& importsServicesLeafOrFallback swarmServicesOnlyNoStore
&& importsServicesLeafOrFallback store
# Fronting many swarm names: the gateway waits for their leaf.
&& importsServicesLeaf swarmServicesOnly
&& importsServicesLeaf swarmServicesOnlyNoStore
&& importsServicesLeaf store
# Fronting many swarm names: the gateway waits for their leaf, and
# its import fails without it.
&& waitsForLeaf swarmServicesOnly
&& waitsForLeaf swarmServicesOnlyNoStore
&& failsWithoutLeaf swarmServicesOnly
&& failsWithoutLeaf swarmServicesOnlyNoStore
&& standsInThrowaway store
&& !(failsWithoutLeaf store)
# The store's UI alone: its nginx carries the passthrough to the
# store, so it starts on the fallback with the store sealed.
&&