diff --git a/README.md b/README.md index 8f4a319d..21f04414 100644 --- a/README.md +++ b/README.md @@ -150,7 +150,7 @@ covers spreading a swarm across hosts. ``` `singleHostSwarm` turns on everything an all-on-one-box swarm implies: -the shared services above, the swarm CA, the swarm controller, and +the shared services above, the swarm controller, and `/etc/hosts` entries so the hive's names resolve on the box itself. `claude-code` is unfree — hyperhive scopes the allowance to itself, so there's nothing to set. diff --git a/docs/networking/gateway.md b/docs/networking/gateway.md index 3298e386..18204a94 100644 --- a/docs/networking/gateway.md +++ b/docs/networking/gateway.md @@ -76,11 +76,11 @@ On by default, listening on `httpsPort` (default 443) beside the plain-http `por The issuer is the **secret store's `pki` mount**, not a bare self-signed leaf. `hive-tls-resign.service` (from `hive-tls.nix`) asks the store for a gateway **leaf** through this hive's own role, presenting the hive's own store identity (`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile`), and writes it under `…tls.stateDir` (default `/var/lib/hive-tls`). `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`). On a host that runs no hive the import unit installs the swarm-services leaf there instead, or a throwaway self-signed pair until that leaf lands. [`swarm/ca.md`](../swarm/ca.md) has the role, the identity and what a hive without one does. -⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the source's `0600 root:root` key), and it makes sure **every cert path the config names exists** — when the swarm-services leaf is missing it installs the hive leaf in its place, or on a host that runs no hive a throwaway self-signed pair. nginx refuses a config naming a missing cert file, so without that fallback one missing leaf takes down every vhost, not just one. +⚠️ **Keep the import unit.** It does two jobs nginx needs: it re-modes the key to `0640 root:nginx` (nginx's pre-start `nginx -t` runs as the nginx user and fails on the source's `0600 root:root` key), and it puts a file at **every cert path the config names**, because nginx refuses a config naming a missing cert file and one missing leaf would take down every vhost, not just one. Where the gateway waits for the swarm-services leaf, a missing one fails the import unit, naming `swarm-services-cert.service`, and nginx stays down until that unit issues it. Where it doesn't — a host fronting no swarm name, or only the store's UI, which starts its passthrough with the store sealed — the hive leaf stands in until the services leaf lands, or on a host that runs no hive a throwaway self-signed pair. **Why an issuer, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust, and an agent's build sets its trust store once, before any leaf exists. Agents and federation peers trust the store's root once; leaf rotation never breaks them. -**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root, plus a hive CA's public `ca.pem` and the swarm root where they exist ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle. +**What consumers trust**: `trust-bundle.pem` in the same state dir. It carries the store's `pki` root, plus a hive CA's public `ca.pem` where one exists ([`swarm/ca.md`](../swarm/ca.md#the-trust-bundle)). nginx serves the leaf alone, since its issuer is a self-signed root. Agents (via `security.pki.certificateFiles`), the CI and forge containers and federating peers all read the bundle. **Why on by default**: matrix-dart-sdk (FluffyChat's SDK) fetches `https:///.well-known/matrix/client` and never falls back to plain http, so without TLS the browser client can't bootstrap. diff --git a/docs/swarm/ca.md b/docs/swarm/ca.md index 89bff47c..7b153d6f 100644 --- a/docs/swarm/ca.md +++ b/docs/swarm/ca.md @@ -83,8 +83,9 @@ the file every consumer anchors on. It holds whichever of these exist: issues under that CA; they stay in the bundle so a leaf it signed verifies until `hive-tls-resign` replaces it. Removing them is a deliberate later step. -- the swarm root, `/root.pem`, on the host that generates - it. + +No other anchor goes in: the store's `pki` root is the only root the swarm +has. No hive generates a CA or signs with a CA key. A `/ca-key.pem` stays on disk, unread: @@ -163,46 +164,34 @@ role's `allowed_domains`, enforced by the store at issue time: a hive's identity that leaks can get certificates for that hive's names and no others. -The swarm root's private key never reaches the nix store: the store is -world-readable and content-addressed, so a key committed to a flake is a -key published to everyone who builds it. The swarm distributes only -certificates. +No hive holds a root CA or its key, and nothing under `/var/lib/swarm-ca` +exists: every host deletes that directory at boot and on every deploy. The +swarm distributes only certificates. ## What to hand a peer `hivectl peer-config` prints the `services.hyperhive.swarm.hives.""` block a peer operator pastes into their own config. When this hive's gateway serves a self-signed leaf (detected by the presence of -`/trust-bundle.pem`), it also prints a -one-time `scp` line installing the **swarm root** — -`/root.pem`, not this hive's own CA — on the peer's -host: - -``` -scp /var/lib/swarm-ca/root.pem :/var/lib/swarm-ca/root.pem -``` - -It then prints the secret store's `pki` root itself, read from +`/trust-bundle.pem`), it also prints the +secret store's `pki` root itself, read from `/var/lib/hive-tls/swarm-services-root.pem`. The store issues every hive's -gateway leaf and the swarm's service names under that root, so a peer host -trusting only the swarm root can't verify them. +gateway leaf and the swarm's service names under that root, so it's the one +anchor a peer host needs. -The operator installs the roots **once per swarm host**, not once per peer, +The operator installs the root **once per swarm host**, not once per peer, so a hive joining later needs no edit on the hives already running. A hive whose cert already chains to a public CA has nothing to install — `peer-config` says so instead. ## Distributing the root -The root **key** is a runtime file for the reason in -[Constraints on the material](#constraints-on-the-material). - -The root **certificate** is a runtime file as a _consequence_ — it lives -beside the key under `swarm.ca.stateDir` — and that has a cost worth +The root **certificate** is a runtime file: each hive fetches it from the +store at boot, so nothing can name it at build time. That has a cost worth naming, because it's subtle and it bites at a distance: > **Nothing whose trust store nix assembles at build time can reference -> the swarm root.** The derivation reads `security.pki.certificateFiles` +> the store's root.** The derivation reads `security.pki.certificateFiles` > inside itself; the root doesn't exist there. Two consumers, and only one of them is fine: @@ -231,7 +220,7 @@ Two consumers, and only one of them is fine: > fixed. The same caveat applies to `hive-forge` (Go) for the same > reason. -To put the root on another host, copy the certificate to the same path -there (`scp /root.pem :/root.pem`). One anchor -per host, not one per peer: a hive joining later needs no edit on the -hives already running, which is the whole point of the hierarchy. +A hive in the swarm fetches the root itself. A host outside it gets the +certificate from `hivectl peer-config` ([What to hand a peer](#what-to-hand-a-peer)). +One anchor per host, not one per peer: a hive joining later needs no edit +on the hives already running. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 93789f78..1aab6fcf 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -43,8 +43,6 @@ neither is a renaming of the other. | secret | generated by | lives at | hive elsewhere | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm root CA cert | `swarm-ca.nix` first-boot unit, when the operator sets `autoConfigure` | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | -| swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | @@ -445,17 +443,12 @@ the hive domain rather than a child** — `forge.` next to `.) -> String { - const HEAD: &str = "# 2. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\ - # the swarm's service names are issued under it:\n"; + const HEAD: &str = "# 1. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\ + # every hive's gateway leaf and the swarm's service names are issued under it:\n"; match pem.map(str::trim_end).filter(|p| !p.is_empty()) { Some(p) => format!( "{HEAD}{p}\n\ @@ -205,8 +191,8 @@ fn store_root_step(pem: Option<&str>) -> String { } /// `peer-config` — print the `swarm.hives.""` block a peer -/// operator pastes to federate with THIS hive, plus the swarm-root and -/// store-root steps when this hive serves a self-signed chain. Reads +/// operator pastes to federate with THIS hive, plus the store-root step +/// when this hive serves a self-signed chain. Reads /// local state only (the TLS trust bundle, the store root + the wg key); /// prints, never mutates. pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: Option<&str>) { @@ -219,17 +205,10 @@ pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: O .and_then(|k| wg_pubkey(&k).ok()); if self_signed { - // One anchor for the whole swarm, installed once per host — not - // a file per peer. That is the point of the hierarchy: a hive - // joining later needs no edit on the hives already running. - println!("# 1. install the SWARM ROOT on the peer host (once, not per hive):"); - println!("scp {SWARM_CA_ROOT_PATH} :{SWARM_CA_ROOT_PATH}"); - println!("# (skip if that host already has the swarm root)"); - println!(); let store_root = std::fs::read_to_string(STORE_PKI_ROOT_PATH).ok(); print!("{}", store_root_step(store_root.as_deref())); println!(); - println!("# 3. paste into the peer hive's NixOS config:"); + println!("# 2. paste into the peer hive's NixOS config:"); } else { println!("# paste into the peer hive's NixOS config:"); } diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 95351c5c..4dfac220 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -47,7 +47,6 @@ ./swarm-authelia.nix ./swarm-bao-service.nix ./swarm-bao.nix - ./swarm-ca.nix ./swarm-ci-runner.nix ./swarm-secret-publisher.nix ./swarm-nats-service.nix @@ -65,6 +64,7 @@ ./swarm-victoriametrics.nix ./swarm-wireguard.nix ./swarm.nix + ./swarm-ca-removed.nix ./swarm-peers-removed.nix ./swarm-required-services.nix ]; diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index 70eaad2e..0f2795b9 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -4,7 +4,7 @@ # different kinds of fact and only one of them varies per machine: # # swarm.* — swarm-wide truth. The swarm's name, domain, hives, peers, -# CA, and where each service lives. **Identical on every +# and where each service lives. **Identical on every # host**, byte for byte; a hive needs all of it to be a # *client* of the swarm. # deploy.* — this machine's deployment decisions. Necessarily different diff --git a/nix/host-modules/hive-c0re/default.nix b/nix/host-modules/hive-c0re/default.nix index 89f79de2..2a68c74a 100644 --- a/nix/host-modules/hive-c0re/default.nix +++ b/nix/host-modules/hive-c0re/default.nix @@ -128,10 +128,10 @@ in ./theme.nix # Hive-CA trust for this daemon's outbound TLS. The matrix provisioning # URL (`swarm.matrix.apiUrl`) defaults to https on the homeserver's - # gateway vhost, which serves a leaf chaining to the store's services - # root or the swarm root — CAs generated at runtime, which no - # build-time trust store can contain. The queue's TLS is verified - # against `HIVE_C0RE_OIDC_CA_FILE` (../hive-tls.nix), not this bundle. + # gateway vhost, which serves a leaf chaining to the store's pki root — + # a CA generated at runtime, which no build-time trust store can + # contain. The queue's TLS is verified against `HIVE_C0RE_OIDC_CA_FILE` + # (../hive-tls.nix), not this bundle. # # ⚠️ `SSL_CERT_FILE` REPLACES the trust store, and unlike the other # consumers of this helper, hive-c0re already makes a *public*-TLS call: diff --git a/nix/host-modules/hive-forge/service.nix b/nix/host-modules/hive-forge/service.nix index f5c650b4..bb7a4cd0 100644 --- a/nix/host-modules/hive-forge/service.nix +++ b/nix/host-modules/hive-forge/service.nix @@ -100,8 +100,8 @@ in ⚠️ To use the per-hive name `forge.''${services.hyperhive.domain}` instead, pin it here. Certificates follow either way: the - swarm-services sub-CA is name-constrained to the configured - names (see `./swarm-ca.nix`), not to a fixed tree. + store's `swarm-services` role issues for the configured names, + not for a fixed tree. Set to a full hostname (`git.example.com`, `forge.internal.lan`, etc.) for a bespoke vhost shape — the diff --git a/nix/host-modules/hive-gateway/default.nix b/nix/host-modules/hive-gateway/default.nix index 3ccaec6c..a5dadd5b 100644 --- a/nix/host-modules/hive-gateway/default.nix +++ b/nix/host-modules/hive-gateway/default.nix @@ -29,6 +29,16 @@ let # The hive leaf exists only where a hive runs; ../hive-tls.nix issues it # under that same condition. runsHive = config.services.hyperhive.deploy.hive-controller.enable; + # Whether the cert import `Requires=` the services leaf's issuing unit + # (../hive-tls.nix decides). Where it does, a missing leaf fails the + # import rather than anything standing in for it. + svcLeafRequired = lib.elem "hive-gateway-self-signed-cert.service" ( + config.systemd.services.swarm-services-cert.requiredBy or [ ] + ); + svcLeafMissing = '' + echo "no swarm-services leaf at ${tlsStateDir}/swarm-services.pem, and this gateway serves swarm names only with it." >&2 + echo "swarm-services-cert.service issues it from the secret store's pki mount; its journal says why it has not." >&2 + exit 1''; # Every vhost claiming `default_server`, ours and the operator's # alike. Computed once so the assertion below and the message it @@ -252,10 +262,9 @@ in # 🚨 DO NOT "simplify" this into pointing nginx at the CA dir. It does # TWO jobs: it re-modes the leaf to the 0640 root:nginx copy below (a # 0600 key fails nginx's pre-start config test and blocks the unit), - # and it guarantees every cert path the config names exists — which is - # what the swarm-services fallback at the bottom of the script is for. - # A leaf that never issues therefore takes the WHOLE gateway down, not - # one vhost. Both mechanisms, and what it cost the time this was + # and it puts a file at every cert path the config names or fails: one + # nginx cannot find takes the WHOLE gateway down, not one vhost. Both + # mechanisms, and what it cost the time this was # collapsed: `docs/networking/gateway.md::Self-signed TLS (default)`. # # ALWAYS runs (no ConditionPathExists) and is idempotent, because it @@ -307,49 +316,51 @@ in # because the hive leaf's role allows only its own hive's # domain, and the service names are siblings of it. # - # Absent is a normal state, not a failure: a host fronting - # no swarm name has none, and issuance can also fail on a - # host that wants one. - # - # ⚠️ When it is absent the HIVE leaf goes to this path - # anyway, and that fallback is load-bearing rather than - # tidy. nginx refuses to load a config naming a cert file - # that does not exist — `cannot load certificate … no such - # file` fails the pre-start test, so the vhost does not - # degrade, the ENTIRE gateway dies and takes the forge, the - # dashboard and matrix with it. Serving the hive leaf on a - # swarm-service name is a name mismatch: browsers warn, - # strict clients refuse, everything else keeps working, and - # the operator gets a bad cert instead of no hive. - # - # Measured, not theorised: this exact path took pr1ma's - # gateway down when the services sub-CA failed to issue. + # Where the gateway waits for it, a missing leaf fails this + # unit and nginx with it. Elsewhere — no swarm name here, or + # only the store's UI, whose host must start its passthrough + # with the store sealed — the hive leaf stands in, because + # nginx refuses a config naming a missing cert file, until + # `swarm-services-cert` issues one and re-runs this unit. if [ -s ${tlsStateDir}/swarm-services.pem ]; then install -m 0644 ${tlsStateDir}/swarm-services.pem ${svcCert} install -m 0640 -g nginx ${tlsStateDir}/swarm-services-key.pem ${svcKey} else - echo "no swarm-services leaf — serving the hive leaf on those names (mismatch, not an outage)" >&2 - install -m 0644 ${tlsStateDir}/gateway.pem ${svcCert} - install -m 0640 -g nginx ${tlsStateDir}/gateway-key.pem ${svcKey} + ${ + if svcLeafRequired then + svcLeafMissing + else + '' + echo "no swarm-services leaf yet — serving the hive leaf on those names until swarm-services-cert.service issues one" >&2 + install -m 0644 ${tlsStateDir}/gateway.pem ${svcCert} + install -m 0640 -g nginx ${tlsStateDir}/gateway-key.pem ${svcKey}'' + } fi'' else '' # No hive here, so no hive leaf: every vhost, the catch-all - # included, serves the swarm-services leaf. Until it lands a - # throwaway self-signed pair stands in, because nginx refuses - # a config naming a missing cert file. A store-only host does - # not wait for the leaf, so its stream passthrough comes up with - # the store sealed. `swarm-services-cert` re-runs this unit - # once the leaf lands. + # included, serves the swarm-services leaf. Where the gateway + # waits for it, a missing leaf fails this unit. A store-only + # host does not wait, so its stream passthrough comes up with + # the store sealed: a throwaway self-signed pair stands in, + # because nginx refuses a config naming a missing cert file, + # until `swarm-services-cert` issues the leaf and re-runs this + # unit. if [ -s ${tlsStateDir}/swarm-services.pem ] && [ -s ${tlsStateDir}/swarm-services-key.pem ]; then cert=${tlsStateDir}/swarm-services.pem key=${tlsStateDir}/swarm-services-key.pem else - echo "no swarm-services leaf yet — serving a throwaway self-signed certificate until it lands" >&2 - cert=${tlsDir}/fallback.pem - key=${tlsDir}/fallback-key.pem - (umask 077; openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days 30 \ - -subj "/CN=hyperhive-gateway-fallback" -keyout "$key" -out "$cert") + ${ + if svcLeafRequired then + svcLeafMissing + else + '' + echo "no swarm-services leaf yet — serving a throwaway self-signed certificate until it lands" >&2 + cert=${tlsDir}/fallback.pem + key=${tlsDir}/fallback-key.pem + (umask 077; openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days 30 \ + -subj "/CN=hyperhive-gateway-fallback" -keyout "$key" -out "$cert")'' + } fi install -m 0644 "$cert" ${tlsCert} install -m 0640 -g nginx "$key" ${tlsKey} diff --git a/nix/host-modules/hive-matrix.nix b/nix/host-modules/hive-matrix.nix index 756134ad..e79d4659 100644 --- a/nix/host-modules/hive-matrix.nix +++ b/nix/host-modules/hive-matrix.nix @@ -12,7 +12,7 @@ let hyperhiveDomain = config.services.hyperhive.domain; # Same runtime→build-time bridge hive-forge and swarm-ci-runner cross: - # binds the hive trust bundle (which folds in the swarm root) into the + # binds the hive trust bundle (which folds in the store's root) into the # container and orders the container after `hive-tls-ca.service`. The # assembled bundle itself comes from `caTrust.trustBundle`, imported in # the container config below. @@ -783,10 +783,9 @@ in # Swarm-internal trust reaches tuwunel at RUNTIME, not via # `security.pki.certificateFiles`. That option is read when the - # system is BUILT, and the swarm root is deliberately a runtime - # file (`swarm.ca.stateDir`) because its key must never enter the - # store — so there is nothing build-time to name. The bind-mount - # above plus the bundle service below are what replaced it. + # system is BUILT, and the store's root is a runtime file fetched + # from the store at boot — so there is nothing build-time to name. + # The bind-mount above plus the bundle service below carry it. # tuwunel hard-fails to boot if `/etc/resolv.conf` has no # `nameserver` line (`Failed to configure DNS resolver ... no diff --git a/nix/host-modules/hive-tls.nix b/nix/host-modules/hive-tls.nix index 49ee6d65..b31609d6 100644 --- a/nix/host-modules/hive-tls.nix +++ b/nix/host-modules/hive-tls.nix @@ -8,7 +8,6 @@ let cfg = config.services.hyperhive.deploy.hive-controller.tls; hyperhiveCfg = config.services.hyperhive; gatewayCfg = config.services.hyperhive.gateway; - swarmCaCfg = config.services.hyperhive.swarm.ca; domain = hyperhiveCfg.domain; baoCfg = hyperhiveCfg.swarm.bao; baoDeploy = hyperhiveCfg.deploy.bao; @@ -346,15 +345,6 @@ in }; config = lib.mkIf active { - # 🚫 There is deliberately NO warning that these service names are - # outside this hive's domain and this host cannot sign for them. The - # services leaf comes from the secret store's `pki` mount, which every - # hive reaches over the network with its own identity, so holding the - # swarm root key does not decide whether a hive can serve a - # swarm-service name. Such a warning would fire on every rebuild of - # every hive that is not the CA host, about a fallback that does not - # happen. - # The state dir and the trust bundle. Never waits on the store: the # store's container is ordered after this unit, which writes the # bundle it bind-mounts, so the leaf is `hive-tls-resign`'s. @@ -366,8 +356,6 @@ in # turn `Requires=`. Order against the unit that reads the dir. before = [ "hive-gateway-self-signed-cert.service" ]; requiredBy = [ "hive-gateway-self-signed-cert.service" ]; - # The bundle carries the swarm root where this host generates it. - # # `swarm-services-cert` is NOT ordered against this unit in either # direction, and the absence is load-bearing — see the cycle spelled # out at that unit. It runs after the store's container, the container @@ -381,7 +369,6 @@ in # "$svcroot" ]` guard is what makes that state legal rather than fatal, # and `swarm-services-cert` restarts this unit once the root it wrote # is new — so the bundle converges instead of being a boot behind. - after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ]; serviceConfig = { Type = "oneshot"; RemainAfterExit = true; @@ -396,7 +383,6 @@ in ca="$d/ca.pem" prev="$d/ca-previous.pem" - root=${lib.escapeShellArg "${swarmCaCfg.stateDir}/root.pem"} svcroot="$d/swarm-services-root.pem" # --- Trust bundle: what a consumer must TRUST. Why it is a set of @@ -423,7 +409,6 @@ in anchors=() if [ -s "$ca" ]; then anchors+=("$ca"); fi if [ -s "$prev" ]; then anchors+=("$prev"); fi - if [ -s "$root" ]; then anchors+=("$root"); fi if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi # `cat` with no arguments reads stdin, so no anchor at all is an # empty file until `swarm-services-cert` writes the root. @@ -834,10 +819,7 @@ in # flattening the chain: the issuer here is a self-signed root, so # appending it would ship an anchor a client either already holds # — in which case it is redundant — or does not, in which case - # shipping it proves nothing. The old file appended the services - # sub-CA because that WAS an intermediate, and a client anchored - # on the swarm root could not build the middle of the chain by - # itself. + # shipping it proves nothing. mv -f "$d/swarm-services-leaf.new" "$svcleaf" mv -f "$d/swarm-services-key.pem.new" "$d/swarm-services-key.pem" chmod 0644 "$svcleaf" diff --git a/nix/host-modules/local-defaults.nix b/nix/host-modules/local-defaults.nix index 5f91fac0..aebeb562 100644 --- a/nix/host-modules/local-defaults.nix +++ b/nix/host-modules/local-defaults.nix @@ -32,7 +32,6 @@ in toggles that an all-on-one-box deployment implies: the swarm's shared services (`services.hyperhive.deploy.allSwarmServices`), the swarm - CA (`services.hyperhive.swarm.ca.autoConfigure`), the swarm controller (`services.hyperhive.deploy.swarm-controller.enable`), and the host's `/etc/hosts` entries for the names this hive serves (`services.hyperhive.gateway.localHostsEntry`) — with no real DNS @@ -71,11 +70,8 @@ in # break every agent's access to the forge. config.services.hyperhive.gateway.localHostsEntry = lib.mkDefault cfg.deploy.singleHostSwarm; - # Out of the `swarm` attrset below, because it is a `deploy.*` option now - # (./deploy.nix): "does THIS host run the swarm's services" is a per-host - # decision. Written as a path rather than folded into a second - # `config.services.hyperhive.deploy = { … }` attrset, for the same reason - # the ⚠️ below gives about `swarm`. + # A `deploy.*` option (./deploy.nix): "does THIS host run the swarm's + # services" is a per-host decision. config.services.hyperhive.deploy.allSwarmServices = lib.mkDefault cfg.deploy.singleHostSwarm; # The queue's auth-callout nkeys. Generating them is safe exactly @@ -86,15 +82,6 @@ in config.services.hyperhive.deploy.nats.autoGenerateCallout = lib.mkDefault cfg.deploy.singleHostSwarm; - # ⚠️ Every `swarm.*` default this mode sets lives INSIDE this attrset, not - # as a second `config.services.hyperhive.swarm.…` path beside it — written - # that way the two definitions of `swarm` collide and the nested one is - # silently lost. The gate caught exactly that on the controller's queue URL, - # when this mode still set it: mode on, `natsUrl` still "". - config.services.hyperhive.swarm = { - ca.autoConfigure = lib.mkDefault cfg.deploy.singleHostSwarm; - }; - # The controller is asserted by the MODE and by nothing else. Its own # option stays `default = false` precisely because running it is a # statement about swarm topology — but "this box is the whole @@ -103,11 +90,8 @@ in # instead would be wrong: a hive in a larger swarm can legitimately # want the shared services without being the host that controls them. # - # Sits outside the `swarm` attrset above because it is a `deploy.*` - # option (./deploy.nix): "does THIS host run the controller" is exactly - # the per-host fact `swarm.*` may not carry. The ⚠️ collision note above - # does not apply here — that one is about two definitions of `swarm` - # itself, and this is a different top-level path. + # A `deploy.*` option (./deploy.nix): "does THIS host run the controller" + # is exactly the per-host fact `swarm.*` may not carry. config.services.hyperhive.deploy.swarm-controller.enable = lib.mkDefault cfg.deploy.singleHostSwarm; # Where the operator drops the token that writes the swarm's first bao @@ -122,8 +106,7 @@ in # substitute for the other: one unseals the store, this one authorises the # first write. # - # Outside the `swarm` attrset above for the same reason the controller's - # options are — a path on this host is a `deploy.*` fact. + # A path on this host is a `deploy.*` fact, like the controller's options. config.services.hyperhive.deploy.bao.bootstrapTokenFile = lib.mkIf cfg.deploy.singleHostSwarm ( lib.mkDefault "/var/lib/swarm-bao-bootstrap/grant.token" ); diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index b5ceb34b..05d21c90 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -597,17 +597,15 @@ let # # The mount's own root is generated into it by the bootstrap unit below and # its private key never leaves — `root/generate/internal` keeps it inside - # the store, which is the whole reason the mount exists rather than another - # key beside ./swarm-ca.nix's. + # the store. # # An option rather than a literal because ./hive-tls.nix issues through it # and has to spell it the same way. servicesPkiMountPath = baoDeploy.servicesPkiMountPath; # Subject of the root generated into that mount. A label for a human reading - # a chain, not an identity anything authenticates against — same fall-through - # ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set - # neither a swarm name nor a domain must still evaluate. + # a chain, not an identity anything authenticates against. Falls through so + # a hive that has set neither a swarm name nor a domain still evaluates. servicesPkiRootLabel = if hyperhiveCfg.swarm.name != null then hyperhiveCfg.swarm.name @@ -624,11 +622,9 @@ let servicesPkiRootDir = "/var/lib/swarm-bao-services-pki"; servicesPkiRootFile = "${servicesPkiRootDir}/services-root.pem"; - # ~30y, matching ./swarm-ca.nix's root rather than its services sub-CA. This - # is an ANCHOR: an operator adds it to a browser's trust store by hand, and - # every rotation is a manual re-trust on every machine that holds it. The - # sub-CA's short window bought cheap rotation because the root above it - # absorbed the churn; nothing sits above this one. + # ~30y. This is an ANCHOR: an operator adds it to a browser's trust store by + # hand, and every rotation is a manual re-trust on every machine that holds + # it; nothing sits above this one to absorb the churn. servicesPkiRootTtl = "262800h"; # The leaf's window, pinned on the role rather than left to the mount's @@ -645,16 +641,16 @@ let # The role every swarm-service certificate will be issued through, and the # whole narrowing of this mount. A role rather than a name-constrained - # sub-CA: the engine enforces the same "these names and no others" at issue - # time that ./swarm-ca.nix builds into x509 `nameConstraints` today, so a - # second CA in the chain would re-implement a check that already happens. + # sub-CA: the engine enforces "these names and no others" at issue time, so + # x509 `nameConstraints` on a second CA in the chain would re-implement a + # check that already happens. servicesPkiRoleName = baoDeploy.servicesPkiRoleName; # The names that role may issue for. Read out of the swarm-tier option - # rather than assembled here, exactly as ./swarm-ca.nix:45 and - # ./hive-tls.nix:16 read it — the thing that narrows what may be issued and - # the leaf that carries those names as SANs must agree, and a second module - # composing its own list is how they stop agreeing. + # rather than assembled here, exactly as ./hive-tls.nix reads it — the thing + # that narrows what may be issued and the leaf that carries those names as + # SANs must agree, and a second module composing its own list is how they + # stop agreeing. swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains; # The identity that asks this mount for a certificate. A FOURTH principal @@ -1374,7 +1370,7 @@ in clientCaFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; - example = "/var/lib/swarm-ca/root.pem"; + example = "/var/lib/swarm-bao-pki/ca.pem"; description = '' Authority the store validates hive **client** certificates against. @@ -1385,7 +1381,7 @@ in On a hive that runs the store, a glue module supplies the CA it minted for exactly this, as a `mkDefault`. Point this at something - else — the swarm root, an operator's own CA — and yours wins. + else — an operator's own CA — and yours wins. `null` leaves client-certificate verification off, which is only appropriate where something else authenticates the connection. @@ -2806,21 +2802,18 @@ in # # Every narrowing is load-bearing, and three of them turn OFF a # default that is on: - # * `allow_subdomains` — the x509 constraint ./swarm-ca.nix writes - # permits a whole subtree per name; a swarm service is an exact - # hostname, so the role is the narrower of the two. + # * `allow_subdomains` — a swarm service is an exact hostname, not + # a subtree. # * `allow_localhost` — bao's own default is `true`, which would # make this role issue for a name nobody configured. - # * `allow_ip_sans` — the sub-CA excludes both IP subtrees - # explicitly (./swarm-ca.nix:260-276); an unconstrained name type - # is a name type this issuer is unconstrained for. + # * `allow_ip_sans` — an unconstrained name type is a name type + # this issuer is unconstrained for. # `server_flag` alone because that is what today's leaf carries # (`extendedKeyUsage=serverAuth`, ./hive-tls.nix:144), and rsa:4096 # because that is the key the current implementation generates. # # An empty `swarm.serviceDomains` renders an empty `allowed_domains`, - # which with `allow_any_name=false` is a role that issues nothing — - # the same outcome as ./swarm-ca.nix's "nothing to issue for" exit, + # which with `allow_any_name=false` is a role that issues nothing, # reached without a branch. bao write ${lib.escapeShellArg "${servicesPkiMountPath}/roles/${servicesPkiRoleName}"} \ allowed_domains=${lib.escapeShellArg (lib.concatStringsSep "," swarmServiceDomains)} \ diff --git a/nix/host-modules/swarm-ca-removed.nix b/nix/host-modules/swarm-ca-removed.nix new file mode 100644 index 00000000..b09e65c2 --- /dev/null +++ b/nix/host-modules/swarm-ca-removed.nix @@ -0,0 +1,33 @@ +# Removal shim for `services.hyperhive.swarm.ca.*` and the host-held swarm +# root it generated. +# +# The secret store's `pki` root is the only trust anchor: it issues every +# hive's gateway leaf and the swarm's service names, and its key never leaves +# the store. Nothing generates, reads or bundles a root under +# `/var/lib/swarm-ca`, so a definition of these options fails evaluation with +# a message, and the directory is deleted on every host. +# +# Self-contained and deletable, like ./swarm-peers-removed.nix: one file to +# remove when the deprecation window closes. +{ lib, ... }: +let + removed = + name: + lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "ca" name ] '' + The host-held swarm root CA is gone: the secret store's pki root is the + only trust anchor, and nothing generates or reads a root under + /var/lib/swarm-ca. Remove this definition. See docs/swarm/ca.md. + ''; +in +{ + imports = map removed [ + "autoConfigure" + "stateDir" + "validityDays" + ]; + + # The root's private key is a secret outside the store, and the + # certificate beside it is an anchor nothing issues under. The option's + # default path only: a relocated `stateDir` is the operator's to delete. + systemd.tmpfiles.rules = [ "R /var/lib/swarm-ca" ]; +} diff --git a/nix/host-modules/swarm-ca.nix b/nix/host-modules/swarm-ca.nix deleted file mode 100644 index f30c38a5..00000000 --- a/nix/host-modules/swarm-ca.nix +++ /dev/null @@ -1,161 +0,0 @@ -# The swarm root CA: the anchor a whole swarm shares, and the issuer of -# each hive's own CA (which is where it gets used — see ./hive-tls.nix). -# -# One root rather than per-peer pinning, because cross-hive trust is then -# O(1): trust the root once and every present *and future* peer validates, -# instead of every hive having to name every other one. -# -# Two provisioning modes share ONE structure — what differs is who puts -# the artifacts on disk, never what the artifacts are. Which mode does -# what, and what moving between them costs: -# `docs/swarm/ca.md::Two provisioning modes, one structure`. -# -# ⚠️ Root key AND root cert are runtime files, never nix options. The key -# for the obvious reason; the cert as a consequence, and that one costs -# something real — nothing whose trust store is built at build time can -# name it: `docs/swarm/ca.md::Distributing the root`. -# -# ⚠️ This root does not issue the swarm's SERVICE certificates; there is no -# `swarm-services-ca` sub-CA under it. Those come out of the secret store's -# own `pki` mount, anchored on a root generated inside it (./swarm-bao.nix), -# and ./hive-tls.nix asks for them over the network instead of signing them -# here. This root has one job: the anchor each hive's own CA is issued under. -{ - lib, - config, - pkgs, - ... -}: -let - cfg = config.services.hyperhive.swarm.ca; - hyperhiveCfg = config.services.hyperhive; - - # The subject CN is a label for a human reading a chain, not an - # identity anything authenticates against. Fall through swarm name → - # hive domain → a constant so a hive that has set neither still - # evaluates; a missing `domain` is reported by its own assertion in - # hive-network.nix, and shouldn't also surface here as a null. - swarmLabel = - if hyperhiveCfg.swarm.name != null then - hyperhiveCfg.swarm.name - else if hyperhiveCfg.domain != null then - hyperhiveCfg.domain - else - "hyperhive"; -in -{ - options.services.hyperhive.swarm.ca = { - autoConfigure = lib.mkOption { - type = lib.types.bool; - default = false; - example = true; - description = '' - Run the whole swarm CA on this one host: generate the swarm - root when it is missing, and issue this hive's CA under it. - - `services.hyperhive.deploy.singleHostSwarm` turns this on as - part of the all-on-one-box mode. Set it here directly to run the - CA on a host that is not otherwise all-local. - - **Off by default, deliberately.** A swarm's services and its - hives can live on different hosts, and this host has no way to - tell whether it is the one holding the root — so the swarm CA - is something an operator sets up, not something a host decides - it is. Turn this on for an all-on-one-host deployment (dev - boxes, single-hive swarms) and get the hierarchy for free. - - With it off, both artifacts are operator-provided: the root - under `stateDir`, and this hive's CA under - `services.hyperhive.deploy.hive-controller.tls.stateDir`. A hive - given neither keeps the self-signed CA it has always had — it - simply isn't part of a swarm's trust hierarchy, which is the - correct outcome for a hive nobody has federated yet. - ''; - }; - - stateDir = lib.mkOption { - type = lib.types.str; - default = "/var/lib/swarm-ca"; - description = '' - Host directory holding the swarm root CA: `root.pem` (the - anchor, safe to distribute — copy it to this same path on every - other host in the swarm) and `root-key.pem` - (0600, the one file that must never reach the nix store or - another host). The directory itself is 0700: nothing reads - out of it but the hive CA issuance in `hive-tls.nix`. - - Moving the swarm CA to its own host is a matter of moving this - directory and setting `autoConfigure = false` here. - ''; - }; - - validityDays = lib.mkOption { - type = lib.types.int; - default = 10950; - description = '' - Validity window of the swarm root CA in days (default ~30y). - Deliberately long: the root must outlive the hive CAs it issued, or those chains - expire out from under hives that are still perfectly happy with - their own intermediate. Rotating a root is the one operation in - this system with no partial-failure mode — it invalidates every - peer at once, paced by the slowest peer's rebuild — so it is - never automatic and this window is meant to be uneventful. - ''; - }; - }; - - config = lib.mkIf cfg.autoConfigure { - systemd.services.swarm-ca = { - description = "Generate the swarm root CA when absent"; - wantedBy = [ "multi-user.target" ]; - path = [ pkgs.openssl ]; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - UMask = "0077"; - # Pin the journal identity (else it's the `script` store-path wrapper). - SyslogIdentifier = "swarm-ca"; - }; - script = '' - set -euo pipefail - d=${lib.escapeShellArg cfg.stateDir} - install -d -m 0700 "$d" - - root="$d/root.pem" - rootk="$d/root-key.pem" - - # Note the asymmetry with the hive CA in hive-tls.nix, which - # regenerates itself once expired: a root is never replaced - # automatically, not even an expired one. Consumers hold this - # cert, so replacing it is a swarm-wide flag day that wants an - # operator running it deliberately, with both roots trusted - # across the overlap. - if [ -s "$root" ] && [ -s "$rootk" ]; then - echo "swarm root CA already present at $root — leaving it alone" - exit 0 - fi - - # Half a root is not a root. Generating a fresh key beside an - # already-distributed cert (or the reverse) leaves every - # consumer trusting an anchor that no longer signs anything — - # and it would look like it worked. - if [ -e "$root" ] || [ -e "$rootk" ]; then - echo "swarm root CA half-provisioned ($root / $rootk) — refusing to generate over it" >&2 - exit 1 - fi - - echo "generating swarm root CA at $root" - # pathlen:1 — the root signs hive CAs, which sign leaves. One - # intermediate below the root and no deeper. - openssl req -x509 -newkey rsa:4096 -nodes -sha256 \ - -days ${toString cfg.validityDays} \ - -keyout "$rootk" -out "$root" \ - -subj "/CN=swarm-ca ${swarmLabel}" \ - -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \ - -addext "keyUsage=critical,keyCertSign,cRLSign" - chmod 0600 "$rootk" - chmod 0644 "$root" - ''; - }; - }; -} diff --git a/nix/host-modules/swarm-controller.nix b/nix/host-modules/swarm-controller.nix index 4c2b716e..d7cc0c3a 100644 --- a/nix/host-modules/swarm-controller.nix +++ b/nix/host-modules/swarm-controller.nix @@ -669,7 +669,7 @@ in hiveClientCaFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; - example = "/var/lib/swarm-ca/root.pem"; + example = "/var/lib/swarm-bao-pki/ca.pem"; description = '' Authority whose leaves are hive client certificates, so that this daemon can create each hive's cert-auth role. The store matches a diff --git a/nix/host-modules/swarm-peers-removed.nix b/nix/host-modules/swarm-peers-removed.nix index dc0b1b36..4e3a1560 100644 --- a/nix/host-modules/swarm-peers-removed.nix +++ b/nix/host-modules/swarm-peers-removed.nix @@ -57,11 +57,10 @@ in services.hyperhive.swarm.peers..caCert is removed and ignored, and has no replacement in services.hyperhive.swarm.hives. - Trust inside a swarm now derives from the swarm root CA - (services.hyperhive.swarm.ca — see docs/swarm/ca.md): every hive - under it chains to it, so a per-hive CA is dead weight. What this - genuinely drops is trusting a hive whose root this swarm does NOT - own — another swarm's, or one keeping its own CA. + Trust inside a swarm derives from the secret store's pki root (see + docs/swarm/ca.md): every hive chains to it, so a per-hive CA is dead + weight. What this genuinely drops is trusting a hive whose root this + swarm does NOT own — another swarm's, or one keeping its own CA. Still set on: ${lib.concatStringsSep ", " withCaCert} ''; diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 2051f116..9a719fa6 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -72,13 +72,13 @@ let # `serviceDomains` below is this set sorted + deduplicated. # # ⚠️ These are NOT required to be under `swarm.domain`. An earlier - # revision asserted that, reasoning that the services sub-CA is - # constrained to the swarm's tree — but the sub-CA is constrained to - # the **configured names** (./swarm-ca.nix) and the swarm root carries - # no name constraints at all, so any configured name is issuable. The - # assertion encoded an intended shape, not a property of the code, and - # it rejected the supported migration path: a hive pinning its old - # `forge.` while joining a swarm at a different apex. + # revision asserted that, reasoning that the services issuer is + # constrained to the swarm's tree — but the store's `swarm-services` + # role admits exactly the **configured names** (./swarm-bao.nix), so any + # configured name is issuable. The assertion encoded an intended shape, + # not a property of the code, and it rejected the supported migration + # path: a hive pinning its old `forge.` while joining a swarm + # at a different apex. # # Every name is listed whether or not this host runs the service: the # store's issuing role is built from this list on the store's host and @@ -226,9 +226,9 @@ in visible = false; internal = true; description = '' - Removed. Trust inside a swarm comes from the swarm root CA - (`services.hyperhive.swarm.ca`, docs/swarm/ca.md), which - replaces per-hive leaf pinning entirely. + Removed. Trust inside a swarm comes from the secret store's + pki root (docs/swarm/ca.md), which replaces per-hive leaf + pinning entirely. ''; }; }; @@ -369,9 +369,9 @@ in existed to serve — nothing else ever consumed it. There is no replacement, and none is needed for a hive inside - this swarm: trust comes from the swarm root CA - (services.hyperhive.swarm.ca — see docs/swarm/ca.md), which - every hive chains to, so one anchor replaces per-hive pinning. + this swarm: trust comes from the secret store's pki root (see + docs/swarm/ca.md), which every hive chains to, so one anchor + replaces per-hive pinning. What that genuinely drops is trusting a hive whose root this swarm does NOT own — another swarm's, or one keeping its own CA. That is a cross-swarm problem and wants a mechanism diff --git a/nix/module-eval/hive-tls.nix b/nix/module-eval/hive-tls.nix index 08775278..cf544b9b 100644 --- a/nix/module-eval/hive-tls.nix +++ b/nix/module-eval/hive-tls.nix @@ -4,7 +4,8 @@ # The swarm-services leaf's renewal: a timer that really re-runs the # issuance, at a threshold read off the store role's lifetime. And the # store's root on a hive that asks it for no leaf. And the gateway leaf, -# issued by the store through the hive's own role, with no hive CA. +# issued by the store through the hive's own role, with no hive CA and no +# host-held swarm root. { pkgs, lib, @@ -21,6 +22,7 @@ let nixosSystem ; }) + hive hiveWithController runGroup ; @@ -208,10 +210,9 @@ let !(holds noService) && holds allLocal; } { - # The controls are the store's PKI unit, which still mints its own CA - # with `openssl req`, and the swarm root's unit, which still reads - # `root-key.pem`: absence here is not a matcher that cannot match. - name = "no hive TLS unit generates a CA, signs with a CA key or reads the swarm root key"; + # The control is the store's PKI unit, which still mints its own CA + # with `openssl req`: absence here is not a matcher that cannot match. + name = "no hive TLS unit generates a CA or signs with a CA key"; ok = let hasCaWork = @@ -219,16 +220,106 @@ let lib.any (p: lib.hasInfix p s) [ "openssl req" "-CAkey" - "root-key.pem" ]; pkiScript = units.swarm-bao-pki.script; - rootScript = units.swarm-ca.script; in !(hasCaWork caUnit.script) && !(hasCaWork leafUnit.script) && !(hasCaWork noService.systemd.services.hive-tls-ca.script) - && lib.hasInfix "openssl req" pkiScript - && lib.hasInfix "root-key.pem" rootScript; + && lib.hasInfix "openssl req" pkiScript; + } + { + # Every unit's script, environment and ordering, on the all-local host + # and on a hive that runs no store. The control is the same scan finding + # the store root's file, so an empty result is not a scan that reads + # nothing. + name = "no unit generates, reads or waits on a host-held swarm root, and every host deletes its directory"; + ok = + let + texts = + m: + lib.concatMap (u: [ + (u.script or "") + (builtins.toJSON (u.environment or { })) + (toString ((u.after or [ ]) ++ (u.requires or [ ]) ++ (u.wants or [ ]))) + ]) (lib.attrValues m.systemd.services); + mentions = m: needle: lib.any (lib.hasInfix needle) (texts m); + gone = + m: + !(m.systemd.services ? swarm-ca) + && !(mentions m "swarm-ca.service") + && !(mentions m "/var/lib/swarm-ca") + && !(mentions m "root-key.pem") + && lib.elem "R /var/lib/swarm-ca" m.systemd.tmpfiles.rules; + in + gone allLocal + && gone noService + && mentions allLocal "swarm-services-root.pem" + && mentions noService "swarm-services-root.pem"; + } + { + # Three slots and no fourth: a root slot beside them is an anchor + # nothing issues under. + name = "the trust bundle anchors on the store's root and a hive CA's public certificates only"; + ok = + let + slots = lib.filter (l: lib.hasPrefix "if [ -s " l && lib.hasInfix "anchors+=(" l) ( + lib.splitString "\n" caUnit.script + ); + in + lib.length slots == 3 + && lib.hasInfix ''if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi'' caUnit.script + && !(lib.hasInfix ''anchors+=("$root")'' caUnit.script); + } + { + # The control is the all-local host, which defines none of them. + name = "a definition of a removed swarm.ca option fails evaluation, naming the option and the store's root"; + ok = + let + refused = + m: opt: + lib.any ( + a: + !a.assertion + && lib.hasInfix "services.hyperhive.swarm.ca.${opt}" a.message + && lib.hasInfix "pki root is the" a.message + ) m.assertions; + setting = { + autoConfigure = hive { swarm.ca.autoConfigure = true; }; + stateDir = hive { swarm.ca.stateDir = "/var/lib/swarm-ca"; }; + validityDays = hive { swarm.ca.validityDays = 10950; }; + }; + in + lib.all (opt: refused setting.${opt} opt) (lib.attrNames setting) + && !(lib.any (opt: refused allLocal opt) (lib.attrNames setting)); + } + { + # Where the import `Requires=` the services leaf's issuing unit, a + # missing leaf fails it; elsewhere the hive leaf stands in. The control + # is a hive fronting no swarm name, which does not wait. + name = "a hive gateway that waits for the services leaf fails its cert import without it, naming the issuing unit"; + ok = + let + imp = m: m.systemd.services.hive-gateway-self-signed-cert.script; + waits = + m: + lib.elem "hive-gateway-self-signed-cert.service" m.systemd.services.swarm-services-cert.requiredBy; + standIn = "install -m 0644 /var/lib/hive-tls/gateway.pem /var/lib/hive-gateway/tls/swarm-services.pem"; + refusal = "swarm-services-cert.service issues it"; + # The `else` branch alone, up to the `fi` that closes it. + missingBranch = + m: + builtins.head ( + lib.splitString "\nfi" (lib.last (lib.splitString "swarm-services.pem ]; then" (imp m))) + ); + in + waits allLocal + && lib.hasInfix refusal (missingBranch allLocal) + && lib.hasSuffix "exit 1" (lib.removeSuffix "\n" (lib.trim (missingBranch allLocal))) + && !(lib.hasInfix standIn (imp allLocal)) + && !(waits noService) + && lib.hasInfix standIn (missingBranch noService) + && !(lib.hasInfix refusal (imp noService)); } { # Nothing reads or removes a `ca-key.pem`, so a rollback finds the diff --git a/nix/module-eval/swarm-services-switch.nix b/nix/module-eval/swarm-services-switch.nix index f46d1cd8..77a1a772 100644 --- a/nix/module-eval/swarm-services-switch.nix +++ b/nix/module-eval/swarm-services-switch.nix @@ -287,30 +287,50 @@ let lib.elem "hive-gateway-self-signed-cert.service" s.swarm-services-cert.requiredBy && lib.elem "hive-gateway-self-signed-cert.service" s.swarm-services-cert.before; # The cert it installs on the default and swarm-services paths: - # the services leaf when present, a throwaway pair otherwise. - importsServicesLeafOrFallback = + # the services leaf when present. + importsServicesLeaf = m: let imp = m.systemd.services.hive-gateway-self-signed-cert; in lib.hasInfix "cert=/var/lib/hive-tls/swarm-services.pem" imp.script - && lib.hasInfix "openssl req -x509" imp.script - && lib.hasInfix "cert=/var/lib/hive-gateway/tls/fallback.pem" imp.script && lib.hasInfix ''install -m 0644 "$cert" /var/lib/hive-gateway/tls/cert.pem'' imp.script - && lib.hasInfix ''install -m 0644 "$cert" /var/lib/hive-gateway/tls/swarm-services.pem'' imp.script + && lib.hasInfix ''install -m 0644 "$cert" /var/lib/hive-gateway/tls/swarm-services.pem'' imp.script; + # Without it: a throwaway pair where the gateway does not wait for + # the leaf, a failure naming the issuing unit where it does. + standsInThrowaway = + m: + let + imp = m.systemd.services.hive-gateway-self-signed-cert; + in + lib.hasInfix "openssl req -x509" imp.script + && lib.hasInfix "cert=/var/lib/hive-gateway/tls/fallback.pem" imp.script && lib.any (p: lib.hasPrefix "openssl" (p.pname or (p.name or ""))) imp.path; + failsWithoutLeaf = + m: + let + s = m.systemd.services.hive-gateway-self-signed-cert.script; + in + lib.hasInfix "swarm-services-cert.service issues it" s + && lib.hasInfix "exit 1" s + && !(lib.hasInfix "fallback.pem" s); store = storeOnlyNoHive; hs = swarmServicesHere.systemd.services; in noHive swarmServicesOnly && noHive swarmServicesOnlyNoStore && noHive store - && importsServicesLeafOrFallback swarmServicesOnly - && importsServicesLeafOrFallback swarmServicesOnlyNoStore - && importsServicesLeafOrFallback store - # Fronting many swarm names: the gateway waits for their leaf. + && importsServicesLeaf swarmServicesOnly + && importsServicesLeaf swarmServicesOnlyNoStore + && importsServicesLeaf store + # Fronting many swarm names: the gateway waits for their leaf, and + # its import fails without it. && waitsForLeaf swarmServicesOnly && waitsForLeaf swarmServicesOnlyNoStore + && failsWithoutLeaf swarmServicesOnly + && failsWithoutLeaf swarmServicesOnlyNoStore + && standsInThrowaway store + && !(failsWithoutLeaf store) # The store's UI alone: its nginx carries the passthrough to the # store, so it starts on the fallback with the store sealed. &&