diff --git a/docs/swarm/ca.md b/docs/swarm/ca.md index 0778d1d1..839453ca 100644 --- a/docs/swarm/ca.md +++ b/docs/swarm/ca.md @@ -37,26 +37,42 @@ Moving the swarm CA onto its own host is then a matter of moving `services.hyperhive.swarm.ca.stateDir` and leaving `autoConfigure` off — there is no second code path to switch to. -### The warning about uncovered service names +### The swarm's service names come from the secret store The swarm's service names (`swarm.forge.domain`, `swarm.matrix.gatewayHost`, `swarm.authelia.domain`) default to siblings of the hive domain — `forge.`, not `forge.`. The hive CA's leaf is a **single-label** wildcard -over the hive's own domain, so it can't cover them; only the -swarm-services leaf can, and this host signs that one only under -`autoConfigure`. A hive with neither serves the hive leaf on those names -and every client sees a name mismatch. +over the hive's own domain, so it can't cover them. -`hive-tls` emits an eval-time **warning** naming the uncovered names. -It's deliberately not an assertion: this module can see what _it_ is -able to issue, but not an operator-installed sub-CA in -`swarm.ca.stateDir`, an external ACME setup, or a certificate delivered -by any other means. If you have arranged one, expect the warning and -ignore it. Otherwise either install the sub-CA, or pin the -names back under the hive domain — a supported migration, since the -sub-CA carries a name constraint on the _configured_ names and the -swarm root carries no name constraints at all. +That leaf is issued by the secret store's `pki` mount instead, from a +root generated inside it. `swarm-services-cert.service` logs in with +this host's services-issuer certificate, calls +`pki/issue/swarm-services`, and writes the result to +`/var/lib/hive-tls/swarm-services.pem` — the path the gateway already +copies from. Every hive does this with its own identity, so holding the +swarm root's private key stopped being what decides whether a hive can +serve its swarm's names. + +The role's `allowed_domains` is read from the same +`services.hyperhive.swarm.serviceDomains` the SANs are built from, so +asking for a name nobody configured is a refusal from the store naming +that name — not a certificate quietly issued for it. + +**The root's public certificate is a file, on every hive:** +`/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the +leaf it signed and folded into `trust-bundle.pem`. On the host running +the store it is also at +`/var/lib/swarm-bao-services-pki/services-root.pem`. That is the file to +hand a browser, and no store login is needed to read it — which matters, +because every store listener demands a client certificate. + +**The root is generated once and never again.** The granting unit asks +the mount whether it already has an issuer (`bao list pki/issuers`) +before calling `pki/root/generate/internal`, so a rebuild or a reboot +re-asserts the role and the grant without touching the anchor. A root +that changed per boot would invalidate every certificate issued under it +and every browser taught to trust it. ## Constraints on the material diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index b5f3c38a..65facb85 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -45,7 +45,7 @@ neither is a renaming of the other. | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | swarm root CA cert | `swarm-ca.nix` first-boot unit, when the operator sets `autoConfigure` | `/var/lib/swarm-ca/root.pem` | operator copies the **cert** in; it's public | | swarm root CA key | same unit | `/var/lib/swarm-ca/root-key.pem`, `0600` | stays on whichever host holds it — see the constraint below | -| swarm-services sub-CA (cert + key) | `swarm-ca.nix`, signed by the root | `/var/lib/swarm-ca/services-ca{,-key}.pem` | issued where the root lives | +| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** is published at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | diff --git a/docs/swarm/ui.md b/docs/swarm/ui.md index d054220d..ed5c0009 100644 --- a/docs/swarm/ui.md +++ b/docs/swarm/ui.md @@ -91,12 +91,12 @@ own options, the way `swarm-ui.nix` and `swarm-authelia.nix` do. ⚠️ The certificate one is the hardest to predict and the most visible when -missed. `serviceDomains` is _both_ the services sub-CA's -`nameConstraints` set and the leaf's SAN list, and the apex is a -**sibling** of `forge.` / `chat.` / `auth.`, not a -parent — no CA in the hierarchy issues for it implicitly. Left out, the -vhost falls back to the hive leaf and the swarm's front page opens with a -name mismatch. +missed. `serviceDomains` is _both_ the `allowed_domains` the secret +store's `pki/roles/swarm-services` narrows to and the leaf's SAN list, +and the apex is a **sibling** of `forge.` / `chat.` / +`auth.`, not a parent — no CA in the hierarchy issues for it +implicitly. Left out, the vhost falls back to the hive leaf and the +swarm's front page opens with a name mismatch. ## Quick links diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 01030d24..25755fdb 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -30,6 +30,7 @@ ./glue-matrix-ctl-bao-identity.nix ./glue-queue-agent-credential.nix ./glue-secret-publisher-bao-identity.nix + ./glue-services-issuer-bao-identity.nix ./glue-swarm-bao-otel-oidc-client.nix ./glue-swarm-otel-oidc-client.nix ./swarm-authelia.nix diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix index 1afaed01..a0e68661 100644 --- a/nix/host-modules/glue-bao-tls.nix +++ b/nix/host-modules/glue-bao-tls.nix @@ -208,6 +208,21 @@ in [ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \ ${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth + + # The identity a hive presents to ask the store's `pki` mount for the + # swarm-services certificate its gateway serves. Minted here like the + # three above, and the reason is the sharpest of the four: this leaf + # is what OPENS the mint, so it cannot come out of it. A certificate + # authority that issues the credential admitting you to it is the + # cycle this file's header exists to keep out — which is exactly why + # the swarm-services root moved into the store and this one did not. + # + # Its own CN, not the reader's: the hive's own leaf reads every secret + # its policy names, while this principal may do one thing — `update` + # on `pki/issue/swarm-services` — and giving a renewal unit the wider + # credential would undo the separation the role was created for. + [ -s ${pkiDir}/services-issuer.pem ] || ${signLeaf} ${pkiDir} services-issuer \ + ${lib.escapeShellArg deployCfg.bao.servicesIssuerCommonName} "" clientAuth ''; }; }; diff --git a/nix/host-modules/glue-services-issuer-bao-identity.nix b/nix/host-modules/glue-services-issuer-bao-identity.nix new file mode 100644 index 00000000..44298c0e --- /dev/null +++ b/nix/host-modules/glue-services-issuer-bao-identity.nix @@ -0,0 +1,43 @@ +# Glue: point the swarm-services certificate unit at the bao leaf minted for +# it. +# +# ONE PAIRING PER FILE — the services-issuer principal ← bao, and nothing +# else. Deleting this leaves a hive that takes an operator-provided path to +# that credential, which is what any deployment not minting its own already +# does. +# +# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the +# leaf, because the thing that owns a private key owns issuing from it. What +# belongs here is the pairing: which paths `swarm-services-cert` reads. +# +# ⚠️ Gated on the leaf existing, not on the store running here — the same rule +# ./glue-controller-bao-identity.nix states, and it matters more for this +# principal than for any of its siblings: every hive in a swarm serves the +# swarm's service names and therefore needs this certificate, while exactly +# one of them hosts the store. +# +# Everything is `mkDefault`. An operator naming their own paths wins. +{ + lib, + config, + ... +}: +let + hyperhiveCfg = config.services.hyperhive; + deployCfg = hyperhiveCfg.deploy; + baoDeploy = deployCfg.bao; + + # Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own + # path rather than repeating that file's directory literal: an operator who + # moves the PKI moves both, and the two cannot drift apart. + haveMintedPki = baoDeploy.clientCertFile != null; + pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null; +in +{ + config = lib.mkIf (hyperhiveCfg.enable && haveMintedPki) { + services.hyperhive.deploy.hive-controller.tls = { + baoClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem"; + baoClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem"; + }; + }; +} diff --git a/nix/host-modules/hive-tls.nix b/nix/host-modules/hive-tls.nix index 2ebe6b67..28b1f133 100644 --- a/nix/host-modules/hive-tls.nix +++ b/nix/host-modules/hive-tls.nix @@ -10,26 +10,22 @@ let gatewayCfg = config.services.hyperhive.gateway; swarmCaCfg = config.services.hyperhive.swarm.ca; domain = hyperhiveCfg.domain; - # Derived once in ./swarm.nix and read here + in ./swarm-ca.nix, so - # the names this leaf carries as SANs and the names the sub-CA is - # constrained to cannot disagree. + baoCfg = hyperhiveCfg.swarm.bao; + baoDeploy = hyperhiveCfg.deploy.bao; + + # The endpoint the services leaf comes out of. Read off the store's own + # options rather than spelled here: ./swarm-bao.nix enables the mount, + # generates its root and writes the grant that names this exact path, and a + # second module composing its own is how the grant and the caller stop + # agreeing. + baoServicesPkiMount = baoDeploy.servicesPkiMountPath; + baoServicesPkiRole = baoDeploy.servicesPkiRoleName; + + # Derived once in ./swarm.nix and read here + in ./swarm-bao.nix, so + # the names this leaf carries as SANs and the names the store's + # `pki/roles/swarm-services` is narrowed to cannot disagree. swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains; - # True for the names `signHiveLeaf` below actually covers: the hive - # domain itself, or ONE label under it. `*.` is a single-label - # wildcard — `a.b.` does not match it — so the depth check is - # the whole point rather than a nicety. - coveredByHiveLeaf = - n: - n == domain - || (lib.hasSuffix ".${domain}" n && !lib.hasInfix "." (lib.removeSuffix ".${domain}" n)); - - # Service names this host can serve a *matching* certificate for, and - # the ones it can't. A name is issuable here when the hive leaf covers - # it, or when this host issues the swarm-services leaf — which needs - # the swarm root's private key, i.e. `swarm.ca.autoConfigure`. - uncoveredServiceDomains = lib.filter (n: !coveredByHiveLeaf n) swarmServiceDomains; - # The host-managed hive CA is the trust anchor for self-signed mode. # It is only stood up when the gateway actually serves a self-signed # cert: the gateway must be in self-signed mode. `domain` is required @@ -182,57 +178,38 @@ let "$d/ca.pem" "$d/ca-key.pem" ''; - # The swarm-services leaf: signed by the services sub-CA, covering the - # swarm's service names. Those are *siblings* of the hive domain, not - # children, so the hive CA is name-constrained out of them and cannot - # sign this however its SAN list is written. - # - # Skipped when the sub-CA isn't on disk: it only exists where the - # swarm CA is autoconfigured, and a hive that gets its certs from its - # operator has nothing for this to do. - signServicesLeaf = lib.optionalString (swarmServiceDomains != [ ]) '' - servicesCa=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"} - servicesCaKey=${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca-key.pem"} - if [ -s "$servicesCa" ] && [ -s "$servicesCaKey" ]; then - ${signLeafScript} "$d" swarm-services \ - ${lib.escapeShellArg (builtins.head swarmServiceDomains)} \ - ${lib.escapeShellArg (lib.concatMapStringsSep "," (n: "DNS:${n}") swarmServiceDomains)} \ - "$servicesCa" "$servicesCaKey" - else - echo "no swarm-services sub-CA at $servicesCa — skipping the services leaf" - fi - ''; - - # Shared by BOTH units that decide whether to re-sign. It lives here - # rather than in one of them because the two guards have to agree: they - # answer the same question at different times (`hive-tls-ca` at service - # activation, i.e. on the deploy; `hive-tls-resign` from a weekly timer - # for a host that stays up long enough to drift). A rule implemented in - # one and not the other is worse than one implemented in neither — it - # looks fixed and only fires on whichever path you did not take, which - # is exactly how a corrected `serviceDomains` still served a stale leaf. + # Shared by every unit that decides whether to re-issue a leaf. It + # lives here rather than in one of them because the guards have to + # agree: they answer the same question at different times + # (`hive-tls-ca` at service activation, i.e. on the deploy; + # `hive-tls-resign` from a weekly timer for a host that stays up long + # enough to drift; `swarm-services-cert` for the leaf the store + # issues). A rule implemented in one and not the others is worse than + # one implemented in none — it looks fixed and only fires on whichever + # path you did not take, which is exactly how a corrected + # `serviceDomains` still served a stale leaf. # # Expects `$d` (state dir) to be set; defines `$leaf`-adjacent names and # `covers`. leafCoverage = '' svcleaf="$d/swarm-services.pem" + svcroot="$d/swarm-services-root.pem" hiveNames=${lib.escapeShellArg "${domain} *.${domain}"} svcNames=${lib.escapeShellArg (lib.concatStringsSep " " swarmServiceDomains)} - # The services leaf is only expected where the sub-CA exists; - # elsewhere its absence is the correct state, not a stale leaf. + # A hive with no configured service names has no services leaf to + # hold, and its absence there is the correct state rather than a + # stale one. Anywhere else it is expected: the leaf comes from the + # secret store's `pki` mount now, not from a sub-CA that exists on + # one host in the swarm, so "this host cannot issue it" is no longer + # one of the answers. want_svc=${if swarmServiceDomains == [ ] then "0" else "1"} - if [ ! -s ${lib.escapeShellArg "${swarmCaCfg.stateDir}/services-ca.pem"} ]; then - want_svc=0 - fi # Expiry is not the only way a leaf goes wrong. One signed when the # configured name set was smaller stays valid for its whole lifetime # while omitting every name added since — so a config change can # evaluate, build and deploy cleanly while the gateway keeps serving a - # certificate that does not cover the new service. The services sub-CA - # already reconciles this way (its `.names` comparison in - # ../swarm-ca.nix); the leaves did not. + # certificate that does not cover the new service. # # Reads the names out of the CERTIFICATE, not a sidecar file: the pem # is what nginx serves, and a bookkeeping file drifts from it the @@ -254,13 +231,20 @@ let [ "$_missing" = 0 ] } - # True when every leaf this host is supposed to hold is present and - # carries its configured names. Expiry is the callers' own business — - # they use different windows. + # True when the leaf this host SIGNS is present and carries its + # configured names. Expiry is the callers' own business — they use + # different windows. + # + # ⚠️ The services leaf is deliberately NOT part of this any more, and + # the omission is load-bearing rather than a simplification. Its two + # callers sign under the hive CA's key; the services leaf comes out + # of the secret store, which neither of them can reach. A guard that + # went on reporting it stale would make `hive-tls-ca` re-sign the + # hive leaf forever over a file it has no way to fix. The + # `swarm-services-cert` unit below holds the same rule for that leaf, + # using the `covers` helper above so the two cannot drift. leavesCoverNames() { - covers "$leaf" "$hiveNames" || return 1 - [ "$want_svc" = 0 ] && return 0 - covers "$svcleaf" "$svcNames" + covers "$leaf" "$hiveNames" } ''; in @@ -313,6 +297,44 @@ in ''; }; + baoClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/services-issuer.pem"; + description = '' + Certificate `swarm-services-cert` presents to the secret store when it + asks for the swarm-services leaf the gateway serves. + + Its subject has to be + {option}`services.hyperhive.deploy.bao.servicesIssuerCommonName` — cert + auth matches on the common name, and this is the one role whose policy + permits `pki/issue/swarm-services`. + + A **separate** credential from + {option}`services.hyperhive.deploy.bao.clientCertFile`, which is this + hive's own reader identity and whose policy reads every secret the + hive is entitled to. A unit that renews one server certificate has no + business holding that. + + On a hive that mints its own store PKI a glue module supplies the leaf + as a `mkDefault`; everywhere else it is the credential an operator + places, for the reason the store's own client certificate is. + + A path, never a value. + ''; + }; + + baoClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/services-issuer-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.hive-controller.tls.baoClientCertFile`. + Both or neither — a certificate with no key authenticates nothing. + ''; + }; + leafValidityDays = lib.mkOption { type = lib.types.int; default = 30; @@ -333,40 +355,17 @@ in }; config = lib.mkIf active { - # A swarm service name that is a sibling of the hive domain rather - # than a child needs the swarm-services leaf, and this host only - # signs that one when it holds the swarm root key. Without it nginx - # falls back to the hive leaf on those names and every client sees a - # name mismatch — on a config that evaluates and deploys cleanly. - # - # ⚠️ A WARNING, NOT AN ASSERTION, and the distinction is the point: - # this module can see what *it* can issue; it cannot see an - # operator-installed services sub-CA, an external ACME setup, or a - # cert delivered by any other means. A rebuild must not be blocked - # by a conclusion this host is not in a position to reach. Report - # the observation and let the operator judge it. - # - # (`certDir` / ACME modes don't reach here at all: `active` is - # self-signed-only, and there the operator's cert decides.) - warnings = lib.optional (uncoveredServiceDomains != [ ] && !swarmCaCfg.autoConfigure) '' - services.hyperhive: these swarm service names are outside this - hive's domain (${domain}), so the hive CA's leaf does not cover - them: - - ${lib.concatMapStringsSep "\n" (n: " ${n}") uncoveredServiceDomains} - - services.hyperhive.swarm.ca.autoConfigure is false, so this host - does not sign the swarm-services leaf either, and the gateway will - serve the hive leaf on those names — a certificate-name mismatch - for browsers and for agents' git-over-https. - - If you have already arranged certificates for them — an - operator-installed services sub-CA in ${swarmCaCfg.stateDir}, or - services.hyperhive.gateway.tls.{certDir,acme} — this is expected - and you can ignore it. Otherwise pin the names back under - ${domain} (services.hyperhive.swarm.{forge.domain, - matrix.gatewayHost, authelia.domain}) or install the sub-CA. - ''; + # 🚫 The "these service names are outside this hive's domain and this + # host cannot sign for them" warning is GONE, and its absence is the + # deliverable rather than a tidy-up. It fired on + # `!swarm.ca.autoConfigure` — i.e. "this host does not hold the swarm + # root key" — which was the whole reason a hive could end up serving + # its own leaf on a swarm-service name. The services leaf now comes + # from the secret store's `pki` mount, which every hive reaches over + # the network with its own identity, so holding the root key stopped + # being the thing that decides. Left in place it would warn, on every + # rebuild of every hive that is not the CA host, about a fallback + # that no longer happens. # Generate (and rotate) the hive CA + gateway leaf before anything # serves it. Idempotent: the CA is created once and reused; the leaf @@ -381,26 +380,22 @@ in # against the unit that reads the file. before = [ "hive-gateway-self-signed-cert.service" ]; requiredBy = [ "hive-gateway-self-signed-cert.service" ]; - # The issuance below needs the swarm root key on disk, and (for the - # services leaf) the services sub-CA it signs under. When this host - # generates them (single-host swarm) both units must have run first; - # when the operator provides the material there is no unit to wait - # for, so the dependency is conditional rather than a unit that - # exists and does nothing. + # The issuance below needs the swarm root key on disk. When this + # host generates it (single-host swarm) that unit must have run + # first; when the operator provides the material there is no unit + # to wait for, so the dependency is conditional rather than a unit + # that exists and does nothing. # - # Without waiting for swarm-services-ca specifically, this unit races - # it: if hive-tls-ca finishes first, it finds no services-ca.pem yet, - # silently skips signing the services leaf (the same as "operator - # hasn't set one up"), and the gateway comes up with a vhost pointed - # at a cert that was never written. - after = lib.optionals swarmCaCfg.autoConfigure [ - "swarm-ca.service" - "swarm-services-ca.service" - ]; - requires = lib.optionals swarmCaCfg.autoConfigure [ - "swarm-ca.service" - "swarm-services-ca.service" - ]; + # `swarm-services-cert` is ordered before this one rather than the + # other way round, and not because this unit needs the leaf — it + # never touches it. The trust bundle written at the end of this + # script anchors on every root a consumer must hold, and the + # services root is one of them, so the file has to be on disk + # before the `cat`. Get it wrong and the bundle is a boot behind: + # correct-looking, and missing the anchor for exactly the names the + # gateway serves with it. + after = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ]; + requires = lib.optionals swarmCaCfg.autoConfigure [ "swarm-ca.service" ]; path = [ pkgs.openssl ]; serviceConfig = { Type = "oneshot"; @@ -496,23 +491,14 @@ in # up to a week late and does nothing for the rebuild that changed # the names in the first place. # - # The name check has to include the SERVICES leaf even though the - # condition is written around the hive one, because both are signed - # in this block: a fresh `gateway.pem` otherwise suppresses the - # re-sign of a `swarm-services.pem` that is missing or stale, which - # is what left a corrected `serviceDomains` still mis-served. resign=0 { [ -s "$leaf" ] && [ -s "$leafk" ]; } || resign=1 openssl x509 -in "$leaf" -noout -checkend 2592000 >/dev/null 2>&1 || resign=1 leavesCoverNames || resign=1 - if [ "$want_svc" = 1 ] && [ ! -s "$svcleaf" ]; then - resign=1 - fi if [ "$resign" = 1 ]; then echo "signing gateway leaf at $leaf (missing, near expiry, or missing a configured name)" ${signHiveLeaf} - ${signServicesLeaf} fi # --- Trust bundle: what a consumer must TRUST, as opposed to @@ -534,10 +520,19 @@ in # hive where the optional file is legitimately absent. The # reflexive `|| true` is worse; it also swallows a real failure to # read the hive CA. + # + # `swarm-services-root.pem` joins the set for the reason the swarm + # root is in it: it is an ANCHOR a consumer terminates at, not an + # intermediate. It is also what keeps this change from being a + # flag day — the swarm root stays in the bundle beside it, so a + # leaf still chaining to the old services sub-CA and one issued by + # the store both verify against the same file, and a hive can be + # rebuilt before or after its peers. bundle="$d/trust-bundle.pem" anchors=("$ca") if [ -s "$prev" ]; then anchors+=("$prev"); fi if [ -s "$root" ]; then anchors+=("$root"); fi + if [ -s "$svcroot" ]; then anchors+=("$svcroot"); fi cat "''${anchors[@]}" > "$bundle" chmod 0644 "$bundle" ''; @@ -582,12 +577,17 @@ in d=${lib.escapeShellArg cfg.stateDir} leaf="$d/gateway.pem" - # ⚠️ EVERY leaf this host issues must be covered here. A leaf that + # ⚠️ EVERY leaf this host SIGNS must be covered here. A leaf that # first-boot issuance creates and this unit does not know about # looks perfect for its entire validity and then expires with no - # warning — the failure is invisible until it is total. `svcleaf` - # and the name checks come from the shared snippet, so this unit - # and `hive-tls-ca` cannot disagree about what a good leaf is. + # warning — the failure is invisible until it is total. The name + # checks come from the shared snippet, so this unit and + # `hive-tls-ca` cannot disagree about what a good leaf is. + # + # The services leaf is not one of them any longer: it is issued + # by the secret store, whose key this unit does not hold and + # cannot re-sign under. Renewing it is `swarm-services-cert`'s + # job, at boot, which is a cadence its own issue owns. ${leafCoverage} # Re-sign only when a leaf is within half its validity of expiry. @@ -598,19 +598,17 @@ in [ -s "$1" ] && openssl x509 -in "$1" -noout -checkend "$halflife" >/dev/null 2>&1 } - if fresh "$leaf" && { [ "$want_svc" = 0 ] || fresh "$svcleaf"; } \ - && leavesCoverNames; then + if fresh "$leaf" && leavesCoverNames; then echo "leaves valid, and covering the configured names — no resign needed" exit 0 fi echo "a leaf is missing, near expiry, or missing a configured name — re-signing" - before="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)" + before="$(sha256sum "$leaf" 2>/dev/null || true)" ${signHiveLeaf} - ${signServicesLeaf} - after="$(sha256sum "$leaf" "$svcleaf" 2>/dev/null || true)" + after="$(sha256sum "$leaf" 2>/dev/null || true)" if [ "$before" != "$after" ]; then echo "gateway leaf rotated — re-importing and reloading nginx" systemctl restart hive-gateway-self-signed-cert.service @@ -621,6 +619,256 @@ in ''; }; + # The swarm-services leaf, issued by the secret store's `pki` mount. + # + # ⚠️ This unit replaces a chain, not a command. What used to happen + # was two openssl hops on whichever host held the swarm root key: a + # name-constrained `swarm-services` sub-CA (../swarm-ca.nix), then a + # leaf signed under it here. Both are gone. The store's + # `pki/roles/swarm-services` carries the same narrowing as an + # `allowed_domains` list enforced at issue time, so the intermediate + # that used to encode it in x509 `nameConstraints` has nothing left + # to express — and with the root inside the mount there is nothing + # for it to be an intermediate OF. + # + # The names it may carry are not this unit's to assert: the role + # refuses anything outside `allowed_domains`, which is read from the + # same `swarm.serviceDomains` the SANs below are built from. A + # mismatch is a refusal from the store naming the offending name, + # not a certificate quietly issued for something nobody configured. + systemd.services.swarm-services-cert = { + description = "Issue the swarm-services TLS leaf from the secret store's PKI"; + wantedBy = [ "multi-user.target" ]; + # ⚠️ The ordering the old shape got by ACCIDENT, written down. + # Nothing used to declare "no TLS until the store is up" — it held + # because `swarm-services-ca.service` happened to block the gateway. + # Now it is a real edge: the gateway's certificate comes out of the + # store, so the store being up is a precondition rather than a + # coincidence, and `requiredBy` is what makes a store that never + # comes up an outage that says so instead of a gateway serving the + # wrong name. + before = [ + "hive-tls-ca.service" + "hive-gateway-self-signed-cert.service" + ]; + requiredBy = [ "hive-gateway-self-signed-cert.service" ]; + # The store's container, where it runs here. On a hive that reads a + # store hosted elsewhere no such unit exists and systemd ignores + # the name, which is the correct behaviour rather than a gap: what + # this unit actually needs is the store reachable, and on a remote + # store nothing local can order against that. + after = [ + "container@${baoCfg.machine}.service" + "swarm-bao-services-issuer-policy.service" + ]; + wants = [ "container@${baoCfg.machine}.service" ]; + path = [ + baoDeploy.package + pkgs.jq + pkgs.openssl + pkgs.coreutils + pkgs.systemd + ]; + # Sized like the store's own granting units, and for the same + # reason: under `seal = "shamir"` an operator unseals BY HAND, and + # the login below fails for as long as that takes. 2880 × 30s is + # 24h inside a 25h window — `StartLimit*` are `[Unit]` settings, so + # the window must exceed `RestartSec × burst` or it closes between + # attempts and the burst is never reached. + startLimitBurst = 2880; + startLimitIntervalSec = 90000; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + UMask = "0077"; + SyslogIdentifier = "swarm-services-cert"; + Restart = "on-failure"; + RestartSec = 30; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + } + // lib.optionalAttrs (cfg.baoClientCertFile != null) { + BAO_CLIENT_CERT = cfg.baoClientCertFile; + } + // lib.optionalAttrs (cfg.baoClientKeyFile != null) { + BAO_CLIENT_KEY = cfg.baoClientKeyFile; + } + # Absent means the system trust store, which is what a deployment + # with a real CA wants and what a self-signed one must not be left + # with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + d=${lib.escapeShellArg cfg.stateDir} + install -d -m 0755 "$d" + + ${leafCoverage} + + if [ "$want_svc" = 0 ]; then + echo "no swarm service domains configured — nothing to issue" + exit 0 + fi + + # Same rule as the hive leaf's, at a different cadence: re-issue + # when the file is missing, within 30 days of expiry, or no + # longer carrying every configured name. The name check is what + # makes adding a swarm service take effect on the rebuild that + # added it rather than whenever the certificate happens to lapse. + reissue=0 + { [ -s "$svcleaf" ] && [ -s "$d/swarm-services-key.pem" ]; } || reissue=1 + openssl x509 -in "$svcleaf" -noout -checkend 2592000 >/dev/null 2>&1 || reissue=1 + covers "$svcleaf" "$svcNames" || reissue=1 + [ -s "$svcroot" ] || reissue=1 + + if [ "$reissue" = 0 ]; then + echo "swarm-services leaf valid and covering the configured names — leaving it alone" + exit 0 + fi + + ${ + if cfg.baoClientCertFile == null || cfg.baoClientKeyFile == null then + '' + echo "no bao client certificate configured for this host, so the gateway's" >&2 + echo "swarm-services certificate cannot be requested from the store." >&2 + echo "Set services.hyperhive.deploy.hive-controller.tls.baoClient{Cert,Key}File" >&2 + echo "to a leaf the store's CA signed with CN=${baoDeploy.servicesIssuerCommonName}." >&2 + exit 1'' + else + "" + } + + # `bao`'s own message is the only thing separating an unreachable + # store from a refused identity from a role that would not issue + # these names, and this unit retries on all three — so it reports + # which one rather than asserting all three in a sentence of ours. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting: the + # `BAO_CLIENT_*` variables only decide which certificate the + # handshake presents. Without a token `bao` asks its token + # helper, and that is a `sh` this unit's `path` does not carry — + # `-token-only` answers on stdout and skips the helper. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to the swarm secret store with this host's services-issuer certificate." >&2 + cat "$err" >&2 + exit 1 + fi + export BAO_TOKEN + + echo "requesting the swarm-services leaf from the store" + resp="$(mktemp "$d/swarm-services.json.XXXXXX")" + trap 'rm -f "$err" "$resp"' EXIT + + # ONE call, and the response carries all three artefacts — + # certificate, private key and issuing CA. Splitting it into a + # local CSR plus `pki/sign` would buy a key that never crosses + # the wire, and cost the thing that makes this unit worth having: + # the store, not this host, decides what the certificate says. + if ! bao write -format=json \ + ${lib.escapeShellArg "${baoServicesPkiMount}/issue/${baoServicesPkiRole}"} \ + common_name=${lib.escapeShellArg (builtins.head swarmServiceDomains)} \ + alt_names=${lib.escapeShellArg (lib.concatStringsSep "," swarmServiceDomains)} \ + > "$resp" 2>"$err"; then + echo "the store refused to issue the swarm-services certificate." >&2 + cat "$err" >&2 + exit 1 + fi + + # Three fields out of ONE issuance, which is why the response is + # captured and then split rather than asked for three times with + # `bao write -field=…`: each of those calls would mint a separate + # certificate, and the key would belong to a different one than + # the leaf. + # + # `jq -r` into a file the shell opened, never into a variable and + # never into an argv — the private key does not become visible in + # /proc on its way to disk. + # + # `// empty` on every one of them: `jq -r` renders a missing + # field as the four characters `null`, which is a non-empty file + # and would sail past the check below into a certificate slot. + umask 077 + jq -r '.data.private_key // empty' < "$resp" > "$d/swarm-services-key.pem.new" + jq -r '.data.certificate // empty' < "$resp" > "$d/swarm-services-leaf.new" + jq -r '.data.issuing_ca // empty' < "$resp" > "$svcroot.new" + + for f in "$d/swarm-services-key.pem.new" "$d/swarm-services-leaf.new" "$svcroot.new"; do + if [ ! -s "$f" ]; then + echo "the store's response was missing a field: $f is empty" >&2 + exit 1 + fi + done + + # nginx serves this file verbatim. It carries the leaf ALONE and + # not leaf-plus-issuer, which is the one visible consequence of + # flattening the chain: the issuer here is a self-signed root, so + # appending it would ship an anchor a client either already holds + # — in which case it is redundant — or does not, in which case + # shipping it proves nothing. The old file appended the services + # sub-CA because that WAS an intermediate, and a client anchored + # on the swarm root could not build the middle of the chain by + # itself. + mv -f "$d/swarm-services-leaf.new" "$svcleaf" + mv -f "$d/swarm-services-key.pem.new" "$d/swarm-services-key.pem" + chmod 0644 "$svcleaf" + chmod 0600 "$d/swarm-services-key.pem" + + # The anchor, world-readable, beside the leaf it signed. This is + # the file an operator adds to a browser's trust store, and the + # one `hive-tls-ca` folds into `trust-bundle.pem`. Public + # certificate material only — the private half never leaves the + # store's `pki` mount, which is what `root/generate/internal` + # buys. + rootchanged=0 + if ! cmp -s "$svcroot.new" "$svcroot"; then + rootchanged=1 + fi + mv -f "$svcroot.new" "$svcroot" + chmod 0644 "$svcroot" + + # Propagation, for the RETRY path only. Ordered before both of + # these, so on a normal boot they have not run yet, `is-active` + # is false, and ordering alone does the work. What this covers is + # the store coming up hours after the gateway did: nginx serves a + # *copy* of the leaf, so re-issuing the source changes nothing + # until the copy is remade. + # + # ⚠️ `--no-block`, and it is not a preference. This unit declares + # `Before=` both of these, so a blocking `systemctl restart` + # enqueues a job that systemd will not start until this unit is + # active — and this unit is not active until its ExecStart + # returns, which is waiting on that job. A deadlock, held until + # the 24h retry window's `TimeoutStartSec` fires. Queueing the + # job and letting it run once we exit is the only ordering that + # terminates. + # + # The cost is real and is the reason `hive-tls-resign` does NOT + # do this: a propagation that fails after we return does not fail + # us, so the leaf can rotate on disk while nginx keeps the old + # copy. That unit blocks because nothing orders it before its + # consumers; this one cannot. + if systemctl is-active --quiet nginx.service; then + echo "swarm-services leaf rotated — re-importing and reloading nginx" + systemctl restart --no-block hive-gateway-self-signed-cert.service + systemctl reload --no-block nginx.service + fi + + # The bundle is assembled by `hive-tls-ca`, which is ordered + # after this unit and therefore already holds the current root on + # any normal boot. On the retry path it ran a long time ago, so a + # CHANGED root has to reach it — and only a changed one, or every + # boot would bounce a unit with nothing to do. + if [ "$rootchanged" = 1 ] && systemctl is-active --quiet hive-tls-ca.service; then + echo "the services root changed — rebuilding the trust bundle" + systemctl restart --no-block hive-tls-ca.service + fi + ''; + }; + systemd.timers.hive-tls-resign = { description = "Weekly gateway-leaf re-sign and propagation"; wantedBy = [ "timers.target" ]; diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index b02149c5..8abadefd 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -300,15 +300,48 @@ let # self-signed CA under `/var/lib/swarm-bao-pki`, and it stays outside the # store permanently: bao cannot issue the credential that opens bao. # - # Nothing issues through this mount yet — see the role below. - servicesPkiMountPath = "pki"; + # The mount's own root is generated into it by the bootstrap unit below and + # its private key never leaves — `root/generate/internal` keeps it inside + # the store, which is the whole reason the mount exists rather than another + # key beside ./swarm-ca.nix's. + # + # An option rather than a literal because ./hive-tls.nix issues through it + # and has to spell it the same way. + servicesPkiMountPath = baoDeploy.servicesPkiMountPath; + + # Subject of the root generated into that mount. A label for a human reading + # a chain, not an identity anything authenticates against — same fall-through + # ./swarm-ca.nix:29-37 uses, and for the same reason: a hive that has set + # neither a swarm name nor a domain must still evaluate. + servicesPkiRootLabel = + if hyperhiveCfg.swarm.name != null then + hyperhiveCfg.swarm.name + else if hyperhiveCfg.domain != null then + hyperhiveCfg.domain + else + "hyperhive"; + + # Where the store's host publishes that root's PUBLIC certificate. Its own + # directory rather than ./glue-bao-tls.nix's `/var/lib/swarm-bao-pki`: that + # one holds the CA key that opens the store, is 0700 for exactly that + # reason, and this file is the opposite kind of thing — the one piece of + # certificate material meant to be copied out and handed around. + servicesPkiRootDir = "/var/lib/swarm-bao-services-pki"; + servicesPkiRootFile = "${servicesPkiRootDir}/services-root.pem"; + + # ~30y, matching ./swarm-ca.nix's root rather than its services sub-CA. This + # is an ANCHOR: an operator adds it to a browser's trust store by hand, and + # every rotation is a manual re-trust on every machine that holds it. The + # sub-CA's short window bought cheap rotation because the root above it + # absorbed the churn; nothing sits above this one. + servicesPkiRootTtl = "262800h"; # The role every swarm-service certificate will be issued through, and the # whole narrowing of this mount. A role rather than a name-constrained # sub-CA: the engine enforces the same "these names and no others" at issue # time that ./swarm-ca.nix builds into x509 `nameConstraints` today, so a # second CA in the chain would re-implement a check that already happens. - servicesPkiRoleName = "swarm-services"; + servicesPkiRoleName = baoDeploy.servicesPkiRoleName; # The names that role may issue for. Read out of the swarm-tier option # rather than assembled here, exactly as ./swarm-ca.nix:45 and @@ -317,23 +350,22 @@ let # composing its own list is how they stop agreeing. swarmServiceDomains = hyperhiveCfg.swarm.serviceDomains; - # The identity that will ask this mount for a certificate, once something - # does. A THIRD principal rather than either existing one, for the reason - # `secretPublisherCommonName` gives for being a second: the controller may - # rewrite every hive's policy and login role, and a unit whose whole job is - # renewing one server certificate has no business holding that. + # The identity that asks this mount for a certificate. A FOURTH principal + # rather than any existing one, for the reason `secretPublisherCommonName` + # gives for being a second: the controller may rewrite every hive's policy + # and login role, and a unit whose whole job is renewing one server + # certificate has no business holding that. # # `update` and not `create`: bao's issue endpoint is a POST to an existing # path, which is `update` in ACL terms — `create` here would grant nothing # and read as though it did. - # - # ⚠️ NOTHING IS ATTACHED TO THIS POLICY. A bao policy grants only through a - # token that carries it, so until a login role names it this is an inert - # declaration of intent — which is the point: the cert-auth role that - # attaches it needs a leaf carrying its CN, and minting that is - # ./glue-bao-tls.nix's job, one step further along than this one. servicesIssuerPolicyName = "swarm-services-issuer"; + # The subject the login role accepts, and the CN ./glue-bao-tls.nix signs + # the matching leaf with. An option rather than a literal for the reason + # `controllerCn` gives: two files have to spell it identically. + servicesIssuerCn = baoDeploy.servicesIssuerCommonName; + servicesIssuerPolicyText = '' path "${servicesPkiMountPath}/issue/${servicesPkiRoleName}" { capabilities = ["update"] @@ -971,6 +1003,65 @@ in ''; }; + servicesPkiMountPath = lib.mkOption { + type = lib.types.str; + default = "pki"; + description = '' + Mount path of the PKI engine the swarm's **service** certificates are + issued from, and whose root is the anchor those certificates chain to. + + An option rather than a literal because two files have to spell it + identically: this module enables the mount, generates its root and + writes the grant that points at `/issue/`, while ./hive-tls.nix + calls that endpoint. A mount path only one of them knows is a grant + naming a location nobody issues from. + + ⚠️ NOT the store's own client-auth PKI. That one is + ./glue-bao-tls.nix's self-signed CA on disk, and it stays outside the + store permanently — bao cannot issue the credential that opens bao. + ''; + }; + + servicesPkiRoleName = lib.mkOption { + type = lib.types.str; + default = "swarm-services"; + description = '' + Role on {option}`services.hyperhive.deploy.bao.servicesPkiMountPath` + that the swarm-services certificate is issued through, and the whole + narrowing of that mount. + + A role rather than a name-constrained sub-CA: the engine enforces the + same "these names and no others" at issue time, from an + `allowed_domains` read off + {option}`services.hyperhive.swarm.serviceDomains`, that an + intermediate would encode in x509 `nameConstraints` — so a second + authority in the chain would re-implement a check that has already + happened. + ''; + }; + + servicesIssuerCommonName = lib.mkOption { + type = lib.types.str; + default = "swarm-services-issuer"; + example = "swarm-services-issuer.svc"; + description = '' + Subject the store's services-issuer cert-auth role accepts — the + identity a hive presents when it asks the `pki` mount for the + swarm-services certificate its gateway serves. + + A **fourth** identity rather than reuse of any sibling above, and the + reasoning is theirs: this principal's whole grant is one `update` on + `pki/issue/swarm-services`, while the controller's includes rewriting + every hive's policy and login role. Handing a certificate-renewal unit + the controller's leaf would give that separation away in one line. + + ⚠️ Same collision as its siblings, and the same answer: ./swarm.nix + feeds this value into the guard on + {option}`services.hyperhive.swarm.hives`, so a hive named after it + fails evaluation rather than silently receiving this grant. + ''; + }; + matrixCtlHiveName = lib.mkOption { type = lib.types.str; default = toString hyperhiveCfg.hiveName; @@ -1407,6 +1498,7 @@ in "swarm-bao-certs" "swarm-bao-token" "swarm-bao-forwarder-oidc" + "swarm-bao-services-issuer-policy" ]; # 🚫 No `swarm.otel.scrapeTargets.bao` entry any more, and its absence is @@ -1748,21 +1840,69 @@ in *) bao secrets enable -path=${credentialMountPath} kv-v2 ;; esac - # The PKI engine the swarm's service certificates are to come from. - # Asked rather than attempted for the same reason as the mount above: + # The PKI engine the swarm's service certificates come from. Asked + # rather than attempted for the same reason as the mount above: # `secrets enable` errors on a path already in use, so a second # rebuild would fail a unit that has nothing left to do. - # - # ⚠️ Deliberately EMPTY: nothing here generates an issuer into it, so - # the role below can be written but cannot yet issue. That is the - # whole shape of this step — the mount, its narrowing and its grant - # exist, and the binary that uses them arrives separately. Undoing it - # is `bao secrets disable ${servicesPkiMountPath}`. case "$mounts" in *'"${servicesPkiMountPath}/"'*) ;; *) bao secrets enable -path=${servicesPkiMountPath} pki ;; esac + # ⚠️ THE ROOT, AND THE ONE THING THIS UNIT MUST NEVER DO TWICE. + # + # `root/generate/internal` mints a new self-signed CA every time it + # is called. Calling it unconditionally would hand the swarm a + # different anchor on every boot: every certificate issued under the + # previous one stops verifying, and every browser an operator taught + # to trust the old root has to be re-taught. So the generation is + # guarded, and the guard asks BAO rather than looking for a marker + # file — a file on this host's disk is a claim about a store that + # may live in a container, may have been restored from a snapshot, + # and may have had the mount disabled and re-enabled underneath it. + # The mount's own issuer list is the only answer that cannot be + # stale. + # + # `bao list ${servicesPkiMountPath}/issuers` is a 404 (non-zero) on a + # mount with no issuer and a key list once one exists, so its exit + # status IS the question — no output parsing, no error string to + # recognise. The default issuer is what `${servicesPkiMountPath}/issue/` + # signs with, and there is exactly one. + if ! bao list ${lib.escapeShellArg "${servicesPkiMountPath}/issuers"} >/dev/null 2>&1; then + echo "generating the swarm services root into the ${servicesPkiMountPath} mount" + # `-field=issuing_ca` discards the rest of the response. The + # private key is not in it under `internal` and cannot be — that + # is the point of the endpoint — and the certificate is read back + # below from a path that works on every subsequent boot too. + bao write -field=issuing_ca ${lib.escapeShellArg "${servicesPkiMountPath}/root/generate/internal"} \ + common_name=${lib.escapeShellArg "swarm-services-ca ${servicesPkiRootLabel}"} \ + issuer_name=${lib.escapeShellArg servicesPkiRoleName} \ + ttl=${servicesPkiRootTtl} \ + key_type=rsa \ + key_bits=4096 >/dev/null + else + echo "the ${servicesPkiMountPath} mount already has an issuer — leaving it alone" + fi + + # The anchor, as a world-readable file on the store's own host. + # + # Written on EVERY run and not only after a generation, so a store + # whose root predates this unit still publishes it. It is a public + # certificate — the material a browser has to be told to trust — + # and the private half stays in the mount, so 0644 is the correct + # mode rather than a concession. + # + # Why a file at all when `bao read -field=certificate + # ${servicesPkiMountPath}/cert/ca` returns the same bytes: every + # listener sets `tls_require_and_verify_client_cert`, so reading it + # from the store costs a client certificate. An operator adding the + # root to a browser is not necessarily holding one. + install -d -m 0755 ${lib.escapeShellArg servicesPkiRootDir} + bao read -field=certificate ${lib.escapeShellArg "${servicesPkiMountPath}/cert/ca"} \ + > ${lib.escapeShellArg "${servicesPkiRootFile}.new"} + chmod 0644 ${lib.escapeShellArg "${servicesPkiRootFile}.new"} + mv -f ${lib.escapeShellArg "${servicesPkiRootFile}.new"} ${lib.escapeShellArg servicesPkiRootFile} + # `bao write` is an upsert, so this re-asserts the role on every # rebuild rather than failing on one that exists — same shape as the # cert-auth roles below, and the reason the mount needs the `case` @@ -1799,14 +1939,6 @@ in client_flag=false \ key_type=rsa \ key_bits=4096 - - # Idempotent for the same reason the controller's policy above is: a - # rebuild re-asserts it. Written here rather than in a unit of its - # own because it attaches to nothing yet — the sibling-unit rule - # below is about a second *principal* that logs in, and this policy - # has no login role until the leaf carrying its CN exists. - printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} | - bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} - '' + lib.optionalString (baoDeploy.clientCaFile != null) '' @@ -1958,6 +2090,57 @@ in systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders; + # A FOURTH sibling, same shape and same reasons as the two above. This + # one is what turns `swarm-services-issuer` from a declaration into a + # grant: a bao policy reaches nothing until a login role hands it to a + # token, so the `bao write auth/cert/certs/…` below is the whole + # difference between a mount nobody may issue from and one the gateway's + # certificate comes out of. + # + # The policy text moved here from the controller's unit, where it sat + # while it attached to nothing — a policy and the role that carries it + # belong in one place, and now there is a principal to put them with. + systemd.services.swarm-bao-services-issuer-policy = lib.mkIf haveBootstrapToken { + description = "write the swarm services issuer's bao policy and cert-auth role"; + after = [ + "container@${cfg.machine}.service" + "swarm-bao-controller-policy.service" + ]; + wantedBy = [ "multi-user.target" ]; + path = [ + baoCli + pkgs.coreutils + ]; + unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; + # Same unseal wait as its three siblings above, for the reason stated + # there: under `seal = "shamir"` a human unseals by hand. + startLimitBurst = 2880; + startLimitIntervalSec = 90000; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + Restart = "on-failure"; + RestartSec = 30; + }; + script = '' + set -euo pipefail + + BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})" + export BAO_TOKEN + + printf '%s' ${lib.escapeShellArg servicesIssuerPolicyText} | + bao policy write ${lib.escapeShellArg servicesIssuerPolicyName} - + '' + + lib.optionalString (baoDeploy.clientCaFile != null) '' + + bao write auth/cert/certs/${lib.escapeShellArg servicesIssuerPolicyName} \ + certificate=@${tlsDir}/client-ca.pem \ + allowed_common_names=${lib.escapeShellArg servicesIssuerCn} \ + token_policies=${lib.escapeShellArg servicesIssuerPolicyName} \ + display_name=${lib.escapeShellArg servicesIssuerCn} + ''; + }; + # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. diff --git a/nix/host-modules/swarm-ca.nix b/nix/host-modules/swarm-ca.nix index 74869644..a434b655 100644 --- a/nix/host-modules/swarm-ca.nix +++ b/nix/host-modules/swarm-ca.nix @@ -14,6 +14,13 @@ # for the obvious reason; the cert as a consequence, and that one costs # something real — nothing whose trust store is built at build time can # name it: `docs/swarm/ca.md::Distributing the root`. +# +# ⚠️ This root no longer issues the swarm's SERVICE certificates, and the +# `swarm-services-ca` sub-CA that used to sit under it is gone rather than +# moved. Those come out of the secret store's own `pki` mount now, anchored +# on a root generated inside it (./swarm-bao.nix), and ./hive-tls.nix asks +# for them over the network instead of signing them here. What is left here +# is one job: the anchor each hive's own CA is issued under. { lib, config, @@ -36,13 +43,6 @@ let hyperhiveCfg.domain else "hyperhive"; - - # Derived once in ./swarm.nix, read here and by ./hive-tls.nix: the - # CA that name-constrains these and the leaf that carries them as SANs - # must agree exactly, and two modules each assembling the list is how - # they stop agreeing. It is also this unit's *rotation trigger* below, - # which is why the ordering is stable there rather than here. - serviceDomains = hyperhiveCfg.swarm.serviceDomains; in { options.services.hyperhive.swarm.ca = { @@ -90,23 +90,6 @@ in ''; }; - servicesValidityDays = lib.mkOption { - type = lib.types.int; - default = 1825; - description = '' - Validity window of the swarm-services sub-CA in days (~5y). - Deliberately far shorter than the root's: this CA is *meant* to - be re-issued — adding a swarm service changes its name - constraints and rotates it — so a long window buys nothing, and - a short one keeps the rotation path exercised rather than - theoretical. - - Rotating it is cheap in the way rotating the root is not: it - touches only the swarm-service vhosts, and no peer hive holds it - as an anchor. - ''; - }; - validityDays = lib.mkOption { type = lib.types.int; default = 10950; @@ -125,11 +108,8 @@ in config = lib.mkIf (hyperhiveCfg.enable && cfg.autoConfigure) { # A CA that fails to issue is invisible until something makes a TLS call - # hours later, so these two oneshots are worth more than most services. - services.hyperhive.swarm.otel.journaldUnits = [ - "swarm-ca" - "swarm-services-ca" - ]; + # hours later, so this oneshot is worth more than most services. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-ca" ]; systemd.services.swarm-ca = { description = "Generate the swarm root CA when absent"; @@ -183,107 +163,5 @@ in chmod 0644 "$root" ''; }; - - # The swarm-services sub-CA: issues leaves for the swarm's own - # service names, which no hive CA can sign — each of those is - # name-constrained to its own hive's domain, and the service names - # are siblings of it, not children. - # - # Rotation is the point of it being separate (mara: "swarm services - # sub ca that can rotate independently of swarm root ca"): the - # constraint enumerates the exact service names, so adding a service - # re-issues *this* and never touches the root or any hive CA. - systemd.services.swarm-services-ca = { - description = "Issue the swarm-services sub-CA under the swarm root"; - wantedBy = [ "multi-user.target" ]; - after = [ "swarm-ca.service" ]; - requires = [ "swarm-ca.service" ]; - path = [ pkgs.openssl ]; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - UMask = "0077"; - SyslogIdentifier = "swarm-services-ca"; - }; - script = '' - set -euo pipefail - d=${lib.escapeShellArg cfg.stateDir} - root="$d/root.pem" - rootk="$d/root-key.pem" - ca="$d/services-ca.pem" - cak="$d/services-ca-key.pem" - # The name set this CA was last issued for. Comparing against it - # is what makes re-issuance happen exactly when the service - # names change — not every boot, and not never. - names="$d/services-ca.names" - want=${lib.escapeShellArg (lib.concatStringsSep "\n" serviceDomains)} - - if [ -z "$want" ]; then - echo "no swarm service domains configured — nothing to issue for" - exit 0 - fi - - # Same half-provisioned guard as the root: a key beside a cert - # that did not sign it looks like it works and issues nothing - # anyone will trust. - if { [ -e "$ca" ] && [ ! -e "$cak" ]; } || { [ -e "$cak" ] && [ ! -e "$ca" ]; }; then - echo "services sub-CA half-provisioned ($ca / $cak) — refusing to generate over it" >&2 - exit 1 - fi - - if [ -s "$ca" ] && [ -s "$cak" ] && [ -f "$names" ] \ - && [ "$(cat "$names")" = "$want" ]; then - echo "services sub-CA present and covers the configured names — leaving it alone" - exit 0 - fi - - if [ ! -s "$root" ] || [ ! -s "$rootk" ]; then - echo "no swarm root CA at $root — cannot issue the services sub-CA under it" >&2 - exit 1 - fi - - echo "issuing services sub-CA at $ca for: $(echo "$want" | tr '\n' ' ')" - csr="$(mktemp "$d/services-ca.csr.XXXXXX")" - ext="$(mktemp "$d/services-ca.ext.XXXXXX")" - trap 'rm -f "$csr" "$ext"' EXIT - - openssl req -newkey rsa:4096 -nodes -sha256 \ - -keyout "$cak" -out "$csr" \ - -subj "/CN=swarm-services-ca ${swarmLabel}" - - { - # pathlen:0 — this signs leaves and delegates no further. - printf 'basicConstraints=critical,CA:TRUE,pathlen:0\n' - printf 'keyUsage=critical,keyCertSign,cRLSign\n' - printf 'subjectKeyIdentifier=hash\n' - printf 'authorityKeyIdentifier=keyid:always\n' - # Constrained to the exact service names, not to the whole - # swarm domain: a leaked services CA should mint `forge.`, - # `chat.`, `auth.` and nothing else. The IP exclusions are not - # redundant — a DNS constraint says nothing about an - # iPAddress SAN, and an unconstrained name type is a name - # type this CA is unconstrained for. - # - # ⚠️ EVERY entry carries its own `permitted;` / `excluded;` - # prefix. openssl's parser takes the qualifier per subtree, not - # once for a run of them: `permitted;DNS:a,DNS:b` is rejected - # outright with `v2i_NAME_CONSTRAINTS: invalid syntax`, which - # fails the whole unit. The hive CA next door emits exactly one - # permitted name, so the missing-prefix form is accidentally - # valid there and does NOT generalise — this list is always - # longer than one. - printf 'nameConstraints=critical,%s,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:0:0:0:0:0:0:0:0/0:0:0:0:0:0:0:0\n' \ - "$(echo "$want" | sed 's/^/permitted;DNS:/' | paste -sd, -)" - } > "$ext" - - openssl x509 -req -in "$csr" -CA "$root" -CAkey "$rootk" \ - -CAcreateserial -days ${toString cfg.servicesValidityDays} -sha256 \ - -extfile "$ext" -out "$ca" - - printf '%s' "$want" > "$names" - chmod 0600 "$cak" - chmod 0644 "$ca" "$names" - ''; - }; }; } diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 213aa382..67f4854c 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -49,6 +49,7 @@ let deployCfg.bao.matrixCtlCommonName deployCfg.bao.grafanaOidcCommonName deployCfg.bao.otelOidcCommonName + deployCfg.bao.servicesIssuerCommonName ] # The two per-hive readers' subjects, spelled out per hive rather than as the # prefix. The prefix alone would reserve the wrong string: the role for hive