From 5761398d60a1cbb114f8d52f52bd0b3a54d28d7e Mon Sep 17 00:00:00 2001 From: atlas Date: Tue, 6 Oct 2026 09:48:16 +0200 Subject: [PATCH] hive-tls: the gateway leaf uses the services-issuer identity, not a hive's swarm-services-cert issues the swarm-services leaf a self-signed gateway serves, and on a host fronting no swarm name fetches the store's pki root. It is gated on the gateway, not on a hive, yet its leaf pair was named under deploy.hive-controller.tls and defaulted from the hive reader's cert directory, and the root fetch presented the hive's own identity. A gateway host with no hive had no such identity, so that branch was red by construction. - The pair moves to deploy.bao.servicesIssuerClient{Cert,Key}File, beside the store's other host-unit readers, and glue-bao-tls.nix defaults it on the store's host from the leaf it already mints. The old deploy.hive-controller.tls.baoClient{Cert,Key}File paths are mkRenamedOptionModule aliases. glue-services-issuer-bao-identity.nix, which derived the default from deploy.bao.clientCertFile, is gone. - The no-swarm-name root fetch presents the services-issuer pair. The hive identity is now only hive-tls-resign's. - A self-signed gateway with no pair fails evaluation, naming both options. A pair named but unreadable still fails the unit, naming the paths. Refs #4921 --- docs/swarm/ca.md | 46 +++-- nix/host-modules/default.nix | 1 - nix/host-modules/deploy.nix | 11 ++ nix/host-modules/glue-bao-tls.nix | 6 +- .../glue-hive-tls-bao-identity.nix | 4 +- .../glue-services-issuer-bao-identity.nix | 43 ----- nix/host-modules/hive-tls.nix | 181 +++++++++--------- nix/host-modules/swarm-bao.nix | 37 ++++ nix/module-eval/bao-grants.nix | 4 +- nix/module-eval/forge-placement.nix | 4 + nix/module-eval/hive-tls.nix | 140 ++++++++++++-- nix/module-eval/nats-tls.nix | 10 +- nix/module-eval/swarm-otel-identity.nix | 5 +- 13 files changed, 319 insertions(+), 173 deletions(-) delete mode 100644 nix/host-modules/glue-services-issuer-bao-identity.nix diff --git a/docs/swarm/ca.md b/docs/swarm/ca.md index e6ec62ad..d98cfb3e 100644 --- a/docs/swarm/ca.md +++ b/docs/swarm/ca.md @@ -39,11 +39,10 @@ from the store's own client CA, with the subject and set both options, before the host's next deploy. Without it, `hive-tls-resign.service` fails with a message naming both -options, and so does `swarm-services-cert.service` on a hive fronting no -swarm name. The unit checks the identity before the leaf, so it fails even while a +options. The unit checks the identity before the leaf, so it fails even while a valid leaf keeps the gateway up. The identity is the hive's alone: -it issues this hive's leaf and fetches the store's root, and it's never the -identity a swarm service uses. +it issues this hive's leaf and nothing else, and it's never the identity a +swarm service uses. **When it issues.** The unit runs at boot, on every deploy, and from `hive-tls-resign.timer` once a day. It asks the store for a new leaf when @@ -108,12 +107,33 @@ domain, so it can't cover them. The secret store's `pki` mount issues them a leaf of their own, through its `swarm-services` role. `swarm-services-cert.service` logs in with -this host's services-issuer certificate, calls +the services-issuer identity, calls `pki/issue/swarm-services`, and writes the result to `/var/lib/hive-tls/swarm-services.pem` — the path the gateway already -copies from. Every hive does this with its own identity, so the store's -role, not a key on the hive, decides whether a hive can serve its swarm's -names. +copies from. The store's role, not a key on the host, decides which names a +gateway can serve. + +**Every gateway that serves the self-signed leaf needs the services-issuer +identity**, whether or not a hive runs on its host. It's a client certificate +from the store's own client CA, with the subject +`deploy.bao.servicesIssuerCommonName` (`swarm-services-issuer` by default), +and it's named by `services.hyperhive.deploy.bao.servicesIssuerClientCertFile` +and `servicesIssuerClientKeyFile`. Its policy may issue from +`pki/issue/swarm-services` and nothing else, and it's the same certificate on +every gateway host: + +- On the host running the store, `swarm-bao-pki.service` mints it at + `/var/lib/swarm-bao-pki/services-issuer.pem` and the two options default to + it. +- On every other gateway host, copy `services-issuer.pem` and + `services-issuer-key.pem` from the store's host and set both options. + +A host whose gateway serves the self-signed leaf without both options set +fails evaluation, naming them. One that names files it can't read fails +`swarm-services-cert.service`, naming the paths. The old spellings, +`deploy.hive-controller.tls.baoClientCertFile` and `baoClientKeyFile`, are +renames of these two: a config that still sets them evaluates with a warning +naming both paths. `services.hyperhive.swarm.serviceDomains`, every swarm service name and the same list on every host, populates the role's @@ -123,14 +143,12 @@ it. A host's own SANs are narrower still: it asks only for the names it fronts a vhost for (`swarm.localServiceDomains`), so the key on one gateway can't serve a swarm service that runs behind another. -**The root's public certificate is a file, on every hive:** +**The root's public certificate is a file, on every self-signed gateway host:** `/var/lib/hive-tls/swarm-services-root.pem` (0644), written beside the -leaf it signed and folded into `trust-bundle.pem`. A hive fronting no +leaf it signed and folded into `trust-bundle.pem`. A host fronting no swarm name asks for no services leaf and reads the root alone, from the store's -`pki/ca/pem`, presenting its own store identity -(`deploy.hive-controller.tls.baoHiveIdentityCertFile` / -`baoHiveIdentityKeyFile`) for the handshake. Every hive must hold one: -without it, or with either file absent, +`pki/ca/pem`, presenting the services-issuer identity for the handshake +without logging in with it. With either file absent, `swarm-services-cert.service` fails with a message naming both options. On the host running the store it's also at diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 6162a0b0..2a6b737f 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -40,7 +40,6 @@ ./glue-nats-oidc-client.nix ./glue-queue-agent-credential.nix ./glue-secret-publisher-bao-identity.nix - ./glue-services-issuer-bao-identity.nix ./glue-swarm-bao-otel-oidc-client.nix ./glue-swarm-otel-oidc-client.nix ./swarm-authelia-service.nix diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index 446e1d7f..d82e37f3 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -174,6 +174,17 @@ in [ "services" "hyperhive" "deploy" "ci-runner" "jobTimeout" ] ) + # The gateway's services-issuer pair is a swarm-service identity, not a + # hive's, so it sits with the store's other host-unit readers. + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "deploy" "hive-controller" "tls" "baoClientCertFile" ] + [ "services" "hyperhive" "deploy" "bao" "servicesIssuerClientCertFile" ] + ) + (lib.mkRenamedOptionModule + [ "services" "hyperhive" "deploy" "hive-controller" "tls" "baoClientKeyFile" ] + [ "services" "hyperhive" "deploy" "bao" "servicesIssuerClientKeyFile" ] + ) + # The runner toggle sat under `swarm.forge.ci` before `deploy.forgejo.ci`. # Both were per-host, so each forwards straight to its final path. (lib.mkRenamedOptionModule diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix index cced87a9..7a1fd64b 100644 --- a/nix/host-modules/glue-bao-tls.nix +++ b/nix/host-modules/glue-bao-tls.nix @@ -107,6 +107,8 @@ in forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem"; ciRunnerClientCertFile = lib.mkDefault "${pkiDir}/ci-runner.pem"; ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem"; + servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem"; + servicesIssuerClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem"; # The granter: every `swarm-bao-*-policy` unit on this host logs in with # it. ⚠️ Unlike every other leaf here, never the file an operator copies: @@ -224,8 +226,8 @@ in [ -s ${pkiDir}/forwarder-oidc.pem ] || ${signLeaf} ${pkiDir} forwarder-oidc \ ${lib.escapeShellArg deployCfg.bao.forwarderOidcCommonName} "" clientAuth - # The identity a hive presents to ask the store's `pki` mount for the - # swarm-services certificate its gateway serves. Minted here like the + # The identity a gateway host presents to ask the store's `pki` mount + # for the swarm-services certificate it serves. Minted here like the # three above, and the reason is the sharpest of the four: this leaf # is what OPENS the mint, so it cannot come out of it. A certificate # authority that issues the credential admitting you to it is the diff --git a/nix/host-modules/glue-hive-tls-bao-identity.nix b/nix/host-modules/glue-hive-tls-bao-identity.nix index f3a13880..19240669 100644 --- a/nix/host-modules/glue-hive-tls-bao-identity.nix +++ b/nix/host-modules/glue-hive-tls-bao-identity.nix @@ -6,7 +6,7 @@ # that credential, which is what every hive on a host without the store does. # # ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the -# leaf, for the reason ./glue-services-issuer-bao-identity.nix gives. +# leaf, because the thing that owns a private key owns issuing from it. # # Gated on the minted PKI existing, as that file is, and on a hive running # here: a host without one has no hive leaf to ask for. @@ -21,7 +21,7 @@ let baoDeploy = config.services.hyperhive.deploy.bao; # Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own - # path as ./glue-services-issuer-bao-identity.nix derives it. + # path rather than repeating that file's directory literal. haveMintedPki = baoDeploy.clientCertFile != null; pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null; in diff --git a/nix/host-modules/glue-services-issuer-bao-identity.nix b/nix/host-modules/glue-services-issuer-bao-identity.nix deleted file mode 100644 index 32f28f3f..00000000 --- a/nix/host-modules/glue-services-issuer-bao-identity.nix +++ /dev/null @@ -1,43 +0,0 @@ -# Glue: point the swarm-services certificate unit at the bao leaf minted for -# it. -# -# ONE PAIRING PER FILE — the services-issuer principal ← bao, and nothing -# else. Deleting this leaves a hive that takes an operator-provided path to -# that credential, which is what any deployment not minting its own already -# does. -# -# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the -# leaf, because the thing that owns a private key owns issuing from it. What -# belongs here is the pairing: which paths `swarm-services-cert` reads. -# -# ⚠️ Gated on the leaf existing, not on the store running here — the same rule -# ./glue-controller-bao-identity.nix states, and it matters more for this -# principal than for any of its siblings: every hive in a swarm serves the -# swarm's service names and therefore needs this certificate, while exactly -# one of them hosts the store. -# -# Everything is `mkDefault`. An operator naming their own paths wins. -{ - lib, - config, - ... -}: -let - hyperhiveCfg = config.services.hyperhive; - deployCfg = hyperhiveCfg.deploy; - baoDeploy = deployCfg.bao; - - # Where ./glue-bao-tls.nix puts the leaves, derived from the reader's own - # path rather than repeating that file's directory literal: an operator who - # moves the PKI moves both, and the two cannot drift apart. - haveMintedPki = baoDeploy.clientCertFile != null; - pkiDir = if haveMintedPki then builtins.dirOf baoDeploy.clientCertFile else null; -in -{ - config = lib.mkIf haveMintedPki { - services.hyperhive.deploy.hive-controller.tls = { - baoClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem"; - baoClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem"; - }; - }; -} diff --git a/nix/host-modules/hive-tls.nix b/nix/host-modules/hive-tls.nix index 6055615b..465d9190 100644 --- a/nix/host-modules/hive-tls.nix +++ b/nix/host-modules/hive-tls.nix @@ -72,41 +72,77 @@ let else "${baoDeploy.hiveLeafPkiRolePrefix}-${hyperhiveCfg.hiveName}"; - # The hive's own store identity: its subject is - # `-`, and its policy may issue from - # `hiveLeafRole` alone. It also carries the root fetch's handshake. - haveHiveIdentity = cfg.baoHiveIdentityCertFile != null && cfg.baoHiveIdentityKeyFile != null; showPath = p: if p == null then "unset" else p; - hiveIdentityOptions = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile (${showPath cfg.baoHiveIdentityCertFile}) and services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile (${showPath cfg.baoHiveIdentityKeyFile})"; - # Fails the unit, naming both options, when the identity is unset or a file - # is unreadable. Every hive must hold one. - requireHiveIdentity = + # Fails the unit, naming both options, when an identity is unset or a file + # is unreadable. + requireIdentity = + { + holder, + what, + certFile, + keyFile, + certOption, + keyOption, + unsetHint ? "", + }: purpose: - if haveHiveIdentity then + let + options = "${certOption} (${showPath certFile}) and ${keyOption} (${showPath keyFile})"; + in + if certFile != null && keyFile != null then '' missing="" for f in ${ lib.escapeShellArgs [ - cfg.baoHiveIdentityCertFile - cfg.baoHiveIdentityKeyFile + certFile + keyFile ] }; do [ -r "$f" ] || missing="$missing $f" done if [ -n "$missing" ]; then - echo "this hive has no usable store identity, so it cannot ${purpose}: cannot read$missing." >&2 - echo "place the files named by ${hiveIdentityOptions}." >&2 + echo "${holder} has no usable ${what}, so it cannot ${purpose}: cannot read$missing." >&2 + echo "place the files named by ${options}." >&2 exit 1 fi '' else '' - echo "this hive has no store identity, so it cannot ${purpose}." >&2 - echo "set ${hiveIdentityOptions} and place both files." >&2 + echo "${holder} has no ${what}, so it cannot ${purpose}." >&2 + echo "set ${options} and place both files.${unsetHint}" >&2 exit 1 ''; + # The hive's own store identity: its subject is + # `-`, and its policy may issue from + # `hiveLeafRole` alone. Every hive must hold one. + requireHiveIdentity = requireIdentity { + holder = "this hive"; + what = "store identity"; + certFile = cfg.baoHiveIdentityCertFile; + keyFile = cfg.baoHiveIdentityKeyFile; + certOption = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile"; + keyOption = "services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile"; + }; + + # The swarm-wide services-issuer identity `swarm-services-cert` presents, + # whether or not a hive runs here. Its subject is + # `servicesIssuerCommonName`, and its policy may issue from + # `pki/issue/swarm-services` alone. + servicesIssuerCert = baoDeploy.servicesIssuerClientCertFile; + servicesIssuerKey = baoDeploy.servicesIssuerClientKeyFile; + haveServicesIssuer = servicesIssuerCert != null && servicesIssuerKey != null; + requireServicesIssuer = requireIdentity { + holder = "this host"; + what = "services-issuer identity"; + certFile = servicesIssuerCert; + keyFile = servicesIssuerKey; + certOption = "services.hyperhive.deploy.bao.servicesIssuerClientCertFile"; + keyOption = "services.hyperhive.deploy.bao.servicesIssuerClientKeyFile"; + unsetHint = " Its subject must be CN=${baoDeploy.servicesIssuerCommonName}."; + }; + # Shared by every unit that decides whether to re-issue a leaf. It # lives here rather than in one of them because the guards have to # agree: they answer the same question for different leaves @@ -195,11 +231,11 @@ let servicesCertEnvironment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; } - // lib.optionalAttrs (cfg.baoClientCertFile != null) { - BAO_CLIENT_CERT = cfg.baoClientCertFile; + // lib.optionalAttrs (servicesIssuerCert != null) { + BAO_CLIENT_CERT = servicesIssuerCert; } - // lib.optionalAttrs (cfg.baoClientKeyFile != null) { - BAO_CLIENT_KEY = cfg.baoClientKeyFile; + // lib.optionalAttrs (servicesIssuerKey != null) { + BAO_CLIENT_KEY = servicesIssuerKey; } # Absent means the system trust store, which is what a deployment # with a real CA wants and what a self-signed one must not be left @@ -208,9 +244,8 @@ let BAO_CACERT = baoDeploy.serverCaFile; }; - # The root-only fetch. Same store and server CA as `servicesCertEnvironment`, - # but the hive's own identity: a host fronting no swarm name holds no - # services-issuer pair. + # The root-only fetch: the same store, server CA and services-issuer pair + # as `servicesCertEnvironment`, handed to curl. storeRootCurlTls = lib.escapeShellArgs ( lib.optionals (baoDeploy.serverCaFile != null) [ "--cacert" @@ -218,9 +253,9 @@ let ] ++ [ "--cert" - cfg.baoHiveIdentityCertFile + servicesIssuerCert "--key" - cfg.baoHiveIdentityKeyFile + servicesIssuerKey ] ); @@ -262,62 +297,21 @@ in ''; }; - baoClientCertFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/services-issuer.pem"; - description = '' - Certificate `swarm-services-cert` presents to the secret store when it - asks for the swarm-services leaf the gateway serves. - - Its subject has to be - {option}`services.hyperhive.deploy.bao.servicesIssuerCommonName` — cert - auth matches on the common name, and this is the one role whose policy - permits `pki/issue/swarm-services`. - - A **separate** credential from - {option}`services.hyperhive.deploy.bao.clientCertFile`, which is this - hive's own reader identity and whose policy reads every secret the - hive is entitled to, and from - {option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile`, - which the root fetch on a hive fronting no swarm name presents. - - On a hive that mints its own store PKI a glue module supplies the leaf - as a `mkDefault`; everywhere else it is the credential an operator - places, for the reason the store's own client certificate is. - - A path, never a value. - ''; - }; - - baoClientKeyFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/services-issuer-key.pem"; - description = '' - Private key for - {option}`services.hyperhive.deploy.hive-controller.tls.baoClientCertFile`. - Both or neither — a certificate with no key authenticates nothing. - ''; - }; - baoHiveIdentityCertFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; example = "/var/lib/swarm-bao-pki/hive-leaf.pem"; description = '' This hive's own store identity. `hive-tls-resign` presents it to ask - the secret store for the gateway leaf, and `swarm-services-cert` on a - hive fronting no swarm name presents it for the handshake of the root - fetch. + the secret store for the gateway leaf. Its subject has to be `-`; that cert-auth role's policy may issue from this hive's gateway-leaf role and nothing else. - Every hive with a self-signed gateway needs it: without it both units - fail, naming this option and + Every hive with a self-signed gateway needs it: without it that unit + fails, naming this option and {option}`services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityKeyFile`. On the host that runs the store a glue module supplies the leaf as a `mkDefault`; on every other host an operator places it. @@ -339,6 +333,30 @@ in }; config = lib.mkIf active { + assertions = [ + { + assertion = haveServicesIssuer; + message = '' + services.hyperhive.gateway serves the self-signed leaf on this host, + which needs the swarm-services issuer's store identity: set both + + services.hyperhive.deploy.bao.servicesIssuerClientCertFile + services.hyperhive.deploy.bao.servicesIssuerClientKeyFile + + swarm-services-cert.service presents it to ask the swarm secret + store for the swarm-services leaf and to fetch the store's pki root. + + Both default to the pair glue-bao-tls.nix mints when + services.hyperhive.deploy.bao.enable is true (the host that runs + the store). Otherwise copy services-issuer.pem and + services-issuer-key.pem from the store host's /var/lib/swarm-bao-pki/ + and name them in those two options, or give the gateway a + certificate of its own (services.hyperhive.gateway.tls.certDir or + services.hyperhive.gateway.tls.acme.enable). + ''; + } + ]; + # The state dir and the trust bundle. Never waits on the store: the # store's container is ordered after this unit, which writes the # bundle it bind-mounts, so the leaf is `hive-tls-resign`'s. @@ -603,9 +621,9 @@ in # store nothing local can order against that. # # `swarm-bao-pki` mints the services-issuer leaf the login below - # presents, and is absent on a hive holding an operator-placed one — + # presents, and is absent on a host holding an operator-placed one — # ignored there for the same reason, rather than being a dependency - # only some hives may declare. + # only some hosts may declare. after = [ "container@${baoCfg.machine}.service" "swarm-bao-pki.service" @@ -643,13 +661,12 @@ in # bundle: every other hive's swarm-service names chain to it. # `ca/pem` takes no token; the TLS client certificate is the # store's listener's demand, not bao's, so this presents the - # hive's own identity and never logs in with it. - # A hive without a readable identity fails here, naming the - # options: every hive must hold one. + # services-issuer pair and never logs in with it. Without a + # readable pair the unit fails here, naming the options. if [ "$want_svc" = 0 ]; then ${ - if haveHiveIdentity then - requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root" + if haveServicesIssuer then + requireServicesIssuer "fetch the store's ${baoServicesPkiMount} root" + '' echo "no swarm service domains configured — fetching only the store's pki root" err="$(mktemp)" @@ -683,7 +700,7 @@ in fi exit 0'' else - requireHiveIdentity "fetch the store's ${baoServicesPkiMount} root" + requireServicesIssuer "fetch the store's ${baoServicesPkiMount} root" } fi @@ -725,17 +742,7 @@ in exit 0 fi - ${ - if cfg.baoClientCertFile == null || cfg.baoClientKeyFile == null then - '' - echo "no bao client certificate configured for this host, so the gateway's" >&2 - echo "swarm-services certificate cannot be requested from the store." >&2 - echo "Set services.hyperhive.deploy.hive-controller.tls.baoClient{Cert,Key}File" >&2 - echo "to a leaf the store's CA signed with CN=${baoDeploy.servicesIssuerCommonName}." >&2 - exit 1'' - else - "" - } + ${requireServicesIssuer "ask the store for the swarm-services leaf"} # `bao`'s own message is the only thing separating an unreachable # store from a refused identity from a role that would not issue diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index f0916b0b..b849d982 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -1950,6 +1950,43 @@ in ''; }; + servicesIssuerClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/services-issuer.pem"; + description = '' + Certificate `swarm-services-cert` presents to the store: to log in and + ask for the swarm-services leaf the gateway serves, and for the + handshake of the root fetch on a host fronting no swarm name. Its + subject must be + {option}`services.hyperhive.deploy.bao.servicesIssuerCommonName`. + + ⚠️ Neither the reader's + {option}`services.hyperhive.deploy.bao.clientCertFile` nor a hive's own + identity: this one's policy permits `update` on + `pki/issue/swarm-services` and nothing else, and it is the same on + every gateway host, whether or not a hive runs there. + + Required on every host whose gateway serves the self-signed leaf; + unset fails evaluation. + + ./glue-bao-tls.nix points it at the leaf it mints on the store's host. + On a gateway host without the store, it is a file the operator copies + from the store's host. A path, never a value. + ''; + }; + + servicesIssuerClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/services-issuer-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.servicesIssuerClientCertFile`. + A path, never a value. + ''; + }; + granterCommonName = lib.mkOption { type = lib.types.str; default = "bao-granter"; diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 4907d2e4..8ca4dc2b 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -1219,7 +1219,9 @@ let && t.baoHiveIdentityKeyFile == "/var/lib/swarm-bao-pki/hive-leaf-key.pem" # Its own leaf, not the reader's or the services issuer's. && t.baoHiveIdentityCertFile != baoGrantHere.services.hyperhive.deploy.bao.clientCertFile - && t.baoHiveIdentityCertFile != t.baoClientCertFile; + && + t.baoHiveIdentityCertFile + != baoGrantHere.services.hyperhive.deploy.bao.servicesIssuerClientCertFile; } { name = "a gateway-leaf role prefix outside swarm-* is refused, naming the option"; diff --git a/nix/module-eval/forge-placement.nix b/nix/module-eval/forge-placement.nix index ed86dea2..ce13d01d 100644 --- a/nix/module-eval/forge-placement.nix +++ b/nix/module-eval/forge-placement.nix @@ -73,6 +73,8 @@ let deploy.forgejo.ci.jobTimeout = "2h"; deploy.bao.ciRunnerClientCertFile = "/etc/pki/bao-ci-runner.pem"; deploy.bao.ciRunnerClientKeyFile = "/etc/pki/bao-ci-runner-key.pem"; + deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem"; + deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem"; }; # The old name alone, with no store identity placed and none defaulted. ciUnderForgeNoIdentity = hive { @@ -88,6 +90,8 @@ let swarm.forge.ci.package = pkgs.hello; deploy.bao.ciRunnerClientCertFile = "/etc/pki/bao-ci-runner.pem"; deploy.bao.ciRunnerClientKeyFile = "/etc/pki/bao-ci-runner-key.pem"; + deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem"; + deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem"; }; runsForge = m: m.services.hyperhive.deploy.forgejo.enable && m.containers ? hive-forge; diff --git a/nix/module-eval/hive-tls.nix b/nix/module-eval/hive-tls.nix index d01c8cd4..672f5537 100644 --- a/nix/module-eval/hive-tls.nix +++ b/nix/module-eval/hive-tls.nix @@ -41,11 +41,14 @@ let }; # Fronting none of the swarm's names, reading a store it does not run, - # with the reader identity and the hive's own identity placed by hand. + # with the reader identity, the services-issuer pair and the hive's own + # identity placed by hand. noService = hiveWithController { deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem"; + deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem"; + deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem"; deploy.hive-controller.tls.baoHiveIdentityCertFile = "/etc/pki/bao-hive.pem"; deploy.hive-controller.tls.baoHiveIdentityKeyFile = "/etc/pki/bao-hive-key.pem"; }; @@ -55,6 +58,23 @@ let noIdentity = hiveWithController { }; noIdFetch = noIdentity.systemd.services.swarm-services-cert.script; + # A gateway on a host with no hive and no store, fronting no swarm name, + # with the services-issuer pair placed by hand; and the same host without it. + gatewayNoHive = hive { + gateway.enable = true; + deploy.bao.serverCaFile = "/etc/pki/bao-ca.pem"; + deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem"; + deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem"; + }; + gatewayNoHiveNoPair = hive { gateway.enable = true; }; + + # The store's host with the pair still named under its old path. + renamedPair = hiveWithController { + deploy.singleHostSwarm = true; + deploy.hive-controller.tls.baoClientCertFile = "/etc/pki/old-issuer.pem"; + deploy.hive-controller.tls.baoClientKeyFile = "/etc/pki/old-issuer-key.pem"; + }; + # The `want_svc=0` branch of a rendered script, up to the `fi` that closes it (the # only one followed by a blank line). noSvcBranch = @@ -190,42 +210,124 @@ let && lib.hasInfix ''cat "$svcroot" > "$bundle"'' noService.systemd.services.hive-tls-ca.script; } { - # The hive's own identity, not its reader identity: the reader's policy - # reads every secret the hive is entitled to. - name = "the root fetch reaches the store with the hive's own identity and server CA"; + # The services-issuer pair, not the reader identity, whose policy reads + # every secret the hive is entitled to, and not the hive's own. + name = "the root fetch reaches the store with the services-issuer pair and server CA"; ok = lib.hasInfix "--cacert ${lib.escapeShellArg "/etc/pki/bao-ca.pem"}" noSvcFetch && lib.hasInfix ( - "--cert ${lib.escapeShellArg "/etc/pki/bao-hive.pem"}" - + " --key ${lib.escapeShellArg "/etc/pki/bao-hive-key.pem"}" + "--cert ${lib.escapeShellArg "/etc/pki/bao-services-issuer.pem"}" + + " --key ${lib.escapeShellArg "/etc/pki/bao-services-issuer-key.pem"}" ) noSvcFetch - && !(lib.hasInfix "/etc/pki/bao-client.pem" (noSvcBranch noSvcFetch)); + && !(lib.hasInfix "/etc/pki/bao-client.pem" (noSvcBranch noSvcFetch)) + && !(lib.hasInfix "/etc/pki/bao-hive" (noSvcBranch noSvcFetch)); } { # No identity to present: the unit fails, naming both options, rather # than leaving the bundle without the root behind a green unit. The - # control is the same hive with an identity, whose branch fetches. - name = "a hive fronting no swarm name with no store identity fails the unit, naming both options"; + # control is the same hive with the pair, whose branch fetches. + name = "a host fronting no swarm name with no services-issuer pair fails the unit, naming both options"; ok = noIdentity.services.hyperhive.swarm.localServiceDomains == [ ] - && noIdentity.services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile == null + && noIdentity.services.hyperhive.deploy.bao.servicesIssuerClientCertFile == null && !(lib.hasInfix "/v1/pki/ca/pem" noIdFetch) - && lib.hasInfix "baoHiveIdentityCertFile (unset)" (noSvcBranch noIdFetch) - && lib.hasInfix "baoHiveIdentityKeyFile (unset)" (noSvcBranch noIdFetch) + && lib.hasInfix "servicesIssuerClientCertFile (unset)" (noSvcBranch noIdFetch) + && lib.hasInfix "servicesIssuerClientKeyFile (unset)" (noSvcBranch noIdFetch) && lib.hasSuffix "exit 1" (lib.removeSuffix "\n" (noSvcBranch noIdFetch)) && !(lib.hasInfix "exit 0" (noSvcBranch noIdFetch)) && !(lib.hasInfix "not staged" noIdFetch) && lib.hasInfix "/v1/pki/ca/pem" (noSvcBranch noSvcFetch); } { - # An identity named but not on disk fails before curl, naming the paths. - name = "a hive fronting no swarm name whose identity files are absent fails the unit, naming the paths"; + # A pair named but not on disk fails before curl, naming the paths. + name = "a host fronting no swarm name whose services-issuer files are absent fails the unit, naming the paths"; ok = - lib.hasInfix "for f in /etc/pki/bao-hive.pem /etc/pki/bao-hive-key.pem; do" (noSvcBranch noSvcFetch) + lib.hasInfix "for f in /etc/pki/bao-services-issuer.pem /etc/pki/bao-services-issuer-key.pem; do" ( + noSvcBranch noSvcFetch + ) && lib.hasInfix ''[ -r "$f" ] || missing="$missing $f"'' (noSvcBranch noSvcFetch) && lib.hasInfix "cannot read$missing" (noSvcBranch noSvcFetch) - && lib.hasInfix "baoHiveIdentityCertFile (/etc/pki/bao-hive.pem)" (noSvcBranch noSvcFetch) - && lib.hasInfix "baoHiveIdentityKeyFile (/etc/pki/bao-hive-key.pem)" (noSvcBranch noSvcFetch); + && lib.hasInfix "servicesIssuerClientCertFile (/etc/pki/bao-services-issuer.pem)" ( + noSvcBranch noSvcFetch + ) + && lib.hasInfix "servicesIssuerClientKeyFile (/etc/pki/bao-services-issuer-key.pem)" ( + noSvcBranch noSvcFetch + ); + } + { + # A gateway host with no hive, fronting no swarm name, the pair placed + # by hand. The control is the hive host, whose gateway-leaf unit does + # present the hive identity, so the scan can find one. + name = "on a host without a hive the swarm-services units present no hive identity, and the root fetch runs"; + ok = + let + hiveRefs = [ + "baoHiveIdentity" + "hive-leaf" + "/etc/pki/bao-hive" + ]; + refsHive = + units: + lib.any ( + u: lib.any (r: lib.hasInfix r u.script || lib.hasInfix r (builtins.toJSON u.environment)) hiveRefs + ) units; + units = gatewayNoHive.systemd.services; + s = units.swarm-services-cert; + in + !gatewayNoHive.services.hyperhive.deploy.hive-controller.enable + && gatewayNoHive.services.hyperhive.swarm.localServiceDomains == [ ] + && !(refsHive [ + s + units.swarm-services-cert-renew + ]) + && lib.hasInfix "/v1/pki/ca/pem" (noSvcBranch s.script) + && lib.hasInfix "--cert ${lib.escapeShellArg "/etc/pki/bao-services-issuer.pem"}" ( + noSvcBranch s.script + ) + && s.environment.BAO_CLIENT_CERT == "/etc/pki/bao-services-issuer.pem" + && refsHive [ noService.systemd.services.hive-tls-resign ]; + } + { + # The control is the same host with the pair placed, and the store's + # host, where the glue supplies it. + name = "a self-signed gateway with no services-issuer pair fails evaluation, naming both options"; + ok = + let + refused = + m: + lib.any ( + a: + !a.assertion + && lib.hasInfix "services.hyperhive.deploy.bao.servicesIssuerClientCertFile" a.message + && lib.hasInfix "services.hyperhive.deploy.bao.servicesIssuerClientKeyFile" a.message + ) m.assertions; + in + refused gatewayNoHiveNoPair + && refused noIdentity + && !(refused gatewayNoHive) + && !(refused noService) + && !(refused allLocal) + && + allLocal.services.hyperhive.deploy.bao.servicesIssuerClientCertFile + == "/var/lib/swarm-bao-pki/services-issuer.pem"; + } + { + # The old path's value reaches the new option over the store glue's + # `mkDefault`, and the unit presents it. The control is the same store + # host on its defaults. + name = "the services-issuer pair set under deploy.hive-controller.tls reaches deploy.bao and the unit"; + ok = + let + b = renamedPair.services.hyperhive.deploy.bao; + env = renamedPair.systemd.services.swarm-services-cert.environment; + in + b.servicesIssuerClientCertFile == "/etc/pki/old-issuer.pem" + && b.servicesIssuerClientKeyFile == "/etc/pki/old-issuer-key.pem" + && env.BAO_CLIENT_CERT == "/etc/pki/old-issuer.pem" + && env.BAO_CLIENT_KEY == "/etc/pki/old-issuer-key.pem" + && + allLocal.systemd.services.swarm-services-cert.environment.BAO_CLIENT_CERT + == "/var/lib/swarm-bao-pki/services-issuer.pem"; } { # The fetch retries against a sealed store; a gateway `Requires=` on it @@ -469,7 +571,9 @@ let && leafUnit.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/hive-leaf.pem" && leafUnit.environment.BAO_CLIENT_KEY == "/var/lib/swarm-bao-pki/hive-leaf-key.pem" && t.baoHiveIdentityCertFile == leafUnit.environment.BAO_CLIENT_CERT - && leafUnit.environment.BAO_CLIENT_CERT != t.baoClientCertFile + && + leafUnit.environment.BAO_CLIENT_CERT + != allLocal.services.hyperhive.deploy.bao.servicesIssuerClientCertFile && leafUnit.environment.BAO_CLIENT_CERT != allLocal.services.hyperhive.deploy.bao.clientCertFile; } { diff --git a/nix/module-eval/nats-tls.nix b/nix/module-eval/nats-tls.nix index 257843a7..051a8155 100644 --- a/nix/module-eval/nats-tls.nix +++ b/nix/module-eval/nats-tls.nix @@ -67,14 +67,16 @@ let # address set by hand: what every hive but one in a multi-host swarm looks # like. The controller is on too, since it may run away from the queue. # - # The forge's secret and the controller's store identity are what a hive - # away from authelia and the store already has to be handed, and none is an - # address; without them this hive would fail assertions that have nothing to - # do with the queue. + # The forge's secret, the controller's store identity and the gateway's + # services-issuer pair are what a hive away from authelia and the store + # already has to be handed, and none is an address; without them this hive + # would fail assertions that have nothing to do with the queue. remoteSecrets = { deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; deploy.swarm-controller.baoClientCertFile = "/var/lib/swarm-controller/bao-client.pem"; deploy.swarm-controller.baoClientKeyFile = "/var/lib/swarm-controller/bao-client-key.pem"; + deploy.bao.servicesIssuerClientCertFile = "/etc/pki/bao-services-issuer.pem"; + deploy.bao.servicesIssuerClientKeyFile = "/etc/pki/bao-services-issuer-key.pem"; }; remote = hiveWithController ( lib.recursiveUpdate remoteSecrets { deploy.swarm-controller.enable = true; } diff --git a/nix/module-eval/swarm-otel-identity.nix b/nix/module-eval/swarm-otel-identity.nix index 99dcd677..78ae95cb 100644 --- a/nix/module-eval/swarm-otel-identity.nix +++ b/nix/module-eval/swarm-otel-identity.nix @@ -27,9 +27,12 @@ let # hive's telemetry with nothing to push authenticated with, which is the # already-supported degrade `haveCollectorSecret` names above the module's # `let`. What this fixture is for is checking the reading unit itself does - # not render, rather than rendering with an env var nothing filled in. + # not render, rather than rendering with an env var nothing filled in. Its + # gateway has a certificate of its own: a self-signed one needs the + # services-issuer identity. otelNoIdentity = hive { deploy.swarm-otel.enable = true; + gateway.tls.certDir = "/var/lib/acme/t.local"; swarm.authelia.url = "https://auth.example.invalid"; deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; };