diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 681d277c..b5f3c38a 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -250,7 +250,7 @@ round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store exists so the only secrets a host holds out of band are **certificates**, and -everything else is read with them. +a host reads everything else with them. Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: running Grafana requires `swarm.authelia.url` (this module disables its local @@ -319,7 +319,7 @@ internal CA deletes that file and names its own paths in `deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. A hive that reads from a store on **another** machine names the reader's half — `clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand, -plus one leaf per principal it runs (the options are listed below). Those are the +plus one leaf per principal it runs (the table below names the options). Those are the credentials that can't come out of the store, being what opens it; everything else a hive needs does. @@ -336,20 +336,55 @@ each and each holds a leaf, a role and a policy of its own: | `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services//oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | -The first two are written **per hive**, because the path they read carries a hive +The first two exist **per hive**, because the path they read carries a hive name and every hive runs its own reader. Their subjects are `-` and `-`; `swarm.nix` reserves both -composed spellings as hive names, so a hive can't be named into another hive's +composed spellings as hive names, so nobody can name a hive into another hive's role. The other two read a path that names a swarm service rather than a hive, so one role each is enough and their subjects are the flat `deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`. On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults -all eight options, and there is nothing to do. Elsewhere each leaf is issued from -that CA out of band and named here — one file per principal rather than one file +all eight options, and there is nothing to do. Elsewhere you issue each leaf from +that CA out of band and name it here — one file per principal rather than one file shared by four, which is the whole of what this buys. +Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four +readers used to render only where their leaf existed, so a hand-configured +remote-store hive that named the hive's own `clientCertFile` and missed a +principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing +the build. All four now refuse it, each naming its own option pair — see +[what a missing leaf costs](#what-a-missing-leaf-costs) below. + +### What a missing leaf costs + +Each of the four modules asserts its own pair, on the same condition: **this host +already reads the store** — it holds `deploy.bao.clientCertFile` and +`clientKeyFile` — **and the consumer is on, and this principal's pair is null**. +The refusal names the two options, so an operator acts on the message without +opening the nix. + +That condition fires on exactly one shape, the hand-configured remote-store hive +that named seven of the eight. It stays silent everywhere else, and each half of +that matters on its own: + +- **A host with no store identity at all** holds no `clientCertFile` either, so + the first clause is false. That's the supported no-store deployment, and also + the one an operator passes through while bringing a hive up — neither should + fail to evaluate. +- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as + defaults, so the third clause is false. Nothing to set, nothing to refuse. +- **A consumer that's off** — no homeserver, no collector, no Grafana — has no + unit to skip, so the second clause is false. Only the queue-credential reader + has no toggle of its own to check: every hive runs one for its own agents, so + reading the store at all is what asks for it. + +The collector keeps a separate degrade underneath this, unchanged: a host with no +store identity runs a collector with no OIDC client secret and still receives +telemetry. The assertion doesn't touch that shape, because that host has no +`clientCertFile` either. + ## How a reader reaches the store Every reader dials the same URL — `https://bao.:` — and on diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix index ca870766..3c0afcb6 100644 --- a/nix/host-modules/glue-matrix-bao-token.nix +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -49,6 +49,14 @@ let haveClientIdentity = baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null; + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. Only + # used to decide whether a missing per-principal leaf is a mistake or a + # deployment that has no store: a host holding neither is the supported + # no-store shape, and one holding this pair but not the pair above named + # seven of the eight options and stopped. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # Where the token lives in the store. A path, not a convention to guess at: # whoever writes it and whoever reads it must agree, and the agreement # belongs in one visible place. @@ -73,134 +81,178 @@ let matrixMachine = "hive-matrix"; in { - config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { - # Same rule as the unit's own gate: this reader exists on a host that has a - # client identity and a homeserver, which is not every host that runs the - # store, so the store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; + config = lib.mkMerge [ + # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole + # mechanism: the unit's arm is gated on `haveClientIdentity`, so an + # assertion written inside it could never be reached in the state it + # exists to report. + # + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the + # same refusal, named to this principal's own pair. What differs is the + # gate. Grafana asserts wherever Grafana runs, because a Grafana with no + # store identity has no way in at all; a homeserver with no store identity + # is a hive that has no store, which is supported. So this one additionally + # requires `hiveReaderIdentity`: the host demonstrably reads the store, and + # named every option but this pair. + (lib.mkIf (hyperhiveCfg.enable && deployCfg.matrix.enable && hiveReaderIdentity) { + assertions = [ + { + assertion = haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set) and runs a homeserver, so it needs the matrix appservice + token reader's own client identity: set both - systemd.services.swarm-bao-matrix-token = { - description = "fetch the matrix appservice token from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" + services.hyperhive.deploy.bao.matrixTokenClientCertFile + services.hyperhive.deploy.bao.matrixTokenClientKeyFile + + swarm-bao-matrix-token.service fetches this hive's appservice token + out of the store, and without these it is not rendered at all — + leaving the homeserver authenticating hive-c0re against whatever is + already on disk, which is a 401 on every request naming nothing. + + ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is + the hive's, and its grant reads every secret in the store; this role + reads the one appservice-token path. Pointing this option at the + hive's leaf would evaluate, deploy and log in — and undo the split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - before = [ "container@${matrixMachine}.service" ]; - wantedBy = [ "container@${matrixMachine}.service" ]; - path = [ - deployCfg.bao.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal. `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after — so a few short attempts cover it. - # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. - # That unit blocks nothing; this one is `Before=` the homeserver's - # container, and whether that ordering waits across an auto-restart is - # unverified — so an hours-long window would be a bet on an unknown, - # where a minute is not. A store still sealed after it keeps the degrade - # below, as today. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the read below. Stated here rather than - # left to systemd's default, so the number a boot waits on is in - # the file that waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; - BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail + }) - # A sealed or uninitialised store answers on the port and never - # answers the read, so "the store is up" is not the same as "the - # store can answer". `TimeoutStartSec` above is the bound; the - # homeserver only `Wants=` this unit, so hitting it degrades to - # keeping the local token rather than holding up the container. - # `bao`'s own message is the only thing separating a missing value - # from a refused identity from an unreachable host. This unit's - # degraded mode is correct for all three, so it reports which one - # rather than asserting all three in a sentence of ours — a reader - # that cannot say why it read nothing is indistinguishable from a - # broken one. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT + (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { + # Same rule as the unit's own gate: this reader exists on a host that has a + # client identity and a homeserver, which is not every host that runs the + # store, so the store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. + systemd.services.swarm-bao-matrix-token = { + description = "fetch the matrix appservice token from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + before = [ "container@${matrixMachine}.service" ]; + wantedBy = [ "container@${matrixMachine}.service" ]; + path = [ + deployCfg.bao.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal. `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after — so a few short attempts cover it. + # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. + # That unit blocks nothing; this one is `Before=` the homeserver's + # container, and whether that ordering waits across an auto-restart is + # unverified — so an hours-long window would be a bet on an unknown, + # where a minute is not. A store still sealed after it keeps the degrade + # below, as today. # - # Fails LOUDLY, unlike the read below: the three states a login failure - # covers — store not up, sealed, role not written yet — are all things - # a retry fixes, and `Restart=on-failure` above is what retries. Exiting - # 0 here spends the whole boot on a condition that was seconds old. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the read below. Stated here rather than + # left to systemd's default, so the number a boot waits on is in + # the file that waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; + BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # A sealed or uninitialised store answers on the port and never + # answers the read, so "the store is up" is not the same as "the + # store can answer". `TimeoutStartSec` above is the bound; the + # homeserver only `Wants=` this unit, so hitting it degrades to + # keeping the local token rather than holding up the container. + # `bao`'s own message is the only thing separating a missing value + # from a refused identity from an unreachable host. This unit's + # degraded mode is correct for all three, so it reports which one + # rather than asserting all three in a sentence of ours — a reader + # that cannot say why it read nothing is indistinguishable from a + # broken one. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the read below: the three states a login failure + # covers — store not up, sealed, role not written yet — are all things + # a retry fixes, and `Restart=on-failure` above is what retries. Exiting + # 0 here spends the whole boot on a condition that was seconds old. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 fi - exit 1 - fi - export BAO_TOKEN + export BAO_TOKEN - if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then - echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then + echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - if [ -z "$token" ]; then - echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 - exit 0 - fi + if [ -z "$token" ]; then + echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 + exit 0 + fi - umask 077 - printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + umask 077 + printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - # Re-stamp the registration file from the token just written. The - # token is half an agreement — the registration the homeserver loads - # has to carry the same value — so writing the file and stopping - # would leave the homeserver authenticating hive-c0re against - # whatever activation put there: a 401 on every request, naming - # nothing. Unconditional rather than on-change, because this unit - # has no way to know what the registration currently says. - # - # hive-matrix's own renderer rather than a `printf` here, so the - # registration's shape has one home. - ${deployCfg.matrix.appserviceRegistrationScript} - ''; - }; - }; + # Re-stamp the registration file from the token just written. The + # token is half an agreement — the registration the homeserver loads + # has to carry the same value — so writing the file and stopping + # would leave the homeserver authenticating hive-c0re against + # whatever activation put there: a 401 on every request, naming + # nothing. Unconditional rather than on-change, because this unit + # has no way to know what the registration currently says. + # + # hive-matrix's own renderer rather than a `printf` here, so the + # registration's shape has one home. + ${deployCfg.matrix.appserviceRegistrationScript} + ''; + }; + }) + ]; } diff --git a/nix/host-modules/glue-queue-agent-credential.nix b/nix/host-modules/glue-queue-agent-credential.nix index 1950f959..8fa2dc06 100644 --- a/nix/host-modules/glue-queue-agent-credential.nix +++ b/nix/host-modules/glue-queue-agent-credential.nix @@ -51,6 +51,14 @@ let haveClientIdentity = baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null; + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. Only + # used to decide whether a missing per-principal leaf is a mistake or a + # deployment that has no store: a host holding neither is the supported + # no-store shape, and one holding this pair but not the pair above named + # seven of the eight options and stopped. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir; secretFile = "${credentialDir}/secret"; clientIdFile = "${credentialDir}/client_id"; @@ -94,144 +102,192 @@ in }; }; - config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { - # Same rule as the unit's own gate: this reader exists on any host holding - # a client identity, which is not every host that runs the store, so the - # store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; + config = lib.mkMerge [ + # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole + # mechanism: the unit's arm is gated on `haveClientIdentity`, so an + # assertion written inside it could never be reached in the state it + # exists to report. + # + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the + # same refusal, named to this principal's own pair. Where Grafana's gate + # is `deploy.grafana.enable`, this reader has no toggle of its own to + # check: every hive runs one for its own agents, so reading the store at + # all is what asks for it. Hence `hiveReaderIdentity` alone — a host + # holding no hive leaf has no store to read and nothing is missing. + (lib.mkIf (hyperhiveCfg.enable && hiveReaderIdentity) { + assertions = [ + { + assertion = haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set), so it needs the agent queue credential reader's own client + identity: set both - systemd.services.swarm-bao-queue-agent = { - description = "fetch this hive's agent queue credential from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" - ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - # Ordered before hive-c0re, so no agent container renders ahead of an - # attempt at its credential. `Wants=`, not `Requires=`: a store this - # unit can't reach delays hive-c0re's start by its own start-limit - # window (`TimeoutStartSec`, retried up to `startLimitBurst` times - # below) rather than failing it — hive-c0re starts once that window - # elapses, whatever credential is or isn't on disk by then. - before = [ "hive-c0re.service" ]; - wantedBy = [ - "multi-user.target" - "hive-c0re.service" - ]; - path = [ - baoDeploy.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal: `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after, so a few short attempts cover it. - # An hours-long window would be a bet on a store that is sealed, and the - # degrade below is already correct for that. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the reads below. Stated here rather than left to - # systemd's default, so the number a boot waits on is in the file that - # waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; - BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail + services.hyperhive.deploy.bao.queueAgentClientCertFile + services.hyperhive.deploy.bao.queueAgentClientKeyFile - # `bao`'s own message is the only thing separating a missing value from - # a refused identity from an unreachable host. This unit's degraded - # mode is correct for all three, so it reports which one rather than - # asserting all three in a sentence of ours. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT + swarm-bao-queue-agent.service fetches this hive's agent queue + credential out of the store, and without these it is not rendered + at all — leaving hive-c0re with no queue credential and no unit + that would ever have written one. - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. + Every hive runs this reader for its own agents, so unlike the other + three principals there is no per-service toggle that turns it off: + a hive that reads the store at all is a hive that needs this pair. + + ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one + is the hive's, and its grant reads every secret in the store; this + role reads the one queue-credential path. Pointing this option at + the hive's leaf would evaluate, deploy and log in — and undo the + split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } + ]; + }) + + (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { + # Same rule as the unit's own gate: this reader exists on any host holding + # a client identity, which is not every host that runs the store, so the + # store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; + + systemd.services.swarm-bao-queue-agent = { + description = "fetch this hive's agent queue credential from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + # Ordered before hive-c0re, so no agent container renders ahead of an + # attempt at its credential. `Wants=`, not `Requires=`: a store this + # unit can't reach delays hive-c0re's start by its own start-limit + # window (`TimeoutStartSec`, retried up to `startLimitBurst` times + # below) rather than failing it — hive-c0re starts once that window + # elapses, whatever credential is or isn't on disk by then. + before = [ "hive-c0re.service" ]; + wantedBy = [ + "multi-user.target" + "hive-c0re.service" + ]; + path = [ + baoDeploy.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal: `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after, so a few short attempts cover it. + # An hours-long window would be a bet on a store that is sealed, and the + # degrade below is already correct for that. # - # Fails LOUDLY, unlike the reads below: the three states a login - # failure covers — store not up, sealed, role not written yet — are all - # things a retry fixes, and `Restart=on-failure` above is what retries. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the reads below. Stated here rather than left to + # systemd's default, so the number a boot waits on is in the file that + # waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; + BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # `bao`'s own message is the only thing separating a missing value from + # a refused identity from an unreachable host. This unit's degraded + # mode is correct for all three, so it reports which one rather than + # asserting all three in a sentence of ours. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the reads below: the three states a login + # failure covers — store not up, sealed, role not written yet — are all + # things a retry fixes, and `Restart=on-failure` above is what retries. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 fi - exit 1 - fi - export BAO_TOKEN + export BAO_TOKEN - # Two reads of one object rather than one `-format=json` parsed with - # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot - # actually disagree — the client id is derived from this hive's name, so - # a rotation landing between these two calls changes the secret and - # rewrites the same id. - if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # Two reads of one object rather than one `-format=json` parsed with + # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot + # actually disagree — the client id is derived from this hive's name, so + # a rotation landing between these two calls changes the secret and + # rewrites the same id. + if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - # Both or neither, for the reason `QueueConfig::from_env` refuses a - # half-set environment: a client that finds one of the two comes up - # "fine" and never connects. - if [ -z "$secret" ] || [ -z "$client_id" ]; then - echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 - exit 0 - fi + # Both or neither, for the reason `QueueConfig::from_env` refuses a + # half-set environment: a client that finds one of the two comes up + # "fine" and never connects. + if [ -z "$secret" ] || [ -z "$client_id" ]; then + echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 + exit 0 + fi - install -d -m 0755 ${lib.escapeShellArg credentialDir} + install -d -m 0755 ${lib.escapeShellArg credentialDir} - umask 077 - printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} - chmod 0600 ${lib.escapeShellArg secretFile} + umask 077 + printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} + chmod 0600 ${lib.escapeShellArg secretFile} - # `0644` on purpose: an OIDC client id is presented to the token - # endpoint on every connection and is public by construction. - printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} - chmod 0644 ${lib.escapeShellArg clientIdFile} - ''; - }; - }; + # `0644` on purpose: an OIDC client id is presented to the token + # endpoint on every connection and is public by construction. + printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} + chmod 0644 ${lib.escapeShellArg clientIdFile} + ''; + }; + }) + ]; } diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index e0db2ce7..c53bfd21 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -242,12 +242,14 @@ let # A reader of the store is defined by holding a certificate the store # accepts, never by standing next to it — the rule # ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical- - # looking flag, this one stays outside `assertions`: the store-reading unit - # below simply does not render without it, the same choice + # looking flag, this one does not gate an assertion on its own: the + # store-reading unit below simply does not render without it, the same choice # ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for - # their own optional readers, because a collector with no client identity is + # their own readers, because a collector with no client identity is # `haveCollectorSecret = false` above, and that is already a supported, - # merely degraded shape rather than a service with no way in at all. + # merely degraded shape rather than a service with no way in at all. What + # IS asserted is narrower and lives in `assertions` below — see + # `hiveReaderIdentity`. # # 🩸 The collector's OWN leaf, not `clientCertFile` — the hive's, which four # units used to share. Bao matches a cert-auth role on the CN, so one leaf for @@ -257,6 +259,16 @@ let haveClientIdentity = baoDeploy.otelOidcClientCertFile != null && baoDeploy.otelOidcClientKeyFile != null; + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. It is + # what separates the degrade the comment above describes from the mistake the + # assertion below reports: a collector on a host holding NO store identity is + # the supported shape, and one on a host that demonstrably reads the store + # named seven of the eight options and stopped. Before the four-way split + # that second host rendered this unit off `clientCertFile`, so it is a silent + # regression rather than a choice anyone made. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # Where the publisher on authelia's host leaves this client's secret — # composed from the same swarm-wide `clientId` the registration in # ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of @@ -732,11 +744,14 @@ in # value. # # ⚠️ Renders only where `haveClientIdentity` holds, unlike - # ./swarm-grafana.nix's equivalent unit. That module asserts the identity - # because a Grafana with none has no way in at all; this collector without - # one is `haveCollectorSecret = false` above — already a supported, - # merely degraded shape, so the unit that would fetch a credential simply - # does not exist rather than refusing the build for want of one. + # ./swarm-grafana.nix's equivalent unit. That module refuses any host that + # runs Grafana without the identity, because a Grafana with none has no way + # in at all; this collector without one is `haveCollectorSecret = false` + # above — already a supported, merely degraded shape, so the unit that would + # fetch a credential simply does not exist rather than refusing the build + # for want of one. The one case that IS refused is the host that already + # holds the hive's leaf and is missing only this pair, which is a silent + # regression rather than that degrade — `hiveReaderIdentity` above. systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity { description = "fetch the swarm collector's OIDC client secret from the swarm secret store"; # Every one of these names a unit that exists only where the store runs. @@ -860,6 +875,45 @@ in systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; assertions = [ + { + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — + # the same refusal, named to this principal's own pair. What differs is + # the `!hiveReaderIdentity ||` guard, and it is what keeps the degrade + # the flag's own comment describes intact: Grafana refuses any host + # that runs it without a leaf, because a Grafana with no SSO has no way + # in at all, while a collector with none still receives telemetry. So + # this fires only where the host already reads the store and is missing + # this one pair — which before the four-way split rendered the unit off + # `clientCertFile`, and now silently does not. + assertion = !hiveReaderIdentity || haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set) and runs the swarm collector, so it needs the collector's own + client identity: set both + + services.hyperhive.deploy.bao.otelOidcClientCertFile + services.hyperhive.deploy.bao.otelOidcClientKeyFile + + swarm-bao-otel-oidc.service fetches the collector's OIDC client + secret out of the store, and without these it is not rendered at all + — so this collector would authenticate to nothing, quietly, on a host + that has everything it needs to fetch the secret. + + A collector on a host holding NO store identity is a different and + supported shape: it runs without a client secret and still receives + telemetry. That is not this host. + + ⚠️ The collector's OWN leaf, not deploy.bao.clientCertFile. That one + is the hive's, and its grant reads every secret in the store; this + role reads the one path this collector's client secret lives at. + Pointing this option at the hive's leaf would evaluate, deploy and + log in — and undo the split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } # ⚠️ An assertion that this collector has "somewhere to send" was REMOVED # rather than relaxed: it read the store's *per-host* enable, so it # rejected at eval the very deployment the stores are reached by domain diff --git a/nix/module-eval/bao-matrix-reader.nix b/nix/module-eval/bao-matrix-reader.nix index dd9470a2..e000ea65 100644 --- a/nix/module-eval/bao-matrix-reader.nix +++ b/nix/module-eval/bao-matrix-reader.nix @@ -53,6 +53,37 @@ let # needs it, and defining it here rather than importing keeps each group's # fixture set its own, as ./lib.nix asks. matrixNoBaoIdentity = hive { deploy.matrix.enable = true; }; + + # 🩸 The hand-configured remote reader that named seven of the eight options. + # `baoRemoteReader` above is the same deployment done right; this one holds + # the hive's own leaf, so it demonstrably reads the store, and is missing both + # per-principal pairs. Before the four-way split this host rendered both units + # off `clientCertFile` alone, so what it has now is a silent regression rather + # than any shape an operator chose — which is what the refusal arms below are + # about. Kept as one fixture rather than two because both refusals fire on it + # and each arm names which. + baoRemoteReaderMissingLeaves = hive { + deploy.matrix.enable = true; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # The same omission on a hive that does NOT run a homeserver. Separates the + # matrix refusal's `deploy.matrix.enable` clause from the queue refusal, which + # has no toggle to check — an arm below reads exactly one refusal off it. + remoteReaderNoMatrixMissingLeaves = hive { + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # Did a module refuse this host, and over which option. An assertion is a + # config VALUE until something forces it — `.config` never throws — so a + # fixture in a state the module refuses stays evaluable and the refusal reads + # back as data. Matched on the option name the message names rather than on + # its prose, because the option name is the part an operator has to act on + # and a message that stopped naming it would be the actual defect. The same + # helper ./grafana.nix uses for `swarm-grafana.nix`'s own refusal. + refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; cases = [ { # A login failure is the store being unreachable, sealed, or not yet @@ -373,6 +404,63 @@ let in !(g ? admin_execute) || g.admin_execute == [ ]; } + { + # 🩸 The arm the whole refusal exists for. Both readers are gated on their + # own leaf, so the way this regresses is the build going green and three + # units rendering where four should — which no presence check on a + # rendered unit can see, because the unit that is missing is the evidence. + # Read as a refusal naming each option, so an operator acts on the message + # without opening the nix. + name = "a remote reader missing the per-principal leaves is refused, naming both options"; + ok = + refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientCertFile" + && refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientKeyFile" + && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile" + && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile"; + } + { + # The matrix refusal's own gate, which the queue refusal does not have. + # Without this arm the two are indistinguishable on the fixture above. + name = "the queue refusal needs no homeserver, and the matrix refusal stays quiet without one"; + ok = + refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile" + && !(refusedOver remoteReaderNoMatrixMissingLeaves "matrixTokenClientCertFile"); + } + { + # ⚠️ The arm that keeps the refusal from being worse than the silence it + # replaced. A hive with NO store identity is the supported no-store + # deployment and also the state an operator passes through bringing a hive + # up — neither may fail to evaluate. Asserted as "no refusal names any of + # the four options", not as "this one fixture is fine", because the way + # this breaks is a gate widened to the principal's leaf alone. + name = "a hive with no store identity at all is refused over none of the per-principal leaves"; + ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [ + "matrixTokenClientCertFile" + "matrixTokenClientKeyFile" + "queueAgentClientCertFile" + "queueAgentClientKeyFile" + ]; + } + { + # The other half of the same guard, and the one an operator meets far more + # often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults + # all eight, so there is nothing to name and nothing to refuse. Paired with + # the fully-named remote reader, which is the same deployment done by hand. + name = "neither a store host nor a correctly-named remote reader is refused"; + ok = + lib.all + ( + m: + lib.all (option: !(refusedOver m option)) [ + "matrixTokenClientCertFile" + "queueAgentClientCertFile" + ] + ) + [ + baoWithMatrix + baoRemoteReader + ]; + } ]; in runGroup "bao-matrix-reader" cases diff --git a/nix/module-eval/swarm-otel-identity.nix b/nix/module-eval/swarm-otel-identity.nix index 67c89357..99dcd677 100644 --- a/nix/module-eval/swarm-otel-identity.nix +++ b/nix/module-eval/swarm-otel-identity.nix @@ -34,6 +34,40 @@ let deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; }; + # 🩸 The shape the degrade above must not swallow: a collector on a host that + # demonstrably READS the store — it holds the hive's own leaf — and is missing + # only the collector's own pair. Before the four-way split this host rendered + # the reading unit off `clientCertFile` alone, so the silence it gets now is a + # regression rather than the supported degrade `otelNoIdentity` stands for. The + # two fixtures differ in exactly that one pair, which is what lets the arms + # below separate a refusal from a degrade. + otelReaderMissingLeaf = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # The same deployment named in full, which must stay buildable. Pairs with the + # fixture above so the refusal is pinned to the omission and not to reading a + # remote store at all. + otelReaderNamedInFull = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; + }; + + # Did the module refuse this host, and over which option. An assertion is a + # config VALUE until something forces it — `.config` never throws — so a + # fixture in a refused state stays evaluable and the refusal reads back as + # data. Matched on the option name the message names rather than on its prose: + # the option name is the part an operator has to act on, and a message that + # stopped naming it would be the actual defect. + refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; + # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. @@ -162,6 +196,38 @@ let && !(builtins.elem "otlphttp/victoriametrics" used) && lib.all (e: s.exporters ? ${e}) used; } + { + # 🩸 The refusal, and the only way this module's omission is visible at + # all: the reading unit is gated on its own leaf, so the regression is a + # green build with the unit absent — and the missing unit is the evidence. + # Read as a refusal naming both options, so an operator acts on the + # message without opening the nix. + name = "a store reader missing the collector's own leaf is refused, naming both options"; + ok = + refusedOver otelReaderMissingLeaf "otelOidcClientCertFile" + && refusedOver otelReaderMissingLeaf "otelOidcClientKeyFile"; + } + { + # ⚠️ The arm that keeps the refusal from eating the degrade beside it. A + # collector on a host with NO store identity still receives every hive's + # telemetry and is a supported deployment — `otelNoIdentity` is that + # fixture, and it differs from the refused one only in the hive's own + # leaf. Widening the gate to the collector's leaf alone would reject it, + # which is what this pins. + name = "a collector with no store identity at all is not refused over the collector's leaf"; + ok = + !(refusedOver otelNoIdentity "otelOidcClientCertFile") + && !(refusedOver otelNoIdentity "otelOidcClientKeyFile"); + } + { + # The other two shapes that must stay buildable: the same remote reader + # named in full, and a host that mints its own PKI and therefore has all + # eight as defaults from ./host-modules/glue-bao-tls.nix. + name = "neither a fully-named remote reader nor a store host is refused"; + ok = + !(refusedOver otelReaderNamedInFull "otelOidcClientCertFile") + && !(refusedOver otelRemoteAuthelia "otelOidcClientCertFile"); + } ]; in runGroup "swarm-otel-identity" cases