From f1445b4c8b546af78842753b461365a137e03349 Mon Sep 17 00:00:00 2001 From: atlas Date: Mon, 21 Sep 2026 23:44:59 +0200 Subject: [PATCH 1/3] swarm-bao: give each hive-cert consumer its own bao identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four units read one path each out of the store, and all four logged in holding `deploy.bao.clientCertFile` — the hive's own leaf. Bao identifies a principal by the subject of the certificate it presents, so four readers behind one certificate were ONE principal, and the only grant expressible was the union of what the four need: read on `swarm/agents/*`, `swarm/hives//*` and `swarm/services/*`. The unit fetching Grafana's OIDC client secret could fetch every agent credential in the swarm; the one fetching this hive's matrix token could fetch Grafana's. Least privilege was not misconfigured here, it was unrepresentable. Each now holds a leaf, a cert-auth role and a policy of its own, and each policy is the single `secret/data/…` path that unit's own script names — spelled to the leaf, not to a prefix, the way matrix-ctl's already is. Following the four exemplars in-tree rather than building a mechanism: `signLeaf` mints the leaves, `swarm-bao.nix` writes the roles from the bootstrap token, the consumers name their own pair. Two of the four are written PER HIVE and two are not, which is the shape of the paths rather than a preference. A matrix appservice token and a queue credential live under `swarm/hives//` and every hive runs a reader for its own, so one role for all of them would have to be granted `hives/*` — letting one hive read another's, a reach no hive has today. An OIDC client secret lives under `swarm/services//` and a swarm registers each exactly once, so one role each is enough. The per-hive subjects are `-` and swarm.nix reserves every composed spelling as a hive name, so a hive cannot be named into another hive's role. The shared leaf stays: hive-c0re still passes it into its container, the `bao` CLI wrapper still defaults to it, and the three `glue-*-bao-identity.nix` files derive the PKI directory from it. module-eval-bao-grants gains a negative arm per principal — each pins the three stanzas the hive's leaf carried and the two wildcards a later widening would reach for, so a policy that grows fails here rather than in a store. Plus the consuming side: repointing a unit back at the hive's leaf would evaluate, deploy and log in, and silently restore the union. A hive that reads a store on another machine now places one leaf per principal instead of one shared by four. That cost is the point, and docs/swarm/secrets.md lists the pairs. --- docs/getting-started/setup.md | 15 +- docs/swarm/secrets.md | 68 +++- nix/host-modules/glue-bao-tls.nix | 48 +++ nix/host-modules/glue-matrix-bao-token.nix | 15 +- .../glue-queue-agent-credential.nix | 15 +- nix/host-modules/swarm-bao.nix | 342 ++++++++++++++++++ nix/host-modules/swarm-grafana.nix | 28 +- nix/host-modules/swarm-otel.nix | 13 +- nix/host-modules/swarm.nix | 15 +- nix/module-eval/bao-grants.nix | 256 +++++++++++++ nix/module-eval/bao-matrix-reader.nix | 10 + nix/module-eval/grafana.nix | 12 +- nix/module-eval/name-guards.nix | 59 +++ nix/module-eval/swarm-otel-core.nix | 6 + 14 files changed, 859 insertions(+), 43 deletions(-) diff --git a/docs/getting-started/setup.md b/docs/getting-started/setup.md index af3290d3..2225529e 100644 --- a/docs/getting-started/setup.md +++ b/docs/getting-started/setup.md @@ -170,16 +170,19 @@ What else you need depends on restart, with the keys `init` printed. The store serves TLS, and on a hive that deploys it you need do nothing: a -first-boot unit mints a CA of the store's own plus the two leaves it signs — -the store's server certificate and this host's client certificate — and points -`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at the store's -half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's. +first-boot unit mints a CA of the store's own plus the leaves it signs — the +store's server certificate, this host's client certificate, and one per service +principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and +`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and +`.serverCaFile` at the reader's, and each principal's own pair at its own leaf. Those are `mkDefault`s, so naming your own paths wins. Do that when your certificates come from a real internal CA; the store has no opinion about which. A hive that does **not** deploy the store names the reader's three -itself: the operator issues that leaf out of band, and it's the one credential -the store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is +itself, plus a pair for every principal it runs — see +[per-principal identities](../swarm/secrets.md#per-principal-identities) for the +list. The operator issues those leaves out of band; they're the credentials the +store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is **mTLS between services and the store**, a separate trust domain, because a store that took its identity from an authority it itself distributes could never come up before that authority. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 53bce3ad..681d277c 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -88,7 +88,7 @@ where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — only its own pushes to the stores go out unauthenticated and get refused there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without -`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice +`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for their own optional readers, rather than the hard assertion `swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by @@ -249,14 +249,14 @@ straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store -exists so a host holds **one** out-of-band secret, its client certificate, and -reads everything else with it. +exists so the only secrets a host holds out of band are **certificates**, and +everything else is read with them. Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: running Grafana requires `swarm.authelia.url` (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the -only way in), and it requires this host's `deploy.bao.clientCertFile` / -`clientKeyFile`. Each refusal names the option to set. Both used to be silent: +only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile` +/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is @@ -271,7 +271,8 @@ runs the collector, and registration lives separately in `glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so `swarm-otel.nix` doesn't assert -`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it +`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana +does — it doesn't render the reading unit without them, the shape `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for their own optional reads. @@ -290,10 +291,18 @@ A service's secret is one value for the whole swarm rather than one per hive, so it lives under the `services` prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a `deploy.*` fact, per-host by definition — so -there is nothing to scope the grant to. Every hive can therefore read every swarm -service's client secret; that's the cost of a shape where the service's host has -no store identity of its own to present, and it's stated in -`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves +there is nothing to scope the grant to. A hive's **own** leaf can therefore read +every swarm service's client secret, and that's stated in +`swarm-secret-client`'s `policy` module beside the grant itself. + +⚠️ **The readers no longer present it.** Four units used to log in with +`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a +principal by the subject of the certificate it presents — so four readers behind +one leaf were one principal holding the union of their needs. Each now holds a +leaf of its own, and a policy naming only the path that unit reads. See +[per-principal identities](#per-principal-identities) below. + +Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in. @@ -303,14 +312,43 @@ credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the hive CA and the swarm CA both, and has nothing to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix` -mints a CA that signs exactly two things, the store's server certificate and a -reader's client certificate, and distributes nothing. A deployment with a real +mints a CA that signs a fixed, short list of leaves — the store's server +certificate, the hive's own reader certificate, and one per service principal — +and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in `deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. A hive that reads from a store on **another** machine names the reader's half — -`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand. -It's the one credential that can't come out of the store, being what opens it; -everything else a hive needs does. +`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand, +plus one leaf per principal it runs (the options are listed below). Those are the +credentials that can't come out of the store, being what opens it; everything +else a hive needs does. + +### Per-principal identities + +Bao matches a cert-auth role on the certificate's subject, so a certificate is an +identity and sharing one merges the identities. These four units read one path +each and each holds a leaf, a role and a policy of its own: + +| unit | option pair under `deploy.bao.` | reads | +| ------------------------ | -------------------------------------------------------- | -------------------------------------------------- | +| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives//matrix/appservice-token` | +| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives//queue/agent` | +| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services//oidc/client` | +| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | + +The first two are written **per hive**, because the path they read carries a hive +name and every hive runs its own reader. Their subjects are +`-` and +`-`; `swarm.nix` reserves both +composed spellings as hive names, so a hive can't be named into another hive's +role. The other two read a path that names a swarm service rather than a hive, so +one role each is enough and their subjects are the flat +`deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`. + +On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults +all eight options, and there is nothing to do. Elsewhere each leaf is issued from +that CA out of band and named here — one file per principal rather than one file +shared by four, which is the whole of what this buys. ## How a reader reaches the store diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix index 819569bf..1afaed01 100644 --- a/nix/host-modules/glue-bao-tls.nix +++ b/nix/host-modules/glue-bao-tls.nix @@ -84,6 +84,25 @@ in clientCertFile = lib.mkDefault "${pkiDir}/client.pem"; clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem"; serverCaFile = lib.mkDefault "${pkiDir}/ca.pem"; + + # 🩸 Four readers that used to present `client.pem` above, each now + # pointed at a leaf of its own. The leaf is what bao sees, so this pairing + # is the whole of what turns "one principal with the union of four grants" + # into four principals with one grant each — see ./swarm-bao.nix's + # `perHiveReaders` block for the grants themselves. + # + # Defaulted here rather than in four `glue--bao-identity.nix` + # files: those exist where the consumer is a CONTAINER with a + # `deploy..*` namespace of its own to point at. These four are + # host units reading the store, which is the pairing this file already is. + matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem"; + matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem"; + queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem"; + queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem"; + grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem"; + grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem"; + otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem"; + otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem"; }; # Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates @@ -160,6 +179,35 @@ in # container an identity instead of lending it the hive's. [ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \ ${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth + + # 🩸 The four readers that used to present `client.pem` above. Each + # carries its own subject, which is the entire mechanism: bao matches a + # cert-auth role on the CN, so four units sharing one leaf were one + # principal holding the union of four grants, and the union included + # every agent credential in the swarm. + # + # ⚠️ The first two carry THIS hive's name in the subject, unlike the + # three service leaves above. Their grants name one hive's path, because + # a matrix token and a queue credential live under `swarm/hives//` + # and every hive runs a reader for its own; ./swarm-bao.nix writes one + # role per hive in the swarm directory to match. The two OIDC readers + # need no such segment — a client is registered once per swarm. + # + # Minted whether or not the consumer runs here, for the reason the three + # leaves above give: on a hive that does not run the store this is the + # file an operator copies, and a leaf that only appears where its + # consumer does is one nobody can copy from anywhere. + [ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \ + ${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth + + [ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \ + ${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth + + [ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \ + ${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth + + [ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \ + ${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth ''; }; }; diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix index 077c7b86..ca870766 100644 --- a/nix/host-modules/glue-matrix-bao-token.nix +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -38,7 +38,16 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: read on + # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when + # this unit reads one appservice token and nothing else. Its own leaf carries + # `-` and its role grants the single path + # below. + haveClientIdentity = + baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null; # Where the token lives in the store. A path, not a convention to guess at: # whoever writes it and whoever reads it must agree, and the agreement @@ -115,8 +124,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; + BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/glue-queue-agent-credential.nix b/nix/host-modules/glue-queue-agent-credential.nix index 9e57b512..1950f959 100644 --- a/nix/host-modules/glue-queue-agent-credential.nix +++ b/nix/host-modules/glue-queue-agent-credential.nix @@ -40,7 +40,16 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: read on + # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when + # this unit reads one queue credential and nothing else. Its own leaf carries + # `-` and its role grants the single path + # below — still this hive's own, so the narrowing costs no reach. + haveClientIdentity = + baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null; credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir; secretFile = "${credentialDir}/secret"; @@ -141,8 +150,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; + BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index e577a988..8fd7474c 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -340,6 +340,156 @@ let } ''; + # ── the four principals that used to share the hive's own leaf ───────────── + # + # 🩸 Each of the four reads exactly ONE path in the store, and until this + # split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf, + # whose policy (`swarm-secret-client`'s `policy::render`) grants read on + # `swarm/agents/*`, `swarm/hives//*` AND `swarm/services/*`. Four + # principals presenting one certificate are ONE principal to bao, so the union + # of what the four need was the only grant expressible: the unit that fetches + # Grafana's OIDC client secret could fetch every agent credential in the + # swarm, and the unit that fetches this hive's matrix token could fetch + # Grafana's. Least privilege was not misconfigured, it was unrepresentable. + # + # Each now gets its own leaf, its own cert-auth role and a policy holding the + # one `secret/data/…` path that principal's script actually names — spelled to + # the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above + # gives for doing the same. + # + # ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the + # shape of the paths rather than a preference. A matrix appservice token and a + # queue credential live under `swarm/hives//`, and every hive runs a + # reader for its own; an OIDC client secret lives under + # `swarm/services//` and a swarm registers each of those exactly + # once. A single role for a per-hive reader would have to name `hives/*` to + # serve every hive — which would let one hive read another's matrix token, a + # reach no hive has today and one this change exists to remove rather than + # create. + readStanza = path: '' + path "${path}" { + capabilities = ["read"] + } + ''; + + # One reader per hive in the swarm directory. Written from the directory + # rather than from this host's own name because the policy is written where + # the STORE is and the reader runs where its hive is — the same split + # `matrixCtlHiveName` above exists to paper over, answered here by naming + # every hive instead of asking the operator which one. + perHiveReaders = + { rolePrefix, cnPrefix, ... }@spec: + lib.mapAttrsToList (hiveName: _: { + name = "${rolePrefix}-${hiveName}"; + cn = "${cnPrefix}-${hiveName}"; + policyText = readStanza (spec.path hiveName); + }) hyperhiveCfg.swarm.hives; + + matrixTokenReaders = perHiveReaders { + rolePrefix = "swarm-matrix-token"; + cnPrefix = baoDeploy.matrixTokenCommonNamePrefix; + path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token"; + }; + + queueAgentReaders = perHiveReaders { + rolePrefix = "swarm-queue-agent"; + cnPrefix = baoDeploy.queueAgentCommonNamePrefix; + path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent"; + }; + + # Singletons, so a list of one rather than a second shape: the granting unit + # below takes a list either way and the four units stay identical apart from + # the objects they write. + grafanaOidcReaders = [ + { + name = "swarm-grafana-oidc"; + cn = baoDeploy.grafanaOidcCommonName; + policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"; + } + ]; + + otelOidcReaders = [ + { + name = "swarm-otel-oidc"; + cn = baoDeploy.otelOidcCommonName; + policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.otel.clientId}/oidc/client"; + } + ]; + + # The role name IS the policy name, as for the three service principals + # above: the role attaches the policy by spelling it identically, and one + # string for both objects removes the way they drift apart. + readerPolicyWrite = obj: '' + printf '%s' ${lib.escapeShellArg obj.policyText} | + bao policy write ${lib.escapeShellArg obj.name} - + ''; + + # ⚠️ Names are `swarm-[-]`, outside the `hive-*` namespace the + # controller may rewrite, for the reason the three service principals above + # state. A per-hive CN cannot collide with a hive's own leaf either — a hive's + # CN is its bare name and ../reserved-hive-fragments.nix forbids the substring + # `swarm` in one — and ./swarm.nix reserves the concrete per-hive spellings + # anyway, because the prefixes are operator-settable and an operator may spell + # one without that substring. + readerRoleWrite = obj: '' + bao write auth/cert/certs/${lib.escapeShellArg obj.name} \ + certificate=@${tlsDir}/client-ca.pem \ + allowed_common_names=${lib.escapeShellArg obj.cn} \ + token_policies=${lib.escapeShellArg obj.name} \ + display_name=${lib.escapeShellArg obj.cn} + ''; + + # One unit per PRINCIPAL, not one unit for the four: these names are + # operator-facing strings, the same reason `swarm-bao-secret-publisher-policy` + # above gives for not being more script inside the controller's unit. A hive + # whose matrix token never arrives is diagnosed by the state of the unit named + # after it. + # + # `after` and not `requires`, for the reason the publisher's unit states: the + # controller's unit creates the KV and cert-auth mounts this one writes into, + # but a failed oneshot still counts as finished, so ordering plus this unit's + # own retry is what converges. + # + # The role write is inside the client-CA branch and the policy write is not, + # exactly as the three above: with no CA there is no trust anchor for a login + # role, but the policy it would attach is still worth asserting. + readerPolicyUnit = + description: objects: + lib.mkIf haveBootstrapToken { + inherit description; + after = [ + "container@${cfg.machine}.service" + "swarm-bao-controller-policy.service" + ]; + wantedBy = [ "multi-user.target" ]; + path = [ + baoCli + pkgs.coreutils + ]; + unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; + # Same unseal wait as its siblings above, for the reason stated there: + # under `seal = "shamir"` a human unseals by hand. + startLimitBurst = 2880; + startLimitIntervalSec = 90000; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + Restart = "on-failure"; + RestartSec = 30; + }; + script = '' + set -euo pipefail + + BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})" + export BAO_TOKEN + + '' + + lib.concatMapStrings readerPolicyWrite objects + + lib.optionalString (baoDeploy.clientCaFile != null) ( + "\n" + lib.concatMapStrings readerRoleWrite objects + ); + }; + # Every listener serves the same identity: they differ in which address # they answer on, not in who they are. Client verification is separate and # optional — a store with no `clientCaFile` still serves TLS, it just does @@ -846,6 +996,185 @@ in ''; }; + matrixTokenCommonNamePrefix = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-matrix-token"; + example = "swarm-bao-matrix-token.svc"; + description = '' + Prefix of the subject the store's per-hive matrix-token roles accept. + The role for hive `h` accepts `-h` and nothing else, and grants + read on exactly `swarm/hives/h/matrix/appservice-token`. + + A **prefix** rather than one subject because every hive runs its own + reader and the path it reads carries that hive's name. One subject for + all of them would need a `hives/*` grant, which would let one hive read + another hive's appservice token — a reach no hive has today. + + ⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its + fixed-subject siblings are: cert auth trusts the CA and + `allowed_common_names` is the whole narrowing, so a hive whose own leaf + carried one of these spellings would receive that grant. + ''; + }; + + queueAgentCommonNamePrefix = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-queue-agent"; + example = "swarm-bao-queue-agent.svc"; + description = '' + Prefix of the subject the store's per-hive queue-credential roles + accept. The role for hive `h` accepts `-h` and grants read on + exactly `swarm/hives/h/queue/agent` — the client credential that hive's + agent containers authenticate to the swarm queue with. + + Per hive for the same reason + {option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is, + and more sharply: every hive in the swarm runs this reader, so a single + subject would have to be granted `hives/*`. + ''; + }; + + grafanaOidcCommonName = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-grafana-oidc"; + example = "swarm-bao-grafana-oidc.svc"; + description = '' + Subject the store's `swarm-grafana-oidc` cert-auth role accepts — the + identity the unit that fetches Grafana's OIDC client secret presents. + + Its grant is one path, + `swarm/services//oidc/client`, and read only. Not + per hive, unlike the two prefixes above: an OIDC client is registered + once per swarm, so the path names the service and never a hive. + + ⚠️ Same collision as its siblings, and the same answer: ./swarm.nix + feeds this value into the guard on + {option}`services.hyperhive.swarm.hives`. + ''; + }; + + otelOidcCommonName = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-otel-oidc"; + example = "swarm-bao-otel-oidc.svc"; + description = '' + Subject the store's `swarm-otel-oidc` cert-auth role accepts — the + identity the unit that fetches the collector's OIDC client secret + presents. Its grant is one path, + `swarm/services//oidc/client`, and read only. + + A **fourth** identity rather than reuse of + {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two + read different services' client secrets, and a collector is not + entitled to Grafana's. + ''; + }; + + matrixTokenClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/matrix-token.pem"; + description = '' + Certificate the unit that fetches this hive's matrix appservice token + presents to the store. Its subject must be + `-`; cert auth matches on + the CN and the role accepts nothing else. + + ⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`. + Pointing this at the hive's own leaf would evaluate, deploy and log in — + and hand a unit that reads one token a credential that reads every + secret in the store. That is what this option exists to end. + + No default here: ./glue-bao-tls.nix points it at the leaf it mints, + wherever this host mints its own PKI. A deployment that reads a store on + another host names it by hand, alongside the leaf itself — one file per + principal placed out of band instead of one shared by four. + ''; + }; + + matrixTokenClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/matrix-token-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both + or neither — a certificate with no key authenticates nothing. + ''; + }; + + queueAgentClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/queue-agent.pem"; + description = '' + Certificate the unit that fetches this hive's queue client credential + presents to the store. Its subject must be + `-`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + queueAgentClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/queue-agent-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`. + ''; + }; + + grafanaOidcClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/grafana-oidc.pem"; + description = '' + Certificate the unit that fetches Grafana's OIDC client secret presents + to the store. Its subject must be + {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + grafanaOidcClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/grafana-oidc-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile`. + ''; + }; + + otelOidcClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/otel-oidc.pem"; + description = '' + Certificate the unit that fetches the collector's OIDC client secret + presents to the store. Its subject must be + {option}`services.hyperhive.deploy.bao.otelOidcCommonName`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + otelOidcClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/otel-oidc-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.otelOidcClientCertFile`. + ''; + }; + serverCaFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; @@ -1593,6 +1922,19 @@ in ''; }; + # 🩸 The four principals that used to log in as the hive. See the block + # around `perHiveReaders` for what each grant is and why the first two are + # per hive; the four units are identical apart from the objects they + # write, so the shape lives in `readerPolicyUnit` and the difference is + # visible here in one line each. + systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders; + + systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders; + + systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders; + + systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders; + # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 006e3abf..85e5a2d1 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -119,7 +119,14 @@ let # other secret is read with it. A Grafana host without it has not been given # its identity yet, which is a thing to say out loud rather than to route # around by reaching into authelia's tree whenever it happens to be local. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 Grafana's OWN leaf, not `clientCertFile` — the hive's, which four units + # used to share. Bao matches a cert-auth role on the CN, so one leaf for four + # readers was ONE principal holding the union of four grants: this unit could + # read every agent credential in the swarm and every other service's OIDC + # client secret, when what it needs is the one path `storeSecretPath` names. + haveClientIdentity = + baoDeploy.grafanaOidcClientCertFile != null && baoDeploy.grafanaOidcClientKeyFile != null; autheliaUrl = toString hyperhiveCfg.swarm.authelia.url; @@ -451,15 +458,20 @@ in services.hyperhive.deploy.grafana.enable requires this host to hold a swarm-secret-store client identity: set both - services.hyperhive.deploy.bao.clientCertFile - services.hyperhive.deploy.bao.clientKeyFile + services.hyperhive.deploy.bao.grafanaOidcClientCertFile + services.hyperhive.deploy.bao.grafanaOidcClientKeyFile Grafana's OIDC client secret is minted by authelia and read out of the store, on every host that runs Grafana — including the host that runs authelia. That is one delivery route rather than two, and it is - what the store is for: this certificate is the single credential - placed out of band, and every other secret comes from the store with - it. + what the store is for: a certificate is the credential placed out of + band, and every other secret comes from the store with it. + + ⚠️ Grafana's OWN leaf, not deploy.bao.clientCertFile. That one is the + hive's, and its grant reads every secret in the store; this role + reads the one path Grafana's client secret lives at. Pointing this + option at the hive's leaf would evaluate, deploy and log in — and + undo the split. On a hive that runs the store, glue-bao-tls.nix supplies both as defaults and there is nothing to do. Elsewhere the leaf is issued @@ -610,8 +622,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 868df8b7..e0db2ce7 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -248,7 +248,14 @@ let # their own optional readers, because a collector with no client identity is # `haveCollectorSecret = false` above, and that is already a supported, # merely degraded shape rather than a service with no way in at all. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 The collector's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: this unit + # could read every agent credential in the swarm and Grafana's OIDC client + # secret, when what it needs is the one path `storeSecretPath` names. + haveClientIdentity = + baoDeploy.otelOidcClientCertFile != null && baoDeploy.otelOidcClientKeyFile != null; # Where the publisher on authelia's host leaves this client's secret — # composed from the same swarm-wide `clientId` the registration in @@ -775,8 +782,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.otelOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.otelOidcClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index d0e36dcd..213aa382 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -47,7 +47,20 @@ let deployCfg.bao.controllerCommonName deployCfg.bao.secretPublisherCommonName deployCfg.bao.matrixCtlCommonName - ]; + deployCfg.bao.grafanaOidcCommonName + deployCfg.bao.otelOidcCommonName + ] + # The two per-hive readers' subjects, spelled out per hive rather than as the + # prefix. The prefix alone would reserve the wrong string: the role for hive + # `h` accepts `-h`, so the name a hive must not BE is that composed + # spelling, and a hive named it would present a leaf that role accepts. + # + # Both spellings are reserved for every declared hive, including the one the + # CN belongs to — a hive cannot be named after its own reader either. + ++ lib.concatMap (hive: [ + "${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}" + "${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}" + ]) hiveNames; # Public hostnames of the swarm's own services, in declaration order. # `serviceDomains` below is this set sorted + deduplicated. diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 2068620c..c03e0fcf 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -45,6 +45,18 @@ let deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; deploy.bao.clientCaFile = lib.mkForce null; }; + # The store plus every one of the four readers that used to log in as the + # hive. One fixture rather than four: the claim they are four *separate* + # principals is only testable where all four render at once — that is the + # deployment in which two of them sharing a leaf would be invisible. + baoGrantWithConsumers = hive { + deploy.bao.enable = true; + deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; + deploy.matrix.enable = true; + deploy.grafana.enable = true; + deploy.swarm-otel.enable = true; + }; + cases = [ { # Reads the rendered unit on the HOST, which is where the write happens: @@ -196,6 +208,250 @@ let ) && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-matrix-ctl-policy); } + + # ── the four readers that used to share the hive's own leaf ────────────── + # + # 🩸 Until this split all four presented `deploy.bao.clientCertFile`, whose + # policy grants read on `swarm/agents/*`, `swarm/hives//*` AND + # `swarm/services/*`. Four principals behind one certificate are one + # principal to bao, so the only expressible grant was the union: the unit + # fetching Grafana's OIDC secret could fetch every agent credential in the + # swarm. + # + # Every one of these cases carries the same three negative arms, and they + # are the deliverable rather than decoration — a positive arm alone passes + # just as well when the other two stanzas are still there beside it. The + # arms pin what each principal must NOT reach, so a later widening fails + # here instead of being noticed in a store. + { + name = "the matrix-token reader's grant is one hive's appservice token and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-matrix-token-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/hives/h1/matrix/appservice-token\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + # The three stanzas the hive's own leaf carried, none of which this + # principal needs: every agent's credential, every service's OIDC + # client, and the rest of its own hive's tree — including the queue + # credential its sibling reader fetches. + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/services" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/queue" s) + # A `hives/*` wildcard would serve every hive from one role and let any + # hive read any other's token — the reach this split exists to remove, + # not to create. + && !(lib.hasInfix "secret/data/swarm/hives/*" s) + # Nothing may rewrite the policy constraining it, for the reason the + # controller's own `hive-*` narrowing above gives. + && !(lib.hasInfix "sys/policies/acl" s); + } + { + name = "the queue-credential reader's grant is one hive's queue credential and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/services" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s) + && !(lib.hasInfix "secret/data/swarm/hives/*" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # ⚠️ The client id is the path segment, so the negative arm that matters + # for this one is the OTHER service's: `services/*` would have granted + # both, and the two are separate principals precisely because a + # dashboard is not entitled to a collector's credential. + name = "the Grafana OIDC reader's grant is Grafana's own client secret and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/services/swarm-grafana/oidc/client\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/hives" s) + && !(lib.hasInfix "secret/data/swarm/services/*" s) + && !(lib.hasInfix "swarm-collector" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # The mirror of the case above, and the arm naming `swarm-grafana` is why + # these are two principals rather than one `services/*` grant shared. + name = "the collector OIDC reader's grant is the collector's own client secret and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-otel-oidc-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/services/swarm-collector/oidc/client\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/hives" s) + && !(lib.hasInfix "secret/data/swarm/services/*" s) + && !(lib.hasInfix "swarm-grafana" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # 🩸 The half that makes the policies above bind: a policy grants only + # through a token that carries it, and a token is minted by a cert-auth + # role matching a CN. Four distinct subjects is the whole mechanism — one + # subject for four readers is one principal however the policies read. + # + # The per-hive subjects carry the hive name because their paths do; the + # two service subjects do not, because an OIDC client is registered once + # per swarm. Pinned so neither shape is tidied into the other. + name = "each of the four readers logs in under a subject of its own"; + ok = + let + subjectOf = + unit: role: cn: + let + s = baoGrantHere.systemd.services.${unit}.script; + in + lib.hasInfix "auth/cert/certs/${role}" s + && lib.hasInfix "allowed_common_names=${cn}" s + && lib.hasInfix "token_policies=${role}" s + # Outside the `hive-*` namespace the controller may rewrite, for + # the reason the three service principals above state. + && !(lib.hasInfix "auth/cert/certs/hive-" s); + in + subjectOf "swarm-bao-matrix-token-policy" "swarm-matrix-token-h1" "swarm-bao-matrix-token-h1" + && subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1" + && subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc" + && subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"; + } + { + # 🩸 The consuming side, and the arm that would catch the regression that + # costs the most: a unit repointed back at `deploy.bao.clientCertFile` + # evaluates, deploys and logs in — and silently restores the union grant, + # because bao would again see one principal. Nothing about the policies + # above would look wrong. + # + # Each pair is asserted whole: a certificate with no key authenticates + # nothing, so a half-set pair is a reader that does not render. + name = "each of the four readers presents its own leaf, never the hive's"; + ok = + let + b = baoGrantWithConsumers.services.hyperhive.deploy.bao; + hiveLeaf = [ + b.clientCertFile + b.clientKeyFile + ]; + own = [ + b.matrixTokenClientCertFile + b.matrixTokenClientKeyFile + b.queueAgentClientCertFile + b.queueAgentClientKeyFile + b.grafanaOidcClientCertFile + b.grafanaOidcClientKeyFile + b.otelOidcClientCertFile + b.otelOidcClientKeyFile + ]; + envOf = unit: baoGrantWithConsumers.systemd.services.${unit}.environment; + presents = + unit: cert: key: + (envOf unit).BAO_CLIENT_CERT == cert && (envOf unit).BAO_CLIENT_KEY == key; + in + lib.all (p: p != null) own + && !(lib.any (p: lib.elem p hiveLeaf) own) + && lib.length (lib.unique own) == lib.length own + && presents "swarm-bao-matrix-token" b.matrixTokenClientCertFile b.matrixTokenClientKeyFile + && presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile + && presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile + && presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile; + } + { + # The minting side of the same claim. A role matching a subject nothing + # signs is a reader that cannot log in, so the leaves and the roles have + # to be asserted against each other — and the two per-hive leaves carry + # THIS host's hive name, which is what makes one hive's leaf useless + # against another hive's role. + name = "the PKI unit signs a leaf per reader, each under that reader's own subject"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-pki.script; + in + # The basename and the subject are matched separately: `signLeaf` takes + # them as consecutive arguments across a `\` continuation, so one + # literal spanning both would pin this file's line wrapping rather than + # the pairing it means to. + lib.all (lib.flip lib.hasInfix s) [ + "/matrix-token.pem ]" + "swarm-bao-matrix-token-h1 \"\" clientAuth" + "/queue-agent.pem ]" + "swarm-bao-queue-agent-h1 \"\" clientAuth" + "/grafana-oidc.pem ]" + "swarm-bao-grafana-oidc \"\" clientAuth" + "/otel-oidc.pem ]" + "swarm-bao-otel-oidc \"\" clientAuth" + ]; + } + { + # The absence arm: with no client CA there is no trust anchor, so the + # login roles cannot be written — but the policies they would attach are + # still asserted, exactly as the three service principals above behave in + # this deployment. A unit that vanished here would take the policy with + # it and leave nothing to diagnose. + name = "with no client CA the four readers get policies but no login roles"; + ok = + let + units = [ + "swarm-bao-matrix-token-policy" + "swarm-bao-queue-agent-policy" + "swarm-bao-grafana-oidc-policy" + "swarm-bao-otel-oidc-policy" + ]; + scriptOf = unit: baoGrantNoClientCa.systemd.services.${unit}.script; + in + lib.all ( + unit: + (baoGrantNoClientCa.systemd.services ? ${unit}) + && lib.hasInfix "bao policy write" (scriptOf unit) + && !(lib.hasInfix "auth/cert/certs" (scriptOf unit)) + ) units; + } + { + # Same control the three service principals carry: the write needs a + # client certificate and the host is the side that has one, so a unit + # rendered inside the store's container would have neither an identity + # nor a route. Plus the ordering that makes the mounts exist first. + name = "the four readers' granting units are ordered after the mounts and rendered on the host"; + ok = + let + units = [ + "swarm-bao-matrix-token-policy" + "swarm-bao-queue-agent-policy" + "swarm-bao-grafana-oidc-policy" + "swarm-bao-otel-oidc-policy" + ]; + in + lib.all ( + unit: + lib.elem "swarm-bao-controller-policy.service" baoGrantHere.systemd.services.${unit}.after + && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? ${unit}) + ) units; + } + { + # A store host that has not placed a bootstrap token can write no grant at + # all, so none of the four units may exist — the same claim + # `baoGrantNoStore` makes for the controller's, one file over. Without + # this arm `lib.mkIf haveBootstrapToken` could be dropped from the shared + # builder and every other case here would still pass. + name = "without a bootstrap token none of the four readers' granting units render"; + ok = + let + s = baoGrantNoStore.systemd.services; + in + !(s ? swarm-bao-matrix-token-policy) + && !(s ? swarm-bao-queue-agent-policy) + && !(s ? swarm-bao-grafana-oidc-policy) + && !(s ? swarm-bao-otel-oidc-policy); + } { # The policy authorising this route lives in another file, and nothing # else relates the grants to the paths the code actually writes. diff --git a/nix/module-eval/bao-matrix-reader.nix b/nix/module-eval/bao-matrix-reader.nix index 00f2256b..dd9470a2 100644 --- a/nix/module-eval/bao-matrix-reader.nix +++ b/nix/module-eval/bao-matrix-reader.nix @@ -32,10 +32,20 @@ let # nothing here mints a leaf and the operator names one placed by hand. The # deployment this pairing exists to serve, and the one that was previously # inexpressible — the gate asked whether the store was a neighbour. + # + # 🩸 One leaf PER READER, and this fixture is where that cost is visible: the + # two units below identify themselves to the store separately, so an operator + # placing leaves by hand places one for each rather than one for both. Naming + # only `clientCertFile` here would leave neither unit rendered — which is what + # the arms below would then be asserting about. baoRemoteReader = hive { deploy.matrix.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.matrixTokenClientCertFile = "/etc/pki/bao-matrix-token.pem"; + deploy.bao.matrixTokenClientKeyFile = "/etc/pki/bao-matrix-token-key.pem"; + deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem"; + deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem"; }; # A homeserver on a hive with NO store identity at all — neither a local diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index ae825800..fe3cd97e 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -41,6 +41,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; + deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # The same UI with the IdP on ANOTHER host and a store leaf placed by hand. @@ -54,6 +56,8 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; + deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # A Grafana host holding no store identity. This used to be the shape the @@ -148,8 +152,8 @@ let # mean the two assertions had been collapsed into one conjunction. name = "a grafana host with no store identity is refused, naming the options to set"; ok = - grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientCertFile" - && grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientKeyFile" + grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientCertFile" + && grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientKeyFile" && !(grafanaRefusedFor grafanaNoIdentity "swarm.authelia.url"); } { @@ -161,9 +165,9 @@ let name = "neither grafana refusal fires on a correctly configured host, co-located or not"; ok = !(grafanaRefusedFor grafanaWithAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.clientCertFile") + && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.grafanaOidcClientCertFile") && !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.clientCertFile"); + && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.grafanaOidcClientCertFile"); } { # Same 403-not-a-miss reason as the matrix and queue arms below: the diff --git a/nix/module-eval/name-guards.nix b/nix/module-eval/name-guards.nix index 8716f980..e035b0f0 100644 --- a/nix/module-eval/name-guards.nix +++ b/nix/module-eval/name-guards.nix @@ -63,6 +63,35 @@ let swarm.hives.mintctl.domain = "m.t.local"; }; + # The two OIDC-secret readers' subjects, fixed strings like the three above. + hiveNamedAfterGrafanaOidcSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.grafanaOidcCommonName = "gfctl"; + swarm.hives.gfctl.domain = "g.t.local"; + }; + + hiveNamedAfterOtelOidcSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.otelOidcCommonName = "otctl"; + swarm.hives.otctl.domain = "o.t.local"; + }; + + # 🩸 A different shape from every fixture above: the matrix-token and + # queue-credential roles are written PER HIVE, so the subject a hive must not + # be is `-` rather than the prefix itself. Reserving + # only the prefix would leave the composed spelling free, and a hive taking it + # would present a leaf the other hive's role accepts — which is a hive reading + # another hive's queue credential, the exact widening the split exists to + # avoid. + # + # Two hives here, not one: the collision is with the OTHER hive's role. + hiveNamedAfterPerHiveReaderSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.queueAgentCommonNamePrefix = "qr"; + swarm.hives.other.domain = "o.t.local"; + swarm.hives.qr-other.domain = "q.t.local"; + }; + hiveNameWithComposedWord = hive { deploy.swarm-otel.enable = false; swarm.hives."h1-agent".domain = "a.t.local"; @@ -111,6 +140,36 @@ let a: !a.assertion && lib.hasInfix "'mintctl'" a.message ) hiveNamedAfterMatrixCtlSubject.assertions; } + { + # The fourth and fifth, for the reason the case above gives: `certAuthCns` + # is where a role added beside the others registers itself, and nothing + # but a case per element notices when one forgets. These two are the + # subjects of the readers that fetch Grafana's and the collector's OIDC + # client secrets. + name = "a hive named after either OIDC-secret reader's subject is refused too"; + ok = + equalityGuardFired hiveNamedAfterGrafanaOidcSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'gfctl'" a.message + ) hiveNamedAfterGrafanaOidcSubject.assertions + && equalityGuardFired hiveNamedAfterOtelOidcSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'otctl'" a.message + ) hiveNamedAfterOtelOidcSubject.assertions; + } + { + # 🩸 The per-hive half, and the one a prefix-only reservation would miss: + # the role is `-`, so the reserved string has to be composed + # against every declared hive. Here hive `qr-other` collides with the role + # written for hive `other` — a leaf that reads a credential belonging to a + # hive that is not it. + name = "a hive named after another hive's per-hive reader subject is refused"; + ok = + equalityGuardFired hiveNamedAfterPerHiveReaderSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'qr-other'" a.message + ) hiveNamedAfterPerHiveReaderSubject.assertions; + } { # Without this the case above proves nothing: an arm that fires for every # roster is not a guard, and `hives` is non-empty in both fixtures. diff --git a/nix/module-eval/swarm-otel-core.nix b/nix/module-eval/swarm-otel-core.nix index 7428919c..0bbfa10e 100644 --- a/nix/module-eval/swarm-otel-core.nix +++ b/nix/module-eval/swarm-otel-core.nix @@ -32,6 +32,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; deploy.victoriametrics.enable = false; deploy.victorialogs.enable = false; }; @@ -44,6 +46,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; # The same collector with the IdP on ANOTHER host and a store leaf placed by @@ -54,6 +58,8 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; cases = [ { From d3e4951cc84259add8ce69a074d3db75be1e820b Mon Sep 17 00:00:00 2001 From: atlas Date: Wed, 23 Sep 2026 09:56:43 +0200 Subject: [PATCH 2/3] swarm-bao: refuse a remote reader that named seven of the eight leaves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four-way client-cert split gives each store reader its own leaf, and three of the four readers render only where their own leaf exists. On a host that mints its own PKI glue-bao-tls.nix defaults all eight, so there is nothing to do; on a hand-configured remote-store hive, omitting one pair used to mean that unit silently did not render — a privilege- narrowing unit absent from a green build, with the missing unit as the only evidence. Each of the three now asserts its own pair, shaped after swarm-grafana.nix's haveClientIdentity assertion and named to the pair it needs. What differs from Grafana's is the gate: these fire only where the host demonstrably reads the store (it holds deploy.bao.clientCertFile and clientKeyFile) and the consumer is on. A host with no store identity is the supported no-store deployment and still evaluates; the collector's no-secret degrade is untouched, because that host holds no clientCertFile either. Also rewords three passive-voice sentences in docs/swarm/secrets.md that vale flagged, and documents what the refusal costs and where it stays silent. --- docs/swarm/secrets.md | 47 ++- nix/host-modules/glue-matrix-bao-token.nix | 290 +++++++++------- .../glue-queue-agent-credential.nix | 310 +++++++++++------- nix/host-modules/swarm-otel.nix | 72 +++- nix/module-eval/bao-matrix-reader.nix | 88 +++++ nix/module-eval/swarm-otel-identity.nix | 66 ++++ 6 files changed, 612 insertions(+), 261 deletions(-) diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 681d277c..b5f3c38a 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -250,7 +250,7 @@ round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store exists so the only secrets a host holds out of band are **certificates**, and -everything else is read with them. +a host reads everything else with them. Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: running Grafana requires `swarm.authelia.url` (this module disables its local @@ -319,7 +319,7 @@ internal CA deletes that file and names its own paths in `deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. A hive that reads from a store on **another** machine names the reader's half — `clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand, -plus one leaf per principal it runs (the options are listed below). Those are the +plus one leaf per principal it runs (the table below names the options). Those are the credentials that can't come out of the store, being what opens it; everything else a hive needs does. @@ -336,20 +336,55 @@ each and each holds a leaf, a role and a policy of its own: | `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services//oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | -The first two are written **per hive**, because the path they read carries a hive +The first two exist **per hive**, because the path they read carries a hive name and every hive runs its own reader. Their subjects are `-` and `-`; `swarm.nix` reserves both -composed spellings as hive names, so a hive can't be named into another hive's +composed spellings as hive names, so nobody can name a hive into another hive's role. The other two read a path that names a swarm service rather than a hive, so one role each is enough and their subjects are the flat `deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`. On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults -all eight options, and there is nothing to do. Elsewhere each leaf is issued from -that CA out of band and named here — one file per principal rather than one file +all eight options, and there is nothing to do. Elsewhere you issue each leaf from +that CA out of band and name it here — one file per principal rather than one file shared by four, which is the whole of what this buys. +Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four +readers used to render only where their leaf existed, so a hand-configured +remote-store hive that named the hive's own `clientCertFile` and missed a +principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing +the build. All four now refuse it, each naming its own option pair — see +[what a missing leaf costs](#what-a-missing-leaf-costs) below. + +### What a missing leaf costs + +Each of the four modules asserts its own pair, on the same condition: **this host +already reads the store** — it holds `deploy.bao.clientCertFile` and +`clientKeyFile` — **and the consumer is on, and this principal's pair is null**. +The refusal names the two options, so an operator acts on the message without +opening the nix. + +That condition fires on exactly one shape, the hand-configured remote-store hive +that named seven of the eight. It stays silent everywhere else, and each half of +that matters on its own: + +- **A host with no store identity at all** holds no `clientCertFile` either, so + the first clause is false. That's the supported no-store deployment, and also + the one an operator passes through while bringing a hive up — neither should + fail to evaluate. +- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as + defaults, so the third clause is false. Nothing to set, nothing to refuse. +- **A consumer that's off** — no homeserver, no collector, no Grafana — has no + unit to skip, so the second clause is false. Only the queue-credential reader + has no toggle of its own to check: every hive runs one for its own agents, so + reading the store at all is what asks for it. + +The collector keeps a separate degrade underneath this, unchanged: a host with no +store identity runs a collector with no OIDC client secret and still receives +telemetry. The assertion doesn't touch that shape, because that host has no +`clientCertFile` either. + ## How a reader reaches the store Every reader dials the same URL — `https://bao.:` — and on diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix index ca870766..3c0afcb6 100644 --- a/nix/host-modules/glue-matrix-bao-token.nix +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -49,6 +49,14 @@ let haveClientIdentity = baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null; + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. Only + # used to decide whether a missing per-principal leaf is a mistake or a + # deployment that has no store: a host holding neither is the supported + # no-store shape, and one holding this pair but not the pair above named + # seven of the eight options and stopped. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # Where the token lives in the store. A path, not a convention to guess at: # whoever writes it and whoever reads it must agree, and the agreement # belongs in one visible place. @@ -73,134 +81,178 @@ let matrixMachine = "hive-matrix"; in { - config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { - # Same rule as the unit's own gate: this reader exists on a host that has a - # client identity and a homeserver, which is not every host that runs the - # store, so the store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; + config = lib.mkMerge [ + # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole + # mechanism: the unit's arm is gated on `haveClientIdentity`, so an + # assertion written inside it could never be reached in the state it + # exists to report. + # + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the + # same refusal, named to this principal's own pair. What differs is the + # gate. Grafana asserts wherever Grafana runs, because a Grafana with no + # store identity has no way in at all; a homeserver with no store identity + # is a hive that has no store, which is supported. So this one additionally + # requires `hiveReaderIdentity`: the host demonstrably reads the store, and + # named every option but this pair. + (lib.mkIf (hyperhiveCfg.enable && deployCfg.matrix.enable && hiveReaderIdentity) { + assertions = [ + { + assertion = haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set) and runs a homeserver, so it needs the matrix appservice + token reader's own client identity: set both - systemd.services.swarm-bao-matrix-token = { - description = "fetch the matrix appservice token from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" + services.hyperhive.deploy.bao.matrixTokenClientCertFile + services.hyperhive.deploy.bao.matrixTokenClientKeyFile + + swarm-bao-matrix-token.service fetches this hive's appservice token + out of the store, and without these it is not rendered at all — + leaving the homeserver authenticating hive-c0re against whatever is + already on disk, which is a 401 on every request naming nothing. + + ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is + the hive's, and its grant reads every secret in the store; this role + reads the one appservice-token path. Pointing this option at the + hive's leaf would evaluate, deploy and log in — and undo the split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - before = [ "container@${matrixMachine}.service" ]; - wantedBy = [ "container@${matrixMachine}.service" ]; - path = [ - deployCfg.bao.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal. `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after — so a few short attempts cover it. - # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. - # That unit blocks nothing; this one is `Before=` the homeserver's - # container, and whether that ordering waits across an auto-restart is - # unverified — so an hours-long window would be a bet on an unknown, - # where a minute is not. A store still sealed after it keeps the degrade - # below, as today. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the read below. Stated here rather than - # left to systemd's default, so the number a boot waits on is in - # the file that waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; - BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail + }) - # A sealed or uninitialised store answers on the port and never - # answers the read, so "the store is up" is not the same as "the - # store can answer". `TimeoutStartSec` above is the bound; the - # homeserver only `Wants=` this unit, so hitting it degrades to - # keeping the local token rather than holding up the container. - # `bao`'s own message is the only thing separating a missing value - # from a refused identity from an unreachable host. This unit's - # degraded mode is correct for all three, so it reports which one - # rather than asserting all three in a sentence of ours — a reader - # that cannot say why it read nothing is indistinguishable from a - # broken one. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT + (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { + # Same rule as the unit's own gate: this reader exists on a host that has a + # client identity and a homeserver, which is not every host that runs the + # store, so the store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. + systemd.services.swarm-bao-matrix-token = { + description = "fetch the matrix appservice token from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + before = [ "container@${matrixMachine}.service" ]; + wantedBy = [ "container@${matrixMachine}.service" ]; + path = [ + deployCfg.bao.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal. `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after — so a few short attempts cover it. + # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. + # That unit blocks nothing; this one is `Before=` the homeserver's + # container, and whether that ordering waits across an auto-restart is + # unverified — so an hours-long window would be a bet on an unknown, + # where a minute is not. A store still sealed after it keeps the degrade + # below, as today. # - # Fails LOUDLY, unlike the read below: the three states a login failure - # covers — store not up, sealed, role not written yet — are all things - # a retry fixes, and `Restart=on-failure` above is what retries. Exiting - # 0 here spends the whole boot on a condition that was seconds old. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the read below. Stated here rather than + # left to systemd's default, so the number a boot waits on is in + # the file that waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; + BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # A sealed or uninitialised store answers on the port and never + # answers the read, so "the store is up" is not the same as "the + # store can answer". `TimeoutStartSec` above is the bound; the + # homeserver only `Wants=` this unit, so hitting it degrades to + # keeping the local token rather than holding up the container. + # `bao`'s own message is the only thing separating a missing value + # from a refused identity from an unreachable host. This unit's + # degraded mode is correct for all three, so it reports which one + # rather than asserting all three in a sentence of ours — a reader + # that cannot say why it read nothing is indistinguishable from a + # broken one. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the read below: the three states a login failure + # covers — store not up, sealed, role not written yet — are all things + # a retry fixes, and `Restart=on-failure` above is what retries. Exiting + # 0 here spends the whole boot on a condition that was seconds old. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 fi - exit 1 - fi - export BAO_TOKEN + export BAO_TOKEN - if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then - echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then + echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - if [ -z "$token" ]; then - echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 - exit 0 - fi + if [ -z "$token" ]; then + echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 + exit 0 + fi - umask 077 - printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + umask 077 + printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - # Re-stamp the registration file from the token just written. The - # token is half an agreement — the registration the homeserver loads - # has to carry the same value — so writing the file and stopping - # would leave the homeserver authenticating hive-c0re against - # whatever activation put there: a 401 on every request, naming - # nothing. Unconditional rather than on-change, because this unit - # has no way to know what the registration currently says. - # - # hive-matrix's own renderer rather than a `printf` here, so the - # registration's shape has one home. - ${deployCfg.matrix.appserviceRegistrationScript} - ''; - }; - }; + # Re-stamp the registration file from the token just written. The + # token is half an agreement — the registration the homeserver loads + # has to carry the same value — so writing the file and stopping + # would leave the homeserver authenticating hive-c0re against + # whatever activation put there: a 401 on every request, naming + # nothing. Unconditional rather than on-change, because this unit + # has no way to know what the registration currently says. + # + # hive-matrix's own renderer rather than a `printf` here, so the + # registration's shape has one home. + ${deployCfg.matrix.appserviceRegistrationScript} + ''; + }; + }) + ]; } diff --git a/nix/host-modules/glue-queue-agent-credential.nix b/nix/host-modules/glue-queue-agent-credential.nix index 1950f959..8fa2dc06 100644 --- a/nix/host-modules/glue-queue-agent-credential.nix +++ b/nix/host-modules/glue-queue-agent-credential.nix @@ -51,6 +51,14 @@ let haveClientIdentity = baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null; + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. Only + # used to decide whether a missing per-principal leaf is a mistake or a + # deployment that has no store: a host holding neither is the supported + # no-store shape, and one holding this pair but not the pair above named + # seven of the eight options and stopped. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir; secretFile = "${credentialDir}/secret"; clientIdFile = "${credentialDir}/client_id"; @@ -94,144 +102,192 @@ in }; }; - config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { - # Same rule as the unit's own gate: this reader exists on any host holding - # a client identity, which is not every host that runs the store, so the - # store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; + config = lib.mkMerge [ + # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole + # mechanism: the unit's arm is gated on `haveClientIdentity`, so an + # assertion written inside it could never be reached in the state it + # exists to report. + # + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the + # same refusal, named to this principal's own pair. Where Grafana's gate + # is `deploy.grafana.enable`, this reader has no toggle of its own to + # check: every hive runs one for its own agents, so reading the store at + # all is what asks for it. Hence `hiveReaderIdentity` alone — a host + # holding no hive leaf has no store to read and nothing is missing. + (lib.mkIf (hyperhiveCfg.enable && hiveReaderIdentity) { + assertions = [ + { + assertion = haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set), so it needs the agent queue credential reader's own client + identity: set both - systemd.services.swarm-bao-queue-agent = { - description = "fetch this hive's agent queue credential from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" - ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - # Ordered before hive-c0re, so no agent container renders ahead of an - # attempt at its credential. `Wants=`, not `Requires=`: a store this - # unit can't reach delays hive-c0re's start by its own start-limit - # window (`TimeoutStartSec`, retried up to `startLimitBurst` times - # below) rather than failing it — hive-c0re starts once that window - # elapses, whatever credential is or isn't on disk by then. - before = [ "hive-c0re.service" ]; - wantedBy = [ - "multi-user.target" - "hive-c0re.service" - ]; - path = [ - baoDeploy.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal: `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after, so a few short attempts cover it. - # An hours-long window would be a bet on a store that is sealed, and the - # degrade below is already correct for that. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the reads below. Stated here rather than left to - # systemd's default, so the number a boot waits on is in the file that - # waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; - BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail + services.hyperhive.deploy.bao.queueAgentClientCertFile + services.hyperhive.deploy.bao.queueAgentClientKeyFile - # `bao`'s own message is the only thing separating a missing value from - # a refused identity from an unreachable host. This unit's degraded - # mode is correct for all three, so it reports which one rather than - # asserting all three in a sentence of ours. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT + swarm-bao-queue-agent.service fetches this hive's agent queue + credential out of the store, and without these it is not rendered + at all — leaving hive-c0re with no queue credential and no unit + that would ever have written one. - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. + Every hive runs this reader for its own agents, so unlike the other + three principals there is no per-service toggle that turns it off: + a hive that reads the store at all is a hive that needs this pair. + + ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one + is the hive's, and its grant reads every secret in the store; this + role reads the one queue-credential path. Pointing this option at + the hive's leaf would evaluate, deploy and log in — and undo the + split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } + ]; + }) + + (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { + # Same rule as the unit's own gate: this reader exists on any host holding + # a client identity, which is not every host that runs the store, so the + # store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; + + systemd.services.swarm-bao-queue-agent = { + description = "fetch this hive's agent queue credential from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + # Ordered before hive-c0re, so no agent container renders ahead of an + # attempt at its credential. `Wants=`, not `Requires=`: a store this + # unit can't reach delays hive-c0re's start by its own start-limit + # window (`TimeoutStartSec`, retried up to `startLimitBurst` times + # below) rather than failing it — hive-c0re starts once that window + # elapses, whatever credential is or isn't on disk by then. + before = [ "hive-c0re.service" ]; + wantedBy = [ + "multi-user.target" + "hive-c0re.service" + ]; + path = [ + baoDeploy.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal: `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after, so a few short attempts cover it. + # An hours-long window would be a bet on a store that is sealed, and the + # degrade below is already correct for that. # - # Fails LOUDLY, unlike the reads below: the three states a login - # failure covers — store not up, sealed, role not written yet — are all - # things a retry fixes, and `Restart=on-failure` above is what retries. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the reads below. Stated here rather than left to + # systemd's default, so the number a boot waits on is in the file that + # waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; + BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # `bao`'s own message is the only thing separating a missing value from + # a refused identity from an unreachable host. This unit's degraded + # mode is correct for all three, so it reports which one rather than + # asserting all three in a sentence of ours. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the reads below: the three states a login + # failure covers — store not up, sealed, role not written yet — are all + # things a retry fixes, and `Restart=on-failure` above is what retries. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 fi - exit 1 - fi - export BAO_TOKEN + export BAO_TOKEN - # Two reads of one object rather than one `-format=json` parsed with - # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot - # actually disagree — the client id is derived from this hive's name, so - # a rotation landing between these two calls changes the secret and - # rewrites the same id. - if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # Two reads of one object rather than one `-format=json` parsed with + # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot + # actually disagree — the client id is derived from this hive's name, so + # a rotation landing between these two calls changes the secret and + # rewrites the same id. + if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - # Both or neither, for the reason `QueueConfig::from_env` refuses a - # half-set environment: a client that finds one of the two comes up - # "fine" and never connects. - if [ -z "$secret" ] || [ -z "$client_id" ]; then - echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 - exit 0 - fi + # Both or neither, for the reason `QueueConfig::from_env` refuses a + # half-set environment: a client that finds one of the two comes up + # "fine" and never connects. + if [ -z "$secret" ] || [ -z "$client_id" ]; then + echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 + exit 0 + fi - install -d -m 0755 ${lib.escapeShellArg credentialDir} + install -d -m 0755 ${lib.escapeShellArg credentialDir} - umask 077 - printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} - chmod 0600 ${lib.escapeShellArg secretFile} + umask 077 + printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} + chmod 0600 ${lib.escapeShellArg secretFile} - # `0644` on purpose: an OIDC client id is presented to the token - # endpoint on every connection and is public by construction. - printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} - chmod 0644 ${lib.escapeShellArg clientIdFile} - ''; - }; - }; + # `0644` on purpose: an OIDC client id is presented to the token + # endpoint on every connection and is public by construction. + printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} + chmod 0644 ${lib.escapeShellArg clientIdFile} + ''; + }; + }) + ]; } diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index e0db2ce7..c53bfd21 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -242,12 +242,14 @@ let # A reader of the store is defined by holding a certificate the store # accepts, never by standing next to it — the rule # ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical- - # looking flag, this one stays outside `assertions`: the store-reading unit - # below simply does not render without it, the same choice + # looking flag, this one does not gate an assertion on its own: the + # store-reading unit below simply does not render without it, the same choice # ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for - # their own optional readers, because a collector with no client identity is + # their own readers, because a collector with no client identity is # `haveCollectorSecret = false` above, and that is already a supported, - # merely degraded shape rather than a service with no way in at all. + # merely degraded shape rather than a service with no way in at all. What + # IS asserted is narrower and lives in `assertions` below — see + # `hiveReaderIdentity`. # # 🩸 The collector's OWN leaf, not `clientCertFile` — the hive's, which four # units used to share. Bao matches a cert-auth role on the CN, so one leaf for @@ -257,6 +259,16 @@ let haveClientIdentity = baoDeploy.otelOidcClientCertFile != null && baoDeploy.otelOidcClientKeyFile != null; + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. It is + # what separates the degrade the comment above describes from the mistake the + # assertion below reports: a collector on a host holding NO store identity is + # the supported shape, and one on a host that demonstrably reads the store + # named seven of the eight options and stopped. Before the four-way split + # that second host rendered this unit off `clientCertFile`, so it is a silent + # regression rather than a choice anyone made. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # Where the publisher on authelia's host leaves this client's secret — # composed from the same swarm-wide `clientId` the registration in # ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of @@ -732,11 +744,14 @@ in # value. # # ⚠️ Renders only where `haveClientIdentity` holds, unlike - # ./swarm-grafana.nix's equivalent unit. That module asserts the identity - # because a Grafana with none has no way in at all; this collector without - # one is `haveCollectorSecret = false` above — already a supported, - # merely degraded shape, so the unit that would fetch a credential simply - # does not exist rather than refusing the build for want of one. + # ./swarm-grafana.nix's equivalent unit. That module refuses any host that + # runs Grafana without the identity, because a Grafana with none has no way + # in at all; this collector without one is `haveCollectorSecret = false` + # above — already a supported, merely degraded shape, so the unit that would + # fetch a credential simply does not exist rather than refusing the build + # for want of one. The one case that IS refused is the host that already + # holds the hive's leaf and is missing only this pair, which is a silent + # regression rather than that degrade — `hiveReaderIdentity` above. systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity { description = "fetch the swarm collector's OIDC client secret from the swarm secret store"; # Every one of these names a unit that exists only where the store runs. @@ -860,6 +875,45 @@ in systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; assertions = [ + { + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — + # the same refusal, named to this principal's own pair. What differs is + # the `!hiveReaderIdentity ||` guard, and it is what keeps the degrade + # the flag's own comment describes intact: Grafana refuses any host + # that runs it without a leaf, because a Grafana with no SSO has no way + # in at all, while a collector with none still receives telemetry. So + # this fires only where the host already reads the store and is missing + # this one pair — which before the four-way split rendered the unit off + # `clientCertFile`, and now silently does not. + assertion = !hiveReaderIdentity || haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set) and runs the swarm collector, so it needs the collector's own + client identity: set both + + services.hyperhive.deploy.bao.otelOidcClientCertFile + services.hyperhive.deploy.bao.otelOidcClientKeyFile + + swarm-bao-otel-oidc.service fetches the collector's OIDC client + secret out of the store, and without these it is not rendered at all + — so this collector would authenticate to nothing, quietly, on a host + that has everything it needs to fetch the secret. + + A collector on a host holding NO store identity is a different and + supported shape: it runs without a client secret and still receives + telemetry. That is not this host. + + ⚠️ The collector's OWN leaf, not deploy.bao.clientCertFile. That one + is the hive's, and its grant reads every secret in the store; this + role reads the one path this collector's client secret lives at. + Pointing this option at the hive's leaf would evaluate, deploy and + log in — and undo the split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } # ⚠️ An assertion that this collector has "somewhere to send" was REMOVED # rather than relaxed: it read the store's *per-host* enable, so it # rejected at eval the very deployment the stores are reached by domain diff --git a/nix/module-eval/bao-matrix-reader.nix b/nix/module-eval/bao-matrix-reader.nix index dd9470a2..e000ea65 100644 --- a/nix/module-eval/bao-matrix-reader.nix +++ b/nix/module-eval/bao-matrix-reader.nix @@ -53,6 +53,37 @@ let # needs it, and defining it here rather than importing keeps each group's # fixture set its own, as ./lib.nix asks. matrixNoBaoIdentity = hive { deploy.matrix.enable = true; }; + + # 🩸 The hand-configured remote reader that named seven of the eight options. + # `baoRemoteReader` above is the same deployment done right; this one holds + # the hive's own leaf, so it demonstrably reads the store, and is missing both + # per-principal pairs. Before the four-way split this host rendered both units + # off `clientCertFile` alone, so what it has now is a silent regression rather + # than any shape an operator chose — which is what the refusal arms below are + # about. Kept as one fixture rather than two because both refusals fire on it + # and each arm names which. + baoRemoteReaderMissingLeaves = hive { + deploy.matrix.enable = true; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # The same omission on a hive that does NOT run a homeserver. Separates the + # matrix refusal's `deploy.matrix.enable` clause from the queue refusal, which + # has no toggle to check — an arm below reads exactly one refusal off it. + remoteReaderNoMatrixMissingLeaves = hive { + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # Did a module refuse this host, and over which option. An assertion is a + # config VALUE until something forces it — `.config` never throws — so a + # fixture in a state the module refuses stays evaluable and the refusal reads + # back as data. Matched on the option name the message names rather than on + # its prose, because the option name is the part an operator has to act on + # and a message that stopped naming it would be the actual defect. The same + # helper ./grafana.nix uses for `swarm-grafana.nix`'s own refusal. + refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; cases = [ { # A login failure is the store being unreachable, sealed, or not yet @@ -373,6 +404,63 @@ let in !(g ? admin_execute) || g.admin_execute == [ ]; } + { + # 🩸 The arm the whole refusal exists for. Both readers are gated on their + # own leaf, so the way this regresses is the build going green and three + # units rendering where four should — which no presence check on a + # rendered unit can see, because the unit that is missing is the evidence. + # Read as a refusal naming each option, so an operator acts on the message + # without opening the nix. + name = "a remote reader missing the per-principal leaves is refused, naming both options"; + ok = + refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientCertFile" + && refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientKeyFile" + && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile" + && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile"; + } + { + # The matrix refusal's own gate, which the queue refusal does not have. + # Without this arm the two are indistinguishable on the fixture above. + name = "the queue refusal needs no homeserver, and the matrix refusal stays quiet without one"; + ok = + refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile" + && !(refusedOver remoteReaderNoMatrixMissingLeaves "matrixTokenClientCertFile"); + } + { + # ⚠️ The arm that keeps the refusal from being worse than the silence it + # replaced. A hive with NO store identity is the supported no-store + # deployment and also the state an operator passes through bringing a hive + # up — neither may fail to evaluate. Asserted as "no refusal names any of + # the four options", not as "this one fixture is fine", because the way + # this breaks is a gate widened to the principal's leaf alone. + name = "a hive with no store identity at all is refused over none of the per-principal leaves"; + ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [ + "matrixTokenClientCertFile" + "matrixTokenClientKeyFile" + "queueAgentClientCertFile" + "queueAgentClientKeyFile" + ]; + } + { + # The other half of the same guard, and the one an operator meets far more + # often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults + # all eight, so there is nothing to name and nothing to refuse. Paired with + # the fully-named remote reader, which is the same deployment done by hand. + name = "neither a store host nor a correctly-named remote reader is refused"; + ok = + lib.all + ( + m: + lib.all (option: !(refusedOver m option)) [ + "matrixTokenClientCertFile" + "queueAgentClientCertFile" + ] + ) + [ + baoWithMatrix + baoRemoteReader + ]; + } ]; in runGroup "bao-matrix-reader" cases diff --git a/nix/module-eval/swarm-otel-identity.nix b/nix/module-eval/swarm-otel-identity.nix index 67c89357..99dcd677 100644 --- a/nix/module-eval/swarm-otel-identity.nix +++ b/nix/module-eval/swarm-otel-identity.nix @@ -34,6 +34,40 @@ let deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; }; + # 🩸 The shape the degrade above must not swallow: a collector on a host that + # demonstrably READS the store — it holds the hive's own leaf — and is missing + # only the collector's own pair. Before the four-way split this host rendered + # the reading unit off `clientCertFile` alone, so the silence it gets now is a + # regression rather than the supported degrade `otelNoIdentity` stands for. The + # two fixtures differ in exactly that one pair, which is what lets the arms + # below separate a refusal from a degrade. + otelReaderMissingLeaf = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # The same deployment named in full, which must stay buildable. Pairs with the + # fixture above so the refusal is pinned to the omission and not to reading a + # remote store at all. + otelReaderNamedInFull = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; + }; + + # Did the module refuse this host, and over which option. An assertion is a + # config VALUE until something forces it — `.config` never throws — so a + # fixture in a refused state stays evaluable and the refusal reads back as + # data. Matched on the option name the message names rather than on its prose: + # the option name is the part an operator has to act on, and a message that + # stopped naming it would be the actual defect. + refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; + # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. @@ -162,6 +196,38 @@ let && !(builtins.elem "otlphttp/victoriametrics" used) && lib.all (e: s.exporters ? ${e}) used; } + { + # 🩸 The refusal, and the only way this module's omission is visible at + # all: the reading unit is gated on its own leaf, so the regression is a + # green build with the unit absent — and the missing unit is the evidence. + # Read as a refusal naming both options, so an operator acts on the + # message without opening the nix. + name = "a store reader missing the collector's own leaf is refused, naming both options"; + ok = + refusedOver otelReaderMissingLeaf "otelOidcClientCertFile" + && refusedOver otelReaderMissingLeaf "otelOidcClientKeyFile"; + } + { + # ⚠️ The arm that keeps the refusal from eating the degrade beside it. A + # collector on a host with NO store identity still receives every hive's + # telemetry and is a supported deployment — `otelNoIdentity` is that + # fixture, and it differs from the refused one only in the hive's own + # leaf. Widening the gate to the collector's leaf alone would reject it, + # which is what this pins. + name = "a collector with no store identity at all is not refused over the collector's leaf"; + ok = + !(refusedOver otelNoIdentity "otelOidcClientCertFile") + && !(refusedOver otelNoIdentity "otelOidcClientKeyFile"); + } + { + # The other two shapes that must stay buildable: the same remote reader + # named in full, and a host that mints its own PKI and therefore has all + # eight as defaults from ./host-modules/glue-bao-tls.nix. + name = "neither a fully-named remote reader nor a store host is refused"; + ok = + !(refusedOver otelReaderNamedInFull "otelOidcClientCertFile") + && !(refusedOver otelRemoteAuthelia "otelOidcClientCertFile"); + } ]; in runGroup "swarm-otel-identity" cases From 9d339b1b585851766310e63fa922da9e6eea71a8 Mon Sep 17 00:00:00 2001 From: atlas Date: Wed, 23 Sep 2026 09:56:43 +0200 Subject: [PATCH 3/3] module-eval: name the grafana helper the refusal readers are shaped after Comment-only. The helper it points at is `grafanaRefusedFor`, not an unnamed one. --- nix/module-eval/bao-matrix-reader.nix | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/nix/module-eval/bao-matrix-reader.nix b/nix/module-eval/bao-matrix-reader.nix index e000ea65..19a0da5f 100644 --- a/nix/module-eval/bao-matrix-reader.nix +++ b/nix/module-eval/bao-matrix-reader.nix @@ -81,8 +81,9 @@ let # fixture in a state the module refuses stays evaluable and the refusal reads # back as data. Matched on the option name the message names rather than on # its prose, because the option name is the part an operator has to act on - # and a message that stopped naming it would be the actual defect. The same - # helper ./grafana.nix uses for `swarm-grafana.nix`'s own refusal. + # and a message that stopped naming it would be the actual defect. Same shape + # as ./grafana.nix's `grafanaRefusedFor`, which reads `swarm-grafana.nix`'s + # own refusal. refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; cases = [ {