diff --git a/docs/getting-started/setup.md b/docs/getting-started/setup.md index af3290d3..2225529e 100644 --- a/docs/getting-started/setup.md +++ b/docs/getting-started/setup.md @@ -170,16 +170,19 @@ What else you need depends on restart, with the keys `init` printed. The store serves TLS, and on a hive that deploys it you need do nothing: a -first-boot unit mints a CA of the store's own plus the two leaves it signs — -the store's server certificate and this host's client certificate — and points -`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at the store's -half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's. +first-boot unit mints a CA of the store's own plus the leaves it signs — the +store's server certificate, this host's client certificate, and one per service +principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and +`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and +`.serverCaFile` at the reader's, and each principal's own pair at its own leaf. Those are `mkDefault`s, so naming your own paths wins. Do that when your certificates come from a real internal CA; the store has no opinion about which. A hive that does **not** deploy the store names the reader's three -itself: the operator issues that leaf out of band, and it's the one credential -the store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is +itself, plus a pair for every principal it runs — see +[per-principal identities](../swarm/secrets.md#per-principal-identities) for the +list. The operator issues those leaves out of band; they're the credentials the +store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is **mTLS between services and the store**, a separate trust domain, because a store that took its identity from an authority it itself distributes could never come up before that authority. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 53bce3ad..b5f3c38a 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -88,7 +88,7 @@ where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — only its own pushes to the stores go out unauthenticated and get refused there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without -`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice +`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for their own optional readers, rather than the hard assertion `swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by @@ -249,14 +249,14 @@ straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store -exists so a host holds **one** out-of-band secret, its client certificate, and -reads everything else with it. +exists so the only secrets a host holds out of band are **certificates**, and +a host reads everything else with them. Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: running Grafana requires `swarm.authelia.url` (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the -only way in), and it requires this host's `deploy.bao.clientCertFile` / -`clientKeyFile`. Each refusal names the option to set. Both used to be silent: +only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile` +/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is @@ -271,7 +271,8 @@ runs the collector, and registration lives separately in `glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so `swarm-otel.nix` doesn't assert -`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it +`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana +does — it doesn't render the reading unit without them, the shape `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for their own optional reads. @@ -290,10 +291,18 @@ A service's secret is one value for the whole swarm rather than one per hive, so it lives under the `services` prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a `deploy.*` fact, per-host by definition — so -there is nothing to scope the grant to. Every hive can therefore read every swarm -service's client secret; that's the cost of a shape where the service's host has -no store identity of its own to present, and it's stated in -`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves +there is nothing to scope the grant to. A hive's **own** leaf can therefore read +every swarm service's client secret, and that's stated in +`swarm-secret-client`'s `policy` module beside the grant itself. + +⚠️ **The readers no longer present it.** Four units used to log in with +`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a +principal by the subject of the certificate it presents — so four readers behind +one leaf were one principal holding the union of their needs. Each now holds a +leaf of its own, and a policy naming only the path that unit reads. See +[per-principal identities](#per-principal-identities) below. + +Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in. @@ -303,14 +312,78 @@ credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the hive CA and the swarm CA both, and has nothing to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix` -mints a CA that signs exactly two things, the store's server certificate and a -reader's client certificate, and distributes nothing. A deployment with a real +mints a CA that signs a fixed, short list of leaves — the store's server +certificate, the hive's own reader certificate, and one per service principal — +and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in `deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. A hive that reads from a store on **another** machine names the reader's half — -`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand. -It's the one credential that can't come out of the store, being what opens it; -everything else a hive needs does. +`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand, +plus one leaf per principal it runs (the table below names the options). Those are the +credentials that can't come out of the store, being what opens it; everything +else a hive needs does. + +### Per-principal identities + +Bao matches a cert-auth role on the certificate's subject, so a certificate is an +identity and sharing one merges the identities. These four units read one path +each and each holds a leaf, a role and a policy of its own: + +| unit | option pair under `deploy.bao.` | reads | +| ------------------------ | -------------------------------------------------------- | -------------------------------------------------- | +| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives//matrix/appservice-token` | +| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives//queue/agent` | +| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services//oidc/client` | +| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | + +The first two exist **per hive**, because the path they read carries a hive +name and every hive runs its own reader. Their subjects are +`-` and +`-`; `swarm.nix` reserves both +composed spellings as hive names, so nobody can name a hive into another hive's +role. The other two read a path that names a swarm service rather than a hive, so +one role each is enough and their subjects are the flat +`deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`. + +On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults +all eight options, and there is nothing to do. Elsewhere you issue each leaf from +that CA out of band and name it here — one file per principal rather than one file +shared by four, which is the whole of what this buys. + +Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four +readers used to render only where their leaf existed, so a hand-configured +remote-store hive that named the hive's own `clientCertFile` and missed a +principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing +the build. All four now refuse it, each naming its own option pair — see +[what a missing leaf costs](#what-a-missing-leaf-costs) below. + +### What a missing leaf costs + +Each of the four modules asserts its own pair, on the same condition: **this host +already reads the store** — it holds `deploy.bao.clientCertFile` and +`clientKeyFile` — **and the consumer is on, and this principal's pair is null**. +The refusal names the two options, so an operator acts on the message without +opening the nix. + +That condition fires on exactly one shape, the hand-configured remote-store hive +that named seven of the eight. It stays silent everywhere else, and each half of +that matters on its own: + +- **A host with no store identity at all** holds no `clientCertFile` either, so + the first clause is false. That's the supported no-store deployment, and also + the one an operator passes through while bringing a hive up — neither should + fail to evaluate. +- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as + defaults, so the third clause is false. Nothing to set, nothing to refuse. +- **A consumer that's off** — no homeserver, no collector, no Grafana — has no + unit to skip, so the second clause is false. Only the queue-credential reader + has no toggle of its own to check: every hive runs one for its own agents, so + reading the store at all is what asks for it. + +The collector keeps a separate degrade underneath this, unchanged: a host with no +store identity runs a collector with no OIDC client secret and still receives +telemetry. The assertion doesn't touch that shape, because that host has no +`clientCertFile` either. ## How a reader reaches the store diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix index 819569bf..1afaed01 100644 --- a/nix/host-modules/glue-bao-tls.nix +++ b/nix/host-modules/glue-bao-tls.nix @@ -84,6 +84,25 @@ in clientCertFile = lib.mkDefault "${pkiDir}/client.pem"; clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem"; serverCaFile = lib.mkDefault "${pkiDir}/ca.pem"; + + # 🩸 Four readers that used to present `client.pem` above, each now + # pointed at a leaf of its own. The leaf is what bao sees, so this pairing + # is the whole of what turns "one principal with the union of four grants" + # into four principals with one grant each — see ./swarm-bao.nix's + # `perHiveReaders` block for the grants themselves. + # + # Defaulted here rather than in four `glue--bao-identity.nix` + # files: those exist where the consumer is a CONTAINER with a + # `deploy..*` namespace of its own to point at. These four are + # host units reading the store, which is the pairing this file already is. + matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem"; + matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem"; + queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem"; + queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem"; + grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem"; + grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem"; + otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem"; + otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem"; }; # Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates @@ -160,6 +179,35 @@ in # container an identity instead of lending it the hive's. [ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \ ${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth + + # 🩸 The four readers that used to present `client.pem` above. Each + # carries its own subject, which is the entire mechanism: bao matches a + # cert-auth role on the CN, so four units sharing one leaf were one + # principal holding the union of four grants, and the union included + # every agent credential in the swarm. + # + # ⚠️ The first two carry THIS hive's name in the subject, unlike the + # three service leaves above. Their grants name one hive's path, because + # a matrix token and a queue credential live under `swarm/hives//` + # and every hive runs a reader for its own; ./swarm-bao.nix writes one + # role per hive in the swarm directory to match. The two OIDC readers + # need no such segment — a client is registered once per swarm. + # + # Minted whether or not the consumer runs here, for the reason the three + # leaves above give: on a hive that does not run the store this is the + # file an operator copies, and a leaf that only appears where its + # consumer does is one nobody can copy from anywhere. + [ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \ + ${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth + + [ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \ + ${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth + + [ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \ + ${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth + + [ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \ + ${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth ''; }; }; diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix index 077c7b86..3c0afcb6 100644 --- a/nix/host-modules/glue-matrix-bao-token.nix +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -38,7 +38,24 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: read on + # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when + # this unit reads one appservice token and nothing else. Its own leaf carries + # `-` and its role grants the single path + # below. + haveClientIdentity = + baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null; + + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. Only + # used to decide whether a missing per-principal leaf is a mistake or a + # deployment that has no store: a host holding neither is the supported + # no-store shape, and one holding this pair but not the pair above named + # seven of the eight options and stopped. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; # Where the token lives in the store. A path, not a convention to guess at: # whoever writes it and whoever reads it must agree, and the agreement @@ -64,134 +81,178 @@ let matrixMachine = "hive-matrix"; in { - config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { - # Same rule as the unit's own gate: this reader exists on a host that has a - # client identity and a homeserver, which is not every host that runs the - # store, so the store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; + config = lib.mkMerge [ + # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole + # mechanism: the unit's arm is gated on `haveClientIdentity`, so an + # assertion written inside it could never be reached in the state it + # exists to report. + # + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the + # same refusal, named to this principal's own pair. What differs is the + # gate. Grafana asserts wherever Grafana runs, because a Grafana with no + # store identity has no way in at all; a homeserver with no store identity + # is a hive that has no store, which is supported. So this one additionally + # requires `hiveReaderIdentity`: the host demonstrably reads the store, and + # named every option but this pair. + (lib.mkIf (hyperhiveCfg.enable && deployCfg.matrix.enable && hiveReaderIdentity) { + assertions = [ + { + assertion = haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set) and runs a homeserver, so it needs the matrix appservice + token reader's own client identity: set both - systemd.services.swarm-bao-matrix-token = { - description = "fetch the matrix appservice token from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" + services.hyperhive.deploy.bao.matrixTokenClientCertFile + services.hyperhive.deploy.bao.matrixTokenClientKeyFile + + swarm-bao-matrix-token.service fetches this hive's appservice token + out of the store, and without these it is not rendered at all — + leaving the homeserver authenticating hive-c0re against whatever is + already on disk, which is a 401 on every request naming nothing. + + ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is + the hive's, and its grant reads every secret in the store; this role + reads the one appservice-token path. Pointing this option at the + hive's leaf would evaluate, deploy and log in — and undo the split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - before = [ "container@${matrixMachine}.service" ]; - wantedBy = [ "container@${matrixMachine}.service" ]; - path = [ - deployCfg.bao.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal. `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after — so a few short attempts cover it. - # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. - # That unit blocks nothing; this one is `Before=` the homeserver's - # container, and whether that ordering waits across an auto-restart is - # unverified — so an hours-long window would be a bet on an unknown, - # where a minute is not. A store still sealed after it keeps the degrade - # below, as today. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the read below. Stated here rather than - # left to systemd's default, so the number a boot waits on is in - # the file that waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail + }) - # A sealed or uninitialised store answers on the port and never - # answers the read, so "the store is up" is not the same as "the - # store can answer". `TimeoutStartSec` above is the bound; the - # homeserver only `Wants=` this unit, so hitting it degrades to - # keeping the local token rather than holding up the container. - # `bao`'s own message is the only thing separating a missing value - # from a refused identity from an unreachable host. This unit's - # degraded mode is correct for all three, so it reports which one - # rather than asserting all three in a sentence of ours — a reader - # that cannot say why it read nothing is indistinguishable from a - # broken one. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT + (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { + # Same rule as the unit's own gate: this reader exists on a host that has a + # client identity and a homeserver, which is not every host that runs the + # store, so the store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. + systemd.services.swarm-bao-matrix-token = { + description = "fetch the matrix appservice token from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + before = [ "container@${matrixMachine}.service" ]; + wantedBy = [ "container@${matrixMachine}.service" ]; + path = [ + deployCfg.bao.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal. `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after — so a few short attempts cover it. + # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. + # That unit blocks nothing; this one is `Before=` the homeserver's + # container, and whether that ordering waits across an auto-restart is + # unverified — so an hours-long window would be a bet on an unknown, + # where a minute is not. A store still sealed after it keeps the degrade + # below, as today. # - # Fails LOUDLY, unlike the read below: the three states a login failure - # covers — store not up, sealed, role not written yet — are all things - # a retry fixes, and `Restart=on-failure` above is what retries. Exiting - # 0 here spends the whole boot on a condition that was seconds old. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the read below. Stated here rather than + # left to systemd's default, so the number a boot waits on is in + # the file that waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; + BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # A sealed or uninitialised store answers on the port and never + # answers the read, so "the store is up" is not the same as "the + # store can answer". `TimeoutStartSec` above is the bound; the + # homeserver only `Wants=` this unit, so hitting it degrades to + # keeping the local token rather than holding up the container. + # `bao`'s own message is the only thing separating a missing value + # from a refused identity from an unreachable host. This unit's + # degraded mode is correct for all three, so it reports which one + # rather than asserting all three in a sentence of ours — a reader + # that cannot say why it read nothing is indistinguishable from a + # broken one. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the read below: the three states a login failure + # covers — store not up, sealed, role not written yet — are all things + # a retry fixes, and `Restart=on-failure` above is what retries. Exiting + # 0 here spends the whole boot on a condition that was seconds old. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 fi - exit 1 - fi - export BAO_TOKEN + export BAO_TOKEN - if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then - echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then + echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - if [ -z "$token" ]; then - echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 - exit 0 - fi + if [ -z "$token" ]; then + echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 + exit 0 + fi - umask 077 - printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + umask 077 + printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - # Re-stamp the registration file from the token just written. The - # token is half an agreement — the registration the homeserver loads - # has to carry the same value — so writing the file and stopping - # would leave the homeserver authenticating hive-c0re against - # whatever activation put there: a 401 on every request, naming - # nothing. Unconditional rather than on-change, because this unit - # has no way to know what the registration currently says. - # - # hive-matrix's own renderer rather than a `printf` here, so the - # registration's shape has one home. - ${deployCfg.matrix.appserviceRegistrationScript} - ''; - }; - }; + # Re-stamp the registration file from the token just written. The + # token is half an agreement — the registration the homeserver loads + # has to carry the same value — so writing the file and stopping + # would leave the homeserver authenticating hive-c0re against + # whatever activation put there: a 401 on every request, naming + # nothing. Unconditional rather than on-change, because this unit + # has no way to know what the registration currently says. + # + # hive-matrix's own renderer rather than a `printf` here, so the + # registration's shape has one home. + ${deployCfg.matrix.appserviceRegistrationScript} + ''; + }; + }) + ]; } diff --git a/nix/host-modules/glue-queue-agent-credential.nix b/nix/host-modules/glue-queue-agent-credential.nix index 9e57b512..8fa2dc06 100644 --- a/nix/host-modules/glue-queue-agent-credential.nix +++ b/nix/host-modules/glue-queue-agent-credential.nix @@ -40,7 +40,24 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: read on + # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when + # this unit reads one queue credential and nothing else. Its own leaf carries + # `-` and its role grants the single path + # below — still this hive's own, so the narrowing costs no reach. + haveClientIdentity = + baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null; + + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. Only + # used to decide whether a missing per-principal leaf is a mistake or a + # deployment that has no store: a host holding neither is the supported + # no-store shape, and one holding this pair but not the pair above named + # seven of the eight options and stopped. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir; secretFile = "${credentialDir}/secret"; @@ -85,144 +102,192 @@ in }; }; - config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { - # Same rule as the unit's own gate: this reader exists on any host holding - # a client identity, which is not every host that runs the store, so the - # store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; + config = lib.mkMerge [ + # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole + # mechanism: the unit's arm is gated on `haveClientIdentity`, so an + # assertion written inside it could never be reached in the state it + # exists to report. + # + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the + # same refusal, named to this principal's own pair. Where Grafana's gate + # is `deploy.grafana.enable`, this reader has no toggle of its own to + # check: every hive runs one for its own agents, so reading the store at + # all is what asks for it. Hence `hiveReaderIdentity` alone — a host + # holding no hive leaf has no store to read and nothing is missing. + (lib.mkIf (hyperhiveCfg.enable && hiveReaderIdentity) { + assertions = [ + { + assertion = haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set), so it needs the agent queue credential reader's own client + identity: set both - systemd.services.swarm-bao-queue-agent = { - description = "fetch this hive's agent queue credential from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" - ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - # Ordered before hive-c0re, so no agent container renders ahead of an - # attempt at its credential. `Wants=`, not `Requires=`: a store this - # unit can't reach delays hive-c0re's start by its own start-limit - # window (`TimeoutStartSec`, retried up to `startLimitBurst` times - # below) rather than failing it — hive-c0re starts once that window - # elapses, whatever credential is or isn't on disk by then. - before = [ "hive-c0re.service" ]; - wantedBy = [ - "multi-user.target" - "hive-c0re.service" - ]; - path = [ - baoDeploy.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal: `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after, so a few short attempts cover it. - # An hours-long window would be a bet on a store that is sealed, and the - # degrade below is already correct for that. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the reads below. Stated here rather than left to - # systemd's default, so the number a boot waits on is in the file that - # waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail + services.hyperhive.deploy.bao.queueAgentClientCertFile + services.hyperhive.deploy.bao.queueAgentClientKeyFile - # `bao`'s own message is the only thing separating a missing value from - # a refused identity from an unreachable host. This unit's degraded - # mode is correct for all three, so it reports which one rather than - # asserting all three in a sentence of ours. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT + swarm-bao-queue-agent.service fetches this hive's agent queue + credential out of the store, and without these it is not rendered + at all — leaving hive-c0re with no queue credential and no unit + that would ever have written one. - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. + Every hive runs this reader for its own agents, so unlike the other + three principals there is no per-service toggle that turns it off: + a hive that reads the store at all is a hive that needs this pair. + + ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one + is the hive's, and its grant reads every secret in the store; this + role reads the one queue-credential path. Pointing this option at + the hive's leaf would evaluate, deploy and log in — and undo the + split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } + ]; + }) + + (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { + # Same rule as the unit's own gate: this reader exists on any host holding + # a client identity, which is not every host that runs the store, so the + # store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; + + systemd.services.swarm-bao-queue-agent = { + description = "fetch this hive's agent queue credential from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" + ]; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + # Ordered before hive-c0re, so no agent container renders ahead of an + # attempt at its credential. `Wants=`, not `Requires=`: a store this + # unit can't reach delays hive-c0re's start by its own start-limit + # window (`TimeoutStartSec`, retried up to `startLimitBurst` times + # below) rather than failing it — hive-c0re starts once that window + # elapses, whatever credential is or isn't on disk by then. + before = [ "hive-c0re.service" ]; + wantedBy = [ + "multi-user.target" + "hive-c0re.service" + ]; + path = [ + baoDeploy.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal: `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after, so a few short attempts cover it. + # An hours-long window would be a bet on a store that is sealed, and the + # degrade below is already correct for that. # - # Fails LOUDLY, unlike the reads below: the three states a login - # failure covers — store not up, sealed, role not written yet — are all - # things a retry fixes, and `Restart=on-failure` above is what retries. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the reads below. Stated here rather than left to + # systemd's default, so the number a boot waits on is in the file that + # waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; + BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # `bao`'s own message is the only thing separating a missing value from + # a refused identity from an unreachable host. This unit's degraded + # mode is correct for all three, so it reports which one rather than + # asserting all three in a sentence of ours. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the reads below: the three states a login + # failure covers — store not up, sealed, role not written yet — are all + # things a retry fixes, and `Restart=on-failure` above is what retries. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 fi - exit 1 - fi - export BAO_TOKEN + export BAO_TOKEN - # Two reads of one object rather than one `-format=json` parsed with - # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot - # actually disagree — the client id is derived from this hive's name, so - # a rotation landing between these two calls changes the secret and - # rewrites the same id. - if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + # Two reads of one object rather than one `-format=json` parsed with + # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot + # actually disagree — the client id is derived from this hive's name, so + # a rotation landing between these two calls changes the secret and + # rewrites the same id. + if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 + if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 fi - exit 0 - fi - # Both or neither, for the reason `QueueConfig::from_env` refuses a - # half-set environment: a client that finds one of the two comes up - # "fine" and never connects. - if [ -z "$secret" ] || [ -z "$client_id" ]; then - echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 - exit 0 - fi + # Both or neither, for the reason `QueueConfig::from_env` refuses a + # half-set environment: a client that finds one of the two comes up + # "fine" and never connects. + if [ -z "$secret" ] || [ -z "$client_id" ]; then + echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 + exit 0 + fi - install -d -m 0755 ${lib.escapeShellArg credentialDir} + install -d -m 0755 ${lib.escapeShellArg credentialDir} - umask 077 - printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} - chmod 0600 ${lib.escapeShellArg secretFile} + umask 077 + printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} + chmod 0600 ${lib.escapeShellArg secretFile} - # `0644` on purpose: an OIDC client id is presented to the token - # endpoint on every connection and is public by construction. - printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} - chmod 0644 ${lib.escapeShellArg clientIdFile} - ''; - }; - }; + # `0644` on purpose: an OIDC client id is presented to the token + # endpoint on every connection and is public by construction. + printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} + chmod 0644 ${lib.escapeShellArg clientIdFile} + ''; + }; + }) + ]; } diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index e577a988..8fd7474c 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -340,6 +340,156 @@ let } ''; + # ── the four principals that used to share the hive's own leaf ───────────── + # + # 🩸 Each of the four reads exactly ONE path in the store, and until this + # split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf, + # whose policy (`swarm-secret-client`'s `policy::render`) grants read on + # `swarm/agents/*`, `swarm/hives//*` AND `swarm/services/*`. Four + # principals presenting one certificate are ONE principal to bao, so the union + # of what the four need was the only grant expressible: the unit that fetches + # Grafana's OIDC client secret could fetch every agent credential in the + # swarm, and the unit that fetches this hive's matrix token could fetch + # Grafana's. Least privilege was not misconfigured, it was unrepresentable. + # + # Each now gets its own leaf, its own cert-auth role and a policy holding the + # one `secret/data/…` path that principal's script actually names — spelled to + # the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above + # gives for doing the same. + # + # ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the + # shape of the paths rather than a preference. A matrix appservice token and a + # queue credential live under `swarm/hives//`, and every hive runs a + # reader for its own; an OIDC client secret lives under + # `swarm/services//` and a swarm registers each of those exactly + # once. A single role for a per-hive reader would have to name `hives/*` to + # serve every hive — which would let one hive read another's matrix token, a + # reach no hive has today and one this change exists to remove rather than + # create. + readStanza = path: '' + path "${path}" { + capabilities = ["read"] + } + ''; + + # One reader per hive in the swarm directory. Written from the directory + # rather than from this host's own name because the policy is written where + # the STORE is and the reader runs where its hive is — the same split + # `matrixCtlHiveName` above exists to paper over, answered here by naming + # every hive instead of asking the operator which one. + perHiveReaders = + { rolePrefix, cnPrefix, ... }@spec: + lib.mapAttrsToList (hiveName: _: { + name = "${rolePrefix}-${hiveName}"; + cn = "${cnPrefix}-${hiveName}"; + policyText = readStanza (spec.path hiveName); + }) hyperhiveCfg.swarm.hives; + + matrixTokenReaders = perHiveReaders { + rolePrefix = "swarm-matrix-token"; + cnPrefix = baoDeploy.matrixTokenCommonNamePrefix; + path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token"; + }; + + queueAgentReaders = perHiveReaders { + rolePrefix = "swarm-queue-agent"; + cnPrefix = baoDeploy.queueAgentCommonNamePrefix; + path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent"; + }; + + # Singletons, so a list of one rather than a second shape: the granting unit + # below takes a list either way and the four units stay identical apart from + # the objects they write. + grafanaOidcReaders = [ + { + name = "swarm-grafana-oidc"; + cn = baoDeploy.grafanaOidcCommonName; + policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"; + } + ]; + + otelOidcReaders = [ + { + name = "swarm-otel-oidc"; + cn = baoDeploy.otelOidcCommonName; + policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.otel.clientId}/oidc/client"; + } + ]; + + # The role name IS the policy name, as for the three service principals + # above: the role attaches the policy by spelling it identically, and one + # string for both objects removes the way they drift apart. + readerPolicyWrite = obj: '' + printf '%s' ${lib.escapeShellArg obj.policyText} | + bao policy write ${lib.escapeShellArg obj.name} - + ''; + + # ⚠️ Names are `swarm-[-]`, outside the `hive-*` namespace the + # controller may rewrite, for the reason the three service principals above + # state. A per-hive CN cannot collide with a hive's own leaf either — a hive's + # CN is its bare name and ../reserved-hive-fragments.nix forbids the substring + # `swarm` in one — and ./swarm.nix reserves the concrete per-hive spellings + # anyway, because the prefixes are operator-settable and an operator may spell + # one without that substring. + readerRoleWrite = obj: '' + bao write auth/cert/certs/${lib.escapeShellArg obj.name} \ + certificate=@${tlsDir}/client-ca.pem \ + allowed_common_names=${lib.escapeShellArg obj.cn} \ + token_policies=${lib.escapeShellArg obj.name} \ + display_name=${lib.escapeShellArg obj.cn} + ''; + + # One unit per PRINCIPAL, not one unit for the four: these names are + # operator-facing strings, the same reason `swarm-bao-secret-publisher-policy` + # above gives for not being more script inside the controller's unit. A hive + # whose matrix token never arrives is diagnosed by the state of the unit named + # after it. + # + # `after` and not `requires`, for the reason the publisher's unit states: the + # controller's unit creates the KV and cert-auth mounts this one writes into, + # but a failed oneshot still counts as finished, so ordering plus this unit's + # own retry is what converges. + # + # The role write is inside the client-CA branch and the policy write is not, + # exactly as the three above: with no CA there is no trust anchor for a login + # role, but the policy it would attach is still worth asserting. + readerPolicyUnit = + description: objects: + lib.mkIf haveBootstrapToken { + inherit description; + after = [ + "container@${cfg.machine}.service" + "swarm-bao-controller-policy.service" + ]; + wantedBy = [ "multi-user.target" ]; + path = [ + baoCli + pkgs.coreutils + ]; + unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; + # Same unseal wait as its siblings above, for the reason stated there: + # under `seal = "shamir"` a human unseals by hand. + startLimitBurst = 2880; + startLimitIntervalSec = 90000; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + Restart = "on-failure"; + RestartSec = 30; + }; + script = '' + set -euo pipefail + + BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})" + export BAO_TOKEN + + '' + + lib.concatMapStrings readerPolicyWrite objects + + lib.optionalString (baoDeploy.clientCaFile != null) ( + "\n" + lib.concatMapStrings readerRoleWrite objects + ); + }; + # Every listener serves the same identity: they differ in which address # they answer on, not in who they are. Client verification is separate and # optional — a store with no `clientCaFile` still serves TLS, it just does @@ -846,6 +996,185 @@ in ''; }; + matrixTokenCommonNamePrefix = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-matrix-token"; + example = "swarm-bao-matrix-token.svc"; + description = '' + Prefix of the subject the store's per-hive matrix-token roles accept. + The role for hive `h` accepts `-h` and nothing else, and grants + read on exactly `swarm/hives/h/matrix/appservice-token`. + + A **prefix** rather than one subject because every hive runs its own + reader and the path it reads carries that hive's name. One subject for + all of them would need a `hives/*` grant, which would let one hive read + another hive's appservice token — a reach no hive has today. + + ⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its + fixed-subject siblings are: cert auth trusts the CA and + `allowed_common_names` is the whole narrowing, so a hive whose own leaf + carried one of these spellings would receive that grant. + ''; + }; + + queueAgentCommonNamePrefix = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-queue-agent"; + example = "swarm-bao-queue-agent.svc"; + description = '' + Prefix of the subject the store's per-hive queue-credential roles + accept. The role for hive `h` accepts `-h` and grants read on + exactly `swarm/hives/h/queue/agent` — the client credential that hive's + agent containers authenticate to the swarm queue with. + + Per hive for the same reason + {option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is, + and more sharply: every hive in the swarm runs this reader, so a single + subject would have to be granted `hives/*`. + ''; + }; + + grafanaOidcCommonName = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-grafana-oidc"; + example = "swarm-bao-grafana-oidc.svc"; + description = '' + Subject the store's `swarm-grafana-oidc` cert-auth role accepts — the + identity the unit that fetches Grafana's OIDC client secret presents. + + Its grant is one path, + `swarm/services//oidc/client`, and read only. Not + per hive, unlike the two prefixes above: an OIDC client is registered + once per swarm, so the path names the service and never a hive. + + ⚠️ Same collision as its siblings, and the same answer: ./swarm.nix + feeds this value into the guard on + {option}`services.hyperhive.swarm.hives`. + ''; + }; + + otelOidcCommonName = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-otel-oidc"; + example = "swarm-bao-otel-oidc.svc"; + description = '' + Subject the store's `swarm-otel-oidc` cert-auth role accepts — the + identity the unit that fetches the collector's OIDC client secret + presents. Its grant is one path, + `swarm/services//oidc/client`, and read only. + + A **fourth** identity rather than reuse of + {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two + read different services' client secrets, and a collector is not + entitled to Grafana's. + ''; + }; + + matrixTokenClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/matrix-token.pem"; + description = '' + Certificate the unit that fetches this hive's matrix appservice token + presents to the store. Its subject must be + `-`; cert auth matches on + the CN and the role accepts nothing else. + + ⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`. + Pointing this at the hive's own leaf would evaluate, deploy and log in — + and hand a unit that reads one token a credential that reads every + secret in the store. That is what this option exists to end. + + No default here: ./glue-bao-tls.nix points it at the leaf it mints, + wherever this host mints its own PKI. A deployment that reads a store on + another host names it by hand, alongside the leaf itself — one file per + principal placed out of band instead of one shared by four. + ''; + }; + + matrixTokenClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/matrix-token-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both + or neither — a certificate with no key authenticates nothing. + ''; + }; + + queueAgentClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/queue-agent.pem"; + description = '' + Certificate the unit that fetches this hive's queue client credential + presents to the store. Its subject must be + `-`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + queueAgentClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/queue-agent-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`. + ''; + }; + + grafanaOidcClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/grafana-oidc.pem"; + description = '' + Certificate the unit that fetches Grafana's OIDC client secret presents + to the store. Its subject must be + {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + grafanaOidcClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/grafana-oidc-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile`. + ''; + }; + + otelOidcClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/otel-oidc.pem"; + description = '' + Certificate the unit that fetches the collector's OIDC client secret + presents to the store. Its subject must be + {option}`services.hyperhive.deploy.bao.otelOidcCommonName`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + otelOidcClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/otel-oidc-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.otelOidcClientCertFile`. + ''; + }; + serverCaFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; @@ -1593,6 +1922,19 @@ in ''; }; + # 🩸 The four principals that used to log in as the hive. See the block + # around `perHiveReaders` for what each grant is and why the first two are + # per hive; the four units are identical apart from the objects they + # write, so the shape lives in `readerPolicyUnit` and the difference is + # visible here in one line each. + systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders; + + systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders; + + systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders; + + systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders; + # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 006e3abf..85e5a2d1 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -119,7 +119,14 @@ let # other secret is read with it. A Grafana host without it has not been given # its identity yet, which is a thing to say out loud rather than to route # around by reaching into authelia's tree whenever it happens to be local. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 Grafana's OWN leaf, not `clientCertFile` — the hive's, which four units + # used to share. Bao matches a cert-auth role on the CN, so one leaf for four + # readers was ONE principal holding the union of four grants: this unit could + # read every agent credential in the swarm and every other service's OIDC + # client secret, when what it needs is the one path `storeSecretPath` names. + haveClientIdentity = + baoDeploy.grafanaOidcClientCertFile != null && baoDeploy.grafanaOidcClientKeyFile != null; autheliaUrl = toString hyperhiveCfg.swarm.authelia.url; @@ -451,15 +458,20 @@ in services.hyperhive.deploy.grafana.enable requires this host to hold a swarm-secret-store client identity: set both - services.hyperhive.deploy.bao.clientCertFile - services.hyperhive.deploy.bao.clientKeyFile + services.hyperhive.deploy.bao.grafanaOidcClientCertFile + services.hyperhive.deploy.bao.grafanaOidcClientKeyFile Grafana's OIDC client secret is minted by authelia and read out of the store, on every host that runs Grafana — including the host that runs authelia. That is one delivery route rather than two, and it is - what the store is for: this certificate is the single credential - placed out of band, and every other secret comes from the store with - it. + what the store is for: a certificate is the credential placed out of + band, and every other secret comes from the store with it. + + ⚠️ Grafana's OWN leaf, not deploy.bao.clientCertFile. That one is the + hive's, and its grant reads every secret in the store; this role + reads the one path Grafana's client secret lives at. Pointing this + option at the hive's leaf would evaluate, deploy and log in — and + undo the split. On a hive that runs the store, glue-bao-tls.nix supplies both as defaults and there is nothing to do. Elsewhere the leaf is issued @@ -610,8 +622,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 868df8b7..c53bfd21 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -242,13 +242,32 @@ let # A reader of the store is defined by holding a certificate the store # accepts, never by standing next to it — the rule # ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical- - # looking flag, this one stays outside `assertions`: the store-reading unit - # below simply does not render without it, the same choice + # looking flag, this one does not gate an assertion on its own: the + # store-reading unit below simply does not render without it, the same choice # ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for - # their own optional readers, because a collector with no client identity is + # their own readers, because a collector with no client identity is # `haveCollectorSecret = false` above, and that is already a supported, - # merely degraded shape rather than a service with no way in at all. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # merely degraded shape rather than a service with no way in at all. What + # IS asserted is narrower and lives in `assertions` below — see + # `hiveReaderIdentity`. + # + # 🩸 The collector's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: this unit + # could read every agent credential in the swarm and Grafana's OIDC client + # secret, when what it needs is the one path `storeSecretPath` names. + haveClientIdentity = + baoDeploy.otelOidcClientCertFile != null && baoDeploy.otelOidcClientKeyFile != null; + + # Does this host read the store at all — the hive's own leaf, which is the + # one thing a remote-store deployment has always had to place by hand. It is + # what separates the degrade the comment above describes from the mistake the + # assertion below reports: a collector on a host holding NO store identity is + # the supported shape, and one on a host that demonstrably reads the store + # named seven of the eight options and stopped. Before the four-way split + # that second host rendered this unit off `clientCertFile`, so it is a silent + # regression rather than a choice anyone made. + hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; # Where the publisher on authelia's host leaves this client's secret — # composed from the same swarm-wide `clientId` the registration in @@ -725,11 +744,14 @@ in # value. # # ⚠️ Renders only where `haveClientIdentity` holds, unlike - # ./swarm-grafana.nix's equivalent unit. That module asserts the identity - # because a Grafana with none has no way in at all; this collector without - # one is `haveCollectorSecret = false` above — already a supported, - # merely degraded shape, so the unit that would fetch a credential simply - # does not exist rather than refusing the build for want of one. + # ./swarm-grafana.nix's equivalent unit. That module refuses any host that + # runs Grafana without the identity, because a Grafana with none has no way + # in at all; this collector without one is `haveCollectorSecret = false` + # above — already a supported, merely degraded shape, so the unit that would + # fetch a credential simply does not exist rather than refusing the build + # for want of one. The one case that IS refused is the host that already + # holds the hive's leaf and is missing only this pair, which is a silent + # regression rather than that degrade — `hiveReaderIdentity` above. systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity { description = "fetch the swarm collector's OIDC client secret from the swarm secret store"; # Every one of these names a unit that exists only where the store runs. @@ -775,8 +797,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.otelOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.otelOidcClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. @@ -853,6 +875,45 @@ in systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; assertions = [ + { + # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — + # the same refusal, named to this principal's own pair. What differs is + # the `!hiveReaderIdentity ||` guard, and it is what keeps the degrade + # the flag's own comment describes intact: Grafana refuses any host + # that runs it without a leaf, because a Grafana with no SSO has no way + # in at all, while a collector with none still receives telemetry. So + # this fires only where the host already reads the store and is missing + # this one pair — which before the four-way split rendered the unit off + # `clientCertFile`, and now silently does not. + assertion = !hiveReaderIdentity || haveClientIdentity; + message = '' + This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile + is set) and runs the swarm collector, so it needs the collector's own + client identity: set both + + services.hyperhive.deploy.bao.otelOidcClientCertFile + services.hyperhive.deploy.bao.otelOidcClientKeyFile + + swarm-bao-otel-oidc.service fetches the collector's OIDC client + secret out of the store, and without these it is not rendered at all + — so this collector would authenticate to nothing, quietly, on a host + that has everything it needs to fetch the secret. + + A collector on a host holding NO store identity is a different and + supported shape: it runs without a client secret and still receives + telemetry. That is not this host. + + ⚠️ The collector's OWN leaf, not deploy.bao.clientCertFile. That one + is the hive's, and its grant reads every secret in the store; this + role reads the one path this collector's client secret lives at. + Pointing this option at the hive's leaf would evaluate, deploy and + log in — and undo the split. + + On a hive that runs the store, glue-bao-tls.nix supplies both as + defaults and there is nothing to do. Elsewhere the leaf is issued + from that CA out of band and named here — see docs/swarm/secrets.md. + ''; + } # ⚠️ An assertion that this collector has "somewhere to send" was REMOVED # rather than relaxed: it read the store's *per-host* enable, so it # rejected at eval the very deployment the stores are reached by domain diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index d0e36dcd..213aa382 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -47,7 +47,20 @@ let deployCfg.bao.controllerCommonName deployCfg.bao.secretPublisherCommonName deployCfg.bao.matrixCtlCommonName - ]; + deployCfg.bao.grafanaOidcCommonName + deployCfg.bao.otelOidcCommonName + ] + # The two per-hive readers' subjects, spelled out per hive rather than as the + # prefix. The prefix alone would reserve the wrong string: the role for hive + # `h` accepts `-h`, so the name a hive must not BE is that composed + # spelling, and a hive named it would present a leaf that role accepts. + # + # Both spellings are reserved for every declared hive, including the one the + # CN belongs to — a hive cannot be named after its own reader either. + ++ lib.concatMap (hive: [ + "${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}" + "${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}" + ]) hiveNames; # Public hostnames of the swarm's own services, in declaration order. # `serviceDomains` below is this set sorted + deduplicated. diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 2068620c..c03e0fcf 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -45,6 +45,18 @@ let deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; deploy.bao.clientCaFile = lib.mkForce null; }; + # The store plus every one of the four readers that used to log in as the + # hive. One fixture rather than four: the claim they are four *separate* + # principals is only testable where all four render at once — that is the + # deployment in which two of them sharing a leaf would be invisible. + baoGrantWithConsumers = hive { + deploy.bao.enable = true; + deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; + deploy.matrix.enable = true; + deploy.grafana.enable = true; + deploy.swarm-otel.enable = true; + }; + cases = [ { # Reads the rendered unit on the HOST, which is where the write happens: @@ -196,6 +208,250 @@ let ) && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-matrix-ctl-policy); } + + # ── the four readers that used to share the hive's own leaf ────────────── + # + # 🩸 Until this split all four presented `deploy.bao.clientCertFile`, whose + # policy grants read on `swarm/agents/*`, `swarm/hives//*` AND + # `swarm/services/*`. Four principals behind one certificate are one + # principal to bao, so the only expressible grant was the union: the unit + # fetching Grafana's OIDC secret could fetch every agent credential in the + # swarm. + # + # Every one of these cases carries the same three negative arms, and they + # are the deliverable rather than decoration — a positive arm alone passes + # just as well when the other two stanzas are still there beside it. The + # arms pin what each principal must NOT reach, so a later widening fails + # here instead of being noticed in a store. + { + name = "the matrix-token reader's grant is one hive's appservice token and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-matrix-token-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/hives/h1/matrix/appservice-token\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + # The three stanzas the hive's own leaf carried, none of which this + # principal needs: every agent's credential, every service's OIDC + # client, and the rest of its own hive's tree — including the queue + # credential its sibling reader fetches. + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/services" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/queue" s) + # A `hives/*` wildcard would serve every hive from one role and let any + # hive read any other's token — the reach this split exists to remove, + # not to create. + && !(lib.hasInfix "secret/data/swarm/hives/*" s) + # Nothing may rewrite the policy constraining it, for the reason the + # controller's own `hive-*` narrowing above gives. + && !(lib.hasInfix "sys/policies/acl" s); + } + { + name = "the queue-credential reader's grant is one hive's queue credential and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/services" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s) + && !(lib.hasInfix "secret/data/swarm/hives/*" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # ⚠️ The client id is the path segment, so the negative arm that matters + # for this one is the OTHER service's: `services/*` would have granted + # both, and the two are separate principals precisely because a + # dashboard is not entitled to a collector's credential. + name = "the Grafana OIDC reader's grant is Grafana's own client secret and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/services/swarm-grafana/oidc/client\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/hives" s) + && !(lib.hasInfix "secret/data/swarm/services/*" s) + && !(lib.hasInfix "swarm-collector" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # The mirror of the case above, and the arm naming `swarm-grafana` is why + # these are two principals rather than one `services/*` grant shared. + name = "the collector OIDC reader's grant is the collector's own client secret and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-otel-oidc-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/services/swarm-collector/oidc/client\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/hives" s) + && !(lib.hasInfix "secret/data/swarm/services/*" s) + && !(lib.hasInfix "swarm-grafana" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # 🩸 The half that makes the policies above bind: a policy grants only + # through a token that carries it, and a token is minted by a cert-auth + # role matching a CN. Four distinct subjects is the whole mechanism — one + # subject for four readers is one principal however the policies read. + # + # The per-hive subjects carry the hive name because their paths do; the + # two service subjects do not, because an OIDC client is registered once + # per swarm. Pinned so neither shape is tidied into the other. + name = "each of the four readers logs in under a subject of its own"; + ok = + let + subjectOf = + unit: role: cn: + let + s = baoGrantHere.systemd.services.${unit}.script; + in + lib.hasInfix "auth/cert/certs/${role}" s + && lib.hasInfix "allowed_common_names=${cn}" s + && lib.hasInfix "token_policies=${role}" s + # Outside the `hive-*` namespace the controller may rewrite, for + # the reason the three service principals above state. + && !(lib.hasInfix "auth/cert/certs/hive-" s); + in + subjectOf "swarm-bao-matrix-token-policy" "swarm-matrix-token-h1" "swarm-bao-matrix-token-h1" + && subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1" + && subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc" + && subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"; + } + { + # 🩸 The consuming side, and the arm that would catch the regression that + # costs the most: a unit repointed back at `deploy.bao.clientCertFile` + # evaluates, deploys and logs in — and silently restores the union grant, + # because bao would again see one principal. Nothing about the policies + # above would look wrong. + # + # Each pair is asserted whole: a certificate with no key authenticates + # nothing, so a half-set pair is a reader that does not render. + name = "each of the four readers presents its own leaf, never the hive's"; + ok = + let + b = baoGrantWithConsumers.services.hyperhive.deploy.bao; + hiveLeaf = [ + b.clientCertFile + b.clientKeyFile + ]; + own = [ + b.matrixTokenClientCertFile + b.matrixTokenClientKeyFile + b.queueAgentClientCertFile + b.queueAgentClientKeyFile + b.grafanaOidcClientCertFile + b.grafanaOidcClientKeyFile + b.otelOidcClientCertFile + b.otelOidcClientKeyFile + ]; + envOf = unit: baoGrantWithConsumers.systemd.services.${unit}.environment; + presents = + unit: cert: key: + (envOf unit).BAO_CLIENT_CERT == cert && (envOf unit).BAO_CLIENT_KEY == key; + in + lib.all (p: p != null) own + && !(lib.any (p: lib.elem p hiveLeaf) own) + && lib.length (lib.unique own) == lib.length own + && presents "swarm-bao-matrix-token" b.matrixTokenClientCertFile b.matrixTokenClientKeyFile + && presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile + && presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile + && presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile; + } + { + # The minting side of the same claim. A role matching a subject nothing + # signs is a reader that cannot log in, so the leaves and the roles have + # to be asserted against each other — and the two per-hive leaves carry + # THIS host's hive name, which is what makes one hive's leaf useless + # against another hive's role. + name = "the PKI unit signs a leaf per reader, each under that reader's own subject"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-pki.script; + in + # The basename and the subject are matched separately: `signLeaf` takes + # them as consecutive arguments across a `\` continuation, so one + # literal spanning both would pin this file's line wrapping rather than + # the pairing it means to. + lib.all (lib.flip lib.hasInfix s) [ + "/matrix-token.pem ]" + "swarm-bao-matrix-token-h1 \"\" clientAuth" + "/queue-agent.pem ]" + "swarm-bao-queue-agent-h1 \"\" clientAuth" + "/grafana-oidc.pem ]" + "swarm-bao-grafana-oidc \"\" clientAuth" + "/otel-oidc.pem ]" + "swarm-bao-otel-oidc \"\" clientAuth" + ]; + } + { + # The absence arm: with no client CA there is no trust anchor, so the + # login roles cannot be written — but the policies they would attach are + # still asserted, exactly as the three service principals above behave in + # this deployment. A unit that vanished here would take the policy with + # it and leave nothing to diagnose. + name = "with no client CA the four readers get policies but no login roles"; + ok = + let + units = [ + "swarm-bao-matrix-token-policy" + "swarm-bao-queue-agent-policy" + "swarm-bao-grafana-oidc-policy" + "swarm-bao-otel-oidc-policy" + ]; + scriptOf = unit: baoGrantNoClientCa.systemd.services.${unit}.script; + in + lib.all ( + unit: + (baoGrantNoClientCa.systemd.services ? ${unit}) + && lib.hasInfix "bao policy write" (scriptOf unit) + && !(lib.hasInfix "auth/cert/certs" (scriptOf unit)) + ) units; + } + { + # Same control the three service principals carry: the write needs a + # client certificate and the host is the side that has one, so a unit + # rendered inside the store's container would have neither an identity + # nor a route. Plus the ordering that makes the mounts exist first. + name = "the four readers' granting units are ordered after the mounts and rendered on the host"; + ok = + let + units = [ + "swarm-bao-matrix-token-policy" + "swarm-bao-queue-agent-policy" + "swarm-bao-grafana-oidc-policy" + "swarm-bao-otel-oidc-policy" + ]; + in + lib.all ( + unit: + lib.elem "swarm-bao-controller-policy.service" baoGrantHere.systemd.services.${unit}.after + && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? ${unit}) + ) units; + } + { + # A store host that has not placed a bootstrap token can write no grant at + # all, so none of the four units may exist — the same claim + # `baoGrantNoStore` makes for the controller's, one file over. Without + # this arm `lib.mkIf haveBootstrapToken` could be dropped from the shared + # builder and every other case here would still pass. + name = "without a bootstrap token none of the four readers' granting units render"; + ok = + let + s = baoGrantNoStore.systemd.services; + in + !(s ? swarm-bao-matrix-token-policy) + && !(s ? swarm-bao-queue-agent-policy) + && !(s ? swarm-bao-grafana-oidc-policy) + && !(s ? swarm-bao-otel-oidc-policy); + } { # The policy authorising this route lives in another file, and nothing # else relates the grants to the paths the code actually writes. diff --git a/nix/module-eval/bao-matrix-reader.nix b/nix/module-eval/bao-matrix-reader.nix index 00f2256b..19a0da5f 100644 --- a/nix/module-eval/bao-matrix-reader.nix +++ b/nix/module-eval/bao-matrix-reader.nix @@ -32,10 +32,20 @@ let # nothing here mints a leaf and the operator names one placed by hand. The # deployment this pairing exists to serve, and the one that was previously # inexpressible — the gate asked whether the store was a neighbour. + # + # 🩸 One leaf PER READER, and this fixture is where that cost is visible: the + # two units below identify themselves to the store separately, so an operator + # placing leaves by hand places one for each rather than one for both. Naming + # only `clientCertFile` here would leave neither unit rendered — which is what + # the arms below would then be asserting about. baoRemoteReader = hive { deploy.matrix.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.matrixTokenClientCertFile = "/etc/pki/bao-matrix-token.pem"; + deploy.bao.matrixTokenClientKeyFile = "/etc/pki/bao-matrix-token-key.pem"; + deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem"; + deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem"; }; # A homeserver on a hive with NO store identity at all — neither a local @@ -43,6 +53,38 @@ let # needs it, and defining it here rather than importing keeps each group's # fixture set its own, as ./lib.nix asks. matrixNoBaoIdentity = hive { deploy.matrix.enable = true; }; + + # 🩸 The hand-configured remote reader that named seven of the eight options. + # `baoRemoteReader` above is the same deployment done right; this one holds + # the hive's own leaf, so it demonstrably reads the store, and is missing both + # per-principal pairs. Before the four-way split this host rendered both units + # off `clientCertFile` alone, so what it has now is a silent regression rather + # than any shape an operator chose — which is what the refusal arms below are + # about. Kept as one fixture rather than two because both refusals fire on it + # and each arm names which. + baoRemoteReaderMissingLeaves = hive { + deploy.matrix.enable = true; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # The same omission on a hive that does NOT run a homeserver. Separates the + # matrix refusal's `deploy.matrix.enable` clause from the queue refusal, which + # has no toggle to check — an arm below reads exactly one refusal off it. + remoteReaderNoMatrixMissingLeaves = hive { + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # Did a module refuse this host, and over which option. An assertion is a + # config VALUE until something forces it — `.config` never throws — so a + # fixture in a state the module refuses stays evaluable and the refusal reads + # back as data. Matched on the option name the message names rather than on + # its prose, because the option name is the part an operator has to act on + # and a message that stopped naming it would be the actual defect. Same shape + # as ./grafana.nix's `grafanaRefusedFor`, which reads `swarm-grafana.nix`'s + # own refusal. + refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; cases = [ { # A login failure is the store being unreachable, sealed, or not yet @@ -363,6 +405,63 @@ let in !(g ? admin_execute) || g.admin_execute == [ ]; } + { + # 🩸 The arm the whole refusal exists for. Both readers are gated on their + # own leaf, so the way this regresses is the build going green and three + # units rendering where four should — which no presence check on a + # rendered unit can see, because the unit that is missing is the evidence. + # Read as a refusal naming each option, so an operator acts on the message + # without opening the nix. + name = "a remote reader missing the per-principal leaves is refused, naming both options"; + ok = + refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientCertFile" + && refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientKeyFile" + && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile" + && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile"; + } + { + # The matrix refusal's own gate, which the queue refusal does not have. + # Without this arm the two are indistinguishable on the fixture above. + name = "the queue refusal needs no homeserver, and the matrix refusal stays quiet without one"; + ok = + refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile" + && !(refusedOver remoteReaderNoMatrixMissingLeaves "matrixTokenClientCertFile"); + } + { + # ⚠️ The arm that keeps the refusal from being worse than the silence it + # replaced. A hive with NO store identity is the supported no-store + # deployment and also the state an operator passes through bringing a hive + # up — neither may fail to evaluate. Asserted as "no refusal names any of + # the four options", not as "this one fixture is fine", because the way + # this breaks is a gate widened to the principal's leaf alone. + name = "a hive with no store identity at all is refused over none of the per-principal leaves"; + ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [ + "matrixTokenClientCertFile" + "matrixTokenClientKeyFile" + "queueAgentClientCertFile" + "queueAgentClientKeyFile" + ]; + } + { + # The other half of the same guard, and the one an operator meets far more + # often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults + # all eight, so there is nothing to name and nothing to refuse. Paired with + # the fully-named remote reader, which is the same deployment done by hand. + name = "neither a store host nor a correctly-named remote reader is refused"; + ok = + lib.all + ( + m: + lib.all (option: !(refusedOver m option)) [ + "matrixTokenClientCertFile" + "queueAgentClientCertFile" + ] + ) + [ + baoWithMatrix + baoRemoteReader + ]; + } ]; in runGroup "bao-matrix-reader" cases diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index ae825800..fe3cd97e 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -41,6 +41,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; + deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # The same UI with the IdP on ANOTHER host and a store leaf placed by hand. @@ -54,6 +56,8 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; + deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # A Grafana host holding no store identity. This used to be the shape the @@ -148,8 +152,8 @@ let # mean the two assertions had been collapsed into one conjunction. name = "a grafana host with no store identity is refused, naming the options to set"; ok = - grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientCertFile" - && grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientKeyFile" + grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientCertFile" + && grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientKeyFile" && !(grafanaRefusedFor grafanaNoIdentity "swarm.authelia.url"); } { @@ -161,9 +165,9 @@ let name = "neither grafana refusal fires on a correctly configured host, co-located or not"; ok = !(grafanaRefusedFor grafanaWithAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.clientCertFile") + && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.grafanaOidcClientCertFile") && !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.clientCertFile"); + && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.grafanaOidcClientCertFile"); } { # Same 403-not-a-miss reason as the matrix and queue arms below: the diff --git a/nix/module-eval/name-guards.nix b/nix/module-eval/name-guards.nix index 8716f980..e035b0f0 100644 --- a/nix/module-eval/name-guards.nix +++ b/nix/module-eval/name-guards.nix @@ -63,6 +63,35 @@ let swarm.hives.mintctl.domain = "m.t.local"; }; + # The two OIDC-secret readers' subjects, fixed strings like the three above. + hiveNamedAfterGrafanaOidcSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.grafanaOidcCommonName = "gfctl"; + swarm.hives.gfctl.domain = "g.t.local"; + }; + + hiveNamedAfterOtelOidcSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.otelOidcCommonName = "otctl"; + swarm.hives.otctl.domain = "o.t.local"; + }; + + # 🩸 A different shape from every fixture above: the matrix-token and + # queue-credential roles are written PER HIVE, so the subject a hive must not + # be is `-` rather than the prefix itself. Reserving + # only the prefix would leave the composed spelling free, and a hive taking it + # would present a leaf the other hive's role accepts — which is a hive reading + # another hive's queue credential, the exact widening the split exists to + # avoid. + # + # Two hives here, not one: the collision is with the OTHER hive's role. + hiveNamedAfterPerHiveReaderSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.queueAgentCommonNamePrefix = "qr"; + swarm.hives.other.domain = "o.t.local"; + swarm.hives.qr-other.domain = "q.t.local"; + }; + hiveNameWithComposedWord = hive { deploy.swarm-otel.enable = false; swarm.hives."h1-agent".domain = "a.t.local"; @@ -111,6 +140,36 @@ let a: !a.assertion && lib.hasInfix "'mintctl'" a.message ) hiveNamedAfterMatrixCtlSubject.assertions; } + { + # The fourth and fifth, for the reason the case above gives: `certAuthCns` + # is where a role added beside the others registers itself, and nothing + # but a case per element notices when one forgets. These two are the + # subjects of the readers that fetch Grafana's and the collector's OIDC + # client secrets. + name = "a hive named after either OIDC-secret reader's subject is refused too"; + ok = + equalityGuardFired hiveNamedAfterGrafanaOidcSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'gfctl'" a.message + ) hiveNamedAfterGrafanaOidcSubject.assertions + && equalityGuardFired hiveNamedAfterOtelOidcSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'otctl'" a.message + ) hiveNamedAfterOtelOidcSubject.assertions; + } + { + # 🩸 The per-hive half, and the one a prefix-only reservation would miss: + # the role is `-`, so the reserved string has to be composed + # against every declared hive. Here hive `qr-other` collides with the role + # written for hive `other` — a leaf that reads a credential belonging to a + # hive that is not it. + name = "a hive named after another hive's per-hive reader subject is refused"; + ok = + equalityGuardFired hiveNamedAfterPerHiveReaderSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'qr-other'" a.message + ) hiveNamedAfterPerHiveReaderSubject.assertions; + } { # Without this the case above proves nothing: an arm that fires for every # roster is not a guard, and `hives` is non-empty in both fixtures. diff --git a/nix/module-eval/swarm-otel-core.nix b/nix/module-eval/swarm-otel-core.nix index 7428919c..0bbfa10e 100644 --- a/nix/module-eval/swarm-otel-core.nix +++ b/nix/module-eval/swarm-otel-core.nix @@ -32,6 +32,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; deploy.victoriametrics.enable = false; deploy.victorialogs.enable = false; }; @@ -44,6 +46,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; # The same collector with the IdP on ANOTHER host and a store leaf placed by @@ -54,6 +58,8 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; cases = [ { diff --git a/nix/module-eval/swarm-otel-identity.nix b/nix/module-eval/swarm-otel-identity.nix index 67c89357..99dcd677 100644 --- a/nix/module-eval/swarm-otel-identity.nix +++ b/nix/module-eval/swarm-otel-identity.nix @@ -34,6 +34,40 @@ let deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; }; + # 🩸 The shape the degrade above must not swallow: a collector on a host that + # demonstrably READS the store — it holds the hive's own leaf — and is missing + # only the collector's own pair. Before the four-way split this host rendered + # the reading unit off `clientCertFile` alone, so the silence it gets now is a + # regression rather than the supported degrade `otelNoIdentity` stands for. The + # two fixtures differ in exactly that one pair, which is what lets the arms + # below separate a refusal from a degrade. + otelReaderMissingLeaf = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + }; + + # The same deployment named in full, which must stay buildable. Pairs with the + # fixture above so the refusal is pinned to the omission and not to reading a + # remote store at all. + otelReaderNamedInFull = hive { + deploy.swarm-otel.enable = true; + swarm.authelia.url = "https://auth.example.invalid"; + deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; + deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; + }; + + # Did the module refuse this host, and over which option. An assertion is a + # config VALUE until something forces it — `.config` never throws — so a + # fixture in a refused state stays evaluable and the refusal reads back as + # data. Matched on the option name the message names rather than on its prose: + # the option name is the part an operator has to act on, and a message that + # stopped naming it would be the actual defect. + refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; + # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. @@ -162,6 +196,38 @@ let && !(builtins.elem "otlphttp/victoriametrics" used) && lib.all (e: s.exporters ? ${e}) used; } + { + # 🩸 The refusal, and the only way this module's omission is visible at + # all: the reading unit is gated on its own leaf, so the regression is a + # green build with the unit absent — and the missing unit is the evidence. + # Read as a refusal naming both options, so an operator acts on the + # message without opening the nix. + name = "a store reader missing the collector's own leaf is refused, naming both options"; + ok = + refusedOver otelReaderMissingLeaf "otelOidcClientCertFile" + && refusedOver otelReaderMissingLeaf "otelOidcClientKeyFile"; + } + { + # ⚠️ The arm that keeps the refusal from eating the degrade beside it. A + # collector on a host with NO store identity still receives every hive's + # telemetry and is a supported deployment — `otelNoIdentity` is that + # fixture, and it differs from the refused one only in the hive's own + # leaf. Widening the gate to the collector's leaf alone would reject it, + # which is what this pins. + name = "a collector with no store identity at all is not refused over the collector's leaf"; + ok = + !(refusedOver otelNoIdentity "otelOidcClientCertFile") + && !(refusedOver otelNoIdentity "otelOidcClientKeyFile"); + } + { + # The other two shapes that must stay buildable: the same remote reader + # named in full, and a host that mints its own PKI and therefore has all + # eight as defaults from ./host-modules/glue-bao-tls.nix. + name = "neither a fully-named remote reader nor a store host is refused"; + ok = + !(refusedOver otelReaderNamedInFull "otelOidcClientCertFile") + && !(refusedOver otelRemoteAuthelia "otelOidcClientCertFile"); + } ]; in runGroup "swarm-otel-identity" cases