diff --git a/docs/getting-started/setup.md b/docs/getting-started/setup.md index af3290d3..2225529e 100644 --- a/docs/getting-started/setup.md +++ b/docs/getting-started/setup.md @@ -170,16 +170,19 @@ What else you need depends on restart, with the keys `init` printed. The store serves TLS, and on a hive that deploys it you need do nothing: a -first-boot unit mints a CA of the store's own plus the two leaves it signs — -the store's server certificate and this host's client certificate — and points -`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at the store's -half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's. +first-boot unit mints a CA of the store's own plus the leaves it signs — the +store's server certificate, this host's client certificate, and one per service +principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and +`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and +`.serverCaFile` at the reader's, and each principal's own pair at its own leaf. Those are `mkDefault`s, so naming your own paths wins. Do that when your certificates come from a real internal CA; the store has no opinion about which. A hive that does **not** deploy the store names the reader's three -itself: the operator issues that leaf out of band, and it's the one credential -the store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is +itself, plus a pair for every principal it runs — see +[per-principal identities](../swarm/secrets.md#per-principal-identities) for the +list. The operator issues those leaves out of band; they're the credentials the +store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is **mTLS between services and the store**, a separate trust domain, because a store that took its identity from an authority it itself distributes could never come up before that authority. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 53bce3ad..681d277c 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -88,7 +88,7 @@ where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — only its own pushes to the stores go out unauthenticated and get refused there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without -`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice +`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for their own optional readers, rather than the hard assertion `swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by @@ -249,14 +249,14 @@ straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store -exists so a host holds **one** out-of-band secret, its client certificate, and -reads everything else with it. +exists so the only secrets a host holds out of band are **certificates**, and +everything else is read with them. Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: running Grafana requires `swarm.authelia.url` (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the -only way in), and it requires this host's `deploy.bao.clientCertFile` / -`clientKeyFile`. Each refusal names the option to set. Both used to be silent: +only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile` +/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is @@ -271,7 +271,8 @@ runs the collector, and registration lives separately in `glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so `swarm-otel.nix` doesn't assert -`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it +`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana +does — it doesn't render the reading unit without them, the shape `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for their own optional reads. @@ -290,10 +291,18 @@ A service's secret is one value for the whole swarm rather than one per hive, so it lives under the `services` prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a `deploy.*` fact, per-host by definition — so -there is nothing to scope the grant to. Every hive can therefore read every swarm -service's client secret; that's the cost of a shape where the service's host has -no store identity of its own to present, and it's stated in -`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves +there is nothing to scope the grant to. A hive's **own** leaf can therefore read +every swarm service's client secret, and that's stated in +`swarm-secret-client`'s `policy` module beside the grant itself. + +⚠️ **The readers no longer present it.** Four units used to log in with +`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a +principal by the subject of the certificate it presents — so four readers behind +one leaf were one principal holding the union of their needs. Each now holds a +leaf of its own, and a policy naming only the path that unit reads. See +[per-principal identities](#per-principal-identities) below. + +Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in. @@ -303,14 +312,43 @@ credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the hive CA and the swarm CA both, and has nothing to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix` -mints a CA that signs exactly two things, the store's server certificate and a -reader's client certificate, and distributes nothing. A deployment with a real +mints a CA that signs a fixed, short list of leaves — the store's server +certificate, the hive's own reader certificate, and one per service principal — +and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in `deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. A hive that reads from a store on **another** machine names the reader's half — -`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand. -It's the one credential that can't come out of the store, being what opens it; -everything else a hive needs does. +`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand, +plus one leaf per principal it runs (the options are listed below). Those are the +credentials that can't come out of the store, being what opens it; everything +else a hive needs does. + +### Per-principal identities + +Bao matches a cert-auth role on the certificate's subject, so a certificate is an +identity and sharing one merges the identities. These four units read one path +each and each holds a leaf, a role and a policy of its own: + +| unit | option pair under `deploy.bao.` | reads | +| ------------------------ | -------------------------------------------------------- | -------------------------------------------------- | +| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives//matrix/appservice-token` | +| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives//queue/agent` | +| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services//oidc/client` | +| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | + +The first two are written **per hive**, because the path they read carries a hive +name and every hive runs its own reader. Their subjects are +`-` and +`-`; `swarm.nix` reserves both +composed spellings as hive names, so a hive can't be named into another hive's +role. The other two read a path that names a swarm service rather than a hive, so +one role each is enough and their subjects are the flat +`deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`. + +On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults +all eight options, and there is nothing to do. Elsewhere each leaf is issued from +that CA out of band and named here — one file per principal rather than one file +shared by four, which is the whole of what this buys. ## How a reader reaches the store diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix index 819569bf..1afaed01 100644 --- a/nix/host-modules/glue-bao-tls.nix +++ b/nix/host-modules/glue-bao-tls.nix @@ -84,6 +84,25 @@ in clientCertFile = lib.mkDefault "${pkiDir}/client.pem"; clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem"; serverCaFile = lib.mkDefault "${pkiDir}/ca.pem"; + + # 🩸 Four readers that used to present `client.pem` above, each now + # pointed at a leaf of its own. The leaf is what bao sees, so this pairing + # is the whole of what turns "one principal with the union of four grants" + # into four principals with one grant each — see ./swarm-bao.nix's + # `perHiveReaders` block for the grants themselves. + # + # Defaulted here rather than in four `glue--bao-identity.nix` + # files: those exist where the consumer is a CONTAINER with a + # `deploy..*` namespace of its own to point at. These four are + # host units reading the store, which is the pairing this file already is. + matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem"; + matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem"; + queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem"; + queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem"; + grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem"; + grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem"; + otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem"; + otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem"; }; # Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates @@ -160,6 +179,35 @@ in # container an identity instead of lending it the hive's. [ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \ ${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth + + # 🩸 The four readers that used to present `client.pem` above. Each + # carries its own subject, which is the entire mechanism: bao matches a + # cert-auth role on the CN, so four units sharing one leaf were one + # principal holding the union of four grants, and the union included + # every agent credential in the swarm. + # + # ⚠️ The first two carry THIS hive's name in the subject, unlike the + # three service leaves above. Their grants name one hive's path, because + # a matrix token and a queue credential live under `swarm/hives//` + # and every hive runs a reader for its own; ./swarm-bao.nix writes one + # role per hive in the swarm directory to match. The two OIDC readers + # need no such segment — a client is registered once per swarm. + # + # Minted whether or not the consumer runs here, for the reason the three + # leaves above give: on a hive that does not run the store this is the + # file an operator copies, and a leaf that only appears where its + # consumer does is one nobody can copy from anywhere. + [ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \ + ${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth + + [ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \ + ${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth + + [ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \ + ${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth + + [ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \ + ${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth ''; }; }; diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix index 077c7b86..ca870766 100644 --- a/nix/host-modules/glue-matrix-bao-token.nix +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -38,7 +38,16 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: read on + # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when + # this unit reads one appservice token and nothing else. Its own leaf carries + # `-` and its role grants the single path + # below. + haveClientIdentity = + baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null; # Where the token lives in the store. A path, not a convention to guess at: # whoever writes it and whoever reads it must agree, and the agreement @@ -115,8 +124,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; + BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/glue-queue-agent-credential.nix b/nix/host-modules/glue-queue-agent-credential.nix index 9e57b512..1950f959 100644 --- a/nix/host-modules/glue-queue-agent-credential.nix +++ b/nix/host-modules/glue-queue-agent-credential.nix @@ -40,7 +40,16 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: read on + # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when + # this unit reads one queue credential and nothing else. Its own leaf carries + # `-` and its role grants the single path + # below — still this hive's own, so the narrowing costs no reach. + haveClientIdentity = + baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null; credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir; secretFile = "${credentialDir}/secret"; @@ -141,8 +150,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; + BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index e577a988..8fd7474c 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -340,6 +340,156 @@ let } ''; + # ── the four principals that used to share the hive's own leaf ───────────── + # + # 🩸 Each of the four reads exactly ONE path in the store, and until this + # split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf, + # whose policy (`swarm-secret-client`'s `policy::render`) grants read on + # `swarm/agents/*`, `swarm/hives//*` AND `swarm/services/*`. Four + # principals presenting one certificate are ONE principal to bao, so the union + # of what the four need was the only grant expressible: the unit that fetches + # Grafana's OIDC client secret could fetch every agent credential in the + # swarm, and the unit that fetches this hive's matrix token could fetch + # Grafana's. Least privilege was not misconfigured, it was unrepresentable. + # + # Each now gets its own leaf, its own cert-auth role and a policy holding the + # one `secret/data/…` path that principal's script actually names — spelled to + # the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above + # gives for doing the same. + # + # ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the + # shape of the paths rather than a preference. A matrix appservice token and a + # queue credential live under `swarm/hives//`, and every hive runs a + # reader for its own; an OIDC client secret lives under + # `swarm/services//` and a swarm registers each of those exactly + # once. A single role for a per-hive reader would have to name `hives/*` to + # serve every hive — which would let one hive read another's matrix token, a + # reach no hive has today and one this change exists to remove rather than + # create. + readStanza = path: '' + path "${path}" { + capabilities = ["read"] + } + ''; + + # One reader per hive in the swarm directory. Written from the directory + # rather than from this host's own name because the policy is written where + # the STORE is and the reader runs where its hive is — the same split + # `matrixCtlHiveName` above exists to paper over, answered here by naming + # every hive instead of asking the operator which one. + perHiveReaders = + { rolePrefix, cnPrefix, ... }@spec: + lib.mapAttrsToList (hiveName: _: { + name = "${rolePrefix}-${hiveName}"; + cn = "${cnPrefix}-${hiveName}"; + policyText = readStanza (spec.path hiveName); + }) hyperhiveCfg.swarm.hives; + + matrixTokenReaders = perHiveReaders { + rolePrefix = "swarm-matrix-token"; + cnPrefix = baoDeploy.matrixTokenCommonNamePrefix; + path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token"; + }; + + queueAgentReaders = perHiveReaders { + rolePrefix = "swarm-queue-agent"; + cnPrefix = baoDeploy.queueAgentCommonNamePrefix; + path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent"; + }; + + # Singletons, so a list of one rather than a second shape: the granting unit + # below takes a list either way and the four units stay identical apart from + # the objects they write. + grafanaOidcReaders = [ + { + name = "swarm-grafana-oidc"; + cn = baoDeploy.grafanaOidcCommonName; + policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"; + } + ]; + + otelOidcReaders = [ + { + name = "swarm-otel-oidc"; + cn = baoDeploy.otelOidcCommonName; + policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.otel.clientId}/oidc/client"; + } + ]; + + # The role name IS the policy name, as for the three service principals + # above: the role attaches the policy by spelling it identically, and one + # string for both objects removes the way they drift apart. + readerPolicyWrite = obj: '' + printf '%s' ${lib.escapeShellArg obj.policyText} | + bao policy write ${lib.escapeShellArg obj.name} - + ''; + + # ⚠️ Names are `swarm-[-]`, outside the `hive-*` namespace the + # controller may rewrite, for the reason the three service principals above + # state. A per-hive CN cannot collide with a hive's own leaf either — a hive's + # CN is its bare name and ../reserved-hive-fragments.nix forbids the substring + # `swarm` in one — and ./swarm.nix reserves the concrete per-hive spellings + # anyway, because the prefixes are operator-settable and an operator may spell + # one without that substring. + readerRoleWrite = obj: '' + bao write auth/cert/certs/${lib.escapeShellArg obj.name} \ + certificate=@${tlsDir}/client-ca.pem \ + allowed_common_names=${lib.escapeShellArg obj.cn} \ + token_policies=${lib.escapeShellArg obj.name} \ + display_name=${lib.escapeShellArg obj.cn} + ''; + + # One unit per PRINCIPAL, not one unit for the four: these names are + # operator-facing strings, the same reason `swarm-bao-secret-publisher-policy` + # above gives for not being more script inside the controller's unit. A hive + # whose matrix token never arrives is diagnosed by the state of the unit named + # after it. + # + # `after` and not `requires`, for the reason the publisher's unit states: the + # controller's unit creates the KV and cert-auth mounts this one writes into, + # but a failed oneshot still counts as finished, so ordering plus this unit's + # own retry is what converges. + # + # The role write is inside the client-CA branch and the policy write is not, + # exactly as the three above: with no CA there is no trust anchor for a login + # role, but the policy it would attach is still worth asserting. + readerPolicyUnit = + description: objects: + lib.mkIf haveBootstrapToken { + inherit description; + after = [ + "container@${cfg.machine}.service" + "swarm-bao-controller-policy.service" + ]; + wantedBy = [ "multi-user.target" ]; + path = [ + baoCli + pkgs.coreutils + ]; + unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; + # Same unseal wait as its siblings above, for the reason stated there: + # under `seal = "shamir"` a human unseals by hand. + startLimitBurst = 2880; + startLimitIntervalSec = 90000; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + Restart = "on-failure"; + RestartSec = 30; + }; + script = '' + set -euo pipefail + + BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})" + export BAO_TOKEN + + '' + + lib.concatMapStrings readerPolicyWrite objects + + lib.optionalString (baoDeploy.clientCaFile != null) ( + "\n" + lib.concatMapStrings readerRoleWrite objects + ); + }; + # Every listener serves the same identity: they differ in which address # they answer on, not in who they are. Client verification is separate and # optional — a store with no `clientCaFile` still serves TLS, it just does @@ -846,6 +996,185 @@ in ''; }; + matrixTokenCommonNamePrefix = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-matrix-token"; + example = "swarm-bao-matrix-token.svc"; + description = '' + Prefix of the subject the store's per-hive matrix-token roles accept. + The role for hive `h` accepts `-h` and nothing else, and grants + read on exactly `swarm/hives/h/matrix/appservice-token`. + + A **prefix** rather than one subject because every hive runs its own + reader and the path it reads carries that hive's name. One subject for + all of them would need a `hives/*` grant, which would let one hive read + another hive's appservice token — a reach no hive has today. + + ⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its + fixed-subject siblings are: cert auth trusts the CA and + `allowed_common_names` is the whole narrowing, so a hive whose own leaf + carried one of these spellings would receive that grant. + ''; + }; + + queueAgentCommonNamePrefix = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-queue-agent"; + example = "swarm-bao-queue-agent.svc"; + description = '' + Prefix of the subject the store's per-hive queue-credential roles + accept. The role for hive `h` accepts `-h` and grants read on + exactly `swarm/hives/h/queue/agent` — the client credential that hive's + agent containers authenticate to the swarm queue with. + + Per hive for the same reason + {option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is, + and more sharply: every hive in the swarm runs this reader, so a single + subject would have to be granted `hives/*`. + ''; + }; + + grafanaOidcCommonName = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-grafana-oidc"; + example = "swarm-bao-grafana-oidc.svc"; + description = '' + Subject the store's `swarm-grafana-oidc` cert-auth role accepts — the + identity the unit that fetches Grafana's OIDC client secret presents. + + Its grant is one path, + `swarm/services//oidc/client`, and read only. Not + per hive, unlike the two prefixes above: an OIDC client is registered + once per swarm, so the path names the service and never a hive. + + ⚠️ Same collision as its siblings, and the same answer: ./swarm.nix + feeds this value into the guard on + {option}`services.hyperhive.swarm.hives`. + ''; + }; + + otelOidcCommonName = lib.mkOption { + type = lib.types.str; + default = "swarm-bao-otel-oidc"; + example = "swarm-bao-otel-oidc.svc"; + description = '' + Subject the store's `swarm-otel-oidc` cert-auth role accepts — the + identity the unit that fetches the collector's OIDC client secret + presents. Its grant is one path, + `swarm/services//oidc/client`, and read only. + + A **fourth** identity rather than reuse of + {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two + read different services' client secrets, and a collector is not + entitled to Grafana's. + ''; + }; + + matrixTokenClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/matrix-token.pem"; + description = '' + Certificate the unit that fetches this hive's matrix appservice token + presents to the store. Its subject must be + `-`; cert auth matches on + the CN and the role accepts nothing else. + + ⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`. + Pointing this at the hive's own leaf would evaluate, deploy and log in — + and hand a unit that reads one token a credential that reads every + secret in the store. That is what this option exists to end. + + No default here: ./glue-bao-tls.nix points it at the leaf it mints, + wherever this host mints its own PKI. A deployment that reads a store on + another host names it by hand, alongside the leaf itself — one file per + principal placed out of band instead of one shared by four. + ''; + }; + + matrixTokenClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/matrix-token-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both + or neither — a certificate with no key authenticates nothing. + ''; + }; + + queueAgentClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/queue-agent.pem"; + description = '' + Certificate the unit that fetches this hive's queue client credential + presents to the store. Its subject must be + `-`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + queueAgentClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/queue-agent-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`. + ''; + }; + + grafanaOidcClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/grafana-oidc.pem"; + description = '' + Certificate the unit that fetches Grafana's OIDC client secret presents + to the store. Its subject must be + {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + grafanaOidcClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/grafana-oidc-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile`. + ''; + }; + + otelOidcClientCertFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/otel-oidc.pem"; + description = '' + Certificate the unit that fetches the collector's OIDC client secret + presents to the store. Its subject must be + {option}`services.hyperhive.deploy.bao.otelOidcCommonName`. + + ⚠️ Not the hive's own leaf, for the reason + {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. + ''; + }; + + otelOidcClientKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/swarm-bao-pki/otel-oidc-key.pem"; + description = '' + Private key for + {option}`services.hyperhive.deploy.bao.otelOidcClientCertFile`. + ''; + }; + serverCaFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; @@ -1593,6 +1922,19 @@ in ''; }; + # 🩸 The four principals that used to log in as the hive. See the block + # around `perHiveReaders` for what each grant is and why the first two are + # per hive; the four units are identical apart from the objects they + # write, so the shape lives in `readerPolicyUnit` and the difference is + # visible here in one line each. + systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders; + + systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders; + + systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders; + + systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders; + # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 006e3abf..85e5a2d1 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -119,7 +119,14 @@ let # other secret is read with it. A Grafana host without it has not been given # its identity yet, which is a thing to say out loud rather than to route # around by reaching into authelia's tree whenever it happens to be local. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 Grafana's OWN leaf, not `clientCertFile` — the hive's, which four units + # used to share. Bao matches a cert-auth role on the CN, so one leaf for four + # readers was ONE principal holding the union of four grants: this unit could + # read every agent credential in the swarm and every other service's OIDC + # client secret, when what it needs is the one path `storeSecretPath` names. + haveClientIdentity = + baoDeploy.grafanaOidcClientCertFile != null && baoDeploy.grafanaOidcClientKeyFile != null; autheliaUrl = toString hyperhiveCfg.swarm.authelia.url; @@ -451,15 +458,20 @@ in services.hyperhive.deploy.grafana.enable requires this host to hold a swarm-secret-store client identity: set both - services.hyperhive.deploy.bao.clientCertFile - services.hyperhive.deploy.bao.clientKeyFile + services.hyperhive.deploy.bao.grafanaOidcClientCertFile + services.hyperhive.deploy.bao.grafanaOidcClientKeyFile Grafana's OIDC client secret is minted by authelia and read out of the store, on every host that runs Grafana — including the host that runs authelia. That is one delivery route rather than two, and it is - what the store is for: this certificate is the single credential - placed out of band, and every other secret comes from the store with - it. + what the store is for: a certificate is the credential placed out of + band, and every other secret comes from the store with it. + + ⚠️ Grafana's OWN leaf, not deploy.bao.clientCertFile. That one is the + hive's, and its grant reads every secret in the store; this role + reads the one path Grafana's client secret lives at. Pointing this + option at the hive's leaf would evaluate, deploy and log in — and + undo the split. On a hive that runs the store, glue-bao-tls.nix supplies both as defaults and there is nothing to do. Elsewhere the leaf is issued @@ -610,8 +622,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 868df8b7..e0db2ce7 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -248,7 +248,14 @@ let # their own optional readers, because a collector with no client identity is # `haveCollectorSecret = false` above, and that is already a supported, # merely degraded shape rather than a service with no way in at all. - haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # + # 🩸 The collector's OWN leaf, not `clientCertFile` — the hive's, which four + # units used to share. Bao matches a cert-auth role on the CN, so one leaf for + # four readers was ONE principal holding the union of four grants: this unit + # could read every agent credential in the swarm and Grafana's OIDC client + # secret, when what it needs is the one path `storeSecretPath` names. + haveClientIdentity = + baoDeploy.otelOidcClientCertFile != null && baoDeploy.otelOidcClientKeyFile != null; # Where the publisher on authelia's host leaves this client's secret — # composed from the same swarm-wide `clientId` the registration in @@ -775,8 +782,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.clientCertFile; - BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + BAO_CLIENT_CERT = baoDeploy.otelOidcClientCertFile; + BAO_CLIENT_KEY = baoDeploy.otelOidcClientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index d0e36dcd..213aa382 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -47,7 +47,20 @@ let deployCfg.bao.controllerCommonName deployCfg.bao.secretPublisherCommonName deployCfg.bao.matrixCtlCommonName - ]; + deployCfg.bao.grafanaOidcCommonName + deployCfg.bao.otelOidcCommonName + ] + # The two per-hive readers' subjects, spelled out per hive rather than as the + # prefix. The prefix alone would reserve the wrong string: the role for hive + # `h` accepts `-h`, so the name a hive must not BE is that composed + # spelling, and a hive named it would present a leaf that role accepts. + # + # Both spellings are reserved for every declared hive, including the one the + # CN belongs to — a hive cannot be named after its own reader either. + ++ lib.concatMap (hive: [ + "${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}" + "${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}" + ]) hiveNames; # Public hostnames of the swarm's own services, in declaration order. # `serviceDomains` below is this set sorted + deduplicated. diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 2068620c..c03e0fcf 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -45,6 +45,18 @@ let deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; deploy.bao.clientCaFile = lib.mkForce null; }; + # The store plus every one of the four readers that used to log in as the + # hive. One fixture rather than four: the claim they are four *separate* + # principals is only testable where all four render at once — that is the + # deployment in which two of them sharing a leaf would be invisible. + baoGrantWithConsumers = hive { + deploy.bao.enable = true; + deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; + deploy.matrix.enable = true; + deploy.grafana.enable = true; + deploy.swarm-otel.enable = true; + }; + cases = [ { # Reads the rendered unit on the HOST, which is where the write happens: @@ -196,6 +208,250 @@ let ) && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-matrix-ctl-policy); } + + # ── the four readers that used to share the hive's own leaf ────────────── + # + # 🩸 Until this split all four presented `deploy.bao.clientCertFile`, whose + # policy grants read on `swarm/agents/*`, `swarm/hives//*` AND + # `swarm/services/*`. Four principals behind one certificate are one + # principal to bao, so the only expressible grant was the union: the unit + # fetching Grafana's OIDC secret could fetch every agent credential in the + # swarm. + # + # Every one of these cases carries the same three negative arms, and they + # are the deliverable rather than decoration — a positive arm alone passes + # just as well when the other two stanzas are still there beside it. The + # arms pin what each principal must NOT reach, so a later widening fails + # here instead of being noticed in a store. + { + name = "the matrix-token reader's grant is one hive's appservice token and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-matrix-token-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/hives/h1/matrix/appservice-token\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + # The three stanzas the hive's own leaf carried, none of which this + # principal needs: every agent's credential, every service's OIDC + # client, and the rest of its own hive's tree — including the queue + # credential its sibling reader fetches. + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/services" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/queue" s) + # A `hives/*` wildcard would serve every hive from one role and let any + # hive read any other's token — the reach this split exists to remove, + # not to create. + && !(lib.hasInfix "secret/data/swarm/hives/*" s) + # Nothing may rewrite the policy constraining it, for the reason the + # controller's own `hive-*` narrowing above gives. + && !(lib.hasInfix "sys/policies/acl" s); + } + { + name = "the queue-credential reader's grant is one hive's queue credential and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/services" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) + && !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s) + && !(lib.hasInfix "secret/data/swarm/hives/*" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # ⚠️ The client id is the path segment, so the negative arm that matters + # for this one is the OTHER service's: `services/*` would have granted + # both, and the two are separate principals precisely because a + # dashboard is not entitled to a collector's credential. + name = "the Grafana OIDC reader's grant is Grafana's own client secret and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/services/swarm-grafana/oidc/client\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/hives" s) + && !(lib.hasInfix "secret/data/swarm/services/*" s) + && !(lib.hasInfix "swarm-collector" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # The mirror of the case above, and the arm naming `swarm-grafana` is why + # these are two principals rather than one `services/*` grant shared. + name = "the collector OIDC reader's grant is the collector's own client secret and nothing else"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-otel-oidc-policy.script; + in + lib.hasInfix "path \"secret/data/swarm/services/swarm-collector/oidc/client\" {" s + && lib.hasInfix "capabilities = [\"read\"]" s + && !(lib.hasInfix "secret/data/swarm/agents" s) + && !(lib.hasInfix "secret/data/swarm/hives" s) + && !(lib.hasInfix "secret/data/swarm/services/*" s) + && !(lib.hasInfix "swarm-grafana" s) + && !(lib.hasInfix "sys/policies/acl" s); + } + { + # 🩸 The half that makes the policies above bind: a policy grants only + # through a token that carries it, and a token is minted by a cert-auth + # role matching a CN. Four distinct subjects is the whole mechanism — one + # subject for four readers is one principal however the policies read. + # + # The per-hive subjects carry the hive name because their paths do; the + # two service subjects do not, because an OIDC client is registered once + # per swarm. Pinned so neither shape is tidied into the other. + name = "each of the four readers logs in under a subject of its own"; + ok = + let + subjectOf = + unit: role: cn: + let + s = baoGrantHere.systemd.services.${unit}.script; + in + lib.hasInfix "auth/cert/certs/${role}" s + && lib.hasInfix "allowed_common_names=${cn}" s + && lib.hasInfix "token_policies=${role}" s + # Outside the `hive-*` namespace the controller may rewrite, for + # the reason the three service principals above state. + && !(lib.hasInfix "auth/cert/certs/hive-" s); + in + subjectOf "swarm-bao-matrix-token-policy" "swarm-matrix-token-h1" "swarm-bao-matrix-token-h1" + && subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1" + && subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc" + && subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"; + } + { + # 🩸 The consuming side, and the arm that would catch the regression that + # costs the most: a unit repointed back at `deploy.bao.clientCertFile` + # evaluates, deploys and logs in — and silently restores the union grant, + # because bao would again see one principal. Nothing about the policies + # above would look wrong. + # + # Each pair is asserted whole: a certificate with no key authenticates + # nothing, so a half-set pair is a reader that does not render. + name = "each of the four readers presents its own leaf, never the hive's"; + ok = + let + b = baoGrantWithConsumers.services.hyperhive.deploy.bao; + hiveLeaf = [ + b.clientCertFile + b.clientKeyFile + ]; + own = [ + b.matrixTokenClientCertFile + b.matrixTokenClientKeyFile + b.queueAgentClientCertFile + b.queueAgentClientKeyFile + b.grafanaOidcClientCertFile + b.grafanaOidcClientKeyFile + b.otelOidcClientCertFile + b.otelOidcClientKeyFile + ]; + envOf = unit: baoGrantWithConsumers.systemd.services.${unit}.environment; + presents = + unit: cert: key: + (envOf unit).BAO_CLIENT_CERT == cert && (envOf unit).BAO_CLIENT_KEY == key; + in + lib.all (p: p != null) own + && !(lib.any (p: lib.elem p hiveLeaf) own) + && lib.length (lib.unique own) == lib.length own + && presents "swarm-bao-matrix-token" b.matrixTokenClientCertFile b.matrixTokenClientKeyFile + && presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile + && presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile + && presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile; + } + { + # The minting side of the same claim. A role matching a subject nothing + # signs is a reader that cannot log in, so the leaves and the roles have + # to be asserted against each other — and the two per-hive leaves carry + # THIS host's hive name, which is what makes one hive's leaf useless + # against another hive's role. + name = "the PKI unit signs a leaf per reader, each under that reader's own subject"; + ok = + let + s = baoGrantHere.systemd.services.swarm-bao-pki.script; + in + # The basename and the subject are matched separately: `signLeaf` takes + # them as consecutive arguments across a `\` continuation, so one + # literal spanning both would pin this file's line wrapping rather than + # the pairing it means to. + lib.all (lib.flip lib.hasInfix s) [ + "/matrix-token.pem ]" + "swarm-bao-matrix-token-h1 \"\" clientAuth" + "/queue-agent.pem ]" + "swarm-bao-queue-agent-h1 \"\" clientAuth" + "/grafana-oidc.pem ]" + "swarm-bao-grafana-oidc \"\" clientAuth" + "/otel-oidc.pem ]" + "swarm-bao-otel-oidc \"\" clientAuth" + ]; + } + { + # The absence arm: with no client CA there is no trust anchor, so the + # login roles cannot be written — but the policies they would attach are + # still asserted, exactly as the three service principals above behave in + # this deployment. A unit that vanished here would take the policy with + # it and leave nothing to diagnose. + name = "with no client CA the four readers get policies but no login roles"; + ok = + let + units = [ + "swarm-bao-matrix-token-policy" + "swarm-bao-queue-agent-policy" + "swarm-bao-grafana-oidc-policy" + "swarm-bao-otel-oidc-policy" + ]; + scriptOf = unit: baoGrantNoClientCa.systemd.services.${unit}.script; + in + lib.all ( + unit: + (baoGrantNoClientCa.systemd.services ? ${unit}) + && lib.hasInfix "bao policy write" (scriptOf unit) + && !(lib.hasInfix "auth/cert/certs" (scriptOf unit)) + ) units; + } + { + # Same control the three service principals carry: the write needs a + # client certificate and the host is the side that has one, so a unit + # rendered inside the store's container would have neither an identity + # nor a route. Plus the ordering that makes the mounts exist first. + name = "the four readers' granting units are ordered after the mounts and rendered on the host"; + ok = + let + units = [ + "swarm-bao-matrix-token-policy" + "swarm-bao-queue-agent-policy" + "swarm-bao-grafana-oidc-policy" + "swarm-bao-otel-oidc-policy" + ]; + in + lib.all ( + unit: + lib.elem "swarm-bao-controller-policy.service" baoGrantHere.systemd.services.${unit}.after + && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? ${unit}) + ) units; + } + { + # A store host that has not placed a bootstrap token can write no grant at + # all, so none of the four units may exist — the same claim + # `baoGrantNoStore` makes for the controller's, one file over. Without + # this arm `lib.mkIf haveBootstrapToken` could be dropped from the shared + # builder and every other case here would still pass. + name = "without a bootstrap token none of the four readers' granting units render"; + ok = + let + s = baoGrantNoStore.systemd.services; + in + !(s ? swarm-bao-matrix-token-policy) + && !(s ? swarm-bao-queue-agent-policy) + && !(s ? swarm-bao-grafana-oidc-policy) + && !(s ? swarm-bao-otel-oidc-policy); + } { # The policy authorising this route lives in another file, and nothing # else relates the grants to the paths the code actually writes. diff --git a/nix/module-eval/bao-matrix-reader.nix b/nix/module-eval/bao-matrix-reader.nix index 00f2256b..dd9470a2 100644 --- a/nix/module-eval/bao-matrix-reader.nix +++ b/nix/module-eval/bao-matrix-reader.nix @@ -32,10 +32,20 @@ let # nothing here mints a leaf and the operator names one placed by hand. The # deployment this pairing exists to serve, and the one that was previously # inexpressible — the gate asked whether the store was a neighbour. + # + # 🩸 One leaf PER READER, and this fixture is where that cost is visible: the + # two units below identify themselves to the store separately, so an operator + # placing leaves by hand places one for each rather than one for both. Naming + # only `clientCertFile` here would leave neither unit rendered — which is what + # the arms below would then be asserting about. baoRemoteReader = hive { deploy.matrix.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.matrixTokenClientCertFile = "/etc/pki/bao-matrix-token.pem"; + deploy.bao.matrixTokenClientKeyFile = "/etc/pki/bao-matrix-token-key.pem"; + deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem"; + deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem"; }; # A homeserver on a hive with NO store identity at all — neither a local diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index ae825800..fe3cd97e 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -41,6 +41,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; + deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # The same UI with the IdP on ANOTHER host and a store leaf placed by hand. @@ -54,6 +56,8 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; + deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # A Grafana host holding no store identity. This used to be the shape the @@ -148,8 +152,8 @@ let # mean the two assertions had been collapsed into one conjunction. name = "a grafana host with no store identity is refused, naming the options to set"; ok = - grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientCertFile" - && grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientKeyFile" + grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientCertFile" + && grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientKeyFile" && !(grafanaRefusedFor grafanaNoIdentity "swarm.authelia.url"); } { @@ -161,9 +165,9 @@ let name = "neither grafana refusal fires on a correctly configured host, co-located or not"; ok = !(grafanaRefusedFor grafanaWithAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.clientCertFile") + && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.grafanaOidcClientCertFile") && !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.clientCertFile"); + && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.grafanaOidcClientCertFile"); } { # Same 403-not-a-miss reason as the matrix and queue arms below: the diff --git a/nix/module-eval/name-guards.nix b/nix/module-eval/name-guards.nix index 8716f980..e035b0f0 100644 --- a/nix/module-eval/name-guards.nix +++ b/nix/module-eval/name-guards.nix @@ -63,6 +63,35 @@ let swarm.hives.mintctl.domain = "m.t.local"; }; + # The two OIDC-secret readers' subjects, fixed strings like the three above. + hiveNamedAfterGrafanaOidcSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.grafanaOidcCommonName = "gfctl"; + swarm.hives.gfctl.domain = "g.t.local"; + }; + + hiveNamedAfterOtelOidcSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.otelOidcCommonName = "otctl"; + swarm.hives.otctl.domain = "o.t.local"; + }; + + # 🩸 A different shape from every fixture above: the matrix-token and + # queue-credential roles are written PER HIVE, so the subject a hive must not + # be is `-` rather than the prefix itself. Reserving + # only the prefix would leave the composed spelling free, and a hive taking it + # would present a leaf the other hive's role accepts — which is a hive reading + # another hive's queue credential, the exact widening the split exists to + # avoid. + # + # Two hives here, not one: the collision is with the OTHER hive's role. + hiveNamedAfterPerHiveReaderSubject = hive { + deploy.swarm-otel.enable = false; + deploy.bao.queueAgentCommonNamePrefix = "qr"; + swarm.hives.other.domain = "o.t.local"; + swarm.hives.qr-other.domain = "q.t.local"; + }; + hiveNameWithComposedWord = hive { deploy.swarm-otel.enable = false; swarm.hives."h1-agent".domain = "a.t.local"; @@ -111,6 +140,36 @@ let a: !a.assertion && lib.hasInfix "'mintctl'" a.message ) hiveNamedAfterMatrixCtlSubject.assertions; } + { + # The fourth and fifth, for the reason the case above gives: `certAuthCns` + # is where a role added beside the others registers itself, and nothing + # but a case per element notices when one forgets. These two are the + # subjects of the readers that fetch Grafana's and the collector's OIDC + # client secrets. + name = "a hive named after either OIDC-secret reader's subject is refused too"; + ok = + equalityGuardFired hiveNamedAfterGrafanaOidcSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'gfctl'" a.message + ) hiveNamedAfterGrafanaOidcSubject.assertions + && equalityGuardFired hiveNamedAfterOtelOidcSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'otctl'" a.message + ) hiveNamedAfterOtelOidcSubject.assertions; + } + { + # 🩸 The per-hive half, and the one a prefix-only reservation would miss: + # the role is `-`, so the reserved string has to be composed + # against every declared hive. Here hive `qr-other` collides with the role + # written for hive `other` — a leaf that reads a credential belonging to a + # hive that is not it. + name = "a hive named after another hive's per-hive reader subject is refused"; + ok = + equalityGuardFired hiveNamedAfterPerHiveReaderSubject + && lib.any ( + a: !a.assertion && lib.hasInfix "'qr-other'" a.message + ) hiveNamedAfterPerHiveReaderSubject.assertions; + } { # Without this the case above proves nothing: an arm that fires for every # roster is not a guard, and `hives` is non-empty in both fixtures. diff --git a/nix/module-eval/swarm-otel-core.nix b/nix/module-eval/swarm-otel-core.nix index 7428919c..0bbfa10e 100644 --- a/nix/module-eval/swarm-otel-core.nix +++ b/nix/module-eval/swarm-otel-core.nix @@ -32,6 +32,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; deploy.victoriametrics.enable = false; deploy.victorialogs.enable = false; }; @@ -44,6 +46,8 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; # The same collector with the IdP on ANOTHER host and a store leaf placed by @@ -54,6 +58,8 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; + deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; + deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; cases = [ {