diff --git a/docs/getting-started/setup.md b/docs/getting-started/setup.md index 2225529e..af3290d3 100644 --- a/docs/getting-started/setup.md +++ b/docs/getting-started/setup.md @@ -170,19 +170,16 @@ What else you need depends on restart, with the keys `init` printed. The store serves TLS, and on a hive that deploys it you need do nothing: a -first-boot unit mints a CA of the store's own plus the leaves it signs — the -store's server certificate, this host's client certificate, and one per service -principal — and points `deploy.bao.serverCertFile`, `.serverKeyFile` and -`.clientCaFile` at the store's half, `.clientCertFile`, `.clientKeyFile` and -`.serverCaFile` at the reader's, and each principal's own pair at its own leaf. +first-boot unit mints a CA of the store's own plus the two leaves it signs — +the store's server certificate and this host's client certificate — and points +`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at the store's +half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's. Those are `mkDefault`s, so naming your own paths wins. Do that when your certificates come from a real internal CA; the store has no opinion about which. A hive that does **not** deploy the store names the reader's three -itself, plus a pair for every principal it runs — see -[per-principal identities](../swarm/secrets.md#per-principal-identities) for the -list. The operator issues those leaves out of band; they're the credentials the -store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is +itself: the operator issues that leaf out of band, and it's the one credential +the store can't hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is **mTLS between services and the store**, a separate trust domain, because a store that took its identity from an authority it itself distributes could never come up before that authority. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index b5f3c38a..53bce3ad 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -88,7 +88,7 @@ where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — only its own pushes to the stores go out unauthenticated and get refused there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without -`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice +`deploy.bao.clientCertFile` / `clientKeyFile`, the same choice `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for their own optional readers, rather than the hard assertion `swarm-grafana.nix` uses for its non-optional one. The operator can still deliver the secret by @@ -249,14 +249,14 @@ straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store -exists so the only secrets a host holds out of band are **certificates**, and -a host reads everything else with them. +exists so a host holds **one** out-of-band secret, its client certificate, and +reads everything else with it. Two things follow, and `swarm-grafana.nix` asserts both rather than degrading: running Grafana requires `swarm.authelia.url` (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the -only way in), and it requires this host's `deploy.bao.grafanaOidcClientCertFile` -/ `grafanaOidcClientKeyFile`. Each refusal names the option to set. Both used to be silent: +only way in), and it requires this host's `deploy.bao.clientCertFile` / +`clientKeyFile`. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is @@ -271,8 +271,7 @@ runs the collector, and registration lives separately in `glue-swarm-otel-oidc-client.nix` for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so `swarm-otel.nix` doesn't assert -`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana -does — it +`deploy.bao.clientCertFile` / `clientKeyFile` the way Grafana does — it doesn't render the reading unit without them, the shape `glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for their own optional reads. @@ -291,18 +290,10 @@ A service's secret is one value for the whole swarm rather than one per hive, so it lives under the `services` prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a `deploy.*` fact, per-host by definition — so -there is nothing to scope the grant to. A hive's **own** leaf can therefore read -every swarm service's client secret, and that's stated in -`swarm-secret-client`'s `policy` module beside the grant itself. - -⚠️ **The readers no longer present it.** Four units used to log in with -`deploy.bao.clientCertFile`, which is the hive's own leaf, and bao identifies a -principal by the subject of the certificate it presents — so four readers behind -one leaf were one principal holding the union of their needs. Each now holds a -leaf of its own, and a policy naming only the path that unit reads. See -[per-principal identities](#per-principal-identities) below. - -Absence behaves +there is nothing to scope the grant to. Every hive can therefore read every swarm +service's client secret; that's the cost of a shape where the service's host has +no store identity of its own to present, and it's stated in +`swarm-secret-client`'s `policy` module beside the grant itself. Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in. @@ -312,78 +303,14 @@ credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the hive CA and the swarm CA both, and has nothing to do with the gateway's HTTPS certificates either way. `glue-bao-tls.nix` -mints a CA that signs a fixed, short list of leaves — the store's server -certificate, the hive's own reader certificate, and one per service principal — -and distributes nothing. A deployment with a real +mints a CA that signs exactly two things, the store's server certificate and a +reader's client certificate, and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in `deploy.bao.serverCertFile` / `clientCaFile`; the store itself has no opinion. A hive that reads from a store on **another** machine names the reader's half — -`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand, -plus one leaf per principal it runs (the table below names the options). Those are the -credentials that can't come out of the store, being what opens it; everything -else a hive needs does. - -### Per-principal identities - -Bao matches a cert-auth role on the certificate's subject, so a certificate is an -identity and sharing one merges the identities. These four units read one path -each and each holds a leaf, a role and a policy of its own: - -| unit | option pair under `deploy.bao.` | reads | -| ------------------------ | -------------------------------------------------------- | -------------------------------------------------- | -| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives//matrix/appservice-token` | -| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives//queue/agent` | -| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services//oidc/client` | -| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services//oidc/client` | - -The first two exist **per hive**, because the path they read carries a hive -name and every hive runs its own reader. Their subjects are -`-` and -`-`; `swarm.nix` reserves both -composed spellings as hive names, so nobody can name a hive into another hive's -role. The other two read a path that names a swarm service rather than a hive, so -one role each is enough and their subjects are the flat -`deploy.bao.grafanaOidcCommonName` and `deploy.bao.otelOidcCommonName`. - -On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults -all eight options, and there is nothing to do. Elsewhere you issue each leaf from -that CA out of band and name it here — one file per principal rather than one file -shared by four, which is the whole of what this buys. - -Forgetting one of the eight elsewhere isn't a quiet degrade. Three of the four -readers used to render only where their leaf existed, so a hand-configured -remote-store hive that named the hive's own `clientCertFile` and missed a -principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing -the build. All four now refuse it, each naming its own option pair — see -[what a missing leaf costs](#what-a-missing-leaf-costs) below. - -### What a missing leaf costs - -Each of the four modules asserts its own pair, on the same condition: **this host -already reads the store** — it holds `deploy.bao.clientCertFile` and -`clientKeyFile` — **and the consumer is on, and this principal's pair is null**. -The refusal names the two options, so an operator acts on the message without -opening the nix. - -That condition fires on exactly one shape, the hand-configured remote-store hive -that named seven of the eight. It stays silent everywhere else, and each half of -that matters on its own: - -- **A host with no store identity at all** holds no `clientCertFile` either, so - the first clause is false. That's the supported no-store deployment, and also - the one an operator passes through while bringing a hive up — neither should - fail to evaluate. -- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as - defaults, so the third clause is false. Nothing to set, nothing to refuse. -- **A consumer that's off** — no homeserver, no collector, no Grafana — has no - unit to skip, so the second clause is false. Only the queue-credential reader - has no toggle of its own to check: every hive runs one for its own agents, so - reading the store at all is what asks for it. - -The collector keeps a separate degrade underneath this, unchanged: a host with no -store identity runs a collector with no OIDC client secret and still receives -telemetry. The assertion doesn't touch that shape, because that host has no -`clientCertFile` either. +`clientCertFile`, `clientKeyFile`, `serverCaFile` — and places that leaf by hand. +It's the one credential that can't come out of the store, being what opens it; +everything else a hive needs does. ## How a reader reaches the store diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix index 1afaed01..819569bf 100644 --- a/nix/host-modules/glue-bao-tls.nix +++ b/nix/host-modules/glue-bao-tls.nix @@ -84,25 +84,6 @@ in clientCertFile = lib.mkDefault "${pkiDir}/client.pem"; clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem"; serverCaFile = lib.mkDefault "${pkiDir}/ca.pem"; - - # 🩸 Four readers that used to present `client.pem` above, each now - # pointed at a leaf of its own. The leaf is what bao sees, so this pairing - # is the whole of what turns "one principal with the union of four grants" - # into four principals with one grant each — see ./swarm-bao.nix's - # `perHiveReaders` block for the grants themselves. - # - # Defaulted here rather than in four `glue--bao-identity.nix` - # files: those exist where the consumer is a CONTAINER with a - # `deploy..*` namespace of its own to point at. These four are - # host units reading the store, which is the pairing this file already is. - matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem"; - matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem"; - queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem"; - queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem"; - grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem"; - grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem"; - otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem"; - otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem"; }; # Idempotent on ABSENCE, never on content. Re-issuing the CA invalidates @@ -179,35 +160,6 @@ in # container an identity instead of lending it the hive's. [ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \ ${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth - - # 🩸 The four readers that used to present `client.pem` above. Each - # carries its own subject, which is the entire mechanism: bao matches a - # cert-auth role on the CN, so four units sharing one leaf were one - # principal holding the union of four grants, and the union included - # every agent credential in the swarm. - # - # ⚠️ The first two carry THIS hive's name in the subject, unlike the - # three service leaves above. Their grants name one hive's path, because - # a matrix token and a queue credential live under `swarm/hives//` - # and every hive runs a reader for its own; ./swarm-bao.nix writes one - # role per hive in the swarm directory to match. The two OIDC readers - # need no such segment — a client is registered once per swarm. - # - # Minted whether or not the consumer runs here, for the reason the three - # leaves above give: on a hive that does not run the store this is the - # file an operator copies, and a leaf that only appears where its - # consumer does is one nobody can copy from anywhere. - [ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \ - ${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth - - [ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \ - ${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth - - [ -s ${pkiDir}/grafana-oidc.pem ] || ${signLeaf} ${pkiDir} grafana-oidc \ - ${lib.escapeShellArg deployCfg.bao.grafanaOidcCommonName} "" clientAuth - - [ -s ${pkiDir}/otel-oidc.pem ] || ${signLeaf} ${pkiDir} otel-oidc \ - ${lib.escapeShellArg deployCfg.bao.otelOidcCommonName} "" clientAuth ''; }; }; diff --git a/nix/host-modules/glue-matrix-bao-token.nix b/nix/host-modules/glue-matrix-bao-token.nix index 3c0afcb6..077c7b86 100644 --- a/nix/host-modules/glue-matrix-bao-token.nix +++ b/nix/host-modules/glue-matrix-bao-token.nix @@ -38,24 +38,7 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - # - # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four - # units used to share. Bao matches a cert-auth role on the CN, so one leaf for - # four readers was ONE principal holding the union of four grants: read on - # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when - # this unit reads one appservice token and nothing else. Its own leaf carries - # `-` and its role grants the single path - # below. - haveClientIdentity = - baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null; - - # Does this host read the store at all — the hive's own leaf, which is the - # one thing a remote-store deployment has always had to place by hand. Only - # used to decide whether a missing per-principal leaf is a mistake or a - # deployment that has no store: a host holding neither is the supported - # no-store shape, and one holding this pair but not the pair above named - # seven of the eight options and stopped. - hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; # Where the token lives in the store. A path, not a convention to guess at: # whoever writes it and whoever reads it must agree, and the agreement @@ -81,178 +64,134 @@ let matrixMachine = "hive-matrix"; in { - config = lib.mkMerge [ - # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole - # mechanism: the unit's arm is gated on `haveClientIdentity`, so an - # assertion written inside it could never be reached in the state it - # exists to report. - # - # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the - # same refusal, named to this principal's own pair. What differs is the - # gate. Grafana asserts wherever Grafana runs, because a Grafana with no - # store identity has no way in at all; a homeserver with no store identity - # is a hive that has no store, which is supported. So this one additionally - # requires `hiveReaderIdentity`: the host demonstrably reads the store, and - # named every option but this pair. - (lib.mkIf (hyperhiveCfg.enable && deployCfg.matrix.enable && hiveReaderIdentity) { - assertions = [ - { - assertion = haveClientIdentity; - message = '' - This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile - is set) and runs a homeserver, so it needs the matrix appservice - token reader's own client identity: set both + config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { + # Same rule as the unit's own gate: this reader exists on a host that has a + # client identity and a homeserver, which is not every host that runs the + # store, so the store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; - services.hyperhive.deploy.bao.matrixTokenClientCertFile - services.hyperhive.deploy.bao.matrixTokenClientKeyFile - - swarm-bao-matrix-token.service fetches this hive's appservice token - out of the store, and without these it is not rendered at all — - leaving the homeserver authenticating hive-c0re against whatever is - already on disk, which is a 401 on every request naming nothing. - - ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is - the hive's, and its grant reads every secret in the store; this role - reads the one appservice-token path. Pointing this option at the - hive's leaf would evaluate, deploy and log in — and undo the split. - - On a hive that runs the store, glue-bao-tls.nix supplies both as - defaults and there is nothing to do. Elsewhere the leaf is issued - from that CA out of band and named here — see docs/swarm/secrets.md. - ''; - } + systemd.services.swarm-bao-matrix-token = { + description = "fetch the matrix appservice token from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" ]; - }) - - (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity && deployCfg.matrix.enable) { - # Same rule as the unit's own gate: this reader exists on a host that has a - # client identity and a homeserver, which is not every host that runs the - # store, so the store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ]; - - systemd.services.swarm-bao-matrix-token = { - description = "fetch the matrix appservice token from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" - ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - before = [ "container@${matrixMachine}.service" ]; - wantedBy = [ "container@${matrixMachine}.service" ]; - path = [ - deployCfg.bao.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal. `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after — so a few short attempts cover it. - # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. - # That unit blocks nothing; this one is `Before=` the homeserver's - # container, and whether that ordering waits across an auto-restart is - # unverified — so an hours-long window would be a bet on an unknown, - # where a minute is not. A store still sealed after it keeps the degrade - # below, as today. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the read below. Stated here rather than - # left to systemd's default, so the number a boot waits on is in - # the file that waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile; - BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail - - # A sealed or uninitialised store answers on the port and never - # answers the read, so "the store is up" is not the same as "the - # store can answer". `TimeoutStartSec` above is the bound; the - # homeserver only `Wants=` this unit, so hitting it degrades to - # keeping the local token rather than holding up the container. - # `bao`'s own message is the only thing separating a missing value - # from a refused identity from an unreachable host. This unit's - # degraded mode is correct for all three, so it reports which one - # rather than asserting all three in a sentence of ours — a reader - # that cannot say why it read nothing is indistinguishable from a - # broken one. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT - - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. - # - # Fails LOUDLY, unlike the read below: the three states a login failure - # covers — store not up, sealed, role not written yet — are all things - # a retry fixes, and `Restart=on-failure` above is what retries. Exiting - # 0 here spends the whole boot on a condition that was seconds old. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 1 - fi - export BAO_TOKEN - - if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then - echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 0 - fi - - if [ -z "$token" ]; then - echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 - exit 0 - fi - - umask 077 - printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} - - # Re-stamp the registration file from the token just written. The - # token is half an agreement — the registration the homeserver loads - # has to carry the same value — so writing the file and stopping - # would leave the homeserver authenticating hive-c0re against - # whatever activation put there: a 401 on every request, naming - # nothing. Unconditional rather than on-change, because this unit - # has no way to know what the registration currently says. - # - # hive-matrix's own renderer rather than a `printf` here, so the - # registration's shape has one home. - ${deployCfg.matrix.appserviceRegistrationScript} - ''; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + before = [ "container@${matrixMachine}.service" ]; + wantedBy = [ "container@${matrixMachine}.service" ]; + path = [ + deployCfg.bao.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal. `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after — so a few short attempts cover it. + # ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy. + # That unit blocks nothing; this one is `Before=` the homeserver's + # container, and whether that ordering waits across an auto-restart is + # unverified — so an hours-long window would be a bet on an unknown, + # where a minute is not. A store still sealed after it keeps the degrade + # below, as today. + # + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the read below. Stated here rather than + # left to systemd's default, so the number a boot waits on is in + # the file that waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; }; - }) - ]; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.clientCertFile; + BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # A sealed or uninitialised store answers on the port and never + # answers the read, so "the store is up" is not the same as "the + # store can answer". `TimeoutStartSec` above is the bound; the + # homeserver only `Wants=` this unit, so hitting it degrades to + # keeping the local token rather than holding up the container. + # `bao`'s own message is the only thing separating a missing value + # from a refused identity from an unreachable host. This unit's + # degraded mode is correct for all three, so it reports which one + # rather than asserting all three in a sentence of ours — a reader + # that cannot say why it read nothing is indistinguishable from a + # broken one. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the read below: the three states a login failure + # covers — store not up, sealed, role not written yet — are all things + # a retry fixes, and `Restart=on-failure` above is what retries. Exiting + # 0 here spends the whole boot on a condition that was seconds old. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 + fi + export BAO_TOKEN + + if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then + echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 + fi + + if [ -z "$token" ]; then + echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2 + exit 0 + fi + + umask 077 + printf '%s\n' "$token" > ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + chmod 0600 ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} + + # Re-stamp the registration file from the token just written. The + # token is half an agreement — the registration the homeserver loads + # has to carry the same value — so writing the file and stopping + # would leave the homeserver authenticating hive-c0re against + # whatever activation put there: a 401 on every request, naming + # nothing. Unconditional rather than on-change, because this unit + # has no way to know what the registration currently says. + # + # hive-matrix's own renderer rather than a `printf` here, so the + # registration's shape has one home. + ${deployCfg.matrix.appserviceRegistrationScript} + ''; + }; + }; } diff --git a/nix/host-modules/glue-queue-agent-credential.nix b/nix/host-modules/glue-queue-agent-credential.nix index 8fa2dc06..9e57b512 100644 --- a/nix/host-modules/glue-queue-agent-credential.nix +++ b/nix/host-modules/glue-queue-agent-credential.nix @@ -40,24 +40,7 @@ let # What decides whether this unit exists at all. A reader is defined by holding # a certificate the store accepts, and that is true on the store's own host # and on a hive three networks away for exactly the same reason. - # - # 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four - # units used to share. Bao matches a cert-auth role on the CN, so one leaf for - # four readers was ONE principal holding the union of four grants: read on - # `swarm/agents/*` AND `swarm/hives//*` AND `swarm/services/*`, when - # this unit reads one queue credential and nothing else. Its own leaf carries - # `-` and its role grants the single path - # below — still this hive's own, so the narrowing costs no reach. - haveClientIdentity = - baoDeploy.queueAgentClientCertFile != null && baoDeploy.queueAgentClientKeyFile != null; - - # Does this host read the store at all — the hive's own leaf, which is the - # one thing a remote-store deployment has always had to place by hand. Only - # used to decide whether a missing per-principal leaf is a mistake or a - # deployment that has no store: a host holding neither is the supported - # no-store shape, and one holding this pair but not the pair above named - # seven of the eight options and stopped. - hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir; secretFile = "${credentialDir}/secret"; @@ -102,192 +85,144 @@ in }; }; - config = lib.mkMerge [ - # ⚠️ A SEPARATE arm from the unit below, and that separation is the whole - # mechanism: the unit's arm is gated on `haveClientIdentity`, so an - # assertion written inside it could never be reached in the state it - # exists to report. - # - # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the - # same refusal, named to this principal's own pair. Where Grafana's gate - # is `deploy.grafana.enable`, this reader has no toggle of its own to - # check: every hive runs one for its own agents, so reading the store at - # all is what asks for it. Hence `hiveReaderIdentity` alone — a host - # holding no hive leaf has no store to read and nothing is missing. - (lib.mkIf (hyperhiveCfg.enable && hiveReaderIdentity) { - assertions = [ - { - assertion = haveClientIdentity; - message = '' - This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile - is set), so it needs the agent queue credential reader's own client - identity: set both + config = lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { + # Same rule as the unit's own gate: this reader exists on any host holding + # a client identity, which is not every host that runs the store, so the + # store's module cannot name it. + services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; - services.hyperhive.deploy.bao.queueAgentClientCertFile - services.hyperhive.deploy.bao.queueAgentClientKeyFile - - swarm-bao-queue-agent.service fetches this hive's agent queue - credential out of the store, and without these it is not rendered - at all — leaving hive-c0re with no queue credential and no unit - that would ever have written one. - - Every hive runs this reader for its own agents, so unlike the other - three principals there is no per-service toggle that turns it off: - a hive that reads the store at all is a hive that needs this pair. - - ⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one - is the hive's, and its grant reads every secret in the store; this - role reads the one queue-credential path. Pointing this option at - the hive's leaf would evaluate, deploy and log in — and undo the - split. - - On a hive that runs the store, glue-bao-tls.nix supplies both as - defaults and there is nothing to do. Elsewhere the leaf is issued - from that CA out of band and named here — see docs/swarm/secrets.md. - ''; - } + systemd.services.swarm-bao-queue-agent = { + description = "fetch this hive's agent queue credential from the swarm secret store"; + # Every one of these names a unit that exists only where the store runs. + # `Requires=` on an absent unit fails the job outright, so the ordering is + # conditional even though the read is not: off-host there is nothing local + # to wait for, and the timeout below is what bounds the attempt instead. + after = lib.optionals baoDeploy.enable [ + "swarm-bao-pki.service" + "container@${baoCfg.machine}.service" ]; - }) - - (lib.mkIf (hyperhiveCfg.enable && haveClientIdentity) { - # Same rule as the unit's own gate: this reader exists on any host holding - # a client identity, which is not every host that runs the store, so the - # store's module cannot name it. - services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-queue-agent" ]; - - systemd.services.swarm-bao-queue-agent = { - description = "fetch this hive's agent queue credential from the swarm secret store"; - # Every one of these names a unit that exists only where the store runs. - # `Requires=` on an absent unit fails the job outright, so the ordering is - # conditional even though the read is not: off-host there is nothing local - # to wait for, and the timeout below is what bounds the attempt instead. - after = lib.optionals baoDeploy.enable [ - "swarm-bao-pki.service" - "container@${baoCfg.machine}.service" - ]; - wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; - requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; - # Ordered before hive-c0re, so no agent container renders ahead of an - # attempt at its credential. `Wants=`, not `Requires=`: a store this - # unit can't reach delays hive-c0re's start by its own start-limit - # window (`TimeoutStartSec`, retried up to `startLimitBurst` times - # below) rather than failing it — hive-c0re starts once that window - # elapses, whatever credential is or isn't on disk by then. - before = [ "hive-c0re.service" ]; - wantedBy = [ - "multi-user.target" - "hive-c0re.service" - ]; - path = [ - baoDeploy.package - pkgs.coreutils - ]; - # Sized for the race this loses, not for an unseal: `swarm-bao` comes up - # seconds before this unit asks, and the cert-auth role it logs in - # against is written seconds after, so a few short attempts cover it. - # An hours-long window would be a bet on a store that is sealed, and the - # degrade below is already correct for that. - # - # `StartLimit*` are `[Unit]` settings, so they go here and not in - # `serviceConfig` — systemd ignores them under `[Service]`. The window - # has to exceed `RestartSec × burst`. - startLimitBurst = 4; - startLimitIntervalSec = 300; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - # What actually bounds the reads below. Stated here rather than left to - # systemd's default, so the number a boot waits on is in the file that - # waits. - TimeoutStartSec = 30; - Restart = "on-failure"; - RestartSec = 15; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.queueAgentClientCertFile; - BAO_CLIENT_KEY = baoDeploy.queueAgentClientKeyFile; - } - # Absent means the system trust store, which is what a deployment with a - # real CA wants and what a self-signed one must not be left with. - // lib.optionalAttrs (baoDeploy.serverCaFile != null) { - BAO_CACERT = baoDeploy.serverCaFile; - }; - script = '' - set -euo pipefail - - # `bao`'s own message is the only thing separating a missing value from - # a refused identity from an unreachable host. This unit's degraded - # mode is correct for all three, so it reports which one rather than - # asserting all three in a sentence of ours. - err="$(mktemp)" - trap 'rm -f "$err"' EXIT - - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. - # - # Fails LOUDLY, unlike the reads below: the three states a login - # failure covers — store not up, sealed, role not written yet — are all - # things a retry fixes, and `Restart=on-failure` above is what retries. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 1 - fi - export BAO_TOKEN - - # Two reads of one object rather than one `-format=json` parsed with - # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot - # actually disagree — the client id is derived from this hive's name, so - # a rotation landing between these two calls changes the secret and - # rewrites the same id. - if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 0 - fi - - if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then - echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 0 - fi - - # Both or neither, for the reason `QueueConfig::from_env` refuses a - # half-set environment: a client that finds one of the two comes up - # "fine" and never connects. - if [ -z "$secret" ] || [ -z "$client_id" ]; then - echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 - exit 0 - fi - - install -d -m 0755 ${lib.escapeShellArg credentialDir} - - umask 077 - printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} - chmod 0600 ${lib.escapeShellArg secretFile} - - # `0644` on purpose: an OIDC client id is presented to the token - # endpoint on every connection and is public by construction. - printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} - chmod 0644 ${lib.escapeShellArg clientIdFile} - ''; + wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ]; + requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ]; + # Ordered before hive-c0re, so no agent container renders ahead of an + # attempt at its credential. `Wants=`, not `Requires=`: a store this + # unit can't reach delays hive-c0re's start by its own start-limit + # window (`TimeoutStartSec`, retried up to `startLimitBurst` times + # below) rather than failing it — hive-c0re starts once that window + # elapses, whatever credential is or isn't on disk by then. + before = [ "hive-c0re.service" ]; + wantedBy = [ + "multi-user.target" + "hive-c0re.service" + ]; + path = [ + baoDeploy.package + pkgs.coreutils + ]; + # Sized for the race this loses, not for an unseal: `swarm-bao` comes up + # seconds before this unit asks, and the cert-auth role it logs in + # against is written seconds after, so a few short attempts cover it. + # An hours-long window would be a bet on a store that is sealed, and the + # degrade below is already correct for that. + # + # `StartLimit*` are `[Unit]` settings, so they go here and not in + # `serviceConfig` — systemd ignores them under `[Service]`. The window + # has to exceed `RestartSec × burst`. + startLimitBurst = 4; + startLimitIntervalSec = 300; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + # What actually bounds the reads below. Stated here rather than left to + # systemd's default, so the number a boot waits on is in the file that + # waits. + TimeoutStartSec = 30; + Restart = "on-failure"; + RestartSec = 15; }; - }) - ]; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = baoDeploy.clientCertFile; + BAO_CLIENT_KEY = baoDeploy.clientKeyFile; + } + # Absent means the system trust store, which is what a deployment with a + # real CA wants and what a self-signed one must not be left with. + // lib.optionalAttrs (baoDeploy.serverCaFile != null) { + BAO_CACERT = baoDeploy.serverCaFile; + }; + script = '' + set -euo pipefail + + # `bao`'s own message is the only thing separating a missing value from + # a refused identity from an unreachable host. This unit's degraded + # mode is correct for all three, so it reports which one rather than + # asserting all three in a sentence of ours. + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` + # variables above only decide which certificate the TLS handshake + # presents; without a token `bao` asks its token helper instead, and + # that is a `sh` this unit's `path` does not carry. `-token-only` + # answers on stdout and skips the helper on both sides. + # + # Fails LOUDLY, unlike the reads below: the three states a login + # failure covers — store not up, sealed, role not written yet — are all + # things a retry fixes, and `Restart=on-failure` above is what retries. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to swarm-bao with this host's certificate; leaving the queue credential in ${credentialDir} as it is." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 + fi + export BAO_TOKEN + + # Two reads of one object rather than one `-format=json` parsed with + # `jq`: no sibling unit carries `jq` on its `path`, and the pair cannot + # actually disagree — the client id is derived from this hive's name, so + # a rotation landing between these two calls changes the secret and + # rewrites the same id. + if ! secret="$(bao kv get -field=value ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "swarm-bao did not return ${credentialPath}; this hive's agents have no queue credential yet." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 + fi + + if ! client_id="$(bao kv get -field=client_id ${lib.escapeShellArg credentialPath} 2>"$err")"; then + echo "${credentialPath} holds no client_id; the secret alone is not a usable credential, so nothing is written." >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 0 + fi + + # Both or neither, for the reason `QueueConfig::from_env` refuses a + # half-set environment: a client that finds one of the two comes up + # "fine" and never connects. + if [ -z "$secret" ] || [ -z "$client_id" ]; then + echo "swarm-bao returned an empty field of ${credentialPath}; leaving the files as they are." >&2 + exit 0 + fi + + install -d -m 0755 ${lib.escapeShellArg credentialDir} + + umask 077 + printf '%s\n' "$secret" > ${lib.escapeShellArg secretFile} + chmod 0600 ${lib.escapeShellArg secretFile} + + # `0644` on purpose: an OIDC client id is presented to the token + # endpoint on every connection and is public by construction. + printf '%s\n' "$client_id" > ${lib.escapeShellArg clientIdFile} + chmod 0644 ${lib.escapeShellArg clientIdFile} + ''; + }; + }; } diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 8fd7474c..e577a988 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -340,156 +340,6 @@ let } ''; - # ── the four principals that used to share the hive's own leaf ───────────── - # - # 🩸 Each of the four reads exactly ONE path in the store, and until this - # split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf, - # whose policy (`swarm-secret-client`'s `policy::render`) grants read on - # `swarm/agents/*`, `swarm/hives//*` AND `swarm/services/*`. Four - # principals presenting one certificate are ONE principal to bao, so the union - # of what the four need was the only grant expressible: the unit that fetches - # Grafana's OIDC client secret could fetch every agent credential in the - # swarm, and the unit that fetches this hive's matrix token could fetch - # Grafana's. Least privilege was not misconfigured, it was unrepresentable. - # - # Each now gets its own leaf, its own cert-auth role and a policy holding the - # one `secret/data/…` path that principal's script actually names — spelled to - # the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above - # gives for doing the same. - # - # ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the - # shape of the paths rather than a preference. A matrix appservice token and a - # queue credential live under `swarm/hives//`, and every hive runs a - # reader for its own; an OIDC client secret lives under - # `swarm/services//` and a swarm registers each of those exactly - # once. A single role for a per-hive reader would have to name `hives/*` to - # serve every hive — which would let one hive read another's matrix token, a - # reach no hive has today and one this change exists to remove rather than - # create. - readStanza = path: '' - path "${path}" { - capabilities = ["read"] - } - ''; - - # One reader per hive in the swarm directory. Written from the directory - # rather than from this host's own name because the policy is written where - # the STORE is and the reader runs where its hive is — the same split - # `matrixCtlHiveName` above exists to paper over, answered here by naming - # every hive instead of asking the operator which one. - perHiveReaders = - { rolePrefix, cnPrefix, ... }@spec: - lib.mapAttrsToList (hiveName: _: { - name = "${rolePrefix}-${hiveName}"; - cn = "${cnPrefix}-${hiveName}"; - policyText = readStanza (spec.path hiveName); - }) hyperhiveCfg.swarm.hives; - - matrixTokenReaders = perHiveReaders { - rolePrefix = "swarm-matrix-token"; - cnPrefix = baoDeploy.matrixTokenCommonNamePrefix; - path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token"; - }; - - queueAgentReaders = perHiveReaders { - rolePrefix = "swarm-queue-agent"; - cnPrefix = baoDeploy.queueAgentCommonNamePrefix; - path = h: "${credentialMountPath}/data/swarm/hives/${h}/queue/agent"; - }; - - # Singletons, so a list of one rather than a second shape: the granting unit - # below takes a list either way and the four units stay identical apart from - # the objects they write. - grafanaOidcReaders = [ - { - name = "swarm-grafana-oidc"; - cn = baoDeploy.grafanaOidcCommonName; - policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"; - } - ]; - - otelOidcReaders = [ - { - name = "swarm-otel-oidc"; - cn = baoDeploy.otelOidcCommonName; - policyText = readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.otel.clientId}/oidc/client"; - } - ]; - - # The role name IS the policy name, as for the three service principals - # above: the role attaches the policy by spelling it identically, and one - # string for both objects removes the way they drift apart. - readerPolicyWrite = obj: '' - printf '%s' ${lib.escapeShellArg obj.policyText} | - bao policy write ${lib.escapeShellArg obj.name} - - ''; - - # ⚠️ Names are `swarm-[-]`, outside the `hive-*` namespace the - # controller may rewrite, for the reason the three service principals above - # state. A per-hive CN cannot collide with a hive's own leaf either — a hive's - # CN is its bare name and ../reserved-hive-fragments.nix forbids the substring - # `swarm` in one — and ./swarm.nix reserves the concrete per-hive spellings - # anyway, because the prefixes are operator-settable and an operator may spell - # one without that substring. - readerRoleWrite = obj: '' - bao write auth/cert/certs/${lib.escapeShellArg obj.name} \ - certificate=@${tlsDir}/client-ca.pem \ - allowed_common_names=${lib.escapeShellArg obj.cn} \ - token_policies=${lib.escapeShellArg obj.name} \ - display_name=${lib.escapeShellArg obj.cn} - ''; - - # One unit per PRINCIPAL, not one unit for the four: these names are - # operator-facing strings, the same reason `swarm-bao-secret-publisher-policy` - # above gives for not being more script inside the controller's unit. A hive - # whose matrix token never arrives is diagnosed by the state of the unit named - # after it. - # - # `after` and not `requires`, for the reason the publisher's unit states: the - # controller's unit creates the KV and cert-auth mounts this one writes into, - # but a failed oneshot still counts as finished, so ordering plus this unit's - # own retry is what converges. - # - # The role write is inside the client-CA branch and the policy write is not, - # exactly as the three above: with no CA there is no trust anchor for a login - # role, but the policy it would attach is still worth asserting. - readerPolicyUnit = - description: objects: - lib.mkIf haveBootstrapToken { - inherit description; - after = [ - "container@${cfg.machine}.service" - "swarm-bao-controller-policy.service" - ]; - wantedBy = [ "multi-user.target" ]; - path = [ - baoCli - pkgs.coreutils - ]; - unitConfig.ConditionPathExists = baoDeploy.bootstrapTokenFile; - # Same unseal wait as its siblings above, for the reason stated there: - # under `seal = "shamir"` a human unseals by hand. - startLimitBurst = 2880; - startLimitIntervalSec = 90000; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - Restart = "on-failure"; - RestartSec = 30; - }; - script = '' - set -euo pipefail - - BAO_TOKEN="$(cat ${lib.escapeShellArg baoDeploy.bootstrapTokenFile})" - export BAO_TOKEN - - '' - + lib.concatMapStrings readerPolicyWrite objects - + lib.optionalString (baoDeploy.clientCaFile != null) ( - "\n" + lib.concatMapStrings readerRoleWrite objects - ); - }; - # Every listener serves the same identity: they differ in which address # they answer on, not in who they are. Client verification is separate and # optional — a store with no `clientCaFile` still serves TLS, it just does @@ -996,185 +846,6 @@ in ''; }; - matrixTokenCommonNamePrefix = lib.mkOption { - type = lib.types.str; - default = "swarm-bao-matrix-token"; - example = "swarm-bao-matrix-token.svc"; - description = '' - Prefix of the subject the store's per-hive matrix-token roles accept. - The role for hive `h` accepts `-h` and nothing else, and grants - read on exactly `swarm/hives/h/matrix/appservice-token`. - - A **prefix** rather than one subject because every hive runs its own - reader and the path it reads carries that hive's name. One subject for - all of them would need a `hives/*` grant, which would let one hive read - another hive's appservice token — a reach no hive has today. - - ⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its - fixed-subject siblings are: cert auth trusts the CA and - `allowed_common_names` is the whole narrowing, so a hive whose own leaf - carried one of these spellings would receive that grant. - ''; - }; - - queueAgentCommonNamePrefix = lib.mkOption { - type = lib.types.str; - default = "swarm-bao-queue-agent"; - example = "swarm-bao-queue-agent.svc"; - description = '' - Prefix of the subject the store's per-hive queue-credential roles - accept. The role for hive `h` accepts `-h` and grants read on - exactly `swarm/hives/h/queue/agent` — the client credential that hive's - agent containers authenticate to the swarm queue with. - - Per hive for the same reason - {option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is, - and more sharply: every hive in the swarm runs this reader, so a single - subject would have to be granted `hives/*`. - ''; - }; - - grafanaOidcCommonName = lib.mkOption { - type = lib.types.str; - default = "swarm-bao-grafana-oidc"; - example = "swarm-bao-grafana-oidc.svc"; - description = '' - Subject the store's `swarm-grafana-oidc` cert-auth role accepts — the - identity the unit that fetches Grafana's OIDC client secret presents. - - Its grant is one path, - `swarm/services//oidc/client`, and read only. Not - per hive, unlike the two prefixes above: an OIDC client is registered - once per swarm, so the path names the service and never a hive. - - ⚠️ Same collision as its siblings, and the same answer: ./swarm.nix - feeds this value into the guard on - {option}`services.hyperhive.swarm.hives`. - ''; - }; - - otelOidcCommonName = lib.mkOption { - type = lib.types.str; - default = "swarm-bao-otel-oidc"; - example = "swarm-bao-otel-oidc.svc"; - description = '' - Subject the store's `swarm-otel-oidc` cert-auth role accepts — the - identity the unit that fetches the collector's OIDC client secret - presents. Its grant is one path, - `swarm/services//oidc/client`, and read only. - - A **fourth** identity rather than reuse of - {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two - read different services' client secrets, and a collector is not - entitled to Grafana's. - ''; - }; - - matrixTokenClientCertFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/matrix-token.pem"; - description = '' - Certificate the unit that fetches this hive's matrix appservice token - presents to the store. Its subject must be - `-`; cert auth matches on - the CN and the role accepts nothing else. - - ⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`. - Pointing this at the hive's own leaf would evaluate, deploy and log in — - and hand a unit that reads one token a credential that reads every - secret in the store. That is what this option exists to end. - - No default here: ./glue-bao-tls.nix points it at the leaf it mints, - wherever this host mints its own PKI. A deployment that reads a store on - another host names it by hand, alongside the leaf itself — one file per - principal placed out of band instead of one shared by four. - ''; - }; - - matrixTokenClientKeyFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/matrix-token-key.pem"; - description = '' - Private key for - {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both - or neither — a certificate with no key authenticates nothing. - ''; - }; - - queueAgentClientCertFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/queue-agent.pem"; - description = '' - Certificate the unit that fetches this hive's queue client credential - presents to the store. Its subject must be - `-`. - - ⚠️ Not the hive's own leaf, for the reason - {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. - ''; - }; - - queueAgentClientKeyFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/queue-agent-key.pem"; - description = '' - Private key for - {option}`services.hyperhive.deploy.bao.queueAgentClientCertFile`. - ''; - }; - - grafanaOidcClientCertFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/grafana-oidc.pem"; - description = '' - Certificate the unit that fetches Grafana's OIDC client secret presents - to the store. Its subject must be - {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`. - - ⚠️ Not the hive's own leaf, for the reason - {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. - ''; - }; - - grafanaOidcClientKeyFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/grafana-oidc-key.pem"; - description = '' - Private key for - {option}`services.hyperhive.deploy.bao.grafanaOidcClientCertFile`. - ''; - }; - - otelOidcClientCertFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/otel-oidc.pem"; - description = '' - Certificate the unit that fetches the collector's OIDC client secret - presents to the store. Its subject must be - {option}`services.hyperhive.deploy.bao.otelOidcCommonName`. - - ⚠️ Not the hive's own leaf, for the reason - {option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. - ''; - }; - - otelOidcClientKeyFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "/var/lib/swarm-bao-pki/otel-oidc-key.pem"; - description = '' - Private key for - {option}`services.hyperhive.deploy.bao.otelOidcClientCertFile`. - ''; - }; - serverCaFile = lib.mkOption { type = lib.types.nullOr lib.types.str; default = null; @@ -1922,19 +1593,6 @@ in ''; }; - # 🩸 The four principals that used to log in as the hive. See the block - # around `perHiveReaders` for what each grant is and why the first two are - # per hive; the four units are identical apart from the objects they - # write, so the shape lives in `readerPolicyUnit` and the difference is - # visible here in one line each. - systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders; - - systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders; - - systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders; - - systemd.services.swarm-bao-otel-oidc-policy = readerPolicyUnit "write the collector's OIDC-secret-reader bao policy and cert-auth role" otelOidcReaders; - # The CA bind source is written at runtime by a host unit, so the # container has to start after it — otherwise nspawn sets up a mount # over a file that does not exist yet. diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 85e5a2d1..006e3abf 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -119,14 +119,7 @@ let # other secret is read with it. A Grafana host without it has not been given # its identity yet, which is a thing to say out loud rather than to route # around by reaching into authelia's tree whenever it happens to be local. - # - # 🩸 Grafana's OWN leaf, not `clientCertFile` — the hive's, which four units - # used to share. Bao matches a cert-auth role on the CN, so one leaf for four - # readers was ONE principal holding the union of four grants: this unit could - # read every agent credential in the swarm and every other service's OIDC - # client secret, when what it needs is the one path `storeSecretPath` names. - haveClientIdentity = - baoDeploy.grafanaOidcClientCertFile != null && baoDeploy.grafanaOidcClientKeyFile != null; + haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; autheliaUrl = toString hyperhiveCfg.swarm.authelia.url; @@ -458,20 +451,15 @@ in services.hyperhive.deploy.grafana.enable requires this host to hold a swarm-secret-store client identity: set both - services.hyperhive.deploy.bao.grafanaOidcClientCertFile - services.hyperhive.deploy.bao.grafanaOidcClientKeyFile + services.hyperhive.deploy.bao.clientCertFile + services.hyperhive.deploy.bao.clientKeyFile Grafana's OIDC client secret is minted by authelia and read out of the store, on every host that runs Grafana — including the host that runs authelia. That is one delivery route rather than two, and it is - what the store is for: a certificate is the credential placed out of - band, and every other secret comes from the store with it. - - ⚠️ Grafana's OWN leaf, not deploy.bao.clientCertFile. That one is the - hive's, and its grant reads every secret in the store; this role - reads the one path Grafana's client secret lives at. Pointing this - option at the hive's leaf would evaluate, deploy and log in — and - undo the split. + what the store is for: this certificate is the single credential + placed out of band, and every other secret comes from the store with + it. On a hive that runs the store, glue-bao-tls.nix supplies both as defaults and there is nothing to do. Elsewhere the leaf is issued @@ -622,8 +610,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.grafanaOidcClientCertFile; - BAO_CLIENT_KEY = baoDeploy.grafanaOidcClientKeyFile; + BAO_CLIENT_CERT = baoDeploy.clientCertFile; + BAO_CLIENT_KEY = baoDeploy.clientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index c53bfd21..868df8b7 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -242,32 +242,13 @@ let # A reader of the store is defined by holding a certificate the store # accepts, never by standing next to it — the rule # ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical- - # looking flag, this one does not gate an assertion on its own: the - # store-reading unit below simply does not render without it, the same choice + # looking flag, this one stays outside `assertions`: the store-reading unit + # below simply does not render without it, the same choice # ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for - # their own readers, because a collector with no client identity is + # their own optional readers, because a collector with no client identity is # `haveCollectorSecret = false` above, and that is already a supported, - # merely degraded shape rather than a service with no way in at all. What - # IS asserted is narrower and lives in `assertions` below — see - # `hiveReaderIdentity`. - # - # 🩸 The collector's OWN leaf, not `clientCertFile` — the hive's, which four - # units used to share. Bao matches a cert-auth role on the CN, so one leaf for - # four readers was ONE principal holding the union of four grants: this unit - # could read every agent credential in the swarm and Grafana's OIDC client - # secret, when what it needs is the one path `storeSecretPath` names. - haveClientIdentity = - baoDeploy.otelOidcClientCertFile != null && baoDeploy.otelOidcClientKeyFile != null; - - # Does this host read the store at all — the hive's own leaf, which is the - # one thing a remote-store deployment has always had to place by hand. It is - # what separates the degrade the comment above describes from the mistake the - # assertion below reports: a collector on a host holding NO store identity is - # the supported shape, and one on a host that demonstrably reads the store - # named seven of the eight options and stopped. Before the four-way split - # that second host rendered this unit off `clientCertFile`, so it is a silent - # regression rather than a choice anyone made. - hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; + # merely degraded shape rather than a service with no way in at all. + haveClientIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; # Where the publisher on authelia's host leaves this client's secret — # composed from the same swarm-wide `clientId` the registration in @@ -744,14 +725,11 @@ in # value. # # ⚠️ Renders only where `haveClientIdentity` holds, unlike - # ./swarm-grafana.nix's equivalent unit. That module refuses any host that - # runs Grafana without the identity, because a Grafana with none has no way - # in at all; this collector without one is `haveCollectorSecret = false` - # above — already a supported, merely degraded shape, so the unit that would - # fetch a credential simply does not exist rather than refusing the build - # for want of one. The one case that IS refused is the host that already - # holds the hive's leaf and is missing only this pair, which is a silent - # regression rather than that degrade — `hiveReaderIdentity` above. + # ./swarm-grafana.nix's equivalent unit. That module asserts the identity + # because a Grafana with none has no way in at all; this collector without + # one is `haveCollectorSecret = false` above — already a supported, + # merely degraded shape, so the unit that would fetch a credential simply + # does not exist rather than refusing the build for want of one. systemd.services.swarm-bao-otel-oidc = lib.mkIf haveClientIdentity { description = "fetch the swarm collector's OIDC client secret from the swarm secret store"; # Every one of these names a unit that exists only where the store runs. @@ -797,8 +775,8 @@ in }; environment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - BAO_CLIENT_CERT = baoDeploy.otelOidcClientCertFile; - BAO_CLIENT_KEY = baoDeploy.otelOidcClientKeyFile; + BAO_CLIENT_CERT = baoDeploy.clientCertFile; + BAO_CLIENT_KEY = baoDeploy.clientKeyFile; } # Absent means the system trust store, which is what a deployment with a # real CA wants and what a self-signed one must not be left with. @@ -875,45 +853,6 @@ in systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; assertions = [ - { - # Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — - # the same refusal, named to this principal's own pair. What differs is - # the `!hiveReaderIdentity ||` guard, and it is what keeps the degrade - # the flag's own comment describes intact: Grafana refuses any host - # that runs it without a leaf, because a Grafana with no SSO has no way - # in at all, while a collector with none still receives telemetry. So - # this fires only where the host already reads the store and is missing - # this one pair — which before the four-way split rendered the unit off - # `clientCertFile`, and now silently does not. - assertion = !hiveReaderIdentity || haveClientIdentity; - message = '' - This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile - is set) and runs the swarm collector, so it needs the collector's own - client identity: set both - - services.hyperhive.deploy.bao.otelOidcClientCertFile - services.hyperhive.deploy.bao.otelOidcClientKeyFile - - swarm-bao-otel-oidc.service fetches the collector's OIDC client - secret out of the store, and without these it is not rendered at all - — so this collector would authenticate to nothing, quietly, on a host - that has everything it needs to fetch the secret. - - A collector on a host holding NO store identity is a different and - supported shape: it runs without a client secret and still receives - telemetry. That is not this host. - - ⚠️ The collector's OWN leaf, not deploy.bao.clientCertFile. That one - is the hive's, and its grant reads every secret in the store; this - role reads the one path this collector's client secret lives at. - Pointing this option at the hive's leaf would evaluate, deploy and - log in — and undo the split. - - On a hive that runs the store, glue-bao-tls.nix supplies both as - defaults and there is nothing to do. Elsewhere the leaf is issued - from that CA out of band and named here — see docs/swarm/secrets.md. - ''; - } # ⚠️ An assertion that this collector has "somewhere to send" was REMOVED # rather than relaxed: it read the store's *per-host* enable, so it # rejected at eval the very deployment the stores are reached by domain diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 213aa382..d0e36dcd 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -47,20 +47,7 @@ let deployCfg.bao.controllerCommonName deployCfg.bao.secretPublisherCommonName deployCfg.bao.matrixCtlCommonName - deployCfg.bao.grafanaOidcCommonName - deployCfg.bao.otelOidcCommonName - ] - # The two per-hive readers' subjects, spelled out per hive rather than as the - # prefix. The prefix alone would reserve the wrong string: the role for hive - # `h` accepts `-h`, so the name a hive must not BE is that composed - # spelling, and a hive named it would present a leaf that role accepts. - # - # Both spellings are reserved for every declared hive, including the one the - # CN belongs to — a hive cannot be named after its own reader either. - ++ lib.concatMap (hive: [ - "${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}" - "${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}" - ]) hiveNames; + ]; # Public hostnames of the swarm's own services, in declaration order. # `serviceDomains` below is this set sorted + deduplicated. diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index c03e0fcf..2068620c 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -45,18 +45,6 @@ let deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; deploy.bao.clientCaFile = lib.mkForce null; }; - # The store plus every one of the four readers that used to log in as the - # hive. One fixture rather than four: the claim they are four *separate* - # principals is only testable where all four render at once — that is the - # deployment in which two of them sharing a leaf would be invisible. - baoGrantWithConsumers = hive { - deploy.bao.enable = true; - deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token"; - deploy.matrix.enable = true; - deploy.grafana.enable = true; - deploy.swarm-otel.enable = true; - }; - cases = [ { # Reads the rendered unit on the HOST, which is where the write happens: @@ -208,250 +196,6 @@ let ) && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? swarm-bao-matrix-ctl-policy); } - - # ── the four readers that used to share the hive's own leaf ────────────── - # - # 🩸 Until this split all four presented `deploy.bao.clientCertFile`, whose - # policy grants read on `swarm/agents/*`, `swarm/hives//*` AND - # `swarm/services/*`. Four principals behind one certificate are one - # principal to bao, so the only expressible grant was the union: the unit - # fetching Grafana's OIDC secret could fetch every agent credential in the - # swarm. - # - # Every one of these cases carries the same three negative arms, and they - # are the deliverable rather than decoration — a positive arm alone passes - # just as well when the other two stanzas are still there beside it. The - # arms pin what each principal must NOT reach, so a later widening fails - # here instead of being noticed in a store. - { - name = "the matrix-token reader's grant is one hive's appservice token and nothing else"; - ok = - let - s = baoGrantHere.systemd.services.swarm-bao-matrix-token-policy.script; - in - lib.hasInfix "path \"secret/data/swarm/hives/h1/matrix/appservice-token\" {" s - && lib.hasInfix "capabilities = [\"read\"]" s - # The three stanzas the hive's own leaf carried, none of which this - # principal needs: every agent's credential, every service's OIDC - # client, and the rest of its own hive's tree — including the queue - # credential its sibling reader fetches. - && !(lib.hasInfix "secret/data/swarm/agents" s) - && !(lib.hasInfix "secret/data/swarm/services" s) - && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) - && !(lib.hasInfix "secret/data/swarm/hives/h1/queue" s) - # A `hives/*` wildcard would serve every hive from one role and let any - # hive read any other's token — the reach this split exists to remove, - # not to create. - && !(lib.hasInfix "secret/data/swarm/hives/*" s) - # Nothing may rewrite the policy constraining it, for the reason the - # controller's own `hive-*` narrowing above gives. - && !(lib.hasInfix "sys/policies/acl" s); - } - { - name = "the queue-credential reader's grant is one hive's queue credential and nothing else"; - ok = - let - s = baoGrantHere.systemd.services.swarm-bao-queue-agent-policy.script; - in - lib.hasInfix "path \"secret/data/swarm/hives/h1/queue/agent\" {" s - && lib.hasInfix "capabilities = [\"read\"]" s - && !(lib.hasInfix "secret/data/swarm/agents" s) - && !(lib.hasInfix "secret/data/swarm/services" s) - && !(lib.hasInfix "secret/data/swarm/hives/h1/*" s) - && !(lib.hasInfix "secret/data/swarm/hives/h1/matrix" s) - && !(lib.hasInfix "secret/data/swarm/hives/*" s) - && !(lib.hasInfix "sys/policies/acl" s); - } - { - # ⚠️ The client id is the path segment, so the negative arm that matters - # for this one is the OTHER service's: `services/*` would have granted - # both, and the two are separate principals precisely because a - # dashboard is not entitled to a collector's credential. - name = "the Grafana OIDC reader's grant is Grafana's own client secret and nothing else"; - ok = - let - s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script; - in - lib.hasInfix "path \"secret/data/swarm/services/swarm-grafana/oidc/client\" {" s - && lib.hasInfix "capabilities = [\"read\"]" s - && !(lib.hasInfix "secret/data/swarm/agents" s) - && !(lib.hasInfix "secret/data/swarm/hives" s) - && !(lib.hasInfix "secret/data/swarm/services/*" s) - && !(lib.hasInfix "swarm-collector" s) - && !(lib.hasInfix "sys/policies/acl" s); - } - { - # The mirror of the case above, and the arm naming `swarm-grafana` is why - # these are two principals rather than one `services/*` grant shared. - name = "the collector OIDC reader's grant is the collector's own client secret and nothing else"; - ok = - let - s = baoGrantHere.systemd.services.swarm-bao-otel-oidc-policy.script; - in - lib.hasInfix "path \"secret/data/swarm/services/swarm-collector/oidc/client\" {" s - && lib.hasInfix "capabilities = [\"read\"]" s - && !(lib.hasInfix "secret/data/swarm/agents" s) - && !(lib.hasInfix "secret/data/swarm/hives" s) - && !(lib.hasInfix "secret/data/swarm/services/*" s) - && !(lib.hasInfix "swarm-grafana" s) - && !(lib.hasInfix "sys/policies/acl" s); - } - { - # 🩸 The half that makes the policies above bind: a policy grants only - # through a token that carries it, and a token is minted by a cert-auth - # role matching a CN. Four distinct subjects is the whole mechanism — one - # subject for four readers is one principal however the policies read. - # - # The per-hive subjects carry the hive name because their paths do; the - # two service subjects do not, because an OIDC client is registered once - # per swarm. Pinned so neither shape is tidied into the other. - name = "each of the four readers logs in under a subject of its own"; - ok = - let - subjectOf = - unit: role: cn: - let - s = baoGrantHere.systemd.services.${unit}.script; - in - lib.hasInfix "auth/cert/certs/${role}" s - && lib.hasInfix "allowed_common_names=${cn}" s - && lib.hasInfix "token_policies=${role}" s - # Outside the `hive-*` namespace the controller may rewrite, for - # the reason the three service principals above state. - && !(lib.hasInfix "auth/cert/certs/hive-" s); - in - subjectOf "swarm-bao-matrix-token-policy" "swarm-matrix-token-h1" "swarm-bao-matrix-token-h1" - && subjectOf "swarm-bao-queue-agent-policy" "swarm-queue-agent-h1" "swarm-bao-queue-agent-h1" - && subjectOf "swarm-bao-grafana-oidc-policy" "swarm-grafana-oidc" "swarm-bao-grafana-oidc" - && subjectOf "swarm-bao-otel-oidc-policy" "swarm-otel-oidc" "swarm-bao-otel-oidc"; - } - { - # 🩸 The consuming side, and the arm that would catch the regression that - # costs the most: a unit repointed back at `deploy.bao.clientCertFile` - # evaluates, deploys and logs in — and silently restores the union grant, - # because bao would again see one principal. Nothing about the policies - # above would look wrong. - # - # Each pair is asserted whole: a certificate with no key authenticates - # nothing, so a half-set pair is a reader that does not render. - name = "each of the four readers presents its own leaf, never the hive's"; - ok = - let - b = baoGrantWithConsumers.services.hyperhive.deploy.bao; - hiveLeaf = [ - b.clientCertFile - b.clientKeyFile - ]; - own = [ - b.matrixTokenClientCertFile - b.matrixTokenClientKeyFile - b.queueAgentClientCertFile - b.queueAgentClientKeyFile - b.grafanaOidcClientCertFile - b.grafanaOidcClientKeyFile - b.otelOidcClientCertFile - b.otelOidcClientKeyFile - ]; - envOf = unit: baoGrantWithConsumers.systemd.services.${unit}.environment; - presents = - unit: cert: key: - (envOf unit).BAO_CLIENT_CERT == cert && (envOf unit).BAO_CLIENT_KEY == key; - in - lib.all (p: p != null) own - && !(lib.any (p: lib.elem p hiveLeaf) own) - && lib.length (lib.unique own) == lib.length own - && presents "swarm-bao-matrix-token" b.matrixTokenClientCertFile b.matrixTokenClientKeyFile - && presents "swarm-bao-queue-agent" b.queueAgentClientCertFile b.queueAgentClientKeyFile - && presents "swarm-bao-grafana-oidc" b.grafanaOidcClientCertFile b.grafanaOidcClientKeyFile - && presents "swarm-bao-otel-oidc" b.otelOidcClientCertFile b.otelOidcClientKeyFile; - } - { - # The minting side of the same claim. A role matching a subject nothing - # signs is a reader that cannot log in, so the leaves and the roles have - # to be asserted against each other — and the two per-hive leaves carry - # THIS host's hive name, which is what makes one hive's leaf useless - # against another hive's role. - name = "the PKI unit signs a leaf per reader, each under that reader's own subject"; - ok = - let - s = baoGrantHere.systemd.services.swarm-bao-pki.script; - in - # The basename and the subject are matched separately: `signLeaf` takes - # them as consecutive arguments across a `\` continuation, so one - # literal spanning both would pin this file's line wrapping rather than - # the pairing it means to. - lib.all (lib.flip lib.hasInfix s) [ - "/matrix-token.pem ]" - "swarm-bao-matrix-token-h1 \"\" clientAuth" - "/queue-agent.pem ]" - "swarm-bao-queue-agent-h1 \"\" clientAuth" - "/grafana-oidc.pem ]" - "swarm-bao-grafana-oidc \"\" clientAuth" - "/otel-oidc.pem ]" - "swarm-bao-otel-oidc \"\" clientAuth" - ]; - } - { - # The absence arm: with no client CA there is no trust anchor, so the - # login roles cannot be written — but the policies they would attach are - # still asserted, exactly as the three service principals above behave in - # this deployment. A unit that vanished here would take the policy with - # it and leave nothing to diagnose. - name = "with no client CA the four readers get policies but no login roles"; - ok = - let - units = [ - "swarm-bao-matrix-token-policy" - "swarm-bao-queue-agent-policy" - "swarm-bao-grafana-oidc-policy" - "swarm-bao-otel-oidc-policy" - ]; - scriptOf = unit: baoGrantNoClientCa.systemd.services.${unit}.script; - in - lib.all ( - unit: - (baoGrantNoClientCa.systemd.services ? ${unit}) - && lib.hasInfix "bao policy write" (scriptOf unit) - && !(lib.hasInfix "auth/cert/certs" (scriptOf unit)) - ) units; - } - { - # Same control the three service principals carry: the write needs a - # client certificate and the host is the side that has one, so a unit - # rendered inside the store's container would have neither an identity - # nor a route. Plus the ordering that makes the mounts exist first. - name = "the four readers' granting units are ordered after the mounts and rendered on the host"; - ok = - let - units = [ - "swarm-bao-matrix-token-policy" - "swarm-bao-queue-agent-policy" - "swarm-bao-grafana-oidc-policy" - "swarm-bao-otel-oidc-policy" - ]; - in - lib.all ( - unit: - lib.elem "swarm-bao-controller-policy.service" baoGrantHere.systemd.services.${unit}.after - && !(baoGrantHere.containers.swarm-bao.config.systemd.services ? ${unit}) - ) units; - } - { - # A store host that has not placed a bootstrap token can write no grant at - # all, so none of the four units may exist — the same claim - # `baoGrantNoStore` makes for the controller's, one file over. Without - # this arm `lib.mkIf haveBootstrapToken` could be dropped from the shared - # builder and every other case here would still pass. - name = "without a bootstrap token none of the four readers' granting units render"; - ok = - let - s = baoGrantNoStore.systemd.services; - in - !(s ? swarm-bao-matrix-token-policy) - && !(s ? swarm-bao-queue-agent-policy) - && !(s ? swarm-bao-grafana-oidc-policy) - && !(s ? swarm-bao-otel-oidc-policy); - } { # The policy authorising this route lives in another file, and nothing # else relates the grants to the paths the code actually writes. diff --git a/nix/module-eval/bao-matrix-reader.nix b/nix/module-eval/bao-matrix-reader.nix index 19a0da5f..00f2256b 100644 --- a/nix/module-eval/bao-matrix-reader.nix +++ b/nix/module-eval/bao-matrix-reader.nix @@ -32,20 +32,10 @@ let # nothing here mints a leaf and the operator names one placed by hand. The # deployment this pairing exists to serve, and the one that was previously # inexpressible — the gate asked whether the store was a neighbour. - # - # 🩸 One leaf PER READER, and this fixture is where that cost is visible: the - # two units below identify themselves to the store separately, so an operator - # placing leaves by hand places one for each rather than one for both. Naming - # only `clientCertFile` here would leave neither unit rendered — which is what - # the arms below would then be asserting about. baoRemoteReader = hive { deploy.matrix.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.matrixTokenClientCertFile = "/etc/pki/bao-matrix-token.pem"; - deploy.bao.matrixTokenClientKeyFile = "/etc/pki/bao-matrix-token-key.pem"; - deploy.bao.queueAgentClientCertFile = "/etc/pki/bao-queue-agent.pem"; - deploy.bao.queueAgentClientKeyFile = "/etc/pki/bao-queue-agent-key.pem"; }; # A homeserver on a hive with NO store identity at all — neither a local @@ -53,38 +43,6 @@ let # needs it, and defining it here rather than importing keeps each group's # fixture set its own, as ./lib.nix asks. matrixNoBaoIdentity = hive { deploy.matrix.enable = true; }; - - # 🩸 The hand-configured remote reader that named seven of the eight options. - # `baoRemoteReader` above is the same deployment done right; this one holds - # the hive's own leaf, so it demonstrably reads the store, and is missing both - # per-principal pairs. Before the four-way split this host rendered both units - # off `clientCertFile` alone, so what it has now is a silent regression rather - # than any shape an operator chose — which is what the refusal arms below are - # about. Kept as one fixture rather than two because both refusals fire on it - # and each arm names which. - baoRemoteReaderMissingLeaves = hive { - deploy.matrix.enable = true; - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - }; - - # The same omission on a hive that does NOT run a homeserver. Separates the - # matrix refusal's `deploy.matrix.enable` clause from the queue refusal, which - # has no toggle to check — an arm below reads exactly one refusal off it. - remoteReaderNoMatrixMissingLeaves = hive { - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - }; - - # Did a module refuse this host, and over which option. An assertion is a - # config VALUE until something forces it — `.config` never throws — so a - # fixture in a state the module refuses stays evaluable and the refusal reads - # back as data. Matched on the option name the message names rather than on - # its prose, because the option name is the part an operator has to act on - # and a message that stopped naming it would be the actual defect. Same shape - # as ./grafana.nix's `grafanaRefusedFor`, which reads `swarm-grafana.nix`'s - # own refusal. - refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; cases = [ { # A login failure is the store being unreachable, sealed, or not yet @@ -405,63 +363,6 @@ let in !(g ? admin_execute) || g.admin_execute == [ ]; } - { - # 🩸 The arm the whole refusal exists for. Both readers are gated on their - # own leaf, so the way this regresses is the build going green and three - # units rendering where four should — which no presence check on a - # rendered unit can see, because the unit that is missing is the evidence. - # Read as a refusal naming each option, so an operator acts on the message - # without opening the nix. - name = "a remote reader missing the per-principal leaves is refused, naming both options"; - ok = - refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientCertFile" - && refusedOver baoRemoteReaderMissingLeaves "matrixTokenClientKeyFile" - && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientCertFile" - && refusedOver baoRemoteReaderMissingLeaves "queueAgentClientKeyFile"; - } - { - # The matrix refusal's own gate, which the queue refusal does not have. - # Without this arm the two are indistinguishable on the fixture above. - name = "the queue refusal needs no homeserver, and the matrix refusal stays quiet without one"; - ok = - refusedOver remoteReaderNoMatrixMissingLeaves "queueAgentClientCertFile" - && !(refusedOver remoteReaderNoMatrixMissingLeaves "matrixTokenClientCertFile"); - } - { - # ⚠️ The arm that keeps the refusal from being worse than the silence it - # replaced. A hive with NO store identity is the supported no-store - # deployment and also the state an operator passes through bringing a hive - # up — neither may fail to evaluate. Asserted as "no refusal names any of - # the four options", not as "this one fixture is fine", because the way - # this breaks is a gate widened to the principal's leaf alone. - name = "a hive with no store identity at all is refused over none of the per-principal leaves"; - ok = lib.all (option: !(refusedOver matrixNoBaoIdentity option)) [ - "matrixTokenClientCertFile" - "matrixTokenClientKeyFile" - "queueAgentClientCertFile" - "queueAgentClientKeyFile" - ]; - } - { - # The other half of the same guard, and the one an operator meets far more - # often: on the store's own host ./host-modules/glue-bao-tls.nix mkDefaults - # all eight, so there is nothing to name and nothing to refuse. Paired with - # the fully-named remote reader, which is the same deployment done by hand. - name = "neither a store host nor a correctly-named remote reader is refused"; - ok = - lib.all - ( - m: - lib.all (option: !(refusedOver m option)) [ - "matrixTokenClientCertFile" - "queueAgentClientCertFile" - ] - ) - [ - baoWithMatrix - baoRemoteReader - ]; - } ]; in runGroup "bao-matrix-reader" cases diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index fe3cd97e..ae825800 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -41,8 +41,6 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; - deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # The same UI with the IdP on ANOTHER host and a store leaf placed by hand. @@ -56,8 +54,6 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.grafanaOidcClientCertFile = "/etc/pki/bao-grafana-oidc.pem"; - deploy.bao.grafanaOidcClientKeyFile = "/etc/pki/bao-grafana-oidc-key.pem"; }; # A Grafana host holding no store identity. This used to be the shape the @@ -152,8 +148,8 @@ let # mean the two assertions had been collapsed into one conjunction. name = "a grafana host with no store identity is refused, naming the options to set"; ok = - grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientCertFile" - && grafanaRefusedFor grafanaNoIdentity "deploy.bao.grafanaOidcClientKeyFile" + grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientCertFile" + && grafanaRefusedFor grafanaNoIdentity "deploy.bao.clientKeyFile" && !(grafanaRefusedFor grafanaNoIdentity "swarm.authelia.url"); } { @@ -165,9 +161,9 @@ let name = "neither grafana refusal fires on a correctly configured host, co-located or not"; ok = !(grafanaRefusedFor grafanaWithAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.grafanaOidcClientCertFile") + && !(grafanaRefusedFor grafanaWithAuthelia "deploy.bao.clientCertFile") && !(grafanaRefusedFor grafanaRemoteAuthelia "services.hyperhive.swarm.authelia.url") - && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.grafanaOidcClientCertFile"); + && !(grafanaRefusedFor grafanaRemoteAuthelia "deploy.bao.clientCertFile"); } { # Same 403-not-a-miss reason as the matrix and queue arms below: the diff --git a/nix/module-eval/name-guards.nix b/nix/module-eval/name-guards.nix index e035b0f0..8716f980 100644 --- a/nix/module-eval/name-guards.nix +++ b/nix/module-eval/name-guards.nix @@ -63,35 +63,6 @@ let swarm.hives.mintctl.domain = "m.t.local"; }; - # The two OIDC-secret readers' subjects, fixed strings like the three above. - hiveNamedAfterGrafanaOidcSubject = hive { - deploy.swarm-otel.enable = false; - deploy.bao.grafanaOidcCommonName = "gfctl"; - swarm.hives.gfctl.domain = "g.t.local"; - }; - - hiveNamedAfterOtelOidcSubject = hive { - deploy.swarm-otel.enable = false; - deploy.bao.otelOidcCommonName = "otctl"; - swarm.hives.otctl.domain = "o.t.local"; - }; - - # 🩸 A different shape from every fixture above: the matrix-token and - # queue-credential roles are written PER HIVE, so the subject a hive must not - # be is `-` rather than the prefix itself. Reserving - # only the prefix would leave the composed spelling free, and a hive taking it - # would present a leaf the other hive's role accepts — which is a hive reading - # another hive's queue credential, the exact widening the split exists to - # avoid. - # - # Two hives here, not one: the collision is with the OTHER hive's role. - hiveNamedAfterPerHiveReaderSubject = hive { - deploy.swarm-otel.enable = false; - deploy.bao.queueAgentCommonNamePrefix = "qr"; - swarm.hives.other.domain = "o.t.local"; - swarm.hives.qr-other.domain = "q.t.local"; - }; - hiveNameWithComposedWord = hive { deploy.swarm-otel.enable = false; swarm.hives."h1-agent".domain = "a.t.local"; @@ -140,36 +111,6 @@ let a: !a.assertion && lib.hasInfix "'mintctl'" a.message ) hiveNamedAfterMatrixCtlSubject.assertions; } - { - # The fourth and fifth, for the reason the case above gives: `certAuthCns` - # is where a role added beside the others registers itself, and nothing - # but a case per element notices when one forgets. These two are the - # subjects of the readers that fetch Grafana's and the collector's OIDC - # client secrets. - name = "a hive named after either OIDC-secret reader's subject is refused too"; - ok = - equalityGuardFired hiveNamedAfterGrafanaOidcSubject - && lib.any ( - a: !a.assertion && lib.hasInfix "'gfctl'" a.message - ) hiveNamedAfterGrafanaOidcSubject.assertions - && equalityGuardFired hiveNamedAfterOtelOidcSubject - && lib.any ( - a: !a.assertion && lib.hasInfix "'otctl'" a.message - ) hiveNamedAfterOtelOidcSubject.assertions; - } - { - # 🩸 The per-hive half, and the one a prefix-only reservation would miss: - # the role is `-`, so the reserved string has to be composed - # against every declared hive. Here hive `qr-other` collides with the role - # written for hive `other` — a leaf that reads a credential belonging to a - # hive that is not it. - name = "a hive named after another hive's per-hive reader subject is refused"; - ok = - equalityGuardFired hiveNamedAfterPerHiveReaderSubject - && lib.any ( - a: !a.assertion && lib.hasInfix "'qr-other'" a.message - ) hiveNamedAfterPerHiveReaderSubject.assertions; - } { # Without this the case above proves nothing: an arm that fires for every # roster is not a guard, and `hives` is non-empty in both fixtures. diff --git a/nix/module-eval/swarm-otel-core.nix b/nix/module-eval/swarm-otel-core.nix index 0bbfa10e..7428919c 100644 --- a/nix/module-eval/swarm-otel-core.nix +++ b/nix/module-eval/swarm-otel-core.nix @@ -32,8 +32,6 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; - deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; deploy.victoriametrics.enable = false; deploy.victorialogs.enable = false; }; @@ -46,8 +44,6 @@ let deploy.authelia.enable = true; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; - deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; # The same collector with the IdP on ANOTHER host and a store leaf placed by @@ -58,8 +54,6 @@ let swarm.authelia.url = "https://auth.example.invalid"; deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; - deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; }; cases = [ { diff --git a/nix/module-eval/swarm-otel-identity.nix b/nix/module-eval/swarm-otel-identity.nix index 99dcd677..67c89357 100644 --- a/nix/module-eval/swarm-otel-identity.nix +++ b/nix/module-eval/swarm-otel-identity.nix @@ -34,40 +34,6 @@ let deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; }; - # 🩸 The shape the degrade above must not swallow: a collector on a host that - # demonstrably READS the store — it holds the hive's own leaf — and is missing - # only the collector's own pair. Before the four-way split this host rendered - # the reading unit off `clientCertFile` alone, so the silence it gets now is a - # regression rather than the supported degrade `otelNoIdentity` stands for. The - # two fixtures differ in exactly that one pair, which is what lets the arms - # below separate a refusal from a degrade. - otelReaderMissingLeaf = hive { - deploy.swarm-otel.enable = true; - swarm.authelia.url = "https://auth.example.invalid"; - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - }; - - # The same deployment named in full, which must stay buildable. Pairs with the - # fixture above so the refusal is pinned to the omission and not to reading a - # remote store at all. - otelReaderNamedInFull = hive { - deploy.swarm-otel.enable = true; - swarm.authelia.url = "https://auth.example.invalid"; - deploy.bao.clientCertFile = "/etc/pki/bao-client.pem"; - deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem"; - deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem"; - deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem"; - }; - - # Did the module refuse this host, and over which option. An assertion is a - # config VALUE until something forces it — `.config` never throws — so a - # fixture in a refused state stays evaluable and the refusal reads back as - # data. Matched on the option name the message names rather than on its prose: - # the option name is the part an operator has to act on, and a message that - # stopped naming it would be the actual defect. - refusedOver = m: option: lib.any (a: !a.assertion && lib.hasInfix option a.message) m.assertions; - # authelia somewhere else, the credential delivered by hand. Whether this # collector authenticates must follow the credential, never another # service's placement. @@ -196,38 +162,6 @@ let && !(builtins.elem "otlphttp/victoriametrics" used) && lib.all (e: s.exporters ? ${e}) used; } - { - # 🩸 The refusal, and the only way this module's omission is visible at - # all: the reading unit is gated on its own leaf, so the regression is a - # green build with the unit absent — and the missing unit is the evidence. - # Read as a refusal naming both options, so an operator acts on the - # message without opening the nix. - name = "a store reader missing the collector's own leaf is refused, naming both options"; - ok = - refusedOver otelReaderMissingLeaf "otelOidcClientCertFile" - && refusedOver otelReaderMissingLeaf "otelOidcClientKeyFile"; - } - { - # ⚠️ The arm that keeps the refusal from eating the degrade beside it. A - # collector on a host with NO store identity still receives every hive's - # telemetry and is a supported deployment — `otelNoIdentity` is that - # fixture, and it differs from the refused one only in the hive's own - # leaf. Widening the gate to the collector's leaf alone would reject it, - # which is what this pins. - name = "a collector with no store identity at all is not refused over the collector's leaf"; - ok = - !(refusedOver otelNoIdentity "otelOidcClientCertFile") - && !(refusedOver otelNoIdentity "otelOidcClientKeyFile"); - } - { - # The other two shapes that must stay buildable: the same remote reader - # named in full, and a host that mints its own PKI and therefore has all - # eight as defaults from ./host-modules/glue-bao-tls.nix. - name = "neither a fully-named remote reader nor a store host is refused"; - ok = - !(refusedOver otelReaderNamedInFull "otelOidcClientCertFile") - && !(refusedOver otelRemoteAuthelia "otelOidcClientCertFile"); - } ]; in runGroup "swarm-otel-identity" cases