diff --git a/docs/integrations/matrix.md b/docs/integrations/matrix.md index 849e917d..497a1103 100644 --- a/docs/integrations/matrix.md +++ b/docs/integrations/matrix.md @@ -192,8 +192,12 @@ restarts, so the first boot after the switch already has both halves. sets it fails to evaluate with a message naming the appservice — a hive that never set it (the default) is unaffected. - **A swarm store holding the old `matrix/registration-token` path** is - no longer read at all; the hive uses its locally minted appservice - token until someone `put`s a value at `matrix/appservice-token`. + no longer read at all. The value that matters now lives at + `matrix/appservice-token`, and `swarm-secret-publish` on the authelia + host mints and `put`s it there — the hive uses its locally minted + token only until the first successful read. See + [`../swarm/secrets.md`](../swarm/secrets.md) for how that mint stays + idempotent across runs. diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 79c1bedb..7c17abe4 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -132,14 +132,14 @@ when the system builds. The server names the offending file and refuses to run. ## Hive-level — one of each per hive -| secret | generated by | lives at | -| ---------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -| hive CA cert + key | `hive-tls.nix` first-boot unit | `/ca.pem`, `ca-key.pem` (`0600`) | -| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `/.pem` | -| matrix appservice token | a host activation script, on first boot | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) | -| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/.secret` inside the forge container | -| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/.secret`, handed to tuwunel through `LoadCredential` | -| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `/secret` (`0600`) and `/client_id` (`0644`) | +| secret | generated by | lives at | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | +| hive CA cert + key | `hive-tls.nix` first-boot unit | `/ca.pem`, `ca-key.pem` (`0600`) | +| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `/.pem` | +| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) | +| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/.secret` inside the forge container | +| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/.secret`, handed to tuwunel through `LoadCredential` | +| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `/secret` (`0600`) and `/client_id` (`0644`) | Both delivery units wait for authelia's first boot to mint the secret — a bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip @@ -162,10 +162,32 @@ nobody. Every failure path — no such key, sealed store, unreachable store, empty value — leaves the locally minted token in place, so a hive with no store behaves exactly as it did before. -The store path is `swarm/hives//matrix/appservice-token`. It was -`…/matrix/registration-token` while the homeserver still took a shared -registration secret; a value left at the old path is read by nothing, and the -hive falls back to its local token until someone `put`s the new one. +The store path is `swarm/hives//matrix/appservice-token`, and the +**producer is `swarm-secret-publish`** — the same unit that copies authelia's +OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s +it there, so the store is the source of truth and every hive converges on the +value it holds. The hive's own activation mint is still there, but it's a +first-boot fallback now: it fires only when the token file is absent, and the +reader overwrites whatever it produced. + +The mint is **idempotent by keeping its own record**, not by asking the store. +The publisher's grant is `create`/`update` under `swarm/hives/*` with no +`read` — write-only on purpose, so a file-copier can't recover every hive's +credentials — which means it can't check whether a hive already has a token. +Instead it keeps the value it minted under its `StateDirectory`, +`/var/lib/swarm-secret-publish/matrix-appservice-token/` (`0700` dir, +`0600` file), and mints only when that file is missing or empty. The `put` +itself runs every time: re-putting the same bytes changes nothing for any +reader, while a mint whose publish failed must not be left as a token this +host holds and no hive can reach. + +Lose the state directory and the next run mints once more and republishes. +That rotates the token, which readers pick up on their next start — nothing +that already registered breaks, because the token authenticates the +appservice rather than living inside any account it created. + +The path was `…/matrix/registration-token` while the homeserver still took a +shared registration secret; a value left at the old path is read by nothing. The **second reader** is the agent containers' queue credential: `glue-queue-agent-credential.nix` lands it as two files, the client secret and diff --git a/nix/host-modules/hive-matrix.nix b/nix/host-modules/hive-matrix.nix index 71485f0b..53aa9c15 100644 --- a/nix/host-modules/hive-matrix.nix +++ b/nix/host-modules/hive-matrix.nix @@ -126,6 +126,15 @@ let # whatever they now hold. **One script with two callers** — the activation # script below and `glue-matrix-bao-token.nix`, which overwrites the # `as_token` with the swarm store's copy and has to re-render afterwards. + # + # ⚠️ The `as_token` mint here is the FIRST-BOOT FALLBACK, not the normal + # route. `swarm-secret-publisher.nix` mints one per hive swarm-side and + # publishes it to the store; the reader above overwrites this file before + # the homeserver's container starts, so what a hive actually runs on is the + # swarm's value. This keeps a hive that has never reached the store working + # — and it fires only when the file is absent, so it never clobbers one the + # store delivered. (`hs_token` has no swarm half: it is how the homeserver + # authenticates to *us*, so it stays hive-local and is minted here for real.) # A second copy of the registration's shape would be a second source of # truth for a format whose mismatch is silent: the homeserver would load a # registration naming a token nobody holds, and every request hive-c0re diff --git a/nix/host-modules/swarm-secret-publisher.nix b/nix/host-modules/swarm-secret-publisher.nix index b373bf58..6f1a51e6 100644 --- a/nix/host-modules/swarm-secret-publisher.nix +++ b/nix/host-modules/swarm-secret-publisher.nix @@ -1,6 +1,7 @@ -# The unit that copies authelia's minted OIDC client secrets into the swarm's -# secret store, so whoever needs one can read it there: a hive its agents' -# credential, a swarm service its own. +# The unit that puts swarm-level secrets into the swarm's secret store, so +# whoever needs one can read it there: a hive its agents' credential, a swarm +# service its own, a hive its matrix appservice token. The OIDC secrets it only +# copies; the appservice token it MINTS, having had no swarm-side producer. # # ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes # into the store even when the service runs beside authelia, because its @@ -14,19 +15,19 @@ # why: the plaintext's other reader lives in a **different container**, and # containers that share this host's network namespace still have separate # filesystem roots, so "the host is the only place both trees are addressable". -# A delivery step therefore runs on the host that mints — not on the store's -# host, and not on the reading hive's. +# A delivery step therefore runs on the minting host, not the store's or the +# reading hive's. # # ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite -# every hive's policy and login role; a unit whose entire job is copying one -# file has no business holding it. ./swarm-bao.nix grants this one -# `create`/`update` under the hive prefix and nothing else. +# every hive's policy and login role; a unit that writes a handful of secrets +# has no business holding it. ./swarm-bao.nix grants this one `create`/`update` +# under the hive prefix and nothing else — no `read`, which is why the mint +# below keeps its own record of what it published. # # ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every # argument is world-readable in /proc for the life of the call — the value is -# passed as `@` and read by bao itself. (`/knowledge/secret-hygiene.md`: -# "a path keeps the secret out of the store, and reading it into a shell -# variable puts it straight into argv".) +# passed as `@` and read by bao itself. Same rule governs the mint: it +# redirects into a file and never holds the token in a shell variable. { pkgs, lib, @@ -71,6 +72,21 @@ let hyperhiveCfg.swarm.grafana.oidc.clientId hyperhiveCfg.swarm.otel.clientId ]; + + # Where this unit keeps the appservice tokens it minted, and the whole reason + # a re-publish is idempotent. The store cannot be that record: ./swarm-bao.nix + # grants this principal `create`/`update` and deliberately no `read`, so "does + # this hive already have a token?" is a question it can only ask locally. + # Without the file every run would mint a fresh value and rotate the swarm's + # token for no reason — and a rotation is not free, because the homeserver + # only reloads the registration naming it when its container restarts. + # + # `StateDirectory=` below is what creates it, so the name there is relative + # and this unit owns the tree. Losing it mints once more and republishes; + # readers pick that up on their next start, and nothing that already + # registered breaks — the token authenticates the appservice, it does not + # live inside any account it created. + matrixTokenDir = "/var/lib/swarm-secret-publish/matrix-appservice-token"; in { options.services.hyperhive.deploy.swarm-secret-publisher = { @@ -82,10 +98,12 @@ in Publish the OIDC client secrets this host mints into the swarm's secret store, so a hive that does not run authelia can read its agents' credential and a swarm service can read its own — from - wherever it runs, this host included. + wherever it runs, this host included. Also mints each hive's matrix + appservice token, which has no other swarm-side producer, and + publishes it the same way. - Defaults to whether this host mints them, which is the only half of - the question that is a property of *this* host. + Defaults to whether this host mints the OIDC secrets, which is the + only half of the question that is a property of *this* host. ⚠️ Deliberately **not** `deploy.bao.enable`. That asks whether the store stands here, and a publisher beside a remote store, holding a @@ -95,11 +113,12 @@ in identity's job (see `baoClientCertFile`), not this option's. Turning it off leaves every hive but this one without its agents' - credential, the swarm's Grafana without any login at all, and its - collector pushing unauthenticated — each secret has exactly one route - and this is the producer's end of it. So the honest reason to set it - false is a deployment delivering those secrets by some other - mechanism it owns. + credential, the swarm's Grafana without any login at all, its + collector pushing unauthenticated, and every hive falling back to the + appservice token its own first boot minted — so two hives never agree + on one. Each secret has exactly one route and this is the producer's + end of it, so the honest reason to set it false is a deployment + delivering those secrets by some other mechanism it owns. ''; }; @@ -143,7 +162,7 @@ in }; systemd.services.swarm-secret-publish = { - description = "publish minted OIDC client secrets to the swarm secret store"; + description = "publish the swarm's minted secrets to the swarm secret store"; wantedBy = [ "multi-user.target" ]; after = [ "network.target" ]; path = [ @@ -159,6 +178,12 @@ in TimeoutStartSec = 60; Restart = "on-failure"; RestartSec = 30; + # The minted appservice tokens, see `matrixTokenDir` above. Named + # relative to /var/lib because that is the form systemd creates; the + # mode is narrowed from systemd's 0755 default because the directory + # listing is the roster of hives and every file in it is a secret. + StateDirectory = "swarm-secret-publish/matrix-appservice-token"; + StateDirectoryMode = "0700"; }; environment = { BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}"; @@ -185,6 +210,7 @@ in published=0 skipped=0 + tokens=0 ${lib.concatMapStringsSep "\n" (hive: '' src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"} @@ -228,7 +254,43 @@ in fi '') serviceClientIds} - echo "published $published client secret(s), skipped $skipped" + # The matrix appservice token, per hive. Unlike the two loops above + # there is nothing to copy: authelia never minted this one, and each + # homeserver's own host minted its own — which is exactly why two hives + # never converged on a value the swarm has to agree on. The mint moves + # here because this unit already holds a store identity and already + # writes under the hive prefix; ../host-modules/hive-matrix.nix keeps + # its activation mint as the first-boot fallback for a hive that has + # never reached the store. + # + # 32 bytes of urandom as lowercase hex, the shape that file has + # generated all along. Written with no trailing newline, unlike that + # file: what goes into the store is the value, and + # ./glue-matrix-bao-token.nix adds the newline the hive's token file + # carries. + ${lib.concatMapStringsSep "\n" (hive: '' + src=${lib.escapeShellArg "${matrixTokenDir}/${hive}"} + if [ ! -s "$src" ]; then + # The subshell keeps the umask local — the loops above rely on + # bao's own file handling and should not inherit one. + ( umask 077; head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$src" ) + echo "minted a matrix appservice token for ${hive}" >&2 + fi + # `value=@$src` for the same reason as both loops above: bao opens + # the file itself, so the token is never an argument of this process. + # + # Put on every run, not only on the run that minted — a put of the + # value already there changes nothing for any reader, while a mint + # whose publish failed must not be left as a token this host holds + # and no hive can reach. The `-s` test above is what keeps the VALUE + # stable across those runs: rotating on every boot would hand the + # homeserver a registration its running container does not carry. + bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/matrix/appservice-token"} \ + value=@"$src" + tokens=$((tokens + 1)) + '') hiveNames} + + echo "published $published client secret(s), skipped $skipped, and $tokens matrix appservice token(s)" ''; }; }; diff --git a/nix/module-eval.nix b/nix/module-eval.nix index 490076d2..7c819347 100644 --- a/nix/module-eval.nix +++ b/nix/module-eval.nix @@ -1477,6 +1477,41 @@ let lib.hasInfix "secret/swarm/hives/h1/queue/agent" s && lib.hasInfix "secret/swarm/hives/h2/queue/agent" s; } + { + # The producer's end of the read `glue-matrix-bao-token.nix` already did. + # Both hives are asserted for the reason the queue case above gives: a + # publisher that knew one name rather than the roster would pass on a + # single-hive fixture and strand the second hive ever added — which is + # the two-hives-never-converge shape this slice exists to close. + name = "the publisher mints an appservice token for every hive and writes it to that hive's matrix path"; + ok = + let + s = secretPublisherHere.systemd.services.swarm-secret-publish.script; + in + lib.hasInfix "secret/swarm/hives/h1/matrix/appservice-token" s + && lib.hasInfix "secret/swarm/hives/h2/matrix/appservice-token" s + && lib.hasInfix "/dev/urandom" s; + } + { + # What makes a re-publish idempotent. This principal is granted + # `create`/`update` and no `read`, so it cannot ask the store whether a + # hive already has a token — with nowhere to keep one, every run would + # mint a fresh value and rotate the swarm's token. A state directory is + # that somewhere, and nothing else in this unit needs one, so its absence + # means exactly this. + # + # The second arm is the mint's own guard: the state file is only written + # when it is missing or empty. Dropping that test leaves a unit that + # still has a state directory and still rotates on every boot. + name = "the publisher keeps the tokens it minted, and mints only when it holds none"; + ok = + let + u = secretPublisherHere.systemd.services.swarm-secret-publish; + in + lib.hasInfix "matrix-appservice-token" (u.serviceConfig.StateDirectory or "") + && u.serviceConfig.StateDirectoryMode or null == "0700" + && lib.hasInfix "if [ ! -s \"$src\" ]" u.script; + } { # A property of the SET, not of one unit: both of these authenticate by # certificate, and `BAO_CLIENT_CERT` is transport rather than identity, so