diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 5643374a..02c7dc54 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -62,7 +62,7 @@ of the cell says how. | `swarm/agents//matrix-recovery/` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | -| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | +| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | | `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index c472703c..ac247935 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -57,7 +57,7 @@ neither is a renaming of the other. | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | @@ -423,7 +423,7 @@ forge host's and Grafana's, which reach more than one: | `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services//oidc/client` | | `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services//oidc/client` | | `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services//oidc/client` | -| `swarm-bao-forge-publisher`, `swarm-bao-forge-oidc`, and the forge's journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` and `swarm/services//oidc/client` | +| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services//oidc/client` and `swarm/services//oidc/client` | The first exists **per hive**, because the path it reads carries a hive name and every hive runs its own reader. Its subject is @@ -449,7 +449,7 @@ shared by eight, which is the whole of what this buys. `swarm-ci-runner.nix` refuses the build wherever the runner runs without its pair, as `swarm-review-classifier.nix` does for the classifier and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without -it `swarm-bao-forge-publisher` and `swarm-bao-forge-oidc` render, fail at start +it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start naming both options, and the controller runs without forge access and the forge without a login source until an operator places the leaf. diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index db108f0e..80dcc827 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -95,9 +95,9 @@ in ) (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "controller" "forgeTokenFile" ] '' The controller reads its forge token from the swarm's secret store, under - its own certificate, and swarm-bao-forge-publisher on the forge's host - puts it there. Remove this definition; the file it named is read by - nothing now and can be deleted. + its own certificate, and forgejo-swarm-controller-account in the forge's + container puts it there. Remove this definition; the file it named is + read by nothing now and can be deleted. '') (lib.mkRenamedOptionModule [ "services" "hyperhive" "swarm" "controller" "authBridgeUrl" ] @@ -268,9 +268,9 @@ in (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "forge" "hostSwarmControllerTokenFile" ] '' The forge publishes the swarm-controller's forge token to the swarm - secret store (swarm-bao-forge-publisher), and the controller reads it - there. Remove this definition; the file it named is read by nothing now - and can be deleted. + secret store (forgejo-swarm-controller-account), and the controller + reads it there. Remove this definition; the file it named is read by + nothing now and can be deleted. '' ) diff --git a/nix/host-modules/glue-bao-readers-policy-order.nix b/nix/host-modules/glue-bao-readers-policy-order.nix index 76e3824b..97d1c3cf 100644 --- a/nix/host-modules/glue-bao-readers-policy-order.nix +++ b/nix/host-modules/glue-bao-readers-policy-order.nix @@ -48,8 +48,6 @@ let swarm-bao-nats-tls = deployCfg.nats.enable; # ./swarm-ci-runner.nix swarm-bao-ci-runner = deployCfg.ci-runner.enable; - # ./hive-forge/default.nix - swarm-bao-forge-publisher = deployCfg.forgejo.enable; }; # Readers that log in under another unit's principal: reader unit → the diff --git a/nix/host-modules/glue-bao-readers-store-order.nix b/nix/host-modules/glue-bao-readers-store-order.nix index b320d7dc..47689399 100644 --- a/nix/host-modules/glue-bao-readers-store-order.nix +++ b/nix/host-modules/glue-bao-readers-store-order.nix @@ -31,7 +31,6 @@ let readersHere = { # ./hive-forge/default.nix swarm-bao-forge-oidc = deployCfg.forgejo.enable; - swarm-bao-forge-publisher = deployCfg.forgejo.enable; # ./hive-matrix.nix swarm-bao-matrix-oidc = deployCfg.matrix.enable; # ./otel.nix: `fetches`, the collector loading the file this unit writes diff --git a/nix/host-modules/hive-forge/default.nix b/nix/host-modules/hive-forge/default.nix index f99f8b1b..1499f220 100644 --- a/nix/host-modules/hive-forge/default.nix +++ b/nix/host-modules/hive-forge/default.nix @@ -55,15 +55,16 @@ let # invite the two to drift rather than prevent it. swarmControllerForgeUser = "swarm-controller"; - # Where the minted token lands inside the forge container — under - # forgejo's own state dir, because the file is the mint's record that it - # has minted and has to survive a reboot. - swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token"; + # The mint's record that it has minted and published the controller's + # token: the token's name, not its value. Inside the forge container, under + # forgejo's own state dir, so it survives a reboot. + swarmControllerTokenNamePath = "/var/lib/forgejo/swarm-controller-token-name"; - # The same file from the host, which is where its publisher reads and watches it. - swarmControllerTokenHostPath = "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"; + # Where the minted token itself used to be written. The mint removes it once + # it has published a token. + oldSwarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token"; - # Where `swarm-bao-forge-publisher` puts it: `controllerForgeTokenLeaf` in + # Where the mint puts the token: `controllerForgeTokenLeaf` in # ../swarm-bao.nix, under the KV mount. controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token"; @@ -93,8 +94,9 @@ let forwarderTelemetryPort = 8898; # The forge host's store identity, handed into the container as system - # credentials (credential id → host source) for the forwarder's fetch unit. - # The host units below keep reading the same files directly. + # credentials (credential id → host source) for the forwarder's fetch unit + # and the controller-token mint. `swarm-bao-forge-oidc` on the host keeps + # reading the same files directly. storeFiles = lib.filterAttrs (_: source: source != null) { forge-bao-cert = deployCfg.bao.forgePublisherClientCertFile; forge-bao-key = deployCfg.bao.forgePublisherClientKeyFile; @@ -104,6 +106,8 @@ let # one word. oneWord = path: builtins.match ".*[[:space:]].*" path == null; forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}"; + mintUnit = "forgejo-swarm-controller-account"; + mintCredential = id: "/run/credentials/${mintUnit}.service/${id}"; # ROOT_URL forgejo advertises in clone links + outbound URLs. # `cfg.domain` doubles as both the forgejo `DOMAIN` setting AND the @@ -165,9 +169,9 @@ in [ "services" "hyperhive" "deploy" "forgejo" "hostSwarmControllerTokenFile" ] '' The forge publishes the swarm-controller's forge token to the swarm - secret store (swarm-bao-forge-publisher), and the controller reads it - there. Remove this definition; the file it named is read by nothing now - and can be deleted. + secret store (forgejo-swarm-controller-account), and the controller + reads it there. Remove this definition; the file it named is read by + nothing now and can be deleted. '' ) ]; @@ -540,7 +544,7 @@ in # same reason the trust bundle below names them: an unordered # resolver write is a race that only shows up on a cold boot. The # forwarder and its secret fetch dial the swarm collector, authelia - # and the store by name. + # and the store by name, and the controller-token mint the store. (import ./../../container-modules/swarm-container-resolver.nix { inherit (networkCfg) bridgeIp; dnsConsumers = [ @@ -548,6 +552,7 @@ in "forgejo-sso-source.service" "opentelemetry-collector.service" "${forwarderUnit}.service" + "${mintUnit}.service" ]; }) @@ -980,34 +985,56 @@ in # swarm-controller — the design requirement was explicit that # this must work even when forge and swarm-controller don't # share a host. - systemd.services.forgejo-swarm-controller-account = { + # + # The token goes straight to the swarm secret store, under the forge + # host's store identity, and never onto disk. Forgejo does not wait + # for this unit. + systemd.services.${mintUnit} = { description = "provision the swarm-controller's forge account + access token"; after = [ "forgejo.service" ]; requires = [ "forgejo.service" ]; wantedBy = [ "multi-user.target" ]; - serviceConfig = { + # ../lib/store-retry.nix: a store that is sealed or unreachable is + # retried, and so is a forge that is not answering yet. + inherit (storeRetry) startLimitBurst startLimitIntervalSec; + serviceConfig = storeRetry.serviceConfig // { Type = "oneshot"; RemainAfterExit = true; User = "forgejo"; Group = "forgejo"; - SyslogIdentifier = "forgejo-swarm-controller-account"; + SyslogIdentifier = mintUnit; + LoadCredential = lib.attrNames storeFiles; + # The forge's answers pass through a temp file. + PrivateTmp = true; + # Each forge call is bounded by curl's own timeouts, and a sealed + # store answers on the port and never answers the write. + TimeoutStartSec = "3min"; }; # `FORGEJO_CUSTOM`, same reason as `forgejo-sso-source` above: # every `forgejo admin` invocation needs it to find the app.ini # the upstream module wrote, not just the auth-source verb. environment = { FORGEJO_CUSTOM = "/var/lib/forgejo/custom"; + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + } + // lib.optionalAttrs havePublisherIdentity { + BAO_CLIENT_CERT = mintCredential "forge-bao-cert"; + BAO_CLIENT_KEY = mintCredential "forge-bao-key"; + } + // lib.optionalAttrs (storeFiles ? forge-bao-server-ca) { + BAO_CACERT = mintCredential "forge-bao-server-ca"; }; path = [ deployCfg.forgejo.package + deployCfg.bao.package pkgs.coreutils + pkgs.curl pkgs.gnugrep + pkgs.jq ]; script = '' set -euo pipefail - ${atomicWriteSecret} - # Idempotent by query, same reasoning as the SSO unit: assert # the effect, not the command's exit code. `--admin`: the # controller creates repos in the agents org on the swarm's @@ -1032,50 +1059,133 @@ in exit 1 fi - token_path=${lib.escapeShellArg swarmControllerTokenPath} - - # Token minting is idempotent by the token FILE's existence, - # not a separate stamp: the file is both the record of "already - # done" and the thing that has to survive, so a state wipe that - # deletes it correctly triggers a re-mint instead of being - # silently masked by a stamp that outlived what it claims exists. - if [ ! -s "$token_path" ]; then - # `write:admin` added for `swarm-controller::forge::Client:: - # ensure_agent_user` (the `CreateForgeUser` jobq node) — - # Forgejo's `admin_create_user` HTTP endpoint refused every - # call with "token does not have at least one of required - # scope(s): [write:admin]" without it, an already-deployed - # swarm hitting this the moment that node shipped. A host - # whose token was minted before this scope was added won't - # pick it up automatically — the `[ ! -s "$token_path" ]` - # guard above only mints when the file is absent, by design - # (see its own comment) — so an existing deployment needs - # its token file deleted to force a re-mint with the new - # scope. - # - # Token name carries a timestamp suffix rather than the - # fixed `swarm-controller-boot` this used to be — a real - # incident, not a hypothetical: deleting the token *file* - # (the remediation above) doesn't delete the token Forgejo - # already has under that name, so re-running this unit hit - # "Command error: access token name has been used already" - # and failed outright. Same fix `hive-c0re::forge::users:: - # mint_token` already uses for its own agent tokens, for - # the identical reason — a monotonic suffix means a re-mint - # can never collide with whatever this account already has. - out=$(forgejo admin user generate-access-token \ - --username ${lib.escapeShellArg swarmControllerForgeUser} \ - --token-name "swarm-controller-boot-$(date +%s)" \ - --scopes "write:repository,write:organization,write:issue,read:user,write:admin") - token=$(printf '%s' "$out" | grep -oE '[0-9a-f]{32,}' | head -n1) - if [ -z "$token" ]; then - echo "no token-shaped word in forgejo's generate-access-token output" >&2 - exit 1 - fi - atomic_write_secret 0600 "" "$token_path" "$token" - echo "minted forge access token for ${swarmControllerForgeUser}" + # Minted once. The marker holds the published token's name and is + # written last, so a marker present means the store has a token + # and every earlier one is revoked. Deleting it forces a re-mint, + # the step for a store that lost the token or for new scopes. + name_path=${lib.escapeShellArg swarmControllerTokenNamePath} + if [ -s "$name_path" ]; then + echo "the forge token $(< "$name_path") for ${swarmControllerForgeUser} is already in the swarm secret store" + exit 0 fi - ''; + '' + + ( + if !havePublisherIdentity then + '' + echo "the forge's host has no forge-publisher store identity, so the swarm-controller's forge token cannot be published: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2 + exit 1 + '' + else + '' + err="$(mktemp)" + body="$(mktemp)" + trap 'rm -f "$err" "$body"' EXIT + + bao_failed() { + echo "$1" >&2 + if [ -s "$err" ]; then + cat "$err" >&2 + else + echo "bao failed without writing a diagnostic." >&2 + fi + exit 1 + } + + # The revocation below goes through the forge's admin API, + # which refuses a token of an account that is not a site + # admin; each retry would then leave one more token live. + if ! forgejo admin user list --admin | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then + echo "forge account ${swarmControllerForgeUser} is not a site admin; promote it by hand. No token minted." >&2 + exit 1 + fi + + # Before the mint, so a store that is down fails the attempt + # before forgejo holds a token nothing can read. + # `-token-only` answers on stdout and skips the token helper, a + # `sh` this unit's `path` does not carry. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + bao_failed "could not log in to swarm-bao with the forge publisher's certificate; no forge token minted for ${swarmControllerForgeUser}." + fi + export BAO_TOKEN + + # `write:admin`: the controller's `CreateForgeUser` jobq node + # (`swarm-controller::forge::Client::ensure_agent_user`) and + # the revocation below call the admin API. + # + # Timestamped: forgejo refuses a name the account already has + # a token under, and the earlier tokens are still there until + # the revocation below. `hive-c0re::forge::users::mint_token` + # names its tokens the same way. + name="swarm-controller-boot-$(date +%s)" + out=$(forgejo admin user generate-access-token \ + --username ${lib.escapeShellArg swarmControllerForgeUser} \ + --token-name "$name" \ + --scopes "write:repository,write:organization,write:issue,read:user,write:admin") + token=$(printf '%s' "$out" | grep -oE '[0-9a-f]{32,}' | head -n1) + if [ -z "$token" ]; then + echo "no token-shaped word in forgejo's generate-access-token output" >&2 + exit 1 + fi + + # On stdin: the token is never an argument of any process. + if ! printf '%s' "$token" | bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=- >/dev/null 2>"$err"; then + bao_failed "swarm-bao refused the write to ${controllerTokenStorePath}; the next attempt mints another token and revokes this one." + fi + echo "published the forge token $name for ${swarmControllerForgeUser} to ${controllerTokenStorePath}" + + # One call to the forge's API as the new token. Prints the + # HTTP status; the body lands in "$body". The token reaches + # curl as a header file, never as an argument. + forge() { + local method="$1" path="$2" + curl --silent --show-error --connect-timeout 5 --max-time 15 \ + -H @<(printf 'Authorization: token %s\n' "$token") \ + -X "$method" -o "$body" -w '%{http_code}' \ + "http://127.0.0.1:${toString cfg.httpPort}/api/v1$path" + } + + # Every other token this unit minted: one whose write to the + # store failed, and the one the store held before the marker + # was lost. The bare `swarm-controller-boot` is the name the + # unit minted under before the timestamp suffix. A failure + # here leaves the marker unwritten, so the retry mints again + # and repeats the revocation. + stale=() + page=1 + while :; do + status="$(forge GET "/admin/users/${swarmControllerForgeUser}/tokens?limit=50&page=$page")" + if [ "$status" != 200 ]; then + echo "the forge answered $status to listing ${swarmControllerForgeUser}'s tokens:" >&2 + cat "$body" >&2 + exit 1 + fi + [ "$(jq length "$body")" -gt 0 ] || break + while IFS= read -r n; do + case "$n" in + "$name") ;; + swarm-controller-boot | swarm-controller-boot-*) stale+=("$n") ;; + esac + done < <(jq -r '.[].name' "$body") + page=$((page + 1)) + done + for n in "''${stale[@]}"; do + status="$(forge DELETE "/admin/users/${swarmControllerForgeUser}/tokens/$n")" + case "$status" in + 204 | 404) ;; + *) + echo "the forge answered $status to revoking ${swarmControllerForgeUser}'s token $n:" >&2 + cat "$body" >&2 + exit 1 + ;; + esac + done + echo "revoked ''${#stale[@]} earlier forge token(s) of ${swarmControllerForgeUser}" + + rm -f ${lib.escapeShellArg oldSwarmControllerTokenPath} + printf '%s\n' "$name" >"$name_path.tmp" + mv -f "$name_path.tmp" "$name_path" + '' + ); }; }; }; @@ -1089,9 +1199,9 @@ in # read both fail loudly, and `storeRetry` retries until the publisher on # authelia's host has written the secret. # - # On the host, where that leaf is: the container holds no store identity. - # A host with no publisher leaf fails here, naming the options, rather than - # rendering no unit. + # On the host, which reads that leaf from its own files. A host with no + # publisher leaf fails here, naming the options, rather than rendering no + # unit. systemd.services.swarm-bao-forge-oidc = { description = "fetch the forge's OIDC client secret from the swarm secret store"; # Ordered after the store where it runs here @@ -1185,122 +1295,6 @@ in systemd.timers.swarm-bao-forge-oidc = storeRefreshTimer "re-read the forge's OIDC client secret from the swarm secret store"; - # The reverse direction of the unit above: publish the swarm-controller's - # freshly-minted forge token OUT of the container into the swarm secret - # store, where the controller reads it wherever it runs. On the host and - # under the forge's own store identity for the reason the unit above gives, - # so nothing here asks where the store or the controller is. - # - # It runs at boot and on a switch that adds or changes it - # (`multi-user.target`), each time the container starts, and each time the - # mint writes the file - # (`systemd.paths.swarm-bao-forge-publisher` below). The mint is a oneshot in - # the container's own systemd, invisible to this one by unit name, so a - # missing file is not an error: the path unit runs this again when the mint - # writes it, the shape of `swarm-secret-publish`. - # - # ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns - # every later start into a no-op, so neither trigger would run it again. - # - # Unconditional, like the in-container unit it publishes from: forge is a - # swarm-wide singleton, so the token always gets minted and published, - # regardless of whether swarm-controller runs on this host, another host, - # or nowhere in this swarm at all. A host with no publisher leaf fails - # here, naming the options, rather than rendering no unit. - systemd.services.swarm-bao-forge-publisher = { - description = "publish the swarm-controller's forge token to the swarm secret store"; - # Also ordered after the store where it runs here: - # ../glue-bao-readers-store-order.nix. - after = [ - "container@hive-forge.service" - ]; - wantedBy = [ - "multi-user.target" - "container@hive-forge.service" - ]; - path = [ - deployCfg.bao.package - pkgs.coreutils - ]; - # ../lib/store-retry.nix: a sealed or unreachable store is retried. - inherit (storeRetry) startLimitBurst startLimitIntervalSec; - serviceConfig = storeRetry.serviceConfig // { - Type = "oneshot"; - SyslogIdentifier = "swarm-bao-forge-publisher"; - # Bounded here so the number a switch waits on is in this file. A sealed - # store answers on the port and never answers the write. - TimeoutStartSec = 60; - }; - environment = { - BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; - } - // lib.optionalAttrs havePublisherIdentity { - BAO_CLIENT_CERT = deployCfg.bao.forgePublisherClientCertFile; - BAO_CLIENT_KEY = deployCfg.bao.forgePublisherClientKeyFile; - } - // lib.optionalAttrs (deployCfg.bao.serverCaFile != null) { - BAO_CACERT = deployCfg.bao.serverCaFile; - }; - script = - if !havePublisherIdentity then - '' - echo "the forge's host has no forge-publisher store identity, so it cannot publish the swarm-controller's forge token: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2 - exit 1 - '' - else - '' - set -euo pipefail - - src=${lib.escapeShellArg swarmControllerTokenHostPath} - - if [ ! -s "$src" ]; then - echo "the forge has not minted $src yet; the path unit publishes it when it does" >&2 - exit 0 - fi - - err="$(mktemp)" - trap 'rm -f "$err"' EXIT - - # `-token-only` answers on stdout and skips the token helper, a - # `sh` this unit's `path` does not carry. - if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then - echo "could not log in to swarm-bao with the forge publisher's certificate; the swarm-controller's forge token is not published." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 1 - fi - export BAO_TOKEN - - # `value=@$src` hands bao the PATH: bao opens the file itself, so - # the token is never an argument of this process. - if ! bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=@"$src" >/dev/null 2>"$err"; then - echo "swarm-bao refused the write to ${controllerTokenStorePath}." >&2 - if [ -s "$err" ]; then - cat "$err" >&2 - else - echo "bao failed without writing a diagnostic." >&2 - fi - exit 1 - fi - echo "published the swarm-controller's forge token to ${controllerTokenStorePath}" - ''; - }; - - # Re-publish when the mint unit writes a token, which it can do without the - # container restarting. The file is the event: the mint runs in the - # container's own systemd, which this one cannot subscribe to. - systemd.paths.swarm-bao-forge-publisher = { - description = "watch the swarm-controller's forge token for a publish"; - wantedBy = [ "multi-user.target" ]; - pathConfig = { - PathChanged = swarmControllerTokenHostPath; - Unit = "swarm-bao-forge-publisher.service"; - }; - }; - systemd.tmpfiles.rules = [ # The OIDC secret's directory, which the container binds at start. "d ${oidcSecretDir} 0700 root root -" diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 4b3f1ec9..ba214c6e 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -1107,9 +1107,10 @@ let } ]; - # The forge host's one store identity. `swarm-bao-forge-publisher` writes - # the controller's forge token under it, the one writer of that path and - # write-only: it copies a file in and never reads one back. + # The forge host's one store identity. The forge's + # `forgejo-swarm-controller-account`, in its container, writes the + # controller's forge token under it, the one writer of that path and + # write-only: it puts the token it mints and never reads one back. # `swarm-bao-forge-oidc` reads the forge's own OIDC client secret, and the # journal forwarder in its container its own, those two leaves and no other # service's. @@ -2016,8 +2017,9 @@ in default = "swarm-forge-publisher"; description = '' Subject the store's `swarm-forge-publisher` cert-auth role accepts: the - forge host's one store identity, presented by `swarm-bao-forge-publisher`, - which publishes the swarm-controller's forge token, and by + forge host's one store identity, presented by + `forgejo-swarm-controller-account` in the forge's container, which + mints and publishes the swarm-controller's forge token, by `swarm-bao-forge-oidc`, which fetches the forge's OIDC client secret, and by the journal forwarder in the forge's container. Its grant is three paths: `swarm/controller/swarm-controller/forge/token`, write @@ -2342,9 +2344,10 @@ in default = null; example = "/var/lib/swarm-bao-pki/forge-publisher.pem"; description = '' - Certificate the forge's host presents to the store, both to publish the - swarm-controller's forge token and to fetch the forge's OIDC client - secret. Its subject must be + Certificate the forge's host presents to the store, to fetch the + forge's OIDC client secret, and that the forge's container presents to + publish the swarm-controller's forge token and to fetch its journal + forwarder's secret. Its subject must be {option}`services.hyperhive.deploy.bao.forgePublisherCommonName`. ./glue-bao-tls.nix points it at the leaf it mints on the store's host. diff --git a/nix/host-modules/swarm-controller.nix b/nix/host-modules/swarm-controller.nix index 1cef50ef..a9e4b1c5 100644 --- a/nix/host-modules/swarm-controller.nix +++ b/nix/host-modules/swarm-controller.nix @@ -305,8 +305,9 @@ in '' The controller reads its forge token from the swarm's secret store, under its own certificate (deploy.swarm-controller.baoClientCertFile), and - swarm-bao-forge-publisher on the forge's host puts it there. Remove this - definition; the file it named is read by nothing now and can be deleted. + forgejo-swarm-controller-account in the forge's container puts it + there. Remove this definition; the file it named is read by nothing now + and can be deleted. '' ) ]; diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index 99b25dee..5e87d6c0 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -1736,10 +1736,11 @@ let && lib.hasInfix "swarm-review-classifier \"\" clientAuth" s; } { - # The forge's host is the one writer of the controller's forge token, - # and reads the forge's own OIDC client secret and its container - # forwarder's under the same leaf. Pinned as the whole list, so an added - # `read` on the token or a write or wildcard on a secret fails. + # The forge's mint, in its container, is the one writer of the + # controller's forge token, and the forge's host and its container + # forwarder read their OIDC client secrets under the same leaf. Pinned + # as the whole list, so an added `read` on the token or a write or + # wildcard on a secret fails. name = "the forge publisher's grant is writing the controller's forge token and reading the forge's and its forwarder's OIDC secrets, and nothing else"; ok = let @@ -1780,7 +1781,8 @@ let } { # The policy units that name the path are the whole list of who reaches - # it: the publisher, which writes it, and the controller, which reads it. + # it: the forge publisher's, which the mint writes it under, and the + # controller's, which reads it. name = "no granting unit but the controller's and the forge publisher's names the controller's forge token"; ok = let diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index 422aa7ae..38837ea0 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -123,11 +123,11 @@ let && centralToggleOff.services.hyperhive.swarm.forge.publicUrl == "https://forge.t.local"; } { - # The controller's token publisher follows where the forge runs + # The controller's token mint follows where the forge runs # (./forge-placement.nix), never the central toggle: a forge host with - # the toggle off still publishes it. - name = "the forge's controller-token publisher does not consult the central toggle"; - ok = forgeHere.systemd.services ? swarm-bao-forge-publisher; + # the toggle off still mints and publishes it. + name = "the forge's controller-token mint does not consult the central toggle"; + ok = forgeHere.containers.hive-forge.config.systemd.services ? forgejo-swarm-controller-account; } { name = "a hive that does not host the swarm's shared services runs none of them"; diff --git a/nix/module-eval/forge-placement.nix b/nix/module-eval/forge-placement.nix index f51bba35..ddc39f5b 100644 --- a/nix/module-eval/forge-placement.nix +++ b/nix/module-eval/forge-placement.nix @@ -107,7 +107,7 @@ let # ./grafana.nix's `grafanaRefusedFor`. refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions; - publisher = m: m.systemd.services.swarm-bao-forge-publisher; + mint = m: m.containers.hive-forge.config.systemd.services.forgejo-swarm-controller-account; oidcFetch = m: m.systemd.services.swarm-bao-forge-oidc; forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings; @@ -125,7 +125,6 @@ let ok = !bare.services.hyperhive.deploy.forgejo.enable && !(bare.containers ? hive-forge) - && !(bare.systemd.services ? swarm-bao-forge-publisher) && !(lib.elem "forge.t.local" bare.services.hyperhive.gateway.localNames); } { @@ -327,55 +326,93 @@ let && refusedOver ciUnderForgeNoIdentity "services.hyperhive.deploy.bao.enable"; } { - # The forge's host publishes the controller's token under its own leaf, - # which the store mints beside it here. `value=@` is what keeps the token - # out of the process's arguments. - name = "the forge's host publishes the controller's forge token to the store under its own leaf"; + # The mint, in the forge's container, writes the controller's token to + # the store under the forge host's leaf, loaded as a credential. On + # stdin, so the token is no process's argument; the copy an earlier mint + # kept on the container's disk goes once a token is published, and no + # host unit reads into the container. + name = "the forge's mint writes the controller's forge token straight to the store and keeps no copy"; ok = let - u = publisher allLocal; + u = mint allLocal; + cred = id: "/run/credentials/forgejo-swarm-controller-account.service/${id}"; in - u.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/forge-publisher.pem" - && u.environment.BAO_CLIENT_KEY == "/var/lib/swarm-bao-pki/forge-publisher-key.pem" - && lib.hasInfix "bao kv put secret/swarm/controller/swarm-controller/forge/token value=@\"$src\"" u.script - && lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token" u.script - && lib.elem "container@hive-forge.service" u.wantedBy - # The deploy that adds the unit starts it only through a target. - && lib.elem "multi-user.target" u.wantedBy + lib.elem "forge-bao-cert" u.serviceConfig.LoadCredential + && lib.elem "forge-bao-key" u.serviceConfig.LoadCredential + && u.environment.BAO_CLIENT_CERT == cred "forge-bao-cert" + && u.environment.BAO_CLIENT_KEY == cred "forge-bao-key" + && lib.elem "--load-credential=forge-bao-cert:/var/lib/swarm-bao-pki/forge-publisher.pem" allLocal.containers.hive-forge.extraFlags + && lib.elem "--load-credential=forge-bao-cert:/etc/pki/forge-publisher.pem" forgeAlone.containers.hive-forge.extraFlags + && lib.hasInfix "printf '%s' \"$token\" | bao kv put secret/swarm/controller/swarm-controller/forge/token value=-" u.script + && !(lib.hasInfix "value=@" u.script) + && !(lib.hasInfix "atomic_write_secret" u.script) + && lib.hasInfix "rm -f /var/lib/forgejo/swarm-controller-token\n" u.script + && !(allLocal.systemd.services ? swarm-bao-forge-publisher) + && !(allLocal.systemd.paths ? swarm-bao-forge-publisher) + && !(forgeAlone.systemd.services ? swarm-bao-forge-publisher) && !(allLocal.systemd.services ? hive-forge-swarm-controller-token); } { - # What makes it re-run. A oneshot left `active (exited)` makes every later - # start a no-op, so `RemainAfterExit` would silence every trigger: the - # targets and the container's start (`wantedBy`, above) and a write of - # the token file. A missing file exits 0, so a switch with the container - # down does not fail. - name = "the forge publisher re-runs on each container start and each write of the token file"; + # The order that keeps a token live only if the store has it: log in + # before the mint, put, revoke every other `swarm-controller-boot-*` + # token, and only then write the marker, which holds the name. A marker + # present skips all of it. + name = "the forge's mint writes its marker only after the put and the revocation of every earlier token"; ok = let - p = allLocal.systemd.paths.swarm-bao-forge-publisher; + s = (mint allLocal).script; + at = needle: lib.stringLength (lib.head (lib.splitString needle s)); + order = l: lib.length l < 2 || (at (lib.elemAt l 0) < at (lib.elemAt l 1) && order (lib.tail l)); + steps = [ + "if [ -s \"$name_path\" ]; then" + "forgejo admin user list --admin" + "bao login -method=cert" + "forgejo admin user generate-access-token" + "bao kv put" + "forge GET \"/admin/users/swarm-controller/tokens?limit=50&page=$page\"" + "forge DELETE \"/admin/users/swarm-controller/tokens/$n\"" + "rm -f /var/lib/forgejo/swarm-controller-token\n" + "mv -f \"$name_path.tmp\" \"$name_path\"" + ]; in - !((publisher allLocal).serviceConfig.RemainAfterExit or false) - && - p.pathConfig.PathChanged - == "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token" - && p.pathConfig.Unit == "swarm-bao-forge-publisher.service" - && lib.hasInfix "the path unit publishes it when it does" (publisher allLocal).script - && lib.hasInfix "exit 0" (publisher allLocal).script - && lib.elem "multi-user.target" p.wantedBy - && !(bare.systemd.paths ? swarm-bao-forge-publisher); + lib.all (step: lib.hasInfix step s) steps + && order steps + && lib.hasInfix "name_path=/var/lib/forgejo/swarm-controller-token-name" s + && lib.hasInfix "printf '%s\\n' \"$name\" >\"$name_path.tmp\"" s + && lib.hasInfix "swarm-controller-boot | swarm-controller-boot-*) stale+=(\"$n\") ;;" s + && lib.hasInfix "-H @<(printf 'Authorization: token %s\\n' \"$token\")" s; + } + { + # A store that is down retries the mint (../host-modules/lib/store-retry.nix), + # and forgejo never waits for it. + name = "the forge's mint retries on failure, and forgejo does not wait for it"; + ok = + let + c = allLocal.containers.hive-forge.config.systemd.services; + u = mint allLocal; + in + u.serviceConfig.Restart == "on-failure" + && u.startLimitBurst == 2880 + && u.serviceConfig.RemainAfterExit + && lib.elem "forgejo.service" u.after + && !(lib.elem "forgejo-swarm-controller-account.service" ( + c.forgejo.after ++ c.forgejo.requires ++ c.forgejo.wants + )) + && lib.elem "forgejo-swarm-controller-account.service" c.swarm-bridge-dns.before; } { # The absence arm: a forge host without the store has no leaf unless one - # is placed, and the unit says which options to set. - name = "a forge host with no publisher leaf renders a publisher that fails naming the options"; + # is placed, and the mint says which options to set before it mints. + name = "a forge host with no publisher leaf renders a mint that fails naming the options, minting nothing"; ok = let - s = (publisher forgeHere).script; + u = mint forgeHere; in - lib.hasInfix "forgePublisherClientCertFile" s - && lib.hasInfix "exit 1" s - && !(lib.hasInfix "bao kv put" s); + !(u.environment ? BAO_CLIENT_CERT) + && lib.hasInfix "forgePublisherClientCertFile" u.script + && lib.hasInfix "exit 1" u.script + && !(lib.hasInfix "forgejo admin user generate-access-token" u.script) + && !(lib.hasInfix "bao kv put" u.script); } { # The forge's OIDC secret has one route, the store, under the forge diff --git a/nix/module-eval/swarm-services-switch.nix b/nix/module-eval/swarm-services-switch.nix index a3c1e948..7e63a99e 100644 --- a/nix/module-eval/swarm-services-switch.nix +++ b/nix/module-eval/swarm-services-switch.nix @@ -819,7 +819,6 @@ let ]; readers = [ "swarm-bao-forge-oidc" - "swarm-bao-forge-publisher" "swarm-bao-matrix-oidc" "swarm-bao-hive-otel-oidc" "swarm-bao-grafana-oidc" @@ -848,7 +847,7 @@ let !(lib.any (n: lib.elem n (s.after ++ s.wants ++ s.requires)) storeUnits); in lib.all waits readers - && lib.length readersElsewhere == 8 + && lib.length readersElsewhere == 7 && lib.all namesNoStoreUnit readersElsewhere && !(lib.any (u: storeOnlyNoHive.systemd.services ? ${u}) readers); } diff --git a/swarm-controller/src/forge.rs b/swarm-controller/src/forge.rs index e405c995..1d3041f0 100644 --- a/swarm-controller/src/forge.rs +++ b/swarm-controller/src/forge.rs @@ -8,9 +8,8 @@ //! (`SWARM_CONTROLLER_FORGE_URL`). The token comes from the swarm's secret //! store, at `swarm_secret_client::forge::controller_token_path`, read under //! this daemon's own certificate and held only in memory. The forge mints it -//! inside its container (`hive-forge/default.nix`'s -//! `forgejo-swarm-controller-account`) and `swarm-bao-forge-publisher`, on the -//! forge's host, puts it there. +//! inside its container and puts it there (`hive-forge/default.nix`'s +//! `forgejo-swarm-controller-account`). //! //! [`enable`] fills the daemon's one client slot: at start when the token is //! there, later from a background re-read when it is not. Every caller reads @@ -241,8 +240,9 @@ async fn fetch_token() -> Result { match stored { Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()), _ => anyhow::bail!( - "nothing at {path}: swarm-bao-forge-publisher, on the host that runs the forge, has \ - not published this daemon's forge token" + "nothing at {path}: forgejo-swarm-controller-account, in the forge's container, has \ + not published this daemon's forge token (if the store lost it, delete /var/lib/forgejo/\ + swarm-controller-token-name in that container and restart the unit)" ), } } diff --git a/swarm-secret-client/src/forge.rs b/swarm-secret-client/src/forge.rs index 3e9bb9e2..f7656f43 100644 --- a/swarm-secret-client/src/forge.rs +++ b/swarm-secret-client/src/forge.rs @@ -14,8 +14,8 @@ //! beside [`crate::matrix`]. //! //! `swarm-controller`'s own forge token ([`controller_token_path`]) is the -//! fourth: the forge mints it, `swarm-bao-forge-publisher` on the forge's host -//! writes it here, and the controller reads it as it starts. +//! fourth: the forge's `forgejo-swarm-controller-account`, in its container, +//! mints it and writes it here, and the controller reads it as it starts. use serde::{Deserialize, Serialize}; @@ -82,14 +82,14 @@ pub fn controller_token_path() -> Result { } /// What [`controller_token_path`] holds. No token name beside the value, unlike -/// [`Credential`]: the publisher copies a file that holds the token alone. +/// [`Credential`]: the mint writes the token alone. /// /// No `Debug` derive: `value` is the forge admin credential. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ControllerToken { - /// The token itself. `swarm-bao-forge-publisher` writes it with - /// `bao kv put … value=@`, so this name is load-bearing for a writer - /// this crate does not control. + /// The token itself. `forgejo-swarm-controller-account` writes it with + /// `bao kv put … value=-`, so this name is load-bearing for a writer this + /// crate does not control. pub value: String, }