diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 6cb22dc7..93e655d6 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -61,7 +61,7 @@ of the cell says how. | `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | -| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | +| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | | `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | | `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | | `swarm/agents//forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index c41c7f77..a7502fa3 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -52,7 +52,7 @@ neither is a renaming of the other. | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | diff --git a/nix/host-modules/hive-forge/default.nix b/nix/host-modules/hive-forge/default.nix index 597c8d53..a129778f 100644 --- a/nix/host-modules/hive-forge/default.nix +++ b/nix/host-modules/hive-forge/default.nix @@ -56,6 +56,9 @@ let # `forgeSecretPath` above. swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token"; + # The same file from the host, which is where its publisher reads and watches it. + swarmControllerTokenHostPath = "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"; + # Where `swarm-bao-forge-publisher` puts it: `controllerForgeTokenLeaf` in # ../swarm-bao.nix, under the KV mount. controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token"; @@ -1057,8 +1060,12 @@ in # It can only depend on the container being up, not on the in-container # oneshot that mints the token (containers run their own systemd instance, # invisible to this one by unit name), so it polls the same bounded way - # `hive-forge-oidc-secret` does. `wantedBy` the container, so every - # container start re-publishes whatever the mint unit left there. + # `hive-forge-oidc-secret` does. It runs whenever the token can have + # changed: each time the container starts (`wantedBy`), and each time the + # file is written (`systemd.paths.swarm-bao-forge-publisher` below). + # + # ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns + # every later start into a no-op, so neither trigger would run it again. # # Unconditional, like the in-container unit it publishes from: forge is a # swarm-wide singleton, so the token always gets minted and published, @@ -1088,7 +1095,6 @@ in inherit (storeRetry) startLimitBurst startLimitIntervalSec; serviceConfig = storeRetry.serviceConfig // { Type = "oneshot"; - RemainAfterExit = true; SyslogIdentifier = "swarm-bao-forge-publisher"; # Longer than the 120s bounded wait below. `DefaultTimeoutStartSec` # is 90s, so without this systemd kills the unit at 90 — before it @@ -1115,7 +1121,7 @@ in '' set -euo pipefail - src=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"} + src=${lib.escapeShellArg swarmControllerTokenHostPath} for _ in $(seq 1 60); do [ -s "$src" ] && break @@ -1157,6 +1163,18 @@ in ''; }; + # Re-publish when the mint unit writes a token, which it can do without the + # container restarting. The file is the event: the mint runs in the + # container's own systemd, which this one cannot subscribe to. + systemd.paths.swarm-bao-forge-publisher = { + description = "watch the swarm-controller's forge token for a publish"; + wantedBy = [ "multi-user.target" ]; + pathConfig = { + PathChanged = swarmControllerTokenHostPath; + Unit = "swarm-bao-forge-publisher.service"; + }; + }; + # Unused: nothing writes or reads this host path, and it may hold a copy of # the controller's forge token. systemd.tmpfiles.rules = [ "r /var/lib/hyperhive-forge/swarm-controller.token" ]; diff --git a/nix/module-eval/forge-placement.nix b/nix/module-eval/forge-placement.nix index 3d0b7fcb..501d52bf 100644 --- a/nix/module-eval/forge-placement.nix +++ b/nix/module-eval/forge-placement.nix @@ -294,6 +294,23 @@ let && lib.elem "container@hive-forge.service" u.wantedBy && !(allLocal.systemd.services ? hive-forge-swarm-controller-token); } + { + # What makes it re-run. A oneshot left `active (exited)` makes every later + # start a no-op, so `RemainAfterExit` would silence both triggers: the + # container's start (`wantedBy`, above) and a write of the token file. + name = "the forge publisher re-runs on each container start and each write of the token file"; + ok = + let + p = allLocal.systemd.paths.swarm-bao-forge-publisher; + in + !((publisher allLocal).serviceConfig.RemainAfterExit or false) + && + p.pathConfig.PathChanged + == "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token" + && p.pathConfig.Unit == "swarm-bao-forge-publisher.service" + && lib.elem "multi-user.target" p.wantedBy + && !(bare.systemd.paths ? swarm-bao-forge-publisher); + } { # The absence arm: a forge host without the store has no leaf unless one # is placed, and the unit says which options to set.