vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-forge: the controller-token mint writes straight to the store, and the on-disk token goes

Browse source 
`forgejo-swarm-controller-account`, in the forge's container, now puts
the swarm-controller's forge token into the swarm secret store itself,
under the forge publisher's leaf. The container already receives that
leaf as nspawn system credentials for its journal forwarder; the mint
loads the same three. The token reaches bao on stdin (`value=-`) and
the forge's API through a header file, never as an argument, and never
touches disk.

The mint's guard is a marker holding the published token's name, at
/var/lib/forgejo/swarm-controller-token-name, written last: after the
put, and after every other `swarm-controller-boot-*` token of the
account is revoked through forgejo's admin API, authenticated as the
new token. So a put that failed, or a marker lost while the store kept
an older token, leaves no live token behind once a mint completes. The
mint logs in to the store and checks that the account is a site admin
before it mints, so a store that is down or an account the admin API
would refuse mints nothing on each retry. Failures retry under
store-retry; forgejo does not wait for the mint. No bao read and no
grant change: the forge publisher keeps create/update on the token.

Removed: the host's `swarm-bao-forge-publisher` service and path unit,
which read the token file through the container's root, and their
entries in the store and policy ordering glue. The forge publisher's
policy unit stays; `swarm-bao-forge-oidc` still uses the leaf.

On deploy, an existing forge host has no marker, so the mint runs once:
it mints a new token, publishes it, revokes the old one, and deletes
the old /var/lib/forgejo/swarm-controller-token. The controller's next
forge call gets a 401, re-reads the store and carries on.
This commit is contained in:
atlas 2026-10-16 14:39:19 +02:00 • committed by mara
commit 141be16c6d
14 changed files with 294 additions and 261 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials.md
View file

@ -62,7 +62,7 @@ of the cell says how.
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |

6
docs/swarm/secrets.md
View file

@ -57,7 +57,7 @@ neither is a renaming of the other.
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
@ -423,7 +423,7 @@ forge host's and Grafana's, which reach more than one:
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` | | `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` | | `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` | | `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
| `swarm-bao-forge-publisher`, `swarm-bao-forge-oidc`, and the forge's journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` | | `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
The first exists **per hive**, because the path it reads carries a hive name The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is and every hive runs its own reader. Its subject is
@ -449,7 +449,7 @@ shared by eight, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as refuses the build wherever the runner runs without its pair, as
`swarm-review-classifier.nix` does for the classifier and `swarm-grafana.nix` `swarm-review-classifier.nix` does for the classifier and `swarm-grafana.nix`
for Grafana. The forge's pair refuses nothing: without for Grafana. The forge's pair refuses nothing: without
it `swarm-bao-forge-publisher` and `swarm-bao-forge-oidc` render, fail at start it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
naming both options, and the controller runs without forge access and the forge naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf. without a login source until an operator places the leaf.

12
nix/host-modules/deploy.nix
View file

@ -95,9 +95,9 @@ in
) )
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "controller" "forgeTokenFile" ] '' (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "controller" "forgeTokenFile" ] ''
The controller reads its forge token from the swarm's secret store, under The controller reads its forge token from the swarm's secret store, under
its own certificate, and swarm-bao-forge-publisher on the forge's host its own certificate, and forgejo-swarm-controller-account in the forge's
puts it there. Remove this definition; the file it named is read by container puts it there. Remove this definition; the file it named is
nothing now and can be deleted. read by nothing now and can be deleted.
'') '')
(lib.mkRenamedOptionModule (lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "controller" "authBridgeUrl" ] [ "services" "hyperhive" "swarm" "controller" "authBridgeUrl" ]
@ -268,9 +268,9 @@ in
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "forge" "hostSwarmControllerTokenFile" ] (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "forge" "hostSwarmControllerTokenFile" ]
'' ''
The forge publishes the swarm-controller's forge token to the swarm The forge publishes the swarm-controller's forge token to the swarm
secret store (swarm-bao-forge-publisher), and the controller reads it secret store (forgejo-swarm-controller-account), and the controller
there. Remove this definition; the file it named is read by nothing now reads it there. Remove this definition; the file it named is read by
and can be deleted. nothing now and can be deleted.
'' ''
) )

2
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -48,8 +48,6 @@ let
swarm-bao-nats-tls = deployCfg.nats.enable; swarm-bao-nats-tls = deployCfg.nats.enable;
# ./swarm-ci-runner.nix # ./swarm-ci-runner.nix
swarm-bao-ci-runner = deployCfg.ci-runner.enable; swarm-bao-ci-runner = deployCfg.ci-runner.enable;
# ./hive-forge/default.nix
swarm-bao-forge-publisher = deployCfg.forgejo.enable;
}; };
# Readers that log in under another unit's principal: reader unit → the # Readers that log in under another unit's principal: reader unit → the

1
nix/host-modules/glue-bao-readers-store-order.nix
View file

@ -31,7 +31,6 @@ let
readersHere = { readersHere = {
# ./hive-forge/default.nix # ./hive-forge/default.nix
swarm-bao-forge-oidc = deployCfg.forgejo.enable; swarm-bao-forge-oidc = deployCfg.forgejo.enable;
swarm-bao-forge-publisher = deployCfg.forgejo.enable;
# ./hive-matrix.nix # ./hive-matrix.nix
swarm-bao-matrix-oidc = deployCfg.matrix.enable; swarm-bao-matrix-oidc = deployCfg.matrix.enable;
# ./otel.nix: `fetches`, the collector loading the file this unit writes # ./otel.nix: `fetches`, the collector loading the file this unit writes

354
nix/host-modules/hive-forge/default.nix
View file

@ -55,15 +55,16 @@ let
# invite the two to drift rather than prevent it. # invite the two to drift rather than prevent it.
swarmControllerForgeUser = "swarm-controller"; swarmControllerForgeUser = "swarm-controller";
# Where the minted token lands inside the forge container — under # The mint's record that it has minted and published the controller's
# forgejo's own state dir, because the file is the mint's record that it # token: the token's name, not its value. Inside the forge container, under
# has minted and has to survive a reboot. # forgejo's own state dir, so it survives a reboot.
swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token"; swarmControllerTokenNamePath = "/var/lib/forgejo/swarm-controller-token-name";
# The same file from the host, which is where its publisher reads and watches it. # Where the minted token itself used to be written. The mint removes it once
swarmControllerTokenHostPath = "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"; # it has published a token.
oldSwarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token";
# Where `swarm-bao-forge-publisher` puts it: `controllerForgeTokenLeaf` in # Where the mint puts the token: `controllerForgeTokenLeaf` in
# ../swarm-bao.nix, under the KV mount. # ../swarm-bao.nix, under the KV mount.
controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token"; controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token";
@ -93,8 +94,9 @@ let
forwarderTelemetryPort = 8898; forwarderTelemetryPort = 8898;
# The forge host's store identity, handed into the container as system # The forge host's store identity, handed into the container as system
# credentials (credential id → host source) for the forwarder's fetch unit. # credentials (credential id → host source) for the forwarder's fetch unit
# The host units below keep reading the same files directly. # and the controller-token mint. `swarm-bao-forge-oidc` on the host keeps
# reading the same files directly.
storeFiles = lib.filterAttrs (_: source: source != null) { storeFiles = lib.filterAttrs (_: source: source != null) {
forge-bao-cert = deployCfg.bao.forgePublisherClientCertFile; forge-bao-cert = deployCfg.bao.forgePublisherClientCertFile;
forge-bao-key = deployCfg.bao.forgePublisherClientKeyFile; forge-bao-key = deployCfg.bao.forgePublisherClientKeyFile;
@ -104,6 +106,8 @@ let
# one word. # one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null; oneWord = path: builtins.match ".*[[:space:]].*" path == null;
forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}"; forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}";
mintUnit = "forgejo-swarm-controller-account";
mintCredential = id: "/run/credentials/${mintUnit}.service/${id}";
# ROOT_URL forgejo advertises in clone links + outbound URLs. # ROOT_URL forgejo advertises in clone links + outbound URLs.
# `cfg.domain` doubles as both the forgejo `DOMAIN` setting AND the # `cfg.domain` doubles as both the forgejo `DOMAIN` setting AND the
@ -165,9 +169,9 @@ in
[ "services" "hyperhive" "deploy" "forgejo" "hostSwarmControllerTokenFile" ] [ "services" "hyperhive" "deploy" "forgejo" "hostSwarmControllerTokenFile" ]
'' ''
The forge publishes the swarm-controller's forge token to the swarm The forge publishes the swarm-controller's forge token to the swarm
secret store (swarm-bao-forge-publisher), and the controller reads it secret store (forgejo-swarm-controller-account), and the controller
there. Remove this definition; the file it named is read by nothing now reads it there. Remove this definition; the file it named is read by
and can be deleted. nothing now and can be deleted.
'' ''
) )
]; ];
@ -540,7 +544,7 @@ in
# same reason the trust bundle below names them: an unordered # same reason the trust bundle below names them: an unordered
# resolver write is a race that only shows up on a cold boot. The # resolver write is a race that only shows up on a cold boot. The
# forwarder and its secret fetch dial the swarm collector, authelia # forwarder and its secret fetch dial the swarm collector, authelia
# and the store by name. # and the store by name, and the controller-token mint the store.
(import ./../../container-modules/swarm-container-resolver.nix { (import ./../../container-modules/swarm-container-resolver.nix {
inherit (networkCfg) bridgeIp; inherit (networkCfg) bridgeIp;
dnsConsumers = [ dnsConsumers = [
@ -548,6 +552,7 @@ in
"forgejo-sso-source.service" "forgejo-sso-source.service"
"opentelemetry-collector.service" "opentelemetry-collector.service"
"${forwarderUnit}.service" "${forwarderUnit}.service"
"${mintUnit}.service"
]; ];
}) })
@ -980,34 +985,56 @@ in
# swarm-controller — the design requirement was explicit that # swarm-controller — the design requirement was explicit that
# this must work even when forge and swarm-controller don't # this must work even when forge and swarm-controller don't
# share a host. # share a host.
systemd.services.forgejo-swarm-controller-account = { #
# The token goes straight to the swarm secret store, under the forge
# host's store identity, and never onto disk. Forgejo does not wait
# for this unit.
systemd.services.${mintUnit} = {
description = "provision the swarm-controller's forge account + access token"; description = "provision the swarm-controller's forge account + access token";
after = [ "forgejo.service" ]; after = [ "forgejo.service" ];
requires = [ "forgejo.service" ]; requires = [ "forgejo.service" ];
wantedBy = [ "multi-user.target" ]; wantedBy = [ "multi-user.target" ];
serviceConfig = { # ../lib/store-retry.nix: a store that is sealed or unreachable is
# retried, and so is a forge that is not answering yet.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
User = "forgejo"; User = "forgejo";
Group = "forgejo"; Group = "forgejo";
SyslogIdentifier = "forgejo-swarm-controller-account"; SyslogIdentifier = mintUnit;
LoadCredential = lib.attrNames storeFiles;
# The forge's answers pass through a temp file.
PrivateTmp = true;
# Each forge call is bounded by curl's own timeouts, and a sealed
# store answers on the port and never answers the write.
TimeoutStartSec = "3min";
}; };
# `FORGEJO_CUSTOM`, same reason as `forgejo-sso-source` above: # `FORGEJO_CUSTOM`, same reason as `forgejo-sso-source` above:
# every `forgejo admin` invocation needs it to find the app.ini # every `forgejo admin` invocation needs it to find the app.ini
# the upstream module wrote, not just the auth-source verb. # the upstream module wrote, not just the auth-source verb.
environment = { environment = {
FORGEJO_CUSTOM = "/var/lib/forgejo/custom"; FORGEJO_CUSTOM = "/var/lib/forgejo/custom";
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs havePublisherIdentity {
BAO_CLIENT_CERT = mintCredential "forge-bao-cert";
BAO_CLIENT_KEY = mintCredential "forge-bao-key";
}
// lib.optionalAttrs (storeFiles ? forge-bao-server-ca) {
BAO_CACERT = mintCredential "forge-bao-server-ca";
}; };
path = [ path = [
deployCfg.forgejo.package deployCfg.forgejo.package
deployCfg.bao.package
pkgs.coreutils pkgs.coreutils
pkgs.curl
pkgs.gnugrep pkgs.gnugrep
pkgs.jq
]; ];
script = '' script = ''
set -euo pipefail set -euo pipefail
${atomicWriteSecret}
# Idempotent by query, same reasoning as the SSO unit: assert # Idempotent by query, same reasoning as the SSO unit: assert
# the effect, not the command's exit code. `--admin`: the # the effect, not the command's exit code. `--admin`: the
# controller creates repos in the agents org on the swarm's # controller creates repos in the agents org on the swarm's
@ -1032,50 +1059,133 @@ in
exit 1 exit 1
fi fi
token_path=${lib.escapeShellArg swarmControllerTokenPath} # Minted once. The marker holds the published token's name and is
# written last, so a marker present means the store has a token
# Token minting is idempotent by the token FILE's existence, # and every earlier one is revoked. Deleting it forces a re-mint,
# not a separate stamp: the file is both the record of "already # the step for a store that lost the token or for new scopes.
# done" and the thing that has to survive, so a state wipe that name_path=${lib.escapeShellArg swarmControllerTokenNamePath}
# deletes it correctly triggers a re-mint instead of being if [ -s "$name_path" ]; then
# silently masked by a stamp that outlived what it claims exists. echo "the forge token $(< "$name_path") for ${swarmControllerForgeUser} is already in the swarm secret store"
if [ ! -s "$token_path" ]; then exit 0
# `write:admin` added for `swarm-controller::forge::Client::
# ensure_agent_user` (the `CreateForgeUser` jobq node) —
# Forgejo's `admin_create_user` HTTP endpoint refused every
# call with "token does not have at least one of required
# scope(s): [write:admin]" without it, an already-deployed
# swarm hitting this the moment that node shipped. A host
# whose token was minted before this scope was added won't
# pick it up automatically — the `[ ! -s "$token_path" ]`
# guard above only mints when the file is absent, by design
# (see its own comment) — so an existing deployment needs
# its token file deleted to force a re-mint with the new
# scope.
#
# Token name carries a timestamp suffix rather than the
# fixed `swarm-controller-boot` this used to be — a real
# incident, not a hypothetical: deleting the token *file*
# (the remediation above) doesn't delete the token Forgejo
# already has under that name, so re-running this unit hit
# "Command error: access token name has been used already"
# and failed outright. Same fix `hive-c0re::forge::users::
# mint_token` already uses for its own agent tokens, for
# the identical reason — a monotonic suffix means a re-mint
# can never collide with whatever this account already has.
out=$(forgejo admin user generate-access-token \
--username ${lib.escapeShellArg swarmControllerForgeUser} \
--token-name "swarm-controller-boot-$(date +%s)" \
--scopes "write:repository,write:organization,write:issue,read:user,write:admin")
token=$(printf '%s' "$out" | grep -oE '[0-9a-f]{32,}' | head -n1)
if [ -z "$token" ]; then
echo "no token-shaped word in forgejo's generate-access-token output" >&2
exit 1
fi
atomic_write_secret 0600 "" "$token_path" "$token"
echo "minted forge access token for ${swarmControllerForgeUser}"
fi fi
''; ''
+ (
if !havePublisherIdentity then
''
echo "the forge's host has no forge-publisher store identity, so the swarm-controller's forge token cannot be published: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2
exit 1
''
else
''
err="$(mktemp)"
body="$(mktemp)"
trap 'rm -f "$err" "$body"' EXIT
bao_failed() {
echo "$1" >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
}
# The revocation below goes through the forge's admin API,
# which refuses a token of an account that is not a site
# admin; each retry would then leave one more token live.
if ! forgejo admin user list --admin | grep -qE "[[:space:]]${swarmControllerForgeUser}[[:space:]]"; then
echo "forge account ${swarmControllerForgeUser} is not a site admin; promote it by hand. No token minted." >&2
exit 1
fi
# Before the mint, so a store that is down fails the attempt
# before forgejo holds a token nothing can read.
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
bao_failed "could not log in to swarm-bao with the forge publisher's certificate; no forge token minted for ${swarmControllerForgeUser}."
fi
export BAO_TOKEN
# `write:admin`: the controller's `CreateForgeUser` jobq node
# (`swarm-controller::forge::Client::ensure_agent_user`) and
# the revocation below call the admin API.
#
# Timestamped: forgejo refuses a name the account already has
# a token under, and the earlier tokens are still there until
# the revocation below. `hive-c0re::forge::users::mint_token`
# names its tokens the same way.
name="swarm-controller-boot-$(date +%s)"
out=$(forgejo admin user generate-access-token \
--username ${lib.escapeShellArg swarmControllerForgeUser} \
--token-name "$name" \
--scopes "write:repository,write:organization,write:issue,read:user,write:admin")
token=$(printf '%s' "$out" | grep -oE '[0-9a-f]{32,}' | head -n1)
if [ -z "$token" ]; then
echo "no token-shaped word in forgejo's generate-access-token output" >&2
exit 1
fi
# On stdin: the token is never an argument of any process.
if ! printf '%s' "$token" | bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=- >/dev/null 2>"$err"; then
bao_failed "swarm-bao refused the write to ${controllerTokenStorePath}; the next attempt mints another token and revokes this one."
fi
echo "published the forge token $name for ${swarmControllerForgeUser} to ${controllerTokenStorePath}"
# One call to the forge's API as the new token. Prints the
# HTTP status; the body lands in "$body". The token reaches
# curl as a header file, never as an argument.
forge() {
local method="$1" path="$2"
curl --silent --show-error --connect-timeout 5 --max-time 15 \
-H @<(printf 'Authorization: token %s\n' "$token") \
-X "$method" -o "$body" -w '%{http_code}' \
"http://127.0.0.1:${toString cfg.httpPort}/api/v1$path"
}
# Every other token this unit minted: one whose write to the
# store failed, and the one the store held before the marker
# was lost. The bare `swarm-controller-boot` is the name the
# unit minted under before the timestamp suffix. A failure
# here leaves the marker unwritten, so the retry mints again
# and repeats the revocation.
stale=()
page=1
while :; do
status="$(forge GET "/admin/users/${swarmControllerForgeUser}/tokens?limit=50&page=$page")"
if [ "$status" != 200 ]; then
echo "the forge answered $status to listing ${swarmControllerForgeUser}'s tokens:" >&2
cat "$body" >&2
exit 1
fi
[ "$(jq length "$body")" -gt 0 ] || break
while IFS= read -r n; do
case "$n" in
"$name") ;;
swarm-controller-boot | swarm-controller-boot-*) stale+=("$n") ;;
esac
done < <(jq -r '.[].name' "$body")
page=$((page + 1))
done
for n in "''${stale[@]}"; do
status="$(forge DELETE "/admin/users/${swarmControllerForgeUser}/tokens/$n")"
case "$status" in
204 | 404) ;;
*)
echo "the forge answered $status to revoking ${swarmControllerForgeUser}'s token $n:" >&2
cat "$body" >&2
exit 1
;;
esac
done
echo "revoked ''${#stale[@]} earlier forge token(s) of ${swarmControllerForgeUser}"
rm -f ${lib.escapeShellArg oldSwarmControllerTokenPath}
printf '%s\n' "$name" >"$name_path.tmp"
mv -f "$name_path.tmp" "$name_path"
''
);
}; };
}; };
}; };
@ -1089,9 +1199,9 @@ in
# read both fail loudly, and `storeRetry` retries until the publisher on # read both fail loudly, and `storeRetry` retries until the publisher on
# authelia's host has written the secret. # authelia's host has written the secret.
# #
# On the host, where that leaf is: the container holds no store identity. # On the host, which reads that leaf from its own files. A host with no
# A host with no publisher leaf fails here, naming the options, rather than # publisher leaf fails here, naming the options, rather than rendering no
# rendering no unit. # unit.
systemd.services.swarm-bao-forge-oidc = { systemd.services.swarm-bao-forge-oidc = {
description = "fetch the forge's OIDC client secret from the swarm secret store"; description = "fetch the forge's OIDC client secret from the swarm secret store";
# Ordered after the store where it runs here # Ordered after the store where it runs here
@ -1185,122 +1295,6 @@ in
systemd.timers.swarm-bao-forge-oidc = storeRefreshTimer "re-read the forge's OIDC client secret from the swarm secret store"; systemd.timers.swarm-bao-forge-oidc = storeRefreshTimer "re-read the forge's OIDC client secret from the swarm secret store";
# The reverse direction of the unit above: publish the swarm-controller's
# freshly-minted forge token OUT of the container into the swarm secret
# store, where the controller reads it wherever it runs. On the host and
# under the forge's own store identity for the reason the unit above gives,
# so nothing here asks where the store or the controller is.
#
# It runs at boot and on a switch that adds or changes it
# (`multi-user.target`), each time the container starts, and each time the
# mint writes the file
# (`systemd.paths.swarm-bao-forge-publisher` below). The mint is a oneshot in
# the container's own systemd, invisible to this one by unit name, so a
# missing file is not an error: the path unit runs this again when the mint
# writes it, the shape of `swarm-secret-publish`.
#
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
# every later start into a no-op, so neither trigger would run it again.
#
# Unconditional, like the in-container unit it publishes from: forge is a
# swarm-wide singleton, so the token always gets minted and published,
# regardless of whether swarm-controller runs on this host, another host,
# or nowhere in this swarm at all. A host with no publisher leaf fails
# here, naming the options, rather than rendering no unit.
systemd.services.swarm-bao-forge-publisher = {
description = "publish the swarm-controller's forge token to the swarm secret store";
# Also ordered after the store where it runs here:
# ../glue-bao-readers-store-order.nix.
after = [
"container@hive-forge.service"
];
wantedBy = [
"multi-user.target"
"container@hive-forge.service"
];
path = [
deployCfg.bao.package
pkgs.coreutils
];
# ../lib/store-retry.nix: a sealed or unreachable store is retried.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
SyslogIdentifier = "swarm-bao-forge-publisher";
# Bounded here so the number a switch waits on is in this file. A sealed
# store answers on the port and never answers the write.
TimeoutStartSec = 60;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs havePublisherIdentity {
BAO_CLIENT_CERT = deployCfg.bao.forgePublisherClientCertFile;
BAO_CLIENT_KEY = deployCfg.bao.forgePublisherClientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
script =
if !havePublisherIdentity then
''
echo "the forge's host has no forge-publisher store identity, so it cannot publish the swarm-controller's forge token: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2
exit 1
''
else
''
set -euo pipefail
src=${lib.escapeShellArg swarmControllerTokenHostPath}
if [ ! -s "$src" ]; then
echo "the forge has not minted $src yet; the path unit publishes it when it does" >&2
exit 0
fi
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with the forge publisher's certificate; the swarm-controller's forge token is not published." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# `value=@$src` hands bao the PATH: bao opens the file itself, so
# the token is never an argument of this process.
if ! bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=@"$src" >/dev/null 2>"$err"; then
echo "swarm-bao refused the write to ${controllerTokenStorePath}." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
echo "published the swarm-controller's forge token to ${controllerTokenStorePath}"
'';
};
# Re-publish when the mint unit writes a token, which it can do without the
# container restarting. The file is the event: the mint runs in the
# container's own systemd, which this one cannot subscribe to.
systemd.paths.swarm-bao-forge-publisher = {
description = "watch the swarm-controller's forge token for a publish";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = swarmControllerTokenHostPath;
Unit = "swarm-bao-forge-publisher.service";
};
};
systemd.tmpfiles.rules = [ systemd.tmpfiles.rules = [
# The OIDC secret's directory, which the container binds at start. # The OIDC secret's directory, which the container binds at start.
"d ${oidcSecretDir} 0700 root root -" "d ${oidcSecretDir} 0700 root root -"

19
nix/host-modules/swarm-bao.nix
View file

@ -1107,9 +1107,10 @@ let
} }
]; ];
# The forge host's one store identity. `swarm-bao-forge-publisher` writes # The forge host's one store identity. The forge's
# the controller's forge token under it, the one writer of that path and # `forgejo-swarm-controller-account`, in its container, writes the
# write-only: it copies a file in and never reads one back. # controller's forge token under it, the one writer of that path and
# write-only: it puts the token it mints and never reads one back.
# `swarm-bao-forge-oidc` reads the forge's own OIDC client secret, and the # `swarm-bao-forge-oidc` reads the forge's own OIDC client secret, and the
# journal forwarder in its container its own, those two leaves and no other # journal forwarder in its container its own, those two leaves and no other
# service's. # service's.
@ -2016,8 +2017,9 @@ in
default = "swarm-forge-publisher"; default = "swarm-forge-publisher";
description = '' description = ''
Subject the store's `swarm-forge-publisher` cert-auth role accepts: the Subject the store's `swarm-forge-publisher` cert-auth role accepts: the
forge host's one store identity, presented by `swarm-bao-forge-publisher`, forge host's one store identity, presented by
which publishes the swarm-controller's forge token, and by `forgejo-swarm-controller-account` in the forge's container, which
mints and publishes the swarm-controller's forge token, by
`swarm-bao-forge-oidc`, which fetches the forge's OIDC client secret, `swarm-bao-forge-oidc`, which fetches the forge's OIDC client secret,
and by the journal forwarder in the forge's container. Its grant is and by the journal forwarder in the forge's container. Its grant is
three paths: `swarm/controller/swarm-controller/forge/token`, write three paths: `swarm/controller/swarm-controller/forge/token`, write
@ -2342,9 +2344,10 @@ in
default = null; default = null;
example = "/var/lib/swarm-bao-pki/forge-publisher.pem"; example = "/var/lib/swarm-bao-pki/forge-publisher.pem";
description = '' description = ''
Certificate the forge's host presents to the store, both to publish the Certificate the forge's host presents to the store, to fetch the
swarm-controller's forge token and to fetch the forge's OIDC client forge's OIDC client secret, and that the forge's container presents to
secret. Its subject must be publish the swarm-controller's forge token and to fetch its journal
forwarder's secret. Its subject must be
{option}`services.hyperhive.deploy.bao.forgePublisherCommonName`. {option}`services.hyperhive.deploy.bao.forgePublisherCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host. ./glue-bao-tls.nix points it at the leaf it mints on the store's host.

5
nix/host-modules/swarm-controller.nix
View file

@ -305,8 +305,9 @@ in
'' ''
The controller reads its forge token from the swarm's secret store, under The controller reads its forge token from the swarm's secret store, under
its own certificate (deploy.swarm-controller.baoClientCertFile), and its own certificate (deploy.swarm-controller.baoClientCertFile), and
swarm-bao-forge-publisher on the forge's host puts it there. Remove this forgejo-swarm-controller-account in the forge's container puts it
definition; the file it named is read by nothing now and can be deleted. there. Remove this definition; the file it named is read by nothing now
and can be deleted.
'' ''
) )
]; ];

12
nix/module-eval/bao-grants.nix
View file

@ -1736,10 +1736,11 @@ let
&& lib.hasInfix "swarm-review-classifier \"\" clientAuth" s; && lib.hasInfix "swarm-review-classifier \"\" clientAuth" s;
} }
{ {
# The forge's host is the one writer of the controller's forge token, # The forge's mint, in its container, is the one writer of the
# and reads the forge's own OIDC client secret and its container # controller's forge token, and the forge's host and its container
# forwarder's under the same leaf. Pinned as the whole list, so an added # forwarder read their OIDC client secrets under the same leaf. Pinned
# `read` on the token or a write or wildcard on a secret fails. # as the whole list, so an added `read` on the token or a write or
# wildcard on a secret fails.
name = "the forge publisher's grant is writing the controller's forge token and reading the forge's and its forwarder's OIDC secrets, and nothing else"; name = "the forge publisher's grant is writing the controller's forge token and reading the forge's and its forwarder's OIDC secrets, and nothing else";
ok = ok =
let let
@ -1780,7 +1781,8 @@ let
} }
{ {
# The policy units that name the path are the whole list of who reaches # The policy units that name the path are the whole list of who reaches
# it: the publisher, which writes it, and the controller, which reads it. # it: the forge publisher's, which the mint writes it under, and the
# controller's, which reads it.
name = "no granting unit but the controller's and the forge publisher's names the controller's forge token"; name = "no granting unit but the controller's and the forge publisher's names the controller's forge token";
ok = ok =
let let

8
nix/module-eval/core-toggle.nix
View file

@ -123,11 +123,11 @@ let
&& centralToggleOff.services.hyperhive.swarm.forge.publicUrl == "https://forge.t.local"; && centralToggleOff.services.hyperhive.swarm.forge.publicUrl == "https://forge.t.local";
} }
{ {
# The controller's token publisher follows where the forge runs # The controller's token mint follows where the forge runs
# (./forge-placement.nix), never the central toggle: a forge host with # (./forge-placement.nix), never the central toggle: a forge host with
# the toggle off still publishes it. # the toggle off still mints and publishes it.
name = "the forge's controller-token publisher does not consult the central toggle"; name = "the forge's controller-token mint does not consult the central toggle";
ok = forgeHere.systemd.services ? swarm-bao-forge-publisher; ok = forgeHere.containers.hive-forge.config.systemd.services ? forgejo-swarm-controller-account;
} }
{ {
name = "a hive that does not host the swarm's shared services runs none of them"; name = "a hive that does not host the swarm's shared services runs none of them";

109
nix/module-eval/forge-placement.nix
View file

@ -107,7 +107,7 @@ let
# ./grafana.nix's `grafanaRefusedFor`. # ./grafana.nix's `grafanaRefusedFor`.
refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions; refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions;
publisher = m: m.systemd.services.swarm-bao-forge-publisher; mint = m: m.containers.hive-forge.config.systemd.services.forgejo-swarm-controller-account;
oidcFetch = m: m.systemd.services.swarm-bao-forge-oidc; oidcFetch = m: m.systemd.services.swarm-bao-forge-oidc;
forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings; forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings;
@ -125,7 +125,6 @@ let
ok = ok =
!bare.services.hyperhive.deploy.forgejo.enable !bare.services.hyperhive.deploy.forgejo.enable
&& !(bare.containers ? hive-forge) && !(bare.containers ? hive-forge)
&& !(bare.systemd.services ? swarm-bao-forge-publisher)
&& !(lib.elem "forge.t.local" bare.services.hyperhive.gateway.localNames); && !(lib.elem "forge.t.local" bare.services.hyperhive.gateway.localNames);
} }
{ {
@ -327,55 +326,93 @@ let
&& refusedOver ciUnderForgeNoIdentity "services.hyperhive.deploy.bao.enable"; && refusedOver ciUnderForgeNoIdentity "services.hyperhive.deploy.bao.enable";
} }
{ {
# The forge's host publishes the controller's token under its own leaf, # The mint, in the forge's container, writes the controller's token to
# which the store mints beside it here. `value=@` is what keeps the token # the store under the forge host's leaf, loaded as a credential. On
# out of the process's arguments. # stdin, so the token is no process's argument; the copy an earlier mint
name = "the forge's host publishes the controller's forge token to the store under its own leaf"; # kept on the container's disk goes once a token is published, and no
# host unit reads into the container.
name = "the forge's mint writes the controller's forge token straight to the store and keeps no copy";
ok = ok =
let let
u = publisher allLocal; u = mint allLocal;
cred = id: "/run/credentials/forgejo-swarm-controller-account.service/${id}";
in in
u.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/forge-publisher.pem" lib.elem "forge-bao-cert" u.serviceConfig.LoadCredential
&& u.environment.BAO_CLIENT_KEY == "/var/lib/swarm-bao-pki/forge-publisher-key.pem" && lib.elem "forge-bao-key" u.serviceConfig.LoadCredential
&& lib.hasInfix "bao kv put secret/swarm/controller/swarm-controller/forge/token value=@\"$src\"" u.script && u.environment.BAO_CLIENT_CERT == cred "forge-bao-cert"
&& lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token" u.script && u.environment.BAO_CLIENT_KEY == cred "forge-bao-key"
&& lib.elem "container@hive-forge.service" u.wantedBy && lib.elem "--load-credential=forge-bao-cert:/var/lib/swarm-bao-pki/forge-publisher.pem" allLocal.containers.hive-forge.extraFlags
# The deploy that adds the unit starts it only through a target. && lib.elem "--load-credential=forge-bao-cert:/etc/pki/forge-publisher.pem" forgeAlone.containers.hive-forge.extraFlags
&& lib.elem "multi-user.target" u.wantedBy && lib.hasInfix "printf '%s' \"$token\" | bao kv put secret/swarm/controller/swarm-controller/forge/token value=-" u.script
&& !(lib.hasInfix "value=@" u.script)
&& !(lib.hasInfix "atomic_write_secret" u.script)
&& lib.hasInfix "rm -f /var/lib/forgejo/swarm-controller-token\n" u.script
&& !(allLocal.systemd.services ? swarm-bao-forge-publisher)
&& !(allLocal.systemd.paths ? swarm-bao-forge-publisher)
&& !(forgeAlone.systemd.services ? swarm-bao-forge-publisher)
&& !(allLocal.systemd.services ? hive-forge-swarm-controller-token); && !(allLocal.systemd.services ? hive-forge-swarm-controller-token);
} }
{ {
# What makes it re-run. A oneshot left `active (exited)` makes every later # The order that keeps a token live only if the store has it: log in
# start a no-op, so `RemainAfterExit` would silence every trigger: the # before the mint, put, revoke every other `swarm-controller-boot-*`
# targets and the container's start (`wantedBy`, above) and a write of # token, and only then write the marker, which holds the name. A marker
# the token file. A missing file exits 0, so a switch with the container # present skips all of it.
# down does not fail. name = "the forge's mint writes its marker only after the put and the revocation of every earlier token";
name = "the forge publisher re-runs on each container start and each write of the token file";
ok = ok =
let let
p = allLocal.systemd.paths.swarm-bao-forge-publisher; s = (mint allLocal).script;
at = needle: lib.stringLength (lib.head (lib.splitString needle s));
order = l: lib.length l < 2 || (at (lib.elemAt l 0) < at (lib.elemAt l 1) && order (lib.tail l));
steps = [
"if [ -s \"$name_path\" ]; then"
"forgejo admin user list --admin"
"bao login -method=cert"
"forgejo admin user generate-access-token"
"bao kv put"
"forge GET \"/admin/users/swarm-controller/tokens?limit=50&page=$page\""
"forge DELETE \"/admin/users/swarm-controller/tokens/$n\""
"rm -f /var/lib/forgejo/swarm-controller-token\n"
"mv -f \"$name_path.tmp\" \"$name_path\""
];
in in
!((publisher allLocal).serviceConfig.RemainAfterExit or false) lib.all (step: lib.hasInfix step s) steps
&& && order steps
p.pathConfig.PathChanged && lib.hasInfix "name_path=/var/lib/forgejo/swarm-controller-token-name" s
== "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token" && lib.hasInfix "printf '%s\\n' \"$name\" >\"$name_path.tmp\"" s
&& p.pathConfig.Unit == "swarm-bao-forge-publisher.service" && lib.hasInfix "swarm-controller-boot | swarm-controller-boot-*) stale+=(\"$n\") ;;" s
&& lib.hasInfix "the path unit publishes it when it does" (publisher allLocal).script && lib.hasInfix "-H @<(printf 'Authorization: token %s\\n' \"$token\")" s;
&& lib.hasInfix "exit 0" (publisher allLocal).script }
&& lib.elem "multi-user.target" p.wantedBy {
&& !(bare.systemd.paths ? swarm-bao-forge-publisher); # A store that is down retries the mint (../host-modules/lib/store-retry.nix),
# and forgejo never waits for it.
name = "the forge's mint retries on failure, and forgejo does not wait for it";
ok =
let
c = allLocal.containers.hive-forge.config.systemd.services;
u = mint allLocal;
in
u.serviceConfig.Restart == "on-failure"
&& u.startLimitBurst == 2880
&& u.serviceConfig.RemainAfterExit
&& lib.elem "forgejo.service" u.after
&& !(lib.elem "forgejo-swarm-controller-account.service" (
c.forgejo.after ++ c.forgejo.requires ++ c.forgejo.wants
))
&& lib.elem "forgejo-swarm-controller-account.service" c.swarm-bridge-dns.before;
} }
{ {
# The absence arm: a forge host without the store has no leaf unless one # The absence arm: a forge host without the store has no leaf unless one
# is placed, and the unit says which options to set. # is placed, and the mint says which options to set before it mints.
name = "a forge host with no publisher leaf renders a publisher that fails naming the options"; name = "a forge host with no publisher leaf renders a mint that fails naming the options, minting nothing";
ok = ok =
let let
s = (publisher forgeHere).script; u = mint forgeHere;
in in
lib.hasInfix "forgePublisherClientCertFile" s !(u.environment ? BAO_CLIENT_CERT)
&& lib.hasInfix "exit 1" s && lib.hasInfix "forgePublisherClientCertFile" u.script
&& !(lib.hasInfix "bao kv put" s); && lib.hasInfix "exit 1" u.script
&& !(lib.hasInfix "forgejo admin user generate-access-token" u.script)
&& !(lib.hasInfix "bao kv put" u.script);
} }
{ {
# The forge's OIDC secret has one route, the store, under the forge # The forge's OIDC secret has one route, the store, under the forge

3
nix/module-eval/swarm-services-switch.nix
View file

@ -819,7 +819,6 @@ let
]; ];
readers = [ readers = [
"swarm-bao-forge-oidc" "swarm-bao-forge-oidc"
"swarm-bao-forge-publisher"
"swarm-bao-matrix-oidc" "swarm-bao-matrix-oidc"
"swarm-bao-hive-otel-oidc" "swarm-bao-hive-otel-oidc"
"swarm-bao-grafana-oidc" "swarm-bao-grafana-oidc"
@ -848,7 +847,7 @@ let
!(lib.any (n: lib.elem n (s.after ++ s.wants ++ s.requires)) storeUnits); !(lib.any (n: lib.elem n (s.after ++ s.wants ++ s.requires)) storeUnits);
in in
lib.all waits readers lib.all waits readers
&& lib.length readersElsewhere == 8 && lib.length readersElsewhere == 7
&& lib.all namesNoStoreUnit readersElsewhere && lib.all namesNoStoreUnit readersElsewhere
&& !(lib.any (u: storeOnlyNoHive.systemd.services ? ${u}) readers); && !(lib.any (u: storeOnlyNoHive.systemd.services ? ${u}) readers);
} }

10
swarm-controller/src/forge.rs
View file

@ -8,9 +8,8 @@
//! (`SWARM_CONTROLLER_FORGE_URL`). The token comes from the swarm's secret //! (`SWARM_CONTROLLER_FORGE_URL`). The token comes from the swarm's secret
//! store, at `swarm_secret_client::forge::controller_token_path`, read under //! store, at `swarm_secret_client::forge::controller_token_path`, read under
//! this daemon's own certificate and held only in memory. The forge mints it //! this daemon's own certificate and held only in memory. The forge mints it
//! inside its container (`hive-forge/default.nix`'s //! inside its container and puts it there (`hive-forge/default.nix`'s
//! `forgejo-swarm-controller-account`) and `swarm-bao-forge-publisher`, on the //! `forgejo-swarm-controller-account`).
//! forge's host, puts it there.
//! //!
//! [`enable`] fills the daemon's one client slot: at start when the token is //! [`enable`] fills the daemon's one client slot: at start when the token is
//! there, later from a background re-read when it is not. Every caller reads //! there, later from a background re-read when it is not. Every caller reads
@ -241,8 +240,9 @@ async fn fetch_token() -> Result<String> {
match stored { match stored {
Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()), Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()),
_ => anyhow::bail!( _ => anyhow::bail!(
"nothing at {path}: swarm-bao-forge-publisher, on the host that runs the forge, has \ "nothing at {path}: forgejo-swarm-controller-account, in the forge's container, has \
not published this daemon's forge token" not published this daemon's forge token (if the store lost it, delete /var/lib/forgejo/\
swarm-controller-token-name in that container and restart the unit)"
), ),
} }
} }

12
swarm-secret-client/src/forge.rs
View file

@ -14,8 +14,8 @@
//! beside [`crate::matrix`]. //! beside [`crate::matrix`].
//! //!
//! `swarm-controller`'s own forge token ([`controller_token_path`]) is the //! `swarm-controller`'s own forge token ([`controller_token_path`]) is the
//! fourth: the forge mints it, `swarm-bao-forge-publisher` on the forge's host //! fourth: the forge's `forgejo-swarm-controller-account`, in its container,
//! writes it here, and the controller reads it as it starts. //! mints it and writes it here, and the controller reads it as it starts.
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
@ -82,14 +82,14 @@ pub fn controller_token_path() -> Result<String, Error> {
} }
/// What [`controller_token_path`] holds. No token name beside the value, unlike /// What [`controller_token_path`] holds. No token name beside the value, unlike
/// [`Credential`]: the publisher copies a file that holds the token alone. /// [`Credential`]: the mint writes the token alone.
/// ///
/// No `Debug` derive: `value` is the forge admin credential. /// No `Debug` derive: `value` is the forge admin credential.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ControllerToken { pub struct ControllerToken {
/// The token itself. `swarm-bao-forge-publisher` writes it with /// The token itself. `forgejo-swarm-controller-account` writes it with
/// `bao kv put … value=@<file>`, so this name is load-bearing for a writer /// `bao kv put … value=-`, so this name is load-bearing for a writer this
/// this crate does not control. /// crate does not control.
pub value: String, pub value: String,
} }