vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-controller reads its forge token from the store, not a host file

Browse source 
The controller's forge token never entered bao: the forge minted it in its
container, hive-forge-swarm-controller-token copied it to a host file, and the
controller LoadCredential'ed that file. forgeTokenFile defaulted to that path
only where the forge ran, so a controller on any other host had no forge
access unless an operator copied the file by hand.

Now the forge's host publishes it and the controller reads it:

- swarm-bao-forge-publisher (replaces hive-forge-swarm-controller-token)
  waits for the in-container token, logs in with its own forge-publisher leaf
  and writes it to swarm/controller/swarm-controller/forge/token. It runs on
  every forge container start, the old unit's trigger.
- New principal swarm-forge-publisher: leaf minted by glue-bao-tls.nix,
  policy create/update on that one path, written by
  swarm-bao-forge-publisher-policy, CN reserved as a hive name.
- The controller policy gains read on that path. swarm-controller reads it
  at start under its own certificate with the OIDC secret's retry, and runs
  without forge access if it still cannot.
- forgeTokenFile, its LoadCredential and the forgeEnv gate are gone;
  SWARM_CONTROLLER_FORGE_URL is always set. hostSwarmControllerTokenFile is
  removed, and a tmpfiles rule deletes the old host copy on the forge host.

The in-container file stays: the mint unit's only-once guard reads it, and
it is the publisher's source.

Refs #4998
This commit is contained in:
atlas 2026-10-08 13:23:17 +02:00
commit 121c4678aa
18 changed files with 580 additions and 277 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/swarm/credentials.md
View file

@ -56,11 +56,12 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts | ❌ the controller reads it at start, so a replaced token reaches it with the next restart |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |

24
docs/swarm/secrets.md
View file

@ -52,6 +52,7 @@ neither is a renaming of the other.
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
@ -382,32 +383,37 @@ next controller restart.
### Per-principal identities ### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. These five units read one path identity and sharing one merges the identities. These six units reach one path
each and each holds a leaf, a role and a policy of its own: each and each holds a leaf, a role and a policy of its own:
| unit | option pair under `deploy.bao.` | reads | | unit | option pair under `deploy.bao.` | reads, or writes |
| -------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- | | --------------------------- | -------------------------------------------------------------- | -------------------------------------------------------- |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` | | `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` | | `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` | | `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` | | `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` |
| `swarm-bao-forge-publisher` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token` |
The first exists **per hive**, because the path it reads carries a hive name The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is and every hive runs its own reader. Its subject is
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that `<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
composed spelling as a hive name, so nobody can name a hive into another hive's composed spelling as a hive name, so nobody can name a hive into another hive's
role. The other four read a path that names a swarm service rather than a hive, role. The other five reach a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`, `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName` and `deploy.bao.ciRunnerCommonName`. `deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName` and
`deploy.bao.forgePublisherCommonName`. The last is the only writer: the other
five read.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all five and defaults On a host that mints its own PKI, `glue-bao-tls.nix` signs all six and defaults
all ten options, and there is nothing to do. Elsewhere you issue each leaf from all twelve options, and there is nothing to do. Elsewhere you issue each leaf from
that CA out of band and name it here — one file per principal rather than one file that CA out of band and name it here — one file per principal rather than one file
shared by five, which is the whole of what this buys. `swarm-ci-runner.nix` shared by six, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as refuses the build wherever the runner runs without its pair, as
`swarm-grafana.nix` does for Grafana. `swarm-grafana.nix` does for Grafana. `swarm-bao-forge-publisher` refuses
nothing: without its pair it renders, fails at start naming both options, and
the controller runs without forge access until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of On a host without the store, where the forwarder never runs, forgetting one of
the other six isn't a quiet degrade. A hand-configured remote-store hive that the other six isn't a quiet degrade. A hand-configured remote-store hive that

20
nix/host-modules/deploy.nix
View file

@ -70,10 +70,12 @@ in
[ "services" "hyperhive" "swarm" "controller" "enable" ] [ "services" "hyperhive" "swarm" "controller" "enable" ]
[ "services" "hyperhive" "deploy" "swarm-controller" "enable" ] [ "services" "hyperhive" "deploy" "swarm-controller" "enable" ]
) )
(lib.mkRenamedOptionModule (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "controller" "forgeTokenFile" ] ''
[ "services" "hyperhive" "swarm" "controller" "forgeTokenFile" ] The controller reads its forge token from the swarm's secret store, under
[ "services" "hyperhive" "deploy" "swarm-controller" "forgeTokenFile" ] its own certificate, and swarm-bao-forge-publisher on the forge's host
) puts it there. Remove this definition; the file it named is read by
nothing now and can be deleted.
'')
(lib.mkRenamedOptionModule (lib.mkRenamedOptionModule
[ "services" "hyperhive" "swarm" "controller" "authBridgeUrl" ] [ "services" "hyperhive" "swarm" "controller" "authBridgeUrl" ]
[ "services" "hyperhive" "deploy" "swarm-controller" "authBridgeUrl" ] [ "services" "hyperhive" "deploy" "swarm-controller" "authBridgeUrl" ]
@ -231,9 +233,13 @@ in
[ "services" "hyperhive" "swarm" "forge" "sso" "clientSecretFile" ] [ "services" "hyperhive" "swarm" "forge" "sso" "clientSecretFile" ]
[ "services" "hyperhive" "deploy" "forgejo" "sso" "clientSecretFile" ] [ "services" "hyperhive" "deploy" "forgejo" "sso" "clientSecretFile" ]
) )
(lib.mkRenamedOptionModule (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "forge" "hostSwarmControllerTokenFile" ]
[ "services" "hyperhive" "swarm" "forge" "hostSwarmControllerTokenFile" ] ''
[ "services" "hyperhive" "deploy" "forgejo" "hostSwarmControllerTokenFile" ] The forge publishes the swarm-controller's forge token to the swarm
secret store (swarm-bao-forge-publisher), and the controller reads it
there. Remove this definition; the file it named is read by nothing now
and can be deleted.
''
) )
# The homeserver, split the same way. `swarm.matrix` keeps what it IS to # The homeserver, split the same way. `swarm.matrix` keeps what it IS to

2
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -48,6 +48,8 @@ let
swarm-bao-nats-tls = deployCfg.nats.enable; swarm-bao-nats-tls = deployCfg.nats.enable;
# ./swarm-ci-runner.nix # ./swarm-ci-runner.nix
swarm-bao-ci-runner = deployCfg.ci-runner.enable; swarm-bao-ci-runner = deployCfg.ci-runner.enable;
# ./hive-forge/default.nix
swarm-bao-forge-publisher = deployCfg.forgejo.enable;
}; };
# Readers that log in under another unit's principal: reader unit → the # Readers that log in under another unit's principal: reader unit → the

8
nix/host-modules/glue-bao-tls.nix
View file

@ -107,6 +107,8 @@ in
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem"; forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
ciRunnerClientCertFile = lib.mkDefault "${pkiDir}/ci-runner.pem"; ciRunnerClientCertFile = lib.mkDefault "${pkiDir}/ci-runner.pem";
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem"; ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem"; servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
servicesIssuerClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem"; servicesIssuerClientKeyFile = lib.mkDefault "${pkiDir}/services-issuer-key.pem";
@ -266,6 +268,12 @@ in
[ -s ${pkiDir}/ci-runner.pem ] || ${signLeaf} ${pkiDir} ci-runner \ [ -s ${pkiDir}/ci-runner.pem ] || ${signLeaf} ${pkiDir} ci-runner \
${lib.escapeShellArg deployCfg.bao.ciRunnerCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.ciRunnerCommonName} "" clientAuth
# The forge's token publisher. Minted here for the matrix-ctl leaf's
# reason: the forge is a swarm singleton, so on a forge host without the
# store this is the file an operator copies.
[ -s ${pkiDir}/forge-publisher.pem ] || ${signLeaf} ${pkiDir} forge-publisher \
${lib.escapeShellArg deployCfg.bao.forgePublisherCommonName} "" clientAuth
${lib.optionalString deployCfg.hive-controller.enable '' ${lib.optionalString deployCfg.hive-controller.enable ''
# This hive's gateway-leaf identity, which opens # This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its # `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its

155
nix/host-modules/hive-forge/default.nix
View file

@ -56,6 +56,16 @@ let
# `forgeSecretPath` above. # `forgeSecretPath` above.
swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token"; swarmControllerTokenPath = "/var/lib/forgejo/swarm-controller-token";
# Where `swarm-bao-forge-publisher` puts it: `controllerForgeTokenLeaf` in
# ../swarm-bao.nix, under the KV mount.
controllerTokenStorePath = "secret/swarm/controller/swarm-controller/forge/token";
baoCfg = config.services.hyperhive.swarm.bao;
storeRetry = import ../lib/store-retry.nix { };
havePublisherIdentity =
deployCfg.bao.forgePublisherClientCertFile != null
&& deployCfg.bao.forgePublisherClientKeyFile != null;
caTrust = import ../lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; caTrust = import ../lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
# ROOT_URL forgejo advertises in clone links + outbound URLs. # ROOT_URL forgejo advertises in clone links + outbound URLs.
@ -101,7 +111,18 @@ in
# `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/` survives # `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/` survives
# restart. See `docs/networking/gateway.md::hive-forge container shape`. # restart. See `docs/networking/gateway.md::hive-forge container shape`.
imports = [ ./service.nix ]; imports = [
./service.nix
(lib.mkRemovedOptionModule
[ "services" "hyperhive" "deploy" "forgejo" "hostSwarmControllerTokenFile" ]
''
The forge publishes the swarm-controller's forge token to the swarm
secret store (swarm-bao-forge-publisher), and the controller reads it
there. Remove this definition; the file it named is read by nothing now
and can be deleted.
''
)
];
# What ./service.nix declares is what the forge IS from any hive's point of # What ./service.nix declares is what the forge IS from any hive's point of
# view: the names and ports it answers on, the URLs it advertises, and the # view: the names and ports it answers on, the URLs it advertises, and the
@ -226,28 +247,6 @@ in
eval error. eval error.
''; '';
}; };
hostSwarmControllerTokenFile = lib.mkOption {
type = lib.types.str;
default = "/var/lib/hyperhive-forge/swarm-controller.token";
description = ''
Host path where this forge deposits the freshly-minted forge
access token for the swarm's `swarm-controller` account (see
`systemd.services.forgejo-swarm-controller-account` inside the
forge container, and `hive-forge-swarm-controller-token` on the
host, which copies the token out).
Same role for this token as
`services.hyperhive.deploy.authelia.hostClientSecretDir` plays
for the OIDC secret: a **host**-local path (not inside any
container), read directly by `swarm-controller.nix`'s
`LoadCredential` when the controller runs on this same host.
On any other host the token has to get there somehow — copy it
out of this path with whatever secret management this
deployment already uses, the same shape `swarm.nix`'s own
`clientSecretFile` documents for the analogous cross-host case.
'';
};
}; };
config = lib.mkIf deployCfg.forgejo.enable { config = lib.mkIf deployCfg.forgejo.enable {
@ -1049,39 +1048,74 @@ in
''; '';
}; };
# The reverse direction of the unit above: collect the # The reverse direction of the unit above: publish the swarm-controller's
# swarm-controller's freshly-minted forge token OUT of the container # freshly-minted forge token OUT of the container into the swarm secret
# onto a host path other units/hosts can read. Same "only the host can # store, where the controller reads it wherever it runs. On the host for the
# see both trees" reasoning, same ordering constraint — this can only # reason `hive-forge-oidc-secret` gives, and under the forge's own store
# depend on the container being up (`container@hive-forge.service`), # identity, so nothing here asks where the store or the controller is.
# not on the specific in-container oneshot that mints the token
# (containers run their own systemd instance, invisible to this one by
# unit name) — so it polls the same bounded way
# `hive-forge-oidc-secret` does while waiting on authelia above.
# #
# Unconditional, like the in-container unit it collects from: forge is # It can only depend on the container being up, not on the in-container
# a swarm-wide singleton, so the token always gets minted and always # oneshot that mints the token (containers run their own systemd instance,
# gets collected here, regardless of whether swarm-controller runs on # invisible to this one by unit name), so it polls the same bounded way
# this host, another host, or nowhere in this swarm at all. # `hive-forge-oidc-secret` does. `wantedBy` the container, so every
systemd.services.hive-forge-swarm-controller-token = { # container start re-publishes whatever the mint unit left there.
description = "collect the swarm-controller's forge token onto the host"; #
after = [ "container@hive-forge.service" ]; # Unconditional, like the in-container unit it publishes from: forge is a
# swarm-wide singleton, so the token always gets minted and published,
# regardless of whether swarm-controller runs on this host, another host,
# or nowhere in this swarm at all. A host with no publisher leaf fails
# here, naming the options, rather than rendering no unit.
systemd.services.swarm-bao-forge-publisher = {
description = "publish the swarm-controller's forge token to the swarm secret store";
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job.
after = [
"container@hive-forge.service"
]
++ lib.optionals deployCfg.bao.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
wantedBy = [ "container@hive-forge.service" ]; wantedBy = [ "container@hive-forge.service" ];
serviceConfig = { path = [
deployCfg.bao.package
pkgs.coreutils
];
# ../lib/store-retry.nix: a sealed or unreachable store is retried, and
# so is a token the forge has not minted yet.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
SyslogIdentifier = "hive-forge-swarm-controller-token"; SyslogIdentifier = "swarm-bao-forge-publisher";
# Longer than the 120s bounded wait below. `DefaultTimeoutStartSec` # Longer than the 120s bounded wait below. `DefaultTimeoutStartSec`
# is 90s, so without this systemd kills the unit at 90 — before it # is 90s, so without this systemd kills the unit at 90 — before it
# can emit the message naming the file it was waiting for. # can emit the message naming the file it was waiting for.
TimeoutStartSec = "180s"; TimeoutStartSec = "180s";
}; };
path = [ pkgs.coreutils ]; environment = {
script = '' BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs havePublisherIdentity {
BAO_CLIENT_CERT = deployCfg.bao.forgePublisherClientCertFile;
BAO_CLIENT_KEY = deployCfg.bao.forgePublisherClientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
script =
if !havePublisherIdentity then
''
echo "the forge's host has no forge-publisher store identity, so it cannot publish the swarm-controller's forge token: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2
exit 1
''
else
''
set -euo pipefail set -euo pipefail
src=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"} src=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${swarmControllerTokenPath}"}
dst=${lib.escapeShellArg deployCfg.forgejo.hostSwarmControllerTokenFile}
for _ in $(seq 1 60); do for _ in $(seq 1 60); do
[ -s "$src" ] && break [ -s "$src" ] && break
@ -1092,10 +1126,41 @@ in
exit 1 exit 1
fi fi
install -D -m 0400 "$src" "$dst" err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with the forge publisher's certificate; the swarm-controller's forge token is not published." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
# `value=@$src` hands bao the PATH: bao opens the file itself, so
# the token is never an argument of this process.
if ! bao kv put ${lib.escapeShellArg controllerTokenStorePath} value=@"$src" >/dev/null 2>"$err"; then
echo "swarm-bao refused the write to ${controllerTokenStorePath}." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
echo "published the swarm-controller's forge token to ${controllerTokenStorePath}"
''; '';
}; };
# Unused: nothing writes or reads this host path, and it may hold a copy of
# the controller's forge token.
systemd.tmpfiles.rules = [ "r /var/lib/hyperhive-forge/swarm-controller.token" ];
networking.firewall = lib.mkIf deployCfg.forgejo.openFirewall { networking.firewall = lib.mkIf deployCfg.forgejo.openFirewall {
allowedTCPPorts = [ allowedTCPPorts = [
cfg.httpPort cfg.httpPort

73
nix/host-modules/swarm-bao.nix
View file

@ -369,9 +369,9 @@ let
# Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read` # Each hive's matrix sender token (`matrix_account::hive_sender`) grants `read`
# for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason. # for the same keep-if-live reason, and `+` for the same glob-only-as-last-segment reason.
# #
# The swarm appservice token and its own OIDC client secret, read-only: it # Three leaves it reads and never writes: the swarm appservice token
# uses both and writes neither. matrix-ctl publishes the token # (matrix-ctl, `matrixCtlPolicyText` below), its own OIDC client secret (the
# (`matrixCtlPolicyText` below), the publisher the secret. # publisher) and its own forge token (the forge's host, `forgePublisherWriters`).
# #
# The agent PKI grant is its only path on that mount: it can ask the one role # The agent PKI grant is its only path on that mount: it can ask the one role
# for a certificate, not write that role or reach the issuer. # for a certificate, not write that role or reach the issuer.
@ -445,6 +445,10 @@ let
capabilities = ["read"] capabilities = ["read"]
} }
path "${credentialMountPath}/data/${controllerForgeTokenLeaf}" {
capabilities = ["read"]
}
path "${agentPkiMountPath}/issue/${agentPkiRoleName}" { path "${agentPkiMountPath}/issue/${agentPkiRoleName}" {
capabilities = ["update"] capabilities = ["update"]
} }
@ -545,6 +549,11 @@ let
# it; the controller reads it and holds no other copy. # it; the controller reads it and holds no other copy.
controllerClientLeaf = "swarm/controller/swarm-controller/oidc/client"; controllerClientLeaf = "swarm/controller/swarm-controller/oidc/client";
# The controller's own forge token, the nix half of
# `swarm_secret_client::forge::controller_token_path`. The forge's host writes
# it; the controller reads it and holds no other copy.
controllerForgeTokenLeaf = "swarm/controller/swarm-controller/forge/token";
# The forge's runner registration token, the nix half of # The forge's runner registration token, the nix half of
# `swarm_secret_client::ci::registration_token_path`. The controller writes # `swarm_secret_client::ci::registration_token_path`. The controller writes
# it and cannot read it back (`forge::runner_token` keeps the value it last # it and cannot read it back (`forge::runner_token` keeps the value it last
@ -922,6 +931,21 @@ let
} }
]; ];
# The forge's host, publishing the controller's forge token. The one writer
# of the path, and write-only like the secret publisher: it copies a file in
# and never reads one back.
forgePublisherWriters = [
{
name = "swarm-forge-publisher";
cn = baoDeploy.forgePublisherCommonName;
policyText = ''
path "${credentialMountPath}/data/${controllerForgeTokenLeaf}" {
capabilities = ["create", "update"]
}
'';
}
];
# The role name IS the policy name, as for the three service principals # The role name IS the policy name, as for the three service principals
# above: the role attaches the policy by spelling it identically, and one # above: the role attaches the policy by spelling it identically, and one
# string for both objects removes the way they drift apart. # string for both objects removes the way they drift apart.
@ -1721,6 +1745,20 @@ in
''; '';
}; };
forgePublisherCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-forge-publisher";
description = ''
Subject the store's `swarm-forge-publisher` cert-auth role accepts: the
identity `swarm-bao-forge-publisher`, the unit on the forge's host that
publishes the swarm-controller's forge token, presents. Its grant is
one path, `swarm/controller/swarm-controller/forge/token`, and write
only.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
'';
};
queueAgentCommonNamePrefix = lib.mkOption { queueAgentCommonNamePrefix = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "swarm-bao-queue-agent"; default = "swarm-bao-queue-agent";
@ -1954,6 +1992,32 @@ in
''; '';
}; };
forgePublisherClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/forge-publisher.pem";
description = ''
Certificate the unit that publishes the swarm-controller's forge token
presents to the store. Its subject must be
{option}`services.hyperhive.deploy.bao.forgePublisherCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
On a forge host without the store, it is a file the operator copies
from the store's host. A path, never a value.
'';
};
forgePublisherClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/forge-publisher-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.forgePublisherClientCertFile`.
A path, never a value.
'';
};
servicesIssuerClientCertFile = lib.mkOption { servicesIssuerClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = null; default = null;
@ -3053,6 +3117,8 @@ in
systemd.services.swarm-bao-ci-runner-policy = readerPolicyUnit "write the CI runner's registration-token-reader bao policy and cert-auth role" ciRunnerReaders; systemd.services.swarm-bao-ci-runner-policy = readerPolicyUnit "write the CI runner's registration-token-reader bao policy and cert-auth role" ciRunnerReaders;
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
# A FOURTH sibling, same shape and same reasons as the two above. This # A FOURTH sibling, same shape and same reasons as the two above. This
# one is what turns `swarm-services-issuer` from a declaration into a # one is what turns `swarm-services-issuer` from a declaration into a
# grant: a bao policy reaches nothing until a login role hands it to a # grant: a bao policy reaches nothing until a login role hands it to a
@ -3258,6 +3324,7 @@ in
"swarm-bao-nats-tls-policy.service" "swarm-bao-nats-tls-policy.service"
"swarm-bao-hive-leaf-policy.service" "swarm-bao-hive-leaf-policy.service"
"swarm-bao-ci-runner-policy.service" "swarm-bao-ci-runner-policy.service"
"swarm-bao-forge-publisher-policy.service"
] ]
++ granterAfter; ++ granterAfter;
requires = [ "swarm-bao-pki.service" ]; requires = [ "swarm-bao-pki.service" ];

92
nix/host-modules/swarm-controller.nix
View file

@ -189,24 +189,11 @@ let
SWARM_CONTROLLER_OIDC_CLIENT_ID = queueClientId; SWARM_CONTROLLER_OIDC_CLIENT_ID = queueClientId;
}; };
# Independent of `queueEnv` on purpose, and now for a simpler reason # Unconditional: every swarm has a forge, reached by name wherever it runs.
# than when this was written: the queue's own co-location guard # The token is not here: the daemon reads it from the swarm's secret store at
# (`queueLocal`) is gone — its coordinates are options with assertions, # start (`forge.rs`), and runs without forge access when it cannot.
# so a split-host controller fails at eval instead of silently losing forgeEnv = {
# its identity. Forge access still has nothing to do with where authelia
# and NATS run, only with whether `forgeTokenFile` resolves to a real
# file — which that option's own default already handles
# (forge-local vs. operator-copied). Gating a second time here would
# just repeat that option's own logic under a different name.
#
# ⚠️ The asymmetry is deliberate: the queue is REQUIRED for a controller
# and the forge is not, so absence is an error for one and a supported
# shape for the other.
forgeEnv = lib.optionalAttrs (deployCfg.swarm-controller.forgeTokenFile != null) {
SWARM_CONTROLLER_FORGE_URL = "https://${forgeCfg.domain}"; SWARM_CONTROLLER_FORGE_URL = "https://${forgeCfg.domain}";
# `%d` is systemd's credentials directory: root reads the plaintext at
# unit start, the daemon's own user sees a 0400 copy.
SWARM_CONTROLLER_FORGE_TOKEN_FILE = "%d/forge-token";
# `agent-configs` org avatar for the forge-objects pass # `agent-configs` org avatar for the forge-objects pass
# (`forge/objects.rs`). Only meaningful with forge access, hence here. # (`forge/objects.rs`). Only meaningful with forge access, hence here.
SWARM_CONTROLLER_CONFIG_ORG_AVATAR_PNG = SWARM_CONTROLLER_CONFIG_ORG_AVATAR_PNG =
@ -268,8 +255,8 @@ let
# `vcs_metrics`/`hive_jobq_metrics` (this daemon's OTLP push exporters) # `vcs_metrics`/`hive_jobq_metrics` (this daemon's OTLP push exporters)
# read the standard `OTEL_EXPORTER_OTLP_ENDPOINT` var and are silent, # read the standard `OTEL_EXPORTER_OTLP_ENDPOINT` var and are silent,
# graceful no-ops without it — same shape as `forgeEnv`/`authBridgeEnv` # graceful no-ops without it — same shape as `authBridgeEnv` above,
# above, genuinely optional and gated on the option resolving rather # genuinely optional and gated on the option resolving rather
# than assumed. # than assumed.
# #
# By NAME through the gateway, never a loopback literal or a bare # By NAME through the gateway, never a loopback literal or a bare
@ -294,8 +281,7 @@ let
# HERE"): the collector is reachable by name wherever it runs, so a # HERE"): the collector is reachable by name wherever it runs, so a
# controller not co-located with it now exports over https instead of # controller not co-located with it now exports over https instead of
# exporting nothing — the graceful-absence case left is "no swarm-otel # exporting nothing — the graceful-absence case left is "no swarm-otel
# anywhere in this swarm at all," which `forgeEnv` already models the # anywhere in this swarm at all."
# same way.
otelSwarmCfg = config.services.hyperhive.swarm.otel; otelSwarmCfg = config.services.hyperhive.swarm.otel;
otelEnv = lib.optionalAttrs deployCfg.swarm-otel.enable { otelEnv = lib.optionalAttrs deployCfg.swarm-otel.enable {
OTEL_EXPORTER_OTLP_ENDPOINT = "https://${otelSwarmCfg.domain}/${otelSwarmCfg.producerName}"; OTEL_EXPORTER_OTLP_ENDPOINT = "https://${otelSwarmCfg.domain}/${otelSwarmCfg.producerName}";
@ -352,6 +338,14 @@ in
be deleted. be deleted.
'' ''
) )
(lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "swarm-controller" "forgeTokenFile" ]
''
The controller reads its forge token from the swarm's secret store, under
its own certificate (deploy.swarm-controller.baoClientCertFile), and
swarm-bao-forge-publisher on the forge's host puts it there. Remove this
definition; the file it named is read by nothing now and can be deleted.
''
)
]; ];
options.services.hyperhive.swarm.controller = { options.services.hyperhive.swarm.controller = {
@ -554,40 +548,6 @@ in
''; '';
}; };
forgeTokenFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
# Forge's own delivery path when the forge runs here, since nothing
# writes that path anywhere else. `null` otherwise, so a controller
# away from the forge logs that it has no forge access instead of
# waiting on a file that never appears. The controller's units are
# the thing that decides whether the file is ever read: they only
# exist under `deploy.swarm-controller.enable`.
default = if deployCfg.forgejo.enable then deployCfg.forgejo.hostSwarmControllerTokenFile else null;
defaultText = lib.literalExpression ''
if config.services.hyperhive.deploy.forgejo.enable
then config.services.hyperhive.deploy.forgejo.hostSwarmControllerTokenFile
else null
'';
example = "/var/lib/secrets/swarm-controller-forge.token";
description = ''
Path to this host's copy of the swarm-controller's forge access
token (see `hive-forge/default.nix`'s
`forgejo-swarm-controller-account` + `hive-forge-swarm-controller-token`
units, which mint and collect it onto forge's own host).
Defaults to forge's own delivery path when
{option}`services.hyperhive.deploy.forgejo.enable` is set on this
host, and to `null` otherwise. Set it explicitly when the forge
runs on another host — copy the token out of forge's
{option}`services.hyperhive.deploy.forgejo.hostSwarmControllerTokenFile`
with whatever secret management this deployment already uses,
the same shape `swarm.nix`'s `clientSecretFile` documents for
its own cross-host case. `null` means no forge access — the
daemon logs that and continues without it, the same
graceful-absence shape the queue coordinates already use.
'';
};
authBridgeUrl = lib.mkOption { authBridgeUrl = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = "${autheliaCfg.url}/bridge"; default = "${autheliaCfg.url}/bridge";
@ -606,7 +566,7 @@ in
already covers it. `null` means no agent-identity support: already covers it. `null` means no agent-identity support:
`CreateIdentity` jobs fail with a clear "no auth bridge `CreateIdentity` jobs fail with a clear "no auth bridge
configured here" error rather than the daemon refusing to configured here" error rather than the daemon refusing to
start, the same graceful-absence shape `forgeTokenFile` uses. start, the same graceful-absence shape forge access has.
''; '';
}; };
@ -758,21 +718,21 @@ in
systemd.services.swarm-controller = { systemd.services.swarm-controller = {
description = "hyperhive swarm-level controller daemon"; description = "hyperhive swarm-level controller daemon";
wantedBy = [ "multi-user.target" ]; wantedBy = [ "multi-user.target" ];
after = [ "network.target" ]; # Where the forge runs here too, the first read waits out one publish
# attempt. `After=` on a unit this host does not define is a no-op.
after = [
"network.target"
"swarm-bao-forge-publisher.service"
];
serviceConfig = { serviceConfig = {
ExecStart = "${deployCfg.swarm-controller.package}/bin/swarm-controller"; ExecStart = "${deployCfg.swarm-controller.package}/bin/swarm-controller";
# The forge token is optional: a controller with no forge access
# still serves its HTTP surface, and that IS a supported shape.
LoadCredential =
lib.optional (
deployCfg.swarm-controller.forgeTokenFile != null
) "forge-token:${deployCfg.swarm-controller.forgeTokenFile}"
# The store identity, same shape and same reason as hive-c0re's: the # The store identity, same shape and same reason as hive-c0re's: the
# key is root-owned `0600` and this daemon runs as `swarm-controller`, # key is root-owned `0600` and this daemon runs as `swarm-controller`,
# so it never gets read access to the original file. # so it never gets read access to the original file.
++ lib.optionals haveBaoIdentity [ LoadCredential =
lib.optionals haveBaoIdentity [
"bao-client.pem:${deployCfg.swarm-controller.baoClientCertFile}" "bao-client.pem:${deployCfg.swarm-controller.baoClientCertFile}"
"bao-client-key.pem:${deployCfg.swarm-controller.baoClientKeyFile}" "bao-client-key.pem:${deployCfg.swarm-controller.baoClientKeyFile}"
] ]
@ -846,8 +806,8 @@ in
# #
# They differ in how absence is prevented: the queue's is checked by # They differ in how absence is prevented: the queue's is checked by
# the assertions above, because a controller without a queue is # the assertions above, because a controller without a queue is
# broken rather than lighter; forge's and the auth bridge's are # broken rather than lighter; the auth bridge's is genuinely optional
# genuinely optional and stay gated on their own option resolving. # and stays gated on its own option resolving. Forge's is always set.
environment = { environment = {
SWARM_CONTROLLER_SOCKET = deployCfg.swarm-controller.socketPath; SWARM_CONTROLLER_SOCKET = deployCfg.swarm-controller.socketPath;
# The swarm's hive directory, JSON-encoded — the full directory # The swarm's hive directory, JSON-encoded — the full directory

1
nix/host-modules/swarm.nix
View file

@ -55,6 +55,7 @@ let
deployCfg.bao.granterCommonName deployCfg.bao.granterCommonName
deployCfg.bao.natsAuthCommonName deployCfg.bao.natsAuthCommonName
deployCfg.bao.ciRunnerCommonName deployCfg.bao.ciRunnerCommonName
deployCfg.bao.forgePublisherCommonName
] ]
# The per-hive subjects, spelled out per hive rather than as the prefix. # The per-hive subjects, spelled out per hive rather than as the prefix.
# The prefix alone would reserve the wrong string: the role for hive # The prefix alone would reserve the wrong string: the role for hive

21
nix/module-eval/bao-controller.nix
View file

@ -197,6 +197,27 @@ let
&& builtins.elem "bao-client.pem:/var/lib/swarm-bao-pki/controller.pem" s.swarm-controller.serviceConfig.LoadCredential && builtins.elem "bao-client.pem:/var/lib/swarm-bao-pki/controller.pem" s.swarm-controller.serviceConfig.LoadCredential
&& builtins.elem "bao-client-key.pem:/var/lib/swarm-bao-pki/controller-key.pem" s.swarm-controller.serviceConfig.LoadCredential; && builtins.elem "bao-client-key.pem:/var/lib/swarm-bao-pki/controller-key.pem" s.swarm-controller.serviceConfig.LoadCredential;
} }
{
# The forge token comes from the store, so the unit is handed no file for
# it, and neither fixture runs the forge: a controller reaches it by name
# wherever it runs.
name = "the controller is handed no forge token file, and is pointed at the forge away from it";
ok =
lib.all
(
m:
let
u = m.systemd.services.swarm-controller;
in
!(lib.any (c: lib.hasPrefix "forge-token:" c) u.serviceConfig.LoadCredential)
&& !(u.environment ? SWARM_CONTROLLER_FORGE_TOKEN_FILE)
&& u.environment.SWARM_CONTROLLER_FORGE_URL == "https://forge.t.local"
)
[
baoControllerHere
controllerNoStore
];
}
{ {
# A hive's cert-auth role carries the authority by value, so the daemon # A hive's cert-auth role carries the authority by value, so the daemon
# has to be handed the file rather than a path into the store's own # has to be handed the file rather than a path into the store's own

66
nix/module-eval/bao-grants.nix
View file

@ -170,7 +170,7 @@ let
_: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile _: u: (u.environment.BAO_CLIENT_CERT or null) == granterCertFile
) baoGrantWithConsumers.systemd.services; ) baoGrantWithConsumers.systemd.services;
# The fourteen units that write a `swarm-*` grant, by name, for the discovery # The fifteen units that write a `swarm-*` grant, by name, for the discovery
# control below. # control below.
grantingUnitNames = [ grantingUnitNames = [
"swarm-bao-controller-policy" "swarm-bao-controller-policy"
@ -187,6 +187,7 @@ let
"swarm-bao-operator-viewer-policy" "swarm-bao-operator-viewer-policy"
"swarm-bao-hive-leaf-policy" "swarm-bao-hive-leaf-policy"
"swarm-bao-ci-runner-policy" "swarm-bao-ci-runner-policy"
"swarm-bao-forge-publisher-policy"
]; ];
# Comment lines dropped first: both the HCL and the scripts explain # Comment lines dropped first: both the HCL and the scripts explain
@ -851,24 +852,24 @@ let
} }
{ {
# A store host without the granter's pair writes its grants some other # A store host without the granter's pair writes its grants some other
# way, so none of the fourteen units may exist. Without this arm # way, so none of the fifteen units may exist. Without this arm
# `lib.mkIf haveGranter` could be dropped from any of them and every other # `lib.mkIf haveGranter` could be dropped from any of them and every other
# case here would still pass. # case here would still pass.
name = "without the granter's pair none of the fourteen granting units render"; name = "without the granter's pair none of the fifteen granting units render";
ok = ok =
let let
s = baoGranterOptOut.systemd.services; s = baoGranterOptOut.systemd.services;
in in
lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ]) lib.all (unit: !(s ? ${unit})) (grantingUnitNames ++ [ "swarm-bao-granter-role" ])
# The control: the same store with the pair renders all fourteen. # The control: the same store with the pair renders all fifteen.
&& lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames; && lib.all (unit: baoGrantHere.systemd.services ? ${unit}) grantingUnitNames;
} }
{ {
# 🩸 What replaced the silent skip. With no bootstrap token the fourteen still # 🩸 What replaced the silent skip. With no bootstrap token the fifteen still
# render, and a refused granter fails them with the step that fixes it. # render, and a refused granter fails them with the step that fixes it.
# A store host that never named a token is told to name one, since the # A store host that never named a token is told to name one, since the
# unit that sets the granter up renders only where it has. # unit that sets the granter up renders only where it has.
name = "a store host without a bootstrap token renders the fourteen, each failing loudly with the one-time step"; name = "a store host without a bootstrap token renders the fifteen, each failing loudly with the one-time step";
ok = ok =
let let
s = baoGranterNoToken.systemd.services; s = baoGranterNoToken.systemd.services;
@ -1371,6 +1372,55 @@ let
in in
lib.hasInfix "/ci-runner.pem ]" s && lib.hasInfix "swarm-ci-runner \"\" clientAuth" s; lib.hasInfix "/ci-runner.pem ]" s && lib.hasInfix "swarm-ci-runner \"\" clientAuth" s;
} }
{
# The forge's host is the one writer of the controller's forge token.
# Pinned as the whole stanza, so an added `read` fails: it copies a file
# in and never reads one back.
name = "the forge publisher's grant is writing the controller's forge token and nothing else";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-forge-publisher-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/forge/token\" {\n capabilities = [\"create\", \"update\"]\n}" s
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1
&& lib.hasInfix "auth/cert/certs/swarm-forge-publisher" s
&& lib.hasInfix "allowed_common_names=swarm-forge-publisher" s
&& lib.hasInfix "token_policies=swarm-forge-publisher" s;
}
{
# The controller's side: read on the one leaf, pinned as the whole
# stanza so an added write fails.
name = "the controller reads its own forge token and cannot write it";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/forge/token\" {\n capabilities = [\"read\"]\n}" s
&& !(lib.hasInfix "secret/data/swarm/controller/*" s);
}
{
# The policy units that name the path are the whole list of who reaches
# it: the publisher, which writes it, and the controller, which reads it.
name = "no granting unit but the controller's and the forge publisher's names the controller's forge token";
ok =
let
naming = lib.filter (
u: lib.hasInfix "swarm-controller/forge/token" baoGrantHere.systemd.services.${u}.script
) grantingUnitNames;
in
lib.sort lib.lessThan naming == [
"swarm-bao-controller-policy"
"swarm-bao-forge-publisher-policy"
];
}
{
name = "the PKI unit signs the forge publisher's leaf under its own subject";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
in
lib.hasInfix "/forge-publisher.pem ]" s && lib.hasInfix "swarm-forge-publisher \"\" clientAuth" s;
}
{ {
# Revocation, and the reason it is a stanza of its own: `delete` on the # Revocation, and the reason it is a stanza of its own: `delete` on the
# `data/` path soft-deletes the newest version and leaves earlier ones # `data/` path soft-deletes the newest version and leaves earlier ones
@ -1503,8 +1553,8 @@ let
} }
{ {
# What makes the case above mean something: discovery by the granter's # What makes the case above mean something: discovery by the granter's
# certificate reaches all fourteen units, and each yields calls. # certificate reaches all fifteen units, and each yields calls.
name = "the granter-policy check sees all fourteen granting units, and parses calls from each"; name = "the granter-policy check sees all fifteen granting units, and parses calls from each";
ok = ok =
lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames lib.sort lib.lessThan (lib.attrNames granterUnits) == lib.sort lib.lessThan grantingUnitNames
&& lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits) && lib.all (u: baoCalls u.script != [ ]) (lib.attrValues granterUnits)

10
nix/module-eval/core-toggle.nix
View file

@ -76,16 +76,16 @@ let
&& centralToggleOff.services.hyperhive.swarm.forge.publicUrl == "https://forge.t.local"; && centralToggleOff.services.hyperhive.swarm.forge.publicUrl == "https://forge.t.local";
} }
{ {
# The controller's token path follows where the forge runs # The controller's token publisher follows where the forge runs
# (./forge-placement.nix), never the central toggle: a forge host with # (./forge-placement.nix), never the central toggle: a forge host with
# the toggle off still renders the path. # the toggle off still publishes it.
name = "the swarm controller's forgeTokenFile default does not consult the central toggle"; name = "the forge's controller-token publisher does not consult the central toggle";
ok = ok =
(hive { (hive {
deploy.hive-controller.enable = false; deploy.hive-controller.enable = false;
deploy.forgejo.enable = true; deploy.forgejo.enable = true;
}).services.hyperhive.deploy.swarm-controller.forgeTokenFile }).systemd.services
== "/var/lib/hyperhive-forge/swarm-controller.token"; ? swarm-bao-forge-publisher;
} }
{ {
name = "a hive that does not host the swarm's shared services runs none of them"; name = "a hive that does not host the swarm's shared services runs none of them";

44
nix/module-eval/forge-placement.nix
View file

@ -104,8 +104,7 @@ let
# ./grafana.nix's `grafanaRefusedFor`. # ./grafana.nix's `grafanaRefusedFor`.
refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions; refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions;
tokenFile = m: m.services.hyperhive.deploy.swarm-controller.forgeTokenFile; publisher = m: m.systemd.services.swarm-bao-forge-publisher;
forgePath = "/var/lib/hyperhive-forge/swarm-controller.token";
forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings; forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings;
@ -122,7 +121,7 @@ let
ok = ok =
!bare.services.hyperhive.deploy.forgejo.enable !bare.services.hyperhive.deploy.forgejo.enable
&& !(bare.containers ? hive-forge) && !(bare.containers ? hive-forge)
&& !(bare.systemd.services ? hive-forge-swarm-controller-token) && !(bare.systemd.services ? swarm-bao-forge-publisher)
&& !(lib.elem "forge.t.local" bare.services.hyperhive.gateway.localNames) && !(lib.elem "forge.t.local" bare.services.hyperhive.gateway.localNames)
&& !(refusedOver bare "deploy.forgejo.sso.clientSecretFile"); && !(refusedOver bare "deploy.forgejo.sso.clientSecretFile");
} }
@ -280,14 +279,39 @@ let
&& refusedOver ciUnderForgeNoIdentity "services.hyperhive.deploy.bao.enable"; && refusedOver ciUnderForgeNoIdentity "services.hyperhive.deploy.bao.enable";
} }
{ {
# Nothing writes the delivery path away from the forge, and a # The forge's host publishes the controller's token under its own leaf,
# `LoadCredential=` naming a missing path is fatal to the unit. # which the store mints beside it here. `value=@` is what keeps the token
name = "the controller's forge token defaults to the delivery path only where the forge runs"; # out of the process's arguments.
name = "the forge's host publishes the controller's forge token to the store under its own leaf";
ok = ok =
tokenFile bare == null let
&& tokenFile servicesForgeElsewhere == null u = publisher allLocal;
&& tokenFile forgeHere == forgePath in
&& tokenFile allLocal == forgePath; u.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/forge-publisher.pem"
&& u.environment.BAO_CLIENT_KEY == "/var/lib/swarm-bao-pki/forge-publisher-key.pem"
&& lib.hasInfix "bao kv put secret/swarm/controller/swarm-controller/forge/token value=@\"$src\"" u.script
&& lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo/swarm-controller-token" u.script
&& lib.elem "container@hive-forge.service" u.wantedBy
&& !(allLocal.systemd.services ? hive-forge-swarm-controller-token);
}
{
# The absence arm: a forge host without the store has no leaf unless one
# is placed, and the unit says which options to set.
name = "a forge host with no publisher leaf renders a publisher that fails naming the options";
ok =
let
s = (publisher forgeHere).script;
in
lib.hasInfix "forgePublisherClientCertFile" s
&& lib.hasInfix "exit 1" s
&& !(lib.hasInfix "bao kv put" s);
}
{
# The host copy is removed where the forge runs: nothing reads it.
name = "the forge's host removes its old copy of the controller's forge token";
ok =
lib.elem "r /var/lib/hyperhive-forge/swarm-controller.token" forgeHere.systemd.tmpfiles.rules
&& !(lib.elem "r /var/lib/hyperhive-forge/swarm-controller.token" bare.systemd.tmpfiles.rules);
} }
{ {
# The only thing that makes a human's forge account: nothing else # The only thing that makes a human's forge account: nothing else

17
nix/module-eval/name-guards.nix
View file

@ -83,6 +83,13 @@ let
swarm.hives.fwctl.domain = "f.t.local"; swarm.hives.fwctl.domain = "f.t.local";
}; };
# The forge publisher's, the one writer of the controller's forge token.
hiveNamedAfterForgePublisherSubject = hive {
deploy.swarm-otel.enable = false;
deploy.bao.forgePublisherCommonName = "fpctl";
swarm.hives.fpctl.domain = "fp.t.local";
};
# The granter's, the one subject whose role may write every `swarm-*` grant. # The granter's, the one subject whose role may write every `swarm-*` grant.
hiveNamedAfterGranterSubject = hive { hiveNamedAfterGranterSubject = hive {
deploy.swarm-otel.enable = false; deploy.swarm-otel.enable = false;
@ -188,6 +195,16 @@ let
a: !a.assertion && lib.hasInfix "'fwctl'" a.message a: !a.assertion && lib.hasInfix "'fwctl'" a.message
) hiveNamedAfterForwarderOidcSubject.assertions; ) hiveNamedAfterForwarderOidcSubject.assertions;
} }
{
# And the forge publisher's: a hive holding a leaf it accepts could
# replace the token the controller provisions every forge object with.
name = "a hive named after the forge publisher's subject is refused too";
ok =
equalityGuardFired hiveNamedAfterForgePublisherSubject
&& lib.any (
a: !a.assertion && lib.hasInfix "'fpctl'" a.message
) hiveNamedAfterForgePublisherSubject.assertions;
}
{ {
# And the granter's, whose role is root-equivalent: a hive holding a leaf # And the granter's, whose role is root-equivalent: a hive holding a leaf
# it accepts could grant itself anything. # it accepts could grant itself anything.

123
swarm-controller/src/forge.rs
View file

@ -4,16 +4,15 @@
//! calls, is premature plumbing; factor out a shared crate later if the //! calls, is premature plumbing; factor out a shared crate later if the
//! duplication actually starts to hurt). //! duplication actually starts to hurt).
//! //!
//! Reads its identity from the credentials `swarm-controller.nix`'s //! The forge's URL comes from `swarm-controller.nix`'s `forgeEnv`
//! `forgeEnv` wires in (`SWARM_CONTROLLER_FORGE_URL` + //! (`SWARM_CONTROLLER_FORGE_URL`). The token comes from the swarm's secret
//! `SWARM_CONTROLLER_FORGE_TOKEN_FILE`, the latter pointing at a //! store, at `swarm_secret_client::forge::controller_token_path`, read once at
//! systemd `LoadCredential`-delivered file) — see that module and //! start under this daemon's own certificate and held only in memory. The
//! `hive-forge/default.nix`'s `forgejo-swarm-controller-account` / //! forge mints it inside its container (`hive-forge/default.nix`'s
//! `hive-forge-swarm-controller-token` units for how the token itself //! `forgejo-swarm-controller-account`) and `swarm-bao-forge-publisher`, on the
//! gets minted and delivered. Absent env (forge not configured on this //! forge's host, puts it there. An unset URL means [`Client::from_env`]
//! host) means [`Client::from_env`] returns `None` — every caller //! returns `None` — every caller treats that as "no forge access here" and
//! treats that as "no forge access here" and continues, the same //! continues, the same graceful-absence shape the queue coordinates use.
//! graceful-absence shape the queue coordinates already use.
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use forgejo_api::structs::{ use forgejo_api::structs::{
@ -130,12 +129,30 @@ const KNOWLEDGE_REPO: &str = "knowledge";
/// admin bypass, without adding a `git` shell-out to this daemon. /// admin bypass, without adding a `git` shell-out to this daemon.
const SWARM_CONTROLLER_FORGE_USER: &str = "swarm-controller"; const SWARM_CONTROLLER_FORGE_USER: &str = "swarm-controller";
/// The env vars `swarm-controller.nix`'s `forgeEnv` sets. Named here /// The env var `swarm-controller.nix`'s `forgeEnv` sets. Named here once so
/// once so [`Client::from_env`] and the nix module can't drift silently /// [`Client::from_env`] and the nix module can't drift silently.
/// — a rename on one side without the other fails loudly (env var
/// absent) rather than silently losing forge access.
const URL_ENV: &str = "SWARM_CONTROLLER_FORGE_URL"; const URL_ENV: &str = "SWARM_CONTROLLER_FORGE_URL";
const TOKEN_FILE_ENV: &str = "SWARM_CONTROLLER_FORGE_TOKEN_FILE";
/// One login and one read. An empty value is refused like an absent one: it
/// would reach the forge as the token and come back as a 401 on the first
/// call, which names neither the store nor the publisher.
async fn fetch_token() -> Result<String> {
let path = swarm_secret_client::forge::controller_token_path()?;
let store = crate::store::connect()
.await
.context("logging in to the swarm secret store")?;
let stored: Option<swarm_secret_client::forge::ControllerToken> = store
.read_optional(&path)
.await
.with_context(|| format!("reading {path}"))?;
match stored {
Some(t) if !t.value.trim().is_empty() => Ok(t.value.trim().to_owned()),
_ => anyhow::bail!(
"nothing at {path}: swarm-bao-forge-publisher, on the host that runs the forge, has \
not published this daemon's forge token"
),
}
}
/// Typed forgejo client, built once at startup from env. /// Typed forgejo client, built once at startup from env.
pub struct Client { pub struct Client {
@ -147,38 +164,29 @@ pub struct Client {
} }
impl Client { impl Client {
/// Build from env. `Ok(None)` (not an error) when neither env var is /// Build from env and the secret store. `Ok(None)` (not an error) when
/// set — forge isn't configured on this host, which is a normal, /// `SWARM_CONTROLLER_FORGE_URL` is unset — no forge is configured for this
/// expected deployment shape (see the module doc comment), not a /// daemon (see the module doc comment).
/// failure. `Err` only for a genuine misconfiguration: one of the ///
/// two env vars present without the other, the token file unreadable, /// # Errors
/// or the URL/client failing to build — those ARE failures, because /// An unparseable URL, or a token the store still would not give up after
/// they mean forge was *meant* to be configured here and isn't /// `queue_identity::RETRY_DELAYS_S`: the store refusing the login, or
/// usable, which should be loud rather than silently downgrading to /// nothing at the path because the forge's host has not published it.
/// "no forge access." pub async fn from_env() -> Result<Option<Self>> {
pub fn from_env() -> Result<Option<Self>> { let Some(url) = std::env::var(URL_ENV).ok() else {
let url = std::env::var(URL_ENV).ok(); return Ok(None);
let token_file = std::env::var(TOKEN_FILE_ENV).ok();
let (url, token_file) = match (url, token_file) {
(None, None) => return Ok(None),
(Some(url), Some(token_file)) => (url, token_file),
(Some(_), None) => {
anyhow::bail!("{URL_ENV} is set but {TOKEN_FILE_ENV} is not — partial forge config")
}
(None, Some(_)) => {
anyhow::bail!("{TOKEN_FILE_ENV} is set but {URL_ENV} is not — partial forge config")
}
}; };
let token = std::fs::read_to_string(&token_file)
.with_context(|| format!("read forge token from {token_file}"))?;
let token = token.trim();
if token.is_empty() {
anyhow::bail!("forge token file {token_file} is empty");
}
let parsed_url = let parsed_url =
url::Url::parse(&url).with_context(|| format!("parse {URL_ENV} ({url}) as a URL"))?; url::Url::parse(&url).with_context(|| format!("parse {URL_ENV} ({url}) as a URL"))?;
let api = let delays = crate::queue_identity::RETRY_DELAYS_S.map(std::time::Duration::from_secs);
Forgejo::new(Auth::Token(token), parsed_url.clone()).context("build forgejo client")?; let token = crate::queue_identity::fetch_with_retry(
&delays,
"the controller's forge token",
fetch_token,
)
.await?;
let api = Forgejo::new(Auth::Token(&token), parsed_url.clone())
.context("build forgejo client")?;
Ok(Some(Self { Ok(Some(Self {
api, api,
url: parsed_url, url: parsed_url,
@ -1295,30 +1303,29 @@ mod tests {
// Both scenarios live in ONE test, not two, deliberately: `cargo // Both scenarios live in ONE test, not two, deliberately: `cargo
// test` runs tests in parallel by default, and two tests mutating // test` runs tests in parallel by default, and two tests mutating
// the same process-global env vars would race each other. A single // the same process-global env var would race each other. A single
// test's body is one thread, so sequential set/clear within it is // test's body is one thread, so sequential set/clear within it is
// race-free without needing a cross-test lock. // race-free without needing a cross-test lock.
#[test] #[tokio::test]
fn from_env_reads_both_vars_or_neither() { async fn from_env_is_none_without_a_url_and_refuses_a_bad_one_before_the_store() {
// SAFETY: test-only mutation of env vars this module itself // SAFETY: test-only mutation of the env var this module itself
// owns (`URL_ENV`/`TOKEN_FILE_ENV`), scoped to this one test // owns (`URL_ENV`), scoped to this one test function so there's no
// function so there's no cross-test race (see comment above). // cross-test race (see comment above).
unsafe { unsafe {
std::env::remove_var(URL_ENV); std::env::remove_var(URL_ENV);
std::env::remove_var(TOKEN_FILE_ENV);
} }
assert!(Client::from_env().unwrap().is_none()); assert!(Client::from_env().await.unwrap().is_none());
unsafe { unsafe {
std::env::set_var(URL_ENV, "https://forge.example.invalid"); std::env::set_var(URL_ENV, "not a url");
} }
// Not `.unwrap_err()`: that needs `Client: Debug` for its panic // Not `.unwrap_err()`: that needs `Client: Debug` for its panic
// message, and the wrapped `forgejo_api::Forgejo` doesn't derive // message, and the wrapped `forgejo_api::Forgejo` doesn't derive
// it. Match directly instead of adding a `Debug` impl nothing // it. Returning at once, with no `BAO_*` set, is what shows the URL
// else needs. // is checked before the store is asked.
match Client::from_env() { match Client::from_env().await {
Err(e) => assert!(e.to_string().contains(TOKEN_FILE_ENV)), Err(e) => assert!(e.to_string().contains(URL_ENV), "{e:#}"),
Ok(_) => panic!("expected an error for a partial forge config"), Ok(_) => panic!("expected an error for an unparseable forge URL"),
} }
unsafe { unsafe {

43
swarm-controller/src/main.rs
View file

@ -313,8 +313,8 @@ async fn run_swarm_node(
}, },
SwarmNodeKind::CreateRepo { agent } => match deps.forge { SwarmNodeKind::CreateRepo { agent } => match deps.forge {
None => Outcome::Failed( None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)" not in the swarm secret store)"
.to_owned(), .to_owned(),
), ),
Some(client) => match client.create_repo(&agent).await { Some(client) => match client.create_repo(&agent).await {
@ -327,8 +327,8 @@ async fn run_swarm_node(
}, },
SwarmNodeKind::CreateForgeUser { agent } => match deps.forge { SwarmNodeKind::CreateForgeUser { agent } => match deps.forge {
None => Outcome::Failed( None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)" not in the swarm secret store)"
.to_owned(), .to_owned(),
), ),
Some(client) => match client.ensure_agent_user(&agent).await { Some(client) => match client.ensure_agent_user(&agent).await {
@ -338,8 +338,8 @@ async fn run_swarm_node(
}, },
SwarmNodeKind::AddRepoMember { agent } => match deps.forge { SwarmNodeKind::AddRepoMember { agent } => match deps.forge {
None => Outcome::Failed( None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)" not in the swarm secret store)"
.to_owned(), .to_owned(),
), ),
Some(client) => match client.add_repo_member(&agent, &agent).await { Some(client) => match client.add_repo_member(&agent, &agent).await {
@ -350,8 +350,8 @@ async fn run_swarm_node(
SwarmNodeKind::GrantDocsRead { agent } => grant_docs_read(deps.forge, &agent).await, SwarmNodeKind::GrantDocsRead { agent } => grant_docs_read(deps.forge, &agent).await,
SwarmNodeKind::InitAgentConfigRepo { agent } => match deps.forge { SwarmNodeKind::InitAgentConfigRepo { agent } => match deps.forge {
None => Outcome::Failed( None => Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset)" not in the swarm secret store)"
.to_owned(), .to_owned(),
), ),
Some(client) => match client.seed_agent_config(&agent, &agent).await { Some(client) => match client.seed_agent_config(&agent, &agent).await {
@ -411,8 +411,8 @@ async fn mint_forge_token(
let Some(client) = forge else { let Some(client) = forge else {
return Outcome::Failed( return Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset), so no agent forge token can be minted" not in the swarm secret store), so no agent forge token can be minted"
.to_owned(), .to_owned(),
); );
}; };
@ -431,8 +431,8 @@ async fn grant_docs_read(
let Some(client) = forge else { let Some(client) = forge else {
return Outcome::Failed( return Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset), so no docs grant can be made" not in the swarm secret store), so no docs grant can be made"
.to_owned(), .to_owned(),
); );
}; };
@ -461,8 +461,8 @@ async fn mint_hive_forge_token(
let Some(client) = forge else { let Some(client) = forge else {
return Outcome::Failed( return Outcome::Failed(
"no forge configured on this host (SWARM_CONTROLLER_FORGE_URL / \ "no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
SWARM_CONTROLLER_FORGE_TOKEN_FILE unset), so no hive forge token can be minted" not in the swarm secret store), so no hive forge token can be minted"
.to_owned(), .to_owned(),
); );
}; };
@ -3017,13 +3017,14 @@ async fn main() -> Result<()> {
None None
} }
}; };
// Same shape again: a controller with no forge configured still serves // Same shape again: a controller with no forge access still serves
// everything else, and the forge-shaped node kinds give an honest // everything else, and the forge-shaped node kinds give an honest
// per-job failure rather than this fn refusing to start. // per-job failure rather than this fn refusing to start. A token missing
let forge_client = match forge::Client::from_env() { // from the store lands here too, after `from_env`'s retries.
let forge_client = match forge::Client::from_env().await {
Ok(client) => client.map(Arc::new), Ok(client) => client.map(Arc::new),
Err(e) => { Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "forge misconfigured; repo provisioning is off"); tracing::warn!(error = %format!("{e:#}"), "no forge access; repo provisioning is off");
None None
} }
}; };
@ -3250,7 +3251,7 @@ mod tests {
/// interesting behavior is in `run_swarm_node`'s executor arm, which /// interesting behavior is in `run_swarm_node`'s executor arm, which
/// this exercises directly). /// this exercises directly).
/// ///
/// Deliberately offline: with both forge env vars unset, /// Deliberately offline: with the forge URL unset,
/// `forge::Client::from_env` returns `Ok(None)` (see that module's doc /// `forge::Client::from_env` returns `Ok(None)` (see that module's doc
/// comment), so this exercises the whole claim → run → complete path /// comment), so this exercises the whole claim → run → complete path
/// through `hive_jobq::scheduler::Scheduler` without a real forge /// through `hive_jobq::scheduler::Scheduler` without a real forge
@ -3259,7 +3260,7 @@ mod tests {
/// actual forge instance and isn't something a unit test in this crate /// actual forge instance and isn't something a unit test in this crate
/// can reach. /// can reach.
/// ///
/// SAFETY: single-threaded mutation of the two `forge` env vars this /// SAFETY: single-threaded mutation of the `forge` env var this
/// test itself owns, restored before returning — no other test in this /// test itself owns, restored before returning — no other test in this
/// crate reads them. /// crate reads them.
/// One roster, one agent name, two hives — the accepted one and a /// One roster, one agent name, two hives — the accepted one and a
@ -4177,7 +4178,6 @@ mod tests {
async fn create_repo_node_runs_end_to_end_and_fails_without_forge_configured() { async fn create_repo_node_runs_end_to_end_and_fails_without_forge_configured() {
unsafe { unsafe {
std::env::remove_var("SWARM_CONTROLLER_FORGE_URL"); std::env::remove_var("SWARM_CONTROLLER_FORGE_URL");
std::env::remove_var("SWARM_CONTROLLER_FORGE_TOKEN_FILE");
} }
let mut sched = hive_jobq::scheduler::Scheduler::new( let mut sched = hive_jobq::scheduler::Scheduler::new(
@ -4231,7 +4231,6 @@ mod tests {
async fn create_forge_user_node_runs_end_to_end_and_fails_without_forge_configured() { async fn create_forge_user_node_runs_end_to_end_and_fails_without_forge_configured() {
unsafe { unsafe {
std::env::remove_var("SWARM_CONTROLLER_FORGE_URL"); std::env::remove_var("SWARM_CONTROLLER_FORGE_URL");
std::env::remove_var("SWARM_CONTROLLER_FORGE_TOKEN_FILE");
} }
let mut sched = hive_jobq::scheduler::Scheduler::new( let mut sched = hive_jobq::scheduler::Scheduler::new(

23
swarm-controller/src/queue_identity.rs
View file

@ -25,7 +25,7 @@ const PREFIX: &str = "SWARM_CONTROLLER";
/// Seconds to wait after each failed fetch before the next one. About a minute /// Seconds to wait after each failed fetch before the next one. About a minute
/// in all, which rides out a store restarting alongside this daemon. A longer /// in all, which rides out a store restarting alongside this daemon. A longer
/// outage fails the start, and the unit's `Restart=` tries again from the top. /// outage fails the start, and the unit's `Restart=` tries again from the top.
const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30]; pub(crate) const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30];
static IDENTITY: OnceLock<Option<QueueConfig>> = OnceLock::new(); static IDENTITY: OnceLock<Option<QueueConfig>> = OnceLock::new();
@ -61,7 +61,9 @@ pub async fn init() -> Result<()> {
None => None, None => None,
Some(coordinates) => { Some(coordinates) => {
let delays = RETRY_DELAYS_S.map(Duration::from_secs); let delays = RETRY_DELAYS_S.map(Duration::from_secs);
Some(coordinates.with_secret(fetch_with_retry(&delays, fetch_secret).await?)) Some(coordinates.with_secret(
fetch_with_retry(&delays, "the queue client secret", fetch_secret).await?,
))
} }
}; };
IDENTITY IDENTITY
@ -118,8 +120,13 @@ async fn fetch_secret() -> Result<String> {
} }
/// Run `attempt`, waiting each of `delays` after a failure, then once more. /// Run `attempt`, waiting each of `delays` after a failure, then once more.
/// Answers with the first success, or the last attempt's error. /// Answers with the first success, or the last attempt's error. `what` names the
async fn fetch_with_retry<F, Fut>(delays: &[Duration], mut attempt: F) -> Result<String> /// secret in the log line and the error.
pub(crate) async fn fetch_with_retry<F, Fut>(
delays: &[Duration],
what: &str,
mut attempt: F,
) -> Result<String>
where where
F: FnMut() -> Fut, F: FnMut() -> Fut,
Fut: Future<Output = Result<String>>, Fut: Future<Output = Result<String>>,
@ -131,7 +138,7 @@ where
tracing::warn!( tracing::warn!(
error = %format!("{e:#}"), error = %format!("{e:#}"),
retry_in = ?delay, retry_in = ?delay,
"fetching the queue client secret from the swarm secret store failed" "fetching {what} from the swarm secret store failed"
); );
tokio::time::sleep(*delay).await; tokio::time::sleep(*delay).await;
} }
@ -139,7 +146,7 @@ where
} }
attempt().await.with_context(|| { attempt().await.with_context(|| {
format!( format!(
"fetching the queue client secret from the swarm secret store, after {} attempts", "fetching {what} from the swarm secret store, after {} attempts",
delays.len() + 1 delays.len() + 1
) )
}) })
@ -217,7 +224,7 @@ mod tests {
#[tokio::test] #[tokio::test]
async fn a_store_that_comes_back_within_the_window_is_waited_for() { async fn a_store_that_comes_back_within_the_window_is_waited_for() {
let mut calls = 0; let mut calls = 0;
let got = fetch_with_retry(&[Duration::ZERO; 3], || { let got = fetch_with_retry(&[Duration::ZERO; 3], "a secret", || {
calls += 1; calls += 1;
let n = calls; let n = calls;
async move { async move {
@ -236,7 +243,7 @@ mod tests {
#[tokio::test] #[tokio::test]
async fn a_store_that_stays_down_fails_after_one_attempt_per_delay_plus_one() { async fn a_store_that_stays_down_fails_after_one_attempt_per_delay_plus_one() {
let mut calls = 0; let mut calls = 0;
let err = fetch_with_retry(&[Duration::ZERO; 3], || { let err = fetch_with_retry(&[Duration::ZERO; 3], "a secret", || {
calls += 1; calls += 1;
async { bail!("store sealed") } async { bail!("store sealed") }
}) })

64
swarm-secret-client/src/forge.rs
View file

@ -1,7 +1,7 @@
//! The forge agreement: where agents' and hives' forge tokens live in the store, //! The forge agreement: where agents' and hives' forge tokens live in the store,
//! and what the objects at those paths hold. //! and what the objects at those paths hold.
//! //!
//! Three kinds. The agent's own token on the swarm's forge //! Four kinds. The agent's own token on the swarm's forge
//! ([`agent_token_path`]): `swarm-controller` mints it with the forge's admin //! ([`agent_token_path`]): `swarm-controller` mints it with the forge's admin
//! API, and `nix/agent-modules/forge-token.nix` reads it back under the agent's //! API, and `nix/agent-modules/forge-token.nix` reads it back under the agent's
//! own certificate. And the agent's accounts on external forges //! own certificate. And the agent's accounts on external forges
@ -12,6 +12,10 @@
//! hive's own forge account, and hive-c0re reads it under the hive's //! hive's own forge account, and hive-c0re reads it under the hive's
//! certificate. Neither end is senior, so both halves are stated once, here, //! certificate. Neither end is senior, so both halves are stated once, here,
//! beside [`crate::matrix`]. //! beside [`crate::matrix`].
//!
//! `swarm-controller`'s own forge token ([`controller_token_path`]) is the
//! fourth: the forge mints it, `swarm-bao-forge-publisher` on the forge's host
//! writes it here, and the controller reads it as it starts.
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
@ -62,6 +66,33 @@ pub fn hive_token_path(hive: &str) -> Result<String, Error> {
Ok(format!("{prefix}/forge/token")) Ok(format!("{prefix}/forge/token"))
} }
/// The path holding `swarm-controller`'s own forge token: the admin account it
/// provisions every agent's and hive's forge objects with.
///
/// Under [`Kind::Controller`], which no hive's policy reads. The nix half is
/// `swarm-bao.nix`'s `controllerForgeTokenLeaf`, which the publisher's write
/// grant and the controller's read grant both name.
///
/// # Errors
/// Never in practice: the name segment is a constant. The `Result` is
/// [`principal_prefix`]'s.
pub fn controller_token_path() -> Result<String, Error> {
let prefix = principal_prefix(Kind::Controller, crate::matrix::CONTROLLER)?;
Ok(format!("{prefix}/forge/token"))
}
/// What [`controller_token_path`] holds. No token name beside the value, unlike
/// [`Credential`]: the publisher copies a file that holds the token alone.
///
/// No `Debug` derive: `value` is the forge admin credential.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ControllerToken {
/// The token itself. `swarm-bao-forge-publisher` writes it with
/// `bao kv put … value=@<file>`, so this name is load-bearing for a writer
/// this crate does not control.
pub value: String,
}
/// What an agent's forge-token path, and a hive's [`hive_token_path`], holds. /// What an agent's forge-token path, and a hive's [`hive_token_path`], holds.
/// ///
/// No `Debug` derive: `value` is a live forge credential, and a derived /// No `Debug` derive: `value` is a live forge credential, and a derived
@ -148,6 +179,37 @@ mod tests {
); );
} }
#[test]
fn the_controller_token_is_under_the_controller_prefix() {
// Spelled out: `swarm-bao.nix` grants exactly this string.
assert_eq!(
controller_token_path().expect("a constant segment is legal"),
"swarm/controller/swarm-controller/forge/token"
);
}
#[test]
fn no_hive_policy_reaches_the_controller_token() {
let policy = crate::policy::render("alpha").expect("legal");
let covers = |path: &str| {
policy.lines().any(|line| {
line.strip_prefix("path \"")
.and_then(|rest| rest.split_once("\" {"))
.and_then(|(p, _)| p.strip_suffix('*'))
.is_some_and(|prefix| path.starts_with(prefix))
})
};
assert!(!covers(&format!(
"secret/data/{}",
controller_token_path().expect("legal")
)));
// The control: the same matcher finds the hive's own token.
assert!(covers(&format!(
"secret/data/{}",
hive_token_path("alpha").expect("legal")
)));
}
#[test] #[test]
fn no_forge_key_is_also_a_directory() { fn no_forge_key_is_also_a_directory() {
// KV-v2 cannot hold a key whose name is also a prefix of another key. // KV-v2 cannot hold a key whose name is also a prefix of another key.