vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-builder: build each agent config PR head and post a commit status

Browse source 
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
This commit is contained in:
atlas 2026-10-12 22:04:51 +02:00
commit b68b74d687
43 changed files with 2452 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

7
CLAUDE.md
View file

@ -202,6 +202,13 @@ hand-maintained per-file tree drifts out of sync with the code.
for agents: nginx sends it `/select/logsql/` requests carrying an agent for agents: nginx sends it `/select/logsql/` requests carrying an agent
token, it answers 200 or 401 via `swarm-agent-verify`. Runs on the log token, it answers 200 or 401 via `swarm-agent-verify`. Runs on the log
store's host (`swarm-victorialogs.nix`). store's host (`swarm-victorialogs.nix`).
- **`swarm-builder/`** — the swarm builder's build daemon, in its own
container where `deploy.builder.enable` is set. Takes an agent config PR's
head from `swarm-controller` over the queue, builds it on its host's agent
base through the host nix daemon (`nix/builder/check-flake`), and replies
with the store path or the error tail; the controller writes the commit
status. Holds a read-only forge token. Design:
[`docs/swarm/builder.md`](docs/swarm/builder.md).
- **`swarmctl/`** — swarm-level operator CLI, installed by the - **`swarmctl/`** — swarm-level operator CLI, installed by the
swarm-controller module on the host that runs the daemon. Runs as swarm-controller module on the host that runs the daemon. Runs as
**root and acts directly** — it serves no socket, publishes no HTTP **root and acts directly** — it serves no socket, publishes no HTTP

15
Cargo.lock generated
View file

@ -4829,6 +4829,21 @@ dependencies = [
"serde_json", "serde_json",
] ]
[[package]]
name = "swarm-builder"
version = "0.1.0"
dependencies = [
"anyhow",
"async-nats",
"futures-util",
"hive-log",
"serde_json",
"swarm-queue-client",
"swarm-secret-client",
"tokio",
"tracing",
]
[[package]] [[package]]
name = "swarm-controller" name = "swarm-controller"
version = "0.1.0" version = "0.1.0"

1
Cargo.toml
View file

@ -27,6 +27,7 @@ members = [
"hivectl", "hivectl",
"swarm-agent-verify", "swarm-agent-verify",
"swarm-authelia-bridge", "swarm-authelia-bridge",
"swarm-builder",
"swarm-authelia-bridge-sock", "swarm-authelia-bridge-sock",
"swarm-controller", "swarm-controller",
"swarm-matrix-client", "swarm-matrix-client",

2
docs/README.md
View file

@ -89,6 +89,8 @@ declarations.
- **How will a PR get checked against its repo's own yes/no rules before - **How will a PR get checked against its repo's own yes/no rules before
a full review?** → [`swarm/review-classifier.md`](swarm/review-classifier.md) a full review?** → [`swarm/review-classifier.md`](swarm/review-classifier.md)
(planned, not implemented yet). (planned, not implemented yet).
- **Does an agent config PR build before it's merged?** →
[`swarm/builder.md`](swarm/builder.md).
## Scheduler, CI, observability ## Scheduler, CI, observability

81
docs/swarm/builder.md Normal file
View file

@ -0,0 +1,81 @@
# Swarm builder
The swarm builder checks that an agent config PR builds before an operator
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
push, the builder builds the PR's head on the agent base, and the result shows as a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
Where it runs and how it's enabled: [`services.md`](services.md#builder).
## What happens on a config PR
1. The forge delivers the `pull_request` event on the `config-pr` webhook
swarm-controller already registers on `agent-configs`. An `opened`,
`reopened` or `synchronized` action on an open PR queues one `build_agent`
node carrying the agent, the head commit and the PR number. An edit, a
label or a close builds nothing.
2. The node sets the status to `pending`, then sends the build as a request on
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
reply.
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
with every input overridden:
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
deployed with, the same two a hive on that host renders into its meta
flake (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`).
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
A build has one hour.
4. The reply is the system closure's store path, or the last lines `nix`
printed. The node writes the status on the head commit:
| Outcome | Status | Description | Node |
| ----------------------------------------- | --------- | --------------------------- | ------ |
| built | `success` | the store path | done |
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
| no builder listening, or no reply in time | `error` | why | failed |
The full tail is in the failed node's error on the Jobs page and in the
builder's journal line for the build.
## What the check is, and isn't
- It composes the config the way a hive's meta flake does
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
what a hive adds: its CA bundle, docs source, `claude-code` path, ports and
OTEL settings, and the `follows` that point the config's own inputs at the
hive's. A green check can therefore still fail on a hive, and the reverse.
- The agent base is the builder's host's, not the hive's. Hives on another
hyperhive or nixpkgs pin build something else.
- The builder pushes nothing and records nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches.
- It doesn't gate anything. A merge deploys whatever the status says.
- A lost webhook delivery isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
## Credentials
| What | Where | Who reads it |
| ----------------------- | ------------------------------------------------- | ---------------------------------------- |
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start |
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The builder holds no write credential; swarm-controller writes the
status with its own forge token. Details per path:
[`credentials.md`](credentials.md).
## Isolation
The container has its own network namespace on the bridge, as the CI runner
does, so a config's evaluation can't reach a host-loopback service. Evaluation
is pure and runs in the container. Builds go to the host's nix daemon as an
untrusted client: a config can set no daemon option, and its derivations build
in the host's sandbox. A fixed-output derivation still has the network, as it
does for any build on that host.

45
docs/swarm/credentials.md
View file

@ -55,28 +55,29 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | | `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

11
docs/swarm/secrets.md
View file

@ -1,3 +1,5 @@
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
# Swarm secrets: what exists, and where each one lives # Swarm secrets: what exists, and where each one lives
A swarm generates its credentials in three different places and reads them A swarm generates its credentials in three different places and reads them
@ -400,9 +402,10 @@ next controller restart.
### Per-principal identities ### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these seven leaves has identity and sharing one merges the identities. Each of these eight leaves has
a role and a policy of its own, and reaches one path, except the review a role and a policy of its own, and reaches one path, except the review
classifier's, the forge host's and Grafana's, which reach two each: classifier's, the builder's, the forge host's and Grafana's, which reach two
each:
| unit | option pair under `deploy.bao.` | reads, or writes | | unit | option pair under `deploy.bao.` | reads, or writes |
| --------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | --------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
@ -418,11 +421,11 @@ The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is and every hive runs its own reader. Its subject is
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that `<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
composed spelling as a hive name, so nobody can name a hive into another hive's composed spelling as a hive name, so nobody can name a hive into another hive's
role. The other six reach a path that names a swarm service rather than a hive, role. The other seven reach a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`, `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`, `deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName` and `deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName` and
`deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one `deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one
leaf two units share: the forge's host holds one store identity for the token it leaf two units share: the forge's host holds one store identity for the token it
publishes and the OIDC client secret it reads. publishes and the OIDC client secret it reads.

20
docs/swarm/services.md
View file

@ -241,6 +241,26 @@ On a host without the store, that leaf is the one file to copy. Registration,
the container and the threat model are in the container and the threat model are in
[`../scheduler/ci.md`](../scheduler/ci.md). [`../scheduler/ci.md`](../scheduler/ci.md).
### Builder
The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
has the builder's store identity, on the same terms as the review classifier.
Set `deploy.builder.enable` directly to override either way.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
swarm-controller queues a `build_agent` job. The builder builds that config at
the PR's head on the agent base its own host runs, through the
host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
### Review classifier ### Review classifier
The review classifier's try-out service, in a `swarm-review-classifier` The review classifier's try-out service, in a `swarm-review-classifier`

4
docs/trust-boundary/security.md
View file

@ -132,6 +132,10 @@ checkpoints**, not about sandboxing the agent from its own tools:
[`config-changes.md`](../agent-lifecycle/config-changes.md)), and [`config-changes.md`](../agent-lifecycle/config-changes.md)), and
scheduled prompts are operator-only, set at swarm level scheduled prompts are operator-only, set at swarm level
(see [`schedules.md`](../swarm/schedules.md)). (see [`schedules.md`](../swarm/schedules.md)).
Before that merge, the swarm builder evaluates an unmerged config PR
off-hive, on any host that sets `deploy.builder.enable`: it builds the
PR's head with a read-only forge token, in its own netns, as an untrusted
client of its host's nix daemon (see [`builder.md`](../swarm/builder.md)).
### Capability = accepted risk ### Capability = accepted risk

3
flake.nix
View file

@ -166,6 +166,9 @@
services.hyperhive.deploy.review-classifier.package = services.hyperhive.deploy.review-classifier.package =
lib.mkDefault lib.mkDefault
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-review-classifier; self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-review-classifier;
services.hyperhive.deploy.builder.package =
lib.mkDefault
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-builder;
services.hyperhive.deploy.authelia.bridgePackage = services.hyperhive.deploy.authelia.bridgePackage =
lib.mkDefault lib.mkDefault
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-authelia-bridge; self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-authelia-bridge;

26
nix/builder/check-flake/flake.nix Normal file
View file

@ -0,0 +1,26 @@
# The swarm builder's build check: one agent config on hyperhive's agent base,
# composed the way a hive's meta flake composes it, minus the values a hive
# adds (its CA, docs source, claude-code path and ports).
#
# Every input is replaced at build time with `--override-input`
# (swarm-builder/src/nix_build.rs), so the URLs below are never fetched and
# the flake has no lock.
{
description = "swarm-builder: an agent config built on agent-base";
inputs = {
nixpkgs.url = "flake:nixpkgs";
hyperhive.url = "flake:hyperhive";
hyperhive.inputs.nixpkgs.follows = "nixpkgs";
agent.url = "flake:agent";
};
outputs =
{ hyperhive, agent, ... }:
{
packages.x86_64-linux.check =
(hyperhive.nixosConfigurations.agent-base.extendModules {
modules = [ agent.nixosModules.default ];
}).config.system.build.toplevel;
};
}

4
nix/checks.nix
View file

@ -44,6 +44,10 @@ in
inherit (pkgs) lib; inherit (pkgs) lib;
shared = moduleEvalShared; shared = moduleEvalShared;
}; };
module-eval-builder = import ./module-eval/builder.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-nats-authelia = import ./module-eval/nats-authelia.nix { module-eval-nats-authelia = import ./module-eval/nats-authelia.nix {
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;

2
nix/host-modules/default.nix
View file

@ -44,6 +44,7 @@
./glue-nats-oidc-client.nix ./glue-nats-oidc-client.nix
./glue-otel-scrape-targets.nix ./glue-otel-scrape-targets.nix
./glue-queue-agent-credential.nix ./glue-queue-agent-credential.nix
./glue-builder-oidc-client.nix
./glue-review-classifier-oidc-client.nix ./glue-review-classifier-oidc-client.nix
./glue-secret-publisher-authelia.nix ./glue-secret-publisher-authelia.nix
./glue-secret-publisher-bao-identity.nix ./glue-secret-publisher-bao-identity.nix
@ -53,6 +54,7 @@
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao-service.nix ./swarm-bao-service.nix
./swarm-bao.nix ./swarm-bao.nix
./swarm-builder.nix
./swarm-ci-runner.nix ./swarm-ci-runner.nix
./swarm-review-classifier.nix ./swarm-review-classifier.nix
./swarm-secret-publisher.nix ./swarm-secret-publisher.nix

18
nix/host-modules/deploy.nix
View file

@ -679,6 +679,24 @@ in
''; '';
}; };
builder.enable = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Run the swarm builder in a `swarm-builder` container on this host. It
builds each agent config PR's head on this host's agent base, through
this host's nix daemon, and swarm-controller writes the result as a
`swarm-builder` commit status on the PR.
On by default where
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
host has the builder's store identity (the pair
{option}`services.hyperhive.deploy.bao.builderClientCertFile` and
`builderClientKeyFile`, minted on the store's host).
'';
};
hive-controller.enable = lib.mkOption { hive-controller.enable = lib.mkOption {
type = lib.types.bool; type = lib.types.bool;
default = false; default = false;

9
nix/host-modules/glue-bao-tls.nix
View file

@ -115,6 +115,8 @@ in
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem"; ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
reviewClassifierClientCertFile = lib.mkDefault "${pkiDir}/review-classifier.pem"; reviewClassifierClientCertFile = lib.mkDefault "${pkiDir}/review-classifier.pem";
reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem"; reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem";
builderClientCertFile = lib.mkDefault "${pkiDir}/builder.pem";
builderClientKeyFile = lib.mkDefault "${pkiDir}/builder-key.pem";
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem"; forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem"; forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem"; servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
@ -290,6 +292,13 @@ in
[ -s ${pkiDir}/review-classifier.pem ] || ${signLeaf} ${pkiDir} review-classifier \ [ -s ${pkiDir}/review-classifier.pem ] || ${signLeaf} ${pkiDir} review-classifier \
${lib.escapeShellArg deployCfg.bao.reviewClassifierCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.reviewClassifierCommonName} "" clientAuth
# The swarm builder's, which reads its forge token and queue client
# secret. Minted here for the matrix-ctl leaf's reason: the builder is a
# swarm service, so on a builder host without the store this is the
# file an operator copies.
[ -s ${pkiDir}/builder.pem ] || ${signLeaf} ${pkiDir} builder \
${lib.escapeShellArg deployCfg.bao.builderCommonName} "" clientAuth
# The forge's token publisher. Minted here for the matrix-ctl leaf's # The forge's token publisher. Minted here for the matrix-ctl leaf's
# reason: the forge is a swarm singleton, so on a forge host without the # reason: the forge is a swarm singleton, so on a forge host without the
# store this is the file an operator copies. # store this is the file an operator copies.

34
nix/host-modules/glue-builder-oidc-client.nix Normal file
View file

@ -0,0 +1,34 @@
# Glue: register the swarm builder as an OIDC client wherever authelia runs.
#
# ONE PAIRING PER FILE — swarm-builder ← authelia, and nothing else. Deleting
# this leaves a builder whose token requests authelia answers with a 401, and
# no minted secret for swarm-secret-publish to put in the store.
#
# ⚠️ Gated on authelia being HERE, and NOT on this host running the builder,
# for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS
# host's provider config. Registered whether or not any host enables the
# builder: nothing in `swarm.*` records that, so a swarm without one carries
# one unused client and one unused minted secret.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
clientId = hyperhiveCfg.swarm.builder.clientId;
in
{
config = lib.mkIf deployCfg.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = clientId;
description = "HyperHive swarm builder";
# `client_credentials`: a daemon authenticating as itself.
kind = "machine";
audience = [ clientId ];
}
];
};
}

68
nix/host-modules/swarm-bao.nix
View file

@ -441,6 +441,10 @@ let
capabilities = ["create", "read", "update"] capabilities = ["create", "read", "update"]
} }
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
capabilities = ["create", "read", "update"]
}
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" { path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
capabilities = ["read"] capabilities = ["read"]
} }
@ -492,6 +496,8 @@ let
# The review classifier's forge token is denied too: the controller posts # The review classifier's forge token is denied too: the controller posts
# the classifier's reviews with whatever token is there, so a write to it # the classifier's reviews with whatever token is there, so a write to it
# picks the account those reviews come from. # picks the account those reviews come from.
# So is the builder's: a write to it picks the account the builder fetches
# config repos as.
secretPublisherPolicyText = '' secretPublisherPolicyText = ''
path "${credentialMountPath}/data/swarm/hives/*" { path "${credentialMountPath}/data/swarm/hives/*" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
@ -521,6 +527,10 @@ let
capabilities = ["deny"] capabilities = ["deny"]
} }
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
capabilities = ["deny"]
}
path "${credentialMountPath}/data/${controllerClientLeaf}" { path "${credentialMountPath}/data/${controllerClientLeaf}" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
@ -588,6 +598,12 @@ let
# the UI's OIDC login; the classifier reads it on every run. # the UI's OIDC login; the classifier reads it on every run.
classifierApiKeyLeaf = "swarm/services/review-classifier/anthropic-api-key"; classifierApiKeyLeaf = "swarm/services/review-classifier/anthropic-api-key";
# The swarm builder's forge token, the nix half of
# `swarm_secret_client::builder::forge_token_path`. The controller mints it
# and reads it back before deciding whether to re-mint; the builder reads it
# to fetch config repos.
builderForgeTokenLeaf = "swarm/services/builder/forge/token";
# The UI's OIDC client secret. The publisher writes it; the granter reads it # The UI's OIDC client secret. The publisher writes it; the granter reads it
# into the `oidc` auth method's config, which is the only copy bao uses. # into the `oidc` auth method's config, which is the only copy bao uses.
baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client"; baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client";
@ -998,6 +1014,18 @@ let
} }
]; ];
# The swarm builder: its forge token, read once per build, and its own queue
# client secret, read at start. Both by the daemon inside its container.
builderReaders = [
{
name = "swarm-builder";
cn = baoDeploy.builderCommonName;
policyText =
readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}"
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client";
}
];
# The forge host's one store identity. `swarm-bao-forge-publisher` writes # The forge host's one store identity. `swarm-bao-forge-publisher` writes
# the controller's forge token under it, the one writer of that path and # the controller's forge token under it, the one writer of that path and
# write-only: it copies a file in and never reads one back. # write-only: it copies a file in and never reads one back.
@ -1848,6 +1876,19 @@ in
''; '';
}; };
builderCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-builder";
description = ''
Subject the store's `swarm-builder` cert-auth role accepts: the
identity the swarm builder presents from inside its container. Its
grant is two paths, read only: `swarm/services/builder/forge/token`
and its own queue client secret.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
'';
};
forgePublisherCommonName = lib.mkOption { forgePublisherCommonName = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "swarm-forge-publisher"; default = "swarm-forge-publisher";
@ -2123,6 +2164,31 @@ in
''; '';
}; };
builderClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/builder.pem";
description = ''
Certificate the swarm builder presents to the store. Its subject must
be {option}`services.hyperhive.deploy.bao.builderCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
On a builder host without the store, it is a file the operator copies
from the store's host. A path, never a value.
'';
};
builderClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/builder-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.builderClientCertFile`.
A path, never a value.
'';
};
forgePublisherClientCertFile = lib.mkOption { forgePublisherClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = null; default = null;
@ -3201,6 +3267,8 @@ in
systemd.services.swarm-bao-review-classifier-policy = readerPolicyUnit "write the review classifier's API-key-reader bao policy and cert-auth role" reviewClassifierReaders; systemd.services.swarm-bao-review-classifier-policy = readerPolicyUnit "write the review classifier's API-key-reader bao policy and cert-auth role" reviewClassifierReaders;
systemd.services.swarm-bao-builder-policy = readerPolicyUnit "write the swarm builder's forge-token-reader bao policy and cert-auth role" builderReaders;
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters; systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
# A FOURTH sibling, same shape and same reasons as the two above. This # A FOURTH sibling, same shape and same reasons as the two above. This

231
nix/host-modules/swarm-builder.nix Normal file
View file

@ -0,0 +1,231 @@
# The swarm builder, in a `swarm-builder` container on the host that sets
# `deploy.builder.enable`, which `deploy.allSwarmServices` defaults on where
# the host has the builder's store identity. It builds agent configs for
# swarm-controller: the controller sends a config PR's head over the queue,
# and the daemon builds that config on this host's agent base and replies.
# docs/swarm/builder.md has the design.
#
# ⚠️ Its own network namespace on the bridge, as ./swarm-ci-runner.nix has: a
# config PR is unreviewed input to a nix evaluation, so the container reaches
# the swarm services by name and nothing on the host's loopback.
#
# Builds go through the HOST nix-daemon, as the CI runner's do. The daemon
# runs as a dynamic user, an untrusted daemon client: a config can set no
# daemon option, and its builds run in the host's sandbox.
#
# The container holds one credential, its own store leaf, which nspawn hands in
# as a system credential with no copy on disk. The daemon reads everything else
# from the store with it: its queue client secret at start and its forge token
# on every build.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
cfg = hyperhiveCfg.deploy.builder;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
natsCfg = hyperhiveCfg.swarm.nats;
autheliaCfg = hyperhiveCfg.swarm.authelia;
forgeCfg = hyperhiveCfg.swarm.forge;
gatewayCfg = hyperhiveCfg.gateway;
networkCfg = hyperhiveCfg.network;
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
machine = "swarm-builder";
unit = "swarm-builder";
clientId = hyperhiveCfg.swarm.builder.clientId;
# The daemon reaches authelia, the store and the queue by name over TLS, and
# `git` reaches the forge; in self-signed mode their leaves chain to the hive
# CA.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
caBundleModule = caTrust.trustBundle {
inherit pkgs;
name = machine;
consumers = [ unit ];
};
haveClientIdentity =
baoDeploy.builderClientCertFile != null && baoDeploy.builderClientKeyFile != null;
# Credential id → host source, as ./swarm-review-classifier.nix hands its
# leaf in.
storeFiles = lib.filterAttrs (_: source: source != null) {
builder-bao-cert = baoDeploy.builderClientCertFile;
builder-bao-key = baoDeploy.builderClientKeyFile;
builder-bao-server-ca = baoDeploy.serverCaFile;
};
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
credential = id: "/run/credentials/${unit}.service/${id}";
# The `nix` the host daemon runs, so client and daemon speak one protocol.
nix = config.nix.package;
# Strings, not paths: both are already store paths, and the flake reference
# the daemon passes must be the same one a hive's meta flake names.
hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake;
nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake;
privateNetwork = true;
in
{
options.services.hyperhive.swarm.builder.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-builder";
description = ''
The OAuth2 client id the swarm builder presents to the swarm queue.
Read-only: the host running authelia registers it, the queue's
auth-callout responder grants it the build subject, and the secret
publisher carries its secret to the store, all from this one value.
'';
};
options.services.hyperhive.deploy.builder.package = lib.mkOption {
type = lib.types.package;
defaultText = lib.literalExpression "hyperhive.packages.\${system}.swarm-builder";
description = "The swarm builder daemon package.";
};
config = lib.mkIf cfg.enable {
assertions = [
{
assertion = haveClientIdentity;
message = ''
services.hyperhive.deploy.builder.enable requires the builder's
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.builderClientCertFile
services.hyperhive.deploy.bao.builderClientKeyFile
The builder reads its queue client secret and its forge token with it.
Both default to the pair glue-bao-tls.nix mints when
services.hyperhive.deploy.bao.enable is true (the host that runs the
store). Otherwise copy builder.pem and builder-key.pem from the store
host's /var/lib/swarm-bao-pki/ and name them in those two options.
'';
}
{
assertion = lib.all oneWord (lib.attrValues storeFiles);
message = ''
The swarm builder's store identity paths must not contain whitespace:
nspawn's --load-credential receives them through nixos-container's
unquoted extra flags.
'';
}
];
# The container hangs off the bridge, takes its address from the gateway's
# DHCP and resolves through its dnsmasq.
services.hyperhive.network.enable = lib.mkDefault true;
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
# nspawn fails to start the container when a credential source is
# missing, so where this host mints the leaf the container waits for it.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
{ after = [ "swarm-bao-pki.service" ]; }
];
containers.${machine} = {
autoStart = true;
# The daemon keeps nothing but `git`'s fetch cache, which a restart may
# drop.
ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [
"--link-journal=host"
]
++ lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
inherit privateNetwork;
hostBridge = networkCfg.bridgeName;
bindMounts = {
# The directory rather than the socket file, so the mount survives a
# daemon restart replacing the inode.
"/nix/var/nix/daemon-socket" = {
hostPath = "/nix/var/nix/daemon-socket";
isReadOnly = false;
};
}
// caTrust.bindMount;
config =
{ lib, ... }:
{
imports = [
../container-modules/swarm-container.nix
caBundleModule
];
services.hyperhive.swarmContainer = {
inherit privateNetwork;
writesOwnResolvConf = false;
};
networking.nameservers = [ networkCfg.bridgeIp ];
networking.interfaces.eth0.useDHCP = true;
# Both the service and its socket are off: the socket unit owns
# `/nix/var/nix/daemon-socket/socket` and would race the bind mount.
systemd.globalEnvironment.NIX_REMOTE = "daemon";
systemd.services.nix-daemon.enable = false;
systemd.sockets.nix-daemon.enable = false;
systemd.services.${unit} = {
description = "build agent configs for swarm-controller";
wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ];
after = [ "network-online.target" ];
# nix fetches the config repo with `git`.
path = [ pkgs.git ];
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = credential "builder-bao-cert";
BAO_CLIENT_KEY = credential "builder-bao-key";
NIX_CONFIG = "experimental-features = nix-command flakes";
# nix's fetch cache, under `CacheDirectory`; the dynamic user has
# no home of its own.
HOME = "/var/cache/${unit}";
XDG_CACHE_HOME = "/var/cache/${unit}";
SWARM_BUILDER_NATS_URL = "tls://${natsCfg.domain}:${toString natsCfg.port}";
SWARM_BUILDER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
SWARM_BUILDER_OIDC_CLIENT_ID = clientId;
SWARM_BUILDER_NIX = "${nix}/bin/nix";
SWARM_BUILDER_CHECK_FLAKE = "${../builder/check-flake}";
SWARM_BUILDER_HYPERHIVE_FLAKE = hyperhiveFlake;
SWARM_BUILDER_NIXPKGS_FLAKE = nixpkgsFlake;
SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}";
SWARM_BUILDER_CONFIG_ORG = "agent-configs";
}
// lib.optionalAttrs (storeFiles ? builder-bao-server-ca) {
BAO_CACERT = credential "builder-bao-server-ca";
}
// lib.optionalAttrs caTrust.useSelfSigned {
SWARM_BUILDER_OIDC_CA_FILE = caTrust.bundlePathFor machine;
# `git`'s trust store, which the bundle's system CAs keep whole.
GIT_SSL_CAINFO = caTrust.bundlePathFor machine;
};
serviceConfig = {
ExecStart = "${cfg.package}/bin/swarm-builder";
DynamicUser = true;
CacheDirectory = unit;
# Bare ids: the system credentials nspawn handed in.
LoadCredential = lib.attrNames storeFiles;
Restart = "on-failure";
RestartSec = "10s";
SyslogIdentifier = unit;
};
};
};
};
};
}

2
nix/host-modules/swarm-nats.nix
View file

@ -683,6 +683,8 @@ in
"--reader-client ${lib.escapeShellArg controllerCfg.queueClientId}" "--reader-client ${lib.escapeShellArg controllerCfg.queueClientId}"
# Hears try-outs and answers each by reply, and nothing else. # Hears try-outs and answers each by reply, and nothing else.
"--review-classifier-client ${lib.escapeShellArg config.services.hyperhive.swarm.review-classifier.clientId}" "--review-classifier-client ${lib.escapeShellArg config.services.hyperhive.swarm.review-classifier.clientId}"
# Hears builds and answers each by reply, and nothing else.
"--builder-client ${lib.escapeShellArg config.services.hyperhive.swarm.builder.clientId}"
# What an agent may publish to, keyed on its own name: its # What an agent may publish to, keyed on its own name: its
# terminal rows and its turn-state header # terminal rows and its turn-state header
# (`hive-agent::swarm_agent_state`), two subject families so a # (`hive-agent::swarm_agent_state`), two subject families so a

7
nix/host-modules/swarm-required-services.nix
View file

@ -117,4 +117,11 @@ in
&& deployCfg.bao.reviewClassifierClientCertFile != null && deployCfg.bao.reviewClassifierClientCertFile != null
&& deployCfg.bao.reviewClassifierClientKeyFile != null && deployCfg.bao.reviewClassifierClientKeyFile != null
); );
# The swarm builder, defaulted on the same terms as the review classifier.
config.services.hyperhive.deploy.builder.enable = lib.mkDefault (
deployCfg.allSwarmServices
&& deployCfg.bao.builderClientCertFile != null
&& deployCfg.bao.builderClientKeyFile != null
);
} }

2
nix/host-modules/swarm-secret-publisher.nix
View file

@ -94,6 +94,8 @@ let
hyperhiveCfg.swarm.forge.sso.clientId hyperhiveCfg.swarm.forge.sso.clientId
# The review classifier, which reads it itself inside its container. # The review classifier, which reads it itself inside its container.
hyperhiveCfg.swarm.review-classifier.clientId hyperhiveCfg.swarm.review-classifier.clientId
# The swarm builder, which reads it itself inside its container.
hyperhiveCfg.swarm.builder.clientId
]; ];
# The swarm controller's OIDC client, which it reads back from the store # The swarm controller's OIDC client, which it reads back from the store

1
nix/host-modules/swarm.nix
View file

@ -57,6 +57,7 @@ let
deployCfg.bao.ciRunnerCommonName deployCfg.bao.ciRunnerCommonName
deployCfg.bao.logsAuthCommonName deployCfg.bao.logsAuthCommonName
deployCfg.bao.reviewClassifierCommonName deployCfg.bao.reviewClassifierCommonName
deployCfg.bao.builderCommonName
deployCfg.bao.forgePublisherCommonName deployCfg.bao.forgePublisherCommonName
] ]
# The per-hive subjects, spelled out per hive rather than as the prefix. # The per-hive subjects, spelled out per hive rather than as the prefix.

53
nix/module-eval/bao-grants.nix
View file

@ -189,6 +189,7 @@ let
"swarm-bao-logs-auth-policy" "swarm-bao-logs-auth-policy"
"swarm-bao-forge-publisher-policy" "swarm-bao-forge-publisher-policy"
"swarm-bao-review-classifier-policy" "swarm-bao-review-classifier-policy"
"swarm-bao-builder-policy"
]; ];
# Comment lines dropped first: both the HCL and the scripts explain # Comment lines dropped first: both the HCL and the scripts explain
@ -1491,6 +1492,58 @@ let
"swarm-bao-controller-policy" "swarm-bao-controller-policy"
]; ];
} }
{
# The builder reads its forge token and its own queue client secret,
# each read-only and spelled to the leaf: `services/*` would reach every
# swarm service's secret.
name = "the swarm builder's grant is its forge token and its queue client secret, and nothing else";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-builder-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"read\"]\n}" s
&& lib.hasInfix "path \"secret/data/swarm/services/swarm-builder/oidc/client\" {\n capabilities = [\"read\"]\n}" s
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
&& lib.hasInfix "auth/cert/certs/swarm-builder" s
&& lib.hasInfix "allowed_common_names=swarm-builder" s
&& lib.hasInfix "token_policies=swarm-builder" s;
}
{
# `forge::builder_account` mints the token and reads it back to decide
# on a re-mint. Pinned as the whole stanza, and spelled to the leaf.
name = "the controller's grant on the swarm builder is its forge token alone";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
in
lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"create\", \"read\", \"update\"]\n}" s
&& !(lib.hasInfix "secret/data/swarm/services/builder/*" s);
}
{
# The token sits under the publisher's `services/*` write, and the
# builder fetches config repos with whatever is there. Its OIDC leaf is
# the control: still the prefix's.
name = "the publisher may not write the swarm builder's forge token";
ok =
let
grants = grantsIn baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy.script;
in
(grantFor grants "secret/data/swarm/services/builder/forge/token").caps == [ "deny" ]
&&
(grantFor grants "secret/data/swarm/services/swarm-builder/oidc/client").caps == [
"create"
"update"
];
}
{
name = "the PKI unit signs the swarm builder's leaf under its own subject";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
in
lib.hasInfix "/builder.pem ]" s && lib.hasInfix "swarm-builder \"\" clientAuth" s;
}
{ {
name = "the PKI unit signs the CI runner's leaf under its own subject"; name = "the PKI unit signs the CI runner's leaf under its own subject";
ok = ok =

131
nix/module-eval/builder.nix Normal file
View file

@ -0,0 +1,131 @@
# `checks.module-eval-builder` — see ./lib.nix for the shared rationale (why
# this suite exists, naming convention, "evaluates not executes"). The swarm
# builder's container, its default and its queue and IdP registrations; its
# store grants are in ./bao-grants.nix beside every other principal's.
{
pkgs,
lib,
self,
nixosSystem,
}:
let
inherit
(import ./lib.nix {
inherit
pkgs
lib
self
nixosSystem
;
})
hive
hiveWithController
runGroup
;
bare = hiveWithController { };
# On the store's host, which mints the builder's store identity.
withBuilder = hive {
deploy.bao.enable = true;
deploy.builder.enable = true;
};
servicesHere = hiveWithController { deploy.allSwarmServices = true; };
servicesBaoElsewhere = hive {
deploy.allSwarmServices = true;
deploy.bao.enable = false;
};
natsHere = hive {
swarm.nats.enable = true;
swarm.nats.authPackage = pkgs.emptyDirectory;
};
autheliaHere = hive {
deploy.bao.enable = true;
deploy.authelia.enable = true;
};
container = withBuilder.containers.swarm-builder;
inner = container.config;
svc = inner.systemd.services.swarm-builder;
env = svc.environment;
cases = [
{
name = "the swarm builder's container exists only where its host enables it";
ok = !(bare.containers ? swarm-builder) && withBuilder.containers ? swarm-builder;
}
{
# Its module asserts a store identity that only the store's host mints,
# so with the store elsewhere the default stays off rather than failing
# that host's eval. `nats` is the control that the switch is on at all.
name = "the shared services turn the builder on with the store's leaf and leave it off without";
ok =
servicesHere.services.hyperhive.deploy.builder.enable
&& servicesHere.containers ? swarm-builder
&& !servicesBaoElsewhere.services.hyperhive.deploy.builder.enable
&& !(servicesBaoElsewhere.containers ? swarm-builder)
&& servicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
}
{
# A config PR is unreviewed input to an evaluation: the container keeps
# its own netns, and reaches the host's nix daemon through its socket
# alone, with no daemon of its own racing the bind.
name = "the builder's container has its own netns and builds only through the host's daemon socket";
ok =
container.privateNetwork
&& inner.services.hyperhive.swarmContainer.privateNetwork
&& container.bindMounts."/nix/var/nix/daemon-socket".hostPath == "/nix/var/nix/daemon-socket"
&& !inner.systemd.services.nix-daemon.enable
&& !inner.systemd.sockets.nix-daemon.enable
&& inner.systemd.globalEnvironment.NIX_REMOTE == "daemon";
}
{
# The daemon holds its store leaf, handed in by nspawn without a copy on
# disk; the forge token is read per build and reaches only `git`.
name = "the builder loads only its store leaf, runs as a dynamic user and carries no token in its unit";
ok =
lib.elem "builder-bao-key" svc.serviceConfig.LoadCredential
&& env.BAO_CLIENT_KEY == "/run/credentials/swarm-builder.service/builder-bao-key"
&& env.SWARM_BUILDER_OIDC_CLIENT_ID == "swarm-builder"
&& lib.elem "--load-credential=builder-bao-key:/var/lib/swarm-bao-pki/builder-key.pem" container.extraFlags
&& svc.serviceConfig.DynamicUser
&& !(lib.any (lib.hasSuffix "TOKEN") (lib.attrNames env));
}
{
# The check builds on the sources a hive on this host would render into
# its meta flake, so a green check means the same base.
name = "the builder builds on the hyperhive and nixpkgs this host's hives use";
ok =
env.SWARM_BUILDER_HYPERHIVE_FLAKE == withBuilder.services.hyperhive.c0re.hyperhiveFlake
&& env.SWARM_BUILDER_NIXPKGS_FLAKE == withBuilder.services.hyperhive.c0re.nixpkgsFlake
&& lib.hasSuffix "-check-flake" env.SWARM_BUILDER_CHECK_FLAKE
&& env.SWARM_BUILDER_CONFIG_ORG == "agent-configs"
&& env.SWARM_BUILDER_FORGE_URL == "https://${withBuilder.services.hyperhive.swarm.forge.domain}";
}
{
# The responder denies a client id it does not know, and a NATS denial
# arrives as a timeout.
name = "the queue's responder grants the builder's client its subject";
ok = lib.hasInfix "--builder-client swarm-builder" natsHere.containers.swarm-nats.config.systemd.services.swarm-nats-auth.serviceConfig.ExecStart;
}
{
name = "the builder's client is registered as a machine, and the publisher carries its secret";
ok =
let
c = lib.findFirst (
c: c.id == "swarm-builder"
) null autheliaHere.services.hyperhive.swarm.authelia.oidc.clients;
in
c != null
&& c.kind == "machine"
&& lib.hasInfix "secret/swarm/services/swarm-builder/oidc/client" (
autheliaHere.systemd.services.swarm-secret-publish.script
);
}
];
in
runGroup "builder" cases

2
nix/module-eval/swarm-services-switch.nix
View file

@ -472,7 +472,7 @@ let
# that lost a container can't pass by comparing fewer of them. # that lost a container can't pass by comparing fewer of them.
name = "every service container restates its host-side privateNetwork"; name = "every service container restates its host-side privateNetwork";
ok = ok =
lib.length (lib.attrNames serviceContainers) == 11 lib.length (lib.attrNames serviceContainers) == 12
&& lib.all ( && lib.all (
c: c.privateNetwork == (c.config.services.hyperhive.swarmContainer.privateNetwork or null) c: c.privateNetwork == (c.config.services.hyperhive.swarmContainer.privateNetwork or null)
) (lib.attrValues serviceContainers); ) (lib.attrValues serviceContainers);

1
nix/packages/default.nix
View file

@ -177,6 +177,7 @@ in
# The review classifier's daemon. Out of `daemonBins` for the responder's # The review classifier's daemon. Out of `daemonBins` for the responder's
# reason: it runs inside its own container, not on the host. # reason: it runs inside its own container, not on the host.
swarm-review-classifier = mkBinPackage "swarm-review-classifier" "hyperhive review classifier try-out daemon"; swarm-review-classifier = mkBinPackage "swarm-review-classifier" "hyperhive review classifier try-out daemon";
swarm-builder = mkBinPackage "swarm-builder" "hyperhive swarm builder: agent config build checks";
# The only process allowed to write swarm-authelia's users database — # The only process allowed to write swarm-authelia's users database —
# same "runs *inside* a container, not on the host" placement as # same "runs *inside* a container, not on the host" placement as

26
swarm-builder/Cargo.toml Normal file
View file

@ -0,0 +1,26 @@
[package]
name = "swarm-builder"
version.workspace = true
readme = "README.md"
edition.workspace = true
[[bin]]
name = "swarm-builder"
path = "src/main.rs"
[dependencies]
anyhow.workspace = true
async-nats.workspace = true
# StreamExt::next on the subscription.
futures-util.workspace = true
hive-log.workspace = true
serde_json.workspace = true
# The build subject and the shapes the controller sends and reads back.
swarm-queue-client.workspace = true
# The forge token and queue client secret paths, and the store login.
swarm-secret-client.workspace = true
tokio.workspace = true
tracing.workspace = true
[lints]
workspace = true

31
swarm-builder/README.md Normal file
View file

@ -0,0 +1,31 @@
# swarm-builder
The swarm builder's build service, in the `swarm-builder` container on the
host that sets `deploy.builder.enable`. The design is
[`docs/swarm/builder.md`](../docs/swarm/builder.md).
swarm-controller sends one request per config PR head: an agent, a commit and
the PR. This service builds that agent config on the agent base this host was
deployed with, through the host's nix daemon, and answers with the system
closure's store path or the tail of what `nix` printed. The controller turns
the answer into a `swarm-builder` commit status on the PR's head.
## What a build is
- `nix build` of `nix/builder/check-flake`'s `check` output, with its three
inputs overridden: `hyperhive` and `nixpkgs` to this host's own sources, and
`agent` to `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<rev>`.
- The evaluation runs here, pure; the builds run in the host daemon's sandbox
as an untrusted client, so a config can neither set daemon options nor read
this service's environment.
- Nothing is pushed and nothing is recorded: the closure stays in the host's
store until it is collected.
## Credentials
The container holds one credential: its own store certificate, which nspawn
hands in at start without a copy on disk. With it the service reads its
queue client secret at start and its read-only forge token at every build.
`git` gets the token through a credential helper reading the child's
environment, never in a URL or an argument. Its store role reads exactly
those two paths.

190
swarm-builder/src/main.rs Normal file
View file

@ -0,0 +1,190 @@
//! The swarm builder's build service.
//!
//! Connects to the swarm queue as its own OIDC client, takes build requests
//! from swarm-controller on [`swarm_queue_client::builder::SUBJECT`], builds
//! each agent config at its commit, answers on the request's reply subject,
//! and logs one line per build. It writes nothing to the forge: the
//! controller turns the reply into a commit status.
//!
//! The store identity comes from the `BAO_*` environment. The queue client
//! secret is read at start and again when authelia refuses it, the forge token
//! once per build, both from the store under that identity.
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
use anyhow::{Context, Result, bail};
use futures_util::StreamExt as _;
use swarm_queue_client::builder::{BUILD_DEADLINE, BuildAgent, QUEUE_GROUP, Reply, SUBJECT};
use swarm_queue_client::stored_secret::StoredSecret;
use swarm_queue_client::{ClientSecret, QueueConfig};
use swarm_secret_client::forge::Credential;
mod nix_build;
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
/// writes it, matching the leaf's subject.
const CERT_ROLE: &str = "swarm-builder";
/// The environment prefix `swarm-builder.nix` sets.
const PREFIX: &str = "SWARM_BUILDER";
/// Seconds to wait after each failed read of the queue client secret. A
/// longer outage fails the start, and the unit's `Restart=` tries again.
const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30];
#[tokio::main]
async fn main() -> Result<()> {
hive_log::init();
swarm_queue_client::install_crypto_provider();
let var = |name: &str| {
let full = format!("{PREFIX}_{name}");
std::env::var(&full).with_context(|| format!("{full} is unset"))
};
let client_id = var("OIDC_CLIENT_ID")?;
let builder = Arc::new(nix_build::Builder {
nix: PathBuf::from(var("NIX")?),
check_flake: var("CHECK_FLAKE")?,
hyperhive: var("HYPERHIVE_FLAKE")?,
nixpkgs: var("NIXPKGS_FLAKE")?,
forge_url: var("FORGE_URL")?.trim_end_matches('/').to_owned(),
config_org: var("CONFIG_ORG")?,
deadline: BUILD_DEADLINE,
});
let secret = client_secret_with_retry(&client_id).await?;
let reread_id = client_id.clone();
let secret = StoredSecret::new(client_id.clone(), secret, move || {
let client_id = reread_id.clone();
async move { client_secret(&client_id).await.map_err(Into::into) }
});
let queue = swarm_queue_client::connect(QueueConfig {
url: var("NATS_URL")?,
token_endpoint: var("OIDC_TOKEN_ENDPOINT")?,
client_id,
client_secret: ClientSecret::Stored(secret),
ca_file: std::env::var(format!("{PREFIX}_OIDC_CA_FILE"))
.ok()
.map(PathBuf::from),
})
.await?;
let mut requests = queue
.queue_subscribe(SUBJECT, QUEUE_GROUP.to_owned())
.await
.with_context(|| format!("subscribing to {SUBJECT}"))?;
tracing::info!(subject = SUBJECT, "builder: awaiting builds");
while let Some(msg) = requests.next().await {
let Some(reply_to) = msg.reply.clone() else {
tracing::warn!("a build arrived with no reply subject; dropping it");
continue;
};
let queue = queue.clone();
let builder = Arc::clone(&builder);
tokio::spawn(async move {
let reply = handle(&msg.payload, &builder).await;
match serde_json::to_vec(&reply) {
Ok(body) => {
if let Err(e) = queue.publish(reply_to, body.into()).await {
tracing::warn!(error = %e, "publishing a build's reply failed");
}
}
Err(e) => tracing::warn!(error = %e, "encoding a build's reply failed"),
}
});
}
bail!("the subscription to {SUBJECT} ended")
}
/// Decode one request, build it and log the build.
async fn handle(payload: &[u8], builder: &nix_build::Builder) -> Reply {
let job: BuildAgent = match serde_json::from_slice(payload) {
Ok(job) => job,
Err(e) => {
tracing::warn!(error = %e, "an undecodable build; answering with the error");
return Reply::Failed {
error_tail: format!("the builder could not decode the request: {e}"),
};
}
};
let reply = match forge_token().await {
Ok(token) => builder.build(&job, &token).await,
Err(e) => Reply::Failed {
error_tail: format!("{e:#}"),
},
};
log_build(&job, &reply);
reply
}
/// The one journal line a build leaves.
fn log_build(job: &BuildAgent, reply: &Reply) {
match reply {
Reply::Built { out_path } => tracing::info!(
agent = %job.agent,
rev = %job.rev,
pr = job.pr,
%out_path,
"agent config built"
),
Reply::Failed { error_tail } => tracing::warn!(
agent = %job.agent,
rev = %job.rev,
pr = job.pr,
%error_tail,
"agent config build failed"
),
}
}
/// The forge token the builder fetches config repos with, read on every
/// build so a rotation takes effect on the next one.
async fn forge_token() -> Result<String> {
let path = swarm_secret_client::builder::forge_token_path()?;
let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE)
.await
.context("logging in to the secret store")?;
let stored: Option<Credential> = store
.read_optional(&path)
.await
.with_context(|| format!("reading {path}"))?;
match stored {
Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()),
_ => bail!("nothing at {path}: swarm-controller has not minted the builder's forge token"),
}
}
/// The queue client secret, read from the store, retried through a store that
/// is restarting alongside this service.
async fn client_secret_with_retry(client_id: &str) -> Result<String> {
for delay in RETRY_DELAYS_S.map(Duration::from_secs) {
match client_secret(client_id).await {
Ok(secret) => return Ok(secret),
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), retry_in = ?delay, "reading the queue client secret failed");
tokio::time::sleep(delay).await;
}
}
}
client_secret(client_id).await
}
async fn client_secret(client_id: &str) -> Result<String> {
use swarm_secret_client::builder::{ClientSecret, client_secret_path};
let path = client_secret_path(client_id)?;
let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE)
.await
.context("logging in to the secret store")?;
let stored: Option<ClientSecret> = store
.read_optional(&path)
.await
.with_context(|| format!("reading {path}"))?;
match stored {
Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()),
_ => bail!(
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \
published this service's client secret"
),
}
}

334
swarm-builder/src/nix_build.rs Normal file
View file

@ -0,0 +1,334 @@
//! One build: `nix build` of the check flake with its three inputs pointed at
//! this host's hyperhive and nixpkgs and at the agent's config commit.
//!
//! The config repo is private, so `git` fetches it with the builder's forge
//! token. The token reaches `git` through a credential helper that reads it
//! from the child's environment: it is in no argument, no URL and no lock, so
//! no error line `nix` prints can carry it into the reply.
use std::path::PathBuf;
use std::process::Stdio;
use std::time::Duration;
use swarm_queue_client::builder::{BuildAgent, Reply};
use tokio::io::AsyncReadExt as _;
/// The most of `nix`'s stderr a failed build's reply carries, from its end.
const TAIL_LINES: usize = 30;
const TAIL_BYTES: usize = 4000;
/// The variable the credential helper reads the token from, in the child only.
const TOKEN_VAR: &str = "SWARM_BUILDER_GIT_TOKEN";
/// What every build is run with, fixed at start.
pub struct Builder {
/// The `nix` binary.
pub nix: PathBuf,
/// The check flake's directory in the store.
pub check_flake: String,
/// The hyperhive source this host was deployed from.
pub hyperhive: String,
/// The nixpkgs source this host was deployed from.
pub nixpkgs: String,
/// `https://<forge>`, with no trailing slash.
pub forge_url: String,
/// The org the config repos are in.
pub config_org: String,
/// How long one build may run.
pub deadline: Duration,
}
impl Builder {
/// Build `job` with `token` as the forge credential.
pub async fn build(&self, job: &BuildAgent, token: &str) -> Reply {
if let Err(error) = validate(job) {
return Reply::Failed { error_tail: error };
}
let mut cmd = tokio::process::Command::new(&self.nix);
cmd.args(self.args(job))
.env(TOKEN_VAR, token)
.env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_CONFIG_COUNT", "1")
.env("GIT_CONFIG_KEY_0", "credential.helper")
.env("GIT_CONFIG_VALUE_0", credential_helper())
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.kill_on_drop(true);
run(cmd, self.deadline).await
}
/// The `nix build` arguments for `job`.
fn args(&self, job: &BuildAgent) -> Vec<String> {
[
"build",
"--no-link",
"--print-out-paths",
"--no-write-lock-file",
"--option",
"accept-flake-config",
"false",
"--print-build-logs",
]
.into_iter()
.map(str::to_owned)
.chain([
format!("{}#check", self.check_flake),
"--override-input".to_owned(),
"hyperhive".to_owned(),
self.hyperhive.clone(),
"--override-input".to_owned(),
"nixpkgs".to_owned(),
self.nixpkgs.clone(),
"--override-input".to_owned(),
"agent".to_owned(),
self.agent_url(job),
])
.collect()
}
/// The config commit as a flake reference, fetched through the PR's head
/// ref.
fn agent_url(&self, job: &BuildAgent) -> String {
format!(
"git+{}/{}/{}?ref=refs/pull/{}/head&rev={}",
self.forge_url, self.config_org, job.agent, job.pr, job.rev
)
}
}
/// A `git` credential helper answering every `get` with the builder's login
/// and the token from [`TOKEN_VAR`].
fn credential_helper() -> String {
format!(
"!f() {{ test \"$1\" = get && printf 'username=builder\\npassword=%s\\n' \"${TOKEN_VAR}\"; }}; f"
)
}
/// Refuse a job whose fields would change the reference's meaning: an agent
/// is one `[a-z0-9-]` segment, and a rev is a full commit id.
fn validate(job: &BuildAgent) -> Result<(), String> {
let agent_ok = !job.agent.is_empty()
&& job
.agent
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-');
if !agent_ok {
return Err(format!("{:?} is not an agent name", job.agent));
}
let rev_ok = job.rev.len() == 40
&& job
.rev
.bytes()
.all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b));
if !rev_ok {
return Err(format!("{:?} is not a full commit id", job.rev));
}
if job.pr == 0 {
return Err("PR 0 does not exist".to_owned());
}
Ok(())
}
/// Run `cmd` to its end or `deadline`, and read the store path it printed or
/// the tail of what it said.
async fn run(mut cmd: tokio::process::Command, deadline: Duration) -> Reply {
let mut child = match cmd.spawn() {
Ok(child) => child,
Err(e) => {
return Reply::Failed {
error_tail: format!("starting nix: {e}"),
};
}
};
let mut stdout = child.stdout.take().expect("piped");
let mut stderr = child.stderr.take().expect("piped");
let work = async {
let (mut out, mut err) = (Vec::new(), Vec::new());
let (o, e, status) = tokio::join!(
stdout.read_to_end(&mut out),
stderr.read_to_end(&mut err),
child.wait()
);
o.and(e).map(|_| ())?;
status.map(|s| (s, out, err))
};
match tokio::time::timeout(deadline, work).await {
Err(_) => Reply::Failed {
error_tail: format!(
"the build did not finish within {} minutes",
deadline.as_secs() / 60
),
},
Ok(Err(e)) => Reply::Failed {
error_tail: format!("waiting for nix: {e}"),
},
Ok(Ok((status, out, err))) => {
let out_path = String::from_utf8_lossy(&out)
.lines()
.rfind(|l| !l.trim().is_empty())
.map(|l| l.trim().to_owned());
match out_path {
Some(out_path) if status.success() => Reply::Built { out_path },
_ => Reply::Failed {
error_tail: tail(&String::from_utf8_lossy(&err), status),
},
}
}
}
}
/// The last [`TAIL_LINES`] lines of `stderr`, at most [`TAIL_BYTES`] bytes,
/// or the exit status when it said nothing.
fn tail(stderr: &str, status: std::process::ExitStatus) -> String {
let lines: Vec<&str> = stderr.lines().filter(|l| !l.trim().is_empty()).collect();
if lines.is_empty() {
return format!("nix exited with {status} and printed nothing");
}
let joined = lines[lines.len().saturating_sub(TAIL_LINES)..].join("\n");
let mut start = joined.len().saturating_sub(TAIL_BYTES);
while !joined.is_char_boundary(start) {
start += 1;
}
joined[start..].to_owned()
}
#[cfg(test)]
mod tests {
use super::*;
use swarm_queue_client::builder::BUILD_DEADLINE;
fn job() -> BuildAgent {
BuildAgent {
agent: "atlas".to_owned(),
rev: "0123456789abcdef0123456789abcdef01234567".to_owned(),
pr: 7,
}
}
fn builder() -> Builder {
Builder {
check_flake: "/nix/store/c-check".to_owned(),
hyperhive: "/nix/store/h-source".to_owned(),
nixpkgs: "/nix/store/n-source".to_owned(),
forge_url: "https://forge.example".to_owned(),
config_org: "agent-configs".to_owned(),
nix: PathBuf::from("nix"),
deadline: BUILD_DEADLINE,
}
}
#[test]
fn the_agent_is_fetched_at_its_commit_through_the_prs_ref() {
assert_eq!(
builder().agent_url(&job()),
"git+https://forge.example/agent-configs/atlas?ref=refs/pull/7/head\
&rev=0123456789abcdef0123456789abcdef01234567"
);
}
#[test]
fn every_input_is_overridden_and_no_lock_is_written() {
let args = builder().args(&job());
let overridden: Vec<&str> = args
.windows(2)
.filter(|w| w[0] == "--override-input")
.map(|w| w[1].as_str())
.collect();
assert_eq!(overridden, ["hyperhive", "nixpkgs", "agent"]);
assert!(args.iter().any(|a| a == "/nix/store/c-check#check"));
assert!(args.iter().any(|a| a == "--no-write-lock-file"));
}
#[test]
fn the_token_is_in_no_argument() {
let args = builder().args(&job()).join(" ");
assert!(!args.contains("password"));
assert!(credential_helper().contains(&format!("${TOKEN_VAR}")));
}
#[test]
fn a_job_that_would_bend_the_reference_is_refused() {
for bad in [
BuildAgent {
agent: "a?rev=x".to_owned(),
..job()
},
BuildAgent {
agent: String::new(),
..job()
},
BuildAgent {
rev: "main".to_owned(),
..job()
},
BuildAgent { pr: 0, ..job() },
] {
assert!(validate(&bad).is_err(), "{bad:?}");
}
assert!(validate(&job()).is_ok());
}
#[test]
fn the_tail_keeps_the_end_of_a_long_log() {
use std::os::unix::process::ExitStatusExt as _;
let log = (0..100)
.map(|i| format!("line {i}"))
.collect::<Vec<_>>()
.join("\n");
let t = tail(&log, std::process::ExitStatus::from_raw(256));
assert!(t.ends_with("line 99"));
assert!(!t.contains("line 69\n"));
assert!(t.starts_with("line 70"));
let quiet = tail("", std::process::ExitStatus::from_raw(256));
assert!(quiet.contains("printed nothing"));
}
fn sh(script: &str, deadline: Duration) -> impl Future<Output = Reply> {
let mut cmd = tokio::process::Command::new("sh");
cmd.args(["-c", script])
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.kill_on_drop(true);
run(cmd, deadline)
}
#[tokio::test]
async fn a_success_replies_with_the_last_path_printed() {
let reply = sh(
"echo warming >&2; echo /nix/store/abc-nixos-system",
Duration::from_secs(30),
)
.await;
assert_eq!(
reply,
Reply::Built {
out_path: "/nix/store/abc-nixos-system".to_owned()
}
);
}
#[tokio::test]
async fn a_failure_replies_with_what_nix_said_last() {
let reply = sh(
"echo first >&2; echo 'error: attribute missing' >&2; exit 1",
Duration::from_secs(30),
)
.await;
let Reply::Failed { error_tail } = reply else {
panic!("{reply:?}")
};
assert!(error_tail.ends_with("error: attribute missing"));
}
#[tokio::test]
async fn a_build_past_its_deadline_is_stopped() {
let reply = sh("sleep 30", Duration::from_millis(200)).await;
assert!(
matches!(&reply, Reply::Failed { error_tail } if error_tail.contains("did not finish")),
"{reply:?}"
);
}
}

304
swarm-controller/src/builder.rs Normal file
View file

@ -0,0 +1,304 @@
//! The `BuildAgent` node: send a config PR's head to the swarm builder over the
//! queue, and turn its answer into the [`CONTEXT`] commit status on that head.
//!
//! The status is the check's only record besides the node itself: nothing is
//! pushed, and the closure the builder made stays in its host's store. A red
//! status blocks nothing; the merge deploys as before.
use std::future::Future;
use forgejo_api::structs::CommitStatusState;
use hive_jobq::scheduler::Outcome;
use swarm_queue_client::builder::{BuildAgent, REPLY_DEADLINE, Reply, SUBJECT};
use crate::forge::{self, CONFIG_ORG};
/// The commit status context the check writes.
pub const CONTEXT: &str = "swarm-builder";
/// Where a build is sent: the queue in production, a fake in the tests.
/// `Err` is a build that never reached a builder or never came back.
pub(crate) trait BuildSink {
fn build(&self, request: Vec<u8>) -> impl Future<Output = Result<Reply, String>> + Send;
}
impl BuildSink for async_nats::Client {
async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
use async_nats::RequestErrorKind;
let sent = async_nats::Request::new()
.payload(request.into())
.timeout(Some(REPLY_DEADLINE));
match self.send_request(SUBJECT, sent).await {
Ok(msg) => serde_json::from_slice(&msg.payload)
.map_err(|e| format!("the builder's reply did not decode: {e}")),
Err(e) => Err(match e.kind() {
RequestErrorKind::NoResponders => "no swarm builder is listening: \
deploy.builder.enable is set on no host, or its service is not connected \
to the swarm queue"
.to_owned(),
RequestErrorKind::TimedOut => format!(
"the swarm builder sent no answer within {} minutes",
REPLY_DEADLINE.as_secs() / 60
),
_ => format!("sending the build to the swarm builder failed: {e}"),
}),
}
}
}
/// Where a status is written: the forge in production, a fake in the tests.
pub(crate) trait StatusSink {
fn set(
&self,
agent: &str,
rev: &str,
state: CommitStatusState,
description: &str,
) -> impl Future<Output = anyhow::Result<()>> + Send;
}
impl StatusSink for forge::Client {
async fn set(
&self,
agent: &str,
rev: &str,
state: CommitStatusState,
description: &str,
) -> anyhow::Result<()> {
forge::within_forge_deadline(
"build_agent status",
self.create_commit_status(CONFIG_ORG, agent, rev, CONTEXT, state, description),
)
.await
.unwrap_or_else(|| Err(anyhow::anyhow!("no answer from the forge")))
}
}
/// Build `job` and write its status. The node fails when the build does, so
/// the Jobs page shows red where the PR does.
pub(crate) async fn run(
queue: Option<&impl BuildSink>,
statuses: Option<&impl StatusSink>,
job: BuildAgent,
) -> Outcome {
let Some(statuses) = statuses else {
return Outcome::Failed(
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token not in the \
swarm secret store), so no build status can be written"
.to_owned(),
);
};
let set = |state, description: String| {
let job = &job;
async move {
statuses
.set(&job.agent, &job.rev, state, &description)
.await
.map_err(|e| format!("{e:#}"))
}
};
let Some(queue) = queue else {
let error = "no swarm queue is configured on this host, so no build can be sent";
return settle(set(CommitStatusState::Error, error.to_owned()).await, error);
};
if let Err(e) = set(
CommitStatusState::Pending,
"building on the swarm builder".to_owned(),
)
.await
{
tracing::warn!(agent = %job.agent, rev = %job.rev, error = %e, "build_agent: the pending status was not written");
}
let request = match serde_json::to_vec(&job) {
Ok(request) => request,
Err(e) => return Outcome::Failed(format!("encoding the build: {e}")),
};
match queue.build(request).await {
Ok(Reply::Built { out_path }) => {
tracing::info!(agent = %job.agent, rev = %job.rev, %out_path, "build_agent: built");
match set(CommitStatusState::Success, out_path).await {
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(format!("built, but the status was not written: {e}")),
}
}
Ok(Reply::Failed { error_tail }) => {
let last = last_line(&error_tail);
settle(
set(CommitStatusState::Failure, last.to_owned()).await,
&format!("the build failed: {error_tail}"),
)
}
Err(error) => settle(set(CommitStatusState::Error, error.clone()).await, &error),
}
}
/// A failed node carrying `error`, and the status write's own failure when
/// there was one.
fn settle(written: Result<(), String>, error: &str) -> Outcome {
match written {
Ok(()) => Outcome::Failed(error.to_owned()),
Err(e) => Outcome::Failed(format!("{error} (and the status was not written: {e})")),
}
}
/// The last non-blank line of `tail`, which for `nix` is the error itself.
fn last_line(tail: &str) -> &str {
tail.lines()
.rev()
.map(str::trim)
.find(|l| !l.is_empty())
.unwrap_or("the build failed")
}
#[cfg(test)]
mod tests {
use std::sync::Mutex;
use super::*;
struct FakeQueue(Result<Reply, String>);
impl BuildSink for FakeQueue {
async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
let sent: BuildAgent = serde_json::from_slice(&request).expect("a build request");
assert_eq!(sent, job());
self.0.clone()
}
}
#[derive(Default)]
struct FakeStatuses(Mutex<Vec<(CommitStatusState, String)>>);
impl StatusSink for FakeStatuses {
async fn set(
&self,
agent: &str,
rev: &str,
state: CommitStatusState,
description: &str,
) -> anyhow::Result<()> {
assert_eq!((agent, rev), ("atlas", "abc"));
self.0.lock().unwrap().push((state, description.to_owned()));
Ok(())
}
}
fn job() -> BuildAgent {
BuildAgent {
agent: "atlas".to_owned(),
rev: "abc".to_owned(),
pr: 3,
}
}
async fn statuses_after(queue: Option<FakeQueue>) -> (Outcome, Vec<CommitStatusState>) {
let statuses = FakeStatuses::default();
let outcome = run(queue.as_ref(), Some(&statuses), job()).await;
let states = statuses.0.into_inner().unwrap();
(outcome, states.into_iter().map(|(s, _)| s).collect())
}
#[tokio::test]
async fn a_built_config_goes_pending_then_green() {
let (outcome, states) = statuses_after(Some(FakeQueue(Ok(Reply::Built {
out_path: "/nix/store/x".to_owned(),
}))))
.await;
assert!(matches!(outcome, Outcome::Done));
assert_eq!(
states,
[CommitStatusState::Pending, CommitStatusState::Success]
);
}
#[tokio::test]
async fn a_failed_build_goes_red_with_its_last_line_and_fails_the_node() {
let statuses = FakeStatuses::default();
let queue = FakeQueue(Ok(Reply::Failed {
error_tail: "evaluating\nerror: boom\n".to_owned(),
}));
let outcome = run(Some(&queue), Some(&statuses), job()).await;
assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("error: boom")));
let written = statuses.0.into_inner().unwrap();
assert_eq!(
written.last(),
Some(&(CommitStatusState::Failure, "error: boom".to_owned()))
);
}
#[tokio::test]
async fn no_builder_is_an_error_status_not_a_failure() {
let (outcome, states) =
statuses_after(Some(FakeQueue(Err("no swarm builder".to_owned())))).await;
assert!(matches!(outcome, Outcome::Failed(_)));
assert_eq!(
states,
[CommitStatusState::Pending, CommitStatusState::Error]
);
}
#[tokio::test]
async fn no_queue_is_an_error_status() {
let (outcome, states) = statuses_after(None).await;
assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no swarm queue")));
assert_eq!(states, [CommitStatusState::Error]);
}
#[tokio::test]
async fn no_forge_fails_the_node_without_sending_the_build() {
struct Unreachable;
impl BuildSink for Unreachable {
async fn build(&self, _: Vec<u8>) -> Result<Reply, String> {
panic!("a build with nowhere to report must not be sent")
}
}
let outcome = run(Some(&Unreachable), None::<&FakeStatuses>, job()).await;
assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no forge access")));
}
/// The queue half against a real server: no builder listening names the
/// option, and a listener's reply comes back decoded.
#[tokio::test]
async fn a_build_reaches_a_listening_builder_and_fails_without_one() {
use futures_util::StreamExt as _;
let nats = crate::test_nats::TestNats::start();
let client = nats.client().await;
let unanswered = client.build(b"{}".to_vec()).await;
assert!(
matches!(&unanswered, Err(e) if e.contains("deploy.builder.enable")),
"{unanswered:?}"
);
let builder = nats.client().await;
let mut requests = builder
.queue_subscribe(SUBJECT, swarm_queue_client::builder::QUEUE_GROUP.to_owned())
.await
.expect("subscribe");
// A message the builder sends itself and receives back proves the
// server processed the subscription before it.
let mut probe = builder
.subscribe("builder-test.probe")
.await
.expect("subscribe");
builder
.publish("builder-test.probe", "".into())
.await
.expect("publish");
probe.next().await.expect("the probe comes back");
let built = Reply::Built {
out_path: "/nix/store/x".to_owned(),
};
let answer = built.clone();
tokio::spawn(async move {
let msg = requests.next().await.expect("a request");
let reply = serde_json::to_vec(&answer).expect("encodes");
builder
.publish(msg.reply.expect("a reply subject"), reply.into())
.await
.expect("publish");
});
assert_eq!(client.build(b"{}".to_vec()).await, Ok(built));
}
}

86
swarm-controller/src/config_pr.rs
View file

@ -16,8 +16,10 @@
//! instead of waiting up to `POLL_INTERVAL`. //! instead of waiting up to `POLL_INTERVAL`.
//! //!
//! [`merged`] reads the same delivery for a merge, which the webhook handler //! [`merged`] reads the same delivery for a merge, which the webhook handler
//! turns into a deploy. The poll has no counterpart: it lists open PRs only, //! turns into a deploy, and [`changed_head`] reads it for a PR's new head,
//! so a merge whose delivery is lost deploys nothing. //! which the handler turns into a build check (`crate::builder`). The poll
//! has no counterpart to either: a merge whose delivery is lost deploys
//! nothing, and a head whose delivery is lost is not built.
use std::collections::HashMap; use std::collections::HashMap;
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
@ -58,12 +60,17 @@ struct WebhookPullRequest {
/// hive builds its config repo's `main`. /// hive builds its config repo's `main`.
#[serde(default)] #[serde(default)]
base: Option<WebhookBranch>, base: Option<WebhookBranch>,
/// The PR's branch, whose `sha` is the commit a build check builds.
#[serde(default)]
head: Option<WebhookBranch>,
} }
#[derive(Deserialize)] #[derive(Deserialize)]
struct WebhookBranch { struct WebhookBranch {
#[serde(rename = "ref")] #[serde(rename = "ref")]
name: String, name: String,
#[serde(default)]
sha: Option<String>,
} }
/// A config PR that was just merged: whose config, and the commit to deploy. /// A config PR that was just merged: whose config, and the commit to deploy.
@ -104,6 +111,43 @@ pub fn merged(body: &[u8]) -> Option<MergedConfigPr> {
}) })
} }
/// A config PR whose head just changed: whose config, the head commit, and
/// the PR.
#[derive(Debug, PartialEq, Eq)]
pub struct ChangedConfigPr {
pub agent: String,
pub rev: String,
pub pr: u64,
}
/// The new head a verified `ConfigPr` delivery reports, if it reports one: a
/// PR opened, reopened, or pushed to (`synchronized`). Edits and labels leave
/// the head as it was and build nothing. A body that does not parse is
/// `None`; [`ConfigPrCache::apply_webhook_delivery`] already logs it.
pub fn changed_head(body: &[u8]) -> Option<ChangedConfigPr> {
let payload: ConfigPrWebhookPayload = serde_json::from_slice(body).ok()?;
if !matches!(
payload.action.as_str(),
"opened" | "reopened" | "synchronized"
) || payload.pull_request.state != "open"
{
return None;
}
let Some(rev) = payload.pull_request.head.and_then(|h| h.sha) else {
tracing::warn!(
agent = %payload.repository.name,
pr = payload.pull_request.number,
"config-pr webhook: PR carries no head sha; nothing built"
);
return None;
};
Some(ChangedConfigPr {
agent: payload.repository.name,
rev,
pr: payload.pull_request.number,
})
}
#[derive(Deserialize)] #[derive(Deserialize)]
struct WebhookRepository { struct WebhookRepository {
/// The config repo's name IS the agent's name — same convention /// The config repo's name IS the agent's name — same convention
@ -369,4 +413,42 @@ mod tests {
.expect("prior entry must survive a bad delivery"); .expect("prior entry must survive a bad delivery");
assert_eq!(status.pr_number, 5); assert_eq!(status.pr_number, 5);
} }
fn head_changed(action: &str) -> Vec<u8> {
serde_json::json!({
"action": action,
"pull_request": {
"number": 5,
"state": "open",
"head": { "ref": "fix", "sha": "def456" },
"base": { "ref": "main", "sha": "abc123" },
},
"repository": { "name": "damocles" },
})
.to_string()
.into_bytes()
}
#[test]
fn an_opened_reopened_or_pushed_pr_names_its_head() {
for action in ["opened", "reopened", "synchronized"] {
assert_eq!(
super::changed_head(&head_changed(action)),
Some(super::ChangedConfigPr {
agent: "damocles".to_owned(),
rev: "def456".to_owned(),
pr: 5,
}),
"{action}"
);
}
}
#[test]
fn an_edit_a_label_or_a_close_builds_nothing() {
for action in ["edited", "label_updated", "closed"] {
assert_eq!(super::changed_head(&head_changed(action)), None, "{action}");
}
assert_eq!(super::changed_head(&closed(true)), None);
}
} }

102
swarm-controller/src/forge.rs
View file

@ -21,12 +21,12 @@
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use forgejo_api::structs::{ use forgejo_api::structs::{
AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation, AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
ChangeFileOperationOperation, ChangeFilesOptions, CreateBranchProtectionOption, ChangeFileOperationOperation, ChangeFilesOptions, CommitStatusState,
CreateHookOption, CreateHookOptionConfig, CreateHookOptionType, CreateRepoOption, CreateBranchProtectionOption, CreateHookOption, CreateHookOptionConfig, CreateHookOptionType,
CreateUserOption, EditUserOption, GetTreeQuery, GitEntry, IssueListIssuesQuery, CreateRepoOption, CreateStatusOption, CreateUserOption, EditUserOption, GetTreeQuery, GitEntry,
IssueListIssuesQueryState, IssueListIssuesQueryType, RepoDownloadPullDiffOrPatchQuery, IssueListIssuesQuery, IssueListIssuesQueryState, IssueListIssuesQueryType,
RepoGetContentsQuery, RepoGetPullRequestCommitsQuery, RepoListPullRequestsQuery, RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery, RepoGetPullRequestCommitsQuery,
RepoListPullRequestsQueryState, RepoSearchQuery, StateType, RepoListPullRequestsQuery, RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
}; };
use forgejo_api::{ApiErrorKind, Auth, Forgejo, ForgejoError}; use forgejo_api::{ApiErrorKind, Auth, Forgejo, ForgejoError};
use futures_util::{StreamExt as _, TryStreamExt as _}; use futures_util::{StreamExt as _, TryStreamExt as _};
@ -40,6 +40,7 @@ use crate::webhook::DeliveryKind;
use swarm_queue_client::review_classifier::Commit; use swarm_queue_client::review_classifier::Commit;
pub mod agent_token; pub mod agent_token;
pub mod builder_account;
pub mod classifier_account; pub mod classifier_account;
pub mod docs_grant; pub mod docs_grant;
pub mod hive_token; pub mod hive_token;
@ -48,6 +49,24 @@ pub mod objects;
pub mod runner_token; pub mod runner_token;
pub mod site_admin; pub mod site_admin;
/// The longest commit status description [`Client::create_commit_status`]
/// sends, in characters. The forge keeps more; its PR view shows one line.
pub const STATUS_DESCRIPTION_MAX: usize = 200;
/// `description` cut to [`STATUS_DESCRIPTION_MAX`] characters, an ellipsis
/// marking a cut.
fn status_description(description: &str) -> String {
if description.chars().count() <= STATUS_DESCRIPTION_MAX {
return description.to_owned();
}
let mut cut: String = description
.chars()
.take(STATUS_DESCRIPTION_MAX - 1)
.collect();
cut.push('…');
cut
}
/// An agent's open config-PR, as [`Client::list_open_config_prs`] reports it /// An agent's open config-PR, as [`Client::list_open_config_prs`] reports it
/// and `GET /api/agents/{name}/config-pr` serves it. /// and `GET /api/agents/{name}/config-pr` serves it.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)] #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)]
@ -163,7 +182,7 @@ const OPERATORS_TEAM: &str = "operators";
/// The merge gate survives the distinction: [`objects`] provisions the /// The merge gate survives the distinction: [`objects`] provisions the
/// `operators` team in **both** orgs precisely so branch protection can be /// `operators` team in **both** orgs precisely so branch protection can be
/// applied in either. /// applied in either.
const CONFIG_ORG: &str = "agent-configs"; pub(crate) const CONFIG_ORG: &str = "agent-configs";
/// The hive-wide knowledge repo, where the `push` hook lives. Same values as /// The hive-wide knowledge repo, where the `push` hook lives. Same values as
/// `hive-c0re::workers::knowledge::{ORG, REPO}`. /// `hive-c0re::workers::knowledge::{ORG, REPO}`.
@ -1127,6 +1146,38 @@ impl Client {
Ok(rows.into_iter().flatten().collect()) Ok(rows.into_iter().flatten().collect())
} }
/// Set the commit status named `context` on `sha` of `owner/repo` to
/// `state`, with `description` cut to [`STATUS_DESCRIPTION_MAX`]
/// characters.
///
/// # Errors
/// The forge refused the status.
pub async fn create_commit_status(
&self,
owner: &str,
repo: &str,
sha: &str,
context: &str,
state: CommitStatusState,
description: &str,
) -> Result<()> {
self.api
.repo_create_status(
owner,
repo,
sha,
CreateStatusOption {
context: Some(context.to_owned()),
description: Some(status_description(description)),
state: Some(state),
target_url: None,
},
)
.await
.with_context(|| format!("set the {context} status on {sha} of {owner}/{repo}"))?;
Ok(())
}
/// What the review classifier reads of `owner/repo`'s PR `number`: its /// What the review classifier reads of `owner/repo`'s PR `number`: its
/// head commit, its commits oldest first with their full messages, and /// head commit, its commits oldest first with their full messages, and
/// its unified diff. /// its unified diff.
@ -2368,4 +2419,41 @@ mod tests {
.unwrap_err(); .unwrap_err();
assert!(!no_repo(&tree), "{tree:#}"); assert!(!no_repo(&tree), "{tree:#}");
} }
#[test]
fn a_long_status_description_is_cut_with_an_ellipsis() {
assert_eq!(status_description("built"), "built");
let long = "é".repeat(STATUS_DESCRIPTION_MAX + 5);
let cut = status_description(&long);
assert_eq!(cut.chars().count(), STATUS_DESCRIPTION_MAX);
assert!(cut.ends_with('…'));
}
#[tokio::test]
async fn a_commit_status_is_posted_on_the_commit() {
let (client, seen) = stub_forge(
(201, r#"{"id":1}"#),
(404, r#"{"message":"not stubbed"}"#),
(404, r#"{"message":"not stubbed"}"#),
)
.await;
client
.create_commit_status(
CONFIG_ORG,
"atlas",
"abc",
"swarm-builder",
CommitStatusState::Failure,
"error: boom",
)
.await
.expect("the stub accepts the status");
let seen = seen.lock().unwrap();
let (method, path, body) = seen.first().expect("one request");
assert_eq!(method, "POST");
assert_eq!(path, "/api/v1/repos/agent-configs/atlas/statuses/abc");
assert_eq!(body["context"], "swarm-builder");
assert_eq!(body["state"], "failure");
assert_eq!(body["description"], "error: boom");
}
} }

284
swarm-controller/src/forge/builder_account.rs Normal file
View file

@ -0,0 +1,284 @@
//! The swarm builder's read-only forge account, [`LOGIN`]: created here with
//! the admin API, put in the `hives` team of `agent-configs` (read on every
//! config repo, including ones created later), and given one token, stored
//! at `swarm_secret_client::builder::forge_token_path`, where the builder
//! reads it under its own store identity to fetch the config it builds.
//!
//! The builder never writes to the forge: its build results become commit
//! statuses through this daemon's own client (`crate::builder`).
//!
//! The token decision is [`super::agent_token`]'s, under this module's name
//! and scopes, so a stored token the forge still lists with the same last
//! eight characters and scopes is kept, and a pass whose reads fail queues
//! nothing.
use std::sync::Arc;
use anyhow::{Context, Result, bail};
use forgejo_api::structs::RepoSearchQuery;
use forgejo_api::{Auth, Forgejo};
use swarm_secret_client::builder::forge_token_path;
use swarm_secret_client::forge;
use super::Client;
use super::agent_token::{Decision, Observed, classify_named, is_not_found};
use super::hive_token::TeamSeen;
/// The builder's forge login.
pub const LOGIN: &str = "builder";
/// The forge's name for the account's one token.
pub const TOKEN_NAME: &str = "swarm-builder";
/// What the token may do: read repositories. Which repositories is the
/// account's business, and the account is only in the `hives` team.
pub const TOKEN_SCOPES: &str = "read:repository";
/// How often [`spawn`] re-checks the account and its token.
const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
/// The email this module creates the account with, and the one it requires
/// before acting on an account named [`LOGIN`]. Under its own domain, so it
/// equals no agent's or hive's account email.
const EMAIL: &str = "builder@services.hyperhive.local";
/// Whether a pass that found `seen`, and `member` for the account's place in
/// the `hives` team, queues a `MintBuilderForgeToken` job.
/// [`Observed::Unknown`] never does, so a forge or store outage can't rotate
/// the token, and neither does a [`Decision::Conflict`], which the job would
/// refuse. A kept token queues only when the account is known to be out of
/// the team; that job finds the token current and does not rotate it.
pub fn needs_job(seen: Observed, member: Option<bool>) -> bool {
match seen {
Observed::NoForgeUser | Observed::Decided(Decision::Mint(_)) => true,
Observed::Decided(Decision::Keep) => member == Some(false),
Observed::Unknown | Observed::Decided(Decision::Conflict) => false,
}
}
impl Client {
/// Make sure [`LOGIN`] exists, is in the `hives` team, and holds a live
/// [`TOKEN_NAME`] token in the store, minting one when it does not. The
/// whole job of the `MintBuilderForgeToken` node.
///
/// # Errors
/// When the store or the forge refuses a step, when an account named
/// [`LOGIN`] exists with another email, or when the read-back does not
/// show the new token.
pub async fn ensure_builder_forge_token(&self) -> Result<()> {
let path = forge_token_path()?;
self.ensure_builder_user().await?;
let team = self.ensure_hives_team().await?;
self.api
.org_add_team_member(team, LOGIN)
.await
.with_context(|| format!("adding {LOGIN} to the hives team"))?;
let store = crate::store::connect()
.await
.context("logging in to the swarm secret store")?;
let (stored, listed) = self.read_builder_token(&store, &path).await?;
let reason = match classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed) {
Decision::Keep => {
tracing::debug!("builder forge token is current; left as it is");
return Ok(());
}
Decision::Conflict => bail!(
"the forge lists more than one {TOKEN_NAME:?} token for {LOGIN:?}; delete them \
by id and re-run"
),
Decision::Mint(reason) => reason,
};
let credential = self
.replace_token(&store, LOGIN, TOKEN_NAME, TOKEN_SCOPES, &listed, &path)
.await?;
Forgejo::new(Auth::Token(&credential.value), self.url.clone())
.context("building a forge client with the new token")?
.repo_search(RepoSearchQuery::default())
.await
.with_context(|| format!("using {LOGIN}'s new forge token"))?;
let (stored, listed) = self.read_builder_token(&store, &path).await?;
let now = classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed);
if now != Decision::Keep {
bail!("{LOGIN}'s new forge token does not read back as current: {now:?}");
}
tracing::info!(?reason, %path, "builder forge token minted and stored");
Ok(())
}
/// The stored token at `path` and every token the forge lists for
/// [`LOGIN`], which must exist.
async fn read_builder_token(
&self,
store: &swarm_secret_client::SecretStore,
path: &str,
) -> Result<(
Option<forge::Credential>,
Vec<forgejo_api::structs::AccessToken>,
)> {
let stored = store
.read_optional(path)
.await
.with_context(|| format!("reading {path}"))?;
let Some(listed) = self.list_agent_tokens(LOGIN).await? else {
bail!("the forge has no user {LOGIN:?} after this job created it");
};
Ok((stored, listed))
}
/// Create [`LOGIN`] if there is none, refuse one with another email, and
/// lock it out of creating repos.
async fn ensure_builder_user(&self) -> Result<()> {
self.create_user(LOGIN, EMAIL).await?;
let user = self
.api
.user_get(LOGIN)
.await
.with_context(|| format!("read forge user {LOGIN}"))?;
if user.email.as_deref() != Some(EMAIL) {
bail!(
"forge account `{LOGIN}` exists but is not the swarm builder's (email {:?}, \
expected `{EMAIL}`); refusing to give it read on every config repo",
user.email
);
}
self.api
.admin_edit_user(LOGIN, super::repo_creation_lockdown(LOGIN))
.await
.with_context(|| format!("disable repo creation for forge user {LOGIN}"))?;
Ok(())
}
/// What the store and the forge say about the account's token, and
/// whether it is in the `hives` team (`None` when that read failed).
async fn observe_builder(&self) -> Result<(Observed, Option<bool>)> {
let path = forge_token_path()?;
let store = crate::store::connect()
.await
.context("logging in to the swarm secret store")?;
let stored = match store.read_optional::<forge::Credential>(&path).await {
Ok(stored) => stored,
Err(e) => {
tracing::warn!(error = %e, "builder forge token: store read failed");
return Ok((Observed::Unknown, None));
}
};
let seen = match self.list_agent_tokens(LOGIN).await {
Ok(Some(listed)) => Observed::Decided(classify_named(
TOKEN_NAME,
TOKEN_SCOPES,
stored.as_ref(),
&listed,
)),
Ok(None) => Observed::NoForgeUser,
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "builder forge token: forge read failed");
Observed::Unknown
}
};
let member = match self.observe_hives_team().await {
TeamSeen::Matches(id) | TeamSeen::Drifted(id) => {
match self.api.org_list_team_member(id, LOGIN).await {
Ok(_) => Some(true),
Err(e) if is_not_found(&e) => Some(false),
Err(e) => {
tracing::warn!(error = %e, "builder forge token: team membership read failed");
None
}
}
}
TeamSeen::Absent => Some(false),
TeamSeen::Unknown => None,
};
Ok((seen, member))
}
}
/// Check the account now and every [`RECONCILE_INTERVAL`] after, and call
/// `enqueue`, which inserts a `MintBuilderForgeToken` node, when it needs a
/// job. A pass that fails is logged and retried on the next tick.
pub fn spawn(client: Arc<Client>, enqueue: impl Fn() + Send + 'static) {
tokio::spawn(async move {
let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
loop {
ticker.tick().await;
let Some(outcome) =
super::within_forge_deadline("builder forge token", client.observe_builder()).await
else {
continue;
};
match outcome {
Ok((seen, member)) if needs_job(seen, member) => {
tracing::info!(?seen, ?member, "builder forge token: queueing a job");
enqueue();
}
Ok((seen, member)) => {
tracing::debug!(?seen, ?member, "builder forge token: no job");
}
Err(e) => tracing::warn!(
error = %format!("{e:#}"),
retry_in_s = RECONCILE_INTERVAL.as_secs(),
"builder forge token: pass failed; retrying next tick"
),
}
}
});
}
#[cfg(test)]
mod tests {
use super::super::agent_token::MintReason;
use super::*;
#[test]
fn the_token_reads_repositories_and_nothing_else() {
assert_eq!(TOKEN_SCOPES, "read:repository");
assert_ne!(TOKEN_NAME, super::super::agent_token::AGENT_TOKEN_NAME);
assert_ne!(TOKEN_NAME, super::super::hive_token::HIVE_TOKEN_NAME);
assert_ne!(TOKEN_NAME, super::super::classifier_account::TOKEN_NAME);
}
#[test]
fn a_mint_a_missing_account_or_a_lost_membership_queues() {
let mint = Observed::Decided(Decision::Mint(MintReason::NotStored));
let keep = Observed::Decided(Decision::Keep);
assert!(needs_job(Observed::NoForgeUser, None));
assert!(needs_job(mint, Some(true)));
assert!(needs_job(keep, Some(false)));
assert!(!needs_job(keep, Some(true)));
assert!(!needs_job(keep, None));
assert!(!needs_job(
Observed::Decided(Decision::Conflict),
Some(false)
));
assert!(!needs_job(Observed::Unknown, Some(false)));
}
#[test]
fn the_accounts_email_is_no_agents_or_hives() {
assert_ne!(EMAIL, super::super::agent_email(LOGIN));
assert_ne!(EMAIL, super::super::legacy_agent_email(LOGIN));
assert!(!LOGIN.starts_with(hive_types::RESERVED_AGENT_PREFIX));
}
/// An existing account named like the builder with an agent's email is
/// refused before the lockdown, so it never joins the team.
#[tokio::test]
async fn the_job_refuses_an_account_that_is_not_the_builders() {
use super::super::tests::{lockdown_patch, stub_forge};
let (client, seen) = stub_forge(
(409, r#"{"message":"user already exists"}"#),
(200, r#"{"email":"builder@hyperhive.local"}"#),
(200, "{}"),
)
.await;
let err = client
.ensure_builder_user()
.await
.expect_err("an agent's account is not the builder's");
assert!(format!("{err:#}").contains("not the swarm builder's"));
assert!(lockdown_patch(&seen, LOGIN).is_none());
}
}

4
swarm-controller/src/forge/hive_token.rs
View file

@ -235,7 +235,7 @@ impl Client {
/// The id of [`HIVES_TEAM`] in `agent-configs`, creating it or /// The id of [`HIVES_TEAM`] in `agent-configs`, creating it or
/// correcting its settings first when needed. Its members are not /// correcting its settings first when needed. Its members are not
/// touched here. /// touched here.
async fn ensure_hives_team(&self) -> Result<i64> { pub(super) async fn ensure_hives_team(&self) -> Result<i64> {
match self.observe_hives_team().await { match self.observe_hives_team().await {
TeamSeen::Matches(id) => Ok(id), TeamSeen::Matches(id) => Ok(id),
TeamSeen::Drifted(id) => { TeamSeen::Drifted(id) => {
@ -292,7 +292,7 @@ impl Client {
} }
} }
async fn observe_hives_team(&self) -> TeamSeen { pub(super) async fn observe_hives_team(&self) -> TeamSeen {
match self.api.org_list_teams(CONFIG_ORG).await { match self.api.org_list_teams(CONFIG_ORG).await {
Ok((_, teams)) => teams Ok((_, teams)) => teams
.into_iter() .into_iter()

115
swarm-controller/src/main.rs
View file

@ -49,6 +49,7 @@ mod agent_renewal;
mod agent_state_stream; mod agent_state_stream;
mod agent_status; mod agent_status;
mod auth; mod auth;
mod builder;
mod config_pr; mod config_pr;
mod forge; mod forge;
mod forge_account; mod forge_account;
@ -153,6 +154,17 @@ enum SwarmNodeKind {
/// live token in the swarm store, where this daemon reads it to post the /// live token in the swarm store, where this daemon reads it to post the
/// classifier's reviews. See `forge::classifier_account`. /// classifier's reviews. See `forge::classifier_account`.
MintClassifierForgeToken, MintClassifierForgeToken,
/// Make sure the swarm builder's read-only forge account exists, is in
/// the `hives` team, and holds a live token in the swarm store, where the
/// builder reads it to fetch config repos. See `forge::builder_account`.
MintBuilderForgeToken,
/// Build `agent`'s config at `rev`, the head of its config PR `pr`, on
/// the swarm builder, and write the result as a commit status on `rev`.
/// See `builder`.
///
/// Carries no hive: the check builds on the builder's own agent base,
/// not on any hive.
BuildAgent { agent: String, rev: String, pr: u64 },
/// Re-mint `agent`'s queue secret if it is still stored and old enough. /// Re-mint `agent`'s queue secret if it is still stored and old enough.
/// See `agent_renewal`. /// See `agent_renewal`.
/// ///
@ -194,6 +206,8 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
SwarmNodeKind::MintHiveSenderToken { .. } => "mint_hive_sender_token".to_owned(), SwarmNodeKind::MintHiveSenderToken { .. } => "mint_hive_sender_token".to_owned(),
SwarmNodeKind::MintHiveForgeToken { .. } => "mint_hive_forge_token".to_owned(), SwarmNodeKind::MintHiveForgeToken { .. } => "mint_hive_forge_token".to_owned(),
SwarmNodeKind::MintClassifierForgeToken => "mint_classifier_forge_token".to_owned(), SwarmNodeKind::MintClassifierForgeToken => "mint_classifier_forge_token".to_owned(),
SwarmNodeKind::MintBuilderForgeToken => "mint_builder_forge_token".to_owned(),
SwarmNodeKind::BuildAgent { .. } => "build_agent".to_owned(),
SwarmNodeKind::RenewAgentQueueCredential { .. } => { SwarmNodeKind::RenewAgentQueueCredential { .. } => {
"renew_agent_queue_credential".to_owned() "renew_agent_queue_credential".to_owned()
} }
@ -224,7 +238,12 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
} }
SwarmNodeKind::MintHiveSenderToken { hive } SwarmNodeKind::MintHiveSenderToken { hive }
| SwarmNodeKind::MintHiveForgeToken { hive } => serde_json::json!({ "hive": hive }), | SwarmNodeKind::MintHiveForgeToken { hive } => serde_json::json!({ "hive": hive }),
SwarmNodeKind::MintClassifierForgeToken => serde_json::json!({}), SwarmNodeKind::MintClassifierForgeToken | SwarmNodeKind::MintBuilderForgeToken => {
serde_json::json!({})
}
SwarmNodeKind::BuildAgent { agent, rev, pr } => {
serde_json::json!({ "agent": agent, "rev": rev, "pr": pr })
}
SwarmNodeKind::TriggerDeploy { hive, agent, rev } => { SwarmNodeKind::TriggerDeploy { hive, agent, rev } => {
serde_json::json!({ "agent": agent, "hive": hive, "rev": rev }) serde_json::json!({ "agent": agent, "hive": hive, "rev": rev })
} }
@ -382,6 +401,10 @@ async fn run_swarm_node(
} }
SwarmNodeKind::MintHiveForgeToken { hive } => mint_hive_forge_token(forge, &hive).await, SwarmNodeKind::MintHiveForgeToken { hive } => mint_hive_forge_token(forge, &hive).await,
SwarmNodeKind::MintClassifierForgeToken => mint_classifier_forge_token(forge).await, SwarmNodeKind::MintClassifierForgeToken => mint_classifier_forge_token(forge).await,
SwarmNodeKind::MintBuilderForgeToken => mint_builder_forge_token(forge).await,
SwarmNodeKind::BuildAgent { agent, rev, pr } => {
build_agent(&deps, forge, agent, rev, pr).await
}
SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await, SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await,
SwarmNodeKind::SetAgentWanted { hive, agent } => match deps.wanted { SwarmNodeKind::SetAgentWanted { hive, agent } => match deps.wanted {
None => Outcome::Failed( None => Outcome::Failed(
@ -536,6 +559,42 @@ async fn mint_hive_forge_token(
} }
} }
/// The `BuildAgent` arm, lifted out for `clippy::too_many_lines`.
async fn build_agent(
deps: &WorkerDeps,
forge: Option<Arc<forge::Client>>,
agent: String,
rev: String,
pr: u64,
) -> hive_jobq::scheduler::Outcome {
let job = swarm_queue_client::builder::BuildAgent { agent, rev, pr };
builder::run(deps.queue.as_ref(), forge.as_deref(), job).await
}
/// The `MintBuilderForgeToken` arm, lifted out for the same reason.
async fn mint_builder_forge_token(
forge: Option<Arc<forge::Client>>,
) -> hive_jobq::scheduler::Outcome {
use hive_jobq::scheduler::Outcome;
let Some(client) = forge else {
return Outcome::Failed(
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
not in the swarm secret store), so no builder forge token can be minted"
.to_owned(),
);
};
match forge_node(
"mint_builder_forge_token",
client.ensure_builder_forge_token(),
)
.await
{
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(format!("{e:#}")),
}
}
/// The `MintClassifierForgeToken` arm, lifted out for the same reason. /// The `MintClassifierForgeToken` arm, lifted out for the same reason.
async fn mint_classifier_forge_token( async fn mint_classifier_forge_token(
forge: Option<Arc<forge::Client>>, forge: Option<Arc<forge::Client>>,
@ -1897,6 +1956,33 @@ fn claiming_hives(
.collect() .collect()
} }
/// Queue a build check of a config PR's new head. Every head is built, on
/// whichever hive the agent runs: the check builds on the builder's own
/// agent base.
fn queue_config_build(state: &AppState, changed: config_pr::ChangedConfigPr) {
let config_pr::ChangedConfigPr { agent, rev, pr } = changed;
let inserted = state
.jobq
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert_job(None, |b| {
vec![
b.node(SwarmNodeKind::BuildAgent {
agent: agent.clone(),
rev: rev.clone(),
pr,
})
.guid(),
]
});
match inserted {
Ok(_) => tracing::info!(%agent, %rev, pr, "config-pr head: build queued"),
Err(e) => {
tracing::warn!(%agent, %rev, pr, error = %e, "config-pr head: queueing the build failed");
}
}
}
/// Queue the deploy of a merged config PR on the one hive that runs its agent. /// Queue the deploy of a merged config PR on the one hive that runs its agent.
/// ///
/// The hive is found by scanning every hive's wanted state, since nothing /// The hive is found by scanning every hive's wanted state, since nothing
@ -2467,6 +2553,20 @@ fn queue_classifier_forge_token_mint(
.map_err(|e| anyhow::anyhow!("{e}")) .map_err(|e| anyhow::anyhow!("{e}"))
} }
/// Insert one `MintBuilderForgeToken` job and return its node's id.
/// `forge::builder_account::spawn`'s periodic pass is the one caller.
fn queue_builder_forge_token_mint(
sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>,
) -> Result<Vec<hive_jobq::NodeId>> {
sched
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert_job(None, |b| {
vec![b.node(SwarmNodeKind::MintBuilderForgeToken).guid()]
})
.map_err(|e| anyhow::anyhow!("{e}"))
}
/// The homeserver agents' own matrix accounts are minted on. Unset is this /// The homeserver agents' own matrix accounts are minted on. Unset is this
/// controller creating agents with no matrix account, which the mint node /// controller creating agents with no matrix account, which the mint node
/// reports by name rather than `main` refusing to start. /// reports by name rather than `main` refusing to start.
@ -2509,7 +2609,8 @@ fn spawn_matrix_account_backfill(
} }
/// Start the forge's periodic passes — the swarm-wide objects, the runner /// Start the forge's periodic passes — the swarm-wide objects, the runner
/// registration token, hives' read tokens, agents' `internal/docs` grants, and /// registration token, hives' and the builder's read tokens, agents'
/// `internal/docs` grants, and
/// agents' tokens (which also /// agents' tokens (which also
/// sweeps legacy tokens; see `forge::legacy_tokens`) — /// sweeps legacy tokens; see `forge::legacy_tokens`) —
/// once there is a forge client. Lifted out of `main` for /// once there is a forge client. Lifted out of `main` for
@ -2535,6 +2636,12 @@ fn spawn_forge_workers(
} }
}); });
let sched = Arc::clone(jobq); let sched = Arc::clone(jobq);
forge::builder_account::spawn(Arc::clone(&client), move || {
if let Err(e) = queue_builder_forge_token_mint(&sched) {
tracing::warn!(error = %format!("{e:#}"), "builder forge token: queueing failed");
}
});
let sched = Arc::clone(jobq);
forge::docs_grant::spawn(Arc::clone(&client), move |agents| { forge::docs_grant::spawn(Arc::clone(&client), move |agents| {
if let Err(e) = queue_docs_grants(&sched, agents) { if let Err(e) = queue_docs_grants(&sched, agents) {
tracing::warn!(error = %format!("{e:#}"), "docs grants: queueing failed"); tracing::warn!(error = %format!("{e:#}"), "docs grants: queueing failed");
@ -4262,6 +4369,10 @@ mod tests {
"mint_classifier_forge_token", "mint_classifier_forge_token",
SwarmNodeKind::MintClassifierForgeToken, SwarmNodeKind::MintClassifierForgeToken,
), ),
(
"mint_builder_forge_token",
SwarmNodeKind::MintBuilderForgeToken,
),
]; ];
for (label, kind) in nodes { for (label, kind) in nodes {
let mut sched = hive_jobq::scheduler::Scheduler::new( let mut sched = hive_jobq::scheduler::Scheduler::new(

3
swarm-controller/src/webhook.rs
View file

@ -418,6 +418,9 @@ pub(super) async fn post_webhook_forge(
if let Some(merged) = crate::config_pr::merged(&body) { if let Some(merged) = crate::config_pr::merged(&body) {
super::queue_merged_config_deploy(&state, merged).await; super::queue_merged_config_deploy(&state, merged).await;
} }
if let Some(changed) = crate::config_pr::changed_head(&body) {
super::queue_config_build(&state, changed);
}
} }
DeliveryKind::VcsActivity => { DeliveryKind::VcsActivity => {
if let Some(activity) = crate::vcs_metrics::parse_push(&body) { if let Some(activity) = crate::vcs_metrics::parse_push(&body) {

8
swarm-nats-auth/src/main.rs
View file

@ -162,6 +162,11 @@ struct Args {
#[arg(long = "review-classifier-client")] #[arg(long = "review-classifier-client")]
review_classifier_clients: Vec<String>, review_classifier_clients: Vec<String>,
/// Client ids that answer the swarm builder's build requests.
/// Repeatable, empty by default, which denies such a client.
#[arg(long = "builder-client")]
builder_clients: Vec<String>,
/// Role on the secret store's `cert` auth mount this responder logs in /// Role on the secret store's `cert` auth mount this responder logs in
/// as to read the callout keys, its own OIDC client secret and agent /// as to read the callout keys, its own OIDC client secret and agent
/// credentials. The store is found /// credentials. The store is found
@ -226,7 +231,8 @@ async fn serve(args: &Args) -> anyhow::Result<()> {
args.hive_publish_subjects.clone(), args.hive_publish_subjects.clone(),
)? )?
.with_agent_token_subjects(args.agent_token_publish_subjects.clone())? .with_agent_token_subjects(args.agent_token_publish_subjects.clone())?
.with_review_classifiers(args.review_classifier_clients.clone()); .with_review_classifiers(args.review_classifier_clients.clone())
.with_builders(args.builder_clients.clone());
let http = reqwest::Client::new(); let http = reqwest::Client::new();
let settings = swarm_secret_client::client::Settings::from_env() let settings = swarm_secret_client::client::Settings::from_env()
.context("reading the secret store's address and this responder's identity")?; .context("reading the secret store's address and this responder's identity")?;

82
swarm-nats-auth/src/policy.rs
View file

@ -50,6 +50,7 @@ pub struct Policy {
extra_hive_subjects: Vec<String>, extra_hive_subjects: Vec<String>,
agent_token_subjects: Vec<String>, agent_token_subjects: Vec<String>,
review_classifiers: Vec<String>, review_classifiers: Vec<String>,
builders: Vec<String>,
} }
/// Placeholder replaced with the hive's own name in `extra_hive_subjects`. /// Placeholder replaced with the hive's own name in `extra_hive_subjects`.
@ -120,6 +121,7 @@ impl Policy {
extra_hive_subjects, extra_hive_subjects,
agent_token_subjects: Vec::new(), agent_token_subjects: Vec::new(),
review_classifiers: Vec::new(), review_classifiers: Vec::new(),
builders: Vec::new(),
}) })
} }
@ -148,6 +150,13 @@ impl Policy {
self self
} }
/// Set the client ids that answer the swarm builder's build requests.
#[must_use]
pub fn with_builders(mut self, clients: Vec<String>) -> Self {
self.builders = clients;
self
}
/// The permissions for an agent whose own credential was verified, or /// The permissions for an agent whose own credential was verified, or
/// `None` when none are configured. /// `None` when none are configured.
/// ///
@ -209,6 +218,9 @@ impl Policy {
if self.review_classifiers.iter().any(|c| c == client_id) { if self.review_classifiers.iter().any(|c| c == client_id) {
return Some(Self::review_classifier_permissions(client_id)); return Some(Self::review_classifier_permissions(client_id));
} }
if self.builders.iter().any(|c| c == client_id) {
return Some(Self::builder_permissions(client_id));
}
None None
} }
@ -229,6 +241,20 @@ impl Policy {
} }
} }
/// What the builder may do: hear build requests and answer each on its
/// reply subject, for as long as a build may take. Its publish list is
/// its own inbox, for [`Self::review_classifier_permissions`]' reason.
fn builder_permissions(client_id: &str) -> Permissions {
Permissions {
publish: vec![Self::inbox(client_id)],
subscribe: Some(vec![
swarm_queue_client::builder::SUBJECT.to_owned(),
Self::inbox(client_id),
]),
responses: Some(swarm_queue_client::builder::REPLY_DEADLINE),
}
}
/// What a hive may subscribe to: the two events the controller sends it, /// What a hive may subscribe to: the two events the controller sends it,
/// and its own reply inbox. /// and its own reply inbox.
/// ///
@ -562,6 +588,8 @@ impl Policy {
// A review classifier try-out, sent as a request: the reply comes // A review classifier try-out, sent as a request: the reply comes
// back on this client's inbox, which it already subscribes to. // back on this client's inbox, which it already subscribes to.
swarm_queue_client::review_classifier::SUBJECT.to_owned(), swarm_queue_client::review_classifier::SUBJECT.to_owned(),
// A build of an agent config, sent as a request the same way.
swarm_queue_client::builder::SUBJECT.to_owned(),
// The wanted-state buckets — one per hive, all created and written // The wanted-state buckets — one per hive, all created and written
// by this single client. // by this single client.
// //
@ -1261,6 +1289,60 @@ mod tests {
} }
} }
const BUILDER: &str = "swarm-builder";
#[test]
fn the_builder_hears_its_subject_and_inbox_only_and_answers_by_reply() {
use swarm_queue_client::builder::{REPLY_DEADLINE, SUBJECT};
use swarm_queue_client::client_inbox_prefix;
let p = policy()
.with_builders(vec![BUILDER.to_owned()])
.permissions(BUILDER)
.expect("a configured builder is admitted");
let inbox = client_inbox_prefix(BUILDER);
assert!(may_subscribe(&p, SUBJECT));
assert!(may_subscribe(&p, &reply(&inbox)));
for refused in [
">".to_owned(),
"$SWARM.>".to_owned(),
swarm_queue_client::review_classifier::SUBJECT.to_owned(),
reply(&client_inbox_prefix("swarm-controller")),
] {
assert!(
!may_subscribe(&p, &refused),
"the builder may subscribe to {refused}"
);
}
assert_eq!(p.publish, vec![format!("{inbox}.>")]);
assert_eq!(p.responses, Some(REPLY_DEADLINE));
// The builder's id is nothing until it is configured.
assert_eq!(policy().permissions(BUILDER), None);
}
#[test]
fn only_the_reader_may_send_a_build() {
use swarm_queue_client::builder::SUBJECT;
let p = with_classifier().with_builders(vec![BUILDER.to_owned()]);
let reader = p.permissions("swarm-controller").expect("the reader");
assert!(reader.publish.iter().any(|g| covers(g, SUBJECT)));
for (who, grant) in [
("a hive", p.permissions("hive-alpha").expect("a hive")),
("the builder", p.permissions(BUILDER).expect("admitted")),
(
"the classifier",
p.permissions(CLASSIFIER).expect("admitted"),
),
] {
assert!(
!grant.publish.iter().any(|g| covers(g, SUBJECT)),
"{who} may send a build: {:?}",
grant.publish
);
}
}
#[test] #[test]
fn the_prefix_alone_names_no_hive() { fn the_prefix_alone_names_no_hive() {
// `hive-` would otherwise produce `$KV.hive-status.`, a subject nobody // `hive-` would otherwise produce `$KV.hive-status.`, a subject nobody

75
swarm-queue-client/src/builder.rs Normal file
View file

@ -0,0 +1,75 @@
//! The swarm builder's builds: the subject swarm-controller sends a build on,
//! and what travels each way.
//!
//! Three crates name these: the controller sends the request, the builder
//! answers it, and the auth-callout responder grants the one subject to each.
use std::time::Duration;
use serde::{Deserialize, Serialize};
/// The subject every build goes to. The controller is its one publisher and
/// the builder its one subscriber; the builder answers on the request's reply
/// subject.
pub const SUBJECT: &str = "$SWARM.builder.build";
/// The queue group the builder subscribes in, so a second instance shares the
/// requests instead of running each one twice.
pub const QUEUE_GROUP: &str = "builder";
/// The longest one build may take before the builder stops it and answers
/// with a timeout.
pub const BUILD_DEADLINE: Duration = Duration::from_hours(1);
/// How long the controller waits for the reply, and the queue lets the
/// builder answer: the build at its deadline, plus a minute for the rest.
pub const REPLY_DEADLINE: Duration = Duration::from_mins(61);
/// One build: an agent's config repo at one commit, on the builder's own
/// agent base.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct BuildAgent {
/// The agent, which is also its config repo's name.
pub agent: String,
/// The commit to build.
pub rev: String,
/// The PR the commit is the head of; its `refs/pull/<pr>/head` is the ref
/// the builder fetches the commit through.
pub pr: u64,
}
/// The builder's answer to one [`BuildAgent`].
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "state", rename_all = "kebab-case")]
pub enum Reply {
/// The config built; `out_path` is the system closure's store path.
Built { out_path: String },
/// The build did not finish: the last lines `nix` printed, or why the
/// builder could not start it.
Failed { error_tail: String },
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_wire_tags_the_reply_by_state() {
let json = serde_json::to_value(Reply::Built {
out_path: "/nix/store/x".to_owned(),
})
.expect("encodes");
assert_eq!(json["state"], "built");
assert_eq!(json["out_path"], "/nix/store/x");
let json = serde_json::to_value(Reply::Failed {
error_tail: "e".to_owned(),
})
.expect("encodes");
assert_eq!(json["state"], "failed");
}
#[test]
fn the_reply_deadline_outlasts_the_build() {
assert!(REPLY_DEADLINE > BUILD_DEADLINE);
}
}

3
swarm-queue-client/src/lib.rs
View file

@ -404,6 +404,9 @@ pub mod review_classifier;
pub mod stored_secret; pub mod stored_secret;
/// The builder's build subject and the request and reply it carries.
pub mod builder;
/// Only the fields this needs; authelia returns several. /// Only the fields this needs; authelia returns several.
#[derive(serde::Deserialize)] #[derive(serde::Deserialize)]
struct TokenResponse { struct TokenResponse {

64
swarm-secret-client/src/builder.rs Normal file
View file

@ -0,0 +1,64 @@
//! The swarm builder agreement: where its forge token and its queue client
//! secret live in the store, and what each object holds.
//!
//! swarm-controller mints the forge token; the builder reads it, and its queue
//! client secret, under its own certificate. Its read stanza in
//! `swarm-bao.nix` names exactly these two paths.
use serde::{Deserialize, Serialize};
use crate::{
Error,
path::{Kind, principal_prefix},
};
/// The builder's name segment under [`Kind::Service`].
const SERVICE: &str = "builder";
/// The path holding the token of the builder's read-only forge account, as a
/// [`crate::forge::Credential`].
///
/// # Errors
/// Never in practice: the name segment is a constant. The `Result` is
/// [`principal_prefix`]'s.
pub fn forge_token_path() -> Result<String, Error> {
let prefix = principal_prefix(Kind::Service, SERVICE)?;
Ok(format!("{prefix}/forge/token"))
}
/// The path holding the queue client secret of the builder's OIDC client
/// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's.
///
/// # Errors
/// [`Error::PathSegment`] when `client_id` is not a single path segment.
pub fn client_secret_path(client_id: &str) -> Result<String, Error> {
let prefix = principal_prefix(Kind::Service, client_id)?;
Ok(format!("{prefix}/oidc/client"))
}
/// What [`client_secret_path`] holds.
///
/// No `Debug`: `value` authenticates the builder at the queue.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ClientSecret {
/// The secret, under the field name the publisher writes.
pub value: String,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_paths_are_the_ones_the_store_grants() {
// Spelled out: `swarm-bao.nix` grants each literal.
assert_eq!(
forge_token_path().expect("a constant segment is legal"),
"swarm/services/builder/forge/token"
);
assert_eq!(
client_secret_path("swarm-builder").expect("a plain id is legal"),
"swarm/services/swarm-builder/oidc/client"
);
}
}

3
swarm-secret-client/src/lib.rs
View file

@ -6,7 +6,7 @@
//! deployment's environment into a logged-in client ([`client`]), and, per kind //! deployment's environment into a logged-in client ([`client`]), and, per kind
//! of secret, the path it lives at together with the fields it holds //! of secret, the path it lives at together with the fields it holds
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`], [`ci`], //! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`], [`ci`],
//! [`review_classifier`]). Each of those is a thing the controller //! [`review_classifier`], [`builder`]). Each of those is a thing the controller
//! and a hive must say identically, so it is said once here. //! and a hive must say identically, so it is said once here.
//! //!
//! [`policy`] is the same kind of agreement seen from the other side: which of //! [`policy`] is the same kind of agreement seen from the other side: which of
@ -24,6 +24,7 @@
//! value inside it, and its doc explains why that is not circular. //! value inside it, and its doc explains why that is not circular.
pub mod acp; pub mod acp;
pub mod builder;
pub mod ci; pub mod ci;
pub mod client; pub mod client;
pub mod forge; pub mod forge;