swarm-builder: build each agent config PR head and post a commit status
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
This commit is contained in:
atlas
parent
aece1c26d0
commit
b68b74d687
43 changed files with 2452 additions and 42 deletions
|
|
@ -202,6 +202,13 @@ hand-maintained per-file tree drifts out of sync with the code.
|
|||
for agents: nginx sends it `/select/logsql/` requests carrying an agent
|
||||
token, it answers 200 or 401 via `swarm-agent-verify`. Runs on the log
|
||||
store's host (`swarm-victorialogs.nix`).
|
||||
- **`swarm-builder/`** — the swarm builder's build daemon, in its own
|
||||
container where `deploy.builder.enable` is set. Takes an agent config PR's
|
||||
head from `swarm-controller` over the queue, builds it on its host's agent
|
||||
base through the host nix daemon (`nix/builder/check-flake`), and replies
|
||||
with the store path or the error tail; the controller writes the commit
|
||||
status. Holds a read-only forge token. Design:
|
||||
[`docs/swarm/builder.md`](docs/swarm/builder.md).
|
||||
- **`swarmctl/`** — swarm-level operator CLI, installed by the
|
||||
swarm-controller module on the host that runs the daemon. Runs as
|
||||
**root and acts directly** — it serves no socket, publishes no HTTP
|
||||
|
|
|
|||
15
Cargo.lock
generated
15
Cargo.lock
generated
|
|
@ -4829,6 +4829,21 @@ dependencies = [
|
|||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "swarm-builder"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-nats",
|
||||
"futures-util",
|
||||
"hive-log",
|
||||
"serde_json",
|
||||
"swarm-queue-client",
|
||||
"swarm-secret-client",
|
||||
"tokio",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "swarm-controller"
|
||||
version = "0.1.0"
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ members = [
|
|||
"hivectl",
|
||||
"swarm-agent-verify",
|
||||
"swarm-authelia-bridge",
|
||||
"swarm-builder",
|
||||
"swarm-authelia-bridge-sock",
|
||||
"swarm-controller",
|
||||
"swarm-matrix-client",
|
||||
|
|
|
|||
|
|
@ -89,6 +89,8 @@ declarations.
|
|||
- **How will a PR get checked against its repo's own yes/no rules before
|
||||
a full review?** → [`swarm/review-classifier.md`](swarm/review-classifier.md)
|
||||
(planned, not implemented yet).
|
||||
- **Does an agent config PR build before it's merged?** →
|
||||
[`swarm/builder.md`](swarm/builder.md).
|
||||
|
||||
## Scheduler, CI, observability
|
||||
|
||||
|
|
|
|||
81
docs/swarm/builder.md
Normal file
81
docs/swarm/builder.md
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# Swarm builder
|
||||
|
||||
The swarm builder checks that an agent config PR builds before an operator
|
||||
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
|
||||
push, the builder builds the PR's head on the agent base, and the result shows as a
|
||||
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
||||
swarm UI's Jobs page.
|
||||
|
||||
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
||||
|
||||
## What happens on a config PR
|
||||
|
||||
1. The forge delivers the `pull_request` event on the `config-pr` webhook
|
||||
swarm-controller already registers on `agent-configs`. An `opened`,
|
||||
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
||||
node carrying the agent, the head commit and the PR number. An edit, a
|
||||
label or a close builds nothing.
|
||||
2. The node sets the status to `pending`, then sends the build as a request on
|
||||
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
||||
reply.
|
||||
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
||||
with every input overridden:
|
||||
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
||||
deployed with, the same two a hive on that host renders into its meta
|
||||
flake (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`).
|
||||
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
|
||||
A build has one hour.
|
||||
4. The reply is the system closure's store path, or the last lines `nix`
|
||||
printed. The node writes the status on the head commit:
|
||||
|
||||
| Outcome | Status | Description | Node |
|
||||
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
||||
| built | `success` | the store path | done |
|
||||
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
||||
| no builder listening, or no reply in time | `error` | why | failed |
|
||||
|
||||
The full tail is in the failed node's error on the Jobs page and in the
|
||||
builder's journal line for the build.
|
||||
|
||||
## What the check is, and isn't
|
||||
|
||||
- It composes the config the way a hive's meta flake does
|
||||
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
|
||||
what a hive adds: its CA bundle, docs source, `claude-code` path, ports and
|
||||
OTEL settings, and the `follows` that point the config's own inputs at the
|
||||
hive's. A green check can therefore still fail on a hive, and the reverse.
|
||||
- The agent base is the builder's host's, not the hive's. Hives on another
|
||||
hyperhive or nixpkgs pin build something else.
|
||||
- The builder pushes nothing and records nothing. The closure stays in the builder
|
||||
host's store until it's collected, so on a host that also runs the hive, the
|
||||
hive's own build can reuse whatever matches.
|
||||
- It doesn't gate anything. A merge deploys whatever the status says.
|
||||
- A lost webhook delivery isn't retried: the head isn't built until the next
|
||||
push. The jobs graph is in memory, so a controller restart drops a build in
|
||||
flight and leaves its status `pending`.
|
||||
|
||||
## Credentials
|
||||
|
||||
| What | Where | Who reads it |
|
||||
| ----------------------- | ------------------------------------------------- | ---------------------------------------- |
|
||||
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
||||
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start |
|
||||
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
||||
|
||||
The token belongs to the `builder` forge account, which swarm-controller
|
||||
creates, puts in the `hives` team of `agent-configs` (read on every config
|
||||
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
|
||||
gets the token through a credential helper reading the child's environment:
|
||||
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
||||
status. The builder holds no write credential; swarm-controller writes the
|
||||
status with its own forge token. Details per path:
|
||||
[`credentials.md`](credentials.md).
|
||||
|
||||
## Isolation
|
||||
|
||||
The container has its own network namespace on the bridge, as the CI runner
|
||||
does, so a config's evaluation can't reach a host-loopback service. Evaluation
|
||||
is pure and runs in the container. Builds go to the host's nix daemon as an
|
||||
untrusted client: a config can set no daemon option, and its derivations build
|
||||
in the host's sandbox. A fixed-output derivation still has the network, as it
|
||||
does for any build on that host.
|
||||
|
|
@ -55,28 +55,29 @@ of the cell says how.
|
|||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
|
||||
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
|
||||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
|
||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
|
||||
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
|
||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
||||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
||||
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
|
||||
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
|
||||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
|
||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
|
||||
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
|
||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
||||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
||||
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
||||
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
|
||||
# Swarm secrets: what exists, and where each one lives
|
||||
|
||||
A swarm generates its credentials in three different places and reads them
|
||||
|
|
@ -400,9 +402,10 @@ next controller restart.
|
|||
### Per-principal identities
|
||||
|
||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||
identity and sharing one merges the identities. Each of these seven leaves has
|
||||
identity and sharing one merges the identities. Each of these eight leaves has
|
||||
a role and a policy of its own, and reaches one path, except the review
|
||||
classifier's, the forge host's and Grafana's, which reach two each:
|
||||
classifier's, the builder's, the forge host's and Grafana's, which reach two
|
||||
each:
|
||||
|
||||
| unit | option pair under `deploy.bao.` | reads, or writes |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
|
@ -418,11 +421,11 @@ The first exists **per hive**, because the path it reads carries a hive name
|
|||
and every hive runs its own reader. Its subject is
|
||||
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
||||
composed spelling as a hive name, so nobody can name a hive into another hive's
|
||||
role. The other six reach a path that names a swarm service rather than a hive,
|
||||
role. The other seven reach a path that names a swarm service rather than a hive,
|
||||
so one role each is enough and their subjects are the flat
|
||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||
`deploy.bao.reviewClassifierCommonName` and
|
||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName` and
|
||||
`deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one
|
||||
leaf two units share: the forge's host holds one store identity for the token it
|
||||
publishes and the OIDC client secret it reads.
|
||||
|
|
|
|||
|
|
@ -241,6 +241,26 @@ On a host without the store, that leaf is the one file to copy. Registration,
|
|||
the container and the threat model are in
|
||||
[`../scheduler/ci.md`](../scheduler/ci.md).
|
||||
|
||||
### Builder
|
||||
|
||||
The swarm builder, in a `swarm-builder` container on the host that sets
|
||||
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
|
||||
has the builder's store identity, on the same terms as the review classifier.
|
||||
Set `deploy.builder.enable` directly to override either way.
|
||||
|
||||
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
|
||||
swarm-controller queues a `build_agent` job. The builder builds that config at
|
||||
the PR's head on the agent base its own host runs, through the
|
||||
host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
|
||||
commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
||||
status blocks nothing: the merge still deploys.
|
||||
|
||||
It needs neither the forge nor a hive on its host. It reaches the queue, the
|
||||
secret store and the forge by name, under its own `builder` leaf, and fetches
|
||||
the private config repo with the read-only token of its own `builder` forge
|
||||
account. On a host without the store, that leaf is the one file to copy, as
|
||||
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
|
||||
|
||||
### Review classifier
|
||||
|
||||
The review classifier's try-out service, in a `swarm-review-classifier`
|
||||
|
|
|
|||
|
|
@ -132,6 +132,10 @@ checkpoints**, not about sandboxing the agent from its own tools:
|
|||
[`config-changes.md`](../agent-lifecycle/config-changes.md)), and
|
||||
scheduled prompts are operator-only, set at swarm level
|
||||
(see [`schedules.md`](../swarm/schedules.md)).
|
||||
Before that merge, the swarm builder evaluates an unmerged config PR
|
||||
off-hive, on any host that sets `deploy.builder.enable`: it builds the
|
||||
PR's head with a read-only forge token, in its own netns, as an untrusted
|
||||
client of its host's nix daemon (see [`builder.md`](../swarm/builder.md)).
|
||||
|
||||
### Capability = accepted risk
|
||||
|
||||
|
|
|
|||
|
|
@ -166,6 +166,9 @@
|
|||
services.hyperhive.deploy.review-classifier.package =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-review-classifier;
|
||||
services.hyperhive.deploy.builder.package =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-builder;
|
||||
services.hyperhive.deploy.authelia.bridgePackage =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-authelia-bridge;
|
||||
|
|
|
|||
26
nix/builder/check-flake/flake.nix
Normal file
26
nix/builder/check-flake/flake.nix
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# The swarm builder's build check: one agent config on hyperhive's agent base,
|
||||
# composed the way a hive's meta flake composes it, minus the values a hive
|
||||
# adds (its CA, docs source, claude-code path and ports).
|
||||
#
|
||||
# Every input is replaced at build time with `--override-input`
|
||||
# (swarm-builder/src/nix_build.rs), so the URLs below are never fetched and
|
||||
# the flake has no lock.
|
||||
{
|
||||
description = "swarm-builder: an agent config built on agent-base";
|
||||
|
||||
inputs = {
|
||||
nixpkgs.url = "flake:nixpkgs";
|
||||
hyperhive.url = "flake:hyperhive";
|
||||
hyperhive.inputs.nixpkgs.follows = "nixpkgs";
|
||||
agent.url = "flake:agent";
|
||||
};
|
||||
|
||||
outputs =
|
||||
{ hyperhive, agent, ... }:
|
||||
{
|
||||
packages.x86_64-linux.check =
|
||||
(hyperhive.nixosConfigurations.agent-base.extendModules {
|
||||
modules = [ agent.nixosModules.default ];
|
||||
}).config.system.build.toplevel;
|
||||
};
|
||||
}
|
||||
|
|
@ -44,6 +44,10 @@ in
|
|||
inherit (pkgs) lib;
|
||||
shared = moduleEvalShared;
|
||||
};
|
||||
module-eval-builder = import ./module-eval/builder.nix {
|
||||
inherit pkgs self nixosSystem;
|
||||
inherit (pkgs) lib;
|
||||
};
|
||||
module-eval-nats-authelia = import ./module-eval/nats-authelia.nix {
|
||||
inherit pkgs self nixosSystem;
|
||||
inherit (pkgs) lib;
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@
|
|||
./glue-nats-oidc-client.nix
|
||||
./glue-otel-scrape-targets.nix
|
||||
./glue-queue-agent-credential.nix
|
||||
./glue-builder-oidc-client.nix
|
||||
./glue-review-classifier-oidc-client.nix
|
||||
./glue-secret-publisher-authelia.nix
|
||||
./glue-secret-publisher-bao-identity.nix
|
||||
|
|
@ -53,6 +54,7 @@
|
|||
./swarm-authelia.nix
|
||||
./swarm-bao-service.nix
|
||||
./swarm-bao.nix
|
||||
./swarm-builder.nix
|
||||
./swarm-ci-runner.nix
|
||||
./swarm-review-classifier.nix
|
||||
./swarm-secret-publisher.nix
|
||||
|
|
|
|||
|
|
@ -679,6 +679,24 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
builder.enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
example = true;
|
||||
description = ''
|
||||
Run the swarm builder in a `swarm-builder` container on this host. It
|
||||
builds each agent config PR's head on this host's agent base, through
|
||||
this host's nix daemon, and swarm-controller writes the result as a
|
||||
`swarm-builder` commit status on the PR.
|
||||
|
||||
On by default where
|
||||
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
||||
host has the builder's store identity (the pair
|
||||
{option}`services.hyperhive.deploy.bao.builderClientCertFile` and
|
||||
`builderClientKeyFile`, minted on the store's host).
|
||||
'';
|
||||
};
|
||||
|
||||
hive-controller.enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
|
|
|
|||
|
|
@ -115,6 +115,8 @@ in
|
|||
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
|
||||
reviewClassifierClientCertFile = lib.mkDefault "${pkiDir}/review-classifier.pem";
|
||||
reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem";
|
||||
builderClientCertFile = lib.mkDefault "${pkiDir}/builder.pem";
|
||||
builderClientKeyFile = lib.mkDefault "${pkiDir}/builder-key.pem";
|
||||
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
|
||||
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
|
||||
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
|
||||
|
|
@ -290,6 +292,13 @@ in
|
|||
[ -s ${pkiDir}/review-classifier.pem ] || ${signLeaf} ${pkiDir} review-classifier \
|
||||
${lib.escapeShellArg deployCfg.bao.reviewClassifierCommonName} "" clientAuth
|
||||
|
||||
# The swarm builder's, which reads its forge token and queue client
|
||||
# secret. Minted here for the matrix-ctl leaf's reason: the builder is a
|
||||
# swarm service, so on a builder host without the store this is the
|
||||
# file an operator copies.
|
||||
[ -s ${pkiDir}/builder.pem ] || ${signLeaf} ${pkiDir} builder \
|
||||
${lib.escapeShellArg deployCfg.bao.builderCommonName} "" clientAuth
|
||||
|
||||
# The forge's token publisher. Minted here for the matrix-ctl leaf's
|
||||
# reason: the forge is a swarm singleton, so on a forge host without the
|
||||
# store this is the file an operator copies.
|
||||
|
|
|
|||
34
nix/host-modules/glue-builder-oidc-client.nix
Normal file
34
nix/host-modules/glue-builder-oidc-client.nix
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# Glue: register the swarm builder as an OIDC client wherever authelia runs.
|
||||
#
|
||||
# ONE PAIRING PER FILE — swarm-builder ← authelia, and nothing else. Deleting
|
||||
# this leaves a builder whose token requests authelia answers with a 401, and
|
||||
# no minted secret for swarm-secret-publish to put in the store.
|
||||
#
|
||||
# ⚠️ Gated on authelia being HERE, and NOT on this host running the builder,
|
||||
# for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS
|
||||
# host's provider config. Registered whether or not any host enables the
|
||||
# builder: nothing in `swarm.*` records that, so a swarm without one carries
|
||||
# one unused client and one unused minted secret.
|
||||
{
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
deployCfg = hyperhiveCfg.deploy;
|
||||
clientId = hyperhiveCfg.swarm.builder.clientId;
|
||||
in
|
||||
{
|
||||
config = lib.mkIf deployCfg.authelia.enable {
|
||||
services.hyperhive.swarm.authelia.oidc.clients = [
|
||||
{
|
||||
id = clientId;
|
||||
description = "HyperHive swarm builder";
|
||||
# `client_credentials`: a daemon authenticating as itself.
|
||||
kind = "machine";
|
||||
audience = [ clientId ];
|
||||
}
|
||||
];
|
||||
};
|
||||
}
|
||||
|
|
@ -441,6 +441,10 @@ let
|
|||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
|
||||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
|
@ -492,6 +496,8 @@ let
|
|||
# The review classifier's forge token is denied too: the controller posts
|
||||
# the classifier's reviews with whatever token is there, so a write to it
|
||||
# picks the account those reviews come from.
|
||||
# So is the builder's: a write to it picks the account the builder fetches
|
||||
# config repos as.
|
||||
secretPublisherPolicyText = ''
|
||||
path "${credentialMountPath}/data/swarm/hives/*" {
|
||||
capabilities = ["create", "update"]
|
||||
|
|
@ -521,6 +527,10 @@ let
|
|||
capabilities = ["deny"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
|
||||
capabilities = ["deny"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${controllerClientLeaf}" {
|
||||
capabilities = ["create", "update"]
|
||||
}
|
||||
|
|
@ -588,6 +598,12 @@ let
|
|||
# the UI's OIDC login; the classifier reads it on every run.
|
||||
classifierApiKeyLeaf = "swarm/services/review-classifier/anthropic-api-key";
|
||||
|
||||
# The swarm builder's forge token, the nix half of
|
||||
# `swarm_secret_client::builder::forge_token_path`. The controller mints it
|
||||
# and reads it back before deciding whether to re-mint; the builder reads it
|
||||
# to fetch config repos.
|
||||
builderForgeTokenLeaf = "swarm/services/builder/forge/token";
|
||||
|
||||
# The UI's OIDC client secret. The publisher writes it; the granter reads it
|
||||
# into the `oidc` auth method's config, which is the only copy bao uses.
|
||||
baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client";
|
||||
|
|
@ -998,6 +1014,18 @@ let
|
|||
}
|
||||
];
|
||||
|
||||
# The swarm builder: its forge token, read once per build, and its own queue
|
||||
# client secret, read at start. Both by the daemon inside its container.
|
||||
builderReaders = [
|
||||
{
|
||||
name = "swarm-builder";
|
||||
cn = baoDeploy.builderCommonName;
|
||||
policyText =
|
||||
readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}"
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
||||
# The forge host's one store identity. `swarm-bao-forge-publisher` writes
|
||||
# the controller's forge token under it, the one writer of that path and
|
||||
# write-only: it copies a file in and never reads one back.
|
||||
|
|
@ -1848,6 +1876,19 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
builderCommonName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-builder";
|
||||
description = ''
|
||||
Subject the store's `swarm-builder` cert-auth role accepts: the
|
||||
identity the swarm builder presents from inside its container. Its
|
||||
grant is two paths, read only: `swarm/services/builder/forge/token`
|
||||
and its own queue client secret.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
'';
|
||||
};
|
||||
|
||||
forgePublisherCommonName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-forge-publisher";
|
||||
|
|
@ -2123,6 +2164,31 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
builderClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/builder.pem";
|
||||
description = ''
|
||||
Certificate the swarm builder presents to the store. Its subject must
|
||||
be {option}`services.hyperhive.deploy.bao.builderCommonName`.
|
||||
|
||||
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
|
||||
On a builder host without the store, it is a file the operator copies
|
||||
from the store's host. A path, never a value.
|
||||
'';
|
||||
};
|
||||
|
||||
builderClientKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/builder-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.bao.builderClientCertFile`.
|
||||
A path, never a value.
|
||||
'';
|
||||
};
|
||||
|
||||
forgePublisherClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
|
|
@ -3201,6 +3267,8 @@ in
|
|||
|
||||
systemd.services.swarm-bao-review-classifier-policy = readerPolicyUnit "write the review classifier's API-key-reader bao policy and cert-auth role" reviewClassifierReaders;
|
||||
|
||||
systemd.services.swarm-bao-builder-policy = readerPolicyUnit "write the swarm builder's forge-token-reader bao policy and cert-auth role" builderReaders;
|
||||
|
||||
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
|
||||
|
||||
# A FOURTH sibling, same shape and same reasons as the two above. This
|
||||
|
|
|
|||
231
nix/host-modules/swarm-builder.nix
Normal file
231
nix/host-modules/swarm-builder.nix
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
# The swarm builder, in a `swarm-builder` container on the host that sets
|
||||
# `deploy.builder.enable`, which `deploy.allSwarmServices` defaults on where
|
||||
# the host has the builder's store identity. It builds agent configs for
|
||||
# swarm-controller: the controller sends a config PR's head over the queue,
|
||||
# and the daemon builds that config on this host's agent base and replies.
|
||||
# docs/swarm/builder.md has the design.
|
||||
#
|
||||
# ⚠️ Its own network namespace on the bridge, as ./swarm-ci-runner.nix has: a
|
||||
# config PR is unreviewed input to a nix evaluation, so the container reaches
|
||||
# the swarm services by name and nothing on the host's loopback.
|
||||
#
|
||||
# Builds go through the HOST nix-daemon, as the CI runner's do. The daemon
|
||||
# runs as a dynamic user, an untrusted daemon client: a config can set no
|
||||
# daemon option, and its builds run in the host's sandbox.
|
||||
#
|
||||
# The container holds one credential, its own store leaf, which nspawn hands in
|
||||
# as a system credential with no copy on disk. The daemon reads everything else
|
||||
# from the store with it: its queue client secret at start and its forge token
|
||||
# on every build.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
cfg = hyperhiveCfg.deploy.builder;
|
||||
baoCfg = hyperhiveCfg.swarm.bao;
|
||||
baoDeploy = hyperhiveCfg.deploy.bao;
|
||||
natsCfg = hyperhiveCfg.swarm.nats;
|
||||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||||
forgeCfg = hyperhiveCfg.swarm.forge;
|
||||
gatewayCfg = hyperhiveCfg.gateway;
|
||||
networkCfg = hyperhiveCfg.network;
|
||||
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
|
||||
|
||||
machine = "swarm-builder";
|
||||
unit = "swarm-builder";
|
||||
clientId = hyperhiveCfg.swarm.builder.clientId;
|
||||
|
||||
# The daemon reaches authelia, the store and the queue by name over TLS, and
|
||||
# `git` reaches the forge; in self-signed mode their leaves chain to the hive
|
||||
# CA.
|
||||
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
|
||||
caBundleModule = caTrust.trustBundle {
|
||||
inherit pkgs;
|
||||
name = machine;
|
||||
consumers = [ unit ];
|
||||
};
|
||||
|
||||
haveClientIdentity =
|
||||
baoDeploy.builderClientCertFile != null && baoDeploy.builderClientKeyFile != null;
|
||||
|
||||
# Credential id → host source, as ./swarm-review-classifier.nix hands its
|
||||
# leaf in.
|
||||
storeFiles = lib.filterAttrs (_: source: source != null) {
|
||||
builder-bao-cert = baoDeploy.builderClientCertFile;
|
||||
builder-bao-key = baoDeploy.builderClientKeyFile;
|
||||
builder-bao-server-ca = baoDeploy.serverCaFile;
|
||||
};
|
||||
# nixos-container expands `extraFlags` unquoted, so a source path must be
|
||||
# one word.
|
||||
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
|
||||
credential = id: "/run/credentials/${unit}.service/${id}";
|
||||
|
||||
# The `nix` the host daemon runs, so client and daemon speak one protocol.
|
||||
nix = config.nix.package;
|
||||
|
||||
# Strings, not paths: both are already store paths, and the flake reference
|
||||
# the daemon passes must be the same one a hive's meta flake names.
|
||||
hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake;
|
||||
nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake;
|
||||
|
||||
privateNetwork = true;
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.swarm.builder.clientId = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
readOnly = true;
|
||||
default = "swarm-builder";
|
||||
description = ''
|
||||
The OAuth2 client id the swarm builder presents to the swarm queue.
|
||||
Read-only: the host running authelia registers it, the queue's
|
||||
auth-callout responder grants it the build subject, and the secret
|
||||
publisher carries its secret to the store, all from this one value.
|
||||
'';
|
||||
};
|
||||
|
||||
options.services.hyperhive.deploy.builder.package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
defaultText = lib.literalExpression "hyperhive.packages.\${system}.swarm-builder";
|
||||
description = "The swarm builder daemon package.";
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
assertions = [
|
||||
{
|
||||
assertion = haveClientIdentity;
|
||||
message = ''
|
||||
services.hyperhive.deploy.builder.enable requires the builder's
|
||||
swarm-secret-store client identity: set both
|
||||
|
||||
services.hyperhive.deploy.bao.builderClientCertFile
|
||||
services.hyperhive.deploy.bao.builderClientKeyFile
|
||||
|
||||
The builder reads its queue client secret and its forge token with it.
|
||||
|
||||
Both default to the pair glue-bao-tls.nix mints when
|
||||
services.hyperhive.deploy.bao.enable is true (the host that runs the
|
||||
store). Otherwise copy builder.pem and builder-key.pem from the store
|
||||
host's /var/lib/swarm-bao-pki/ and name them in those two options.
|
||||
'';
|
||||
}
|
||||
{
|
||||
assertion = lib.all oneWord (lib.attrValues storeFiles);
|
||||
message = ''
|
||||
The swarm builder's store identity paths must not contain whitespace:
|
||||
nspawn's --load-credential receives them through nixos-container's
|
||||
unquoted extra flags.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
# The container hangs off the bridge, takes its address from the gateway's
|
||||
# DHCP and resolves through its dnsmasq.
|
||||
services.hyperhive.network.enable = lib.mkDefault true;
|
||||
services.hyperhive.gateway.enable = lib.mkDefault true;
|
||||
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
|
||||
|
||||
# nspawn fails to start the container when a credential source is
|
||||
# missing, so where this host mints the leaf the container waits for it.
|
||||
systemd.services."container@${machine}" = lib.mkMerge [
|
||||
caTrust.containerOrdering
|
||||
{ after = [ "swarm-bao-pki.service" ]; }
|
||||
];
|
||||
|
||||
containers.${machine} = {
|
||||
autoStart = true;
|
||||
# The daemon keeps nothing but `git`'s fetch cache, which a restart may
|
||||
# drop.
|
||||
ephemeral = false;
|
||||
# Journal files on the host, not inside the container: nixpkgs hardcodes
|
||||
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
|
||||
extraFlags = [
|
||||
"--link-journal=host"
|
||||
]
|
||||
++ lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
|
||||
inherit privateNetwork;
|
||||
hostBridge = networkCfg.bridgeName;
|
||||
|
||||
bindMounts = {
|
||||
# The directory rather than the socket file, so the mount survives a
|
||||
# daemon restart replacing the inode.
|
||||
"/nix/var/nix/daemon-socket" = {
|
||||
hostPath = "/nix/var/nix/daemon-socket";
|
||||
isReadOnly = false;
|
||||
};
|
||||
}
|
||||
// caTrust.bindMount;
|
||||
|
||||
config =
|
||||
{ lib, ... }:
|
||||
{
|
||||
imports = [
|
||||
../container-modules/swarm-container.nix
|
||||
caBundleModule
|
||||
];
|
||||
|
||||
services.hyperhive.swarmContainer = {
|
||||
inherit privateNetwork;
|
||||
writesOwnResolvConf = false;
|
||||
};
|
||||
networking.nameservers = [ networkCfg.bridgeIp ];
|
||||
networking.interfaces.eth0.useDHCP = true;
|
||||
|
||||
# Both the service and its socket are off: the socket unit owns
|
||||
# `/nix/var/nix/daemon-socket/socket` and would race the bind mount.
|
||||
systemd.globalEnvironment.NIX_REMOTE = "daemon";
|
||||
systemd.services.nix-daemon.enable = false;
|
||||
systemd.sockets.nix-daemon.enable = false;
|
||||
|
||||
systemd.services.${unit} = {
|
||||
description = "build agent configs for swarm-controller";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
# nix fetches the config repo with `git`.
|
||||
path = [ pkgs.git ];
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
BAO_CLIENT_CERT = credential "builder-bao-cert";
|
||||
BAO_CLIENT_KEY = credential "builder-bao-key";
|
||||
NIX_CONFIG = "experimental-features = nix-command flakes";
|
||||
# nix's fetch cache, under `CacheDirectory`; the dynamic user has
|
||||
# no home of its own.
|
||||
HOME = "/var/cache/${unit}";
|
||||
XDG_CACHE_HOME = "/var/cache/${unit}";
|
||||
SWARM_BUILDER_NATS_URL = "tls://${natsCfg.domain}:${toString natsCfg.port}";
|
||||
SWARM_BUILDER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
|
||||
SWARM_BUILDER_OIDC_CLIENT_ID = clientId;
|
||||
SWARM_BUILDER_NIX = "${nix}/bin/nix";
|
||||
SWARM_BUILDER_CHECK_FLAKE = "${../builder/check-flake}";
|
||||
SWARM_BUILDER_HYPERHIVE_FLAKE = hyperhiveFlake;
|
||||
SWARM_BUILDER_NIXPKGS_FLAKE = nixpkgsFlake;
|
||||
SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}";
|
||||
SWARM_BUILDER_CONFIG_ORG = "agent-configs";
|
||||
}
|
||||
// lib.optionalAttrs (storeFiles ? builder-bao-server-ca) {
|
||||
BAO_CACERT = credential "builder-bao-server-ca";
|
||||
}
|
||||
// lib.optionalAttrs caTrust.useSelfSigned {
|
||||
SWARM_BUILDER_OIDC_CA_FILE = caTrust.bundlePathFor machine;
|
||||
# `git`'s trust store, which the bundle's system CAs keep whole.
|
||||
GIT_SSL_CAINFO = caTrust.bundlePathFor machine;
|
||||
};
|
||||
serviceConfig = {
|
||||
ExecStart = "${cfg.package}/bin/swarm-builder";
|
||||
DynamicUser = true;
|
||||
CacheDirectory = unit;
|
||||
# Bare ids: the system credentials nspawn handed in.
|
||||
LoadCredential = lib.attrNames storeFiles;
|
||||
Restart = "on-failure";
|
||||
RestartSec = "10s";
|
||||
SyslogIdentifier = unit;
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
@ -683,6 +683,8 @@ in
|
|||
"--reader-client ${lib.escapeShellArg controllerCfg.queueClientId}"
|
||||
# Hears try-outs and answers each by reply, and nothing else.
|
||||
"--review-classifier-client ${lib.escapeShellArg config.services.hyperhive.swarm.review-classifier.clientId}"
|
||||
# Hears builds and answers each by reply, and nothing else.
|
||||
"--builder-client ${lib.escapeShellArg config.services.hyperhive.swarm.builder.clientId}"
|
||||
# What an agent may publish to, keyed on its own name: its
|
||||
# terminal rows and its turn-state header
|
||||
# (`hive-agent::swarm_agent_state`), two subject families so a
|
||||
|
|
|
|||
|
|
@ -117,4 +117,11 @@ in
|
|||
&& deployCfg.bao.reviewClassifierClientCertFile != null
|
||||
&& deployCfg.bao.reviewClassifierClientKeyFile != null
|
||||
);
|
||||
|
||||
# The swarm builder, defaulted on the same terms as the review classifier.
|
||||
config.services.hyperhive.deploy.builder.enable = lib.mkDefault (
|
||||
deployCfg.allSwarmServices
|
||||
&& deployCfg.bao.builderClientCertFile != null
|
||||
&& deployCfg.bao.builderClientKeyFile != null
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -94,6 +94,8 @@ let
|
|||
hyperhiveCfg.swarm.forge.sso.clientId
|
||||
# The review classifier, which reads it itself inside its container.
|
||||
hyperhiveCfg.swarm.review-classifier.clientId
|
||||
# The swarm builder, which reads it itself inside its container.
|
||||
hyperhiveCfg.swarm.builder.clientId
|
||||
];
|
||||
|
||||
# The swarm controller's OIDC client, which it reads back from the store
|
||||
|
|
|
|||
|
|
@ -57,6 +57,7 @@ let
|
|||
deployCfg.bao.ciRunnerCommonName
|
||||
deployCfg.bao.logsAuthCommonName
|
||||
deployCfg.bao.reviewClassifierCommonName
|
||||
deployCfg.bao.builderCommonName
|
||||
deployCfg.bao.forgePublisherCommonName
|
||||
]
|
||||
# The per-hive subjects, spelled out per hive rather than as the prefix.
|
||||
|
|
|
|||
|
|
@ -189,6 +189,7 @@ let
|
|||
"swarm-bao-logs-auth-policy"
|
||||
"swarm-bao-forge-publisher-policy"
|
||||
"swarm-bao-review-classifier-policy"
|
||||
"swarm-bao-builder-policy"
|
||||
];
|
||||
|
||||
# Comment lines dropped first: both the HCL and the scripts explain
|
||||
|
|
@ -1491,6 +1492,58 @@ let
|
|||
"swarm-bao-controller-policy"
|
||||
];
|
||||
}
|
||||
{
|
||||
# The builder reads its forge token and its own queue client secret,
|
||||
# each read-only and spelled to the leaf: `services/*` would reach every
|
||||
# swarm service's secret.
|
||||
name = "the swarm builder's grant is its forge token and its queue client secret, and nothing else";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-builder-policy.script;
|
||||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"read\"]\n}" s
|
||||
&& lib.hasInfix "path \"secret/data/swarm/services/swarm-builder/oidc/client\" {\n capabilities = [\"read\"]\n}" s
|
||||
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2
|
||||
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
|
||||
&& lib.hasInfix "auth/cert/certs/swarm-builder" s
|
||||
&& lib.hasInfix "allowed_common_names=swarm-builder" s
|
||||
&& lib.hasInfix "token_policies=swarm-builder" s;
|
||||
}
|
||||
{
|
||||
# `forge::builder_account` mints the token and reads it back to decide
|
||||
# on a re-mint. Pinned as the whole stanza, and spelled to the leaf.
|
||||
name = "the controller's grant on the swarm builder is its forge token alone";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-controller-policy.script;
|
||||
in
|
||||
lib.hasInfix "path \"secret/data/swarm/services/builder/forge/token\" {\n capabilities = [\"create\", \"read\", \"update\"]\n}" s
|
||||
&& !(lib.hasInfix "secret/data/swarm/services/builder/*" s);
|
||||
}
|
||||
{
|
||||
# The token sits under the publisher's `services/*` write, and the
|
||||
# builder fetches config repos with whatever is there. Its OIDC leaf is
|
||||
# the control: still the prefix's.
|
||||
name = "the publisher may not write the swarm builder's forge token";
|
||||
ok =
|
||||
let
|
||||
grants = grantsIn baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy.script;
|
||||
in
|
||||
(grantFor grants "secret/data/swarm/services/builder/forge/token").caps == [ "deny" ]
|
||||
&&
|
||||
(grantFor grants "secret/data/swarm/services/swarm-builder/oidc/client").caps == [
|
||||
"create"
|
||||
"update"
|
||||
];
|
||||
}
|
||||
{
|
||||
name = "the PKI unit signs the swarm builder's leaf under its own subject";
|
||||
ok =
|
||||
let
|
||||
s = baoGrantHere.systemd.services.swarm-bao-pki.script;
|
||||
in
|
||||
lib.hasInfix "/builder.pem ]" s && lib.hasInfix "swarm-builder \"\" clientAuth" s;
|
||||
}
|
||||
{
|
||||
name = "the PKI unit signs the CI runner's leaf under its own subject";
|
||||
ok =
|
||||
|
|
|
|||
131
nix/module-eval/builder.nix
Normal file
131
nix/module-eval/builder.nix
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# `checks.module-eval-builder` — see ./lib.nix for the shared rationale (why
|
||||
# this suite exists, naming convention, "evaluates not executes"). The swarm
|
||||
# builder's container, its default and its queue and IdP registrations; its
|
||||
# store grants are in ./bao-grants.nix beside every other principal's.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
self,
|
||||
nixosSystem,
|
||||
}:
|
||||
let
|
||||
inherit
|
||||
(import ./lib.nix {
|
||||
inherit
|
||||
pkgs
|
||||
lib
|
||||
self
|
||||
nixosSystem
|
||||
;
|
||||
})
|
||||
hive
|
||||
hiveWithController
|
||||
runGroup
|
||||
;
|
||||
|
||||
bare = hiveWithController { };
|
||||
|
||||
# On the store's host, which mints the builder's store identity.
|
||||
withBuilder = hive {
|
||||
deploy.bao.enable = true;
|
||||
deploy.builder.enable = true;
|
||||
};
|
||||
|
||||
servicesHere = hiveWithController { deploy.allSwarmServices = true; };
|
||||
|
||||
servicesBaoElsewhere = hive {
|
||||
deploy.allSwarmServices = true;
|
||||
deploy.bao.enable = false;
|
||||
};
|
||||
|
||||
natsHere = hive {
|
||||
swarm.nats.enable = true;
|
||||
swarm.nats.authPackage = pkgs.emptyDirectory;
|
||||
};
|
||||
|
||||
autheliaHere = hive {
|
||||
deploy.bao.enable = true;
|
||||
deploy.authelia.enable = true;
|
||||
};
|
||||
|
||||
container = withBuilder.containers.swarm-builder;
|
||||
inner = container.config;
|
||||
svc = inner.systemd.services.swarm-builder;
|
||||
env = svc.environment;
|
||||
|
||||
cases = [
|
||||
{
|
||||
name = "the swarm builder's container exists only where its host enables it";
|
||||
ok = !(bare.containers ? swarm-builder) && withBuilder.containers ? swarm-builder;
|
||||
}
|
||||
{
|
||||
# Its module asserts a store identity that only the store's host mints,
|
||||
# so with the store elsewhere the default stays off rather than failing
|
||||
# that host's eval. `nats` is the control that the switch is on at all.
|
||||
name = "the shared services turn the builder on with the store's leaf and leave it off without";
|
||||
ok =
|
||||
servicesHere.services.hyperhive.deploy.builder.enable
|
||||
&& servicesHere.containers ? swarm-builder
|
||||
&& !servicesBaoElsewhere.services.hyperhive.deploy.builder.enable
|
||||
&& !(servicesBaoElsewhere.containers ? swarm-builder)
|
||||
&& servicesBaoElsewhere.services.hyperhive.deploy.nats.enable;
|
||||
}
|
||||
{
|
||||
# A config PR is unreviewed input to an evaluation: the container keeps
|
||||
# its own netns, and reaches the host's nix daemon through its socket
|
||||
# alone, with no daemon of its own racing the bind.
|
||||
name = "the builder's container has its own netns and builds only through the host's daemon socket";
|
||||
ok =
|
||||
container.privateNetwork
|
||||
&& inner.services.hyperhive.swarmContainer.privateNetwork
|
||||
&& container.bindMounts."/nix/var/nix/daemon-socket".hostPath == "/nix/var/nix/daemon-socket"
|
||||
&& !inner.systemd.services.nix-daemon.enable
|
||||
&& !inner.systemd.sockets.nix-daemon.enable
|
||||
&& inner.systemd.globalEnvironment.NIX_REMOTE == "daemon";
|
||||
}
|
||||
{
|
||||
# The daemon holds its store leaf, handed in by nspawn without a copy on
|
||||
# disk; the forge token is read per build and reaches only `git`.
|
||||
name = "the builder loads only its store leaf, runs as a dynamic user and carries no token in its unit";
|
||||
ok =
|
||||
lib.elem "builder-bao-key" svc.serviceConfig.LoadCredential
|
||||
&& env.BAO_CLIENT_KEY == "/run/credentials/swarm-builder.service/builder-bao-key"
|
||||
&& env.SWARM_BUILDER_OIDC_CLIENT_ID == "swarm-builder"
|
||||
&& lib.elem "--load-credential=builder-bao-key:/var/lib/swarm-bao-pki/builder-key.pem" container.extraFlags
|
||||
&& svc.serviceConfig.DynamicUser
|
||||
&& !(lib.any (lib.hasSuffix "TOKEN") (lib.attrNames env));
|
||||
}
|
||||
{
|
||||
# The check builds on the sources a hive on this host would render into
|
||||
# its meta flake, so a green check means the same base.
|
||||
name = "the builder builds on the hyperhive and nixpkgs this host's hives use";
|
||||
ok =
|
||||
env.SWARM_BUILDER_HYPERHIVE_FLAKE == withBuilder.services.hyperhive.c0re.hyperhiveFlake
|
||||
&& env.SWARM_BUILDER_NIXPKGS_FLAKE == withBuilder.services.hyperhive.c0re.nixpkgsFlake
|
||||
&& lib.hasSuffix "-check-flake" env.SWARM_BUILDER_CHECK_FLAKE
|
||||
&& env.SWARM_BUILDER_CONFIG_ORG == "agent-configs"
|
||||
&& env.SWARM_BUILDER_FORGE_URL == "https://${withBuilder.services.hyperhive.swarm.forge.domain}";
|
||||
}
|
||||
{
|
||||
# The responder denies a client id it does not know, and a NATS denial
|
||||
# arrives as a timeout.
|
||||
name = "the queue's responder grants the builder's client its subject";
|
||||
ok = lib.hasInfix "--builder-client swarm-builder" natsHere.containers.swarm-nats.config.systemd.services.swarm-nats-auth.serviceConfig.ExecStart;
|
||||
}
|
||||
{
|
||||
name = "the builder's client is registered as a machine, and the publisher carries its secret";
|
||||
ok =
|
||||
let
|
||||
c = lib.findFirst (
|
||||
c: c.id == "swarm-builder"
|
||||
) null autheliaHere.services.hyperhive.swarm.authelia.oidc.clients;
|
||||
in
|
||||
c != null
|
||||
&& c.kind == "machine"
|
||||
&& lib.hasInfix "secret/swarm/services/swarm-builder/oidc/client" (
|
||||
autheliaHere.systemd.services.swarm-secret-publish.script
|
||||
);
|
||||
}
|
||||
];
|
||||
in
|
||||
runGroup "builder" cases
|
||||
|
|
@ -472,7 +472,7 @@ let
|
|||
# that lost a container can't pass by comparing fewer of them.
|
||||
name = "every service container restates its host-side privateNetwork";
|
||||
ok =
|
||||
lib.length (lib.attrNames serviceContainers) == 11
|
||||
lib.length (lib.attrNames serviceContainers) == 12
|
||||
&& lib.all (
|
||||
c: c.privateNetwork == (c.config.services.hyperhive.swarmContainer.privateNetwork or null)
|
||||
) (lib.attrValues serviceContainers);
|
||||
|
|
|
|||
|
|
@ -177,6 +177,7 @@ in
|
|||
# The review classifier's daemon. Out of `daemonBins` for the responder's
|
||||
# reason: it runs inside its own container, not on the host.
|
||||
swarm-review-classifier = mkBinPackage "swarm-review-classifier" "hyperhive review classifier try-out daemon";
|
||||
swarm-builder = mkBinPackage "swarm-builder" "hyperhive swarm builder: agent config build checks";
|
||||
|
||||
# The only process allowed to write swarm-authelia's users database —
|
||||
# same "runs *inside* a container, not on the host" placement as
|
||||
|
|
|
|||
26
swarm-builder/Cargo.toml
Normal file
26
swarm-builder/Cargo.toml
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
[package]
|
||||
name = "swarm-builder"
|
||||
version.workspace = true
|
||||
readme = "README.md"
|
||||
edition.workspace = true
|
||||
|
||||
[[bin]]
|
||||
name = "swarm-builder"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
async-nats.workspace = true
|
||||
# StreamExt::next on the subscription.
|
||||
futures-util.workspace = true
|
||||
hive-log.workspace = true
|
||||
serde_json.workspace = true
|
||||
# The build subject and the shapes the controller sends and reads back.
|
||||
swarm-queue-client.workspace = true
|
||||
# The forge token and queue client secret paths, and the store login.
|
||||
swarm-secret-client.workspace = true
|
||||
tokio.workspace = true
|
||||
tracing.workspace = true
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
31
swarm-builder/README.md
Normal file
31
swarm-builder/README.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# swarm-builder
|
||||
|
||||
The swarm builder's build service, in the `swarm-builder` container on the
|
||||
host that sets `deploy.builder.enable`. The design is
|
||||
[`docs/swarm/builder.md`](../docs/swarm/builder.md).
|
||||
|
||||
swarm-controller sends one request per config PR head: an agent, a commit and
|
||||
the PR. This service builds that agent config on the agent base this host was
|
||||
deployed with, through the host's nix daemon, and answers with the system
|
||||
closure's store path or the tail of what `nix` printed. The controller turns
|
||||
the answer into a `swarm-builder` commit status on the PR's head.
|
||||
|
||||
## What a build is
|
||||
|
||||
- `nix build` of `nix/builder/check-flake`'s `check` output, with its three
|
||||
inputs overridden: `hyperhive` and `nixpkgs` to this host's own sources, and
|
||||
`agent` to `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<rev>`.
|
||||
- The evaluation runs here, pure; the builds run in the host daemon's sandbox
|
||||
as an untrusted client, so a config can neither set daemon options nor read
|
||||
this service's environment.
|
||||
- Nothing is pushed and nothing is recorded: the closure stays in the host's
|
||||
store until it is collected.
|
||||
|
||||
## Credentials
|
||||
|
||||
The container holds one credential: its own store certificate, which nspawn
|
||||
hands in at start without a copy on disk. With it the service reads its
|
||||
queue client secret at start and its read-only forge token at every build.
|
||||
`git` gets the token through a credential helper reading the child's
|
||||
environment, never in a URL or an argument. Its store role reads exactly
|
||||
those two paths.
|
||||
190
swarm-builder/src/main.rs
Normal file
190
swarm-builder/src/main.rs
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
//! The swarm builder's build service.
|
||||
//!
|
||||
//! Connects to the swarm queue as its own OIDC client, takes build requests
|
||||
//! from swarm-controller on [`swarm_queue_client::builder::SUBJECT`], builds
|
||||
//! each agent config at its commit, answers on the request's reply subject,
|
||||
//! and logs one line per build. It writes nothing to the forge: the
|
||||
//! controller turns the reply into a commit status.
|
||||
//!
|
||||
//! The store identity comes from the `BAO_*` environment. The queue client
|
||||
//! secret is read at start and again when authelia refuses it, the forge token
|
||||
//! once per build, both from the store under that identity.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result, bail};
|
||||
use futures_util::StreamExt as _;
|
||||
use swarm_queue_client::builder::{BUILD_DEADLINE, BuildAgent, QUEUE_GROUP, Reply, SUBJECT};
|
||||
use swarm_queue_client::stored_secret::StoredSecret;
|
||||
use swarm_queue_client::{ClientSecret, QueueConfig};
|
||||
use swarm_secret_client::forge::Credential;
|
||||
|
||||
mod nix_build;
|
||||
|
||||
/// The cert-auth role this service logs in to the store as. `swarm-bao.nix`
|
||||
/// writes it, matching the leaf's subject.
|
||||
const CERT_ROLE: &str = "swarm-builder";
|
||||
|
||||
/// The environment prefix `swarm-builder.nix` sets.
|
||||
const PREFIX: &str = "SWARM_BUILDER";
|
||||
|
||||
/// Seconds to wait after each failed read of the queue client secret. A
|
||||
/// longer outage fails the start, and the unit's `Restart=` tries again.
|
||||
const RETRY_DELAYS_S: [u64; 6] = [1, 2, 4, 8, 16, 30];
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
hive_log::init();
|
||||
swarm_queue_client::install_crypto_provider();
|
||||
let var = |name: &str| {
|
||||
let full = format!("{PREFIX}_{name}");
|
||||
std::env::var(&full).with_context(|| format!("{full} is unset"))
|
||||
};
|
||||
let client_id = var("OIDC_CLIENT_ID")?;
|
||||
let builder = Arc::new(nix_build::Builder {
|
||||
nix: PathBuf::from(var("NIX")?),
|
||||
check_flake: var("CHECK_FLAKE")?,
|
||||
hyperhive: var("HYPERHIVE_FLAKE")?,
|
||||
nixpkgs: var("NIXPKGS_FLAKE")?,
|
||||
forge_url: var("FORGE_URL")?.trim_end_matches('/').to_owned(),
|
||||
config_org: var("CONFIG_ORG")?,
|
||||
deadline: BUILD_DEADLINE,
|
||||
});
|
||||
let secret = client_secret_with_retry(&client_id).await?;
|
||||
let reread_id = client_id.clone();
|
||||
let secret = StoredSecret::new(client_id.clone(), secret, move || {
|
||||
let client_id = reread_id.clone();
|
||||
async move { client_secret(&client_id).await.map_err(Into::into) }
|
||||
});
|
||||
let queue = swarm_queue_client::connect(QueueConfig {
|
||||
url: var("NATS_URL")?,
|
||||
token_endpoint: var("OIDC_TOKEN_ENDPOINT")?,
|
||||
client_id,
|
||||
client_secret: ClientSecret::Stored(secret),
|
||||
ca_file: std::env::var(format!("{PREFIX}_OIDC_CA_FILE"))
|
||||
.ok()
|
||||
.map(PathBuf::from),
|
||||
})
|
||||
.await?;
|
||||
let mut requests = queue
|
||||
.queue_subscribe(SUBJECT, QUEUE_GROUP.to_owned())
|
||||
.await
|
||||
.with_context(|| format!("subscribing to {SUBJECT}"))?;
|
||||
tracing::info!(subject = SUBJECT, "builder: awaiting builds");
|
||||
|
||||
while let Some(msg) = requests.next().await {
|
||||
let Some(reply_to) = msg.reply.clone() else {
|
||||
tracing::warn!("a build arrived with no reply subject; dropping it");
|
||||
continue;
|
||||
};
|
||||
let queue = queue.clone();
|
||||
let builder = Arc::clone(&builder);
|
||||
tokio::spawn(async move {
|
||||
let reply = handle(&msg.payload, &builder).await;
|
||||
match serde_json::to_vec(&reply) {
|
||||
Ok(body) => {
|
||||
if let Err(e) = queue.publish(reply_to, body.into()).await {
|
||||
tracing::warn!(error = %e, "publishing a build's reply failed");
|
||||
}
|
||||
}
|
||||
Err(e) => tracing::warn!(error = %e, "encoding a build's reply failed"),
|
||||
}
|
||||
});
|
||||
}
|
||||
bail!("the subscription to {SUBJECT} ended")
|
||||
}
|
||||
|
||||
/// Decode one request, build it and log the build.
|
||||
async fn handle(payload: &[u8], builder: &nix_build::Builder) -> Reply {
|
||||
let job: BuildAgent = match serde_json::from_slice(payload) {
|
||||
Ok(job) => job,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "an undecodable build; answering with the error");
|
||||
return Reply::Failed {
|
||||
error_tail: format!("the builder could not decode the request: {e}"),
|
||||
};
|
||||
}
|
||||
};
|
||||
let reply = match forge_token().await {
|
||||
Ok(token) => builder.build(&job, &token).await,
|
||||
Err(e) => Reply::Failed {
|
||||
error_tail: format!("{e:#}"),
|
||||
},
|
||||
};
|
||||
log_build(&job, &reply);
|
||||
reply
|
||||
}
|
||||
|
||||
/// The one journal line a build leaves.
|
||||
fn log_build(job: &BuildAgent, reply: &Reply) {
|
||||
match reply {
|
||||
Reply::Built { out_path } => tracing::info!(
|
||||
agent = %job.agent,
|
||||
rev = %job.rev,
|
||||
pr = job.pr,
|
||||
%out_path,
|
||||
"agent config built"
|
||||
),
|
||||
Reply::Failed { error_tail } => tracing::warn!(
|
||||
agent = %job.agent,
|
||||
rev = %job.rev,
|
||||
pr = job.pr,
|
||||
%error_tail,
|
||||
"agent config build failed"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The forge token the builder fetches config repos with, read on every
|
||||
/// build so a rotation takes effect on the next one.
|
||||
async fn forge_token() -> Result<String> {
|
||||
let path = swarm_secret_client::builder::forge_token_path()?;
|
||||
let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE)
|
||||
.await
|
||||
.context("logging in to the secret store")?;
|
||||
let stored: Option<Credential> = store
|
||||
.read_optional(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
match stored {
|
||||
Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()),
|
||||
_ => bail!("nothing at {path}: swarm-controller has not minted the builder's forge token"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The queue client secret, read from the store, retried through a store that
|
||||
/// is restarting alongside this service.
|
||||
async fn client_secret_with_retry(client_id: &str) -> Result<String> {
|
||||
for delay in RETRY_DELAYS_S.map(Duration::from_secs) {
|
||||
match client_secret(client_id).await {
|
||||
Ok(secret) => return Ok(secret),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), retry_in = ?delay, "reading the queue client secret failed");
|
||||
tokio::time::sleep(delay).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
client_secret(client_id).await
|
||||
}
|
||||
|
||||
async fn client_secret(client_id: &str) -> Result<String> {
|
||||
use swarm_secret_client::builder::{ClientSecret, client_secret_path};
|
||||
|
||||
let path = client_secret_path(client_id)?;
|
||||
let store = swarm_secret_client::SecretStore::from_env(CERT_ROLE)
|
||||
.await
|
||||
.context("logging in to the secret store")?;
|
||||
let stored: Option<ClientSecret> = store
|
||||
.read_optional(&path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
match stored {
|
||||
Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()),
|
||||
_ => bail!(
|
||||
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \
|
||||
published this service's client secret"
|
||||
),
|
||||
}
|
||||
}
|
||||
334
swarm-builder/src/nix_build.rs
Normal file
334
swarm-builder/src/nix_build.rs
Normal file
|
|
@ -0,0 +1,334 @@
|
|||
//! One build: `nix build` of the check flake with its three inputs pointed at
|
||||
//! this host's hyperhive and nixpkgs and at the agent's config commit.
|
||||
//!
|
||||
//! The config repo is private, so `git` fetches it with the builder's forge
|
||||
//! token. The token reaches `git` through a credential helper that reads it
|
||||
//! from the child's environment: it is in no argument, no URL and no lock, so
|
||||
//! no error line `nix` prints can carry it into the reply.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::process::Stdio;
|
||||
use std::time::Duration;
|
||||
|
||||
use swarm_queue_client::builder::{BuildAgent, Reply};
|
||||
use tokio::io::AsyncReadExt as _;
|
||||
|
||||
/// The most of `nix`'s stderr a failed build's reply carries, from its end.
|
||||
const TAIL_LINES: usize = 30;
|
||||
const TAIL_BYTES: usize = 4000;
|
||||
|
||||
/// The variable the credential helper reads the token from, in the child only.
|
||||
const TOKEN_VAR: &str = "SWARM_BUILDER_GIT_TOKEN";
|
||||
|
||||
/// What every build is run with, fixed at start.
|
||||
pub struct Builder {
|
||||
/// The `nix` binary.
|
||||
pub nix: PathBuf,
|
||||
/// The check flake's directory in the store.
|
||||
pub check_flake: String,
|
||||
/// The hyperhive source this host was deployed from.
|
||||
pub hyperhive: String,
|
||||
/// The nixpkgs source this host was deployed from.
|
||||
pub nixpkgs: String,
|
||||
/// `https://<forge>`, with no trailing slash.
|
||||
pub forge_url: String,
|
||||
/// The org the config repos are in.
|
||||
pub config_org: String,
|
||||
/// How long one build may run.
|
||||
pub deadline: Duration,
|
||||
}
|
||||
|
||||
impl Builder {
|
||||
/// Build `job` with `token` as the forge credential.
|
||||
pub async fn build(&self, job: &BuildAgent, token: &str) -> Reply {
|
||||
if let Err(error) = validate(job) {
|
||||
return Reply::Failed { error_tail: error };
|
||||
}
|
||||
let mut cmd = tokio::process::Command::new(&self.nix);
|
||||
cmd.args(self.args(job))
|
||||
.env(TOKEN_VAR, token)
|
||||
.env("GIT_TERMINAL_PROMPT", "0")
|
||||
.env("GIT_CONFIG_COUNT", "1")
|
||||
.env("GIT_CONFIG_KEY_0", "credential.helper")
|
||||
.env("GIT_CONFIG_VALUE_0", credential_helper())
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.kill_on_drop(true);
|
||||
run(cmd, self.deadline).await
|
||||
}
|
||||
|
||||
/// The `nix build` arguments for `job`.
|
||||
fn args(&self, job: &BuildAgent) -> Vec<String> {
|
||||
[
|
||||
"build",
|
||||
"--no-link",
|
||||
"--print-out-paths",
|
||||
"--no-write-lock-file",
|
||||
"--option",
|
||||
"accept-flake-config",
|
||||
"false",
|
||||
"--print-build-logs",
|
||||
]
|
||||
.into_iter()
|
||||
.map(str::to_owned)
|
||||
.chain([
|
||||
format!("{}#check", self.check_flake),
|
||||
"--override-input".to_owned(),
|
||||
"hyperhive".to_owned(),
|
||||
self.hyperhive.clone(),
|
||||
"--override-input".to_owned(),
|
||||
"nixpkgs".to_owned(),
|
||||
self.nixpkgs.clone(),
|
||||
"--override-input".to_owned(),
|
||||
"agent".to_owned(),
|
||||
self.agent_url(job),
|
||||
])
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The config commit as a flake reference, fetched through the PR's head
|
||||
/// ref.
|
||||
fn agent_url(&self, job: &BuildAgent) -> String {
|
||||
format!(
|
||||
"git+{}/{}/{}?ref=refs/pull/{}/head&rev={}",
|
||||
self.forge_url, self.config_org, job.agent, job.pr, job.rev
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// A `git` credential helper answering every `get` with the builder's login
|
||||
/// and the token from [`TOKEN_VAR`].
|
||||
fn credential_helper() -> String {
|
||||
format!(
|
||||
"!f() {{ test \"$1\" = get && printf 'username=builder\\npassword=%s\\n' \"${TOKEN_VAR}\"; }}; f"
|
||||
)
|
||||
}
|
||||
|
||||
/// Refuse a job whose fields would change the reference's meaning: an agent
|
||||
/// is one `[a-z0-9-]` segment, and a rev is a full commit id.
|
||||
fn validate(job: &BuildAgent) -> Result<(), String> {
|
||||
let agent_ok = !job.agent.is_empty()
|
||||
&& job
|
||||
.agent
|
||||
.bytes()
|
||||
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-');
|
||||
if !agent_ok {
|
||||
return Err(format!("{:?} is not an agent name", job.agent));
|
||||
}
|
||||
let rev_ok = job.rev.len() == 40
|
||||
&& job
|
||||
.rev
|
||||
.bytes()
|
||||
.all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b));
|
||||
if !rev_ok {
|
||||
return Err(format!("{:?} is not a full commit id", job.rev));
|
||||
}
|
||||
if job.pr == 0 {
|
||||
return Err("PR 0 does not exist".to_owned());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Run `cmd` to its end or `deadline`, and read the store path it printed or
|
||||
/// the tail of what it said.
|
||||
async fn run(mut cmd: tokio::process::Command, deadline: Duration) -> Reply {
|
||||
let mut child = match cmd.spawn() {
|
||||
Ok(child) => child,
|
||||
Err(e) => {
|
||||
return Reply::Failed {
|
||||
error_tail: format!("starting nix: {e}"),
|
||||
};
|
||||
}
|
||||
};
|
||||
let mut stdout = child.stdout.take().expect("piped");
|
||||
let mut stderr = child.stderr.take().expect("piped");
|
||||
let work = async {
|
||||
let (mut out, mut err) = (Vec::new(), Vec::new());
|
||||
let (o, e, status) = tokio::join!(
|
||||
stdout.read_to_end(&mut out),
|
||||
stderr.read_to_end(&mut err),
|
||||
child.wait()
|
||||
);
|
||||
o.and(e).map(|_| ())?;
|
||||
status.map(|s| (s, out, err))
|
||||
};
|
||||
match tokio::time::timeout(deadline, work).await {
|
||||
Err(_) => Reply::Failed {
|
||||
error_tail: format!(
|
||||
"the build did not finish within {} minutes",
|
||||
deadline.as_secs() / 60
|
||||
),
|
||||
},
|
||||
Ok(Err(e)) => Reply::Failed {
|
||||
error_tail: format!("waiting for nix: {e}"),
|
||||
},
|
||||
Ok(Ok((status, out, err))) => {
|
||||
let out_path = String::from_utf8_lossy(&out)
|
||||
.lines()
|
||||
.rfind(|l| !l.trim().is_empty())
|
||||
.map(|l| l.trim().to_owned());
|
||||
match out_path {
|
||||
Some(out_path) if status.success() => Reply::Built { out_path },
|
||||
_ => Reply::Failed {
|
||||
error_tail: tail(&String::from_utf8_lossy(&err), status),
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The last [`TAIL_LINES`] lines of `stderr`, at most [`TAIL_BYTES`] bytes,
|
||||
/// or the exit status when it said nothing.
|
||||
fn tail(stderr: &str, status: std::process::ExitStatus) -> String {
|
||||
let lines: Vec<&str> = stderr.lines().filter(|l| !l.trim().is_empty()).collect();
|
||||
if lines.is_empty() {
|
||||
return format!("nix exited with {status} and printed nothing");
|
||||
}
|
||||
let joined = lines[lines.len().saturating_sub(TAIL_LINES)..].join("\n");
|
||||
let mut start = joined.len().saturating_sub(TAIL_BYTES);
|
||||
while !joined.is_char_boundary(start) {
|
||||
start += 1;
|
||||
}
|
||||
joined[start..].to_owned()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use swarm_queue_client::builder::BUILD_DEADLINE;
|
||||
|
||||
fn job() -> BuildAgent {
|
||||
BuildAgent {
|
||||
agent: "atlas".to_owned(),
|
||||
rev: "0123456789abcdef0123456789abcdef01234567".to_owned(),
|
||||
pr: 7,
|
||||
}
|
||||
}
|
||||
|
||||
fn builder() -> Builder {
|
||||
Builder {
|
||||
check_flake: "/nix/store/c-check".to_owned(),
|
||||
hyperhive: "/nix/store/h-source".to_owned(),
|
||||
nixpkgs: "/nix/store/n-source".to_owned(),
|
||||
forge_url: "https://forge.example".to_owned(),
|
||||
config_org: "agent-configs".to_owned(),
|
||||
nix: PathBuf::from("nix"),
|
||||
deadline: BUILD_DEADLINE,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_agent_is_fetched_at_its_commit_through_the_prs_ref() {
|
||||
assert_eq!(
|
||||
builder().agent_url(&job()),
|
||||
"git+https://forge.example/agent-configs/atlas?ref=refs/pull/7/head\
|
||||
&rev=0123456789abcdef0123456789abcdef01234567"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_input_is_overridden_and_no_lock_is_written() {
|
||||
let args = builder().args(&job());
|
||||
let overridden: Vec<&str> = args
|
||||
.windows(2)
|
||||
.filter(|w| w[0] == "--override-input")
|
||||
.map(|w| w[1].as_str())
|
||||
.collect();
|
||||
assert_eq!(overridden, ["hyperhive", "nixpkgs", "agent"]);
|
||||
assert!(args.iter().any(|a| a == "/nix/store/c-check#check"));
|
||||
assert!(args.iter().any(|a| a == "--no-write-lock-file"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_token_is_in_no_argument() {
|
||||
let args = builder().args(&job()).join(" ");
|
||||
assert!(!args.contains("password"));
|
||||
assert!(credential_helper().contains(&format!("${TOKEN_VAR}")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_job_that_would_bend_the_reference_is_refused() {
|
||||
for bad in [
|
||||
BuildAgent {
|
||||
agent: "a?rev=x".to_owned(),
|
||||
..job()
|
||||
},
|
||||
BuildAgent {
|
||||
agent: String::new(),
|
||||
..job()
|
||||
},
|
||||
BuildAgent {
|
||||
rev: "main".to_owned(),
|
||||
..job()
|
||||
},
|
||||
BuildAgent { pr: 0, ..job() },
|
||||
] {
|
||||
assert!(validate(&bad).is_err(), "{bad:?}");
|
||||
}
|
||||
assert!(validate(&job()).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_tail_keeps_the_end_of_a_long_log() {
|
||||
use std::os::unix::process::ExitStatusExt as _;
|
||||
|
||||
let log = (0..100)
|
||||
.map(|i| format!("line {i}"))
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
let t = tail(&log, std::process::ExitStatus::from_raw(256));
|
||||
assert!(t.ends_with("line 99"));
|
||||
assert!(!t.contains("line 69\n"));
|
||||
assert!(t.starts_with("line 70"));
|
||||
let quiet = tail("", std::process::ExitStatus::from_raw(256));
|
||||
assert!(quiet.contains("printed nothing"));
|
||||
}
|
||||
|
||||
fn sh(script: &str, deadline: Duration) -> impl Future<Output = Reply> {
|
||||
let mut cmd = tokio::process::Command::new("sh");
|
||||
cmd.args(["-c", script])
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.kill_on_drop(true);
|
||||
run(cmd, deadline)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_success_replies_with_the_last_path_printed() {
|
||||
let reply = sh(
|
||||
"echo warming >&2; echo /nix/store/abc-nixos-system",
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(
|
||||
reply,
|
||||
Reply::Built {
|
||||
out_path: "/nix/store/abc-nixos-system".to_owned()
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_failure_replies_with_what_nix_said_last() {
|
||||
let reply = sh(
|
||||
"echo first >&2; echo 'error: attribute missing' >&2; exit 1",
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
let Reply::Failed { error_tail } = reply else {
|
||||
panic!("{reply:?}")
|
||||
};
|
||||
assert!(error_tail.ends_with("error: attribute missing"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_build_past_its_deadline_is_stopped() {
|
||||
let reply = sh("sleep 30", Duration::from_millis(200)).await;
|
||||
assert!(
|
||||
matches!(&reply, Reply::Failed { error_tail } if error_tail.contains("did not finish")),
|
||||
"{reply:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
304
swarm-controller/src/builder.rs
Normal file
304
swarm-controller/src/builder.rs
Normal file
|
|
@ -0,0 +1,304 @@
|
|||
//! The `BuildAgent` node: send a config PR's head to the swarm builder over the
|
||||
//! queue, and turn its answer into the [`CONTEXT`] commit status on that head.
|
||||
//!
|
||||
//! The status is the check's only record besides the node itself: nothing is
|
||||
//! pushed, and the closure the builder made stays in its host's store. A red
|
||||
//! status blocks nothing; the merge deploys as before.
|
||||
|
||||
use std::future::Future;
|
||||
|
||||
use forgejo_api::structs::CommitStatusState;
|
||||
use hive_jobq::scheduler::Outcome;
|
||||
use swarm_queue_client::builder::{BuildAgent, REPLY_DEADLINE, Reply, SUBJECT};
|
||||
|
||||
use crate::forge::{self, CONFIG_ORG};
|
||||
|
||||
/// The commit status context the check writes.
|
||||
pub const CONTEXT: &str = "swarm-builder";
|
||||
|
||||
/// Where a build is sent: the queue in production, a fake in the tests.
|
||||
/// `Err` is a build that never reached a builder or never came back.
|
||||
pub(crate) trait BuildSink {
|
||||
fn build(&self, request: Vec<u8>) -> impl Future<Output = Result<Reply, String>> + Send;
|
||||
}
|
||||
|
||||
impl BuildSink for async_nats::Client {
|
||||
async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
|
||||
use async_nats::RequestErrorKind;
|
||||
|
||||
let sent = async_nats::Request::new()
|
||||
.payload(request.into())
|
||||
.timeout(Some(REPLY_DEADLINE));
|
||||
match self.send_request(SUBJECT, sent).await {
|
||||
Ok(msg) => serde_json::from_slice(&msg.payload)
|
||||
.map_err(|e| format!("the builder's reply did not decode: {e}")),
|
||||
Err(e) => Err(match e.kind() {
|
||||
RequestErrorKind::NoResponders => "no swarm builder is listening: \
|
||||
deploy.builder.enable is set on no host, or its service is not connected \
|
||||
to the swarm queue"
|
||||
.to_owned(),
|
||||
RequestErrorKind::TimedOut => format!(
|
||||
"the swarm builder sent no answer within {} minutes",
|
||||
REPLY_DEADLINE.as_secs() / 60
|
||||
),
|
||||
_ => format!("sending the build to the swarm builder failed: {e}"),
|
||||
}),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Where a status is written: the forge in production, a fake in the tests.
|
||||
pub(crate) trait StatusSink {
|
||||
fn set(
|
||||
&self,
|
||||
agent: &str,
|
||||
rev: &str,
|
||||
state: CommitStatusState,
|
||||
description: &str,
|
||||
) -> impl Future<Output = anyhow::Result<()>> + Send;
|
||||
}
|
||||
|
||||
impl StatusSink for forge::Client {
|
||||
async fn set(
|
||||
&self,
|
||||
agent: &str,
|
||||
rev: &str,
|
||||
state: CommitStatusState,
|
||||
description: &str,
|
||||
) -> anyhow::Result<()> {
|
||||
forge::within_forge_deadline(
|
||||
"build_agent status",
|
||||
self.create_commit_status(CONFIG_ORG, agent, rev, CONTEXT, state, description),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|| Err(anyhow::anyhow!("no answer from the forge")))
|
||||
}
|
||||
}
|
||||
|
||||
/// Build `job` and write its status. The node fails when the build does, so
|
||||
/// the Jobs page shows red where the PR does.
|
||||
pub(crate) async fn run(
|
||||
queue: Option<&impl BuildSink>,
|
||||
statuses: Option<&impl StatusSink>,
|
||||
job: BuildAgent,
|
||||
) -> Outcome {
|
||||
let Some(statuses) = statuses else {
|
||||
return Outcome::Failed(
|
||||
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token not in the \
|
||||
swarm secret store), so no build status can be written"
|
||||
.to_owned(),
|
||||
);
|
||||
};
|
||||
let set = |state, description: String| {
|
||||
let job = &job;
|
||||
async move {
|
||||
statuses
|
||||
.set(&job.agent, &job.rev, state, &description)
|
||||
.await
|
||||
.map_err(|e| format!("{e:#}"))
|
||||
}
|
||||
};
|
||||
let Some(queue) = queue else {
|
||||
let error = "no swarm queue is configured on this host, so no build can be sent";
|
||||
return settle(set(CommitStatusState::Error, error.to_owned()).await, error);
|
||||
};
|
||||
if let Err(e) = set(
|
||||
CommitStatusState::Pending,
|
||||
"building on the swarm builder".to_owned(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::warn!(agent = %job.agent, rev = %job.rev, error = %e, "build_agent: the pending status was not written");
|
||||
}
|
||||
let request = match serde_json::to_vec(&job) {
|
||||
Ok(request) => request,
|
||||
Err(e) => return Outcome::Failed(format!("encoding the build: {e}")),
|
||||
};
|
||||
match queue.build(request).await {
|
||||
Ok(Reply::Built { out_path }) => {
|
||||
tracing::info!(agent = %job.agent, rev = %job.rev, %out_path, "build_agent: built");
|
||||
match set(CommitStatusState::Success, out_path).await {
|
||||
Ok(()) => Outcome::Done,
|
||||
Err(e) => Outcome::Failed(format!("built, but the status was not written: {e}")),
|
||||
}
|
||||
}
|
||||
Ok(Reply::Failed { error_tail }) => {
|
||||
let last = last_line(&error_tail);
|
||||
settle(
|
||||
set(CommitStatusState::Failure, last.to_owned()).await,
|
||||
&format!("the build failed: {error_tail}"),
|
||||
)
|
||||
}
|
||||
Err(error) => settle(set(CommitStatusState::Error, error.clone()).await, &error),
|
||||
}
|
||||
}
|
||||
|
||||
/// A failed node carrying `error`, and the status write's own failure when
|
||||
/// there was one.
|
||||
fn settle(written: Result<(), String>, error: &str) -> Outcome {
|
||||
match written {
|
||||
Ok(()) => Outcome::Failed(error.to_owned()),
|
||||
Err(e) => Outcome::Failed(format!("{error} (and the status was not written: {e})")),
|
||||
}
|
||||
}
|
||||
|
||||
/// The last non-blank line of `tail`, which for `nix` is the error itself.
|
||||
fn last_line(tail: &str) -> &str {
|
||||
tail.lines()
|
||||
.rev()
|
||||
.map(str::trim)
|
||||
.find(|l| !l.is_empty())
|
||||
.unwrap_or("the build failed")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::sync::Mutex;
|
||||
|
||||
use super::*;
|
||||
|
||||
struct FakeQueue(Result<Reply, String>);
|
||||
|
||||
impl BuildSink for FakeQueue {
|
||||
async fn build(&self, request: Vec<u8>) -> Result<Reply, String> {
|
||||
let sent: BuildAgent = serde_json::from_slice(&request).expect("a build request");
|
||||
assert_eq!(sent, job());
|
||||
self.0.clone()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct FakeStatuses(Mutex<Vec<(CommitStatusState, String)>>);
|
||||
|
||||
impl StatusSink for FakeStatuses {
|
||||
async fn set(
|
||||
&self,
|
||||
agent: &str,
|
||||
rev: &str,
|
||||
state: CommitStatusState,
|
||||
description: &str,
|
||||
) -> anyhow::Result<()> {
|
||||
assert_eq!((agent, rev), ("atlas", "abc"));
|
||||
self.0.lock().unwrap().push((state, description.to_owned()));
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn job() -> BuildAgent {
|
||||
BuildAgent {
|
||||
agent: "atlas".to_owned(),
|
||||
rev: "abc".to_owned(),
|
||||
pr: 3,
|
||||
}
|
||||
}
|
||||
|
||||
async fn statuses_after(queue: Option<FakeQueue>) -> (Outcome, Vec<CommitStatusState>) {
|
||||
let statuses = FakeStatuses::default();
|
||||
let outcome = run(queue.as_ref(), Some(&statuses), job()).await;
|
||||
let states = statuses.0.into_inner().unwrap();
|
||||
(outcome, states.into_iter().map(|(s, _)| s).collect())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_built_config_goes_pending_then_green() {
|
||||
let (outcome, states) = statuses_after(Some(FakeQueue(Ok(Reply::Built {
|
||||
out_path: "/nix/store/x".to_owned(),
|
||||
}))))
|
||||
.await;
|
||||
assert!(matches!(outcome, Outcome::Done));
|
||||
assert_eq!(
|
||||
states,
|
||||
[CommitStatusState::Pending, CommitStatusState::Success]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_failed_build_goes_red_with_its_last_line_and_fails_the_node() {
|
||||
let statuses = FakeStatuses::default();
|
||||
let queue = FakeQueue(Ok(Reply::Failed {
|
||||
error_tail: "evaluating\nerror: boom\n".to_owned(),
|
||||
}));
|
||||
let outcome = run(Some(&queue), Some(&statuses), job()).await;
|
||||
assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("error: boom")));
|
||||
let written = statuses.0.into_inner().unwrap();
|
||||
assert_eq!(
|
||||
written.last(),
|
||||
Some(&(CommitStatusState::Failure, "error: boom".to_owned()))
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_builder_is_an_error_status_not_a_failure() {
|
||||
let (outcome, states) =
|
||||
statuses_after(Some(FakeQueue(Err("no swarm builder".to_owned())))).await;
|
||||
assert!(matches!(outcome, Outcome::Failed(_)));
|
||||
assert_eq!(
|
||||
states,
|
||||
[CommitStatusState::Pending, CommitStatusState::Error]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_queue_is_an_error_status() {
|
||||
let (outcome, states) = statuses_after(None).await;
|
||||
assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no swarm queue")));
|
||||
assert_eq!(states, [CommitStatusState::Error]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_forge_fails_the_node_without_sending_the_build() {
|
||||
struct Unreachable;
|
||||
impl BuildSink for Unreachable {
|
||||
async fn build(&self, _: Vec<u8>) -> Result<Reply, String> {
|
||||
panic!("a build with nowhere to report must not be sent")
|
||||
}
|
||||
}
|
||||
let outcome = run(Some(&Unreachable), None::<&FakeStatuses>, job()).await;
|
||||
assert!(matches!(&outcome, Outcome::Failed(e) if e.contains("no forge access")));
|
||||
}
|
||||
|
||||
/// The queue half against a real server: no builder listening names the
|
||||
/// option, and a listener's reply comes back decoded.
|
||||
#[tokio::test]
|
||||
async fn a_build_reaches_a_listening_builder_and_fails_without_one() {
|
||||
use futures_util::StreamExt as _;
|
||||
|
||||
let nats = crate::test_nats::TestNats::start();
|
||||
let client = nats.client().await;
|
||||
let unanswered = client.build(b"{}".to_vec()).await;
|
||||
assert!(
|
||||
matches!(&unanswered, Err(e) if e.contains("deploy.builder.enable")),
|
||||
"{unanswered:?}"
|
||||
);
|
||||
|
||||
let builder = nats.client().await;
|
||||
let mut requests = builder
|
||||
.queue_subscribe(SUBJECT, swarm_queue_client::builder::QUEUE_GROUP.to_owned())
|
||||
.await
|
||||
.expect("subscribe");
|
||||
// A message the builder sends itself and receives back proves the
|
||||
// server processed the subscription before it.
|
||||
let mut probe = builder
|
||||
.subscribe("builder-test.probe")
|
||||
.await
|
||||
.expect("subscribe");
|
||||
builder
|
||||
.publish("builder-test.probe", "".into())
|
||||
.await
|
||||
.expect("publish");
|
||||
probe.next().await.expect("the probe comes back");
|
||||
let built = Reply::Built {
|
||||
out_path: "/nix/store/x".to_owned(),
|
||||
};
|
||||
let answer = built.clone();
|
||||
tokio::spawn(async move {
|
||||
let msg = requests.next().await.expect("a request");
|
||||
let reply = serde_json::to_vec(&answer).expect("encodes");
|
||||
builder
|
||||
.publish(msg.reply.expect("a reply subject"), reply.into())
|
||||
.await
|
||||
.expect("publish");
|
||||
});
|
||||
assert_eq!(client.build(b"{}".to_vec()).await, Ok(built));
|
||||
}
|
||||
}
|
||||
|
|
@ -16,8 +16,10 @@
|
|||
//! instead of waiting up to `POLL_INTERVAL`.
|
||||
//!
|
||||
//! [`merged`] reads the same delivery for a merge, which the webhook handler
|
||||
//! turns into a deploy. The poll has no counterpart: it lists open PRs only,
|
||||
//! so a merge whose delivery is lost deploys nothing.
|
||||
//! turns into a deploy, and [`changed_head`] reads it for a PR's new head,
|
||||
//! which the handler turns into a build check (`crate::builder`). The poll
|
||||
//! has no counterpart to either: a merge whose delivery is lost deploys
|
||||
//! nothing, and a head whose delivery is lost is not built.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
|
@ -58,12 +60,17 @@ struct WebhookPullRequest {
|
|||
/// hive builds its config repo's `main`.
|
||||
#[serde(default)]
|
||||
base: Option<WebhookBranch>,
|
||||
/// The PR's branch, whose `sha` is the commit a build check builds.
|
||||
#[serde(default)]
|
||||
head: Option<WebhookBranch>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct WebhookBranch {
|
||||
#[serde(rename = "ref")]
|
||||
name: String,
|
||||
#[serde(default)]
|
||||
sha: Option<String>,
|
||||
}
|
||||
|
||||
/// A config PR that was just merged: whose config, and the commit to deploy.
|
||||
|
|
@ -104,6 +111,43 @@ pub fn merged(body: &[u8]) -> Option<MergedConfigPr> {
|
|||
})
|
||||
}
|
||||
|
||||
/// A config PR whose head just changed: whose config, the head commit, and
|
||||
/// the PR.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub struct ChangedConfigPr {
|
||||
pub agent: String,
|
||||
pub rev: String,
|
||||
pub pr: u64,
|
||||
}
|
||||
|
||||
/// The new head a verified `ConfigPr` delivery reports, if it reports one: a
|
||||
/// PR opened, reopened, or pushed to (`synchronized`). Edits and labels leave
|
||||
/// the head as it was and build nothing. A body that does not parse is
|
||||
/// `None`; [`ConfigPrCache::apply_webhook_delivery`] already logs it.
|
||||
pub fn changed_head(body: &[u8]) -> Option<ChangedConfigPr> {
|
||||
let payload: ConfigPrWebhookPayload = serde_json::from_slice(body).ok()?;
|
||||
if !matches!(
|
||||
payload.action.as_str(),
|
||||
"opened" | "reopened" | "synchronized"
|
||||
) || payload.pull_request.state != "open"
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let Some(rev) = payload.pull_request.head.and_then(|h| h.sha) else {
|
||||
tracing::warn!(
|
||||
agent = %payload.repository.name,
|
||||
pr = payload.pull_request.number,
|
||||
"config-pr webhook: PR carries no head sha; nothing built"
|
||||
);
|
||||
return None;
|
||||
};
|
||||
Some(ChangedConfigPr {
|
||||
agent: payload.repository.name,
|
||||
rev,
|
||||
pr: payload.pull_request.number,
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct WebhookRepository {
|
||||
/// The config repo's name IS the agent's name — same convention
|
||||
|
|
@ -369,4 +413,42 @@ mod tests {
|
|||
.expect("prior entry must survive a bad delivery");
|
||||
assert_eq!(status.pr_number, 5);
|
||||
}
|
||||
|
||||
fn head_changed(action: &str) -> Vec<u8> {
|
||||
serde_json::json!({
|
||||
"action": action,
|
||||
"pull_request": {
|
||||
"number": 5,
|
||||
"state": "open",
|
||||
"head": { "ref": "fix", "sha": "def456" },
|
||||
"base": { "ref": "main", "sha": "abc123" },
|
||||
},
|
||||
"repository": { "name": "damocles" },
|
||||
})
|
||||
.to_string()
|
||||
.into_bytes()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_opened_reopened_or_pushed_pr_names_its_head() {
|
||||
for action in ["opened", "reopened", "synchronized"] {
|
||||
assert_eq!(
|
||||
super::changed_head(&head_changed(action)),
|
||||
Some(super::ChangedConfigPr {
|
||||
agent: "damocles".to_owned(),
|
||||
rev: "def456".to_owned(),
|
||||
pr: 5,
|
||||
}),
|
||||
"{action}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_edit_a_label_or_a_close_builds_nothing() {
|
||||
for action in ["edited", "label_updated", "closed"] {
|
||||
assert_eq!(super::changed_head(&head_changed(action)), None, "{action}");
|
||||
}
|
||||
assert_eq!(super::changed_head(&closed(true)), None);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -21,12 +21,12 @@
|
|||
use anyhow::{Context, Result};
|
||||
use forgejo_api::structs::{
|
||||
AddCollaboratorOption, AddCollaboratorOptionPermission, ChangeFileOperation,
|
||||
ChangeFileOperationOperation, ChangeFilesOptions, CreateBranchProtectionOption,
|
||||
CreateHookOption, CreateHookOptionConfig, CreateHookOptionType, CreateRepoOption,
|
||||
CreateUserOption, EditUserOption, GetTreeQuery, GitEntry, IssueListIssuesQuery,
|
||||
IssueListIssuesQueryState, IssueListIssuesQueryType, RepoDownloadPullDiffOrPatchQuery,
|
||||
RepoGetContentsQuery, RepoGetPullRequestCommitsQuery, RepoListPullRequestsQuery,
|
||||
RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
|
||||
ChangeFileOperationOperation, ChangeFilesOptions, CommitStatusState,
|
||||
CreateBranchProtectionOption, CreateHookOption, CreateHookOptionConfig, CreateHookOptionType,
|
||||
CreateRepoOption, CreateStatusOption, CreateUserOption, EditUserOption, GetTreeQuery, GitEntry,
|
||||
IssueListIssuesQuery, IssueListIssuesQueryState, IssueListIssuesQueryType,
|
||||
RepoDownloadPullDiffOrPatchQuery, RepoGetContentsQuery, RepoGetPullRequestCommitsQuery,
|
||||
RepoListPullRequestsQuery, RepoListPullRequestsQueryState, RepoSearchQuery, StateType,
|
||||
};
|
||||
use forgejo_api::{ApiErrorKind, Auth, Forgejo, ForgejoError};
|
||||
use futures_util::{StreamExt as _, TryStreamExt as _};
|
||||
|
|
@ -40,6 +40,7 @@ use crate::webhook::DeliveryKind;
|
|||
use swarm_queue_client::review_classifier::Commit;
|
||||
|
||||
pub mod agent_token;
|
||||
pub mod builder_account;
|
||||
pub mod classifier_account;
|
||||
pub mod docs_grant;
|
||||
pub mod hive_token;
|
||||
|
|
@ -48,6 +49,24 @@ pub mod objects;
|
|||
pub mod runner_token;
|
||||
pub mod site_admin;
|
||||
|
||||
/// The longest commit status description [`Client::create_commit_status`]
|
||||
/// sends, in characters. The forge keeps more; its PR view shows one line.
|
||||
pub const STATUS_DESCRIPTION_MAX: usize = 200;
|
||||
|
||||
/// `description` cut to [`STATUS_DESCRIPTION_MAX`] characters, an ellipsis
|
||||
/// marking a cut.
|
||||
fn status_description(description: &str) -> String {
|
||||
if description.chars().count() <= STATUS_DESCRIPTION_MAX {
|
||||
return description.to_owned();
|
||||
}
|
||||
let mut cut: String = description
|
||||
.chars()
|
||||
.take(STATUS_DESCRIPTION_MAX - 1)
|
||||
.collect();
|
||||
cut.push('…');
|
||||
cut
|
||||
}
|
||||
|
||||
/// An agent's open config-PR, as [`Client::list_open_config_prs`] reports it
|
||||
/// and `GET /api/agents/{name}/config-pr` serves it.
|
||||
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ToSchema)]
|
||||
|
|
@ -163,7 +182,7 @@ const OPERATORS_TEAM: &str = "operators";
|
|||
/// The merge gate survives the distinction: [`objects`] provisions the
|
||||
/// `operators` team in **both** orgs precisely so branch protection can be
|
||||
/// applied in either.
|
||||
const CONFIG_ORG: &str = "agent-configs";
|
||||
pub(crate) const CONFIG_ORG: &str = "agent-configs";
|
||||
|
||||
/// The hive-wide knowledge repo, where the `push` hook lives. Same values as
|
||||
/// `hive-c0re::workers::knowledge::{ORG, REPO}`.
|
||||
|
|
@ -1127,6 +1146,38 @@ impl Client {
|
|||
Ok(rows.into_iter().flatten().collect())
|
||||
}
|
||||
|
||||
/// Set the commit status named `context` on `sha` of `owner/repo` to
|
||||
/// `state`, with `description` cut to [`STATUS_DESCRIPTION_MAX`]
|
||||
/// characters.
|
||||
///
|
||||
/// # Errors
|
||||
/// The forge refused the status.
|
||||
pub async fn create_commit_status(
|
||||
&self,
|
||||
owner: &str,
|
||||
repo: &str,
|
||||
sha: &str,
|
||||
context: &str,
|
||||
state: CommitStatusState,
|
||||
description: &str,
|
||||
) -> Result<()> {
|
||||
self.api
|
||||
.repo_create_status(
|
||||
owner,
|
||||
repo,
|
||||
sha,
|
||||
CreateStatusOption {
|
||||
context: Some(context.to_owned()),
|
||||
description: Some(status_description(description)),
|
||||
state: Some(state),
|
||||
target_url: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.with_context(|| format!("set the {context} status on {sha} of {owner}/{repo}"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// What the review classifier reads of `owner/repo`'s PR `number`: its
|
||||
/// head commit, its commits oldest first with their full messages, and
|
||||
/// its unified diff.
|
||||
|
|
@ -2368,4 +2419,41 @@ mod tests {
|
|||
.unwrap_err();
|
||||
assert!(!no_repo(&tree), "{tree:#}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_long_status_description_is_cut_with_an_ellipsis() {
|
||||
assert_eq!(status_description("built"), "built");
|
||||
let long = "é".repeat(STATUS_DESCRIPTION_MAX + 5);
|
||||
let cut = status_description(&long);
|
||||
assert_eq!(cut.chars().count(), STATUS_DESCRIPTION_MAX);
|
||||
assert!(cut.ends_with('…'));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_commit_status_is_posted_on_the_commit() {
|
||||
let (client, seen) = stub_forge(
|
||||
(201, r#"{"id":1}"#),
|
||||
(404, r#"{"message":"not stubbed"}"#),
|
||||
(404, r#"{"message":"not stubbed"}"#),
|
||||
)
|
||||
.await;
|
||||
client
|
||||
.create_commit_status(
|
||||
CONFIG_ORG,
|
||||
"atlas",
|
||||
"abc",
|
||||
"swarm-builder",
|
||||
CommitStatusState::Failure,
|
||||
"error: boom",
|
||||
)
|
||||
.await
|
||||
.expect("the stub accepts the status");
|
||||
let seen = seen.lock().unwrap();
|
||||
let (method, path, body) = seen.first().expect("one request");
|
||||
assert_eq!(method, "POST");
|
||||
assert_eq!(path, "/api/v1/repos/agent-configs/atlas/statuses/abc");
|
||||
assert_eq!(body["context"], "swarm-builder");
|
||||
assert_eq!(body["state"], "failure");
|
||||
assert_eq!(body["description"], "error: boom");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
284
swarm-controller/src/forge/builder_account.rs
Normal file
284
swarm-controller/src/forge/builder_account.rs
Normal file
|
|
@ -0,0 +1,284 @@
|
|||
//! The swarm builder's read-only forge account, [`LOGIN`]: created here with
|
||||
//! the admin API, put in the `hives` team of `agent-configs` (read on every
|
||||
//! config repo, including ones created later), and given one token, stored
|
||||
//! at `swarm_secret_client::builder::forge_token_path`, where the builder
|
||||
//! reads it under its own store identity to fetch the config it builds.
|
||||
//!
|
||||
//! The builder never writes to the forge: its build results become commit
|
||||
//! statuses through this daemon's own client (`crate::builder`).
|
||||
//!
|
||||
//! The token decision is [`super::agent_token`]'s, under this module's name
|
||||
//! and scopes, so a stored token the forge still lists with the same last
|
||||
//! eight characters and scopes is kept, and a pass whose reads fail queues
|
||||
//! nothing.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::{Context, Result, bail};
|
||||
use forgejo_api::structs::RepoSearchQuery;
|
||||
use forgejo_api::{Auth, Forgejo};
|
||||
use swarm_secret_client::builder::forge_token_path;
|
||||
use swarm_secret_client::forge;
|
||||
|
||||
use super::Client;
|
||||
use super::agent_token::{Decision, Observed, classify_named, is_not_found};
|
||||
use super::hive_token::TeamSeen;
|
||||
|
||||
/// The builder's forge login.
|
||||
pub const LOGIN: &str = "builder";
|
||||
|
||||
/// The forge's name for the account's one token.
|
||||
pub const TOKEN_NAME: &str = "swarm-builder";
|
||||
|
||||
/// What the token may do: read repositories. Which repositories is the
|
||||
/// account's business, and the account is only in the `hives` team.
|
||||
pub const TOKEN_SCOPES: &str = "read:repository";
|
||||
|
||||
/// How often [`spawn`] re-checks the account and its token.
|
||||
const RECONCILE_INTERVAL: std::time::Duration = std::time::Duration::from_mins(5);
|
||||
|
||||
/// The email this module creates the account with, and the one it requires
|
||||
/// before acting on an account named [`LOGIN`]. Under its own domain, so it
|
||||
/// equals no agent's or hive's account email.
|
||||
const EMAIL: &str = "builder@services.hyperhive.local";
|
||||
|
||||
/// Whether a pass that found `seen`, and `member` for the account's place in
|
||||
/// the `hives` team, queues a `MintBuilderForgeToken` job.
|
||||
/// [`Observed::Unknown`] never does, so a forge or store outage can't rotate
|
||||
/// the token, and neither does a [`Decision::Conflict`], which the job would
|
||||
/// refuse. A kept token queues only when the account is known to be out of
|
||||
/// the team; that job finds the token current and does not rotate it.
|
||||
pub fn needs_job(seen: Observed, member: Option<bool>) -> bool {
|
||||
match seen {
|
||||
Observed::NoForgeUser | Observed::Decided(Decision::Mint(_)) => true,
|
||||
Observed::Decided(Decision::Keep) => member == Some(false),
|
||||
Observed::Unknown | Observed::Decided(Decision::Conflict) => false,
|
||||
}
|
||||
}
|
||||
|
||||
impl Client {
|
||||
/// Make sure [`LOGIN`] exists, is in the `hives` team, and holds a live
|
||||
/// [`TOKEN_NAME`] token in the store, minting one when it does not. The
|
||||
/// whole job of the `MintBuilderForgeToken` node.
|
||||
///
|
||||
/// # Errors
|
||||
/// When the store or the forge refuses a step, when an account named
|
||||
/// [`LOGIN`] exists with another email, or when the read-back does not
|
||||
/// show the new token.
|
||||
pub async fn ensure_builder_forge_token(&self) -> Result<()> {
|
||||
let path = forge_token_path()?;
|
||||
self.ensure_builder_user().await?;
|
||||
let team = self.ensure_hives_team().await?;
|
||||
self.api
|
||||
.org_add_team_member(team, LOGIN)
|
||||
.await
|
||||
.with_context(|| format!("adding {LOGIN} to the hives team"))?;
|
||||
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
let (stored, listed) = self.read_builder_token(&store, &path).await?;
|
||||
let reason = match classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed) {
|
||||
Decision::Keep => {
|
||||
tracing::debug!("builder forge token is current; left as it is");
|
||||
return Ok(());
|
||||
}
|
||||
Decision::Conflict => bail!(
|
||||
"the forge lists more than one {TOKEN_NAME:?} token for {LOGIN:?}; delete them \
|
||||
by id and re-run"
|
||||
),
|
||||
Decision::Mint(reason) => reason,
|
||||
};
|
||||
let credential = self
|
||||
.replace_token(&store, LOGIN, TOKEN_NAME, TOKEN_SCOPES, &listed, &path)
|
||||
.await?;
|
||||
|
||||
Forgejo::new(Auth::Token(&credential.value), self.url.clone())
|
||||
.context("building a forge client with the new token")?
|
||||
.repo_search(RepoSearchQuery::default())
|
||||
.await
|
||||
.with_context(|| format!("using {LOGIN}'s new forge token"))?;
|
||||
let (stored, listed) = self.read_builder_token(&store, &path).await?;
|
||||
let now = classify_named(TOKEN_NAME, TOKEN_SCOPES, stored.as_ref(), &listed);
|
||||
if now != Decision::Keep {
|
||||
bail!("{LOGIN}'s new forge token does not read back as current: {now:?}");
|
||||
}
|
||||
tracing::info!(?reason, %path, "builder forge token minted and stored");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The stored token at `path` and every token the forge lists for
|
||||
/// [`LOGIN`], which must exist.
|
||||
async fn read_builder_token(
|
||||
&self,
|
||||
store: &swarm_secret_client::SecretStore,
|
||||
path: &str,
|
||||
) -> Result<(
|
||||
Option<forge::Credential>,
|
||||
Vec<forgejo_api::structs::AccessToken>,
|
||||
)> {
|
||||
let stored = store
|
||||
.read_optional(path)
|
||||
.await
|
||||
.with_context(|| format!("reading {path}"))?;
|
||||
let Some(listed) = self.list_agent_tokens(LOGIN).await? else {
|
||||
bail!("the forge has no user {LOGIN:?} after this job created it");
|
||||
};
|
||||
Ok((stored, listed))
|
||||
}
|
||||
|
||||
/// Create [`LOGIN`] if there is none, refuse one with another email, and
|
||||
/// lock it out of creating repos.
|
||||
async fn ensure_builder_user(&self) -> Result<()> {
|
||||
self.create_user(LOGIN, EMAIL).await?;
|
||||
let user = self
|
||||
.api
|
||||
.user_get(LOGIN)
|
||||
.await
|
||||
.with_context(|| format!("read forge user {LOGIN}"))?;
|
||||
if user.email.as_deref() != Some(EMAIL) {
|
||||
bail!(
|
||||
"forge account `{LOGIN}` exists but is not the swarm builder's (email {:?}, \
|
||||
expected `{EMAIL}`); refusing to give it read on every config repo",
|
||||
user.email
|
||||
);
|
||||
}
|
||||
self.api
|
||||
.admin_edit_user(LOGIN, super::repo_creation_lockdown(LOGIN))
|
||||
.await
|
||||
.with_context(|| format!("disable repo creation for forge user {LOGIN}"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// What the store and the forge say about the account's token, and
|
||||
/// whether it is in the `hives` team (`None` when that read failed).
|
||||
async fn observe_builder(&self) -> Result<(Observed, Option<bool>)> {
|
||||
let path = forge_token_path()?;
|
||||
let store = crate::store::connect()
|
||||
.await
|
||||
.context("logging in to the swarm secret store")?;
|
||||
let stored = match store.read_optional::<forge::Credential>(&path).await {
|
||||
Ok(stored) => stored,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "builder forge token: store read failed");
|
||||
return Ok((Observed::Unknown, None));
|
||||
}
|
||||
};
|
||||
let seen = match self.list_agent_tokens(LOGIN).await {
|
||||
Ok(Some(listed)) => Observed::Decided(classify_named(
|
||||
TOKEN_NAME,
|
||||
TOKEN_SCOPES,
|
||||
stored.as_ref(),
|
||||
&listed,
|
||||
)),
|
||||
Ok(None) => Observed::NoForgeUser,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), "builder forge token: forge read failed");
|
||||
Observed::Unknown
|
||||
}
|
||||
};
|
||||
let member = match self.observe_hives_team().await {
|
||||
TeamSeen::Matches(id) | TeamSeen::Drifted(id) => {
|
||||
match self.api.org_list_team_member(id, LOGIN).await {
|
||||
Ok(_) => Some(true),
|
||||
Err(e) if is_not_found(&e) => Some(false),
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "builder forge token: team membership read failed");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
TeamSeen::Absent => Some(false),
|
||||
TeamSeen::Unknown => None,
|
||||
};
|
||||
Ok((seen, member))
|
||||
}
|
||||
}
|
||||
|
||||
/// Check the account now and every [`RECONCILE_INTERVAL`] after, and call
|
||||
/// `enqueue`, which inserts a `MintBuilderForgeToken` node, when it needs a
|
||||
/// job. A pass that fails is logged and retried on the next tick.
|
||||
pub fn spawn(client: Arc<Client>, enqueue: impl Fn() + Send + 'static) {
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
let Some(outcome) =
|
||||
super::within_forge_deadline("builder forge token", client.observe_builder()).await
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
match outcome {
|
||||
Ok((seen, member)) if needs_job(seen, member) => {
|
||||
tracing::info!(?seen, ?member, "builder forge token: queueing a job");
|
||||
enqueue();
|
||||
}
|
||||
Ok((seen, member)) => {
|
||||
tracing::debug!(?seen, ?member, "builder forge token: no job");
|
||||
}
|
||||
Err(e) => tracing::warn!(
|
||||
error = %format!("{e:#}"),
|
||||
retry_in_s = RECONCILE_INTERVAL.as_secs(),
|
||||
"builder forge token: pass failed; retrying next tick"
|
||||
),
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::agent_token::MintReason;
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_token_reads_repositories_and_nothing_else() {
|
||||
assert_eq!(TOKEN_SCOPES, "read:repository");
|
||||
assert_ne!(TOKEN_NAME, super::super::agent_token::AGENT_TOKEN_NAME);
|
||||
assert_ne!(TOKEN_NAME, super::super::hive_token::HIVE_TOKEN_NAME);
|
||||
assert_ne!(TOKEN_NAME, super::super::classifier_account::TOKEN_NAME);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_mint_a_missing_account_or_a_lost_membership_queues() {
|
||||
let mint = Observed::Decided(Decision::Mint(MintReason::NotStored));
|
||||
let keep = Observed::Decided(Decision::Keep);
|
||||
assert!(needs_job(Observed::NoForgeUser, None));
|
||||
assert!(needs_job(mint, Some(true)));
|
||||
assert!(needs_job(keep, Some(false)));
|
||||
assert!(!needs_job(keep, Some(true)));
|
||||
assert!(!needs_job(keep, None));
|
||||
assert!(!needs_job(
|
||||
Observed::Decided(Decision::Conflict),
|
||||
Some(false)
|
||||
));
|
||||
assert!(!needs_job(Observed::Unknown, Some(false)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_accounts_email_is_no_agents_or_hives() {
|
||||
assert_ne!(EMAIL, super::super::agent_email(LOGIN));
|
||||
assert_ne!(EMAIL, super::super::legacy_agent_email(LOGIN));
|
||||
assert!(!LOGIN.starts_with(hive_types::RESERVED_AGENT_PREFIX));
|
||||
}
|
||||
|
||||
/// An existing account named like the builder with an agent's email is
|
||||
/// refused before the lockdown, so it never joins the team.
|
||||
#[tokio::test]
|
||||
async fn the_job_refuses_an_account_that_is_not_the_builders() {
|
||||
use super::super::tests::{lockdown_patch, stub_forge};
|
||||
|
||||
let (client, seen) = stub_forge(
|
||||
(409, r#"{"message":"user already exists"}"#),
|
||||
(200, r#"{"email":"builder@hyperhive.local"}"#),
|
||||
(200, "{}"),
|
||||
)
|
||||
.await;
|
||||
let err = client
|
||||
.ensure_builder_user()
|
||||
.await
|
||||
.expect_err("an agent's account is not the builder's");
|
||||
assert!(format!("{err:#}").contains("not the swarm builder's"));
|
||||
assert!(lockdown_patch(&seen, LOGIN).is_none());
|
||||
}
|
||||
}
|
||||
|
|
@ -235,7 +235,7 @@ impl Client {
|
|||
/// The id of [`HIVES_TEAM`] in `agent-configs`, creating it or
|
||||
/// correcting its settings first when needed. Its members are not
|
||||
/// touched here.
|
||||
async fn ensure_hives_team(&self) -> Result<i64> {
|
||||
pub(super) async fn ensure_hives_team(&self) -> Result<i64> {
|
||||
match self.observe_hives_team().await {
|
||||
TeamSeen::Matches(id) => Ok(id),
|
||||
TeamSeen::Drifted(id) => {
|
||||
|
|
@ -292,7 +292,7 @@ impl Client {
|
|||
}
|
||||
}
|
||||
|
||||
async fn observe_hives_team(&self) -> TeamSeen {
|
||||
pub(super) async fn observe_hives_team(&self) -> TeamSeen {
|
||||
match self.api.org_list_teams(CONFIG_ORG).await {
|
||||
Ok((_, teams)) => teams
|
||||
.into_iter()
|
||||
|
|
|
|||
|
|
@ -49,6 +49,7 @@ mod agent_renewal;
|
|||
mod agent_state_stream;
|
||||
mod agent_status;
|
||||
mod auth;
|
||||
mod builder;
|
||||
mod config_pr;
|
||||
mod forge;
|
||||
mod forge_account;
|
||||
|
|
@ -153,6 +154,17 @@ enum SwarmNodeKind {
|
|||
/// live token in the swarm store, where this daemon reads it to post the
|
||||
/// classifier's reviews. See `forge::classifier_account`.
|
||||
MintClassifierForgeToken,
|
||||
/// Make sure the swarm builder's read-only forge account exists, is in
|
||||
/// the `hives` team, and holds a live token in the swarm store, where the
|
||||
/// builder reads it to fetch config repos. See `forge::builder_account`.
|
||||
MintBuilderForgeToken,
|
||||
/// Build `agent`'s config at `rev`, the head of its config PR `pr`, on
|
||||
/// the swarm builder, and write the result as a commit status on `rev`.
|
||||
/// See `builder`.
|
||||
///
|
||||
/// Carries no hive: the check builds on the builder's own agent base,
|
||||
/// not on any hive.
|
||||
BuildAgent { agent: String, rev: String, pr: u64 },
|
||||
/// Re-mint `agent`'s queue secret if it is still stored and old enough.
|
||||
/// See `agent_renewal`.
|
||||
///
|
||||
|
|
@ -194,6 +206,8 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
|
|||
SwarmNodeKind::MintHiveSenderToken { .. } => "mint_hive_sender_token".to_owned(),
|
||||
SwarmNodeKind::MintHiveForgeToken { .. } => "mint_hive_forge_token".to_owned(),
|
||||
SwarmNodeKind::MintClassifierForgeToken => "mint_classifier_forge_token".to_owned(),
|
||||
SwarmNodeKind::MintBuilderForgeToken => "mint_builder_forge_token".to_owned(),
|
||||
SwarmNodeKind::BuildAgent { .. } => "build_agent".to_owned(),
|
||||
SwarmNodeKind::RenewAgentQueueCredential { .. } => {
|
||||
"renew_agent_queue_credential".to_owned()
|
||||
}
|
||||
|
|
@ -224,7 +238,12 @@ impl hive_jobq_wire::WireNode for SwarmNodeKind {
|
|||
}
|
||||
SwarmNodeKind::MintHiveSenderToken { hive }
|
||||
| SwarmNodeKind::MintHiveForgeToken { hive } => serde_json::json!({ "hive": hive }),
|
||||
SwarmNodeKind::MintClassifierForgeToken => serde_json::json!({}),
|
||||
SwarmNodeKind::MintClassifierForgeToken | SwarmNodeKind::MintBuilderForgeToken => {
|
||||
serde_json::json!({})
|
||||
}
|
||||
SwarmNodeKind::BuildAgent { agent, rev, pr } => {
|
||||
serde_json::json!({ "agent": agent, "rev": rev, "pr": pr })
|
||||
}
|
||||
SwarmNodeKind::TriggerDeploy { hive, agent, rev } => {
|
||||
serde_json::json!({ "agent": agent, "hive": hive, "rev": rev })
|
||||
}
|
||||
|
|
@ -382,6 +401,10 @@ async fn run_swarm_node(
|
|||
}
|
||||
SwarmNodeKind::MintHiveForgeToken { hive } => mint_hive_forge_token(forge, &hive).await,
|
||||
SwarmNodeKind::MintClassifierForgeToken => mint_classifier_forge_token(forge).await,
|
||||
SwarmNodeKind::MintBuilderForgeToken => mint_builder_forge_token(forge).await,
|
||||
SwarmNodeKind::BuildAgent { agent, rev, pr } => {
|
||||
build_agent(&deps, forge, agent, rev, pr).await
|
||||
}
|
||||
SwarmNodeKind::RenewAgentQueueCredential { agent } => renew_queue_credential(&agent).await,
|
||||
SwarmNodeKind::SetAgentWanted { hive, agent } => match deps.wanted {
|
||||
None => Outcome::Failed(
|
||||
|
|
@ -536,6 +559,42 @@ async fn mint_hive_forge_token(
|
|||
}
|
||||
}
|
||||
|
||||
/// The `BuildAgent` arm, lifted out for `clippy::too_many_lines`.
|
||||
async fn build_agent(
|
||||
deps: &WorkerDeps,
|
||||
forge: Option<Arc<forge::Client>>,
|
||||
agent: String,
|
||||
rev: String,
|
||||
pr: u64,
|
||||
) -> hive_jobq::scheduler::Outcome {
|
||||
let job = swarm_queue_client::builder::BuildAgent { agent, rev, pr };
|
||||
builder::run(deps.queue.as_ref(), forge.as_deref(), job).await
|
||||
}
|
||||
|
||||
/// The `MintBuilderForgeToken` arm, lifted out for the same reason.
|
||||
async fn mint_builder_forge_token(
|
||||
forge: Option<Arc<forge::Client>>,
|
||||
) -> hive_jobq::scheduler::Outcome {
|
||||
use hive_jobq::scheduler::Outcome;
|
||||
|
||||
let Some(client) = forge else {
|
||||
return Outcome::Failed(
|
||||
"no forge access here (SWARM_CONTROLLER_FORGE_URL unset, or its token \
|
||||
not in the swarm secret store), so no builder forge token can be minted"
|
||||
.to_owned(),
|
||||
);
|
||||
};
|
||||
match forge_node(
|
||||
"mint_builder_forge_token",
|
||||
client.ensure_builder_forge_token(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(()) => Outcome::Done,
|
||||
Err(e) => Outcome::Failed(format!("{e:#}")),
|
||||
}
|
||||
}
|
||||
|
||||
/// The `MintClassifierForgeToken` arm, lifted out for the same reason.
|
||||
async fn mint_classifier_forge_token(
|
||||
forge: Option<Arc<forge::Client>>,
|
||||
|
|
@ -1897,6 +1956,33 @@ fn claiming_hives(
|
|||
.collect()
|
||||
}
|
||||
|
||||
/// Queue a build check of a config PR's new head. Every head is built, on
|
||||
/// whichever hive the agent runs: the check builds on the builder's own
|
||||
/// agent base.
|
||||
fn queue_config_build(state: &AppState, changed: config_pr::ChangedConfigPr) {
|
||||
let config_pr::ChangedConfigPr { agent, rev, pr } = changed;
|
||||
let inserted = state
|
||||
.jobq
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.insert_job(None, |b| {
|
||||
vec![
|
||||
b.node(SwarmNodeKind::BuildAgent {
|
||||
agent: agent.clone(),
|
||||
rev: rev.clone(),
|
||||
pr,
|
||||
})
|
||||
.guid(),
|
||||
]
|
||||
});
|
||||
match inserted {
|
||||
Ok(_) => tracing::info!(%agent, %rev, pr, "config-pr head: build queued"),
|
||||
Err(e) => {
|
||||
tracing::warn!(%agent, %rev, pr, error = %e, "config-pr head: queueing the build failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Queue the deploy of a merged config PR on the one hive that runs its agent.
|
||||
///
|
||||
/// The hive is found by scanning every hive's wanted state, since nothing
|
||||
|
|
@ -2467,6 +2553,20 @@ fn queue_classifier_forge_token_mint(
|
|||
.map_err(|e| anyhow::anyhow!("{e}"))
|
||||
}
|
||||
|
||||
/// Insert one `MintBuilderForgeToken` job and return its node's id.
|
||||
/// `forge::builder_account::spawn`'s periodic pass is the one caller.
|
||||
fn queue_builder_forge_token_mint(
|
||||
sched: &Mutex<hive_jobq::scheduler::Scheduler<SwarmNodeKind, SwarmResourceKind>>,
|
||||
) -> Result<Vec<hive_jobq::NodeId>> {
|
||||
sched
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.insert_job(None, |b| {
|
||||
vec![b.node(SwarmNodeKind::MintBuilderForgeToken).guid()]
|
||||
})
|
||||
.map_err(|e| anyhow::anyhow!("{e}"))
|
||||
}
|
||||
|
||||
/// The homeserver agents' own matrix accounts are minted on. Unset is this
|
||||
/// controller creating agents with no matrix account, which the mint node
|
||||
/// reports by name rather than `main` refusing to start.
|
||||
|
|
@ -2509,7 +2609,8 @@ fn spawn_matrix_account_backfill(
|
|||
}
|
||||
|
||||
/// Start the forge's periodic passes — the swarm-wide objects, the runner
|
||||
/// registration token, hives' read tokens, agents' `internal/docs` grants, and
|
||||
/// registration token, hives' and the builder's read tokens, agents'
|
||||
/// `internal/docs` grants, and
|
||||
/// agents' tokens (which also
|
||||
/// sweeps legacy tokens; see `forge::legacy_tokens`) —
|
||||
/// once there is a forge client. Lifted out of `main` for
|
||||
|
|
@ -2535,6 +2636,12 @@ fn spawn_forge_workers(
|
|||
}
|
||||
});
|
||||
let sched = Arc::clone(jobq);
|
||||
forge::builder_account::spawn(Arc::clone(&client), move || {
|
||||
if let Err(e) = queue_builder_forge_token_mint(&sched) {
|
||||
tracing::warn!(error = %format!("{e:#}"), "builder forge token: queueing failed");
|
||||
}
|
||||
});
|
||||
let sched = Arc::clone(jobq);
|
||||
forge::docs_grant::spawn(Arc::clone(&client), move |agents| {
|
||||
if let Err(e) = queue_docs_grants(&sched, agents) {
|
||||
tracing::warn!(error = %format!("{e:#}"), "docs grants: queueing failed");
|
||||
|
|
@ -4262,6 +4369,10 @@ mod tests {
|
|||
"mint_classifier_forge_token",
|
||||
SwarmNodeKind::MintClassifierForgeToken,
|
||||
),
|
||||
(
|
||||
"mint_builder_forge_token",
|
||||
SwarmNodeKind::MintBuilderForgeToken,
|
||||
),
|
||||
];
|
||||
for (label, kind) in nodes {
|
||||
let mut sched = hive_jobq::scheduler::Scheduler::new(
|
||||
|
|
|
|||
|
|
@ -418,6 +418,9 @@ pub(super) async fn post_webhook_forge(
|
|||
if let Some(merged) = crate::config_pr::merged(&body) {
|
||||
super::queue_merged_config_deploy(&state, merged).await;
|
||||
}
|
||||
if let Some(changed) = crate::config_pr::changed_head(&body) {
|
||||
super::queue_config_build(&state, changed);
|
||||
}
|
||||
}
|
||||
DeliveryKind::VcsActivity => {
|
||||
if let Some(activity) = crate::vcs_metrics::parse_push(&body) {
|
||||
|
|
|
|||
|
|
@ -162,6 +162,11 @@ struct Args {
|
|||
#[arg(long = "review-classifier-client")]
|
||||
review_classifier_clients: Vec<String>,
|
||||
|
||||
/// Client ids that answer the swarm builder's build requests.
|
||||
/// Repeatable, empty by default, which denies such a client.
|
||||
#[arg(long = "builder-client")]
|
||||
builder_clients: Vec<String>,
|
||||
|
||||
/// Role on the secret store's `cert` auth mount this responder logs in
|
||||
/// as to read the callout keys, its own OIDC client secret and agent
|
||||
/// credentials. The store is found
|
||||
|
|
@ -226,7 +231,8 @@ async fn serve(args: &Args) -> anyhow::Result<()> {
|
|||
args.hive_publish_subjects.clone(),
|
||||
)?
|
||||
.with_agent_token_subjects(args.agent_token_publish_subjects.clone())?
|
||||
.with_review_classifiers(args.review_classifier_clients.clone());
|
||||
.with_review_classifiers(args.review_classifier_clients.clone())
|
||||
.with_builders(args.builder_clients.clone());
|
||||
let http = reqwest::Client::new();
|
||||
let settings = swarm_secret_client::client::Settings::from_env()
|
||||
.context("reading the secret store's address and this responder's identity")?;
|
||||
|
|
|
|||
|
|
@ -50,6 +50,7 @@ pub struct Policy {
|
|||
extra_hive_subjects: Vec<String>,
|
||||
agent_token_subjects: Vec<String>,
|
||||
review_classifiers: Vec<String>,
|
||||
builders: Vec<String>,
|
||||
}
|
||||
|
||||
/// Placeholder replaced with the hive's own name in `extra_hive_subjects`.
|
||||
|
|
@ -120,6 +121,7 @@ impl Policy {
|
|||
extra_hive_subjects,
|
||||
agent_token_subjects: Vec::new(),
|
||||
review_classifiers: Vec::new(),
|
||||
builders: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -148,6 +150,13 @@ impl Policy {
|
|||
self
|
||||
}
|
||||
|
||||
/// Set the client ids that answer the swarm builder's build requests.
|
||||
#[must_use]
|
||||
pub fn with_builders(mut self, clients: Vec<String>) -> Self {
|
||||
self.builders = clients;
|
||||
self
|
||||
}
|
||||
|
||||
/// The permissions for an agent whose own credential was verified, or
|
||||
/// `None` when none are configured.
|
||||
///
|
||||
|
|
@ -209,6 +218,9 @@ impl Policy {
|
|||
if self.review_classifiers.iter().any(|c| c == client_id) {
|
||||
return Some(Self::review_classifier_permissions(client_id));
|
||||
}
|
||||
if self.builders.iter().any(|c| c == client_id) {
|
||||
return Some(Self::builder_permissions(client_id));
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
|
|
@ -229,6 +241,20 @@ impl Policy {
|
|||
}
|
||||
}
|
||||
|
||||
/// What the builder may do: hear build requests and answer each on its
|
||||
/// reply subject, for as long as a build may take. Its publish list is
|
||||
/// its own inbox, for [`Self::review_classifier_permissions`]' reason.
|
||||
fn builder_permissions(client_id: &str) -> Permissions {
|
||||
Permissions {
|
||||
publish: vec![Self::inbox(client_id)],
|
||||
subscribe: Some(vec![
|
||||
swarm_queue_client::builder::SUBJECT.to_owned(),
|
||||
Self::inbox(client_id),
|
||||
]),
|
||||
responses: Some(swarm_queue_client::builder::REPLY_DEADLINE),
|
||||
}
|
||||
}
|
||||
|
||||
/// What a hive may subscribe to: the two events the controller sends it,
|
||||
/// and its own reply inbox.
|
||||
///
|
||||
|
|
@ -562,6 +588,8 @@ impl Policy {
|
|||
// A review classifier try-out, sent as a request: the reply comes
|
||||
// back on this client's inbox, which it already subscribes to.
|
||||
swarm_queue_client::review_classifier::SUBJECT.to_owned(),
|
||||
// A build of an agent config, sent as a request the same way.
|
||||
swarm_queue_client::builder::SUBJECT.to_owned(),
|
||||
// The wanted-state buckets — one per hive, all created and written
|
||||
// by this single client.
|
||||
//
|
||||
|
|
@ -1261,6 +1289,60 @@ mod tests {
|
|||
}
|
||||
}
|
||||
|
||||
const BUILDER: &str = "swarm-builder";
|
||||
|
||||
#[test]
|
||||
fn the_builder_hears_its_subject_and_inbox_only_and_answers_by_reply() {
|
||||
use swarm_queue_client::builder::{REPLY_DEADLINE, SUBJECT};
|
||||
use swarm_queue_client::client_inbox_prefix;
|
||||
|
||||
let p = policy()
|
||||
.with_builders(vec![BUILDER.to_owned()])
|
||||
.permissions(BUILDER)
|
||||
.expect("a configured builder is admitted");
|
||||
let inbox = client_inbox_prefix(BUILDER);
|
||||
assert!(may_subscribe(&p, SUBJECT));
|
||||
assert!(may_subscribe(&p, &reply(&inbox)));
|
||||
for refused in [
|
||||
">".to_owned(),
|
||||
"$SWARM.>".to_owned(),
|
||||
swarm_queue_client::review_classifier::SUBJECT.to_owned(),
|
||||
reply(&client_inbox_prefix("swarm-controller")),
|
||||
] {
|
||||
assert!(
|
||||
!may_subscribe(&p, &refused),
|
||||
"the builder may subscribe to {refused}"
|
||||
);
|
||||
}
|
||||
assert_eq!(p.publish, vec![format!("{inbox}.>")]);
|
||||
assert_eq!(p.responses, Some(REPLY_DEADLINE));
|
||||
// The builder's id is nothing until it is configured.
|
||||
assert_eq!(policy().permissions(BUILDER), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_the_reader_may_send_a_build() {
|
||||
use swarm_queue_client::builder::SUBJECT;
|
||||
|
||||
let p = with_classifier().with_builders(vec![BUILDER.to_owned()]);
|
||||
let reader = p.permissions("swarm-controller").expect("the reader");
|
||||
assert!(reader.publish.iter().any(|g| covers(g, SUBJECT)));
|
||||
for (who, grant) in [
|
||||
("a hive", p.permissions("hive-alpha").expect("a hive")),
|
||||
("the builder", p.permissions(BUILDER).expect("admitted")),
|
||||
(
|
||||
"the classifier",
|
||||
p.permissions(CLASSIFIER).expect("admitted"),
|
||||
),
|
||||
] {
|
||||
assert!(
|
||||
!grant.publish.iter().any(|g| covers(g, SUBJECT)),
|
||||
"{who} may send a build: {:?}",
|
||||
grant.publish
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_prefix_alone_names_no_hive() {
|
||||
// `hive-` would otherwise produce `$KV.hive-status.`, a subject nobody
|
||||
|
|
|
|||
75
swarm-queue-client/src/builder.rs
Normal file
75
swarm-queue-client/src/builder.rs
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
//! The swarm builder's builds: the subject swarm-controller sends a build on,
|
||||
//! and what travels each way.
|
||||
//!
|
||||
//! Three crates name these: the controller sends the request, the builder
|
||||
//! answers it, and the auth-callout responder grants the one subject to each.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// The subject every build goes to. The controller is its one publisher and
|
||||
/// the builder its one subscriber; the builder answers on the request's reply
|
||||
/// subject.
|
||||
pub const SUBJECT: &str = "$SWARM.builder.build";
|
||||
|
||||
/// The queue group the builder subscribes in, so a second instance shares the
|
||||
/// requests instead of running each one twice.
|
||||
pub const QUEUE_GROUP: &str = "builder";
|
||||
|
||||
/// The longest one build may take before the builder stops it and answers
|
||||
/// with a timeout.
|
||||
pub const BUILD_DEADLINE: Duration = Duration::from_hours(1);
|
||||
|
||||
/// How long the controller waits for the reply, and the queue lets the
|
||||
/// builder answer: the build at its deadline, plus a minute for the rest.
|
||||
pub const REPLY_DEADLINE: Duration = Duration::from_mins(61);
|
||||
|
||||
/// One build: an agent's config repo at one commit, on the builder's own
|
||||
/// agent base.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct BuildAgent {
|
||||
/// The agent, which is also its config repo's name.
|
||||
pub agent: String,
|
||||
/// The commit to build.
|
||||
pub rev: String,
|
||||
/// The PR the commit is the head of; its `refs/pull/<pr>/head` is the ref
|
||||
/// the builder fetches the commit through.
|
||||
pub pr: u64,
|
||||
}
|
||||
|
||||
/// The builder's answer to one [`BuildAgent`].
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(tag = "state", rename_all = "kebab-case")]
|
||||
pub enum Reply {
|
||||
/// The config built; `out_path` is the system closure's store path.
|
||||
Built { out_path: String },
|
||||
/// The build did not finish: the last lines `nix` printed, or why the
|
||||
/// builder could not start it.
|
||||
Failed { error_tail: String },
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_wire_tags_the_reply_by_state() {
|
||||
let json = serde_json::to_value(Reply::Built {
|
||||
out_path: "/nix/store/x".to_owned(),
|
||||
})
|
||||
.expect("encodes");
|
||||
assert_eq!(json["state"], "built");
|
||||
assert_eq!(json["out_path"], "/nix/store/x");
|
||||
let json = serde_json::to_value(Reply::Failed {
|
||||
error_tail: "e".to_owned(),
|
||||
})
|
||||
.expect("encodes");
|
||||
assert_eq!(json["state"], "failed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_reply_deadline_outlasts_the_build() {
|
||||
assert!(REPLY_DEADLINE > BUILD_DEADLINE);
|
||||
}
|
||||
}
|
||||
|
|
@ -404,6 +404,9 @@ pub mod review_classifier;
|
|||
|
||||
pub mod stored_secret;
|
||||
|
||||
/// The builder's build subject and the request and reply it carries.
|
||||
pub mod builder;
|
||||
|
||||
/// Only the fields this needs; authelia returns several.
|
||||
#[derive(serde::Deserialize)]
|
||||
struct TokenResponse {
|
||||
|
|
|
|||
64
swarm-secret-client/src/builder.rs
Normal file
64
swarm-secret-client/src/builder.rs
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
//! The swarm builder agreement: where its forge token and its queue client
|
||||
//! secret live in the store, and what each object holds.
|
||||
//!
|
||||
//! swarm-controller mints the forge token; the builder reads it, and its queue
|
||||
//! client secret, under its own certificate. Its read stanza in
|
||||
//! `swarm-bao.nix` names exactly these two paths.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::{
|
||||
Error,
|
||||
path::{Kind, principal_prefix},
|
||||
};
|
||||
|
||||
/// The builder's name segment under [`Kind::Service`].
|
||||
const SERVICE: &str = "builder";
|
||||
|
||||
/// The path holding the token of the builder's read-only forge account, as a
|
||||
/// [`crate::forge::Credential`].
|
||||
///
|
||||
/// # Errors
|
||||
/// Never in practice: the name segment is a constant. The `Result` is
|
||||
/// [`principal_prefix`]'s.
|
||||
pub fn forge_token_path() -> Result<String, Error> {
|
||||
let prefix = principal_prefix(Kind::Service, SERVICE)?;
|
||||
Ok(format!("{prefix}/forge/token"))
|
||||
}
|
||||
|
||||
/// The path holding the queue client secret of the builder's OIDC client
|
||||
/// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's.
|
||||
///
|
||||
/// # Errors
|
||||
/// [`Error::PathSegment`] when `client_id` is not a single path segment.
|
||||
pub fn client_secret_path(client_id: &str) -> Result<String, Error> {
|
||||
let prefix = principal_prefix(Kind::Service, client_id)?;
|
||||
Ok(format!("{prefix}/oidc/client"))
|
||||
}
|
||||
|
||||
/// What [`client_secret_path`] holds.
|
||||
///
|
||||
/// No `Debug`: `value` authenticates the builder at the queue.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ClientSecret {
|
||||
/// The secret, under the field name the publisher writes.
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_paths_are_the_ones_the_store_grants() {
|
||||
// Spelled out: `swarm-bao.nix` grants each literal.
|
||||
assert_eq!(
|
||||
forge_token_path().expect("a constant segment is legal"),
|
||||
"swarm/services/builder/forge/token"
|
||||
);
|
||||
assert_eq!(
|
||||
client_secret_path("swarm-builder").expect("a plain id is legal"),
|
||||
"swarm/services/swarm-builder/oidc/client"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -6,7 +6,7 @@
|
|||
//! deployment's environment into a logged-in client ([`client`]), and, per kind
|
||||
//! of secret, the path it lives at together with the fields it holds
|
||||
//! ([`matrix`], [`queue`], [`mtls`], [`forge`], [`github`], [`acp`], [`ci`],
|
||||
//! [`review_classifier`]). Each of those is a thing the controller
|
||||
//! [`review_classifier`], [`builder`]). Each of those is a thing the controller
|
||||
//! and a hive must say identically, so it is said once here.
|
||||
//!
|
||||
//! [`policy`] is the same kind of agreement seen from the other side: which of
|
||||
|
|
@ -24,6 +24,7 @@
|
|||
//! value inside it, and its doc explains why that is not circular.
|
||||
|
||||
pub mod acp;
|
||||
pub mod builder;
|
||||
pub mod ci;
|
||||
pub mod client;
|
||||
pub mod forge;
|
||||
|
|
|
|||
Loading…
Reference in a new issue