swarm-builder: build each agent config PR head and post a commit status
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).
When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.
The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.
Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.
This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
This commit is contained in:
atlas
parent
aece1c26d0
commit
b68b74d687
43 changed files with 2452 additions and 42 deletions
|
|
@ -202,6 +202,13 @@ hand-maintained per-file tree drifts out of sync with the code.
|
|||
for agents: nginx sends it `/select/logsql/` requests carrying an agent
|
||||
token, it answers 200 or 401 via `swarm-agent-verify`. Runs on the log
|
||||
store's host (`swarm-victorialogs.nix`).
|
||||
- **`swarm-builder/`** — the swarm builder's build daemon, in its own
|
||||
container where `deploy.builder.enable` is set. Takes an agent config PR's
|
||||
head from `swarm-controller` over the queue, builds it on its host's agent
|
||||
base through the host nix daemon (`nix/builder/check-flake`), and replies
|
||||
with the store path or the error tail; the controller writes the commit
|
||||
status. Holds a read-only forge token. Design:
|
||||
[`docs/swarm/builder.md`](docs/swarm/builder.md).
|
||||
- **`swarmctl/`** — swarm-level operator CLI, installed by the
|
||||
swarm-controller module on the host that runs the daemon. Runs as
|
||||
**root and acts directly** — it serves no socket, publishes no HTTP
|
||||
|
|
|
|||
15
Cargo.lock
generated
15
Cargo.lock
generated
|
|
@ -4829,6 +4829,21 @@ dependencies = [
|
|||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "swarm-builder"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-nats",
|
||||
"futures-util",
|
||||
"hive-log",
|
||||
"serde_json",
|
||||
"swarm-queue-client",
|
||||
"swarm-secret-client",
|
||||
"tokio",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "swarm-controller"
|
||||
version = "0.1.0"
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ members = [
|
|||
"hivectl",
|
||||
"swarm-agent-verify",
|
||||
"swarm-authelia-bridge",
|
||||
"swarm-builder",
|
||||
"swarm-authelia-bridge-sock",
|
||||
"swarm-controller",
|
||||
"swarm-matrix-client",
|
||||
|
|
|
|||
|
|
@ -89,6 +89,8 @@ declarations.
|
|||
- **How will a PR get checked against its repo's own yes/no rules before
|
||||
a full review?** → [`swarm/review-classifier.md`](swarm/review-classifier.md)
|
||||
(planned, not implemented yet).
|
||||
- **Does an agent config PR build before it's merged?** →
|
||||
[`swarm/builder.md`](swarm/builder.md).
|
||||
|
||||
## Scheduler, CI, observability
|
||||
|
||||
|
|
|
|||
81
docs/swarm/builder.md
Normal file
81
docs/swarm/builder.md
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# Swarm builder
|
||||
|
||||
The swarm builder checks that an agent config PR builds before an operator
|
||||
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
|
||||
push, the builder builds the PR's head on the agent base, and the result shows as a
|
||||
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
||||
swarm UI's Jobs page.
|
||||
|
||||
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
||||
|
||||
## What happens on a config PR
|
||||
|
||||
1. The forge delivers the `pull_request` event on the `config-pr` webhook
|
||||
swarm-controller already registers on `agent-configs`. An `opened`,
|
||||
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
||||
node carrying the agent, the head commit and the PR number. An edit, a
|
||||
label or a close builds nothing.
|
||||
2. The node sets the status to `pending`, then sends the build as a request on
|
||||
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
||||
reply.
|
||||
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
||||
with every input overridden:
|
||||
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
||||
deployed with, the same two a hive on that host renders into its meta
|
||||
flake (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`).
|
||||
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
|
||||
A build has one hour.
|
||||
4. The reply is the system closure's store path, or the last lines `nix`
|
||||
printed. The node writes the status on the head commit:
|
||||
|
||||
| Outcome | Status | Description | Node |
|
||||
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
||||
| built | `success` | the store path | done |
|
||||
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
||||
| no builder listening, or no reply in time | `error` | why | failed |
|
||||
|
||||
The full tail is in the failed node's error on the Jobs page and in the
|
||||
builder's journal line for the build.
|
||||
|
||||
## What the check is, and isn't
|
||||
|
||||
- It composes the config the way a hive's meta flake does
|
||||
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
|
||||
what a hive adds: its CA bundle, docs source, `claude-code` path, ports and
|
||||
OTEL settings, and the `follows` that point the config's own inputs at the
|
||||
hive's. A green check can therefore still fail on a hive, and the reverse.
|
||||
- The agent base is the builder's host's, not the hive's. Hives on another
|
||||
hyperhive or nixpkgs pin build something else.
|
||||
- The builder pushes nothing and records nothing. The closure stays in the builder
|
||||
host's store until it's collected, so on a host that also runs the hive, the
|
||||
hive's own build can reuse whatever matches.
|
||||
- It doesn't gate anything. A merge deploys whatever the status says.
|
||||
- A lost webhook delivery isn't retried: the head isn't built until the next
|
||||
push. The jobs graph is in memory, so a controller restart drops a build in
|
||||
flight and leaves its status `pending`.
|
||||
|
||||
## Credentials
|
||||
|
||||
| What | Where | Who reads it |
|
||||
| ----------------------- | ------------------------------------------------- | ---------------------------------------- |
|
||||
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
||||
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start |
|
||||
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
||||
|
||||
The token belongs to the `builder` forge account, which swarm-controller
|
||||
creates, puts in the `hives` team of `agent-configs` (read on every config
|
||||
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
|
||||
gets the token through a credential helper reading the child's environment:
|
||||
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
||||
status. The builder holds no write credential; swarm-controller writes the
|
||||
status with its own forge token. Details per path:
|
||||
[`credentials.md`](credentials.md).
|
||||
|
||||
## Isolation
|
||||
|
||||
The container has its own network namespace on the bridge, as the CI runner
|
||||
does, so a config's evaluation can't reach a host-loopback service. Evaluation
|
||||
is pure and runs in the container. Builds go to the host's nix daemon as an
|
||||
untrusted client: a config can set no daemon option, and its derivations build
|
||||
in the host's sandbox. A fixed-output derivation still has the network, as it
|
||||
does for any build on that host.
|
||||
|
|
@ -55,28 +55,29 @@ of the cell says how.
|
|||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
|
||||
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
|
||||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
|
||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
|
||||
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
|
||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
||||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
||||
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
|
||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
|
||||
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
|
||||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
|
||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
|
||||
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
|
||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
||||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
|
||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
||||
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
||||
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
|
||||
# Swarm secrets: what exists, and where each one lives
|
||||
|
||||
A swarm generates its credentials in three different places and reads them
|
||||
|
|
@ -400,9 +402,10 @@ next controller restart.
|
|||
### Per-principal identities
|
||||
|
||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||
identity and sharing one merges the identities. Each of these seven leaves has
|
||||
identity and sharing one merges the identities. Each of these eight leaves has
|
||||
a role and a policy of its own, and reaches one path, except the review
|
||||
classifier's, the forge host's and Grafana's, which reach two each:
|
||||
classifier's, the builder's, the forge host's and Grafana's, which reach two
|
||||
each:
|
||||
|
||||
| unit | option pair under `deploy.bao.` | reads, or writes |
|
||||
| --------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
|
@ -418,11 +421,11 @@ The first exists **per hive**, because the path it reads carries a hive name
|
|||
and every hive runs its own reader. Its subject is
|
||||
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
||||
composed spelling as a hive name, so nobody can name a hive into another hive's
|
||||
role. The other six reach a path that names a swarm service rather than a hive,
|
||||
role. The other seven reach a path that names a swarm service rather than a hive,
|
||||
so one role each is enough and their subjects are the flat
|
||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||
`deploy.bao.reviewClassifierCommonName` and
|
||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName` and
|
||||
`deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one
|
||||
leaf two units share: the forge's host holds one store identity for the token it
|
||||
publishes and the OIDC client secret it reads.
|
||||
|
|
|
|||
|
|
@ -241,6 +241,26 @@ On a host without the store, that leaf is the one file to copy. Registration,
|
|||
the container and the threat model are in
|
||||
[`../scheduler/ci.md`](../scheduler/ci.md).
|
||||
|
||||
### Builder
|
||||
|
||||
The swarm builder, in a `swarm-builder` container on the host that sets
|
||||
`deploy.builder.enable`. `allSwarmServices` defaults it on wherever the host
|
||||
has the builder's store identity, on the same terms as the review classifier.
|
||||
Set `deploy.builder.enable` directly to override either way.
|
||||
|
||||
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
|
||||
swarm-controller queues a `build_agent` job. The builder builds that config at
|
||||
the PR's head on the agent base its own host runs, through the
|
||||
host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
|
||||
commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
||||
status blocks nothing: the merge still deploys.
|
||||
|
||||
It needs neither the forge nor a hive on its host. It reaches the queue, the
|
||||
secret store and the forge by name, under its own `builder` leaf, and fetches
|
||||
the private config repo with the read-only token of its own `builder` forge
|
||||
account. On a host without the store, that leaf is the one file to copy, as
|
||||
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
|
||||
|
||||
### Review classifier
|
||||
|
||||
The review classifier's try-out service, in a `swarm-review-classifier`
|
||||
|
|
|
|||
|
|
@ -132,6 +132,10 @@ checkpoints**, not about sandboxing the agent from its own tools:
|
|||
[`config-changes.md`](../agent-lifecycle/config-changes.md)), and
|
||||
scheduled prompts are operator-only, set at swarm level
|
||||
(see [`schedules.md`](../swarm/schedules.md)).
|
||||
Before that merge, the swarm builder evaluates an unmerged config PR
|
||||
off-hive, on any host that sets `deploy.builder.enable`: it builds the
|
||||
PR's head with a read-only forge token, in its own netns, as an untrusted
|
||||
client of its host's nix daemon (see [`builder.md`](../swarm/builder.md)).
|
||||
|
||||
### Capability = accepted risk
|
||||
|
||||
|
|
|
|||
|
|
@ -166,6 +166,9 @@
|
|||
services.hyperhive.deploy.review-classifier.package =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-review-classifier;
|
||||
services.hyperhive.deploy.builder.package =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-builder;
|
||||
services.hyperhive.deploy.authelia.bridgePackage =
|
||||
lib.mkDefault
|
||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-authelia-bridge;
|
||||
|
|
|
|||
26
nix/builder/check-flake/flake.nix
Normal file
26
nix/builder/check-flake/flake.nix
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# The swarm builder's build check: one agent config on hyperhive's agent base,
|
||||
# composed the way a hive's meta flake composes it, minus the values a hive
|
||||
# adds (its CA, docs source, claude-code path and ports).
|
||||
#
|
||||
# Every input is replaced at build time with `--override-input`
|
||||
# (swarm-builder/src/nix_build.rs), so the URLs below are never fetched and
|
||||
# the flake has no lock.
|
||||
{
|
||||
description = "swarm-builder: an agent config built on agent-base";
|
||||
|
||||
inputs = {
|
||||
nixpkgs.url = "flake:nixpkgs";
|
||||
hyperhive.url = "flake:hyperhive";
|
||||
hyperhive.inputs.nixpkgs.follows = "nixpkgs";
|
||||
agent.url = "flake:agent";
|
||||
};
|
||||
|
||||
outputs =
|
||||
{ hyperhive, agent, ... }:
|
||||
{
|
||||
packages.x86_64-linux.check =
|
||||
(hyperhive.nixosConfigurations.agent-base.extendModules {
|
||||
modules = [ agent.nixosModules.default ];
|
||||
}).config.system.build.toplevel;
|
||||
};
|
||||
}
|
||||
|
|
@ -44,6 +44,10 @@ in
|
|||
inherit (pkgs) lib;
|
||||
shared = moduleEvalShared;
|
||||
};
|
||||
module-eval-builder = import ./module-eval/builder.nix {
|
||||
inherit pkgs self nixosSystem;
|
||||
inherit (pkgs) lib;
|
||||
};
|
||||
module-eval-nats-authelia = import ./module-eval/nats-authelia.nix {
|
||||
inherit pkgs self nixosSystem;
|
||||
inherit (pkgs) lib;
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@
|
|||
./glue-nats-oidc-client.nix
|
||||
./glue-otel-scrape-targets.nix
|
||||
./glue-queue-agent-credential.nix
|
||||
./glue-builder-oidc-client.nix
|
||||
./glue-review-classifier-oidc-client.nix
|
||||
./glue-secret-publisher-authelia.nix
|
||||
./glue-secret-publisher-bao-identity.nix
|
||||
|
|
@ -53,6 +54,7 @@
|
|||
./swarm-authelia.nix
|
||||
./swarm-bao-service.nix
|
||||
./swarm-bao.nix
|
||||
./swarm-builder.nix
|
||||
./swarm-ci-runner.nix
|
||||
./swarm-review-classifier.nix
|
||||
./swarm-secret-publisher.nix
|
||||
|
|
|
|||
|
|
@ -679,6 +679,24 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
builder.enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
example = true;
|
||||
description = ''
|
||||
Run the swarm builder in a `swarm-builder` container on this host. It
|
||||
builds each agent config PR's head on this host's agent base, through
|
||||
this host's nix daemon, and swarm-controller writes the result as a
|
||||
`swarm-builder` commit status on the PR.
|
||||
|
||||
On by default where
|
||||
{option}`services.hyperhive.deploy.allSwarmServices` is set and this
|
||||
host has the builder's store identity (the pair
|
||||
{option}`services.hyperhive.deploy.bao.builderClientCertFile` and
|
||||
`builderClientKeyFile`, minted on the store's host).
|
||||
'';
|
||||
};
|
||||
|
||||
hive-controller.enable = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
|
|
|
|||
|
|
@ -115,6 +115,8 @@ in
|
|||
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
|
||||
reviewClassifierClientCertFile = lib.mkDefault "${pkiDir}/review-classifier.pem";
|
||||
reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem";
|
||||
builderClientCertFile = lib.mkDefault "${pkiDir}/builder.pem";
|
||||
builderClientKeyFile = lib.mkDefault "${pkiDir}/builder-key.pem";
|
||||
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
|
||||
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
|
||||
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
|
||||
|
|
@ -290,6 +292,13 @@ in
|
|||
[ -s ${pkiDir}/review-classifier.pem ] || ${signLeaf} ${pkiDir} review-classifier \
|
||||
${lib.escapeShellArg deployCfg.bao.reviewClassifierCommonName} "" clientAuth
|
||||
|
||||
# The swarm builder's, which reads its forge token and queue client
|
||||
# secret. Minted here for the matrix-ctl leaf's reason: the builder is a
|
||||
# swarm service, so on a builder host without the store this is the
|
||||
# file an operator copies.
|
||||
[ -s ${pkiDir}/builder.pem ] || ${signLeaf} ${pkiDir} builder \
|
||||
${lib.escapeShellArg deployCfg.bao.builderCommonName} "" clientAuth
|
||||
|
||||
# The forge's token publisher. Minted here for the matrix-ctl leaf's
|
||||
# reason: the forge is a swarm singleton, so on a forge host without the
|
||||
# store this is the file an operator copies.
|
||||
|
|
|
|||
34
nix/host-modules/glue-builder-oidc-client.nix
Normal file
34
nix/host-modules/glue-builder-oidc-client.nix
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# Glue: register the swarm builder as an OIDC client wherever authelia runs.
|
||||
#
|
||||
# ONE PAIRING PER FILE — swarm-builder ← authelia, and nothing else. Deleting
|
||||
# this leaves a builder whose token requests authelia answers with a 401, and
|
||||
# no minted secret for swarm-secret-publish to put in the store.
|
||||
#
|
||||
# ⚠️ Gated on authelia being HERE, and NOT on this host running the builder,
|
||||
# for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS
|
||||
# host's provider config. Registered whether or not any host enables the
|
||||
# builder: nothing in `swarm.*` records that, so a swarm without one carries
|
||||
# one unused client and one unused minted secret.
|
||||
{
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
deployCfg = hyperhiveCfg.deploy;
|
||||
clientId = hyperhiveCfg.swarm.builder.clientId;
|
||||
in
|
||||
{
|
||||
config = lib.mkIf deployCfg.authelia.enable {
|
||||
services.hyperhive.swarm.authelia.oidc.clients = [
|
||||
{
|
||||
id = clientId;
|
||||
description = "HyperHive swarm builder";
|
||||
# `client_credentials`: a daemon authenticating as itself.
|
||||
kind = "machine";
|
||||
audience = [ clientId ];
|
||||
}
|
||||
];
|
||||
};
|
||||
}
|
||||
|
|
@ -441,6 +441,10 @@ let
|
|||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
|
||||
capabilities = ["create", "read", "update"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
|
@ -492,6 +496,8 @@ let
|
|||
# The review classifier's forge token is denied too: the controller posts
|
||||
# the classifier's reviews with whatever token is there, so a write to it
|
||||
# picks the account those reviews come from.
|
||||
# So is the builder's: a write to it picks the account the builder fetches
|
||||
# config repos as.
|
||||
secretPublisherPolicyText = ''
|
||||
path "${credentialMountPath}/data/swarm/hives/*" {
|
||||
capabilities = ["create", "update"]
|
||||
|
|
@ -521,6 +527,10 @@ let
|
|||
capabilities = ["deny"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
|
||||
capabilities = ["deny"]
|
||||
}
|
||||
|
||||
path "${credentialMountPath}/data/${controllerClientLeaf}" {
|
||||
capabilities = ["create", "update"]
|
||||
}
|
||||
|
|
@ -588,6 +598,12 @@ let
|
|||
# the UI's OIDC login; the classifier reads it on every run.
|
||||
classifierApiKeyLeaf = "swarm/services/review-classifier/anthropic-api-key";
|
||||
|
||||
# The swarm builder's forge token, the nix half of
|
||||
# `swarm_secret_client::builder::forge_token_path`. The controller mints it
|
||||
# and reads it back before deciding whether to re-mint; the builder reads it
|
||||
# to fetch config repos.
|
||||
builderForgeTokenLeaf = "swarm/services/builder/forge/token";
|
||||
|
||||
# The UI's OIDC client secret. The publisher writes it; the granter reads it
|
||||
# into the `oidc` auth method's config, which is the only copy bao uses.
|
||||
baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client";
|
||||
|
|
@ -998,6 +1014,18 @@ let
|
|||
}
|
||||
];
|
||||
|
||||
# The swarm builder: its forge token, read once per build, and its own queue
|
||||
# client secret, read at start. Both by the daemon inside its container.
|
||||
builderReaders = [
|
||||
{
|
||||
name = "swarm-builder";
|
||||
cn = baoDeploy.builderCommonName;
|
||||
policyText =
|
||||
readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}"
|
||||
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client";
|
||||
}
|
||||
];
|
||||
|
||||
# The forge host's one store identity. `swarm-bao-forge-publisher` writes
|
||||
# the controller's forge token under it, the one writer of that path and
|
||||
# write-only: it copies a file in and never reads one back.
|
||||
|
|
@ -1848,6 +1876,19 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
builderCommonName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-builder";
|
||||
description = ''
|
||||
Subject the store's `swarm-builder` cert-auth role accepts: the
|
||||
identity the swarm builder presents from inside its container. Its
|
||||
grant is two paths, read only: `swarm/services/builder/forge/token`
|
||||
and its own queue client secret.
|
||||
|
||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
||||
'';
|
||||
};
|
||||
|
||||
forgePublisherCommonName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "swarm-forge-publisher";
|
||||
|
|
@ -2123,6 +2164,31 @@ in
|
|||
'';
|
||||
};
|
||||
|
||||
builderClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/builder.pem";
|
||||
description = ''
|
||||
Certificate the swarm builder presents to the store. Its subject must
|
||||
be {option}`services.hyperhive.deploy.bao.builderCommonName`.
|
||||
|
||||
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
|
||||
On a builder host without the store, it is a file the operator copies
|
||||
from the store's host. A path, never a value.
|
||||
'';
|
||||
};
|
||||
|
||||
builderClientKeyFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
example = "/var/lib/swarm-bao-pki/builder-key.pem";
|
||||
description = ''
|
||||
Private key for
|
||||
{option}`services.hyperhive.deploy.bao.builderClientCertFile`.
|
||||
A path, never a value.
|
||||
'';
|
||||
};
|
||||
|
||||
forgePublisherClientCertFile = lib.mkOption {
|
||||
type = lib.types.nullOr lib.types.str;
|
||||
default = null;
|
||||
|
|
@ -3201,6 +3267,8 @@ in
|
|||
|
||||
systemd.services.swarm-bao-review-classifier-policy = readerPolicyUnit "write the review classifier's API-key-reader bao policy and cert-auth role" reviewClassifierReaders;
|
||||
|
||||
systemd.services.swarm-bao-builder-policy = readerPolicyUnit "write the swarm builder's forge-token-reader bao policy and cert-auth role" builderReaders;
|
||||
|
||||
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
|
||||
|
||||
# A FOURTH sibling, same shape and same reasons as the two above. This
|
||||
|
|
|
|||
231
nix/host-modules/swarm-builder.nix
Normal file
231
nix/host-modules/swarm-builder.nix
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
# The swarm builder, in a `swarm-builder` container on the host that sets
|
||||
# `deploy.builder.enable`, which `deploy.allSwarmServices` defaults on where
|
||||
# the host has the builder's store identity. It builds agent configs for
|
||||
# swarm-controller: the controller sends a config PR's head over the queue,
|
||||
# and the daemon builds that config on this host's agent base and replies.
|
||||
# docs/swarm/builder.md has the design.
|
||||
#
|
||||
# ⚠️ Its own network namespace on the bridge, as ./swarm-ci-runner.nix has: a
|
||||
# config PR is unreviewed input to a nix evaluation, so the container reaches
|
||||
# the swarm services by name and nothing on the host's loopback.
|
||||
#
|
||||
# Builds go through the HOST nix-daemon, as the CI runner's do. The daemon
|
||||
# runs as a dynamic user, an untrusted daemon client: a config can set no
|
||||
# daemon option, and its builds run in the host's sandbox.
|
||||
#
|
||||
# The container holds one credential, its own store leaf, which nspawn hands in
|
||||
# as a system credential with no copy on disk. The daemon reads everything else
|
||||
# from the store with it: its queue client secret at start and its forge token
|
||||
# on every build.
|
||||
{
|
||||
pkgs,
|
||||
lib,
|
||||
config,
|
||||
...
|
||||
}:
|
||||
let
|
||||
hyperhiveCfg = config.services.hyperhive;
|
||||
cfg = hyperhiveCfg.deploy.builder;
|
||||
baoCfg = hyperhiveCfg.swarm.bao;
|
||||
baoDeploy = hyperhiveCfg.deploy.bao;
|
||||
natsCfg = hyperhiveCfg.swarm.nats;
|
||||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||||
forgeCfg = hyperhiveCfg.swarm.forge;
|
||||
gatewayCfg = hyperhiveCfg.gateway;
|
||||
networkCfg = hyperhiveCfg.network;
|
||||
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
|
||||
|
||||
machine = "swarm-builder";
|
||||
unit = "swarm-builder";
|
||||
clientId = hyperhiveCfg.swarm.builder.clientId;
|
||||
|
||||
# The daemon reaches authelia, the store and the queue by name over TLS, and
|
||||
# `git` reaches the forge; in self-signed mode their leaves chain to the hive
|
||||
# CA.
|
||||
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
|
||||
caBundleModule = caTrust.trustBundle {
|
||||
inherit pkgs;
|
||||
name = machine;
|
||||
consumers = [ unit ];
|
||||
};
|
||||
|
||||
haveClientIdentity =
|
||||
baoDeploy.builderClientCertFile != null && baoDeploy.builderClientKeyFile != null;
|
||||
|
||||
# Credential id → host source, as ./swarm-review-classifier.nix hands its
|
||||
# leaf in.
|
||||
storeFiles = lib.filterAttrs (_: source: source != null) {
|
||||
builder-bao-cert = baoDeploy.builderClientCertFile;
|
||||
builder-bao-key = baoDeploy.builderClientKeyFile;
|
||||
builder-bao-server-ca = baoDeploy.serverCaFile;
|
||||
};
|
||||
# nixos-container expands `extraFlags` unquoted, so a source path must be
|
||||
# one word.
|
||||
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
|
||||
credential = id: "/run/credentials/${unit}.service/${id}";
|
||||
|
||||
# The `nix` the host daemon runs, so client and daemon speak one protocol.
|
||||
nix = config.nix.package;
|
||||
|
||||
# Strings, not paths: both are already store paths, and the flake reference
|
||||
# the daemon passes must be the same one a hive's meta flake names.
|
||||
hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake;
|
||||
nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake;
|
||||
|
||||
privateNetwork = true;
|
||||
in
|
||||
{
|
||||
options.services.hyperhive.swarm.builder.clientId = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
readOnly = true;
|
||||
default = "swarm-builder";
|
||||
description = ''
|
||||
The OAuth2 client id the swarm builder presents to the swarm queue.
|
||||
Read-only: the host running authelia registers it, the queue's
|
||||
auth-callout responder grants it the build subject, and the secret
|
||||
publisher carries its secret to the store, all from this one value.
|
||||
'';
|
||||
};
|
||||
|
||||
options.services.hyperhive.deploy.builder.package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
defaultText = lib.literalExpression "hyperhive.packages.\${system}.swarm-builder";
|
||||
description = "The swarm builder daemon package.";
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
assertions = [
|
||||
{
|
||||
assertion = haveClientIdentity;
|
||||
message = ''
|
||||
services.hyperhive.deploy.builder.enable requires the builder's
|
||||
swarm-secret-store client identity: set both
|
||||
|
||||
services.hyperhive.deploy.bao.builderClientCertFile
|
||||
services.hyperhive.deploy.bao.builderClientKeyFile
|
||||
|
||||
The builder reads its queue client secret and its forge token with it.
|
||||
|
||||
Both default to the pair glue-bao-tls.nix mints when
|
||||
services.hyperhive.deploy.bao.enable is true (the host that runs the
|
||||
store). Otherwise copy builder.pem and builder-key.pem from the store
|
||||
host's /var/lib/swarm-bao-pki/ and name them in those two options.
|
||||
'';
|
||||
}
|
||||
{
|
||||
assertion = lib.all oneWord (lib.attrValues storeFiles);
|
||||
message = ''
|
||||
The swarm builder's store identity paths must not contain whitespace:
|
||||
nspawn's --load-credential receives them through nixos-container's
|
||||
unquoted extra flags.
|
||||
'';
|
||||
}
|
||||
];
|
||||
|
||||
# The container hangs off the bridge, takes its address from the gateway's
|
||||
# DHCP and resolves through its dnsmasq.
|
||||
services.hyperhive.network.enable = lib.mkDefault true;
|
||||
services.hyperhive.gateway.enable = lib.mkDefault true;
|
||||
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
|
||||
|
||||
# nspawn fails to start the container when a credential source is
|
||||
# missing, so where this host mints the leaf the container waits for it.
|
||||
systemd.services."container@${machine}" = lib.mkMerge [
|
||||
caTrust.containerOrdering
|
||||
{ after = [ "swarm-bao-pki.service" ]; }
|
||||
];
|
||||
|
||||
containers.${machine} = {
|
||||
autoStart = true;
|
||||
# The daemon keeps nothing but `git`'s fetch cache, which a restart may
|
||||
# drop.
|
||||
ephemeral = false;
|
||||
# Journal files on the host, not inside the container: nixpkgs hardcodes
|
||||
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
|
||||
extraFlags = [
|
||||
"--link-journal=host"
|
||||
]
|
||||
++ lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
|
||||
inherit privateNetwork;
|
||||
hostBridge = networkCfg.bridgeName;
|
||||
|
||||
bindMounts = {
|
||||
# The directory rather than the socket file, so the mount survives a
|
||||
# daemon restart replacing the inode.
|
||||
"/nix/var/nix/daemon-socket" = {
|
||||
hostPath = "/nix/var/nix/daemon-socket";
|
||||
isReadOnly = false;
|
||||
};
|
||||
}
|
||||
// caTrust.bindMount;
|
||||
|
||||
config =
|
||||
{ lib, ... }:
|
||||
{
|
||||
imports = [
|
||||
../container-modules/swarm-container.nix
|
||||
caBundleModule
|
||||
];
|
||||
|
||||
services.hyperhive.swarmContainer = {
|
||||
inherit privateNetwork;
|
||||
writesOwnResolvConf = false;
|
||||
};
|
||||
networking.nameservers = [ networkCfg.bridgeIp ];
|
||||
networking.interfaces.eth0.useDHCP = true;
|
||||
|
||||
# Both the service and its socket are off: the socket unit owns
|
||||
# `/nix/var/nix/daemon-socket/socket` and would race the bind mount.
|
||||
systemd.globalEnvironment.NIX_REMOTE = "daemon";
|
||||
systemd.services.nix-daemon.enable = false;
|
||||
systemd.sockets.nix-daemon.enable = false;
|
||||
|
||||
systemd.services.${unit} = {
|
||||
description = "build agent configs for swarm-controller";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
# nix fetches the config repo with `git`.
|
||||
path = [ pkgs.git ];
|
||||
environment = {
|
||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||||
BAO_CLIENT_CERT = credential "builder-bao-cert";
|
||||
BAO_CLIENT_KEY = credential "builder-bao-key";
|
||||
NIX_CONFIG = "experimental-features = nix-command flakes";
|
||||
# nix's fetch cache, under `CacheDirectory`; the dynamic user has
|
||||
# no home of its own.
|
||||
HOME = "/var/cache/${unit}";
|
||||
XDG_CACHE_HOME = "/var/cache/${unit}";
|
||||
SWARM_BUILDER_NATS_URL = "tls://${natsCfg.domain}:${toString natsCfg.port}";
|
||||
SWARM_BUILDER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
|
||||
SWARM_BUILDER_OIDC_CLIENT_ID = clientId;
|
||||
SWARM_BUILDER_NIX = "${nix}/bin/nix";
|
||||
SWARM_BUILDER_CHECK_FLAKE = "${../builder/check-flake}";
|
||||
SWARM_BUILDER_HYPERHIVE_FLAKE = hyperhiveFlake;
|
||||
SWARM_BUILDER_NIXPKGS_FLAKE = nixpkgsFlake;
|
||||
SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}";
|
||||
SWARM_BUILDER_CONFIG_ORG = "agent-configs";
|
||||
}
|
||||
// lib.optionalAttrs (storeFiles ? builder-bao-server-ca) {
|
||||
BAO_CACERT = credential "builder-bao-server-ca";
|
||||
}
|
||||
// lib.optionalAttrs caTrust.useSelfSigned {
|
||||
SWARM_BUILDER_OIDC_CA_FILE = caTrust.bundlePathFor machine;
|
||||
# `git`'s trust store, which the bundle's system CAs keep whole.
|
||||
GIT_SSL_CAINFO = caTrust.bundlePathFor machine;
|
||||
};
|
||||
serviceConfig = {
|
||||
ExecStart = "${cfg.package}/bin/swarm-builder";
|
||||
DynamicUser = true;
|
||||
CacheDirectory = unit;
|
||||
# Bare ids: the system credentials nspawn handed in.
|
||||
LoadCredential = lib.attrNames storeFiles;
|
||||
Restart = "on-failure";
|
||||
RestartSec = "10s";
|
||||
SyslogIdentifier = unit;
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
@ -683,6 +683,8 @@ in
|
|||
"--reader-client ${lib.escapeShellArg controllerCfg.queueClientId}"
|
||||
# Hears try-outs and answers each by reply, and nothing else.
|
||||
"--review-classifier-client ${lib.escapeShellArg config.services.hyperhive.swarm.review-classifier.clientId}"
|
||||
# Hears builds and answers each by reply, and nothing else.
|
||||
"--builder-client ${lib.escapeShellArg config.services.hyperhive.swarm.builder.clientId}"
|
||||
# What an agent may publish to, keyed on its own name: its
|
||||
# terminal rows and its turn-state header
|
||||
# (`hive-agent::swarm_agent_state`), two subject families so a
|
||||
|
|
|
|||
|
|
@ -117,4 +117,11 @@ in
|
|||
&& deployCfg.bao.reviewClassifierClientCertFile != null
|
||||
&& deployCfg.bao.reviewClassifierClientKeyFile != null
|
||||
);
|
||||
|
||||
# The swarm builder, defaulted on the same terms as the review classifier.
|
||||
config.services.hyperhive.deploy.builder.enable = lib.mkDefault (
|
||||
deployCfg.allSwarmServices
|
||||
&& deployCfg.bao.builderClientCertFile != null
|
||||
&& deployCfg.bao.builderClientKeyFile != null
|
||||
);
|
||||
}
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue