vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

Compare commits

base: vinzenz:aece1c26d00e937ccd249ab1af2fefaadf7e0a8c
Branches Tags
vinzenz:main
...
compare: vinzenz:6d7fc0dd9993f986e5007045fa3beae2ddd88022
Branches Tags
vinzenz:main
Author SHA1 Message Date
atlas
6d7fc0dd99 docs: builder.md says the client secret is re-read on refusal too 
The builder reads its queue client secret at start and again when
authelia refuses it. The credentials table in builder.md and the
swarm-builder README still said it was read only at start.
 2026-10-14 03:30:46 +02:00
atlas
47d3ccfede module-eval: bao-grants counts the eighteen granting units 2026-10-14 03:30:46 +02:00
atlas
cafa120d77 swarm-builder: scope the forge token to the forge, one build at a time, opt-in 
argus's review on #5177: the git credential helper answered every `get`,
so a config PR's own flake inputs could have git send the builder's forge
token to any host. The helper is now configured under
`credential.<forge-url>.helper` and itself answers only an https request
whose host is the forge's; a test runs it as git does and checks that
other hosts, http and non-get operations get nothing.

Builds are bounded: every `build_agent` node needs the one `builder` jobq
resource, so one runs at a time and the rest wait pending, and a newer
head cancels its PR's pending build (a running one finishes and writes on
its own commit). The builder also holds a one-permit semaphore around each
build.

`builder` and `review-classifier`, the forge logins swarm-controller owns,
join nix/reserved-names.nix, with a test tying the list to both constants.

The builder is now opt-in like the CI runner: `allSwarmServices` no longer
turns it on, since it runs unreviewed config PRs through the host's nix
daemon. mara had not answered atlas's question on the PR.
 2026-10-14 03:30:46 +02:00
atlas
b68b74d687 swarm-builder: build each agent config PR head and post a commit status 
A new `swarm-builder` service, in its own container on the host that sets
`deploy.builder.enable` (defaulted on by `allSwarmServices` where the host
has the builder's store leaf, like the review classifier).

When an `agent-configs/<agent>` PR is opened, reopened or synchronized,
swarm-controller queues a `build_agent` jobq node. The node sets a
`swarm-builder` commit status to pending on the PR head, sends
`{agent, rev, pr}` to the builder over NATS request-reply
(`$SWARM.builder.build`), and writes success (store path), failure (last
line nix printed) or error (no builder / no reply) on the head. The node
fails when the build does, so the Jobs page matches the PR.

The builder builds nix/builder/check-flake's `check` with every input
overridden: hyperhive and nixpkgs to the host's own
`c0re.hyperhiveFlake`/`c0re.nixpkgsFlake` (what a hive there renders into
its meta flake), and the agent to the PR head on the forge. Evaluation is
pure and runs in the container; builds go to the host nix daemon as an
untrusted client. The private config repo is fetched with the token of a
new read-only `builder` forge account (in the `hives` team), which the
controller mints into `swarm/services/builder/forge/token`; git gets it
from a credential helper reading the child's environment, so it is in no
argument or URL.

Credentials: a `builder` store leaf minted by glue-bao-tls on the store
host, a `swarm-builder` reader policy (forge token + its own OIDC client
secret), an authelia machine client, and a `--builder-client` grant in the
queue's auth-callout responder. The controller gains a grant on the token
path and the secret publisher a deny on it.

This is the agent-config build check atlas scoped on #5081 (slice 3):
nothing is pushed to a cache, nothing is written to KV, and a red status
gates no deploy. The check composes agent-base + the config without the
values a hive adds (CA, docs, claude-code path, ports, input follows), so
it can disagree with a hive's own build.
 2026-10-14 03:30:46 +02:00
42 changed files with 2757 additions and 56 deletions
Download patch file Download diff file Expand all files Collapse all files

7
CLAUDE.md
View file

@ -202,6 +202,13 @@ hand-maintained per-file tree drifts out of sync with the code.
for agents: nginx sends it `/select/logsql/` requests carrying an agent for agents: nginx sends it `/select/logsql/` requests carrying an agent
token, it answers 200 or 401 via `swarm-agent-verify`. Runs on the log token, it answers 200 or 401 via `swarm-agent-verify`. Runs on the log
store's host (`swarm-victorialogs.nix`). store's host (`swarm-victorialogs.nix`).
- **`swarm-builder/`** — the swarm builder's build daemon, in its own
container where `deploy.builder.enable` is set. Takes an agent config PR's
head from `swarm-controller` over the queue, builds it on its host's agent
base through the host nix daemon (`nix/builder/check-flake`), and replies
with the store path or the error tail; the controller writes the commit
status. Holds a read-only forge token. Design:
[`docs/swarm/builder.md`](docs/swarm/builder.md).
- **`swarmctl/`** — swarm-level operator CLI, installed by the - **`swarmctl/`** — swarm-level operator CLI, installed by the
swarm-controller module on the host that runs the daemon. Runs as swarm-controller module on the host that runs the daemon. Runs as
**root and acts directly** — it serves no socket, publishes no HTTP **root and acts directly** — it serves no socket, publishes no HTTP

15
Cargo.lock generated
View file

@ -4829,6 +4829,21 @@ dependencies = [
"serde_json", "serde_json",
] ]
[[package]]
name = "swarm-builder"
version = "0.1.0"
dependencies = [
"anyhow",
"async-nats",
"futures-util",
"hive-log",
"serde_json",
"swarm-queue-client",
"swarm-secret-client",
"tokio",
"tracing",
]
[[package]] [[package]]
name = "swarm-controller" name = "swarm-controller"
version = "0.1.0" version = "0.1.0"

1
Cargo.toml
View file

@ -27,6 +27,7 @@ members = [
"hivectl", "hivectl",
"swarm-agent-verify", "swarm-agent-verify",
"swarm-authelia-bridge", "swarm-authelia-bridge",
"swarm-builder",
"swarm-authelia-bridge-sock", "swarm-authelia-bridge-sock",
"swarm-controller", "swarm-controller",
"swarm-matrix-client", "swarm-matrix-client",

2
docs/README.md
View file

@ -89,6 +89,8 @@ declarations.
- **How will a PR get checked against its repo's own yes/no rules before - **How will a PR get checked against its repo's own yes/no rules before
a full review?** → [`swarm/review-classifier.md`](swarm/review-classifier.md) a full review?** → [`swarm/review-classifier.md`](swarm/review-classifier.md)
(planned, not implemented yet). (planned, not implemented yet).
- **Does an agent config PR build before it's merged?** →
[`swarm/builder.md`](swarm/builder.md).
## Scheduler, CI, observability ## Scheduler, CI, observability

86
docs/swarm/builder.md Normal file
View file

@ -0,0 +1,86 @@
# Swarm builder
The swarm builder checks that an agent config PR builds before an operator
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
push, the builder builds the PR's head on the agent base, and the result shows as a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
Where it runs and how it's enabled: [`services.md`](services.md#builder).
## What happens on a config PR
1. The forge delivers the `pull_request` event on the `config-pr` webhook
swarm-controller already registers on `agent-configs`. An `opened`,
`reopened` or `synchronized` action on an open PR queues one `build_agent`
node carrying the agent, the head commit and the PR number. An edit, a
label or a close builds nothing. Every `build_agent` node needs the one
`builder` slot, so builds run one at a time and the rest wait pending. A
newer head cancels its PR's pending build; a build already running
finishes and writes its status on its own, older commit.
2. The node sets the status to `pending`, then sends the build as a request on
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
reply.
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
with every input overridden:
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
deployed with, the same two a hive on that host renders into its meta
flake (`c0re.hyperhiveFlake`, `c0re.nixpkgsFlake`).
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
A build has one hour.
4. The reply is the system closure's store path, or the last lines `nix`
printed. The node writes the status on the head commit:
| Outcome | Status | Description | Node |
| ----------------------------------------- | --------- | --------------------------- | ------ |
| built | `success` | the store path | done |
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
| no builder listening, or no reply in time | `error` | why | failed |
The full tail is in the failed node's error on the Jobs page and in the
builder's journal line for the build.
## What the check is, and isn't
- It composes the config the way a hive's meta flake does
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
what a hive adds: its CA bundle, docs source, `claude-code` path, ports and
OTEL settings, and the `follows` that point the config's own inputs at the
hive's. A green check can therefore still fail on a hive, and the reverse.
- The agent base is the builder's host's, not the hive's. Hives on another
hyperhive or nixpkgs pin build something else.
- The builder pushes nothing and records nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches.
- It doesn't gate anything. A merge deploys whatever the status says.
- A lost webhook delivery isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
## Credentials
| What | Where | Who reads it |
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers
the helper for the forge's URL only, and the helper answers only an `https`
request for the forge's host: any other host gets nothing. The builder holds no write credential; swarm-controller writes the
status with its own forge token. Details per path:
[`credentials.md`](credentials.md).
## Isolation
The container has its own network namespace on the bridge, as the CI runner
does, so a config's evaluation can't reach a host-loopback service. Evaluation
is pure and runs in the container. Builds go to the host's nix daemon as an
untrusted client: a config can set no daemon option, and its derivations build
in the host's sandbox. A fixed-output derivation still has the network, as it
does for any build on that host.

45
docs/swarm/credentials.md
View file

@ -55,28 +55,29 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | | `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

11
docs/swarm/secrets.md
View file

@ -1,3 +1,5 @@
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
# Swarm secrets: what exists, and where each one lives # Swarm secrets: what exists, and where each one lives
A swarm generates its credentials in three different places and reads them A swarm generates its credentials in three different places and reads them
@ -400,9 +402,10 @@ next controller restart.
### Per-principal identities ### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these seven leaves has identity and sharing one merges the identities. Each of these eight leaves has
a role and a policy of its own, and reaches one path, except the review a role and a policy of its own, and reaches one path, except the review
classifier's, the forge host's and Grafana's, which reach two each: classifier's, the builder's, the forge host's and Grafana's, which reach two
each:
| unit | option pair under `deploy.bao.` | reads, or writes | | unit | option pair under `deploy.bao.` | reads, or writes |
| --------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | --------------------------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
@ -418,11 +421,11 @@ The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is and every hive runs its own reader. Its subject is
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that `<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
composed spelling as a hive name, so nobody can name a hive into another hive's composed spelling as a hive name, so nobody can name a hive into another hive's
role. The other six reach a path that names a swarm service rather than a hive, role. The other seven reach a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`, `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`, `deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
`deploy.bao.reviewClassifierCommonName` and `deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName` and
`deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one `deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one
leaf two units share: the forge's host holds one store identity for the token it leaf two units share: the forge's host holds one store identity for the token it
publishes and the OIDC client secret it reads. publishes and the OIDC client secret it reads.

21
docs/swarm/services.md
View file

@ -241,6 +241,27 @@ On a host without the store, that leaf is the one file to copy. Registration,
the container and the threat model are in the container and the threat model are in
[`../scheduler/ci.md`](../scheduler/ci.md). [`../scheduler/ci.md`](../scheduler/ci.md).
### Builder
The swarm builder, in a `swarm-builder` container on the host that sets
`deploy.builder.enable`. It doesn't derive from `allSwarmServices`: it
evaluates and builds unreviewed config PRs through its host's nix daemon, so
running one is a decision of its own, as for the CI runner.
When an `agent-configs/<agent>` PR opens, reopens or gets a new push,
swarm-controller queues a `build_agent` job. Jobs run one at a time, and a
newer push replaces its PR's job while that job is still waiting. The builder
builds that config at the PR's head on the agent base its own host runs,
through the host's nix daemon, and swarm-controller writes the result as a `swarm-builder`
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge
account. On a host without the store, that leaf is the one file to copy, as
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
### Review classifier ### Review classifier
The review classifier's try-out service, in a `swarm-review-classifier` The review classifier's try-out service, in a `swarm-review-classifier`

4
docs/trust-boundary/security.md
View file

@ -132,6 +132,10 @@ checkpoints**, not about sandboxing the agent from its own tools:
[`config-changes.md`](../agent-lifecycle/config-changes.md)), and [`config-changes.md`](../agent-lifecycle/config-changes.md)), and
scheduled prompts are operator-only, set at swarm level scheduled prompts are operator-only, set at swarm level
(see [`schedules.md`](../swarm/schedules.md)). (see [`schedules.md`](../swarm/schedules.md)).
Before that merge, the swarm builder evaluates an unmerged config PR
off-hive, on any host that sets `deploy.builder.enable`: it builds the
PR's head with a read-only forge token, in its own netns, as an untrusted
client of its host's nix daemon (see [`builder.md`](../swarm/builder.md)).
### Capability = accepted risk ### Capability = accepted risk

3
flake.nix
View file

@ -166,6 +166,9 @@
services.hyperhive.deploy.review-classifier.package = services.hyperhive.deploy.review-classifier.package =
lib.mkDefault lib.mkDefault
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-review-classifier; self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-review-classifier;
services.hyperhive.deploy.builder.package =
lib.mkDefault
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-builder;
services.hyperhive.deploy.authelia.bridgePackage = services.hyperhive.deploy.authelia.bridgePackage =
lib.mkDefault lib.mkDefault
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-authelia-bridge; self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-authelia-bridge;

26
nix/builder/check-flake/flake.nix Normal file
View file

@ -0,0 +1,26 @@
# The swarm builder's build check: one agent config on hyperhive's agent base,
# composed the way a hive's meta flake composes it, minus the values a hive
# adds (its CA, docs source, claude-code path and ports).
#
# Every input is replaced at build time with `--override-input`
# (swarm-builder/src/nix_build.rs), so the URLs below are never fetched and
# the flake has no lock.
{
description = "swarm-builder: an agent config built on agent-base";
inputs = {
nixpkgs.url = "flake:nixpkgs";
hyperhive.url = "flake:hyperhive";
hyperhive.inputs.nixpkgs.follows = "nixpkgs";
agent.url = "flake:agent";
};
outputs =
{ hyperhive, agent, ... }:
{
packages.x86_64-linux.check =
(hyperhive.nixosConfigurations.agent-base.extendModules {
modules = [ agent.nixosModules.default ];
}).config.system.build.toplevel;
};
}

4
nix/checks.nix
View file

@ -44,6 +44,10 @@ in
inherit (pkgs) lib; inherit (pkgs) lib;
shared = moduleEvalShared; shared = moduleEvalShared;
}; };
module-eval-builder = import ./module-eval/builder.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-nats-authelia = import ./module-eval/nats-authelia.nix { module-eval-nats-authelia = import ./module-eval/nats-authelia.nix {
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;

2
nix/host-modules/default.nix
View file

@ -44,6 +44,7 @@
./glue-nats-oidc-client.nix ./glue-nats-oidc-client.nix
./glue-otel-scrape-targets.nix ./glue-otel-scrape-targets.nix
./glue-queue-agent-credential.nix ./glue-queue-agent-credential.nix
./glue-builder-oidc-client.nix
./glue-review-classifier-oidc-client.nix ./glue-review-classifier-oidc-client.nix
./glue-secret-publisher-authelia.nix ./glue-secret-publisher-authelia.nix
./glue-secret-publisher-bao-identity.nix ./glue-secret-publisher-bao-identity.nix
@ -53,6 +54,7 @@
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao-service.nix ./swarm-bao-service.nix
./swarm-bao.nix ./swarm-bao.nix
./swarm-builder.nix
./swarm-ci-runner.nix ./swarm-ci-runner.nix
./swarm-review-classifier.nix ./swarm-review-classifier.nix
./swarm-secret-publisher.nix ./swarm-secret-publisher.nix

20
nix/host-modules/deploy.nix
View file

@ -679,6 +679,26 @@ in
''; '';
}; };
builder.enable = lib.mkOption {
type = lib.types.bool;
default = false;
example = true;
description = ''
Run the swarm builder in a `swarm-builder` container on this host. It
builds each agent config PR's head on this host's agent base, through
this host's nix daemon, and swarm-controller writes the result as a
`swarm-builder` commit status on the PR.
Not derived from
{option}`services.hyperhive.deploy.allSwarmServices`: the builder
evaluates and builds unreviewed config PRs on this host's nix daemon,
so running one is a decision of its own, as for the CI runner. It needs
the builder's store identity
({option}`services.hyperhive.deploy.bao.builderClientCertFile` and
`builderClientKeyFile`), which the store's host mints.
'';
};
hive-controller.enable = lib.mkOption { hive-controller.enable = lib.mkOption {
type = lib.types.bool; type = lib.types.bool;
default = false; default = false;

9
nix/host-modules/glue-bao-tls.nix
View file

@ -115,6 +115,8 @@ in
ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem"; ciRunnerClientKeyFile = lib.mkDefault "${pkiDir}/ci-runner-key.pem";
reviewClassifierClientCertFile = lib.mkDefault "${pkiDir}/review-classifier.pem"; reviewClassifierClientCertFile = lib.mkDefault "${pkiDir}/review-classifier.pem";
reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem"; reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem";
builderClientCertFile = lib.mkDefault "${pkiDir}/builder.pem";
builderClientKeyFile = lib.mkDefault "${pkiDir}/builder-key.pem";
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem"; forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem"; forgePublisherClientKeyFile = lib.mkDefault "${pkiDir}/forge-publisher-key.pem";
servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem"; servicesIssuerClientCertFile = lib.mkDefault "${pkiDir}/services-issuer.pem";
@ -290,6 +292,13 @@ in
[ -s ${pkiDir}/review-classifier.pem ] || ${signLeaf} ${pkiDir} review-classifier \ [ -s ${pkiDir}/review-classifier.pem ] || ${signLeaf} ${pkiDir} review-classifier \
${lib.escapeShellArg deployCfg.bao.reviewClassifierCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.reviewClassifierCommonName} "" clientAuth
# The swarm builder's, which reads its forge token and queue client
# secret. Minted here for the matrix-ctl leaf's reason: the builder is a
# swarm service, so on a builder host without the store this is the
# file an operator copies.
[ -s ${pkiDir}/builder.pem ] || ${signLeaf} ${pkiDir} builder \
${lib.escapeShellArg deployCfg.bao.builderCommonName} "" clientAuth
# The forge's token publisher. Minted here for the matrix-ctl leaf's # The forge's token publisher. Minted here for the matrix-ctl leaf's
# reason: the forge is a swarm singleton, so on a forge host without the # reason: the forge is a swarm singleton, so on a forge host without the
# store this is the file an operator copies. # store this is the file an operator copies.

34
nix/host-modules/glue-builder-oidc-client.nix Normal file
View file

@ -0,0 +1,34 @@
# Glue: register the swarm builder as an OIDC client wherever authelia runs.
#
# ONE PAIRING PER FILE — swarm-builder ← authelia, and nothing else. Deleting
# this leaves a builder whose token requests authelia answers with a 401, and
# no minted secret for swarm-secret-publish to put in the store.
#
# ⚠️ Gated on authelia being HERE, and NOT on this host running the builder,
# for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS
# host's provider config. Registered whether or not any host enables the
# builder: nothing in `swarm.*` records that, so a swarm without one carries
# one unused client and one unused minted secret.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
clientId = hyperhiveCfg.swarm.builder.clientId;
in
{
config = lib.mkIf deployCfg.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = clientId;
description = "HyperHive swarm builder";
# `client_credentials`: a daemon authenticating as itself.
kind = "machine";
audience = [ clientId ];
}
];
};
}

68
nix/host-modules/swarm-bao.nix
View file

@ -441,6 +441,10 @@ let
capabilities = ["create", "read", "update"] capabilities = ["create", "read", "update"]
} }
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
capabilities = ["create", "read", "update"]
}
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" { path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
capabilities = ["read"] capabilities = ["read"]
} }
@ -492,6 +496,8 @@ let
# The review classifier's forge token is denied too: the controller posts # The review classifier's forge token is denied too: the controller posts
# the classifier's reviews with whatever token is there, so a write to it # the classifier's reviews with whatever token is there, so a write to it
# picks the account those reviews come from. # picks the account those reviews come from.
# So is the builder's: a write to it picks the account the builder fetches
# config repos as.
secretPublisherPolicyText = '' secretPublisherPolicyText = ''
path "${credentialMountPath}/data/swarm/hives/*" { path "${credentialMountPath}/data/swarm/hives/*" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
@ -521,6 +527,10 @@ let
capabilities = ["deny"] capabilities = ["deny"]
} }
path "${credentialMountPath}/data/${builderForgeTokenLeaf}" {
capabilities = ["deny"]
}
path "${credentialMountPath}/data/${controllerClientLeaf}" { path "${credentialMountPath}/data/${controllerClientLeaf}" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
@ -588,6 +598,12 @@ let
# the UI's OIDC login; the classifier reads it on every run. # the UI's OIDC login; the classifier reads it on every run.
classifierApiKeyLeaf = "swarm/services/review-classifier/anthropic-api-key"; classifierApiKeyLeaf = "swarm/services/review-classifier/anthropic-api-key";
# The swarm builder's forge token, the nix half of
# `swarm_secret_client::builder::forge_token_path`. The controller mints it
# and reads it back before deciding whether to re-mint; the builder reads it
# to fetch config repos.
builderForgeTokenLeaf = "swarm/services/builder/forge/token";
# The UI's OIDC client secret. The publisher writes it; the granter reads it # The UI's OIDC client secret. The publisher writes it; the granter reads it
# into the `oidc` auth method's config, which is the only copy bao uses. # into the `oidc` auth method's config, which is the only copy bao uses.
baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client"; baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client";
@ -998,6 +1014,18 @@ let
} }
]; ];
# The swarm builder: its forge token, read once per build, and its own queue
# client secret, read at start. Both by the daemon inside its container.
builderReaders = [
{
name = "swarm-builder";
cn = baoDeploy.builderCommonName;
policyText =
readStanza "${credentialMountPath}/data/${builderForgeTokenLeaf}"
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.builder.clientId}/oidc/client";
}
];
# The forge host's one store identity. `swarm-bao-forge-publisher` writes # The forge host's one store identity. `swarm-bao-forge-publisher` writes
# the controller's forge token under it, the one writer of that path and # the controller's forge token under it, the one writer of that path and
# write-only: it copies a file in and never reads one back. # write-only: it copies a file in and never reads one back.
@ -1848,6 +1876,19 @@ in
''; '';
}; };
builderCommonName = lib.mkOption {
type = lib.types.str;
default = "swarm-builder";
description = ''
Subject the store's `swarm-builder` cert-auth role accepts: the
identity the swarm builder presents from inside its container. Its
grant is two paths, read only: `swarm/services/builder/forge/token`
and its own queue client secret.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
'';
};
forgePublisherCommonName = lib.mkOption { forgePublisherCommonName = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "swarm-forge-publisher"; default = "swarm-forge-publisher";
@ -2123,6 +2164,31 @@ in
''; '';
}; };
builderClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/builder.pem";
description = ''
Certificate the swarm builder presents to the store. Its subject must
be {option}`services.hyperhive.deploy.bao.builderCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
On a builder host without the store, it is a file the operator copies
from the store's host. A path, never a value.
'';
};
builderClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/builder-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.builderClientCertFile`.
A path, never a value.
'';
};
forgePublisherClientCertFile = lib.mkOption { forgePublisherClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = null; default = null;
@ -3201,6 +3267,8 @@ in
systemd.services.swarm-bao-review-classifier-policy = readerPolicyUnit "write the review classifier's API-key-reader bao policy and cert-auth role" reviewClassifierReaders; systemd.services.swarm-bao-review-classifier-policy = readerPolicyUnit "write the review classifier's API-key-reader bao policy and cert-auth role" reviewClassifierReaders;
systemd.services.swarm-bao-builder-policy = readerPolicyUnit "write the swarm builder's forge-token-reader bao policy and cert-auth role" builderReaders;
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters; systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
# A FOURTH sibling, same shape and same reasons as the two above. This # A FOURTH sibling, same shape and same reasons as the two above. This

232
nix/host-modules/swarm-builder.nix Normal file
View file

@ -0,0 +1,232 @@
# The swarm builder, in a `swarm-builder` container on the host that sets
# `deploy.builder.enable`, which nothing turns on for the operator: it runs
# unreviewed config PRs through the host's nix daemon. It builds agent configs
# for swarm-controller: the controller sends a config PR's head over the
# queue, and the daemon builds that config on this host's agent base and
# replies.
# docs/swarm/builder.md has the design.
#
# ⚠️ Its own network namespace on the bridge, as ./swarm-ci-runner.nix has: a
# config PR is unreviewed input to a nix evaluation, so the container reaches
# the swarm services by name and nothing on the host's loopback.
#
# Builds go through the HOST nix-daemon, as the CI runner's do. The daemon
# runs as a dynamic user, an untrusted daemon client: a config can set no
# daemon option, and its builds run in the host's sandbox.
#
# The container holds one credential, its own store leaf, which nspawn hands in
# as a system credential with no copy on disk. The daemon reads everything else
# from the store with it: its queue client secret at start and its forge token
# on every build.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
cfg = hyperhiveCfg.deploy.builder;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = hyperhiveCfg.deploy.bao;
natsCfg = hyperhiveCfg.swarm.nats;
autheliaCfg = hyperhiveCfg.swarm.authelia;
forgeCfg = hyperhiveCfg.swarm.forge;
gatewayCfg = hyperhiveCfg.gateway;
networkCfg = hyperhiveCfg.network;
tlsCfg = hyperhiveCfg.deploy.hive-controller.tls;
machine = "swarm-builder";
unit = "swarm-builder";
clientId = hyperhiveCfg.swarm.builder.clientId;
# The daemon reaches authelia, the store and the queue by name over TLS, and
# `git` reaches the forge; in self-signed mode their leaves chain to the hive
# CA.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
caBundleModule = caTrust.trustBundle {
inherit pkgs;
name = machine;
consumers = [ unit ];
};
haveClientIdentity =
baoDeploy.builderClientCertFile != null && baoDeploy.builderClientKeyFile != null;
# Credential id → host source, as ./swarm-review-classifier.nix hands its
# leaf in.
storeFiles = lib.filterAttrs (_: source: source != null) {
builder-bao-cert = baoDeploy.builderClientCertFile;
builder-bao-key = baoDeploy.builderClientKeyFile;
builder-bao-server-ca = baoDeploy.serverCaFile;
};
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
credential = id: "/run/credentials/${unit}.service/${id}";
# The `nix` the host daemon runs, so client and daemon speak one protocol.
nix = config.nix.package;
# Strings, not paths: both are already store paths, and the flake reference
# the daemon passes must be the same one a hive's meta flake names.
hyperhiveFlake = hyperhiveCfg.c0re.hyperhiveFlake;
nixpkgsFlake = hyperhiveCfg.c0re.nixpkgsFlake;
privateNetwork = true;
in
{
options.services.hyperhive.swarm.builder.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-builder";
description = ''
The OAuth2 client id the swarm builder presents to the swarm queue.
Read-only: the host running authelia registers it, the queue's
auth-callout responder grants it the build subject, and the secret
publisher carries its secret to the store, all from this one value.
'';
};
options.services.hyperhive.deploy.builder.package = lib.mkOption {
type = lib.types.package;
defaultText = lib.literalExpression "hyperhive.packages.\${system}.swarm-builder";
description = "The swarm builder daemon package.";
};
config = lib.mkIf cfg.enable {
assertions = [
{
assertion = haveClientIdentity;
message = ''
services.hyperhive.deploy.builder.enable requires the builder's
swarm-secret-store client identity: set both
services.hyperhive.deploy.bao.builderClientCertFile
services.hyperhive.deploy.bao.builderClientKeyFile
The builder reads its queue client secret and its forge token with it.
Both default to the pair glue-bao-tls.nix mints when
services.hyperhive.deploy.bao.enable is true (the host that runs the
store). Otherwise copy builder.pem and builder-key.pem from the store
host's /var/lib/swarm-bao-pki/ and name them in those two options.
'';
}
{
assertion = lib.all oneWord (lib.attrValues storeFiles);
message = ''
The swarm builder's store identity paths must not contain whitespace:
nspawn's --load-credential receives them through nixos-container's
unquoted extra flags.
'';
}
];
# The container hangs off the bridge, takes its address from the gateway's
# DHCP and resolves through its dnsmasq.
services.hyperhive.network.enable = lib.mkDefault true;
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
# nspawn fails to start the container when a credential source is
# missing, so where this host mints the leaf the container waits for it.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
{ after = [ "swarm-bao-pki.service" ]; }
];
containers.${machine} = {
autoStart = true;
# The daemon keeps nothing but `git`'s fetch cache, which a restart may
# drop.
ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [
"--link-journal=host"
]
++ lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
inherit privateNetwork;
hostBridge = networkCfg.bridgeName;
bindMounts = {
# The directory rather than the socket file, so the mount survives a
# daemon restart replacing the inode.
"/nix/var/nix/daemon-socket" = {
hostPath = "/nix/var/nix/daemon-socket";
isReadOnly = false;
};
}
// caTrust.bindMount;
config =
{ lib, ... }:
{
imports = [
../container-modules/swarm-container.nix
caBundleModule
];
services.hyperhive.swarmContainer = {
inherit privateNetwork;
writesOwnResolvConf = false;
};
networking.nameservers = [ networkCfg.bridgeIp ];
networking.interfaces.eth0.useDHCP = true;
# Both the service and its socket are off: the socket unit owns
# `/nix/var/nix/daemon-socket/socket` and would race the bind mount.
systemd.globalEnvironment.NIX_REMOTE = "daemon";
systemd.services.nix-daemon.enable = false;
systemd.sockets.nix-daemon.enable = false;
systemd.services.${unit} = {
description = "build agent configs for swarm-controller";
wantedBy = [ "multi-user.target" ];
wants = [ "network-online.target" ];
after = [ "network-online.target" ];
# nix fetches the config repo with `git`.
path = [ pkgs.git ];
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = credential "builder-bao-cert";
BAO_CLIENT_KEY = credential "builder-bao-key";
NIX_CONFIG = "experimental-features = nix-command flakes";
# nix's fetch cache, under `CacheDirectory`; the dynamic user has
# no home of its own.
HOME = "/var/cache/${unit}";
XDG_CACHE_HOME = "/var/cache/${unit}";
SWARM_BUILDER_NATS_URL = "tls://${natsCfg.domain}:${toString natsCfg.port}";
SWARM_BUILDER_OIDC_TOKEN_ENDPOINT = "${autheliaCfg.url}/api/oidc/token";
SWARM_BUILDER_OIDC_CLIENT_ID = clientId;
SWARM_BUILDER_NIX = "${nix}/bin/nix";
SWARM_BUILDER_CHECK_FLAKE = "${../builder/check-flake}";
SWARM_BUILDER_HYPERHIVE_FLAKE = hyperhiveFlake;
SWARM_BUILDER_NIXPKGS_FLAKE = nixpkgsFlake;
SWARM_BUILDER_FORGE_URL = "https://${forgeCfg.domain}";
SWARM_BUILDER_CONFIG_ORG = "agent-configs";
}
// lib.optionalAttrs (storeFiles ? builder-bao-server-ca) {
BAO_CACERT = credential "builder-bao-server-ca";
}
// lib.optionalAttrs caTrust.useSelfSigned {
SWARM_BUILDER_OIDC_CA_FILE = caTrust.bundlePathFor machine;
# `git`'s trust store, which the bundle's system CAs keep whole.
GIT_SSL_CAINFO = caTrust.bundlePathFor machine;
};
serviceConfig = {
ExecStart = "${cfg.package}/bin/swarm-builder";
DynamicUser = true;
CacheDirectory = unit;
# Bare ids: the system credentials nspawn handed in.
LoadCredential = lib.attrNames storeFiles;
Restart = "on-failure";
RestartSec = "10s";
SyslogIdentifier = unit;
};
};
};
};
};
}

2
nix/host-modules/swarm-nats.nix
View file

@ -683,6 +683,8 @@ in
"--reader-client ${lib.escapeShellArg controllerCfg.queueClientId}" "--reader-client ${lib.escapeShellArg controllerCfg.queueClientId}"
# Hears try-outs and answers each by reply, and nothing else. # Hears try-outs and answers each by reply, and nothing else.
"--review-classifier-client ${lib.escapeShellArg config.services.hyperhive.swarm.review-classifier.clientId}" "--review-classifier-client ${lib.escapeShellArg config.services.hyperhive.swarm.review-classifier.clientId}"
# Hears builds and answers each by reply, and nothing else.
"--builder-client ${lib.escapeShellArg config.services.hyperhive.swarm.builder.clientId}"
# What an agent may publish to, keyed on its own name: its # What an agent may publish to, keyed on its own name: its
# terminal rows and its turn-state header # terminal rows and its turn-state header
# (`hive-agent::swarm_agent_state`), two subject families so a # (`hive-agent::swarm_agent_state`), two subject families so a

2
nix/host-modules/swarm-secret-publisher.nix
View file

@ -94,6 +94,8 @@ let
hyperhiveCfg.swarm.forge.sso.clientId hyperhiveCfg.swarm.forge.sso.clientId
# The review classifier, which reads it itself inside its container. # The review classifier, which reads it itself inside its container.
hyperhiveCfg.swarm.review-classifier.clientId hyperhiveCfg.swarm.review-classifier.clientId
# The swarm builder, which reads it itself inside its container.
hyperhiveCfg.swarm.builder.clientId
]; ];
# The swarm controller's OIDC client, which it reads back from the store # The swarm controller's OIDC client, which it reads back from the store

Some files were not shown because too many files have changed in this diff Show more