Watch

atlas 458a046b8d run attic as a swarm service with deploy.attic.enable

2026-10-27 13:04:17 +01:00

23 KiB

Raw Blame History

Swarm builder

The swarm builder checks that an agent config PR builds before an operator merges it. Each time an agent-configs/<agent> PR opens, reopens or gets a new push, the builder builds the PR's head on the agent base, and the result shows as a swarm-builder commit status on the PR and as a build_agent node on the swarm UI's Jobs page.

The builder also builds each agent from internal/meta before the swarm sends its deploy, and a failed build holds the deploy back: see Deploy gate.

Where it runs and how it's enabled: services.md.

What happens on a config PR

The forge delivers the pull_request event on the config-pr webhook swarm-controller already registers on agent-configs. An opened, reopened or synchronized action on an open PR queues one build_agent node carrying the agent, the head commit and the PR number. An edit, a label or a close builds nothing. Every build_agent node needs the one builder slot, so builds run one at a time and the rest wait pending. A newer head cancels its PR's pending build; a build already running finishes and writes its status on its own, older commit.
The node sets the status to pending, then sends the build as a request on the queue subject $SWARM.builder.build and waits up to 61 minutes for the reply. The builder runs one build at a time and refuses a request that has waited 30 seconds for its turn, such as one behind a build a controller restart orphaned: a build started later could still run when the 61 minutes end. Reading the forge token before a build has 20 seconds, and the rest of the minute covers the reply's trip back.
The builder runs nix build of nix/builder/check-flake's check output with every input overridden:
- hyperhive and nixpkgs: the sources the builder's own host was deployed with (c0re.hyperhiveFlake, c0re.nixpkgsFlake). A hive's meta flake pins hyperhive by forge URL and rev instead.
- agent: git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>. A build has one hour.

The reply is the system closure's store path, or the last lines nix printed. The node writes the status on the head commit:

Outcome	Status	Description	Node
built	`success`	the store path	done
`nix` failed or timed out	`failure`	the last line `nix` printed	failed
the builder busy for 30 seconds	`failure`	why	failed
no builder listening, or no reply in time	`error`	why	failed

The controller records every build that gets a final status, and the status links to that record: see Build records. The full tail is also in the failed node's error on the Jobs page and in the builder's journal line for the build, as the field F_ERROR_TAIL:

$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json

The builder holds the last 64 KiB of each of nix's output streams while the build runs, and the reply is the last 30 lines of that (at most 4000 bytes), so a long build log never grows its memory.

Build records

swarm-controller records each build when it writes the final status: built, failed, or no build ran or came back. The newest 100 are one JSON array under the key recent in the builder-runs KV bucket on the swarm queue, which only the controller reads or writes. It creates the bucket on first use and adds each build with a compare-and-set write, so the records survive a controller restart. Recording the 101st drops the oldest. A record the controller can't decode stays in the bucket, out of the list, and counts toward the 100.

Each record has:

id: <agent>-<first 12 characters of the commit>-<started_unix>
repo (agent-configs/<agent>), pr and head_sha
started_unix and finished_unix
outcome: built, failed or error, the status's success, failure or error
out_path for built, error_tail (the last 30 lines nix printed) for failed, and detail (why) for error

GET /api/builder/runs lists them, newest first, and GET /api/builder/runs/<id> returns one, or 404 when it isn't among the newest 100. Both are behind the swarm UI vhost's admins gate, and answer 503 when the controller has no swarm queue or can't open the bucket, 500 when it can't read it.

The final status links to https://<swarm domain>/jobs?run=<id>, the swarm UI's jobs tab, where the builds are a card under the job graph. Selecting Details on a red swarm-builder check opens that build's card, with nix's tail, in the browser. The raw record stays at GET /api/builder/runs/<id>. A status has no link when the record wasn't stored (the controller logs the failed write) or when the controller doesn't serve the swarm vhost (SWARM_CONTROLLER_PUBLIC_URL unset). The pending status has no link.

What the check is, and isn't

It composes the config the way a hive's meta flake does (agent-base.extendModules with the config's nixosModules.default), minus what a hive adds: its CA bundle, claude-code path, ports and OTEL settings, and the follows that point the config's own inputs at the hive's. A green check can therefore still fail on a hive, and the reverse.
The agent base is the builder's host's, not the hive's. Hives on another hyperhive or nixpkgs pin build something else.
The check pushes nothing. The closure stays in the builder host's store until it's collected, so on a host that also runs the hive, the hive's own build can reuse whatever matches. A meta build can push to a binary cache.
The PR check gates nothing. A merge deploys whatever the status says. The deploy gate is a separate build, from internal/meta.
A lost webhook delivery isn't retried: the head isn't built until the next push. The jobs graph is in memory, so a controller restart drops a build in flight and leaves its status pending.

Archetype evaluation

The builder also answers $SWARM.builder.archetypes: it evaluates the meta of every agent archetype its own hyperhive source declares, with no forge token, and answers with the JSON. An evaluation doesn't wait for a build in progress and has five minutes; the controller waits six. The builder logs each evaluation's time as elapsed_ms, and a timeout's error carries the last lines nix printed. What the controller does with the answer: archetypes.md.

Meta-flake lock

The builder also answers $SWARM.builder.lock-meta. The request carries a rendered flake.nix. The builder writes it into a scratch directory, runs nix flake lock there with the same forge credential helper its builds use, answers with the flake.lock that nix wrote, and removes the directory. A lock doesn't wait for a build in progress and has ten minutes. The builder commits nothing.

swarm-controller sends the request. It renders internal/meta's flake.nix with the hive-meta-flake renderer, all over https. Each custom agent that a hive places pins to its agent-configs/<agent> main; the controller omits and logs one whose repo has no commit on main, so it doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own running rev (services.hyperhive.c0re.hyperhiveRev, from self.rev):

the first commit pins hyperhive's forge main;
later, the pin moves to the controller's rev only when the forge's compare API shows that rev is a strict descendant of the committed pin: commits ahead of it, and none the pin has that it lacks. The compare endpoint counts histories with no common ancestor as zero commits ahead, so the check needs both directions;
an older controller, a rev on another line, a controller built from a dirty tree or a forge that can't answer leaves the pin. A merge to hyperhive's main moves nothing.

The controller commits flake.nix and the returned flake.lock to internal/meta in one commit, and gives each hive's forge account read on the repo. It re-renders when it starts, and again every ten minutes while the latest update hasn't succeeded (a tick adds nothing while one is pending or running, or after a success), before it deploys a merged config PR, and before the first deploy of a new agent. Those deploys run only after the update succeeded and the deploy gate built the agent: a hive builds only from internal/meta, so after a failed update the scheduler skips the deploy node, and the graph shows the update as failed and the deploy as skipped. Each ten-minute retry of a failed update declares a fresh build and deploy behind it for every agent whose latest deploy the failed update skipped, so the run that succeeds sends them (see Config changes). A render equal to the committed flake.nix sends no lock.

The controller sends a deploy to every agent a hive places whenever the pin in internal/meta differs from the pin it last sent, so they rebuild on the new hyperhive. It keeps that pin as the key pin in the meta-fanout KV bucket, which only the controller reads or writes, and a RecordFanout job writes the key once every deploy of the fan-out has finished or the deploy gate has skipped it. The check runs after each update and once when the controller starts.

The first commit sends a deploy, because the key holds no pin yet.
A controller that starts with no key, as after the upgrade that added it, sends one round. A deploy is idempotent per pin, so the round is harmless.
A controller that dies after the commit, or a deploy that fails, leaves the key behind. The next update or start sends the deploys again.
A deploy the gate skipped counts as sent. A failed build holds only its own agent back, and the key still moves, so one broken agent doesn't resend the healthy ones on every retry. The timer declares a build and deploy again for a deploy skipped because its build failed.
A restart that finds the key equal to the pin sends nothing.
When the bucket won't open, or its key won't read, the controller logs a warning and sends deploys only when its own update moved the pin. That case writes no key.

Deploy gate

Every deploy the swarm sends waits for a build of its agent from internal/meta, the flake a hive builds from. Four paths send deploys, and each declares the same build and deploy pair: a merged config PR, a new agent, the pin fan-out and the ten-minute retry. The build_meta node comes before the trigger_deploy node and needs the one builder slot, so it queues behind PR checks. The deploy runs only when the build is green.

The build_meta node reads internal/meta's head commit when it runs, because most paths declare it before the update that commits that head. It sends the agent and that commit on $SWARM.builder.build-meta and waits up to 61 minutes. The builder takes the same one-build slot as PR checks, and refuses a request that waited 30 seconds for it. It runs nix build of nixosConfigurations.<agent>.config.system.build.toplevel from git+https://<forge>/internal/meta.git?rev=<head>, with no input overridden: the committed flake.lock decides every input. Like the hive's build, it uses that attribute, so a green build here means the flake evaluates and builds for the builder's platform. The hive doesn't reuse the result yet: it still builds locally, from whatever head it reads at deploy time, which can be newer than the one the swarm built.

The node ends done or failed. Every failure skips the deploy and shows on the Jobs page as a failed build_meta node, and the timer builds it again:

What happened	`build_meta` node	Deploy
built	done, closure recorded	runs
`nix` exited with an error	failed, with `nix`'s last lines	skipped
`nix` hit its one-hour deadline, or couldn't start	failed, `nix timed out: …` or `starting nix: …`	skipped
no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token	failed, `no verdict: …`	skipped
the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure	failed, `no verdict: …`	skipped

On every ten-minute round the timer declares a build and deploy again for each agent whose latest build failed, without a meta update when the latest update succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out still records the pin when the gate skips a deploy, so one failing agent doesn't resend the healthy ones. A controller that updates before its builder host does is the same case: the old builder has no subscriber for the new subject, so the controller gets no responders at once, and the deploys go out after the builder updates. Each round adds a build and deploy node per failing agent.

When the build belongs to a merged config PR and nix exited with an error or hit its deadline, the controller also comments once on that PR, with its forge account: the agent, the internal/meta commit of the first red build, nix's last lines (at most 4000 bytes), and that the deploy waits and the swarm builds it again every round. A comment carries a marker line with the agent, and the controller posts only if the PR holds no comment of its own with that marker; a copy of the marker in someone else's comment is neither counted nor edited. The retry rounds stay quiet whatever commit they build, and the latest error stays on the Jobs page. A green build of the agent for that PR rewrites the marker and appends the build's commit to the comment, so a later red comments again. A no verdict failure, and a build that isn't a merged PR's (a new agent, the pin fan-out), leave no comment. When the forge refuses the comment, the controller logs a warning and the node ends as before.

The controller exports the gauge swarm.deploys.held: the agents whose latest build failed, with the label reason set to failed or unavailable. The Grafana alert hh-deploys-held fires when the sum stays above zero for 30 minutes, so a dead builder or a config that fails to build shows in Grafana before anyone notices an agent not updating.

On a built reply the controller writes the agent's key in the agent-closures KV bucket: the internal/meta commit, the closure's store path, the builder's platform and the time. Only the controller writes the bucket, and the builder holds no grant on it. A failed build writes nothing and leaves the previous key. The record's cached field says whether the builder pushed the closure to the binary cache. Nothing reads the bucket yet. The closure stays unrooted in the builder host's store.

The controller gives the builder forge account read on internal/meta when it updates the flake, and again before each build, since the repo is private. A grant the forge refuses fails the build with no verdict.

Binary cache

With services.hyperhive.swarm.attic.domain and cache set, the builder pushes each green meta build's closure to that cache, an attic server inside or outside the swarm, before it replies. It runs attic push swarm:<cache> <out path> once per build, not attic watch, and the builder pushes again when the controller runs a build again. The push has ten minutes, or less when the reply is due sooner, because it must end inside the 61 minutes the controller waits. attic checks the server's certificate against the system roots and pins nothing, so a Let's Encrypt certificate works.

Set both options or neither: evaluation refuses a domain without a cache name. The cache name follows attic's rule, 1 to 50 letters, digits, -, _ or +, starting with a letter or digit, and evaluation refuses any other name.

The push token is the swarm's own cache's to create, or the operator's to write for an external one. The builder reads it on every push from secret/swarm/services/attic/push-token (field value), writes it to a file and an attic config naming that file (token-file) in the unit's private runtime directory (/run/swarm-builder, mode 0700, both files 0600), points attic at the config with XDG_CONFIG_HOME and removes both after the push. The token is in no argument or variable, so no process listing shows it.

The reply's cached says whether the push finished. A push that fails, times out or can't read the token leaves the build green with cached false and logs a warning with the reason; the deploy goes ahead and the next build of the agent tries again. With no cache set, the builder pushes nothing and cached is always false.

Mixed versions:

controller	builder	what happens
new	new	`cached` in the reply and in the `agent-closures` record
new	old	the reply has no `cached`, which reads as `false`
old	new	the controller ignores `cached`; its record has no such field

A record written before the field existed also reads as false.

The swarm's own cache

services.hyperhive.deploy.attic.enable runs an attic server in a swarm-attic container behind the gateway, at attic.<swarm domain>. It defaults on where you set deploy.allSwarmServices and the host holds the cache's store identity (deploy.bao.atticClient{Cert,Key}File, which the store's host mints). You write no secret: on first start, two host units log in to the store as the cache's own principal and create what's missing.

swarm-bao-attic-secret creates the key atticd signs tokens with at secret/swarm/services/attic/jwt-secret and writes it where the container reads it.
swarm-bao-attic-tokens creates the cache and mints the push token, which the builder reads, at secret/swarm/services/attic/push-token, and a read-only pull token for the hives at secret/swarm/services/attic/pull-token.

A unit creates each secret only while the store holds none, so a second start mints nothing. Both tokens last ten years and nothing renews them. To replace one, write the new token to its path in the store: nothing mints over a value that exists. A new signing key invalidates both tokens.

With the switch on, swarm.attic.domain and swarm.attic.cache default to attic.<swarm domain> and swarm, so the builder on that host pushes to it. Naming either option yourself wins over its default. The defaults apply only on the host that runs the cache: on any other host, name both options too, with the same values.

Storage is local disk in the container's state (/var/lib/swarm-attic). A store path nobody added or fetched for 14 days goes at the next garbage collection, which runs every 12 hours (deploy.attic.retentionPeriod). The next build of that agent pushes a closure the cache lost.

The gateway terminates TLS with a swarm services certificate and forwards to 127.0.0.1:8910 (deploy.attic.port) with no login in front: attic checks its own tokens, and a nix client can't follow a login redirect. The container logs to its journal only; its journal doesn't reach the swarm log store.

External cache

The cache can be an attic tenant on an instance outside the swarm. To point the swarm at an existing tenant:

On the builder host, set services.hyperhive.swarm.attic.domain to the instance's host name (no scheme, host:port if it isn't on 443) and services.hyperhive.swarm.attic.cache to the cache's name on it.
Seed the push token in the swarm secret store at secret/swarm/services/attic/push-token, field value. Use the store's UI, or bao kv put with value=- and the token on stdin, so the token is in no command line. The token needs push rights on that cache. Until it exists, every push fails and builds stay green with cached false.

The builder checks the instance's certificate against the system roots and pins nothing, so a Let's Encrypt certificate needs no setup. The builder reads the token again on every push, so a new token takes effect on the next build.

Where you host the shared services, deploy.attic.enable defaults on and runs the swarm's own cache. To use an external one, set it to false with lib.mkForce, then set the two options by hand and seed the token as above. The swarm then renders no container, vhost or unit of its own cache, and creates no secret in the store. The store's host still mints the cache's leaf and writes its role, which nothing logs in to.

Credentials

What	Where	Who reads it
store leaf	`deploy.bao.builderClient{Cert,Key}File`	nspawn, into the service as a credential
queue client secret	`secret/swarm/services/swarm-builder/oidc/client`	the builder, at start and when authelia refuses it
forge token (read-only)	`secret/swarm/services/builder/forge/token`	the builder, on every build
cache push token	`secret/swarm/services/attic/push-token`	the builder, on every push

The token belongs to the builder forge account, which swarm-controller creates, puts in the hives team of agent-configs (read on every config repo, and on internal/meta through the meta update) and mints a read:repository token for, in a five-minute pass. git gets the token through a credential helper reading the child's environment: it's in no argument, URL or lock, so no line nix prints can carry it into a status. The same git fetches a config's own inputs, so the builder registers the helper for the forge's URL only, and the helper answers only an https request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the status with its own forge token. Details per path: credentials-reference.md.

Isolation

The container has its own network namespace on the bridge, as the CI runner does, so a config's evaluation can't reach a host-loopback service. Evaluation is pure and runs in the container. Builds go to the host's nix daemon as an untrusted client: a config can set no daemon option, and its derivations build in the host's sandbox. A fixed-output derivation still has the network, as it does for any build on that host.

23 KiB Raw Blame History