548 lines
35 KiB
Markdown
548 lines
35 KiB
Markdown
# Swarm builder
|
|
|
|
The swarm builder checks that an agent config PR builds before an operator
|
|
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
|
|
push, the builder builds the PR's head on the agent base, and the result shows as a
|
|
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
|
swarm UI's Jobs page.
|
|
|
|
The builder also builds each agent from `internal/meta` before the swarm sends
|
|
its deploy, and a failed build holds the deploy back: see
|
|
[Deploy gate](#deploy-gate).
|
|
|
|
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|
|
|
## What happens on a config PR
|
|
|
|
1. The forge delivers the `pull_request` event on the `config-pr` webhook
|
|
swarm-controller already registers on `agent-configs`. An `opened`,
|
|
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
|
node carrying the agent, the head commit and the PR number. An edit, a
|
|
label or a close builds nothing. Every `build_agent` node needs the one
|
|
`builder` slot, so builds run one at a time and the rest wait pending. A
|
|
newer head cancels its PR's pending build; a build already running
|
|
finishes and writes its status on its own, older commit.
|
|
2. The node sets the status to `pending`, then sends the build as a request on
|
|
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
|
reply. The builder runs one build at a time and refuses a request that has
|
|
waited 30 seconds for its turn, such as one behind a build a controller
|
|
restart orphaned: a build started later could still run when the 61
|
|
minutes end. Reading the forge token before a build has 20 seconds, and
|
|
the rest of the minute covers the reply's trip back.
|
|
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
|
with every input overridden:
|
|
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
|
deployed with (`deploy.hyperhiveFlake`, `deploy.builder.nixpkgsFlake`). A hive's meta
|
|
flake pins `hyperhive` by forge URL and rev instead.
|
|
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
|
|
A build has one hour.
|
|
4. The reply is the system closure's store path, or the last lines `nix`
|
|
printed. The node writes the status on the head commit:
|
|
|
|
| Outcome | Status | Description | Node |
|
|
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
|
| built | `success` | the store path | done |
|
|
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
|
| the builder busy for 30 seconds | `failure` | why | failed |
|
|
| no builder listening, or no reply in time | `error` | why | failed |
|
|
|
|
The controller records every build that gets a final status, and the
|
|
status links to that record: see [Build records](#build-records). The
|
|
full tail is also in the failed node's error on the Jobs page and in the
|
|
builder's journal line for the build, as the field `F_ERROR_TAIL`:
|
|
|
|
```console
|
|
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
|
|
```
|
|
|
|
The builder holds the last 64 KiB of each of `nix`'s output streams while the
|
|
build runs, and the reply is the last 30 lines of that (at most 4000
|
|
bytes), so a long build log never grows its memory.
|
|
|
|
## Build records
|
|
|
|
`swarm-controller` records each build when it writes the final status:
|
|
built, failed, or no build ran or came back. The newest 100 are one JSON
|
|
array under the key `recent` in the `builder-runs` KV bucket on the swarm
|
|
queue, which only the controller reads or writes. It creates the bucket on
|
|
first use and adds each build with a compare-and-set write, so the records
|
|
survive a controller restart. Recording the 101st drops the oldest. A record
|
|
the controller can't decode stays in the bucket, out of the list, and counts
|
|
toward the 100.
|
|
|
|
Each record has:
|
|
|
|
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>`
|
|
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`
|
|
- `started_unix` and `finished_unix`
|
|
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
|
|
or `error`
|
|
- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
|
|
`failed`, and `detail` (why) for `error`
|
|
|
|
`GET /api/builder/runs` lists them, newest first, and
|
|
`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
|
|
newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
|
|
when the controller has no swarm queue or can't open the bucket, 500 when it
|
|
can't read it.
|
|
|
|
The final status links to `https://<swarm domain>/jobs?run=<id>`, the swarm
|
|
UI's jobs tab, where the builds are a card under the job graph. Selecting
|
|
**Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record
|
|
stays at `GET /api/builder/runs/<id>`. A status has no link when the record wasn't
|
|
stored (the controller logs the failed write) or when the controller doesn't
|
|
serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
|
|
status has no link.
|
|
|
|
## What the check is, and isn't
|
|
|
|
- It composes the config the way a hive's meta flake does
|
|
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
|
|
what a hive adds: its CA bundle, `claude-code` path, ports and
|
|
OTEL settings, and the `follows` that point the config's own inputs at the
|
|
hive's. A green check can therefore still fail on a hive, and the reverse.
|
|
- The agent base is the builder's host's, not the hive's. Hives on another
|
|
hyperhive or nixpkgs pin build something else.
|
|
- The check pushes nothing. The closure stays in the builder
|
|
host's store until it's collected, so on a host that also runs the hive, the
|
|
hive's own build can reuse whatever matches. A [meta build](#deploy-gate)
|
|
can push to [a binary cache](#binary-cache).
|
|
- The PR check gates nothing. A merge deploys whatever the status says. The
|
|
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
|
|
- A lost webhook delivery isn't retried: the head isn't built until the next
|
|
push. The jobs graph is in memory, so a controller restart drops a build in
|
|
flight and leaves its status `pending`.
|
|
|
|
## Archetype evaluation
|
|
|
|
The builder also answers `$SWARM.builder.archetypes`: it evaluates the
|
|
`meta` of every agent archetype its own hyperhive source declares, with no
|
|
forge token, and answers with the JSON. An evaluation doesn't wait for a
|
|
build in progress and has five minutes; the controller waits six. The builder
|
|
logs each evaluation's time as `elapsed_ms`, and a timeout's error carries the last
|
|
lines nix printed. What the controller does with the answer: [`archetypes.md`](archetypes.md).
|
|
|
|
## Meta-flake lock
|
|
|
|
The builder also answers `$SWARM.builder.lock-meta`. The request carries a
|
|
rendered `flake.nix`. The builder writes it into a scratch directory, runs
|
|
`nix flake lock` there with the same forge credential helper its builds use,
|
|
answers with the `flake.lock` that `nix` wrote, and removes the directory. A
|
|
lock doesn't wait for a build in progress and has ten minutes. The builder
|
|
commits nothing.
|
|
|
|
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
|
|
with the `hive-meta-flake` renderer, all over `https`.
|
|
Each custom agent that a hive places pins to its `agent-configs/<agent>`
|
|
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
|
|
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
|
|
running rev (`services.hyperhive.deploy.hive-controller.hyperhiveRev`, from `self.rev`):
|
|
|
|
- the first commit pins hyperhive's forge `main`;
|
|
- later, the pin moves to the controller's rev only when the forge's compare
|
|
API shows that rev is a strict descendant of the committed pin: commits
|
|
ahead of it, and none the pin has that it lacks. The compare endpoint counts
|
|
histories with no common ancestor as zero commits ahead, so the check needs
|
|
both directions;
|
|
- an older controller, a rev on another line, a controller built from a dirty
|
|
tree or a forge that can't answer leaves the pin. A merge to hyperhive's
|
|
`main` moves nothing.
|
|
|
|
The controller commits `flake.nix` and the returned `flake.lock` to
|
|
`internal/meta` in one commit, and gives each hive's forge account read on the
|
|
repo. It re-renders when it starts, and again every ten minutes while the latest
|
|
update hasn't succeeded (a tick adds nothing while one is pending or running,
|
|
or after a success), before it deploys a merged config PR, and
|
|
before the first deploy of a new agent. Those deploys run only after the
|
|
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
|
|
builds only from `internal/meta`, so after a failed update the scheduler skips
|
|
the deploy node, and the graph shows the update as failed and the deploy as
|
|
skipped. Each ten-minute retry of a failed update declares a fresh build and
|
|
deploy behind it for every agent whose latest deploy the failed update skipped,
|
|
so the run that succeeds sends them (see
|
|
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
|
|
equal to the committed `flake.nix` sends no lock.
|
|
|
|
The controller sends a deploy to every agent a hive places whenever the pin in
|
|
`internal/meta` differs from the pin it last sent, so they rebuild on the new
|
|
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
|
|
which only the controller reads or writes, and a `RecordFanout` job writes the
|
|
key once every deploy of the fan-out has finished or the deploy gate has skipped
|
|
it. The check runs after each update and once when the controller starts.
|
|
|
|
- The first commit sends a deploy, because the key holds no pin yet.
|
|
- A controller that starts with no key, as after the upgrade that added it,
|
|
sends one round. A deploy is idempotent per pin, so the round is harmless.
|
|
- A controller that dies after the commit, or a deploy that fails, leaves the
|
|
key behind. The next update or start sends the deploys again.
|
|
- A deploy the gate skipped counts as sent. A failed build holds only its own
|
|
agent back, and the key still moves, so one broken agent doesn't resend the
|
|
healthy ones on every retry. The timer declares a build and deploy again for
|
|
a deploy skipped because its build failed.
|
|
- A restart that finds the key equal to the pin sends nothing.
|
|
- When the bucket won't open, or its key won't read, the controller logs a
|
|
warning and sends deploys only when its own update moved the pin. That case
|
|
writes no key.
|
|
|
|
## Deploy gate
|
|
|
|
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
|
|
the flake a hive builds from. Four paths send deploys, and each declares the same
|
|
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
|
|
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
|
|
and needs the one `builder` slot, so it queues behind PR checks. The deploy
|
|
runs only when the build is green.
|
|
|
|
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
|
|
most paths declare it before the update that commits that head. It sends the
|
|
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
|
|
minutes. The builder takes the same one-build slot as PR checks, and
|
|
refuses a request that waited 30 seconds for it. It runs `nix build` of
|
|
`nixosConfigurations.<agent>.config.system.build.toplevel` from
|
|
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
|
|
the committed `flake.lock` decides every input. Like the hive's build, it uses
|
|
that attribute, so a green build here means the flake evaluates and builds for
|
|
the builder's platform. A hive [downloads the closure instead of building it](#what-hives-do-with-the-cache)
|
|
when the record is for the commit it builds from; otherwise it builds locally,
|
|
from whatever head it reads at deploy time, which can be newer than the one the
|
|
swarm built.
|
|
|
|
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
|
|
Jobs page as a failed `build_meta` node, and the timer builds it again:
|
|
|
|
| What happened | `build_meta` node | Deploy |
|
|
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
|
|
| built | done, closure recorded | runs |
|
|
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
|
|
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
|
|
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
|
|
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
|
|
|
|
On every ten-minute round the timer declares a build and deploy again for each
|
|
agent whose latest build failed, without a meta update when the latest update
|
|
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
|
|
still records the pin when the gate skips a deploy, so one failing agent doesn't
|
|
resend the healthy ones. A controller that updates before its builder host does
|
|
is the same case: the old builder has no subscriber for the new subject, so the
|
|
controller gets no responders at once, and the deploys go out after the builder
|
|
updates. Each round adds a build and deploy node per failing agent.
|
|
|
|
When the build belongs to a merged config PR and `nix` exited with an error or hit
|
|
its deadline, the controller also comments once on that PR, with its forge
|
|
account: the agent, the `internal/meta` commit of the first red build, `nix`'s last lines (at most 4000
|
|
bytes), and that the deploy waits and the swarm builds it again every round. A comment
|
|
carries a marker line with the agent, and the controller posts only if the PR
|
|
holds no comment of its own with that marker; a copy of the marker in someone else's comment is neither counted nor edited. The retry rounds stay quiet whatever commit
|
|
they build, and the latest error stays on the Jobs page. A green build of the
|
|
agent for that PR rewrites the marker and appends the build's commit to the comment, so a later red comments again.
|
|
A `no verdict` failure, and a build that isn't a
|
|
merged PR's (a new agent, the pin fan-out), leave no comment. When the forge
|
|
refuses the comment, the controller logs a warning and the node ends as before.
|
|
|
|
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
|
|
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
|
|
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
|
|
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
|
|
not updating.
|
|
|
|
On a built reply the controller writes the agent's key in the `agent-closures`
|
|
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
|
|
platform and the time. Only the controller writes the bucket, and the builder
|
|
holds no grant on it. A failed build writes nothing and leaves the previous key.
|
|
The record's `cached` field says whether the builder pushed the closure to
|
|
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
|
|
[report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
|
|
|
|
The controller gives the `builder` forge account read on `internal/meta` when it
|
|
updates the flake, and again before each build, since the repo is private. A
|
|
grant the forge refuses fails the build with no verdict.
|
|
|
|
## Binary cache
|
|
|
|
`services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own
|
|
cache (`attic.<swarm domain>` and `swarm`) on every host, so the builder pushes
|
|
each green meta build's closure to it, or to the cache you override them with,
|
|
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
|
|
not `attic watch`, and the builder pushes again when the controller runs a build again. The
|
|
push has ten minutes, or less when the reply is due sooner, because it must end
|
|
inside the 61 minutes the controller waits. `attic` checks the server's certificate
|
|
against the system roots and pins nothing, so a Let's Encrypt certificate
|
|
works.
|
|
|
|
Set both options to a name or both to null: evaluation refuses one without the
|
|
other. Null in both is how a swarm opts out of pushing.
|
|
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
|
|
starting with a letter or digit, and evaluation refuses any other name.
|
|
|
|
The push token is the swarm's own cache's to create, or the operator's to write for an
|
|
external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to
|
|
a file and an `attic` config naming that file (`token-file`) in the unit's private
|
|
runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
|
|
`attic` at the config with `XDG_CONFIG_HOME` and removes both after the push.
|
|
The token is in no argument or variable, so no process listing shows it.
|
|
|
|
The reply's `cached` says whether the push finished. A push that fails, times out
|
|
or can't read the token leaves the build green with `cached` false and logs a
|
|
warning with the reason; the deploy goes ahead and the next build of the agent
|
|
tries again. With no attic behind the defaults, every push fails this way. With
|
|
both options null, the builder pushes nothing and `cached` is always false.
|
|
|
|
Mixed versions:
|
|
|
|
| controller | builder | what happens |
|
|
| ---------- | ------- | ------------------------------------------------------------- |
|
|
| new | new | `cached` in the reply and in the `agent-closures` record |
|
|
| new | old | the reply has no `cached`, which reads as `false` |
|
|
| old | new | the controller ignores `cached`; its record has no such field |
|
|
|
|
A record written before the field existed also reads as `false`.
|
|
|
|
### Rebuild on a cache miss
|
|
|
|
The cache drops a closure nobody fetched for its retention period, but the
|
|
record in `agent-closures` stays. Every ten minutes `swarm-controller` checks
|
|
each agent a hive places: when the record is from `internal/meta`'s head and
|
|
says the builder pushed it, the controller fetches the narinfo from
|
|
`<url>/<cache>/<hash>.narinfo` with the pull token it reads from
|
|
`secret/swarm/services/attic/` (the same four leaves it
|
|
[copies to the hives](#what-hives-do-with-the-cache); with any missing, or a URL
|
|
that isn't `https`, the pass does nothing). A 404 is a miss, and the controller
|
|
sends the builder `build-meta` for that agent and the record's own commit. The
|
|
builder builds it and pushes as for any meta build. Any other answer or a failed
|
|
request isn't a miss, and rebuilds nothing. The next ordinary build handles a record
|
|
from an older commit, or one with `cached` false.
|
|
|
|
A pass sends at most three builds, one after the other. After an agent's rebuild
|
|
failed, got no answer or built without pushing, the pass skips it for 30 minutes, then
|
|
for twice as long after each further failure, up to 12 hours. The wait is in memory,
|
|
so a controller restart tries every agent again. The first pass runs ten
|
|
minutes after the controller starts.
|
|
|
|
The record keeps naming what hives deploy. A rebuild that lands on the record's
|
|
path writes nothing. A rebuild that lands on another path means the build isn't
|
|
reproducible: the controller logs a warning, and replaces the record only if it
|
|
still holds the same commit and no write came since the controller read it. The
|
|
record otherwise stays as it was.
|
|
|
|
The controller counts each rebuild in `hyperhive.builder.cache_miss_rebuilds` with
|
|
the label `outcome`: `rebuilt`, `path_changed` (the record now names the new path),
|
|
`path_changed_kept` (the record stayed), `not_pushed`, `failed` (`nix` failed) or
|
|
`unavailable` (the builder was busy, absent or gave no answer).
|
|
|
|
Mixed versions: a builder needs no change, since a meta build already names
|
|
its commit. An old controller does nothing about a miss, as before. A hive
|
|
sees nothing new.
|
|
|
|
### The swarm's own cache
|
|
|
|
`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic`
|
|
container behind the gateway, at `attic.<swarm domain>`. It defaults on where
|
|
you set `deploy.allSwarmServices` and the host holds the cache's store identity
|
|
(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write
|
|
no secret: on first start, two host units log in to the store as the cache's own
|
|
principal and create what's missing.
|
|
|
|
- `swarm-bao-attic-secret` creates the key atticd signs tokens with at
|
|
`secret/swarm/services/attic/jwt-secret` and writes it where the container
|
|
reads it.
|
|
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
|
|
the builder reads, at `secret/swarm/services/attic/push-token`, and a
|
|
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
|
|
It also stores the facts a hive pulls with: the cache's address at
|
|
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
|
|
`secret/swarm/services/attic/cache`, and the public key it signs with at
|
|
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
|
|
|
|
A unit creates each secret only while the store holds none, so a second start
|
|
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
|
|
write the new token to its path in the store, in its UI under the OIDC login:
|
|
nothing mints over a value that exists. A new signing key invalidates both tokens.
|
|
|
|
The unit creates the address, name and key once and never updates them. After
|
|
you change `swarm.attic.domain`, or after the server's database goes and the
|
|
cache gets a new key, delete the stale leaf and restart
|
|
`swarm-bao-attic-tokens`.
|
|
|
|
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
|
|
and `swarm` on every host, whether or not it runs the cache, so a builder on any
|
|
host pushes to it.
|
|
|
|
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
|
|
path nobody added or fetched for 14 days goes at the next garbage collection,
|
|
which runs every 12 hours (`deploy.attic.retentionPeriod`). The controller
|
|
[rebuilds a closure the cache lost](#rebuild-on-a-cache-miss).
|
|
|
|
The gateway terminates TLS with a swarm services certificate and forwards to
|
|
`127.0.0.1:8910` (`deploy.attic.port`) with no login in front: attic checks its
|
|
own tokens, and a nix client can't follow a login redirect. The container logs
|
|
to its journal only; its journal doesn't reach the swarm log store.
|
|
|
|
### External cache
|
|
|
|
The cache can be an attic tenant on an instance outside the swarm. To point
|
|
the swarm at an existing tenant:
|
|
|
|
1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the
|
|
instance's host name (no scheme, `host:port` if it isn't on 443) and
|
|
`services.hyperhive.swarm.attic.cache` to the cache's name on it. Set
|
|
both options on every host that reads them.
|
|
2. Seed the push token in the swarm secret store at
|
|
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI under
|
|
the OIDC login, or `bao kv put` with `value=-` and the token on stdin, so the token is in no
|
|
command line. The token needs push rights on that cache. Until it
|
|
exists, every push fails and builds stay green with `cached` false.
|
|
Seed the hives' pull token the same way at
|
|
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
|
|
3. Seed the three facts hives pull with, each as field `value`:
|
|
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
|
|
slash; hives refuse `http://`), `secret/swarm/services/attic/cache` (the cache's name) and
|
|
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
|
|
cache's page or `attic cache info`). Until all four exist, hives report
|
|
`no-credentials`.
|
|
|
|
The builder checks the instance's certificate against the system roots and
|
|
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
|
|
reads the token again on every push, so a new token takes effect on the next
|
|
build.
|
|
|
|
To use an external cache, override the two defaults and don't set
|
|
`deploy.attic.enable` anywhere. Where you host the shared services with
|
|
`allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there.
|
|
The swarm then renders no container, vhost or unit of its own cache, and
|
|
creates no secret in the store. The store's host still mints the cache's
|
|
leaf and writes its role, which nothing logs in to.
|
|
|
|
A host that has `deploy.attic.enable` on and `swarm.attic.domain` set to another
|
|
name gets an evaluation warning that names this fix.
|
|
|
|
## What hives do with the cache
|
|
|
|
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
|
|
leaves (`pull-token`, `url`, `cache`, `public-key`) to
|
|
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
|
|
own prefix. It writes a copy only when the value differs, skips a leaf the
|
|
store doesn't hold, and leaves a copy whose source disappeared.
|
|
|
|
`hive-c0re` reads its four leaves every five minutes under the hive's own store
|
|
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
|
|
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the
|
|
token goes out in a header and in the netrc; a hive that finds another scheme
|
|
writes nothing.
|
|
|
|
`hive-priv` doesn't read those files. The key that must have signed a download and
|
|
the pull token come from `hive-priv-cache-facts`, a root unit that reads the
|
|
hive's own `attic/public-key` and `attic/pull-token` from the store under the
|
|
hive's own store identity every ten minutes and writes them to
|
|
`/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). The cache's
|
|
address is the host's `services.hyperhive.swarm.attic.domain` and `cache`,
|
|
handed to `hive-priv` in its unit's environment. A file `hive-c0re` writes
|
|
decides nothing `hive-priv` trusts.
|
|
|
|
The same pass checks each of the hive's agents. It reads the agent's record in
|
|
`agent-closures`, then fetches the closure's narinfo from
|
|
`<url>/<cache>/<hash>.narinfo` with the pull token as the basic-auth password.
|
|
It checks that a signature by the public key covers the narinfo, in Rust, with
|
|
no help from the host's nix daemon. It downloads nothing. The result is a log
|
|
line (at info when it changes) and the gauge
|
|
`hyperhive.agent.closure_pullable{agent}` (the `hive` label comes from the
|
|
collector, as for every `hyperhive.*` metric): `1` when the hive can pull the
|
|
latest closure, else `0` with a `reason` label. The first check that fails
|
|
decides the reason:
|
|
|
|
| `reason` | Meaning |
|
|
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `no-credentials` | One of the four leaves is missing or unusable (a URL that isn't `https` is unusable). The swarm has no cache, the controller is too old to copy them, or you haven't seeded them yet. |
|
|
| `no-record` | `agent-closures` holds no build for the agent. |
|
|
| `rev-mismatch` | The record's build came from another `internal/meta` commit than its head, or the hive can't read the head. |
|
|
| `not-cached` | The record says the builder didn't push the closure, or the cache answers 404 for the narinfo. |
|
|
| `fetch-failed` | The request failed, or the cache answered with a status other than 200 or 404 (a refused token is a 401 or 403). |
|
|
| `bad-signature` | The narinfo describes another path, or no signature by the public key covers it. |
|
|
|
|
A failed read of the store, the queue or the forge leaves the last result in
|
|
place and raises a warning after three failed passes. A hive reports nothing
|
|
before its first pass.
|
|
|
|
Mixed versions:
|
|
|
|
| controller | hive | what happens |
|
|
| ---------- | ---- | ----------------------------------------------------------------------------------- |
|
|
| new | new | the copies exist and the hive reports per agent |
|
|
| old | new | the controller copies nothing, so the hive reports `no-credentials` for every agent |
|
|
| new | old | the controller writes copies nobody reads |
|
|
| any | any | a swarm with no cache seeds no leaves, and every hive reports `no-credentials` |
|
|
|
|
The signature check follows Nix's fingerprint format but hasn't run against a
|
|
live attic. A `bad-signature` on every agent in the first pass after a deploy
|
|
means check attic's fingerprint format; it says nothing about an attacker. A
|
|
`fetch-failed` on every agent points at the URL, the token or the network.
|
|
|
|
### Rebuilds download the closure
|
|
|
|
A rebuild's `Prebuild` node, which runs while the agent is still up, tries the
|
|
cache before it builds. It reads the agent's record in `agent-closures` and
|
|
goes on only when the record is for the `internal/meta` commit `MetaSync`
|
|
recorded for this rebuild and says the builder pushed the closure. It then
|
|
asks `hive-priv` to realise the record's store path (`RealiseClosure`). `hive-priv` runs
|
|
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
|
|
the host's configured cache address, and the public key and netrc from the
|
|
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
|
|
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
|
|
source, and nix stores the path only when the cache's key signed it. The host's
|
|
nix configuration stays as it was.
|
|
|
|
The deploy log says which way it went:
|
|
|
|
- `prebuild: substituted <path>`: the closure is in the store, and the local build found it there and had nothing to build.
|
|
- `prebuild: the downloaded closure <path> is not the one this hive evaluates, <path>; built the latter locally`, at warn level: the builder and the hive evaluate different closures for the agent, which is worth a look. The download was no use and the rebuild built locally, with the agent still up.
|
|
- `prebuild: built locally: <reason>`: the reason is the first check that failed,
|
|
such as `no-record`, `rev-mismatch`, `not-cached`, or what `hive-priv` answered
|
|
(`not available: …`, for a cache that's down or has no credentials). The rebuild builds as before.
|
|
- The same line at error level, with `no signature by the swarm cache's key` in
|
|
the reason: nix refused the path because no trusted key signed it. The rebuild builds locally
|
|
and `hive-priv` logs the refusal as well.
|
|
|
|
After a download, `Prebuild` still runs the local `nix build` of the agent's toplevel,
|
|
while the agent is up. It's an evaluation and no build when the hive evaluates the path the
|
|
swarm built, and a full build, as without a cache, when it doesn't, so `Swap` never meets a
|
|
build it didn't expect. `Swap` still evaluates and builds in `hive-priv`, and finds the path in
|
|
the store. A stopped agent skips `Prebuild`, so it still builds in `Swap`.
|
|
|
|
Mixed versions:
|
|
|
|
| hive | controller or record | what happens |
|
|
| ---- | -------------------- | ---------------------------------------------------------------------------------- |
|
|
| old | any | builds as before |
|
|
| new | old, or no record | builds locally and logs why |
|
|
| new | cache down | the download fails, the hive builds locally and logs why |
|
|
| new | signature mismatch | nix refuses the path, the hive builds locally and logs an error |
|
|
| new | old `hive-priv` | the verb is unknown to it, so the call fails and the hive builds locally, as above |
|
|
|
|
## Credentials
|
|
|
|
| What | Where | Who reads it |
|
|
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
|
|
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
|
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
|
|
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
|
| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push |
|
|
|
|
The token belongs to the `builder` forge account, which swarm-controller
|
|
creates, puts in the `hives` team of `agent-configs` (read on every config
|
|
repo, and on `internal/meta` through the meta update) and mints a
|
|
`read:repository` token for, in a five-minute pass. `git`
|
|
gets the token through a credential helper reading the child's environment:
|
|
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
|
status. The same `git` fetches a config's own inputs, so the builder registers
|
|
the helper for the forge's URL only, and the helper answers only an `https`
|
|
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
|
|
status with its own forge token. Details per path:
|
|
[`credentials-reference.md`](credentials-reference.md).
|
|
|
|
## Isolation
|
|
|
|
The container has its own network namespace on the bridge, as the CI runner
|
|
does, so a config's evaluation can't reach a host-loopback service. Evaluation
|
|
is pure and runs in the container. Builds go to the host's nix daemon as an
|
|
untrusted client: a config can set no daemon option, and its derivations build
|
|
in the host's sandbox. A fixed-output derivation still has the network, as it
|
|
does for any build on that host.
|