6b800600e2
hive-priv applies only the path its last RealiseClosure returned for that agent. On drift the swap and create record the pins of the commit the closure was built from.
543 lines
35 KiB
Markdown
543 lines
35 KiB
Markdown
# Swarm builder
|
|
|
|
The swarm builder checks that an agent config PR builds before an operator
|
|
merges it. Each time an `agent-configs/<agent>` PR opens, reopens or gets a new
|
|
push, the builder builds the PR's head on the agent base, and the result shows as a
|
|
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
|
swarm UI's Jobs page.
|
|
|
|
The builder also builds each agent from `internal/meta` before the swarm sends
|
|
its deploy, and a failed build holds the deploy back: see
|
|
[Deploy gate](#deploy-gate).
|
|
|
|
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
|
|
|
## What happens on a config PR
|
|
|
|
1. The forge delivers the `pull_request` event on the `config-pr` webhook
|
|
swarm-controller already registers on `agent-configs`. An `opened`,
|
|
`reopened` or `synchronized` action on an open PR queues one `build_agent`
|
|
node carrying the agent, the head commit and the PR number. An edit, a
|
|
label or a close builds nothing. Every `build_agent` node needs the one
|
|
`builder` slot, so builds run one at a time and the rest wait pending. A
|
|
newer head cancels its PR's pending build; a build already running
|
|
finishes and writes its status on its own, older commit.
|
|
2. The node sets the status to `pending`, then sends the build as a request on
|
|
the queue subject `$SWARM.builder.build` and waits up to 61 minutes for the
|
|
reply. The builder runs one build at a time and refuses a request that has
|
|
waited 30 seconds for its turn, such as one behind a build a controller
|
|
restart orphaned: a build started later could still run when the 61
|
|
minutes end. Reading the forge token before a build has 20 seconds, and
|
|
the rest of the minute covers the reply's trip back.
|
|
3. The builder runs `nix build` of `nix/builder/check-flake`'s `check` output
|
|
with every input overridden:
|
|
- `hyperhive` and `nixpkgs`: the sources the builder's own host was
|
|
deployed with (`deploy.hyperhiveFlake`, `deploy.builder.nixpkgsFlake`). A hive's meta
|
|
flake pins `hyperhive` by forge URL and rev instead.
|
|
- `agent`: `git+https://<forge>/agent-configs/<agent>?ref=refs/pull/<pr>/head&rev=<head>`.
|
|
A build has one hour.
|
|
4. The reply is the system closure's store path, or the last lines `nix`
|
|
printed. The node writes the status on the head commit:
|
|
|
|
| Outcome | Status | Description | Node |
|
|
| ----------------------------------------- | --------- | --------------------------- | ------ |
|
|
| built | `success` | the store path | done |
|
|
| `nix` failed or timed out | `failure` | the last line `nix` printed | failed |
|
|
| the builder busy for 30 seconds | `failure` | why | failed |
|
|
| no builder listening, or no reply in time | `error` | why | failed |
|
|
|
|
The controller records every build that gets a final status, and the
|
|
status links to that record: see [Build records](#build-records). The
|
|
full tail is also in the failed node's error on the Jobs page and in the
|
|
builder's journal line for the build, as the field `F_ERROR_TAIL`:
|
|
|
|
```console
|
|
$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-builder.service" _msg:="agent config build failed" | fields _time, F_AGENT, F_PR, F_REV, F_ERROR_TAIL' --format json
|
|
```
|
|
|
|
The builder holds the last 64 KiB of each of `nix`'s output streams while the
|
|
build runs, and the reply is the last 30 lines of that (at most 4000
|
|
bytes), so a long build log never grows its memory.
|
|
|
|
## Build records
|
|
|
|
`swarm-controller` records each build when it writes the final status:
|
|
built, failed, or no build ran or came back. The newest 100 are one JSON
|
|
array under the key `recent` in the `builder-runs` KV bucket on the swarm
|
|
queue, which only the controller reads or writes. It creates the bucket on
|
|
first use and adds each build with a compare-and-set write, so the records
|
|
survive a controller restart. Recording the 101st drops the oldest. A record
|
|
the controller can't decode stays in the bucket, out of the list, and counts
|
|
toward the 100.
|
|
|
|
Each record has:
|
|
|
|
- `id`: `<agent>-<first 12 characters of the commit>-<started_unix>`
|
|
- `repo` (`agent-configs/<agent>`), `pr` and `head_sha`
|
|
- `started_unix` and `finished_unix`
|
|
- `outcome`: `built`, `failed` or `error`, the status's `success`, `failure`
|
|
or `error`
|
|
- `out_path` for `built`, `error_tail` (the last 30 lines `nix` printed) for
|
|
`failed`, and `detail` (why) for `error`
|
|
|
|
`GET /api/builder/runs` lists them, newest first, and
|
|
`GET /api/builder/runs/<id>` returns one, or 404 when it isn't among the
|
|
newest 100. Both are behind the swarm UI vhost's `admins` gate, and answer 503
|
|
when the controller has no swarm queue or can't open the bucket, 500 when it
|
|
can't read it.
|
|
|
|
The final status links to `https://<swarm domain>/jobs?run=<id>`, the swarm
|
|
UI's jobs tab, where the builds are a card under the job graph. Selecting
|
|
**Details** on a red `swarm-builder` check opens that build's card, with `nix`'s tail, in the browser. The raw record
|
|
stays at `GET /api/builder/runs/<id>`. A status has no link when the record wasn't
|
|
stored (the controller logs the failed write) or when the controller doesn't
|
|
serve the swarm vhost (`SWARM_CONTROLLER_PUBLIC_URL` unset). The `pending`
|
|
status has no link.
|
|
|
|
## What the check is, and isn't
|
|
|
|
- It composes the config the way a hive's meta flake does
|
|
(`agent-base.extendModules` with the config's `nixosModules.default`), minus
|
|
what a hive adds: its CA bundle, `claude-code` path, ports and
|
|
OTEL settings, and the `follows` that point the config's own inputs at the
|
|
hive's. A green check can therefore still fail on a hive, and the reverse.
|
|
- The agent base is the builder's host's, not the hive's. Hives on another
|
|
hyperhive or nixpkgs pin build something else.
|
|
- The check pushes nothing. The closure stays in the builder
|
|
host's store until it's collected, so on a host that also runs the hive, the
|
|
hive's own build can reuse whatever matches. A [meta build](#deploy-gate)
|
|
can push to [a binary cache](#binary-cache).
|
|
- The PR check gates nothing. A merge deploys whatever the status says. The
|
|
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
|
|
- A lost webhook delivery isn't retried: the head isn't built until the next
|
|
push. The jobs graph is in memory, so a controller restart drops a build in
|
|
flight and leaves its status `pending`.
|
|
|
|
## Archetype evaluation
|
|
|
|
The builder also answers `$SWARM.builder.archetypes`: it evaluates the
|
|
`meta` of every agent archetype its own hyperhive source declares, with no
|
|
forge token, and answers with the JSON. An evaluation doesn't wait for a
|
|
build in progress and has five minutes; the controller waits six. The builder
|
|
logs each evaluation's time as `elapsed_ms`, and a timeout's error carries the last
|
|
lines nix printed. What the controller does with the answer: [`archetypes.md`](archetypes.md).
|
|
|
|
## Meta-flake lock
|
|
|
|
The builder also answers `$SWARM.builder.lock-meta`. The request carries a
|
|
rendered `flake.nix`. The builder writes it into a scratch directory, runs
|
|
`nix flake lock` there with the same forge credential helper its builds use,
|
|
answers with the `flake.lock` that `nix` wrote, and removes the directory. A
|
|
lock doesn't wait for a build in progress and has ten minutes. The builder
|
|
commits nothing.
|
|
|
|
swarm-controller sends the request. It renders `internal/meta`'s `flake.nix`
|
|
with the `hive-meta-flake` renderer, all over `https`.
|
|
Each custom agent that a hive places pins to its `agent-configs/<agent>`
|
|
`main`; the controller omits and logs one whose repo has no commit on `main`, so it
|
|
doesn't block the rest. Hyperhive pins to one rev, a ratchet on the controller's own
|
|
running rev (`services.hyperhive.deploy.hive-controller.hyperhiveRev`, from `self.rev`):
|
|
|
|
- the first commit pins hyperhive's forge `main`;
|
|
- later, the pin moves to the controller's rev only when the forge's compare
|
|
API shows that rev is a strict descendant of the committed pin: commits
|
|
ahead of it, and none the pin has that it lacks. The compare endpoint counts
|
|
histories with no common ancestor as zero commits ahead, so the check needs
|
|
both directions;
|
|
- an older controller, a rev on another line, a controller built from a dirty
|
|
tree or a forge that can't answer leaves the pin. A merge to hyperhive's
|
|
`main` moves nothing.
|
|
|
|
The controller commits `flake.nix` and the returned `flake.lock` to
|
|
`internal/meta` in one commit, and gives each hive's forge account read on the
|
|
repo. It re-renders when it starts, and again every ten minutes while the latest
|
|
update hasn't succeeded (a tick adds nothing while one is pending or running,
|
|
or after a success), before it deploys a merged config PR, and
|
|
before the first deploy of a new agent. Those deploys run only after the
|
|
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
|
|
builds only from `internal/meta`, so after a failed update the scheduler skips
|
|
the deploy node, and the graph shows the update as failed and the deploy as
|
|
skipped. Each ten-minute retry of a failed update declares a fresh build and
|
|
deploy behind it for every agent whose latest deploy the failed update skipped,
|
|
so the run that succeeds sends them (see
|
|
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
|
|
equal to the committed `flake.nix` sends no lock.
|
|
|
|
The controller sends a deploy to every agent a hive places whenever the pin in
|
|
`internal/meta` differs from the pin it last sent, so they rebuild on the new
|
|
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
|
|
which only the controller reads or writes, and a `RecordFanout` job writes the
|
|
key once every deploy of the fan-out has finished or the deploy gate has skipped
|
|
it. The check runs after each update and once when the controller starts.
|
|
|
|
- The first commit sends a deploy, because the key holds no pin yet.
|
|
- A controller that starts with no key, as after the upgrade that added it,
|
|
sends one round. A deploy is idempotent per pin, so the round is harmless.
|
|
- A controller that dies after the commit, or a deploy that fails, leaves the
|
|
key behind. The next update or start sends the deploys again.
|
|
- A deploy the gate skipped counts as sent. A failed build holds only its own
|
|
agent back, and the key still moves, so one broken agent doesn't resend the
|
|
healthy ones on every retry. The timer declares a build and deploy again for
|
|
a deploy skipped because its build failed.
|
|
- A restart that finds the key equal to the pin sends nothing.
|
|
- When the bucket won't open, or its key won't read, the controller logs a
|
|
warning and sends deploys only when its own update moved the pin. That case
|
|
writes no key.
|
|
|
|
## Deploy gate
|
|
|
|
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
|
|
the flake a hive builds from. Four paths send deploys, and each declares the same
|
|
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
|
|
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
|
|
and needs the one `builder` slot, so it queues behind PR checks. The deploy
|
|
runs only when the build is green.
|
|
|
|
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
|
|
most paths declare it before the update that commits that head. It sends the
|
|
agent and that commit on `$SWARM.builder.build-meta` and waits up to 61
|
|
minutes. The builder takes the same one-build slot as PR checks, and
|
|
refuses a request that waited 30 seconds for it. It runs `nix build` of
|
|
`nixosConfigurations.<agent>.config.system.build.toplevel` from
|
|
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
|
|
the committed `flake.lock` decides every input. A green build here means the
|
|
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
|
|
this build produced and never builds one itself.
|
|
|
|
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
|
|
Jobs page as a failed `build_meta` node, and the timer builds it again:
|
|
|
|
| What happened | `build_meta` node | Deploy |
|
|
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
|
|
| built | done, closure recorded | runs |
|
|
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
|
|
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
|
|
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
|
|
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
|
|
|
|
On every ten-minute round the timer declares a build and deploy again for each
|
|
agent whose latest build failed, without a meta update when the latest update
|
|
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
|
|
still records the pin when the gate skips a deploy, so one failing agent doesn't
|
|
resend the healthy ones. A controller that updates before its builder host does
|
|
is the same case: the old builder has no subscriber for the new subject, so the
|
|
controller gets no responders at once, and the deploys go out after the builder
|
|
updates. Each round adds a build and deploy node per failing agent.
|
|
|
|
When the build belongs to a merged config PR and `nix` exited with an error or hit
|
|
its deadline, the controller also comments once on that PR, with its forge
|
|
account: the agent, the `internal/meta` commit of the first red build, `nix`'s last lines (at most 4000
|
|
bytes), and that the deploy waits and the swarm builds it again every round. A comment
|
|
carries a marker line with the agent, and the controller posts only if the PR
|
|
holds no comment of its own with that marker; a copy of the marker in someone else's comment is neither counted nor edited. The retry rounds stay quiet whatever commit
|
|
they build, and the latest error stays on the Jobs page. A green build of the
|
|
agent for that PR rewrites the marker and appends the build's commit to the comment, so a later red comments again.
|
|
A `no verdict` failure, and a build that isn't a
|
|
merged PR's (a new agent, the pin fan-out), leave no comment. When the forge
|
|
refuses the comment, the controller logs a warning and the node ends as before.
|
|
|
|
The controller exports the gauge `swarm.deploys.held`: the agents whose latest
|
|
build failed, with the label `reason` set to `failed` or `unavailable`. The Grafana
|
|
alert `hh-deploys-held` fires when the sum stays above zero for 30 minutes, so
|
|
a dead builder or a config that fails to build shows in Grafana before anyone notices an agent
|
|
not updating.
|
|
|
|
On a built reply the controller writes the agent's key in the `agent-closures`
|
|
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
|
|
platform and the time. Only the controller writes the bucket, and the builder
|
|
holds no grant on it. A failed build writes nothing and leaves the previous key.
|
|
The record's `cached` field says whether the builder pushed the closure to
|
|
[the binary cache](#binary-cache). Each hive reads its own agents' keys to
|
|
[report whether it can pull them and download them in a rebuild](#what-hives-do-with-the-cache). The closure stays unrooted in the builder host's store.
|
|
|
|
The controller gives the `builder` forge account read on `internal/meta` when it
|
|
updates the flake, and again before each build, since the repo is private. A
|
|
grant the forge refuses fails the build with no verdict.
|
|
|
|
## Binary cache
|
|
|
|
`services.hyperhive.swarm.attic.domain` and `cache` default to the swarm's own
|
|
cache (`attic.<swarm domain>` and `swarm`) on every host, so the builder pushes
|
|
each green meta build's closure to it, or to the cache you override them with,
|
|
before it replies. It runs `attic push swarm:<cache> <out path>` once per build,
|
|
not `attic watch`, and the builder pushes again when the controller runs a build again. The
|
|
push has ten minutes, or less when the reply is due sooner, because it must end
|
|
inside the 61 minutes the controller waits. `attic` checks the server's certificate
|
|
against the system roots and pins nothing, so a Let's Encrypt certificate
|
|
works.
|
|
|
|
Set both options to a name or both to null: evaluation refuses one without the
|
|
other. Null in both is how a swarm opts out of pushing.
|
|
The cache name follows attic's rule, 1 to 50 letters, digits, `-`, `_` or `+`,
|
|
starting with a letter or digit, and evaluation refuses any other name.
|
|
|
|
The push token is the swarm's own cache's to create, or the operator's to write for an
|
|
external one. The builder reads it on every push from `secret/swarm/services/attic/push-token` (field `value`), writes it to
|
|
a file and an `attic` config naming that file (`token-file`) in the unit's private
|
|
runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
|
|
`attic` at the config with `XDG_CONFIG_HOME` and removes both after the push.
|
|
The token is in no argument or variable, so no process listing shows it.
|
|
|
|
The reply's `cached` says whether the push finished. A push that fails, times out
|
|
or can't read the token leaves the build green with `cached` false and logs a
|
|
warning with the reason; the deploy goes ahead and the next build of the agent
|
|
tries again. With no attic behind the defaults, every push fails this way. With
|
|
both options null, the builder pushes nothing and `cached` is always false.
|
|
|
|
Mixed versions:
|
|
|
|
| controller | builder | what happens |
|
|
| ---------- | ------- | ------------------------------------------------------------- |
|
|
| new | new | `cached` in the reply and in the `agent-closures` record |
|
|
| new | old | the reply has no `cached`, which reads as `false` |
|
|
| old | new | the controller ignores `cached`; its record has no such field |
|
|
|
|
A record written before the field existed also reads as `false`.
|
|
|
|
### Rebuild on a cache miss
|
|
|
|
The cache drops a closure nobody fetched for its retention period, but the
|
|
record in `agent-closures` stays. Every ten minutes `swarm-controller` checks
|
|
each agent a hive places: when the record is from `internal/meta`'s head and
|
|
says the builder pushed it, the controller fetches the narinfo from
|
|
`<url>/<cache>/<hash>.narinfo` with the pull token it reads from
|
|
`secret/swarm/services/attic/` (the same four leaves it
|
|
[copies to the hives](#what-hives-do-with-the-cache); with any missing, or a URL
|
|
that isn't `https`, the pass does nothing). A 404 is a miss, and the controller
|
|
sends the builder `build-meta` for that agent and the record's own commit. The
|
|
builder builds it and pushes as for any meta build. Any other answer or a failed
|
|
request isn't a miss, and rebuilds nothing. The next ordinary build handles a record
|
|
from an older commit, or one with `cached` false.
|
|
|
|
A pass sends at most three builds, one after the other. After an agent's rebuild
|
|
failed, got no answer or built without pushing, the pass skips it for 30 minutes, then
|
|
for twice as long after each further failure, up to 12 hours. The wait is in memory,
|
|
so a controller restart tries every agent again. The first pass runs ten
|
|
minutes after the controller starts.
|
|
|
|
The record keeps naming what hives deploy. A rebuild that lands on the record's
|
|
path writes nothing. A rebuild that lands on another path means the build isn't
|
|
reproducible: the controller logs a warning, and replaces the record only if it
|
|
still holds the same commit and no write came since the controller read it. The
|
|
record otherwise stays as it was.
|
|
|
|
The controller counts each rebuild in `hyperhive.builder.cache_miss_rebuilds` with
|
|
the label `outcome`: `rebuilt`, `path_changed` (the record now names the new path),
|
|
`path_changed_kept` (the record stayed), `not_pushed`, `failed` (`nix` failed) or
|
|
`unavailable` (the builder was busy, absent or gave no answer).
|
|
|
|
Mixed versions: a builder needs no change, since a meta build already names
|
|
its commit. An old controller does nothing about a miss, as before. A hive
|
|
sees nothing new.
|
|
|
|
### The swarm's own cache
|
|
|
|
`services.hyperhive.deploy.attic.enable` runs an attic server in a `swarm-attic`
|
|
container behind the gateway, at `attic.<swarm domain>`. It defaults on where
|
|
you set `deploy.allSwarmServices` and the host holds the cache's store identity
|
|
(`deploy.bao.atticClient{Cert,Key}File`, which the store's host mints). You write
|
|
no secret: on first start, two host units log in to the store as the cache's own
|
|
principal and create what's missing.
|
|
|
|
- `swarm-bao-attic-secret` creates the key atticd signs tokens with at
|
|
`secret/swarm/services/attic/jwt-secret` and writes it where the container
|
|
reads it.
|
|
- `swarm-bao-attic-tokens` creates the cache and mints the push token, which
|
|
the builder reads, at `secret/swarm/services/attic/push-token`, and a
|
|
read-only pull token for the hives at `secret/swarm/services/attic/pull-token`.
|
|
It also stores the facts a hive pulls with: the cache's address at
|
|
`secret/swarm/services/attic/url` (`https://<swarm.attic.domain>`), its name at
|
|
`secret/swarm/services/attic/cache`, and the public key it signs with at
|
|
`secret/swarm/services/attic/public-key` (the key `attic cache info` prints).
|
|
|
|
A unit creates each secret only while the store holds none, so a second start
|
|
mints nothing. Both tokens last ten years and nothing renews them. To replace one,
|
|
write the new token to its path in the store, in its UI under the OIDC login:
|
|
nothing mints over a value that exists. A new signing key invalidates both tokens.
|
|
|
|
The unit creates the address, name and key once and never updates them. After
|
|
you change `swarm.attic.domain`, or after the server's database goes and the
|
|
cache gets a new key, delete the stale leaf and restart
|
|
`swarm-bao-attic-tokens`.
|
|
|
|
`swarm.attic.domain` and `swarm.attic.cache` default to `attic.<swarm domain>`
|
|
and `swarm` on every host, whether or not it runs the cache, so a builder on any
|
|
host pushes to it.
|
|
|
|
Storage is local disk in the container's state (`/var/lib/swarm-attic`). A store
|
|
path nobody added or fetched for 14 days goes at the next garbage collection,
|
|
which runs every 12 hours (`deploy.attic.retentionPeriod`). The controller
|
|
[rebuilds a closure the cache lost](#rebuild-on-a-cache-miss).
|
|
|
|
The gateway terminates TLS with a swarm services certificate and forwards to
|
|
`127.0.0.1:8910` (`deploy.attic.port`) with no login in front: attic checks its
|
|
own tokens, and a nix client can't follow a login redirect. The container logs
|
|
to its journal only; its journal doesn't reach the swarm log store.
|
|
|
|
### External cache
|
|
|
|
The cache can be an attic tenant on an instance outside the swarm. To point
|
|
the swarm at an existing tenant:
|
|
|
|
1. Override the defaults: set `services.hyperhive.swarm.attic.domain` to the
|
|
instance's host name (no scheme, `host:port` if it isn't on 443) and
|
|
`services.hyperhive.swarm.attic.cache` to the cache's name on it. Set
|
|
both options on every host that reads them.
|
|
2. Seed the push token in the swarm secret store at
|
|
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI under
|
|
the OIDC login, or `bao kv put` with `value=-` and the token on stdin, so the token is in no
|
|
command line. The token needs push rights on that cache. Until it
|
|
exists, every push fails and builds stay green with `cached` false.
|
|
Seed the hives' pull token the same way at
|
|
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
|
|
3. Seed the three facts hives pull with, each as field `value`:
|
|
`secret/swarm/services/attic/url` (`https://` and the host, no trailing
|
|
slash; hives refuse `http://`), `secret/swarm/services/attic/cache` (the cache's name) and
|
|
`secret/swarm/services/attic/public-key` (`<name>:<base64>`, from the
|
|
cache's page or `attic cache info`). Until all four exist, hives report
|
|
`no-credentials`.
|
|
|
|
The builder checks the instance's certificate against the system roots and
|
|
pins nothing, so a Let's Encrypt certificate needs no setup. The builder
|
|
reads the token again on every push, so a new token takes effect on the next
|
|
build.
|
|
|
|
To use an external cache, override the two defaults and don't set
|
|
`deploy.attic.enable` anywhere. Where you host the shared services with
|
|
`allSwarmServices`, it defaults on, so set it to `lib.mkForce false` there.
|
|
The swarm then renders no container, vhost or unit of its own cache, and
|
|
creates no secret in the store. The store's host still mints the cache's
|
|
leaf and writes its role, which nothing logs in to.
|
|
|
|
A host that has `deploy.attic.enable` on and `swarm.attic.domain` set to another
|
|
name gets an evaluation warning that names this fix.
|
|
|
|
## What hives do with the cache
|
|
|
|
Every five minutes `swarm-controller` copies the four `swarm/services/attic/`
|
|
leaves (`pull-token`, `url`, `cache`, `public-key`) to
|
|
`swarm/hives/<hive>/attic/<leaf>` for each hive, because a hive reads only its
|
|
own prefix. It writes a copy only when the value differs, skips a leaf the
|
|
store doesn't hold, and leaves a copy whose source disappeared.
|
|
|
|
`hive-c0re` reads its four leaves every five minutes under the hive's own store
|
|
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
|
|
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the
|
|
token goes out in a header and in the netrc; a hive that finds another scheme
|
|
writes nothing.
|
|
|
|
`hive-priv` doesn't read those files. The key that must have signed a download and
|
|
the pull token come from `hive-priv-cache-facts`, a root unit that reads the
|
|
hive's own `attic/public-key` and `attic/pull-token` from the store under the
|
|
hive's own store identity every ten minutes and writes them to
|
|
`/run/hyperhive-priv/attic/` (`0700`, root; each file `0600`). The cache's
|
|
address is the host's `services.hyperhive.swarm.attic.domain` and `cache`,
|
|
handed to `hive-priv` in its unit's environment. A file `hive-c0re` writes
|
|
decides nothing `hive-priv` trusts.
|
|
|
|
The same pass checks each of the hive's agents. It reads the agent's record in
|
|
`agent-closures`, then fetches the closure's narinfo from
|
|
`<url>/<cache>/<hash>.narinfo` with the pull token as the basic-auth password.
|
|
It checks that a signature by the public key covers the narinfo, in Rust, with
|
|
no help from the host's nix daemon. It downloads nothing. The result is a log
|
|
line (at info when it changes) and the gauge
|
|
`hyperhive.agent.closure_pullable{agent}` (the `hive` label comes from the
|
|
collector, as for every `hyperhive.*` metric): `1` when the hive can pull the
|
|
latest closure, else `0` with a `reason` label. The first check that fails
|
|
decides the reason:
|
|
|
|
| `reason` | Meaning |
|
|
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `no-credentials` | One of the four leaves is missing or unusable (a URL that isn't `https` is unusable). The swarm has no cache, the controller is too old to copy them, or you haven't seeded them yet. |
|
|
| `no-record` | `agent-closures` holds no build for the agent. |
|
|
| `rev-mismatch` | The record's build came from another `internal/meta` commit than its head, or the hive can't read the head. |
|
|
| `not-cached` | The record says the builder didn't push the closure, or the cache answers 404 for the narinfo. |
|
|
| `fetch-failed` | The request failed, or the cache answered with a status other than 200 or 404 (a refused token is a 401 or 403). |
|
|
| `bad-signature` | The narinfo describes another path, or no signature by the public key covers it. |
|
|
|
|
A failed read of the store, the queue or the forge leaves the last result in
|
|
place and raises a warning after three failed passes. A hive reports nothing
|
|
before its first pass.
|
|
|
|
Mixed versions:
|
|
|
|
| controller | hive | what happens |
|
|
| ---------- | ---- | ----------------------------------------------------------------------------------- |
|
|
| new | new | the copies exist and the hive reports per agent |
|
|
| old | new | the controller copies nothing, so the hive reports `no-credentials` for every agent |
|
|
| new | old | the controller writes copies nobody reads |
|
|
| any | any | a swarm with no cache seeds no leaves, and every hive reports `no-credentials` |
|
|
|
|
The signature check follows Nix's fingerprint format but hasn't run against a
|
|
live attic. A `bad-signature` on every agent in the first pass after a deploy
|
|
means check attic's fingerprint format; it says nothing about an attacker. A
|
|
`fetch-failed` on every agent points at the URL, the token or the network.
|
|
|
|
### Rebuilds download the closure
|
|
|
|
A hive builds nothing. A rebuild's `Prebuild` node, which runs while the agent is
|
|
still up, reads the agent's record in `agent-closures` and asks `hive-priv` to
|
|
realise the record's store path (`RealiseClosure`). `hive-priv` runs
|
|
`nix-store --realise` with `--max-jobs 0`, so nix can't build the path, and with
|
|
the host's configured cache address, and the public key and netrc from the
|
|
root-owned `/run/hyperhive-priv/attic/`, as per-call settings. The settings replace the host's `substituters` and
|
|
`trusted-public-keys` and set `require-sigs true`: the swarm cache is the only
|
|
source, and nix stores the path only when the cache's key signed it. The host's
|
|
nix configuration stays as it was.
|
|
|
|
`Swap` then does the same lookup and download, a no-op once the path is in the
|
|
store, and gives `hive-priv` the store path to apply. A stopped agent skips
|
|
`Prebuild`, so `Swap` downloads it. A first deploy's `Create` node does the
|
|
same and creates the container from that path.
|
|
|
|
A rebuild or create has three outcomes, by what the lookup and the download find:
|
|
|
|
| Found | What happens |
|
|
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| a record that the builder pushed, for the commit this hive chose | The hive downloads the closure and deploys it. The hive logs `prebuild: substituted <path>` (`create: substituted <path>` for a create) at info level. |
|
|
| a record that the builder pushed, for another commit (the hive and the builder disagree) | The hive downloads the builder's closure and deploys it, with a warning that names the agent, both commits and the path. The hive evaluates nothing, and records the pins of the commit the builder built the closure from, so the agent's markers say what runs. |
|
|
| no record, or one the hive can't use (not pushed, queue or cache unreachable, download or signature failed) | The rebuild or create fails with an error that names the agent and the reason, logged at error level, and nothing changes. A running container keeps running unchanged, because `Prebuild` fails before `StopForUpdate`. A create makes no container. |
|
|
|
|
The hive never falls back to a local build. The third row has one owner in the
|
|
code, `without_closure` in `hive-c0re/src/job_queue/exec.rs`, so a change to what
|
|
an unreachable cache costs is one function.
|
|
|
|
Mixed versions:
|
|
|
|
| hive | controller or record | what happens |
|
|
| ---- | -------------------- | ------------------------------------------------------------------------------------------ |
|
|
| old | any | builds locally, as before |
|
|
| new | old, or no record | refuses the create; a running agent keeps running, and the rebuild logs an error naming it |
|
|
| new | cache down | the download fails; same as no record, with the download's error in the log |
|
|
| new | signature mismatch | nix refuses the path; same as no record, and `hive-priv` logs the refusal as well |
|
|
| new | old `hive-priv` | it refuses the request, which has no `meta_rev`, so the swap or create fails |
|
|
|
|
## Credentials
|
|
|
|
| What | Where | Who reads it |
|
|
| ----------------------- | ------------------------------------------------- | -------------------------------------------------- |
|
|
| store leaf | `deploy.bao.builderClient{Cert,Key}File` | nspawn, into the service as a credential |
|
|
| queue client secret | `secret/swarm/services/swarm-builder/oidc/client` | the builder, at start and when authelia refuses it |
|
|
| forge token (read-only) | `secret/swarm/services/builder/forge/token` | the builder, on every build |
|
|
| cache push token | `secret/swarm/services/attic/push-token` | the builder, on every push |
|
|
|
|
The token belongs to the `builder` forge account, which swarm-controller
|
|
creates, puts in the `hives` team of `agent-configs` (read on every config
|
|
repo, and on `internal/meta` through the meta update) and mints a
|
|
`read:repository` token for, in a five-minute pass. `git`
|
|
gets the token through a credential helper reading the child's environment:
|
|
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
|
status. The same `git` fetches a config's own inputs, so the builder registers
|
|
the helper for the forge's URL only, and the helper answers only an `https`
|
|
request for the forge's host: any other host gets nothing. The builder holds no forge write credential; swarm-controller writes the
|
|
status with its own forge token. Details per path:
|
|
[`credentials-reference.md`](credentials-reference.md).
|
|
|
|
## Isolation
|
|
|
|
The container has its own network namespace on the bridge, as the CI runner
|
|
does, so a config's evaluation can't reach a host-loopback service. Evaluation
|
|
is pure and runs in the container. Builds go to the host's nix daemon as an
|
|
untrusted client: a config can set no daemon option, and its derivations build
|
|
in the host's sandbox. A fixed-output derivation still has the network, as it
|
|
does for any build on that host.
|