Watch
0
0
Fork
You've already forked hyperhive
0

gate each deploy on a swarm build of the agent from internal/meta

This commit is contained in:
atlas 2026-10-26 16:40:06 +01:00
commit 5e64048e53
3 changed files with 1570 additions and 178 deletions

View file

@ -149,7 +149,9 @@ every agent any hive places (the pins:
every agent from it. While the repo has no commit that pins an agent's current
config, that agent's build fails and names the agent. The controller queues
its update of the repo before the deploy request for a merged config commit
and for a new agent, and sends the deploy only when that update succeeded.
and for a new agent, and sends the deploy only when that update succeeded and
the swarm builder built the agent from the updated repo (the
[deploy gate](../swarm/builder.md#deploy-gate)). A red build skips the deploy.
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
default-branch head with the hive's read token (`forge::reader_token()`)

View file

@ -6,6 +6,10 @@ push, the builder builds the PR's head on the agent base, and the result shows a
`swarm-builder` commit status on the PR and as a `build_agent` node on the
swarm UI's Jobs page.
The builder also builds each agent from `internal/meta` before the swarm sends
its deploy, and a red build holds the deploy back: see
[Deploy gate](#deploy-gate).
Where it runs and how it's enabled: [`services.md`](services.md#builder).
## What happens on a config PR
@ -102,7 +106,8 @@ status has no link.
- The builder pushes nothing. The closure stays in the builder
host's store until it's collected, so on a host that also runs the hive, the
hive's own build can reuse whatever matches.
- It doesn't gate anything. A merge deploys whatever the status says.
- The PR check gates nothing. A merge deploys whatever the status says. The
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
- A lost webhook delivery isn't retried: the head isn't built until the next
push. The jobs graph is in memory, so a controller restart drops a build in
flight and leaves its status `pending`.
@ -148,11 +153,12 @@ repo. It re-renders when it starts, and again every ten minutes while the latest
update hasn't succeeded (a tick adds nothing while one is pending or running,
or after a success), before it deploys a merged config PR, and
before the first deploy of a new agent. Those deploys run only after the
update succeeded: a hive builds only from `internal/meta`, so after a failed
update the scheduler skips the deploy node, and the graph shows the update as
failed and the deploy as skipped. Each ten-minute retry of a failed update
declares a fresh deploy behind it for every agent whose latest deploy the failed update
skipped, so the run that succeeds sends them (see
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
builds only from `internal/meta`, so after a failed update the scheduler skips
the deploy node, and the graph shows the update as failed and the deploy as
skipped. Each ten-minute retry of a failed update declares a fresh build and
deploy behind it for every agent whose latest deploy the failed update skipped,
so the run that succeeds sends them (see
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
equal to the committed `flake.nix` sends no lock.
@ -160,19 +166,72 @@ The controller sends a deploy to every agent a hive places whenever the pin in
`internal/meta` differs from the pin it last sent, so they rebuild on the new
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
which only the controller reads or writes, and a `RecordFanout` job writes the
key after every deploy of the fan-out succeeds. The check runs after each update
and once when the controller starts.
key once every deploy of the fan-out has finished or the deploy gate has skipped
it. The check runs after each update and once when the controller starts.
- The first commit sends a deploy, because the key holds no pin yet.
- A controller that starts with no key, as after the upgrade that added it,
sends one round. A deploy is idempotent per pin, so the round is harmless.
- A controller that dies after the commit, or a deploy that fails, leaves the
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A red build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because the build gave no verdict.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
writes no key.
## Deploy gate
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
the flake a hive builds from. Four paths send deploys, and each declares the same
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
and needs the one `builder` slot, so it queues behind PR checks. The deploy
runs only when the build is green.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
agent and that commit on `.builder.build-meta` and waits up to 61
minutes. The builder takes the same one-build slot as PR checks, and
refuses a request that waited 30 seconds for it. It runs `nix build` of
`nixosConfigurations.<agent>.config.system.build.toplevel` from
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
the committed `flake.lock` decides every input. Like the hive's build, it uses
that attribute, so a green build here means the flake evaluates and builds for
the builder's platform. The hive doesn't reuse the result yet: it still builds
locally, from whatever head it reads at deploy time, which can be newer than
the one the swarm built.
The node ends one of three ways:
| Builder's answer | Node | Deploy | Timer |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------- | --------------------------------- |
| built | done, closure recorded | runs | |
| `nix` failed or timed out (a red build) | failed, with `nix`'s last lines | skipped | declares nothing |
| no builder listening, no reply in time, or the builder couldn't start `nix` (busy for 30 seconds, no forge token) | failed, `not built, will be retried` | skipped | declares a build and deploy again |
A red build is a verdict on the config: a config merge queues its own build and
deploy. A build that gave no verdict isn't, so the ten-minute timer declares
it again, without a meta update when the latest update succeeded, until the
builder answers. That covers a controller that updates before its builder
host does: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each retry adds a build and deploy node per affected agent while the
builder stays away.
On a built reply the controller writes the agent's key in the `agent-closures`
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
platform and the time. Only the controller writes the bucket, and the builder
holds no grant on it. A red build writes nothing and leaves the previous key.
Nothing reads the bucket yet. The closure stays unrooted in the builder host's
store.
The controller gives the `builder` forge account read on `internal/meta` when it
updates the flake, since the repo is private.
## Credentials
| What | Where | Who reads it |
@ -183,7 +242,8 @@ and once when the controller starts.
The token belongs to the `builder` forge account, which swarm-controller
creates, puts in the `hives` team of `agent-configs` (read on every config
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
repo, and on `internal/meta` through the meta update) and mints a
`read:repository` token for, in a five-minute pass. `git`
gets the token through a credential helper reading the child's environment:
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
status. The same `git` fetches a config's own inputs, so the builder registers

View file

@ -422,6 +422,11 @@ through the host's nix daemon, and swarm-controller writes the result as a `swar
commit status on the head. The job shows on the swarm UI's Jobs page. A red
status blocks nothing: the merge still deploys.
Separately, the builder builds each agent from `internal/meta` before the swarm
sends its deploy, and a red build holds that deploy back (`build_meta` job;
[deploy gate](builder.md#deploy-gate)). Those jobs share the one-build slot with
the PR checks.
It needs neither the forge nor a hive on its host. It reaches the queue, the
secret store and the forge by name, under its own `builder` leaf, and fetches
the private config repo with the read-only token of its own `builder` forge