mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm: a deploy goes out only once its closure is cached and the hive has its cache facts

Browse source
This commit is contained in:
atlas 2026-11-07 11:53:25 +01:00
commit 79c1b6d81d
1 changed files with 432 additions and 92 deletions
Download patch file Download diff file Expand all files Collapse all files

73
docs/4-implementation/swarm/builder.md
View file

@ -196,8 +196,8 @@ it. The check runs after each update and once when the controller starts.
key behind. The next update or start sends the deploys again.
- A deploy the gate skipped counts as sent. A failed build holds only its own
agent back, and the key still moves, so one broken agent doesn't resend the
healthy ones on every retry. The timer declares a build and deploy again for
a deploy skipped because its build failed.
healthy ones on every retry. The timer declares a build, a copy and a deploy
again for a deploy skipped because its build or its facts copy failed.
- A restart that finds the key equal to the pin sends nothing.
- When the bucket won't open, or its key won't read, the controller logs a
warning and sends deploys only when its own update moved the pin. That case
@ -205,12 +205,14 @@ it. The check runs after each update and once when the controller starts.
## Deploy gate
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
the flake the swarm builds from. Four paths send deploys, and each declares the same
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
and needs the one `builder` slot, so it queues behind PR checks. The deploy
runs only when the build is green.
Every deploy the swarm sends waits for two things: a build of its agent from
`internal/meta`, the flake the swarm builds from, with the closure pushed to the
cache, and a copy of the cache facts to the hive it goes to. Four paths send
deploys, and each declares the same three nodes: a merged config PR, a new
agent, the pin fan-out and the ten-minute retry. The `build_meta` node and the
`copy_attic_facts` node come before the `trigger_deploy` node. The build needs
the one `builder` slot, so it queues behind PR checks. The deploy runs only when
both have finished.
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
most paths declare it before the update that commits that head. It sends the
@ -223,25 +225,41 @@ the committed `flake.lock` decides every input. A green build here means the
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
this build produced and never builds one itself.
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
Jobs page as a failed `build_meta` node, and the timer builds it again:
A build whose closure the builder didn't push (`cached` false in the reply: the
push failed or timed out, the token was unreadable or the swarm has no cache)
ends `failed` with `no verdict: …`, because a hive that gets a deploy for it
can't download the closure and refuses the deploy. The controller still writes
the record, with `cached` false, so the [cache-miss pass](#rebuild-on-a-cache-miss)
and the cache probe see it.
The `build_meta` node ends `done` or `failed`. Every failure skips the deploy and
shows on the Jobs page as a failed `build_meta` node, and the timer builds it
again:
| What happened | `build_meta` node | Deploy |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
| built | done, closure recorded | runs |
| built and pushed to the cache | done, closure recorded | runs |
| built, but the builder didn't push it | failed, `no verdict: …`, closure recorded | skipped |
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
On every ten-minute round the timer declares a build and deploy again for each
agent whose latest build failed, without a meta update when the latest update
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
The `copy_attic_facts` node copies the swarm's four cache facts to the one hive
the deploy goes to ([below](#what-hives-do-with-the-cache)), with the code the
five-minute pass uses. It ends `done` when the hive's copies are current or
written, including when the swarm stores no facts, and `failed` with the cause
when it can't read, log in to or write the store, or the copy takes over
two minutes. A failed copy skips the deploy.
On every ten-minute round the timer declares a build, a copy and a deploy again
for each agent whose latest build or copy failed, without a meta update when the
latest update succeeded, until both succeed. Nothing limits the retries. The pin fan-out
still records the pin when the gate skips a deploy, so one failing agent doesn't
resend the healthy ones. A controller that updates before its builder host does
is the same case: the old builder has no subscriber for the new subject, so the
controller gets no responders at once, and the deploys go out after the builder
updates. Each round adds a build and deploy node per failing agent.
updates. Each round adds a build, a copy and a deploy node per failing agent.
A merged config PR whose webhook delivery never arrives would leave the agent
on its old config. Every five minutes, after the config-PR scan, the controller
@ -335,11 +353,14 @@ runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
The token is in no argument or variable, so no process listing shows it.
The reply's `cached` says whether the push finished. A push that fails, times out
or can't read the token leaves the build green with `cached` false and logs a
warning with the reason; the deploy goes ahead, and the controller sends the
same build again until the push lands ([below](#rebuild-on-a-cache-miss)). With
no attic behind the defaults, every push fails this way. With both options null,
the builder pushes nothing and `cached` is always false.
or can't read the token leaves the reply `Built` with `cached` false and logs a
warning with the reason. The controller ends that build's `build_meta` node
`failed`, which holds the deploy, and sends the same build again on the next
ten-minute round until the push lands ([deploy gate](#deploy-gate)); the
[cache-miss pass](#rebuild-on-a-cache-miss) also sends it for a record that says
the builder didn't push. With no attic behind the defaults, every push fails this
way. With both options null, the builder pushes nothing and `cached` is always
false, so no deploy goes out.
Mixed versions:
@ -351,6 +372,11 @@ Mixed versions:
A record written before the field existed also reads as `false`.
The deploy gate is controller-only and changes no message. A new controller
holds the deploy of a closure the builder didn't push; an old controller sends
it, and the hive refuses it. Hives need no change: a hive of any version gets
fewer deploys, and none for a closure it couldn't download.
### Rebuild on a cache miss
The cache drops a closure nobody fetched for its retention period, but the
@ -461,7 +487,7 @@ the swarm at an existing tenant:
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI under
the OIDC login, or `bao kv put` with `value=-` and the token on stdin, so the token is in no
command line. The token needs push rights on that cache. Until it
exists, every push fails and builds stay green with `cached` false.
exists, every push fails and no deploy goes out.
Seed the hives' pull token the same way at
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
3. Seed the three facts hives pull with, each as field `value`:
@ -494,6 +520,11 @@ leaves (`pull-token`, `url`, `cache`, `public-key`) to
own prefix. It writes a copy only when the value differs, skips a leaf the
store doesn't hold, and leaves a copy whose source disappeared.
A deploy waits for the same copy: its `copy_attic_facts` node copies for the one
hive the deploy goes to, so the facts are in the hive's prefix before the swarm tells the hive
to deploy ([deploy gate](#deploy-gate)). That puts the facts in the
store only; the hive reads them from there on its own schedule.
`hive-c0re` reads its four leaves every five minutes under the hive's own store
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the