swarm: a deploy goes out only once its closure is cached and the hive has its cache facts
This commit is contained in:
atlas
parent
0e84fe8bc4
commit
79c1b6d81d
4 changed files with 432 additions and 92 deletions
|
|
@ -141,7 +141,7 @@ agent. The controller queues
|
|||
its update of the repo before the deploy request for a merged config commit
|
||||
and for a new agent, and sends the deploy only when that update succeeded and
|
||||
the swarm builder built the agent from the updated repo (the
|
||||
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
|
||||
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). A failed build, a closure the builder didn't push or a failed copy of the hive's cache facts skips the deploy, and the timer declares them again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
|
||||
|
||||
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
|
||||
agent's record in `agent-closures` declares, whichever `internal/meta` commit
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/` | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
|
||||
| this hive's cache pull facts | `swarm-controller`, into the secret store at `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}`, copied every five minutes from `swarm/services/attic/`, and before each deploy to the hive | `hive-c0re` reads them under the hive's own store identity and writes `/run/hyperhive/attic/netrc` (the pull token), `substituter` and `public-key` (tmpfs, each `0600` `hive-core`). Only the probe reads them; `hive-priv` doesn't. |
|
||||
|
||||
# Credentials: the per-secret reference
|
||||
|
||||
|
|
@ -50,12 +50,12 @@ the rest of the cell says how.
|
|||
| `swarm/services/review-classifier/acp-provider` | you, in the store's UI, field `api_key` | `swarm-review-classifier` | ❌ write a new key | ✅ read on every run |
|
||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every review |
|
||||
| `swarm/services/builder/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-builder`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every build |
|
||||
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the build stays green with `cached` false | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
|
||||
| `swarm/services/attic/push-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the token on stdin | `swarm-builder` | ❌ write a new token; a token the cache refuses fails the push, and the agent's deploy is held until a push lands | ✅ read on every push; with both `swarm.attic` options null, nothing reads it |
|
||||
| `swarm/services/attic/pull-token` | `swarm-bao-attic-tokens` on the cache's host, when none is stored; read-only, ten years; with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, to copy it and to check narinfos | ❌ write a new token | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/services/attic/url` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`https://<swarm.attic.domain>`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (scheme included, no trailing slash) | `swarm-controller`, which copies it to each hive | ❌ delete it and restart `swarm-bao-attic-tokens`, or write the new value | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/services/attic/cache` | `swarm-bao-attic-tokens` on the cache's host, when none is stored (`swarm.attic.cache`); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin | `swarm-controller`, which copies it to each hive | ❌ as the URL | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/services/attic/public-key` | `swarm-bao-attic-tokens` on the cache's host, when none is stored and again whenever the cache reports another key (the key `attic cache info` prints); with an external cache, you, in the store's UI (field `value`), or `bao kv put` with `value=-` and the value on stdin (`<name>:<base64>`) | `swarm-controller`, which copies it to each hive | ✅ on the cache's host, the unit's next run replaces any other value, a hand edit included, with the cache's key; with an external cache, replace it yourself | ✅ copied to the hives every 5 minutes |
|
||||
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
|
||||
| `swarm/hives/<hive>/attic/{pull-token,url,cache,public-key}` | `swarm-controller`, from the four `swarm/services/attic/` leaves, every 5 minutes, and before each deploy to the hive; it skips a leaf whose source holds nothing, and a copy whose source is gone stays | the hive (`hive-c0re`) | ✅ a changed source is written over the copy in the next pass | ✅ read every 5 minutes |
|
||||
| `swarm/services/attic/jwt-secret` | `swarm-bao-attic-secret` on the cache's host, when none is stored | `swarm-bao-attic-secret`, `swarm-bao-attic-tokens` | ❌ a new key invalidates both tokens | ❌ atticd restarts when the file changes |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
|
|
|||
|
|
@ -487,7 +487,7 @@ commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
|||
status blocks nothing: the merge still deploys.
|
||||
|
||||
Separately, the builder builds each agent from `internal/meta` before the swarm
|
||||
sends its deploy, and a failed build holds that deploy back (`build_meta` job;
|
||||
sends its deploy, and a failed or unpushed build holds that deploy back (`build_meta` job;
|
||||
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). Those jobs share the one-build slot with
|
||||
the PR checks.
|
||||
|
||||
|
|
|
|||
|
|
@ -196,8 +196,8 @@ it. The check runs after each update and once when the controller starts.
|
|||
key behind. The next update or start sends the deploys again.
|
||||
- A deploy the gate skipped counts as sent. A failed build holds only its own
|
||||
agent back, and the key still moves, so one broken agent doesn't resend the
|
||||
healthy ones on every retry. The timer declares a build and deploy again for
|
||||
a deploy skipped because its build failed.
|
||||
healthy ones on every retry. The timer declares a build, a copy and a deploy
|
||||
again for a deploy skipped because its build or its facts copy failed.
|
||||
- A restart that finds the key equal to the pin sends nothing.
|
||||
- When the bucket won't open, or its key won't read, the controller logs a
|
||||
warning and sends deploys only when its own update moved the pin. That case
|
||||
|
|
@ -205,12 +205,14 @@ it. The check runs after each update and once when the controller starts.
|
|||
|
||||
## Deploy gate
|
||||
|
||||
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
|
||||
the flake the swarm builds from. Four paths send deploys, and each declares the same
|
||||
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
|
||||
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
|
||||
and needs the one `builder` slot, so it queues behind PR checks. The deploy
|
||||
runs only when the build is green.
|
||||
Every deploy the swarm sends waits for two things: a build of its agent from
|
||||
`internal/meta`, the flake the swarm builds from, with the closure pushed to the
|
||||
cache, and a copy of the cache facts to the hive it goes to. Four paths send
|
||||
deploys, and each declares the same three nodes: a merged config PR, a new
|
||||
agent, the pin fan-out and the ten-minute retry. The `build_meta` node and the
|
||||
`copy_attic_facts` node come before the `trigger_deploy` node. The build needs
|
||||
the one `builder` slot, so it queues behind PR checks. The deploy runs only when
|
||||
both have finished.
|
||||
|
||||
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
|
||||
most paths declare it before the update that commits that head. It sends the
|
||||
|
|
@ -223,25 +225,41 @@ the committed `flake.lock` decides every input. A green build here means the
|
|||
flake evaluates and builds for the builder's platform. A hive [downloads the closure](#rebuilds-download-the-closure)
|
||||
this build produced and never builds one itself.
|
||||
|
||||
The node ends `done` or `failed`. Every failure skips the deploy and shows on the
|
||||
Jobs page as a failed `build_meta` node, and the timer builds it again:
|
||||
A build whose closure the builder didn't push (`cached` false in the reply: the
|
||||
push failed or timed out, the token was unreadable or the swarm has no cache)
|
||||
ends `failed` with `no verdict: …`, because a hive that gets a deploy for it
|
||||
can't download the closure and refuses the deploy. The controller still writes
|
||||
the record, with `cached` false, so the [cache-miss pass](#rebuild-on-a-cache-miss)
|
||||
and the cache probe see it.
|
||||
|
||||
The `build_meta` node ends `done` or `failed`. Every failure skips the deploy and
|
||||
shows on the Jobs page as a failed `build_meta` node, and the timer builds it
|
||||
again:
|
||||
|
||||
| What happened | `build_meta` node | Deploy |
|
||||
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ------- |
|
||||
| built | done, closure recorded | runs |
|
||||
| built and pushed to the cache | done, closure recorded | runs |
|
||||
| built, but the builder didn't push it | failed, `no verdict: …`, closure recorded | skipped |
|
||||
| `nix` exited with an error | failed, with `nix`'s last lines | skipped |
|
||||
| `nix` hit its one-hour deadline, or couldn't start | failed, `nix timed out: …` or `starting nix: …` | skipped |
|
||||
| no builder listening, no reply in 61 minutes, a reply that doesn't decode, the builder's slot busy for 30 seconds, or no forge token | failed, `no verdict: …` | skipped |
|
||||
| the controller couldn't read the `internal/meta` head, find a commit there, grant the builder read on the repo, or write the closure | failed, `no verdict: …` | skipped |
|
||||
|
||||
On every ten-minute round the timer declares a build and deploy again for each
|
||||
agent whose latest build failed, without a meta update when the latest update
|
||||
succeeded, until a build succeeds. Nothing limits the retries. The pin fan-out
|
||||
The `copy_attic_facts` node copies the swarm's four cache facts to the one hive
|
||||
the deploy goes to ([below](#what-hives-do-with-the-cache)), with the code the
|
||||
five-minute pass uses. It ends `done` when the hive's copies are current or
|
||||
written, including when the swarm stores no facts, and `failed` with the cause
|
||||
when it can't read, log in to or write the store, or the copy takes over
|
||||
two minutes. A failed copy skips the deploy.
|
||||
|
||||
On every ten-minute round the timer declares a build, a copy and a deploy again
|
||||
for each agent whose latest build or copy failed, without a meta update when the
|
||||
latest update succeeded, until both succeed. Nothing limits the retries. The pin fan-out
|
||||
still records the pin when the gate skips a deploy, so one failing agent doesn't
|
||||
resend the healthy ones. A controller that updates before its builder host does
|
||||
is the same case: the old builder has no subscriber for the new subject, so the
|
||||
controller gets no responders at once, and the deploys go out after the builder
|
||||
updates. Each round adds a build and deploy node per failing agent.
|
||||
updates. Each round adds a build, a copy and a deploy node per failing agent.
|
||||
|
||||
A merged config PR whose webhook delivery never arrives would leave the agent
|
||||
on its old config. Every five minutes, after the config-PR scan, the controller
|
||||
|
|
@ -335,11 +353,14 @@ runtime directory (`/run/swarm-builder`, mode `0700`, both files `0600`), points
|
|||
The token is in no argument or variable, so no process listing shows it.
|
||||
|
||||
The reply's `cached` says whether the push finished. A push that fails, times out
|
||||
or can't read the token leaves the build green with `cached` false and logs a
|
||||
warning with the reason; the deploy goes ahead, and the controller sends the
|
||||
same build again until the push lands ([below](#rebuild-on-a-cache-miss)). With
|
||||
no attic behind the defaults, every push fails this way. With both options null,
|
||||
the builder pushes nothing and `cached` is always false.
|
||||
or can't read the token leaves the reply `Built` with `cached` false and logs a
|
||||
warning with the reason. The controller ends that build's `build_meta` node
|
||||
`failed`, which holds the deploy, and sends the same build again on the next
|
||||
ten-minute round until the push lands ([deploy gate](#deploy-gate)); the
|
||||
[cache-miss pass](#rebuild-on-a-cache-miss) also sends it for a record that says
|
||||
the builder didn't push. With no attic behind the defaults, every push fails this
|
||||
way. With both options null, the builder pushes nothing and `cached` is always
|
||||
false, so no deploy goes out.
|
||||
|
||||
Mixed versions:
|
||||
|
||||
|
|
@ -351,6 +372,11 @@ Mixed versions:
|
|||
|
||||
A record written before the field existed also reads as `false`.
|
||||
|
||||
The deploy gate is controller-only and changes no message. A new controller
|
||||
holds the deploy of a closure the builder didn't push; an old controller sends
|
||||
it, and the hive refuses it. Hives need no change: a hive of any version gets
|
||||
fewer deploys, and none for a closure it couldn't download.
|
||||
|
||||
### Rebuild on a cache miss
|
||||
|
||||
The cache drops a closure nobody fetched for its retention period, but the
|
||||
|
|
@ -461,7 +487,7 @@ the swarm at an existing tenant:
|
|||
`secret/swarm/services/attic/push-token`, field `value`. Use the store's UI under
|
||||
the OIDC login, or `bao kv put` with `value=-` and the token on stdin, so the token is in no
|
||||
command line. The token needs push rights on that cache. Until it
|
||||
exists, every push fails and builds stay green with `cached` false.
|
||||
exists, every push fails and no deploy goes out.
|
||||
Seed the hives' pull token the same way at
|
||||
`secret/swarm/services/attic/pull-token`, with read rights on that cache.
|
||||
3. Seed the three facts hives pull with, each as field `value`:
|
||||
|
|
@ -494,6 +520,11 @@ leaves (`pull-token`, `url`, `cache`, `public-key`) to
|
|||
own prefix. It writes a copy only when the value differs, skips a leaf the
|
||||
store doesn't hold, and leaves a copy whose source disappeared.
|
||||
|
||||
A deploy waits for the same copy: its `copy_attic_facts` node copies for the one
|
||||
hive the deploy goes to, so the facts are in the hive's prefix before the swarm tells the hive
|
||||
to deploy ([deploy gate](#deploy-gate)). That puts the facts in the
|
||||
store only; the hive reads them from there on its own schedule.
|
||||
|
||||
`hive-c0re` reads its four leaves every five minutes under the hive's own store
|
||||
identity and writes `/run/hyperhive/attic/netrc`, `substituter` and
|
||||
`public-key` (tmpfs, each `0600`). It accepts only an `https` URL, since the
|
||||
|
|
|
|||
Loading…
Reference in a new issue