gate each deploy on a swarm build of the agent from internal/meta
This commit is contained in:
parent
3db835fb57
commit
5e64048e53
3 changed files with 1570 additions and 178 deletions
|
|
@ -149,7 +149,9 @@ every agent any hive places (the pins:
|
|||
every agent from it. While the repo has no commit that pins an agent's current
|
||||
config, that agent's build fails and names the agent. The controller queues
|
||||
its update of the repo before the deploy request for a merged config commit
|
||||
and for a new agent, and sends the deploy only when that update succeeded.
|
||||
and for a new agent, and sends the deploy only when that update succeeded and
|
||||
the swarm builder built the agent from the updated repo (the
|
||||
[deploy gate](../swarm/builder.md#deploy-gate)). A red build skips the deploy.
|
||||
|
||||
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
|
||||
default-branch head with the hive's read token (`forge::reader_token()`)
|
||||
|
|
|
|||
|
|
@ -6,6 +6,10 @@ push, the builder builds the PR's head on the agent base, and the result shows a
|
|||
`swarm-builder` commit status on the PR and as a `build_agent` node on the
|
||||
swarm UI's Jobs page.
|
||||
|
||||
The builder also builds each agent from `internal/meta` before the swarm sends
|
||||
its deploy, and a red build holds the deploy back: see
|
||||
[Deploy gate](#deploy-gate).
|
||||
|
||||
Where it runs and how it's enabled: [`services.md`](services.md#builder).
|
||||
|
||||
## What happens on a config PR
|
||||
|
|
@ -102,7 +106,8 @@ status has no link.
|
|||
- The builder pushes nothing. The closure stays in the builder
|
||||
host's store until it's collected, so on a host that also runs the hive, the
|
||||
hive's own build can reuse whatever matches.
|
||||
- It doesn't gate anything. A merge deploys whatever the status says.
|
||||
- The PR check gates nothing. A merge deploys whatever the status says. The
|
||||
[deploy gate](#deploy-gate) is a separate build, from `internal/meta`.
|
||||
- A lost webhook delivery isn't retried: the head isn't built until the next
|
||||
push. The jobs graph is in memory, so a controller restart drops a build in
|
||||
flight and leaves its status `pending`.
|
||||
|
|
@ -148,11 +153,12 @@ repo. It re-renders when it starts, and again every ten minutes while the latest
|
|||
update hasn't succeeded (a tick adds nothing while one is pending or running,
|
||||
or after a success), before it deploys a merged config PR, and
|
||||
before the first deploy of a new agent. Those deploys run only after the
|
||||
update succeeded: a hive builds only from `internal/meta`, so after a failed
|
||||
update the scheduler skips the deploy node, and the graph shows the update as
|
||||
failed and the deploy as skipped. Each ten-minute retry of a failed update
|
||||
declares a fresh deploy behind it for every agent whose latest deploy the failed update
|
||||
skipped, so the run that succeeds sends them (see
|
||||
update succeeded and the [deploy gate](#deploy-gate) built the agent: a hive
|
||||
builds only from `internal/meta`, so after a failed update the scheduler skips
|
||||
the deploy node, and the graph shows the update as failed and the deploy as
|
||||
skipped. Each ten-minute retry of a failed update declares a fresh build and
|
||||
deploy behind it for every agent whose latest deploy the failed update skipped,
|
||||
so the run that succeeds sends them (see
|
||||
[Config changes](../agent-lifecycle/config-changes.md#the-swarms-meta-flake)). A render
|
||||
equal to the committed `flake.nix` sends no lock.
|
||||
|
||||
|
|
@ -160,19 +166,72 @@ The controller sends a deploy to every agent a hive places whenever the pin in
|
|||
`internal/meta` differs from the pin it last sent, so they rebuild on the new
|
||||
hyperhive. It keeps that pin as the key `pin` in the `meta-fanout` KV bucket,
|
||||
which only the controller reads or writes, and a `RecordFanout` job writes the
|
||||
key after every deploy of the fan-out succeeds. The check runs after each update
|
||||
and once when the controller starts.
|
||||
key once every deploy of the fan-out has finished or the deploy gate has skipped
|
||||
it. The check runs after each update and once when the controller starts.
|
||||
|
||||
- The first commit sends a deploy, because the key holds no pin yet.
|
||||
- A controller that starts with no key, as after the upgrade that added it,
|
||||
sends one round. A deploy is idempotent per pin, so the round is harmless.
|
||||
- A controller that dies after the commit, or a deploy that fails, leaves the
|
||||
key behind. The next update or start sends the deploys again.
|
||||
- A deploy the gate skipped counts as sent. A red build holds only its own
|
||||
agent back, and the key still moves, so one broken agent doesn't resend the
|
||||
healthy ones on every retry. The timer declares a build and deploy again for
|
||||
a deploy skipped because the build gave no verdict.
|
||||
- A restart that finds the key equal to the pin sends nothing.
|
||||
- When the bucket won't open, or its key won't read, the controller logs a
|
||||
warning and sends deploys only when its own update moved the pin. That case
|
||||
writes no key.
|
||||
|
||||
## Deploy gate
|
||||
|
||||
Every deploy the swarm sends waits for a build of its agent from `internal/meta`,
|
||||
the flake a hive builds from. Four paths send deploys, and each declares the same
|
||||
build and deploy pair: a merged config PR, a new agent, the pin fan-out and the
|
||||
ten-minute retry. The `build_meta` node comes before the `trigger_deploy` node
|
||||
and needs the one `builder` slot, so it queues behind PR checks. The deploy
|
||||
runs only when the build is green.
|
||||
|
||||
The `build_meta` node reads `internal/meta`'s head commit when it runs, because
|
||||
most paths declare it before the update that commits that head. It sends the
|
||||
agent and that commit on `.builder.build-meta` and waits up to 61
|
||||
minutes. The builder takes the same one-build slot as PR checks, and
|
||||
refuses a request that waited 30 seconds for it. It runs `nix build` of
|
||||
`nixosConfigurations.<agent>.config.system.build.toplevel` from
|
||||
`git+https://<forge>/internal/meta.git?rev=<head>`, with no input overridden:
|
||||
the committed `flake.lock` decides every input. Like the hive's build, it uses
|
||||
that attribute, so a green build here means the flake evaluates and builds for
|
||||
the builder's platform. The hive doesn't reuse the result yet: it still builds
|
||||
locally, from whatever head it reads at deploy time, which can be newer than
|
||||
the one the swarm built.
|
||||
|
||||
The node ends one of three ways:
|
||||
|
||||
| Builder's answer | Node | Deploy | Timer |
|
||||
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------- | --------------------------------- |
|
||||
| built | done, closure recorded | runs | |
|
||||
| `nix` failed or timed out (a red build) | failed, with `nix`'s last lines | skipped | declares nothing |
|
||||
| no builder listening, no reply in time, or the builder couldn't start `nix` (busy for 30 seconds, no forge token) | failed, `not built, will be retried` | skipped | declares a build and deploy again |
|
||||
|
||||
A red build is a verdict on the config: a config merge queues its own build and
|
||||
deploy. A build that gave no verdict isn't, so the ten-minute timer declares
|
||||
it again, without a meta update when the latest update succeeded, until the
|
||||
builder answers. That covers a controller that updates before its builder
|
||||
host does: the old builder has no subscriber for the new subject, so the
|
||||
controller gets no responders at once, and the deploys go out after the builder
|
||||
updates. Each retry adds a build and deploy node per affected agent while the
|
||||
builder stays away.
|
||||
|
||||
On a built reply the controller writes the agent's key in the `agent-closures`
|
||||
KV bucket: the `internal/meta` commit, the closure's store path, the builder's
|
||||
platform and the time. Only the controller writes the bucket, and the builder
|
||||
holds no grant on it. A red build writes nothing and leaves the previous key.
|
||||
Nothing reads the bucket yet. The closure stays unrooted in the builder host's
|
||||
store.
|
||||
|
||||
The controller gives the `builder` forge account read on `internal/meta` when it
|
||||
updates the flake, since the repo is private.
|
||||
|
||||
## Credentials
|
||||
|
||||
| What | Where | Who reads it |
|
||||
|
|
@ -183,7 +242,8 @@ and once when the controller starts.
|
|||
|
||||
The token belongs to the `builder` forge account, which swarm-controller
|
||||
creates, puts in the `hives` team of `agent-configs` (read on every config
|
||||
repo) and mints a `read:repository` token for, in a five-minute pass. `git`
|
||||
repo, and on `internal/meta` through the meta update) and mints a
|
||||
`read:repository` token for, in a five-minute pass. `git`
|
||||
gets the token through a credential helper reading the child's environment:
|
||||
it's in no argument, URL or lock, so no line `nix` prints can carry it into a
|
||||
status. The same `git` fetches a config's own inputs, so the builder registers
|
||||
|
|
|
|||
|
|
@ -422,6 +422,11 @@ through the host's nix daemon, and swarm-controller writes the result as a `swar
|
|||
commit status on the head. The job shows on the swarm UI's Jobs page. A red
|
||||
status blocks nothing: the merge still deploys.
|
||||
|
||||
Separately, the builder builds each agent from `internal/meta` before the swarm
|
||||
sends its deploy, and a red build holds that deploy back (`build_meta` job;
|
||||
[deploy gate](builder.md#deploy-gate)). Those jobs share the one-build slot with
|
||||
the PR checks.
|
||||
|
||||
It needs neither the forge nor a hive on its host. It reaches the queue, the
|
||||
secret store and the forge by name, under its own `builder` leaf, and fetches
|
||||
the private config repo with the read-only token of its own `builder` forge
|
||||
|
|
|
|||
Loading…
Reference in a new issue