defer start-after-rebuild to the fast lane so the build lane frees up (closes #2094)

This commit is contained in:
damocles 2026-07-01 23:37:13 +02:00
commit b191858366
6 changed files with 162 additions and 61 deletions

View file

@ -34,7 +34,7 @@ somewhere."
| Kind | Description |
|------|-------------|
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. |
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see *Deferred start* under the rebuild path below. |
| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. |
| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. |
| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. |
@ -42,7 +42,7 @@ somewhere."
| `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. |
| `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/<agent>`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. |
**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill`. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.)
**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill` via the direct API paths. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard.
### Dedup
@ -185,6 +185,19 @@ Sequence for a running container:
If the container is already stopped, step 1 is skipped (no downtime to shave — no
point evaluating the flake twice).
**Deferred start (queue-dispatched rebuilds):** step 4 can take a while
(container boot), and holding the serialized build lane through it delays the
next queued rebuild's nix build for no reason. Queue-dispatched rebuilds
therefore pass `defer_start``rebuild_no_meta` skips the start and returns
`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead
(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under
it — the same split the graceful-stop path uses for its container stop). The
build-lane entry completes when the profile-swap finishes; a start failure
surfaces on the `Start` entry, which runs with the cold-start fallback. Direct
callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy
flow which verifies the agent comes back up before finalizing) keep the start
inline.
### Cold-start fallback
`start` after `update` can exit non-zero when packages are **removed** between
@ -194,7 +207,10 @@ half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
`start` (clean cold-start, no generation transition, new activation runs cleanly).
Both errors are preserved and surfaced if the cold-start also fails.
Both errors are preserved and surfaced if the cold-start also fails. The fallback
lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild
path and the queue's fast-lane `Start` handler (which the deferred
start-after-rebuild rides).
### Spawn path (new container)