remove the hive's local build of agent closures
A rebuild or create downloads the closure the swarm built and applies it; with none to deploy it fails and changes nothing. hive-priv takes a store path instead of an internal/meta commit and builds nothing.
This commit is contained in:
atlas
parent
70538fb209
commit
4590b04cac
1 changed files with 632 additions and 1351 deletions
|
|
@ -42,11 +42,11 @@ because there is no malformed spec to reject.
|
|||
|
||||
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||
|
||||
| Node | Wraps |
|
||||
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm cached for the `internal/meta` commit it resolves, else builds it locally, and marks the agent's pins |
|
||||
| Node | Wraps |
|
||||
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
|
||||
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the agent's pins |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
|
|
@ -54,7 +54,7 @@ Cheap — no build slot:
|
|||
|
||||
| Node | Behavior |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` build from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
|
||||
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the agent-set refresh (`agent_set::refresh`), and the `internal/meta` commit `Prebuild` and `Swap` expect the swarm's closure to be built from, resolved once and recorded; fails when the repo doesn't pin the agent's current config. Declares no resources |
|
||||
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
|
||||
|
|
@ -186,7 +186,7 @@ resources are free. Resources:
|
|||
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
|
||||
`Prebuild`, `Provision` — they
|
||||
touch the store, not the running container, which is exactly why a
|
||||
stop can land while another DAG's prebuild is still building. Also exempt, for a different
|
||||
stop can land while another DAG's prebuild is still downloading. Also exempt, for a different
|
||||
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
|
||||
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
|
||||
container, but their `AgentWindow` brace holds the lease above them — see
|
||||
|
|
@ -344,17 +344,17 @@ one go, rather than the double-bounce a live `update` would trigger.
|
|||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `MetaSync` — rebuild-dir prep, the agent-set refresh, and the choice of the
|
||||
`internal/meta` commit to build from. Short, and it declares no resources.
|
||||
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch +
|
||||
build happen out-of-band. `nixos-container update` then finds the result
|
||||
cached and skips straight to the profile-swap. Build failures surface
|
||||
here, before `Swap` touches the running container. Skipped when the
|
||||
container is already down — a stopped agent has no uptime to preserve,
|
||||
so `Swap` builds inline instead of paying the double eval.
|
||||
`internal/meta` commit the swarm's closure should come from. Short,
|
||||
and it declares no resources.
|
||||
2. `Prebuild` — download the swarm's closure for the agent **before** stopping.
|
||||
The container keeps serving the previous generation while the download
|
||||
happens out-of-band. A missing or unpullable closure fails the DAG here,
|
||||
before `Swap` touches the running container, and the container keeps
|
||||
running. Skipped when the container is already down — a stopped agent has
|
||||
no uptime to preserve, so `Swap` downloads it.
|
||||
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
4. `Swap` — `nixos-container update` profile-swap from `internal/meta`
|
||||
(near-instant after the prebuild).
|
||||
4. `Swap` — profile-swap onto the swarm's closure (near-instant after the
|
||||
prebuild).
|
||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot.
|
||||
|
|
@ -374,18 +374,10 @@ fallback.
|
|||
|
||||
### Spawn path (new container)
|
||||
|
||||
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
|
||||
it leaves no container record to clean up. A separate prebuild would just duplicate
|
||||
the eval, so it's skipped. Sequence: `create` from `internal/meta` → write nspawn
|
||||
flags → `systemctl daemon-reload` → `start`.
|
||||
|
||||
### Prebuild attr path
|
||||
|
||||
`nix build` doesn't autoresolve `<flake-root>#<name>` against `nixosConfigurations` the
|
||||
way `nixos-container` does internally. nix requires the explicit attr path
|
||||
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
|
||||
using the bare `<flake-root>#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
||||
or the flake root directly — none of which exist in the rendered meta flake.
|
||||
For a first-time `create`, `nixos-container create` is atomic: it leaves no container
|
||||
record to clean up if it fails. The `Create` node downloads the swarm's closure first,
|
||||
so a spawn with no closure to deploy fails before it creates the container. Sequence:
|
||||
download the closure → `create` from it → write nspawn flags → `systemctl daemon-reload` → `start`.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -396,11 +388,10 @@ limits, build parallelism, and first-spawn latency.
|
|||
|
||||
### Build slots
|
||||
|
||||
`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
|
||||
`buildSlots` (default `1`) sets how many closure-fetching job-queue nodes
|
||||
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
|
||||
concurrently. The default serializes all heavy nix work like the pre-DAG
|
||||
rebuild queue did; raise it on hosts with the cores/RAM to build multiple
|
||||
agent toplevels at once. Per-agent correctness is independent of the count —
|
||||
concurrently. The default serializes all of them; raise it on hosts with
|
||||
the bandwidth and disk to take more agents at once. Per-agent correctness is independent of the count —
|
||||
each agent's container-affecting ops serialize on its lifecycle lease
|
||||
regardless.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue