diff --git a/docs/approvals.md b/docs/approvals.md index b0274283..32daa6d4 100644 --- a/docs/approvals.md +++ b/docs/approvals.md @@ -336,50 +336,49 @@ approval id to retry. Because tags are first-class git objects, rejected and failed trees stay browsable forever — `git log --tags` in the applied repo is the audit trail. -### Dispatch via `rebuild_queue` +### Dispatch via the job queue Long-running approval work — `ApplyCommit`, `UpdateMetaInputs`, `Spawn` — no longer runs inline inside `actions::approve`. Instead -the approval handler enqueues a `QueueEntry` into the global -`rebuild_queue`: +the approval handler submits a DAG to the global job queue +(`docs/coordinator.md::Job queue`): -| `ApprovalKind` | `QueueKind` queued | `QueueSource` | +| `ApprovalKind` | DAG submitted | source | |---|---|---| -| `ApplyCommit` | `Rebuild` | `Approval` | -| `MergeConfigPr` | `Rebuild` | `Approval` | -| `UpdateMetaInputs` | `MetaUpdate` | `Approval` | -| `Spawn` | `Spawn` | `Approval` | +| `ApplyCommit` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` | +| `MergeConfigPr` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` | +| `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` | +| `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` | | `InitConfig` | — runs inline (sub-second git seed) | — | | `SchedulePrompt` | — runs inline (single sqlite insert) | — | -Each queue entry carries the originating `approval_id` so the -worker can re-fetch the approval row when it dispatches, run the -kind-specific pipeline (`run_approval_apply_commit` / -`run_approval_merge_config_pr` / `run_approval_update_meta_inputs` / -`run_approval_spawn`), and -fire the matching `HelperEvent::*` on completion via -`finish_approval`. +The DAG carries the originating `approval_id`. The `ApprovalDeploy` +node runs the kind-specific pipeline (`run_approval_apply_commit` / +`run_approval_merge_config_pr` — the two-phase meta deploy stays +inside `actions.rs`) and fires the matching `HelperEvent::*` via +`finish_approval` itself; `Spawn` and `UpdateMetaInputs` DAGs resolve +through `actions::resolve_approval_dag` when the DAG settles terminal +(a spawn additionally runs the post-spawn forge bookkeeping there). Two visible consequences: - **Operator dashboard**: after clicking APPR0VE the work-in-progress - shows up on the *rebuild queue* card (`POST /api/state.rebuild_queue` + shows up on the *rebuild queue* card (`/api/state.rebuild_queue` + live `rebuild_queue_changed` events), not on the approvals panel (which already moved the row to "approved"). A long meta-update - cascade renders as a parent entry with one child per per-agent - rebuild — see `docs/web-ui.md` for the layout. -- **Cancellation**: the dashboard's *× cancel* button on a `Queued` - entry calls `POST /api/rebuild-queue/{id}/cancel`, which flips the - entry to `Cancelled` before the worker dispatches it. Returns - `{"cancelled": true}` on success, `{"cancelled": false}` if the - entry already left `Queued` (running / done / failed) — terminal - states can't be retroactively rewritten. + cascade renders as a parent DAG with one child rebuild per affected + agent — see `docs/web-ui.md` for the layout. +- **Cancellation**: the dashboard's *× cancel* button on a still-queued + DAG calls `POST /api/rebuild-queue/{id}/cancel`, which flips it to + `Cancelled` before any node runs (and fails the approval row instead + of leaving it dangling). Returns `{"cancelled": true}` on success, + `{"cancelled": false}` once any node started — terminal states can't + be retroactively rewritten. -`QueueSource::Approval` carries the `approval_id` so a tail-end -build failure surfaces back as a failed approval row, not just a -silent queue entry. `QueueSource::Manual` (dashboard ↻ R3BU1LD) -and `QueueSource::AutoUpdate` (boot-time sweep) use the same -queue but skip the approval row plumbing. +The `approval` source + `approval_id` mean a tail-end build failure +surfaces back as a failed approval row, not just a silent queue +entry. `manual` (dashboard ↻ R3BU1LD) and `auto_update` (boot +reconcile) DAGs use the same queue but skip the approval plumbing. ### Forge mirror diff --git a/docs/coordinator.md b/docs/coordinator.md index 539aa159..3cf09fdb 100644 --- a/docs/coordinator.md +++ b/docs/coordinator.md @@ -6,102 +6,164 @@ and `docs/persistence.md`. --- -## Rebuild queue +## Job queue -Every long-running container/meta operation (rebuild, meta-update, first-spawn) -goes through the global rebuild queue (`hive-c0re/src/rebuild_queue.rs`). A single -background worker drains it in FIFO order so two `nixos-container update` runs on -the same agent never overlap, and a fresh agent rebuild never races a meta-update's -lock bump. +Every container/meta operation (rebuild, meta-update, first-spawn, power +changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`) +as a **DAG of primitive nodes**. One scheduler task drives all DAGs; +concurrency comes from the resource classes below, not from multiple workers. +The old special cases — the graceful-stop watcher thread, the deferred-start +fast-lane follow-up, the meta-update cascade pre-enqueue — are all just DAG +*shapes* now. -### Why one queue +### Two levels: DAG and node -Before the rebuild queue landed, four independent call paths could fire -`auto_update::rebuild_agent` concurrently: +The **DAG** is the unit of dedup / cancel / approval-resolution and the +dashboard group; the **node** is the unit of scheduling / execution / +build-log / step label. Deps are intra-DAG edges only (`AfterOk` by default: +the dep must succeed, a failed/cancelled dep cancels the dependent — +cancel-downstream). Cross-DAG ordering comes from the per-agent lease + dedup, +never from edges between DAGs. Submit-time validation (petgraph `toposort`) +rejects cyclic specs outright, fixing the old queue's "circular dep silently +deadlocks" caveat. -- Dashboard manual rebuild button -- `update-all` / `meta-update` cascade -- Approval handler (apply-commit / spawn) -- Startup auto-update sweep +### Node inventory (primitives) -Nothing serialised them. `nix-daemon` serialises the actual store ops, but the rest -of `rebuild_agent` (token sync, kick, rescan, lock-bump emit) interleaved -unpredictably. The single-worker queue gives operators a visible, ordered runway and -lets the UI render "what's about to happen" instead of "something might be happening -somewhere." +Nix-heavy — hold one of the `buildSlots` permits for the node's duration: -### Queue kinds +| Node | Wraps | +| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving | +| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped) + the post-rebuild bookkeeping tail (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan) | +| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) | +| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion | +| `ApprovalDeploy` | the opaque apply-commit / merge-config-PR pipeline (see _Approvals_ below) | -| Kind | Description | -| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see _Deferred start_ under the rebuild path below. | -| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. | -| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. | -| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. | -| `Restart` | Stop + start a container without touching config (~5-10s). Routed through the queue so it serialises against in-flight rebuilds for the same agent — prevents a restart racing a rebuild mid-flight. Sources: dashboard ↺ button, the `restart` MCP tool. | -| `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. | -| `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. | +Cheap — no build slot: -**Intentionally not queued** (sub-second ops): the _hard_ `start`, `stop`, `kill` via the direct API paths. (A _graceful_ stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard. +| Node | Behavior | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | +| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | +| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn | +| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way | +| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload | +| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots | -### Dedup +There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation +with its commit under its internal `META_LOCK` mutex, so a standalone commit +node would open a dirty-working-tree window between nodes. That same +`META_LOCK` is also why the scheduler needs no meta-repo resource class — any +executor touching the meta repo serializes inside `meta.rs`. -Enqueueing `(kind, agent)` that already has a `Queued` entry returns the existing -entry's id and appends the new request as an "also requested by …" line. Running -entries do not dedup — a re-queue during a run is legitimate (something changed -since the current run started). +### Every operation as a DAG -### Sources +```text +rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-any) Reconcile(a) +graceful-stop(a): [wanted=Offline] Signal(a) → Drain(a) → Reconcile(a) +restart(a): StopForUpdate(a) → Reconcile(a) (wanted unchanged) +start(a): [wanted=Up] Reconcile(a) (stale rev ⇒ upgraded to rebuild) +stop(a): [wanted=Offline] Reconcile(a) +spawn(a): [wanted=Up] Create(a) → WriteDropin(a) → Reconcile(a) +perm-change(a): WritePermFile(a) → «rebuild subgraph» +meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected agent» +boot: (if any rev marker stale) MetaLock(hyperhive) → «fan-out rebuild»; + plus Reconcile(a) for every drifted agent +``` -| Source | Meaning | -| -------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| `Manual` | Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, an agent MCP tool). | -| `AutoUpdate` | Legacy startup-sweep source (flat, no parent). Replaced by `StartupSweep` for new boots. | -| `StartupSweep` | Child of a `StartupSweep` parent entry; boot-time per-agent rebuild with the sweep as the visual group header. | -| `Approval` | Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`. | +Notable collapses: -### Cascade parent tracking +- **`rebuild` is one uniform shape** — no `was_running` branch. + `StopForUpdate` noops when already down; the tail `Reconcile` auto-noops the + start when `wanted = Offline` (a rebuild of a deliberately-stopped agent + leaves it stopped). +- **The swap-failure recovery-start is structural**: `Reconcile` deps on + `Swap` with the one `AfterAny` edge in the system — it runs after `Swap` + terminal ok *or* fail, bringing a wanted-up agent back on its old config. +- **Deferred start is automatic**: `Reconcile` holds no build slot, so the + next DAG's `Prebuild` starts as soon as `Swap` frees the slot. +- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a + whole-hive graceful stop fires every agent's signal immediately and all + drains overlap; each DAG's tail `Reconcile` does the actual stop. +- **The meta-update cascade fans out on completion**: `MetaLock`'s executor + computes the affected agent set after the bump lands and appends child + `rebuild` DAGs (`parent_id` set, `relock = false` so the children don't + revert the bump). A failed bump fans out nothing — no cancel-children dance. -`MetaUpdate` and `StartupSweep` entries fan out `Rebuild` children, each carrying -`parent_id = `. The dashboard groups children under their parent -in the queue panel so the operator sees the whole cascade as a tree, -not a flat list. +### Desired-state (spec vs status) -### Step labels +Per-agent power *intent* — `wanted: Up | Offline` — is durable in +`/var/lib/hyperhive/db/agent_power.sqlite` (`hive-c0re/src/power.rs`). +`container_view` remains the observed *status*; `Reconcile` nodes converge the +two. Setting `wanted` is never a queued node: the submit layer +(`job_queue/submit.rs`) writes the row synchronously, then submits the DAG +whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins. +Power toggles never commit to the meta repo. Direct (non-queued) power paths — +`hivectl stop/start`, the admin-socket kill, the MCP kill tool — write +`wanted` too, so reconciles never undo an operator's stop. Agents without a +row are seeded from observed state on first touch (running ⇒ `Up`); destroy +removes the row. -Each queue entry has a mutable `step: Option` field that the worker updates -as it progresses through lifecycle phases (`"nix build"`, `"nixos-container stop"`, -`"nixos-container update"`, `"nixos-container start"`). The dashboard polls -`/api/state` and renders the current step beneath the running entry so the operator -can see which phase is taking time. +### Scheduler semantics -### Dependency tracking +A node is **ready** when it's `Queued`, every dep is satisfied, and its +resources are free. Resources: -Each entry carries a `depends_on: Vec` field. The worker skips entries whose -dependencies are not yet resolved — a dependency is resolved when its id is either -in the queue as a terminal entry (`Done` / `Failed` / `Cancelled`) or no longer in -the queue at all (evicted by `trim_history`, which only evicts terminals). +1. **Build slots** — `services.hyperhive.c0re.buildSlots` permits (default 1), + held by nix-heavy nodes for the node's duration. +2. **Per-agent lifecycle lease** — DAG-scoped: acquired at the DAG's first + container-affecting node (`StopForUpdate`, `Swap`, `Signal`, `Drain`, + `Reconcile`, `WriteDropin`, `Create`, `ApprovalDeploy`), held until the DAG + is terminal, so two lifecycle DAGs for one agent never interleave their + container ops. **Lease-exempt**: `Prebuild`, `MetaLock`, `WritePermFile` — + they touch the store / meta, not the running container, which is exactly + why a stop can land while another DAG's prebuild is still building. -Use cases: +Among simultaneously-ready nodes competing for a resource, DAG-submit order +wins (FIFO) so bulk operations drain predictably. The scheduler also owns the +DAG-lifetime transient guard (dashboard pill + crash-watch suppression), +created on lease acquisition and dropped when the DAG settles terminal. -- Chain a `Rebuild` after an explicit prerequisite step without coupling them through - the `parent_id` cascade mechanism. -- Sequence a `PermChange` + `Rebuild` pair where the rebuild must not start until the - perm-file write commits (already handled by the single-worker FIFO today, but - `depends_on` allows explicit cross-kind sequencing when parallel workers are added). +The queue is in-memory only and lost on hive-c0re restart — deliberate: +desired state is re-derived at boot from the DB + rev markers (see _Boot +reconcile_), so there is no durable-recovery machinery to go wrong. -`depends_on` is part of the dedup key: two entries with the same `(kind, agent, -parent_id, inputs, approval_id)` but different dep sets are treated as distinct work. +### Dedup, cancel, history -**Worker re-notification**: the worker drains `take_next()` in a tight loop after -each entry finishes. When a dep entry transitions to terminal, the loop re-evaluates -the queue immediately, so downstream entries are unblocked with no extra wakeup. No -additional `notify_one()` call is needed. +Dedup at **DAG granularity**: a repeat submit against a DAG whose roll-up is +still `Queued` with the same `(template, agent, parent_id, approval_id)` — +plus `inputs` for meta-updates and the perm-type discriminant for perm +changes — returns the existing id and appends an "also requested by …" line. +`parent_id` in the key keeps a cascade child from collapsing into a +standalone or sweep rebuild. Running/terminal DAGs never dedup. -**Circular-dep caveat**: if A depends on B and B depends on A, neither entry ever -becomes runnable — the worker skips both indefinitely with no error. Callers must -ensure acyclic dep graphs. Cycle detection is deferred to a future iteration (when -parallel workers make a stuck queue more visible). +Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't +interruptible); `cancel_children` cancels a parent's still-queued child DAGs. +Roll-up state: `Failed` if any node failed, else `Running` / `Queued` / +`Cancelled` / `Done`. The snapshot retains the 5 most recent terminal DAGs +per template. + +### Approvals + +`ApplyCommit` / `MergeConfigPr` approvals ride as single-node +`ApprovalDeploy` DAGs: the two-phase `prepare_deploy` / `finalize_deploy` / +`abort_deploy` meta orchestration stays inside `actions.rs` in v1 +(deliberately not modeled as scheduler nodes) and resolves the approval +itself. `Spawn` and `UpdateMetaInputs` approvals map onto the ordinary +`spawn` / `meta-update` shapes; the scheduler fires +`actions::resolve_approval_dag` exactly once when such a DAG settles +terminal (including cancelled-while-queued, which fails the approval instead +of dangling it). + +### Wire shape + +`RebuildQueueChanged { seq, queue: [DagView…] }` (event name kept). Each +`DagView` carries the old entry-level fields (`id`, `kind` = template string, +roll-up `state`, `agent`, `source`, `parent_id`, `reason`, timestamps, +`inputs`, `approval_id`) plus `nodes: [NodeView…]` — per-node `kind`, `deps`, +`state`, `step`, `build_log_id`, timestamps, `error`. Step labels and build +logs are **per-node**; the dashboard renders the node chain on each queue +card and keys the live-log panel off the running node. --- @@ -115,31 +177,31 @@ render. --- -## Auto-update sweep +## Boot reconcile -On startup, `auto_update.rs` rebuilds containers that actually need it. Two skip -rules keep boot-time work minimal: +On startup, `auto_update::run` classifies every agent by rev freshness (the +per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/` +vs the current flake path) and persisted `wanted` intent, then: -1. **Stopped containers** are deferred: the startup sweep enqueues nothing for them. - When the operator later starts a stopped container (via the dashboard or the - `start` MCP tool), both `run_start` (queue path) and `handle_start` (socket path) - check the rev marker first — if it's stale, the start is silently upgraded to a - full rebuild+start so the container runs current nix derivations. +1. **Config path** — when *any* marker is stale, submit one `StartupSweep` + DAG: a `MetaLock` (hyperhive input bump, non-fatal) that fans out `Rebuild` + children for the stale agents whose `wanted = Up` (topology-sorted, parents + first). Stale but wanted-offline agents get no boot-time nix work — their + rebuild happens on their next start (the start submit path upgrades a + stale start to rebuild+start), which is also why the lock bump runs even + when every stale agent is offline: those later start-upgrades must build + against the bumped lock. Each child rebuild's tail `Reconcile` brings the + agent (back) up, covering both the running-stale and stopped-but-wanted-up + cases. -2. **Running containers with a matching rev marker** are skipped: if the per-agent - `.{name}.hyperhive-rev` file under `/var/lib/hyperhive/applied/` already holds - the current flake rev, no nix work is needed and the entry is omitted entirely. +2. **Power path** — every agent whose observed state drifted from `wanted` + gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`). -`auto_update::run` enqueues a single `StartupSweep` parent entry (`kind = -startup_sweep`, `agent = "hyperhive"`) followed by per-agent `Rebuild` children -for the agents that do need rebuilding (`source = startup_sweep`, `parent_id = -sweep_id`). The sweep description records the rebuild / deferred / skipped counts -so the operator can see at a glance how much work the boot triggered. The child -rebuilds drain sequentially through the queue; the dashboard renders them nested -under the parent. - -Before the sweep-grouping change, each boot enqueued flat `Rebuild` entries with -`source = AutoUpdate` and no parent — visible but ungrouped. +Booting with no config change performs **no meta commit** — only reconciles. +The sweep reason records the rebuild / deferred / up-to-date counts so the +operator sees at a glance how much work the boot triggered. Agents without an +`agent_power` row are seeded from observed state during classification (the +one-time migration; thereafter the DB is authoritative). ## Meta flake @@ -157,8 +219,13 @@ Key operations: `RequestApplyCommit` path so a failed `nixos-container update` leaves no orphan commit in meta. Prepare writes the new lock without committing; finalize commits with the deploy message; abort restores the lock. -- **`lock_update_hyperhive`** — one-shot for the auto-update path: bumps the - `hyperhive` input lock, commits, cascades agent rebuilds. +- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the + sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits; + the scheduler fans out the agent rebuilds on completion. + +Every public `meta.rs` operation takes the module's internal `META_LOCK` +mutex, so concurrent job-queue nodes (and the approval deploy pipeline) never +race on the repo's `.git/index.lock`. --- @@ -179,34 +246,26 @@ new profile and skips the in-container `switch-to-configuration`. The subsequent `start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in one go, rather than the double-bounce a live `update` would trigger. -Sequence for a running container: +Sequence for a rebuild DAG (each step is its own queue node): -1. `prebuild_toplevel` — build the new `system.build.toplevel` **before** stopping. - The container keeps serving the previous generation while eval + fetch + build - happen out-of-band. `nixos-container update` then finds the result cached and - skips straight to the profile-swap. Build failures surface here, before the - running container is touched. -2. `nixos-container stop` — bring the container down. -3. `nixos-container update --flake meta#` — profile-swap (near-instant after - the prebuild). -4. `nixos-container start` — boot into the new generation; the in-container - activation script transitions old → new. +1. `Prebuild` — build the new `system.build.toplevel` **before** stopping. + The container keeps serving the previous generation while eval + fetch + + build happen out-of-band. `nixos-container update` then finds the result + cached and skips straight to the profile-swap. Build failures surface + here, before the running container is touched. (Runs even for a stopped + container — same total nix work, one uniform DAG shape.) +2. `StopForUpdate` — bring the container down (noop when already stopped). +3. `Swap` — `nixos-container update --flake meta#` profile-swap + (near-instant after the prebuild). +4. `Reconcile` — boot into the new generation when `wanted = Up`; the + in-container activation script transitions old → new. Holds no build + slot, so the next DAG's `Prebuild` overlaps the container boot — the old + "deferred start" split, now structural. -If the container is already stopped, step 1 is skipped (no downtime to shave — no -point evaluating the flake twice). - -**Deferred start (queue-dispatched rebuilds):** step 4 can take a while -(container boot), and holding the serialized build lane through it delays the -next queued rebuild's nix build for no reason. Queue-dispatched rebuilds -therefore pass `defer_start` — `rebuild_no_meta` skips the start and returns -`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead -(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under -it — the same split the graceful-stop path uses for its container stop). The -build-lane entry completes when the profile-swap finishes; a start failure -surfaces on the `Start` entry, which runs with the cold-start fallback. Direct -callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy -flow which verifies the agent comes back up before finalizing) keep the start -inline. +The approval apply-commit pipeline still drives `lifecycle::rebuild_no_meta` +(the fused stop/update/start path with an inline start) inside its +`ApprovalDeploy` node, because it verifies the agent comes back up before +finalizing the deploy tag. ### Cold-start fallback @@ -218,9 +277,8 @@ half-started at that point. Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes) → `start` (clean cold-start, no generation transition, new activation runs cleanly). Both errors are preserved and surfaced if the cold-start also fails. The fallback -lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild -path and the queue's fast-lane `Start` handler (which the deferred -start-after-rebuild rides). +lives in `lifecycle::start_with_fallback`, shared by the apply-commit deploy's +inline start and every `Reconcile` node's start action. ### Spawn path (new container) @@ -241,9 +299,18 @@ or the flake root directly — none of which exist in the rendered meta flake. ## Host-level resource + performance options -Three `services.hyperhive.c0re.*` options tune container resource -limits and first-spawn latency. All three apply uniformly to every -agent container. +A handful of `services.hyperhive.c0re.*` options tune container resource +limits, build parallelism, and first-spawn latency. + +### Build slots + +`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes +(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run +concurrently. The default serializes all heavy nix work like the pre-DAG +rebuild queue did; raise it on hosts with the cores/RAM to build several +agent toplevels at once. Per-agent correctness is independent of the count — +each agent's container-affecting ops serialize on its lifecycle lease +regardless. ### Container resource limits diff --git a/docs/persistence.md b/docs/persistence.md index af900b95..51a911d4 100644 --- a/docs/persistence.md +++ b/docs/persistence.md @@ -2,12 +2,12 @@ Where state lives, what survives what, and how it's bounded. -## Three sqlite databases +## Sqlite databases ### `/var/lib/hyperhive/db/broker.sqlite` (host) -Six tables, all in one file — four queues plus the schedule -header/targets split: +Seven tables, all in one file — four queues, the schedule +header/targets split, and the per-agent power-intent registry: - `messages` — every inter-agent / operator-bound message. `sender / recipient / body / sent_at / delivered_at / acked_at / @@ -44,6 +44,17 @@ header/targets split: last_fired_at_unix / last_result`. `ON DELETE CASCADE` from `scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON` per connection (set at open). +- `agent_power` — one tiny row per agent: `agent PK / wanted (up | + offline) / updated_at` — the durable power *intent* behind the job + queue's desired-state reconciliation + (`docs/coordinator.md::Job queue`; owner: `hive-c0re/src/power.rs`). + Written synchronously by every operator/agent power action + (dashboard start/stop, `hivectl stop`, the MCP kill/start tools, + spawn approval); read by `Reconcile` nodes and the boot reconcile. + Intent survives hive-c0re restarts — in-flight queue work + deliberately does not. Agents without a row are seeded from + observed state on first touch (running ⇒ `up`); destroy deletes + the row. Retention: @@ -63,6 +74,8 @@ Retention: (`cancel_schedule` MCP / dashboard ✗) which tombstones via `cancelled_at_unix`, then `reap_cancelled` drops the row on the next worker pass. +- `agent_power` rows live until the agent is destroyed (one row per + agent — nothing to vacuum). ### `/harness/hyperhive-events.sqlite` (per agent) diff --git a/docs/tools/hivectl-cli.md b/docs/tools/hivectl-cli.md index ace96cb9..44aa1d57 100644 --- a/docs/tools/hivectl-cli.md +++ b/docs/tools/hivectl-cli.md @@ -407,7 +407,7 @@ Stop containers hive-wide in one operator action. Bare `hivectl stop` stops **ev * `--forge` — The forge container (`hive-forge`) * `--gateway` — The gateway container (`hive-gateway`) * `--matrix` — The matrix container (`hive-matrix`) -* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent is enqueued as a `GracefulStop` on the rebuild queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). Applies to agents only +* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only diff --git a/hive-c0re/src/bin/hivectl.rs b/hive-c0re/src/bin/hivectl.rs index 3a5a743a..9816f90b 100644 --- a/hive-c0re/src/bin/hivectl.rs +++ b/hive-c0re/src/bin/hivectl.rs @@ -139,11 +139,11 @@ enum Cmd { #[command(flatten)] scope: ScopeArgs, /// Gracefully quiesce each agent before stopping, instead of a - /// hard stop. Each agent is enqueued as a `GracefulStop` on the - /// rebuild queue: the harness is signalled, runs one - /// stop-checkpoint turn to flush durable `/state`, drains, then - /// the container is stopped (bounded by a 3-min timeout that - /// falls back to a hard stop). Applies to agents only. + /// hard stop. Each agent gets a graceful-stop DAG on the job + /// queue: the harness is signalled, runs one stop-checkpoint + /// turn to flush durable `/state`, drains, then the container + /// is stopped (bounded by a 3-min timeout that falls back to a + /// hard stop). All drains overlap. Applies to agents only. #[arg(long)] graceful: bool, }, diff --git a/hive-c0re/src/coordinator.rs b/hive-c0re/src/coordinator.rs index 2dbf4dd1..868e97af 100644 --- a/hive-c0re/src/coordinator.rs +++ b/hive-c0re/src/coordinator.rs @@ -490,8 +490,7 @@ impl Coordinator { let audit_log = Arc::new(crate::audit_log::AuditLog::open(build_logs_dir).context("open audit_log")?); crate::audit_log::install(audit_log.clone()); - let power = - Arc::new(crate::power::PowerStore::open(build_logs_dir).context("open agent_power")?); + let power = Arc::new(crate::power::PowerStore::open(db_path).context("open agent_power")?); let (dashboard_events, _) = broadcast::channel(DASHBOARD_CHANNEL); let (shutdown_tx, _) = watch::channel(false); Ok(Self { diff --git a/hive-c0re/src/job_queue/exec.rs b/hive-c0re/src/job_queue/exec.rs index 0e9810e8..7c316243 100644 --- a/hive-c0re/src/job_queue/exec.rs +++ b/hive-c0re/src/job_queue/exec.rs @@ -130,7 +130,7 @@ async fn run_swap(coord: &Arc, claim: &Claim, ctx: &Ctx<'_>) -> Res let paths = Coordinator::agent_paths(name, agent_dir); let result = crate::lifecycle::swap_update(name, &hive, &paths, &|step| ctx.step(step), &|log_id| { - ctx.build_log(log_id) + ctx.build_log(log_id); }) .await; match &result { diff --git a/hive-c0re/src/power.rs b/hive-c0re/src/power.rs index dbf71f69..2f23689e 100644 --- a/hive-c0re/src/power.rs +++ b/hive-c0re/src/power.rs @@ -3,8 +3,11 @@ //! `container_view` remains the observed *status*; the job queue's //! `Reconcile` nodes are the mechanism that converges the two. //! -//! Stored in `/var/lib/hyperhive/db/agent_power.sqlite` (one tiny row -//! per agent). Intent persists across hive-c0re restarts; in-flight +//! Stored as the `agent_power` table in the coordinator DB +//! (`/var/lib/hyperhive/db/broker.sqlite`, one tiny row per agent) — +//! same one-file-many-modules pattern as `approvals` / +//! `operator_questions` / `scheduled_prompts`, each with its own +//! connection. Intent persists across hive-c0re restarts; in-flight //! queue work deliberately does not. Setting `wanted` is never a //! queued node: operator/intent actions update the row synchronously //! at request time, then submit the DAG whose terminal `Reconcile` @@ -80,12 +83,21 @@ pub struct PowerStore { } impl PowerStore { - pub fn open(db_dir: &Path) -> Result { - std::fs::create_dir_all(db_dir) - .with_context(|| format!("create agent_power db parent {}", db_dir.display()))?; - let path = db_dir.join("agent_power.sqlite"); - let conn = Connection::open(&path) - .with_context(|| format!("open agent_power db {}", path.display()))?; + /// Open (a connection to) the shared coordinator DB and ensure the + /// `agent_power` table exists. `db_path` is the same sqlite file + /// the broker / approvals / questions stores open. + pub fn open(db_path: &Path) -> Result { + if let Some(parent) = db_path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("create agent_power db parent {}", parent.display()))?; + } + let conn = Connection::open(db_path) + .with_context(|| format!("open agent_power db {}", db_path.display()))?; + // Several modules hold their own connection to this file (the + // broker / approvals / questions pattern); wait out a + // concurrent writer instead of surfacing SQLITE_BUSY. + conn.busy_timeout(std::time::Duration::from_secs(5)) + .context("set agent_power busy_timeout")?; conn.execute_batch(SCHEMA) .context("apply agent_power schema")?; Ok(Self {