docs: job-DAG queue model; fold agent_power table into broker.sqlite
coordinator.md rewrites the queue section (node inventory, DAG shapes, resources, desired-state reconciliation, boot reconcile); approvals.md + persistence.md + hivectl --graceful help updated to match. agent_power lives in broker.sqlite like approvals/questions (own connection + busy timeout) instead of a separate db file.
This commit is contained in:
müde
parent
8349e6f621
commit
604e1c2557
8 changed files with 293 additions and 203 deletions
|
|
@ -336,50 +336,49 @@ approval id to retry. Because tags are first-class git objects,
|
|||
rejected and failed trees stay browsable forever — `git log
|
||||
--tags` in the applied repo is the audit trail.
|
||||
|
||||
### Dispatch via `rebuild_queue`
|
||||
### Dispatch via the job queue
|
||||
|
||||
Long-running approval work — `ApplyCommit`, `UpdateMetaInputs`,
|
||||
`Spawn` — no longer runs inline inside `actions::approve`. Instead
|
||||
the approval handler enqueues a `QueueEntry` into the global
|
||||
`rebuild_queue`:
|
||||
the approval handler submits a DAG to the global job queue
|
||||
(`docs/coordinator.md::Job queue`):
|
||||
|
||||
| `ApprovalKind` | `QueueKind` queued | `QueueSource` |
|
||||
| `ApprovalKind` | DAG submitted | source |
|
||||
|---|---|---|
|
||||
| `ApplyCommit` | `Rebuild` | `Approval` |
|
||||
| `MergeConfigPr` | `Rebuild` | `Approval` |
|
||||
| `UpdateMetaInputs` | `MetaUpdate` | `Approval` |
|
||||
| `Spawn` | `Spawn` | `Approval` |
|
||||
| `ApplyCommit` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` |
|
||||
| `MergeConfigPr` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` |
|
||||
| `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` |
|
||||
| `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` |
|
||||
| `InitConfig` | — runs inline (sub-second git seed) | — |
|
||||
| `SchedulePrompt` | — runs inline (single sqlite insert) | — |
|
||||
|
||||
Each queue entry carries the originating `approval_id` so the
|
||||
worker can re-fetch the approval row when it dispatches, run the
|
||||
kind-specific pipeline (`run_approval_apply_commit` /
|
||||
`run_approval_merge_config_pr` / `run_approval_update_meta_inputs` /
|
||||
`run_approval_spawn`), and
|
||||
fire the matching `HelperEvent::*` on completion via
|
||||
`finish_approval`.
|
||||
The DAG carries the originating `approval_id`. The `ApprovalDeploy`
|
||||
node runs the kind-specific pipeline (`run_approval_apply_commit` /
|
||||
`run_approval_merge_config_pr` — the two-phase meta deploy stays
|
||||
inside `actions.rs`) and fires the matching `HelperEvent::*` via
|
||||
`finish_approval` itself; `Spawn` and `UpdateMetaInputs` DAGs resolve
|
||||
through `actions::resolve_approval_dag` when the DAG settles terminal
|
||||
(a spawn additionally runs the post-spawn forge bookkeeping there).
|
||||
|
||||
Two visible consequences:
|
||||
|
||||
- **Operator dashboard**: after clicking APPR0VE the work-in-progress
|
||||
shows up on the *rebuild queue* card (`POST /api/state.rebuild_queue`
|
||||
shows up on the *rebuild queue* card (`/api/state.rebuild_queue`
|
||||
+ live `rebuild_queue_changed` events), not on the approvals panel
|
||||
(which already moved the row to "approved"). A long meta-update
|
||||
cascade renders as a parent entry with one child per per-agent
|
||||
rebuild — see `docs/web-ui.md` for the layout.
|
||||
- **Cancellation**: the dashboard's *× cancel* button on a `Queued`
|
||||
entry calls `POST /api/rebuild-queue/{id}/cancel`, which flips the
|
||||
entry to `Cancelled` before the worker dispatches it. Returns
|
||||
`{"cancelled": true}` on success, `{"cancelled": false}` if the
|
||||
entry already left `Queued` (running / done / failed) — terminal
|
||||
states can't be retroactively rewritten.
|
||||
cascade renders as a parent DAG with one child rebuild per affected
|
||||
agent — see `docs/web-ui.md` for the layout.
|
||||
- **Cancellation**: the dashboard's *× cancel* button on a still-queued
|
||||
DAG calls `POST /api/rebuild-queue/{id}/cancel`, which flips it to
|
||||
`Cancelled` before any node runs (and fails the approval row instead
|
||||
of leaving it dangling). Returns `{"cancelled": true}` on success,
|
||||
`{"cancelled": false}` once any node started — terminal states can't
|
||||
be retroactively rewritten.
|
||||
|
||||
`QueueSource::Approval` carries the `approval_id` so a tail-end
|
||||
build failure surfaces back as a failed approval row, not just a
|
||||
silent queue entry. `QueueSource::Manual` (dashboard ↻ R3BU1LD)
|
||||
and `QueueSource::AutoUpdate` (boot-time sweep) use the same
|
||||
queue but skip the approval row plumbing.
|
||||
The `approval` source + `approval_id` mean a tail-end build failure
|
||||
surfaces back as a failed approval row, not just a silent queue
|
||||
entry. `manual` (dashboard ↻ R3BU1LD) and `auto_update` (boot
|
||||
reconcile) DAGs use the same queue but skip the approval plumbing.
|
||||
|
||||
### Forge mirror
|
||||
|
||||
|
|
|
|||
|
|
@ -6,102 +6,164 @@ and `docs/persistence.md`.
|
|||
|
||||
---
|
||||
|
||||
## Rebuild queue
|
||||
## Job queue
|
||||
|
||||
Every long-running container/meta operation (rebuild, meta-update, first-spawn)
|
||||
goes through the global rebuild queue (`hive-c0re/src/rebuild_queue.rs`). A single
|
||||
background worker drains it in FIFO order so two `nixos-container update` runs on
|
||||
the same agent never overlap, and a fresh agent rebuild never races a meta-update's
|
||||
lock bump.
|
||||
Every container/meta operation (rebuild, meta-update, first-spawn, power
|
||||
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
||||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||
concurrency comes from the resource classes below, not from multiple workers.
|
||||
The old special cases — the graceful-stop watcher thread, the deferred-start
|
||||
fast-lane follow-up, the meta-update cascade pre-enqueue — are all just DAG
|
||||
*shapes* now.
|
||||
|
||||
### Why one queue
|
||||
### Two levels: DAG and node
|
||||
|
||||
Before the rebuild queue landed, four independent call paths could fire
|
||||
`auto_update::rebuild_agent` concurrently:
|
||||
The **DAG** is the unit of dedup / cancel / approval-resolution and the
|
||||
dashboard group; the **node** is the unit of scheduling / execution /
|
||||
build-log / step label. Deps are intra-DAG edges only (`AfterOk` by default:
|
||||
the dep must succeed, a failed/cancelled dep cancels the dependent —
|
||||
cancel-downstream). Cross-DAG ordering comes from the per-agent lease + dedup,
|
||||
never from edges between DAGs. Submit-time validation (petgraph `toposort`)
|
||||
rejects cyclic specs outright, fixing the old queue's "circular dep silently
|
||||
deadlocks" caveat.
|
||||
|
||||
- Dashboard manual rebuild button
|
||||
- `update-all` / `meta-update` cascade
|
||||
- Approval handler (apply-commit / spawn)
|
||||
- Startup auto-update sweep
|
||||
### Node inventory (primitives)
|
||||
|
||||
Nothing serialised them. `nix-daemon` serialises the actual store ops, but the rest
|
||||
of `rebuild_agent` (token sync, kick, rescan, lock-bump emit) interleaved
|
||||
unpredictably. The single-worker queue gives operators a visible, ordered runway and
|
||||
lets the UI render "what's about to happen" instead of "something might be happening
|
||||
somewhere."
|
||||
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||
|
||||
### Queue kinds
|
||||
| Node | Wraps |
|
||||
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped) + the post-rebuild bookkeeping tail (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan) |
|
||||
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `ApprovalDeploy` | the opaque apply-commit / merge-config-PR pipeline (see _Approvals_ below) |
|
||||
|
||||
| Kind | Description |
|
||||
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see _Deferred start_ under the rebuild path below. |
|
||||
| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. |
|
||||
| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. |
|
||||
| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. |
|
||||
| `Restart` | Stop + start a container without touching config (~5-10s). Routed through the queue so it serialises against in-flight rebuilds for the same agent — prevents a restart racing a rebuild mid-flight. Sources: dashboard ↺ button, the `restart` MCP tool. |
|
||||
| `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. |
|
||||
| `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/<agent>`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. |
|
||||
Cheap — no build slot:
|
||||
|
||||
**Intentionally not queued** (sub-second ops): the _hard_ `start`, `stop`, `kill` via the direct API paths. (A _graceful_ stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard.
|
||||
| Node | Behavior |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
||||
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
||||
|
||||
### Dedup
|
||||
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
|
||||
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
||||
node would open a dirty-working-tree window between nodes. That same
|
||||
`META_LOCK` is also why the scheduler needs no meta-repo resource class — any
|
||||
executor touching the meta repo serializes inside `meta.rs`.
|
||||
|
||||
Enqueueing `(kind, agent)` that already has a `Queued` entry returns the existing
|
||||
entry's id and appends the new request as an "also requested by …" line. Running
|
||||
entries do not dedup — a re-queue during a run is legitimate (something changed
|
||||
since the current run started).
|
||||
### Every operation as a DAG
|
||||
|
||||
### Sources
|
||||
```text
|
||||
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-any) Reconcile(a)
|
||||
graceful-stop(a): [wanted=Offline] Signal(a) → Drain(a) → Reconcile(a)
|
||||
restart(a): StopForUpdate(a) → Reconcile(a) (wanted unchanged)
|
||||
start(a): [wanted=Up] Reconcile(a) (stale rev ⇒ upgraded to rebuild)
|
||||
stop(a): [wanted=Offline] Reconcile(a)
|
||||
spawn(a): [wanted=Up] Create(a) → WriteDropin(a) → Reconcile(a)
|
||||
perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
||||
meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected agent»
|
||||
boot: (if any rev marker stale) MetaLock(hyperhive) → «fan-out rebuild»;
|
||||
plus Reconcile(a) for every drifted agent
|
||||
```
|
||||
|
||||
| Source | Meaning |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Manual` | Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, an agent MCP tool). |
|
||||
| `AutoUpdate` | Legacy startup-sweep source (flat, no parent). Replaced by `StartupSweep` for new boots. |
|
||||
| `StartupSweep` | Child of a `StartupSweep` parent entry; boot-time per-agent rebuild with the sweep as the visual group header. |
|
||||
| `Approval` | Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`. |
|
||||
Notable collapses:
|
||||
|
||||
### Cascade parent tracking
|
||||
- **`rebuild` is one uniform shape** — no `was_running` branch.
|
||||
`StopForUpdate` noops when already down; the tail `Reconcile` auto-noops the
|
||||
start when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
||||
leaves it stopped).
|
||||
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
|
||||
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
|
||||
terminal ok *or* fail, bringing a wanted-up agent back on its old config.
|
||||
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
|
||||
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
|
||||
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
||||
whole-hive graceful stop fires every agent's signal immediately and all
|
||||
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
||||
- **The meta-update cascade fans out on completion**: `MetaLock`'s executor
|
||||
computes the affected agent set after the bump lands and appends child
|
||||
`rebuild` DAGs (`parent_id` set, `relock = false` so the children don't
|
||||
revert the bump). A failed bump fans out nothing — no cancel-children dance.
|
||||
|
||||
`MetaUpdate` and `StartupSweep` entries fan out `Rebuild` children, each carrying
|
||||
`parent_id = <parent_id>`. The dashboard groups children under their parent
|
||||
in the queue panel so the operator sees the whole cascade as a tree,
|
||||
not a flat list.
|
||||
### Desired-state (spec vs status)
|
||||
|
||||
### Step labels
|
||||
Per-agent power *intent* — `wanted: Up | Offline` — is durable in
|
||||
`/var/lib/hyperhive/db/agent_power.sqlite` (`hive-c0re/src/power.rs`).
|
||||
`container_view` remains the observed *status*; `Reconcile` nodes converge the
|
||||
two. Setting `wanted` is never a queued node: the submit layer
|
||||
(`job_queue/submit.rs`) writes the row synchronously, then submits the DAG
|
||||
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
||||
Power toggles never commit to the meta repo. Direct (non-queued) power paths —
|
||||
`hivectl stop/start`, the admin-socket kill, the MCP kill tool — write
|
||||
`wanted` too, so reconciles never undo an operator's stop. Agents without a
|
||||
row are seeded from observed state on first touch (running ⇒ `Up`); destroy
|
||||
removes the row.
|
||||
|
||||
Each queue entry has a mutable `step: Option<String>` field that the worker updates
|
||||
as it progresses through lifecycle phases (`"nix build"`, `"nixos-container stop"`,
|
||||
`"nixos-container update"`, `"nixos-container start"`). The dashboard polls
|
||||
`/api/state` and renders the current step beneath the running entry so the operator
|
||||
can see which phase is taking time.
|
||||
### Scheduler semantics
|
||||
|
||||
### Dependency tracking
|
||||
A node is **ready** when it's `Queued`, every dep is satisfied, and its
|
||||
resources are free. Resources:
|
||||
|
||||
Each entry carries a `depends_on: Vec<u64>` field. The worker skips entries whose
|
||||
dependencies are not yet resolved — a dependency is resolved when its id is either
|
||||
in the queue as a terminal entry (`Done` / `Failed` / `Cancelled`) or no longer in
|
||||
the queue at all (evicted by `trim_history`, which only evicts terminals).
|
||||
1. **Build slots** — `services.hyperhive.c0re.buildSlots` permits (default 1),
|
||||
held by nix-heavy nodes for the node's duration.
|
||||
2. **Per-agent lifecycle lease** — DAG-scoped: acquired at the DAG's first
|
||||
container-affecting node (`StopForUpdate`, `Swap`, `Signal`, `Drain`,
|
||||
`Reconcile`, `WriteDropin`, `Create`, `ApprovalDeploy`), held until the DAG
|
||||
is terminal, so two lifecycle DAGs for one agent never interleave their
|
||||
container ops. **Lease-exempt**: `Prebuild`, `MetaLock`, `WritePermFile` —
|
||||
they touch the store / meta, not the running container, which is exactly
|
||||
why a stop can land while another DAG's prebuild is still building.
|
||||
|
||||
Use cases:
|
||||
Among simultaneously-ready nodes competing for a resource, DAG-submit order
|
||||
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
|
||||
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
|
||||
created on lease acquisition and dropped when the DAG settles terminal.
|
||||
|
||||
- Chain a `Rebuild` after an explicit prerequisite step without coupling them through
|
||||
the `parent_id` cascade mechanism.
|
||||
- Sequence a `PermChange` + `Rebuild` pair where the rebuild must not start until the
|
||||
perm-file write commits (already handled by the single-worker FIFO today, but
|
||||
`depends_on` allows explicit cross-kind sequencing when parallel workers are added).
|
||||
The queue is in-memory only and lost on hive-c0re restart — deliberate:
|
||||
desired state is re-derived at boot from the DB + rev markers (see _Boot
|
||||
reconcile_), so there is no durable-recovery machinery to go wrong.
|
||||
|
||||
`depends_on` is part of the dedup key: two entries with the same `(kind, agent,
|
||||
parent_id, inputs, approval_id)` but different dep sets are treated as distinct work.
|
||||
### Dedup, cancel, history
|
||||
|
||||
**Worker re-notification**: the worker drains `take_next()` in a tight loop after
|
||||
each entry finishes. When a dep entry transitions to terminal, the loop re-evaluates
|
||||
the queue immediately, so downstream entries are unblocked with no extra wakeup. No
|
||||
additional `notify_one()` call is needed.
|
||||
Dedup at **DAG granularity**: a repeat submit against a DAG whose roll-up is
|
||||
still `Queued` with the same `(template, agent, parent_id, approval_id)` —
|
||||
plus `inputs` for meta-updates and the perm-type discriminant for perm
|
||||
changes — returns the existing id and appends an "also requested by …" line.
|
||||
`parent_id` in the key keeps a cascade child from collapsing into a
|
||||
standalone or sweep rebuild. Running/terminal DAGs never dedup.
|
||||
|
||||
**Circular-dep caveat**: if A depends on B and B depends on A, neither entry ever
|
||||
becomes runnable — the worker skips both indefinitely with no error. Callers must
|
||||
ensure acyclic dep graphs. Cycle detection is deferred to a future iteration (when
|
||||
parallel workers make a stuck queue more visible).
|
||||
Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't
|
||||
interruptible); `cancel_children` cancels a parent's still-queued child DAGs.
|
||||
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
|
||||
`Cancelled` / `Done`. The snapshot retains the 5 most recent terminal DAGs
|
||||
per template.
|
||||
|
||||
### Approvals
|
||||
|
||||
`ApplyCommit` / `MergeConfigPr` approvals ride as single-node
|
||||
`ApprovalDeploy` DAGs: the two-phase `prepare_deploy` / `finalize_deploy` /
|
||||
`abort_deploy` meta orchestration stays inside `actions.rs` in v1
|
||||
(deliberately not modeled as scheduler nodes) and resolves the approval
|
||||
itself. `Spawn` and `UpdateMetaInputs` approvals map onto the ordinary
|
||||
`spawn` / `meta-update` shapes; the scheduler fires
|
||||
`actions::resolve_approval_dag` exactly once when such a DAG settles
|
||||
terminal (including cancelled-while-queued, which fails the approval instead
|
||||
of dangling it).
|
||||
|
||||
### Wire shape
|
||||
|
||||
`RebuildQueueChanged { seq, queue: [DagView…] }` (event name kept). Each
|
||||
`DagView` carries the old entry-level fields (`id`, `kind` = template string,
|
||||
roll-up `state`, `agent`, `source`, `parent_id`, `reason`, timestamps,
|
||||
`inputs`, `approval_id`) plus `nodes: [NodeView…]` — per-node `kind`, `deps`,
|
||||
`state`, `step`, `build_log_id`, timestamps, `error`. Step labels and build
|
||||
logs are **per-node**; the dashboard renders the node chain on each queue
|
||||
card and keys the live-log panel off the running node.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -115,31 +177,31 @@ render.
|
|||
|
||||
---
|
||||
|
||||
## Auto-update sweep
|
||||
## Boot reconcile
|
||||
|
||||
On startup, `auto_update.rs` rebuilds containers that actually need it. Two skip
|
||||
rules keep boot-time work minimal:
|
||||
On startup, `auto_update::run` classifies every agent by rev freshness (the
|
||||
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
||||
vs the current flake path) and persisted `wanted` intent, then:
|
||||
|
||||
1. **Stopped containers** are deferred: the startup sweep enqueues nothing for them.
|
||||
When the operator later starts a stopped container (via the dashboard or the
|
||||
`start` MCP tool), both `run_start` (queue path) and `handle_start` (socket path)
|
||||
check the rev marker first — if it's stale, the start is silently upgraded to a
|
||||
full rebuild+start so the container runs current nix derivations.
|
||||
1. **Config path** — when *any* marker is stale, submit one `StartupSweep`
|
||||
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that fans out `Rebuild`
|
||||
children for the stale agents whose `wanted = Up` (topology-sorted, parents
|
||||
first). Stale but wanted-offline agents get no boot-time nix work — their
|
||||
rebuild happens on their next start (the start submit path upgrades a
|
||||
stale start to rebuild+start), which is also why the lock bump runs even
|
||||
when every stale agent is offline: those later start-upgrades must build
|
||||
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
|
||||
agent (back) up, covering both the running-stale and stopped-but-wanted-up
|
||||
cases.
|
||||
|
||||
2. **Running containers with a matching rev marker** are skipped: if the per-agent
|
||||
`.{name}.hyperhive-rev` file under `/var/lib/hyperhive/applied/` already holds
|
||||
the current flake rev, no nix work is needed and the entry is omitted entirely.
|
||||
2. **Power path** — every agent whose observed state drifted from `wanted`
|
||||
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
||||
|
||||
`auto_update::run` enqueues a single `StartupSweep` parent entry (`kind =
|
||||
startup_sweep`, `agent = "hyperhive"`) followed by per-agent `Rebuild` children
|
||||
for the agents that do need rebuilding (`source = startup_sweep`, `parent_id =
|
||||
sweep_id`). The sweep description records the rebuild / deferred / skipped counts
|
||||
so the operator can see at a glance how much work the boot triggered. The child
|
||||
rebuilds drain sequentially through the queue; the dashboard renders them nested
|
||||
under the parent.
|
||||
|
||||
Before the sweep-grouping change, each boot enqueued flat `Rebuild` entries with
|
||||
`source = AutoUpdate` and no parent — visible but ungrouped.
|
||||
Booting with no config change performs **no meta commit** — only reconciles.
|
||||
The sweep reason records the rebuild / deferred / up-to-date counts so the
|
||||
operator sees at a glance how much work the boot triggered. Agents without an
|
||||
`agent_power` row are seeded from observed state during classification (the
|
||||
one-time migration; thereafter the DB is authoritative).
|
||||
|
||||
## Meta flake
|
||||
|
||||
|
|
@ -157,8 +219,13 @@ Key operations:
|
|||
`RequestApplyCommit` path so a failed `nixos-container update` leaves no orphan
|
||||
commit in meta. Prepare writes the new lock without committing; finalize commits
|
||||
with the deploy message; abort restores the lock.
|
||||
- **`lock_update_hyperhive`** — one-shot for the auto-update path: bumps the
|
||||
`hyperhive` input lock, commits, cascades agent rebuilds.
|
||||
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
|
||||
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
|
||||
the scheduler fans out the agent rebuilds on completion.
|
||||
|
||||
Every public `meta.rs` operation takes the module's internal `META_LOCK`
|
||||
mutex, so concurrent job-queue nodes (and the approval deploy pipeline) never
|
||||
race on the repo's `.git/index.lock`.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -179,34 +246,26 @@ new profile and skips the in-container `switch-to-configuration`. The subsequent
|
|||
`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
|
||||
one go, rather than the double-bounce a live `update` would trigger.
|
||||
|
||||
Sequence for a running container:
|
||||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `prebuild_toplevel` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch + build
|
||||
happen out-of-band. `nixos-container update` then finds the result cached and
|
||||
skips straight to the profile-swap. Build failures surface here, before the
|
||||
running container is touched.
|
||||
2. `nixos-container stop` — bring the container down.
|
||||
3. `nixos-container update --flake meta#<name>` — profile-swap (near-instant after
|
||||
the prebuild).
|
||||
4. `nixos-container start` — boot into the new generation; the in-container
|
||||
activation script transitions old → new.
|
||||
1. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch +
|
||||
build happen out-of-band. `nixos-container update` then finds the result
|
||||
cached and skips straight to the profile-swap. Build failures surface
|
||||
here, before the running container is touched. (Runs even for a stopped
|
||||
container — same total nix work, one uniform DAG shape.)
|
||||
2. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
3. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
|
||||
(near-instant after the prebuild).
|
||||
4. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot — the old
|
||||
"deferred start" split, now structural.
|
||||
|
||||
If the container is already stopped, step 1 is skipped (no downtime to shave — no
|
||||
point evaluating the flake twice).
|
||||
|
||||
**Deferred start (queue-dispatched rebuilds):** step 4 can take a while
|
||||
(container boot), and holding the serialized build lane through it delays the
|
||||
next queued rebuild's nix build for no reason. Queue-dispatched rebuilds
|
||||
therefore pass `defer_start` — `rebuild_no_meta` skips the start and returns
|
||||
`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead
|
||||
(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under
|
||||
it — the same split the graceful-stop path uses for its container stop). The
|
||||
build-lane entry completes when the profile-swap finishes; a start failure
|
||||
surfaces on the `Start` entry, which runs with the cold-start fallback. Direct
|
||||
callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy
|
||||
flow which verifies the agent comes back up before finalizing) keep the start
|
||||
inline.
|
||||
The approval apply-commit pipeline still drives `lifecycle::rebuild_no_meta`
|
||||
(the fused stop/update/start path with an inline start) inside its
|
||||
`ApprovalDeploy` node, because it verifies the agent comes back up before
|
||||
finalizing the deploy tag.
|
||||
|
||||
### Cold-start fallback
|
||||
|
||||
|
|
@ -218,9 +277,8 @@ half-started at that point.
|
|||
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
|
||||
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
|
||||
Both errors are preserved and surfaced if the cold-start also fails. The fallback
|
||||
lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild
|
||||
path and the queue's fast-lane `Start` handler (which the deferred
|
||||
start-after-rebuild rides).
|
||||
lives in `lifecycle::start_with_fallback`, shared by the apply-commit deploy's
|
||||
inline start and every `Reconcile` node's start action.
|
||||
|
||||
### Spawn path (new container)
|
||||
|
||||
|
|
@ -241,9 +299,18 @@ or the flake root directly — none of which exist in the rendered meta flake.
|
|||
|
||||
## Host-level resource + performance options
|
||||
|
||||
Three `services.hyperhive.c0re.*` options tune container resource
|
||||
limits and first-spawn latency. All three apply uniformly to every
|
||||
agent container.
|
||||
A handful of `services.hyperhive.c0re.*` options tune container resource
|
||||
limits, build parallelism, and first-spawn latency.
|
||||
|
||||
### Build slots
|
||||
|
||||
`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
|
||||
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
|
||||
concurrently. The default serializes all heavy nix work like the pre-DAG
|
||||
rebuild queue did; raise it on hosts with the cores/RAM to build several
|
||||
agent toplevels at once. Per-agent correctness is independent of the count —
|
||||
each agent's container-affecting ops serialize on its lifecycle lease
|
||||
regardless.
|
||||
|
||||
### Container resource limits
|
||||
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@
|
|||
|
||||
Where state lives, what survives what, and how it's bounded.
|
||||
|
||||
## Three sqlite databases
|
||||
## Sqlite databases
|
||||
|
||||
### `/var/lib/hyperhive/db/broker.sqlite` (host)
|
||||
|
||||
Six tables, all in one file — four queues plus the schedule
|
||||
header/targets split:
|
||||
Seven tables, all in one file — four queues, the schedule
|
||||
header/targets split, and the per-agent power-intent registry:
|
||||
|
||||
- `messages` — every inter-agent / operator-bound message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
|
|
@ -44,6 +44,17 @@ header/targets split:
|
|||
last_fired_at_unix / last_result`. `ON DELETE CASCADE` from
|
||||
`scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
|
||||
per connection (set at open).
|
||||
- `agent_power` — one tiny row per agent: `agent PK / wanted (up |
|
||||
offline) / updated_at` — the durable power *intent* behind the job
|
||||
queue's desired-state reconciliation
|
||||
(`docs/coordinator.md::Job queue`; owner: `hive-c0re/src/power.rs`).
|
||||
Written synchronously by every operator/agent power action
|
||||
(dashboard start/stop, `hivectl stop`, the MCP kill/start tools,
|
||||
spawn approval); read by `Reconcile` nodes and the boot reconcile.
|
||||
Intent survives hive-c0re restarts — in-flight queue work
|
||||
deliberately does not. Agents without a row are seeded from
|
||||
observed state on first touch (running ⇒ `up`); destroy deletes
|
||||
the row.
|
||||
|
||||
Retention:
|
||||
|
||||
|
|
@ -63,6 +74,8 @@ Retention:
|
|||
(`cancel_schedule` MCP / dashboard ✗) which tombstones via
|
||||
`cancelled_at_unix`, then `reap_cancelled` drops the row on
|
||||
the next worker pass.
|
||||
- `agent_power` rows live until the agent is destroyed (one row per
|
||||
agent — nothing to vacuum).
|
||||
|
||||
### `/harness/hyperhive-events.sqlite` (per agent)
|
||||
|
||||
|
|
|
|||
|
|
@ -407,7 +407,7 @@ Stop containers hive-wide in one operator action. Bare `hivectl stop` stops **ev
|
|||
* `--forge` — The forge container (`hive-forge`)
|
||||
* `--gateway` — The gateway container (`hive-gateway`)
|
||||
* `--matrix` — The matrix container (`hive-matrix`)
|
||||
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent is enqueued as a `GracefulStop` on the rebuild queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). Applies to agents only
|
||||
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: the harness is signalled, runs one stop-checkpoint turn to flush durable `/state`, drains, then the container is stopped (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue