diff --git a/docs/coordinator.md b/docs/coordinator.md new file mode 100644 index 00000000..5063356e --- /dev/null +++ b/docs/coordinator.md @@ -0,0 +1,121 @@ +# hive-c0re coordinator internals + +Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems. +For the public API surface (dashboard, socket protocol) see `docs/conventions.md` +and `docs/persistence.md`. + +--- + +## Rebuild queue + +Every long-running container/meta operation (rebuild, meta-update, first-spawn) +goes through the global rebuild queue (`hive-c0re/src/rebuild_queue.rs`). A single +background worker drains it in FIFO order so two `nixos-container update` runs on +the same agent never overlap, and a fresh agent rebuild never races a meta-update's +lock bump. + +### Why one queue + +Before the rebuild queue landed, four independent call paths could fire +`auto_update::rebuild_agent` concurrently: + +- Dashboard manual rebuild button +- `update-all` / `meta-update` cascade +- Approval handler (apply-commit / spawn) +- Startup auto-update sweep + +Nothing serialised them. `nix-daemon` serialises the actual store ops, but the rest +of `rebuild_agent` (token sync, kick, rescan, lock-bump emit) interleaved +unpredictably. The single-worker queue gives operators a visible, ordered runway and +lets the UI render "what's about to happen" instead of "something might be happening +somewhere." + +### Queue kinds + +| Kind | Description | +|------|-------------| +| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. | +| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. | +| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. | +| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. | + +**Intentionally not queued** (sub-second ops; adding them adds dashboard noise +without serving the "one at a time" goal): `start`, `stop`, `restart`, `kill`. + +### Dedup + +Enqueueing `(kind, agent)` that already has a `Queued` entry returns the existing +entry's id and appends the new request as an "also requested by …" line. Running +entries do not dedup — a re-queue during a run is legitimate (something changed +since the current run started). + +### Sources + +| Source | Meaning | +|--------|---------| +| `Manual` | Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, manager tool). | +| `AutoUpdate` | Fired by the startup sweep or a meta-update cascade. | +| `Approval` | Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`. | + +### Cascade parent tracking + +`MetaUpdate` entries fan out `Rebuild` children, each carrying +`parent_id = `. The dashboard groups children under their parent +in the queue panel so the operator sees the whole meta-update cascade as a tree, +not a flat list. + +### Step labels + +Each queue entry has a mutable `step: Option` field that the worker updates +as it progresses through lifecycle phases (`"nix build"`, `"nixos-container stop"`, +`"nixos-container update"`, `"nixos-container start"`). The dashboard polls +`/api/state` and renders the current step beneath the running entry so the operator +can see which phase is taking time. + +--- + +## Container view + +`container_view.rs` maintains an in-memory snapshot of every nixos-container's +systemd service state. It is polled on coordinator startup and re-scanned after +every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects +the actual container status without a live `nixos-container list` call on each +render. + +--- + +## Auto-update sweep + +On startup, `auto_update.rs` rebuilds every known container unconditionally. +`nixos-container update` is a no-op at the nix level when nothing changed (same +store path), so the cost is low and avoids rev-marker staleness — all agents always +need an update pass when any meta commit lands. Each rebuild is enqueued as a +`Rebuild` entry with `source = AutoUpdate` and drains through the global queue. + +## Meta flake + +`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`. +This flake consumes every agent's applied config repo as a flake input and exports +one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so +meta's git log is the system-wide deploy audit trail. + +Key operations: + +- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set, + init the repo on first call, relock if the rendered contents changed, commit. + Called by spawn / destroy / startup migration. +- **`prepare_deploy` + `finalize_deploy` / `abort_deploy`** — two-phase for the + `RequestApplyCommit` path so a failed `nixos-container update` leaves no orphan + commit in meta. Prepare writes the new lock without committing; finalize commits + with the deploy message; abort restores the lock. +- **`lock_update_hyperhive`** — one-shot for the auto-update path: bumps the + `hyperhive` input lock, commits, cascades agent rebuilds. + +--- + +## See also + +- `docs/approvals.md` — approval flow + scheduled prompts +- `docs/persistence.md` — SQLite schema, state-dir layout +- `docs/conventions.md` — wire protocol, recipient sentinels +- `docs/agent-hierarchy.md` — topology and parent/child relations diff --git a/hive-c0re/src/auto_update.rs b/hive-c0re/src/auto_update.rs index 456fd958..36d89e92 100644 --- a/hive-c0re/src/auto_update.rs +++ b/hive-c0re/src/auto_update.rs @@ -1,9 +1,8 @@ //! Startup auto-update: on `hive-c0re serve` boot, rebuild every known //! container unconditionally. `nixos-container update` is a no-op at the -//! nix level when nothing changed (same store path), so the cost of always -//! running it on startup is low and avoids the complexity of rev-marker -//! staleness (issue #179: all agents always needed update when any meta -//! commit landed). +//! nix level when nothing changed (same store path), so the cost is low +//! and avoids rev-marker staleness (all agents always need an update pass +//! when any meta commit lands). See `docs/coordinator.md::Auto-update sweep`. use std::path::{Path, PathBuf}; use std::sync::Arc; diff --git a/hive-c0re/src/rebuild_queue.rs b/hive-c0re/src/rebuild_queue.rs index 6d640b48..eeecaf2c 100644 --- a/hive-c0re/src/rebuild_queue.rs +++ b/hive-c0re/src/rebuild_queue.rs @@ -1,53 +1,8 @@ -//! Global rebuild queue. -//! -//! Every long-running container/meta operation (rebuild, meta-update, -//! first-spawn) goes through this queue. A single background worker -//! drains it in FIFO order so we never overlap two `nixos-container -//! update` runs on the same agent and never start a fresh agent rebuild -//! while a meta-update's lock bump is mid-flight. -//! -//! ## Why one queue -//! -//! Before this module landed, four independent call paths could fire -//! `auto_update::rebuild_agent` concurrently: -//! - dashboard manual rebuild button -//! - `update-all` / `meta-update` cascade -//! - approval handler (apply-commit / spawn) -//! - startup auto-update sweep -//! -//! Nothing serialised them. nix-daemon serialises the actual store -//! ops, but the rest of `rebuild_agent` (token sync, kick, rescan, -//! lock-bump emit) interleaved unpredictably. The single-worker queue -//! gives operators a visible, ordered runway and lets the UI render -//! "what's about to happen" instead of "something might be happening -//! somewhere." -//! -//! ## Scope -//! -//! In-queue kinds: -//! - `Rebuild` — a single-agent rebuild (covers manual / approval-driven / -//! auto-update / meta-update cascade variants — they all funnel here). -//! - `MetaUpdate` — `nix flake update` on the meta flake. The worker -//! runs the lock bump itself, then enqueues a cascade of `Rebuild` -//! entries with `parent_id` set to the meta-update's id. -//! - `Spawn` — first-deploy of an agent (approval-driven). Same -//! serialisation as `Rebuild` from the operator's POV. -//! - `Destroy` — for future use (`destroy --purge` does real I/O); not -//! currently routed through the queue. -//! -//! Out of scope (intentionally not queued — these are sub-second ops -//! and adding them adds visual noise without serving the "one at a -//! time" goal): -//! - `start` / `stop` / `restart` -//! - `kill` -//! -//! ## Dedup -//! -//! Enqueueing `(kind, agent)` that already has a `Queued` entry returns -//! the existing entry's id and appends the new reason as an -//! "also requested by …" line. Running entries do not dedup — a -//! re-queue during a run is legitimate (something changed since the -//! current run started). +//! Global rebuild queue — serialises all long-running container/meta +//! operations (rebuild, meta-update, first-spawn) through a single +//! background worker. Design rationale, kind taxonomy, dedup rules, +//! cascade parent tracking, and step labels: +//! `docs/coordinator.md::Rebuild queue`. use std::collections::VecDeque; use std::sync::Mutex;