docs: add coordinator.md, move rebuild-queue prose from code (#715)
This commit is contained in:
damocles
mara
parent
89030ff1e1
commit
64f11a862d
3 changed files with 129 additions and 54 deletions
121
docs/coordinator.md
Normal file
121
docs/coordinator.md
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
# hive-c0re coordinator internals
|
||||
|
||||
Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems.
|
||||
For the public API surface (dashboard, socket protocol) see `docs/conventions.md`
|
||||
and `docs/persistence.md`.
|
||||
|
||||
---
|
||||
|
||||
## Rebuild queue
|
||||
|
||||
Every long-running container/meta operation (rebuild, meta-update, first-spawn)
|
||||
goes through the global rebuild queue (`hive-c0re/src/rebuild_queue.rs`). A single
|
||||
background worker drains it in FIFO order so two `nixos-container update` runs on
|
||||
the same agent never overlap, and a fresh agent rebuild never races a meta-update's
|
||||
lock bump.
|
||||
|
||||
### Why one queue
|
||||
|
||||
Before the rebuild queue landed, four independent call paths could fire
|
||||
`auto_update::rebuild_agent` concurrently:
|
||||
|
||||
- Dashboard manual rebuild button
|
||||
- `update-all` / `meta-update` cascade
|
||||
- Approval handler (apply-commit / spawn)
|
||||
- Startup auto-update sweep
|
||||
|
||||
Nothing serialised them. `nix-daemon` serialises the actual store ops, but the rest
|
||||
of `rebuild_agent` (token sync, kick, rescan, lock-bump emit) interleaved
|
||||
unpredictably. The single-worker queue gives operators a visible, ordered runway and
|
||||
lets the UI render "what's about to happen" instead of "something might be happening
|
||||
somewhere."
|
||||
|
||||
### Queue kinds
|
||||
|
||||
| Kind | Description |
|
||||
|------|-------------|
|
||||
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. |
|
||||
| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. |
|
||||
| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. |
|
||||
| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. |
|
||||
|
||||
**Intentionally not queued** (sub-second ops; adding them adds dashboard noise
|
||||
without serving the "one at a time" goal): `start`, `stop`, `restart`, `kill`.
|
||||
|
||||
### Dedup
|
||||
|
||||
Enqueueing `(kind, agent)` that already has a `Queued` entry returns the existing
|
||||
entry's id and appends the new request as an "also requested by …" line. Running
|
||||
entries do not dedup — a re-queue during a run is legitimate (something changed
|
||||
since the current run started).
|
||||
|
||||
### Sources
|
||||
|
||||
| Source | Meaning |
|
||||
|--------|---------|
|
||||
| `Manual` | Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, manager tool). |
|
||||
| `AutoUpdate` | Fired by the startup sweep or a meta-update cascade. |
|
||||
| `Approval` | Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`. |
|
||||
|
||||
### Cascade parent tracking
|
||||
|
||||
`MetaUpdate` entries fan out `Rebuild` children, each carrying
|
||||
`parent_id = <meta_update_id>`. The dashboard groups children under their parent
|
||||
in the queue panel so the operator sees the whole meta-update cascade as a tree,
|
||||
not a flat list.
|
||||
|
||||
### Step labels
|
||||
|
||||
Each queue entry has a mutable `step: Option<String>` field that the worker updates
|
||||
as it progresses through lifecycle phases (`"nix build"`, `"nixos-container stop"`,
|
||||
`"nixos-container update"`, `"nixos-container start"`). The dashboard polls
|
||||
`/api/state` and renders the current step beneath the running entry so the operator
|
||||
can see which phase is taking time.
|
||||
|
||||
---
|
||||
|
||||
## Container view
|
||||
|
||||
`container_view.rs` maintains an in-memory snapshot of every nixos-container's
|
||||
systemd service state. It is polled on coordinator startup and re-scanned after
|
||||
every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects
|
||||
the actual container status without a live `nixos-container list` call on each
|
||||
render.
|
||||
|
||||
---
|
||||
|
||||
## Auto-update sweep
|
||||
|
||||
On startup, `auto_update.rs` rebuilds every known container unconditionally.
|
||||
`nixos-container update` is a no-op at the nix level when nothing changed (same
|
||||
store path), so the cost is low and avoids rev-marker staleness — all agents always
|
||||
need an update pass when any meta commit lands. Each rebuild is enqueued as a
|
||||
`Rebuild` entry with `source = AutoUpdate` and drains through the global queue.
|
||||
|
||||
## Meta flake
|
||||
|
||||
`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`.
|
||||
This flake consumes every agent's applied config repo as a flake input and exports
|
||||
one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so
|
||||
meta's git log is the system-wide deploy audit trail.
|
||||
|
||||
Key operations:
|
||||
|
||||
- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set,
|
||||
init the repo on first call, relock if the rendered contents changed, commit.
|
||||
Called by spawn / destroy / startup migration.
|
||||
- **`prepare_deploy` + `finalize_deploy` / `abort_deploy`** — two-phase for the
|
||||
`RequestApplyCommit` path so a failed `nixos-container update` leaves no orphan
|
||||
commit in meta. Prepare writes the new lock without committing; finalize commits
|
||||
with the deploy message; abort restores the lock.
|
||||
- **`lock_update_hyperhive`** — one-shot for the auto-update path: bumps the
|
||||
`hyperhive` input lock, commits, cascades agent rebuilds.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/approvals.md` — approval flow + scheduled prompts
|
||||
- `docs/persistence.md` — SQLite schema, state-dir layout
|
||||
- `docs/conventions.md` — wire protocol, recipient sentinels
|
||||
- `docs/agent-hierarchy.md` — topology and parent/child relations
|
||||
|
|
@ -1,9 +1,8 @@
|
|||
//! Startup auto-update: on `hive-c0re serve` boot, rebuild every known
|
||||
//! container unconditionally. `nixos-container update` is a no-op at the
|
||||
//! nix level when nothing changed (same store path), so the cost of always
|
||||
//! running it on startup is low and avoids the complexity of rev-marker
|
||||
//! staleness (issue #179: all agents always needed update when any meta
|
||||
//! commit landed).
|
||||
//! nix level when nothing changed (same store path), so the cost is low
|
||||
//! and avoids rev-marker staleness (all agents always need an update pass
|
||||
//! when any meta commit lands). See `docs/coordinator.md::Auto-update sweep`.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
|
|
|||
|
|
@ -1,53 +1,8 @@
|
|||
//! Global rebuild queue.
|
||||
//!
|
||||
//! Every long-running container/meta operation (rebuild, meta-update,
|
||||
//! first-spawn) goes through this queue. A single background worker
|
||||
//! drains it in FIFO order so we never overlap two `nixos-container
|
||||
//! update` runs on the same agent and never start a fresh agent rebuild
|
||||
//! while a meta-update's lock bump is mid-flight.
|
||||
//!
|
||||
//! ## Why one queue
|
||||
//!
|
||||
//! Before this module landed, four independent call paths could fire
|
||||
//! `auto_update::rebuild_agent` concurrently:
|
||||
//! - dashboard manual rebuild button
|
||||
//! - `update-all` / `meta-update` cascade
|
||||
//! - approval handler (apply-commit / spawn)
|
||||
//! - startup auto-update sweep
|
||||
//!
|
||||
//! Nothing serialised them. nix-daemon serialises the actual store
|
||||
//! ops, but the rest of `rebuild_agent` (token sync, kick, rescan,
|
||||
//! lock-bump emit) interleaved unpredictably. The single-worker queue
|
||||
//! gives operators a visible, ordered runway and lets the UI render
|
||||
//! "what's about to happen" instead of "something might be happening
|
||||
//! somewhere."
|
||||
//!
|
||||
//! ## Scope
|
||||
//!
|
||||
//! In-queue kinds:
|
||||
//! - `Rebuild` — a single-agent rebuild (covers manual / approval-driven /
|
||||
//! auto-update / meta-update cascade variants — they all funnel here).
|
||||
//! - `MetaUpdate` — `nix flake update` on the meta flake. The worker
|
||||
//! runs the lock bump itself, then enqueues a cascade of `Rebuild`
|
||||
//! entries with `parent_id` set to the meta-update's id.
|
||||
//! - `Spawn` — first-deploy of an agent (approval-driven). Same
|
||||
//! serialisation as `Rebuild` from the operator's POV.
|
||||
//! - `Destroy` — for future use (`destroy --purge` does real I/O); not
|
||||
//! currently routed through the queue.
|
||||
//!
|
||||
//! Out of scope (intentionally not queued — these are sub-second ops
|
||||
//! and adding them adds visual noise without serving the "one at a
|
||||
//! time" goal):
|
||||
//! - `start` / `stop` / `restart`
|
||||
//! - `kill`
|
||||
//!
|
||||
//! ## Dedup
|
||||
//!
|
||||
//! Enqueueing `(kind, agent)` that already has a `Queued` entry returns
|
||||
//! the existing entry's id and appends the new reason as an
|
||||
//! "also requested by …" line. Running entries do not dedup — a
|
||||
//! re-queue during a run is legitimate (something changed since the
|
||||
//! current run started).
|
||||
//! Global rebuild queue — serialises all long-running container/meta
|
||||
//! operations (rebuild, meta-update, first-spawn) through a single
|
||||
//! background worker. Design rationale, kind taxonomy, dedup rules,
|
||||
//! cascade parent tracking, and step labels:
|
||||
//! `docs/coordinator.md::Rebuild queue`.
|
||||
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::Mutex;
|
||||
|
|
|
|||
Loading…
Reference in a new issue