hive-c0re coordinator internals

Architecture notes for the hive-c0re coordinator daemon's internal subsystems. For the public API surface (dashboard, socket protocol) see docs/conventions.md and docs/persistence.md.

Rebuild queue

Every long-running container/meta operation (rebuild, meta-update, first-spawn) goes through the global rebuild queue (hive-c0re/src/rebuild_queue.rs). A single background worker drains it in FIFO order so two nixos-container update runs on the same agent never overlap, and a fresh agent rebuild never races a meta-update's lock bump.

Why one queue

Before the rebuild queue landed, four independent call paths could fire auto_update::rebuild_agent concurrently:

Dashboard manual rebuild button
update-all / meta-update cascade
Approval handler (apply-commit / spawn)
Startup auto-update sweep

Nothing serialised them. nix-daemon serialises the actual store ops, but the rest of rebuild_agent (token sync, kick, rescan, lock-bump emit) interleaved unpredictably. The single-worker queue gives operators a visible, ordered runway and lets the UI render "what's about to happen" instead of "something might be happening somewhere."

Queue kinds

Kind	Description
`Rebuild`	Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path.
`MetaUpdate`	`nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id.
`Spawn`	First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV.
`Destroy`	For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue.

Intentionally not queued (sub-second ops; adding them adds dashboard noise without serving the "one at a time" goal): start, stop, restart, kill.

Dedup

Enqueueing (kind, agent) that already has a Queued entry returns the existing entry's id and appends the new request as an "also requested by …" line. Running entries do not dedup — a re-queue during a run is legitimate (something changed since the current run started).

Sources

Source	Meaning
`Manual`	Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, manager tool).
`AutoUpdate`	Fired by the startup sweep or a meta-update cascade.
`Approval`	Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`.

Cascade parent tracking

MetaUpdate entries fan out Rebuild children, each carrying parent_id = <meta_update_id>. The dashboard groups children under their parent in the queue panel so the operator sees the whole meta-update cascade as a tree, not a flat list.

Step labels

Each queue entry has a mutable step: Option<String> field that the worker updates as it progresses through lifecycle phases ("nix build", "nixos-container stop", "nixos-container update", "nixos-container start"). The dashboard polls /api/state and renders the current step beneath the running entry so the operator can see which phase is taking time.

Container view

container_view.rs maintains an in-memory snapshot of every nixos-container's systemd service state. It is polled on coordinator startup and re-scanned after every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects the actual container status without a live nixos-container list call on each render.

Auto-update sweep

On startup, auto_update.rs rebuilds every known container unconditionally. nixos-container update is a no-op at the nix level when nothing changed (same store path), so the cost is low and avoids rev-marker staleness — all agents always need an update pass when any meta commit lands. Each rebuild is enqueued as a Rebuild entry with source = AutoUpdate and drains through the global queue.

Meta flake

meta.rs owns the single coordinator-managed flake at /var/lib/hyperhive/meta/. This flake consumes every agent's applied config repo as a flake input and exports one nixosConfiguration per agent. Container lifecycle ops drive the lock file so meta's git log is the system-wide deploy audit trail.

Key operations:

sync_agents (idempotent) — render flake.nix for the current agent set, init the repo on first call, relock if the rendered contents changed, commit. Called by spawn / destroy / startup migration.
prepare_deploy + finalize_deploy / abort_deploy — two-phase for the RequestApplyCommit path so a failed nixos-container update leaves no orphan commit in meta. Prepare writes the new lock without committing; finalize commits with the deploy message; abort restores the lock.
lock_update_hyperhive — one-shot for the auto-update path: bumps the hyperhive input lock, commits, cascades agent rebuilds.

5.3 KiB Raw Blame History