Watch

damocles 88cdab411e docs: add container lifecycle section to coordinator.md, trim lifecycle.rs prose

2026-06-01 11:38:42 +02:00

8.1 KiB

Raw Blame History

hive-c0re coordinator internals

Architecture notes for the hive-c0re coordinator daemon's internal subsystems. For the public API surface (dashboard, socket protocol) see docs/conventions.md and docs/persistence.md.

Rebuild queue

Every long-running container/meta operation (rebuild, meta-update, first-spawn) goes through the global rebuild queue (hive-c0re/src/rebuild_queue.rs). A single background worker drains it in FIFO order so two nixos-container update runs on the same agent never overlap, and a fresh agent rebuild never races a meta-update's lock bump.

Why one queue

Before the rebuild queue landed, four independent call paths could fire auto_update::rebuild_agent concurrently:

Dashboard manual rebuild button
update-all / meta-update cascade
Approval handler (apply-commit / spawn)
Startup auto-update sweep

Nothing serialised them. nix-daemon serialises the actual store ops, but the rest of rebuild_agent (token sync, kick, rescan, lock-bump emit) interleaved unpredictably. The single-worker queue gives operators a visible, ordered runway and lets the UI render "what's about to happen" instead of "something might be happening somewhere."

Queue kinds

Kind	Description
`Rebuild`	Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path.
`MetaUpdate`	`nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id.
`Spawn`	First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV.
`Destroy`	For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue.

Intentionally not queued (sub-second ops; adding them adds dashboard noise without serving the "one at a time" goal): start, stop, restart, kill.

Dedup

Enqueueing (kind, agent) that already has a Queued entry returns the existing entry's id and appends the new request as an "also requested by …" line. Running entries do not dedup — a re-queue during a run is legitimate (something changed since the current run started).

Sources

Source	Meaning
`Manual`	Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, manager tool).
`AutoUpdate`	Fired by the startup sweep or a meta-update cascade.
`Approval`	Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`.

Cascade parent tracking

MetaUpdate entries fan out Rebuild children, each carrying parent_id = <meta_update_id>. The dashboard groups children under their parent in the queue panel so the operator sees the whole meta-update cascade as a tree, not a flat list.

Step labels

Each queue entry has a mutable step: Option<String> field that the worker updates as it progresses through lifecycle phases ("nix build", "nixos-container stop", "nixos-container update", "nixos-container start"). The dashboard polls /api/state and renders the current step beneath the running entry so the operator can see which phase is taking time.

Container view

container_view.rs maintains an in-memory snapshot of every nixos-container's systemd service state. It is polled on coordinator startup and re-scanned after every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects the actual container status without a live nixos-container list call on each render.

Auto-update sweep

On startup, auto_update.rs rebuilds every known container unconditionally. nixos-container update is a no-op at the nix level when nothing changed (same store path), so the cost is low and avoids rev-marker staleness — all agents always need an update pass when any meta commit lands. Each rebuild is enqueued as a Rebuild entry with source = AutoUpdate and drains through the global queue.

Meta flake

meta.rs owns the single coordinator-managed flake at /var/lib/hyperhive/meta/. This flake consumes every agent's applied config repo as a flake input and exports one nixosConfiguration per agent. Container lifecycle ops drive the lock file so meta's git log is the system-wide deploy audit trail.

Key operations:

sync_agents (idempotent) — render flake.nix for the current agent set, init the repo on first call, relock if the rendered contents changed, commit. Called by spawn / destroy / startup migration.
prepare_deploy + finalize_deploy / abort_deploy — two-phase for the RequestApplyCommit path so a failed nixos-container update leaves no orphan commit in meta. Prepare writes the new lock without committing; finalize commits with the deploy message; abort restores the lock.
lock_update_hyperhive — one-shot for the auto-update path: bumps the hyperhive input lock, commits, cascades agent rebuilds.

Container lifecycle (`lifecycle.rs`)

Every container operation ultimately calls into lifecycle.rs. Two paths exist: rebuild (existing container) and spawn (first-time creation).

Rebuild path (existing container)

Goal: apply the new system profile and any EXTRA_NSPAWN_FLAGS / drop-in changes in a single start, with minimum downtime.

nixos-container update only runs systemctl reload container@<c> when the container is already up (per isContainerRunning in nixos-container.pl). Stopping first turns update into a boot-style operation: it builds + nix-env --sets the new profile and skips the in-container switch-to-configuration. The subsequent start then applies both the new profile and any EXTRA_NSPAWN_FLAGS changes in one go, rather than the double-bounce a live update would trigger.

Sequence for a running container:

prebuild_toplevel — build the new system.build.toplevel before stopping. The container keeps serving the previous generation while eval + fetch + build happen out-of-band. nixos-container update then finds the result cached and skips straight to the profile-swap. Build failures surface here, before the running container is touched.
nixos-container stop — bring the container down.
nixos-container update --flake meta#<name> — profile-swap (near-instant after the prebuild).
nixos-container start — boot into the new generation; the in-container activation script transitions old → new.

If the container is already stopped, step 1 is skipped (no downtime to shave — no point evaluating the flake twice).

Cold-start fallback

start after update can exit non-zero when packages are removed between generations: the old-generation activation script references units that no longer exist in the new closure, causing systemd to exit non-zero. The container may be half-started at that point.

Fallback: stop (graceful SIGTERM drain) → kill (SIGKILL any lingering processes) → start (clean cold-start, no generation transition, new activation runs cleanly). Both errors are preserved and surfaced if the cold-start also fails.

Spawn path (new container)

For a first-time create, nixos-container create is atomic: if the build fails, no container record is left to clean up. A separate prebuild would just duplicate the eval, so it's skipped. Sequence: create --flake meta#<name> → write nspawn flags → systemctl daemon-reload → start.

Prebuild attr path

nix build does not auto-resolve meta#<name> against nixosConfigurations the way nixos-container does internally. The explicit attr path <flake-root>#nixosConfigurations.<name>.config.system.build.toplevel is required; using the bare meta#<name> ref would make nix look in packages, legacyPackages, or the flake root directly — none of which exist in the rendered meta flake.

8.1 KiB Raw Blame History