|
|
|
|
@ -6,102 +6,164 @@ and `docs/persistence.md`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Rebuild queue
|
|
|
|
|
## Job queue
|
|
|
|
|
|
|
|
|
|
Every long-running container/meta operation (rebuild, meta-update, first-spawn)
|
|
|
|
|
goes through the global rebuild queue (`hive-c0re/src/rebuild_queue.rs`). A single
|
|
|
|
|
background worker drains it in FIFO order so two `nixos-container update` runs on
|
|
|
|
|
the same agent never overlap, and a fresh agent rebuild never races a meta-update's
|
|
|
|
|
lock bump.
|
|
|
|
|
Every container/meta operation (rebuild, meta-update, first-spawn, power
|
|
|
|
|
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
|
|
|
|
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
|
|
|
|
concurrency comes from the resource classes below, not from multiple workers.
|
|
|
|
|
The old special cases — the graceful-stop watcher thread, the deferred-start
|
|
|
|
|
fast-lane follow-up, the meta-update cascade pre-enqueue — are all just DAG
|
|
|
|
|
*shapes* now.
|
|
|
|
|
|
|
|
|
|
### Why one queue
|
|
|
|
|
### Two levels: DAG and node
|
|
|
|
|
|
|
|
|
|
Before the rebuild queue landed, four independent call paths could fire
|
|
|
|
|
`auto_update::rebuild_agent` concurrently:
|
|
|
|
|
The **DAG** is the unit of dedup / cancel / approval-resolution and the
|
|
|
|
|
dashboard group; the **node** is the unit of scheduling / execution /
|
|
|
|
|
build-log / step label. Deps are intra-DAG edges only (`AfterOk` by default:
|
|
|
|
|
the dep must succeed, a failed/cancelled dep cancels the dependent —
|
|
|
|
|
cancel-downstream). Cross-DAG ordering comes from the per-agent lease + dedup,
|
|
|
|
|
never from edges between DAGs. Submit-time validation (petgraph `toposort`)
|
|
|
|
|
rejects cyclic specs outright, fixing the old queue's "circular dep silently
|
|
|
|
|
deadlocks" caveat.
|
|
|
|
|
|
|
|
|
|
- Dashboard manual rebuild button
|
|
|
|
|
- `update-all` / `meta-update` cascade
|
|
|
|
|
- Approval handler (apply-commit / spawn)
|
|
|
|
|
- Startup auto-update sweep
|
|
|
|
|
### Node inventory (primitives)
|
|
|
|
|
|
|
|
|
|
Nothing serialised them. `nix-daemon` serialises the actual store ops, but the rest
|
|
|
|
|
of `rebuild_agent` (token sync, kick, rescan, lock-bump emit) interleaved
|
|
|
|
|
unpredictably. The single-worker queue gives operators a visible, ordered runway and
|
|
|
|
|
lets the UI render "what's about to happen" instead of "something might be happening
|
|
|
|
|
somewhere."
|
|
|
|
|
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
|
|
|
|
|
|
|
|
|
### Queue kinds
|
|
|
|
|
| Node | Wraps |
|
|
|
|
|
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
|
|
| `Prebuild` | meta `sync_agents` + optional per-agent relock + `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving |
|
|
|
|
|
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped) + the post-rebuild bookkeeping tail (rev marker, `Rebuilt` event, forge/matrix sync, kick, rescan) |
|
|
|
|
|
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
|
|
|
|
|
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
|
|
|
|
| `ApprovalDeploy` | the opaque apply-commit / merge-config-PR pipeline (see _Approvals_ below) |
|
|
|
|
|
|
|
|
|
|
| Kind | Description |
|
|
|
|
|
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
|
|
|
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see _Deferred start_ under the rebuild path below. |
|
|
|
|
|
| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. |
|
|
|
|
|
| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. |
|
|
|
|
|
| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. |
|
|
|
|
|
| `Restart` | Stop + start a container without touching config (~5-10s). Routed through the queue so it serialises against in-flight rebuilds for the same agent — prevents a restart racing a rebuild mid-flight. Sources: dashboard ↺ button, the `restart` MCP tool. |
|
|
|
|
|
| `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. |
|
|
|
|
|
| `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/<agent>`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. |
|
|
|
|
|
Cheap — no build slot:
|
|
|
|
|
|
|
|
|
|
**Intentionally not queued** (sub-second ops): the _hard_ `start`, `stop`, `kill` via the direct API paths. (A _graceful_ stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard.
|
|
|
|
|
| Node | Behavior |
|
|
|
|
|
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
|
|
|
|
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
|
|
|
|
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
|
|
|
|
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
|
|
|
|
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
|
|
|
|
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
|
|
|
|
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
|
|
|
|
|
|
|
|
|
### Dedup
|
|
|
|
|
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
|
|
|
|
|
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
|
|
|
|
node would open a dirty-working-tree window between nodes. That same
|
|
|
|
|
`META_LOCK` is also why the scheduler needs no meta-repo resource class — any
|
|
|
|
|
executor touching the meta repo serializes inside `meta.rs`.
|
|
|
|
|
|
|
|
|
|
Enqueueing `(kind, agent)` that already has a `Queued` entry returns the existing
|
|
|
|
|
entry's id and appends the new request as an "also requested by …" line. Running
|
|
|
|
|
entries do not dedup — a re-queue during a run is legitimate (something changed
|
|
|
|
|
since the current run started).
|
|
|
|
|
### Every operation as a DAG
|
|
|
|
|
|
|
|
|
|
### Sources
|
|
|
|
|
```text
|
|
|
|
|
rebuild(a): Prebuild(a) → StopForUpdate(a) → Swap(a) →(after-any) Reconcile(a)
|
|
|
|
|
graceful-stop(a): [wanted=Offline] Signal(a) → Drain(a) → Reconcile(a)
|
|
|
|
|
restart(a): StopForUpdate(a) → Reconcile(a) (wanted unchanged)
|
|
|
|
|
start(a): [wanted=Up] Reconcile(a) (stale rev ⇒ upgraded to rebuild)
|
|
|
|
|
stop(a): [wanted=Offline] Reconcile(a)
|
|
|
|
|
spawn(a): [wanted=Up] Create(a) → WriteDropin(a) → Reconcile(a)
|
|
|
|
|
perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
|
|
|
|
meta-update(inp): MetaLock(inp) → «fan-out rebuild(a) per affected agent»
|
|
|
|
|
boot: (if any rev marker stale) MetaLock(hyperhive) → «fan-out rebuild»;
|
|
|
|
|
plus Reconcile(a) for every drifted agent
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
| Source | Meaning |
|
|
|
|
|
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
|
|
| `Manual` | Operator clicked rebuild / update-all / meta-update on the dashboard, or any other direct human action (CLI, an agent MCP tool). |
|
|
|
|
|
| `AutoUpdate` | Legacy startup-sweep source (flat, no parent). Replaced by `StartupSweep` for new boots. |
|
|
|
|
|
| `StartupSweep` | Child of a `StartupSweep` parent entry; boot-time per-agent rebuild with the sweep as the visual group header. |
|
|
|
|
|
| `Approval` | Triggered by an operator-approved `ApprovalKind::{Spawn, ApplyCommit}`. |
|
|
|
|
|
Notable collapses:
|
|
|
|
|
|
|
|
|
|
### Cascade parent tracking
|
|
|
|
|
- **`rebuild` is one uniform shape** — no `was_running` branch.
|
|
|
|
|
`StopForUpdate` noops when already down; the tail `Reconcile` auto-noops the
|
|
|
|
|
start when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
|
|
|
|
leaves it stopped).
|
|
|
|
|
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
|
|
|
|
|
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
|
|
|
|
|
terminal ok *or* fail, bringing a wanted-up agent back on its old config.
|
|
|
|
|
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
|
|
|
|
|
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
|
|
|
|
|
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
|
|
|
|
whole-hive graceful stop fires every agent's signal immediately and all
|
|
|
|
|
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
|
|
|
|
- **The meta-update cascade fans out on completion**: `MetaLock`'s executor
|
|
|
|
|
computes the affected agent set after the bump lands and appends child
|
|
|
|
|
`rebuild` DAGs (`parent_id` set, `relock = false` so the children don't
|
|
|
|
|
revert the bump). A failed bump fans out nothing — no cancel-children dance.
|
|
|
|
|
|
|
|
|
|
`MetaUpdate` and `StartupSweep` entries fan out `Rebuild` children, each carrying
|
|
|
|
|
`parent_id = <parent_id>`. The dashboard groups children under their parent
|
|
|
|
|
in the queue panel so the operator sees the whole cascade as a tree,
|
|
|
|
|
not a flat list.
|
|
|
|
|
### Desired-state (spec vs status)
|
|
|
|
|
|
|
|
|
|
### Step labels
|
|
|
|
|
Per-agent power *intent* — `wanted: Up | Offline` — is durable in
|
|
|
|
|
`/var/lib/hyperhive/db/agent_power.sqlite` (`hive-c0re/src/power.rs`).
|
|
|
|
|
`container_view` remains the observed *status*; `Reconcile` nodes converge the
|
|
|
|
|
two. Setting `wanted` is never a queued node: the submit layer
|
|
|
|
|
(`job_queue/submit.rs`) writes the row synchronously, then submits the DAG
|
|
|
|
|
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
|
|
|
|
Power toggles never commit to the meta repo. Direct (non-queued) power paths —
|
|
|
|
|
`hivectl stop/start`, the admin-socket kill, the MCP kill tool — write
|
|
|
|
|
`wanted` too, so reconciles never undo an operator's stop. Agents without a
|
|
|
|
|
row are seeded from observed state on first touch (running ⇒ `Up`); destroy
|
|
|
|
|
removes the row.
|
|
|
|
|
|
|
|
|
|
Each queue entry has a mutable `step: Option<String>` field that the worker updates
|
|
|
|
|
as it progresses through lifecycle phases (`"nix build"`, `"nixos-container stop"`,
|
|
|
|
|
`"nixos-container update"`, `"nixos-container start"`). The dashboard polls
|
|
|
|
|
`/api/state` and renders the current step beneath the running entry so the operator
|
|
|
|
|
can see which phase is taking time.
|
|
|
|
|
### Scheduler semantics
|
|
|
|
|
|
|
|
|
|
### Dependency tracking
|
|
|
|
|
A node is **ready** when it's `Queued`, every dep is satisfied, and its
|
|
|
|
|
resources are free. Resources:
|
|
|
|
|
|
|
|
|
|
Each entry carries a `depends_on: Vec<u64>` field. The worker skips entries whose
|
|
|
|
|
dependencies are not yet resolved — a dependency is resolved when its id is either
|
|
|
|
|
in the queue as a terminal entry (`Done` / `Failed` / `Cancelled`) or no longer in
|
|
|
|
|
the queue at all (evicted by `trim_history`, which only evicts terminals).
|
|
|
|
|
1. **Build slots** — `services.hyperhive.c0re.buildSlots` permits (default 1),
|
|
|
|
|
held by nix-heavy nodes for the node's duration.
|
|
|
|
|
2. **Per-agent lifecycle lease** — DAG-scoped: acquired at the DAG's first
|
|
|
|
|
container-affecting node (`StopForUpdate`, `Swap`, `Signal`, `Drain`,
|
|
|
|
|
`Reconcile`, `WriteDropin`, `Create`, `ApprovalDeploy`), held until the DAG
|
|
|
|
|
is terminal, so two lifecycle DAGs for one agent never interleave their
|
|
|
|
|
container ops. **Lease-exempt**: `Prebuild`, `MetaLock`, `WritePermFile` —
|
|
|
|
|
they touch the store / meta, not the running container, which is exactly
|
|
|
|
|
why a stop can land while another DAG's prebuild is still building.
|
|
|
|
|
|
|
|
|
|
Use cases:
|
|
|
|
|
Among simultaneously-ready nodes competing for a resource, DAG-submit order
|
|
|
|
|
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
|
|
|
|
|
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
|
|
|
|
|
created on lease acquisition and dropped when the DAG settles terminal.
|
|
|
|
|
|
|
|
|
|
- Chain a `Rebuild` after an explicit prerequisite step without coupling them through
|
|
|
|
|
the `parent_id` cascade mechanism.
|
|
|
|
|
- Sequence a `PermChange` + `Rebuild` pair where the rebuild must not start until the
|
|
|
|
|
perm-file write commits (already handled by the single-worker FIFO today, but
|
|
|
|
|
`depends_on` allows explicit cross-kind sequencing when parallel workers are added).
|
|
|
|
|
The queue is in-memory only and lost on hive-c0re restart — deliberate:
|
|
|
|
|
desired state is re-derived at boot from the DB + rev markers (see _Boot
|
|
|
|
|
reconcile_), so there is no durable-recovery machinery to go wrong.
|
|
|
|
|
|
|
|
|
|
`depends_on` is part of the dedup key: two entries with the same `(kind, agent,
|
|
|
|
|
parent_id, inputs, approval_id)` but different dep sets are treated as distinct work.
|
|
|
|
|
### Dedup, cancel, history
|
|
|
|
|
|
|
|
|
|
**Worker re-notification**: the worker drains `take_next()` in a tight loop after
|
|
|
|
|
each entry finishes. When a dep entry transitions to terminal, the loop re-evaluates
|
|
|
|
|
the queue immediately, so downstream entries are unblocked with no extra wakeup. No
|
|
|
|
|
additional `notify_one()` call is needed.
|
|
|
|
|
Dedup at **DAG granularity**: a repeat submit against a DAG whose roll-up is
|
|
|
|
|
still `Queued` with the same `(template, agent, parent_id, approval_id)` —
|
|
|
|
|
plus `inputs` for meta-updates and the perm-type discriminant for perm
|
|
|
|
|
changes — returns the existing id and appends an "also requested by …" line.
|
|
|
|
|
`parent_id` in the key keeps a cascade child from collapsing into a
|
|
|
|
|
standalone or sweep rebuild. Running/terminal DAGs never dedup.
|
|
|
|
|
|
|
|
|
|
**Circular-dep caveat**: if A depends on B and B depends on A, neither entry ever
|
|
|
|
|
becomes runnable — the worker skips both indefinitely with no error. Callers must
|
|
|
|
|
ensure acyclic dep graphs. Cycle detection is deferred to a future iteration (when
|
|
|
|
|
parallel workers make a stuck queue more visible).
|
|
|
|
|
Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't
|
|
|
|
|
interruptible); `cancel_children` cancels a parent's still-queued child DAGs.
|
|
|
|
|
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
|
|
|
|
|
`Cancelled` / `Done`. The snapshot retains the 5 most recent terminal DAGs
|
|
|
|
|
per template.
|
|
|
|
|
|
|
|
|
|
### Approvals
|
|
|
|
|
|
|
|
|
|
`ApplyCommit` / `MergeConfigPr` approvals ride as single-node
|
|
|
|
|
`ApprovalDeploy` DAGs: the two-phase `prepare_deploy` / `finalize_deploy` /
|
|
|
|
|
`abort_deploy` meta orchestration stays inside `actions.rs` in v1
|
|
|
|
|
(deliberately not modeled as scheduler nodes) and resolves the approval
|
|
|
|
|
itself. `Spawn` and `UpdateMetaInputs` approvals map onto the ordinary
|
|
|
|
|
`spawn` / `meta-update` shapes; the scheduler fires
|
|
|
|
|
`actions::resolve_approval_dag` exactly once when such a DAG settles
|
|
|
|
|
terminal (including cancelled-while-queued, which fails the approval instead
|
|
|
|
|
of dangling it).
|
|
|
|
|
|
|
|
|
|
### Wire shape
|
|
|
|
|
|
|
|
|
|
`RebuildQueueChanged { seq, queue: [DagView…] }` (event name kept). Each
|
|
|
|
|
`DagView` carries the old entry-level fields (`id`, `kind` = template string,
|
|
|
|
|
roll-up `state`, `agent`, `source`, `parent_id`, `reason`, timestamps,
|
|
|
|
|
`inputs`, `approval_id`) plus `nodes: [NodeView…]` — per-node `kind`, `deps`,
|
|
|
|
|
`state`, `step`, `build_log_id`, timestamps, `error`. Step labels and build
|
|
|
|
|
logs are **per-node**; the dashboard renders the node chain on each queue
|
|
|
|
|
card and keys the live-log panel off the running node.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
@ -115,31 +177,31 @@ render.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Auto-update sweep
|
|
|
|
|
## Boot reconcile
|
|
|
|
|
|
|
|
|
|
On startup, `auto_update.rs` rebuilds containers that actually need it. Two skip
|
|
|
|
|
rules keep boot-time work minimal:
|
|
|
|
|
On startup, `auto_update::run` classifies every agent by rev freshness (the
|
|
|
|
|
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
|
|
|
|
vs the current flake path) and persisted `wanted` intent, then:
|
|
|
|
|
|
|
|
|
|
1. **Stopped containers** are deferred: the startup sweep enqueues nothing for them.
|
|
|
|
|
When the operator later starts a stopped container (via the dashboard or the
|
|
|
|
|
`start` MCP tool), both `run_start` (queue path) and `handle_start` (socket path)
|
|
|
|
|
check the rev marker first — if it's stale, the start is silently upgraded to a
|
|
|
|
|
full rebuild+start so the container runs current nix derivations.
|
|
|
|
|
1. **Config path** — when *any* marker is stale, submit one `StartupSweep`
|
|
|
|
|
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that fans out `Rebuild`
|
|
|
|
|
children for the stale agents whose `wanted = Up` (topology-sorted, parents
|
|
|
|
|
first). Stale but wanted-offline agents get no boot-time nix work — their
|
|
|
|
|
rebuild happens on their next start (the start submit path upgrades a
|
|
|
|
|
stale start to rebuild+start), which is also why the lock bump runs even
|
|
|
|
|
when every stale agent is offline: those later start-upgrades must build
|
|
|
|
|
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
|
|
|
|
|
agent (back) up, covering both the running-stale and stopped-but-wanted-up
|
|
|
|
|
cases.
|
|
|
|
|
|
|
|
|
|
2. **Running containers with a matching rev marker** are skipped: if the per-agent
|
|
|
|
|
`.{name}.hyperhive-rev` file under `/var/lib/hyperhive/applied/` already holds
|
|
|
|
|
the current flake rev, no nix work is needed and the entry is omitted entirely.
|
|
|
|
|
2. **Power path** — every agent whose observed state drifted from `wanted`
|
|
|
|
|
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
|
|
|
|
|
|
|
|
|
`auto_update::run` enqueues a single `StartupSweep` parent entry (`kind =
|
|
|
|
|
startup_sweep`, `agent = "hyperhive"`) followed by per-agent `Rebuild` children
|
|
|
|
|
for the agents that do need rebuilding (`source = startup_sweep`, `parent_id =
|
|
|
|
|
sweep_id`). The sweep description records the rebuild / deferred / skipped counts
|
|
|
|
|
so the operator can see at a glance how much work the boot triggered. The child
|
|
|
|
|
rebuilds drain sequentially through the queue; the dashboard renders them nested
|
|
|
|
|
under the parent.
|
|
|
|
|
|
|
|
|
|
Before the sweep-grouping change, each boot enqueued flat `Rebuild` entries with
|
|
|
|
|
`source = AutoUpdate` and no parent — visible but ungrouped.
|
|
|
|
|
Booting with no config change performs **no meta commit** — only reconciles.
|
|
|
|
|
The sweep reason records the rebuild / deferred / up-to-date counts so the
|
|
|
|
|
operator sees at a glance how much work the boot triggered. Agents without an
|
|
|
|
|
`agent_power` row are seeded from observed state during classification (the
|
|
|
|
|
one-time migration; thereafter the DB is authoritative).
|
|
|
|
|
|
|
|
|
|
## Meta flake
|
|
|
|
|
|
|
|
|
|
@ -157,8 +219,13 @@ Key operations:
|
|
|
|
|
`RequestApplyCommit` path so a failed `nixos-container update` leaves no orphan
|
|
|
|
|
commit in meta. Prepare writes the new lock without committing; finalize commits
|
|
|
|
|
with the deploy message; abort restores the lock.
|
|
|
|
|
- **`lock_update_hyperhive`** — one-shot for the auto-update path: bumps the
|
|
|
|
|
`hyperhive` input lock, commits, cascades agent rebuilds.
|
|
|
|
|
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
|
|
|
|
|
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
|
|
|
|
|
the scheduler fans out the agent rebuilds on completion.
|
|
|
|
|
|
|
|
|
|
Every public `meta.rs` operation takes the module's internal `META_LOCK`
|
|
|
|
|
mutex, so concurrent job-queue nodes (and the approval deploy pipeline) never
|
|
|
|
|
race on the repo's `.git/index.lock`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
@ -179,34 +246,26 @@ new profile and skips the in-container `switch-to-configuration`. The subsequent
|
|
|
|
|
`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
|
|
|
|
|
one go, rather than the double-bounce a live `update` would trigger.
|
|
|
|
|
|
|
|
|
|
Sequence for a running container:
|
|
|
|
|
Sequence for a rebuild DAG (each step is its own queue node):
|
|
|
|
|
|
|
|
|
|
1. `prebuild_toplevel` — build the new `system.build.toplevel` **before** stopping.
|
|
|
|
|
The container keeps serving the previous generation while eval + fetch + build
|
|
|
|
|
happen out-of-band. `nixos-container update` then finds the result cached and
|
|
|
|
|
skips straight to the profile-swap. Build failures surface here, before the
|
|
|
|
|
running container is touched.
|
|
|
|
|
2. `nixos-container stop` — bring the container down.
|
|
|
|
|
3. `nixos-container update --flake meta#<name>` — profile-swap (near-instant after
|
|
|
|
|
the prebuild).
|
|
|
|
|
4. `nixos-container start` — boot into the new generation; the in-container
|
|
|
|
|
activation script transitions old → new.
|
|
|
|
|
1. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
|
|
|
|
The container keeps serving the previous generation while eval + fetch +
|
|
|
|
|
build happen out-of-band. `nixos-container update` then finds the result
|
|
|
|
|
cached and skips straight to the profile-swap. Build failures surface
|
|
|
|
|
here, before the running container is touched. (Runs even for a stopped
|
|
|
|
|
container — same total nix work, one uniform DAG shape.)
|
|
|
|
|
2. `StopForUpdate` — bring the container down (noop when already stopped).
|
|
|
|
|
3. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
|
|
|
|
|
(near-instant after the prebuild).
|
|
|
|
|
4. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
|
|
|
|
in-container activation script transitions old → new. Holds no build
|
|
|
|
|
slot, so the next DAG's `Prebuild` overlaps the container boot — the old
|
|
|
|
|
"deferred start" split, now structural.
|
|
|
|
|
|
|
|
|
|
If the container is already stopped, step 1 is skipped (no downtime to shave — no
|
|
|
|
|
point evaluating the flake twice).
|
|
|
|
|
|
|
|
|
|
**Deferred start (queue-dispatched rebuilds):** step 4 can take a while
|
|
|
|
|
(container boot), and holding the serialized build lane through it delays the
|
|
|
|
|
next queued rebuild's nix build for no reason. Queue-dispatched rebuilds
|
|
|
|
|
therefore pass `defer_start` — `rebuild_no_meta` skips the start and returns
|
|
|
|
|
`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead
|
|
|
|
|
(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under
|
|
|
|
|
it — the same split the graceful-stop path uses for its container stop). The
|
|
|
|
|
build-lane entry completes when the profile-swap finishes; a start failure
|
|
|
|
|
surfaces on the `Start` entry, which runs with the cold-start fallback. Direct
|
|
|
|
|
callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy
|
|
|
|
|
flow which verifies the agent comes back up before finalizing) keep the start
|
|
|
|
|
inline.
|
|
|
|
|
The approval apply-commit pipeline still drives `lifecycle::rebuild_no_meta`
|
|
|
|
|
(the fused stop/update/start path with an inline start) inside its
|
|
|
|
|
`ApprovalDeploy` node, because it verifies the agent comes back up before
|
|
|
|
|
finalizing the deploy tag.
|
|
|
|
|
|
|
|
|
|
### Cold-start fallback
|
|
|
|
|
|
|
|
|
|
@ -218,9 +277,8 @@ half-started at that point.
|
|
|
|
|
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
|
|
|
|
|
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
|
|
|
|
|
Both errors are preserved and surfaced if the cold-start also fails. The fallback
|
|
|
|
|
lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild
|
|
|
|
|
path and the queue's fast-lane `Start` handler (which the deferred
|
|
|
|
|
start-after-rebuild rides).
|
|
|
|
|
lives in `lifecycle::start_with_fallback`, shared by the apply-commit deploy's
|
|
|
|
|
inline start and every `Reconcile` node's start action.
|
|
|
|
|
|
|
|
|
|
### Spawn path (new container)
|
|
|
|
|
|
|
|
|
|
@ -241,9 +299,18 @@ or the flake root directly — none of which exist in the rendered meta flake.
|
|
|
|
|
|
|
|
|
|
## Host-level resource + performance options
|
|
|
|
|
|
|
|
|
|
Three `services.hyperhive.c0re.*` options tune container resource
|
|
|
|
|
limits and first-spawn latency. All three apply uniformly to every
|
|
|
|
|
agent container.
|
|
|
|
|
A handful of `services.hyperhive.c0re.*` options tune container resource
|
|
|
|
|
limits, build parallelism, and first-spawn latency.
|
|
|
|
|
|
|
|
|
|
### Build slots
|
|
|
|
|
|
|
|
|
|
`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
|
|
|
|
|
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
|
|
|
|
|
concurrently. The default serializes all heavy nix work like the pre-DAG
|
|
|
|
|
rebuild queue did; raise it on hosts with the cores/RAM to build several
|
|
|
|
|
agent toplevels at once. Per-agent correctness is independent of the count —
|
|
|
|
|
each agent's container-affecting ops serialize on its lifecycle lease
|
|
|
|
|
regardless.
|
|
|
|
|
|
|
|
|
|
### Container resource limits
|
|
|
|
|
|
|
|
|
|
|
müde