Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).
Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
sibling ordering directly (deps order execution, parent groups a subtree
whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
separate top-level root (AfterAny Prebuild) so it survives the cancel-
cascade of any failed step (recovery-start invariant) and converges to
the persisted `wanted` on a fresh lease. This is the multi-root
correction to the single-root-chain sketch: node0=root broke lease-
exemption (hoisting the lease onto Prebuild) and recovery-reconcile
(root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
same way; per-agent power-op subgraphs stay independent roots so a
multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
DAG terminal node deps AfterAny on every group root and runs once the
whole op rolls up. Drop the old Graph::add_dep terminal wiring.
36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
A per-agent lifecycle lease gates that agent's container globally across
concurrent DAGs, so it should be held for exactly as long as the agent's
work in a DAG is in flight, no longer. settle() previously freed every
lease a DAG held only at whole-DAG terminal, so a multi-agent DAG (a
hive-wide restart) kept agent A's container locked until B and C also
finished, blocking any other DAG wanting A.
Now free each agent's lease the moment its own subgraph within the DAG is
terminal (no live node still targets it), and drop that agent's dashboard
transient pill on the same edge via a new per-agent release channel. A
single-agent DAG is unaffected: its agent's subgraph goes terminal exactly
when the whole DAG does, so behaviour is identical.
append_subgraph is the multi-node/multi-agent generalisation of the
single-node append_node, so the two in-DAG-growth channels collapse to
one: the Reconcile planner now emits its mechanical Start/Stop as a
single-node append_subgraph rooted on the reconcile node (stamping
claim.agent on the NodeSpec, which append_node inherited implicitly).
Removes NodeOutput.append_nodes + its scheduler drain loop and
JobQueue::append_node. No behaviour change — a channel unification.
MetaLock's non-sweep completion now grows one rebuild subgraph per
affected agent into the same DAG (append_subgraph), replacing the
fan-out-child-DAGs + cancel_children dance. Drops NodeOutput.fanout and
scheduler's fanout_specs. meta_update DAG carries Rebuilding transient so
each cascade agent gets crash-watch suppression at Swap (the property the
old child Rebuild DAGs held via their own transient); MetaLock head needs
no lease so the pseudo-agent gets no pill.
append_children/parent_id and child-DAG tests are intentionally left for
the #2453 capstone.
The boot is now ONE DAG, assembled inline in submit_boot_tree — no boot_root
Noop anchor, no per-agent child DAGs, no display-only parent_id grouping: a
sweep MetaLock root (only when something is stale) that grows one rebuild
subgraph per stale agent into the same DAG (via append_subgraph, previous
commit), plus one Reconcile root per drifted agent (independent — a boot
reconcile needs no lock bump).
- submit_boot_tree builds the DagSpec inline; removed the single-use
templates::{boot_root, startup_sweep} builders (inlined per the operator's
"don't force single-use shapes into templates.rs" steer).
- fanout_specs simplified to the meta-update cascade path only — the startup
sweep no longer fans out child DAGs, so its branch was dead.
- test: append_subgraph_roots_on_emitter_and_rebases_local_deps.
Vestigial after this (deliberately left as follow-ups, flagged in the PR):
NodeKind::Noop is now unconstructed (contained to hive-c0re, removable);
Template::StartupSweep is unconstructed but a hive-sh4re wire type
(frontend-coordinated removal).
First half of making the startup sweep one DAG. Adds the runtime
subgraph-append machinery and switches the sweep MetaLock from fanning out
child Rebuild DAGs to growing one rebuild subgraph per stale agent into the
same DAG:
- JobQueue::append_subgraph(dag_id, nodes, dep_on) — the multi-node,
multi-agent generalisation of append_node: rebases a subgraph's local deps
onto the DAG's id space and roots it on the emitting node.
- NodeOutput.append_subgraph: Vec<Vec<NodeSpec>> — the executor→scheduler
channel for it; scheduler drains it before completing the emitting node
(same ordering as append_nodes).
- run_meta_lock sweep branch returns the stale agents' rebuild_nodes
subgraphs via append_subgraph instead of fanout.
Follow-up commit collapses submit_boot_tree (drop boot_root + per-agent
reconcile child DAGs) so the whole boot is one DAG built inline.
Agent was a single field on Dag/DagSpec, making a DAG structurally
one-agent — a multi-agent op could only ever be N separate DAGs. Move it
onto Node/NodeSpec (and the NodeView wire type), drop it from Dag/DagSpec
(and DagView): a DAG can now span agents.
- lifecycle lease keys on the node's agent, still globally exclusive per
agent across all DAGs (Inner.leases unchanged in shape). A DAG holds one
lease per distinct agent it touches; settle() frees each at DAG-terminal
(per-agent-subgraph early release is a follow-up, only observable with
multi-agent DAGs).
- transient guard keyed (dag_id, agent); cancel-revert + Rebuilt events
walk TerminalDag.agents.
- submit-time dedup removed (a multi-agent DAG has no single agent to key
on); every submit enqueues a fresh DAG. Whether dedup needs reintroducing
is tracked in a follow-up sub-issue.
- templates gain a node(agent, kind, deps) helper stamping the agent onto
every node; meta templates stamp "hyperhive".
Templates stay single-agent in this PR — behaviour is unchanged, only the
representation + wire shape. Multi-agent DAG emission (restart/restart-all/
broad stop+start as one DAG) and the SetWanted-as-a-node change are
follow-ups off #2439.
- deploy-window gate (meta::exclusive) + path-limited meta commits:
a perm/lock/topology commit can no longer sweep an ApprovalDeploy's
staged flake.lock and neuter abort_deploy (regression test included)
- cancel surfaces now buffer terminal roll-ups the scheduler drains,
so a queued approval DAG cancelled by the operator resolves its
approval instead of dangling, and cancelled power ops revert their
wanted flip to the observed state
- hivectl restart / restart-all ride the queue (lease serialization,
transient guard) and restart sets wanted=Up like the old kill+start
- exactly one Rebuilt event per rebuild DAG, emitted at terminal
- StopForUpdate pre-seeds a missing agent_power row from the pre-stop
observation so a rebuild can't strand an unknown agent offline
- history trim keeps terminal fan-out parents with live children
- audit_log back on db::open; swarm.js badge for reconcile DAGs
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
