`NodeId` has been globally unique across DAGs since #2801, so the dag id
carried no information the node id didn't. The parameter was already
underscore-prefixed as unused, but still populated by `claim_ready`, carried
through the scheduler's mpsc on every `Claim`, and passed at the call site —
three layers of plumbing feeding a dead argument.
Removing it surfaced four more dead things it had been keeping alive:
`settle_approval_tail`, `settle_rebuild_tail` and `drain_meta_syncs` each took
a dag id they only forwarded to `complete_node`, and one `submit` binding was
never read. Those are deleted rather than underscore-prefixed — prefixing is
what let the original argument survive this long.
`Claim.dag_id` stays: it has live consumers in the tracing spans,
`append_subgraph`'s container guard, `Ctx` for the build-log link,
`first_error`, and the approval-deploy context.
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
`Template` was a DAG-level enum that three different things read back
out: `terminal_hook()` mapped it to a side effect, the retention pass
bucketed history by it, and a tracing field printed it. None of those
needed a *label* — they needed the two facts the label happened to
encode. So the enum was a lossy stand-in for intent, and every new DAG
shape had to pick the variant whose inferred behaviour matched, whether
or not the name fit (`reparent` rode `MetaUpdate` for exactly this
reason, with a 10-line comment apologising for it).
Replace the inference with a declaration: `DagSpec.hook:
Option<HookKind>`. Only the builder assembling a DAG knows why it did
so, so only the builder can say what should happen when it settles.
`run_terminal_hook` becomes a field read, and `reparent`'s apology
becomes `hook: None`.
Hook assignment is byte-identical to the old precedence rule
(`approval_id.is_some()` wins, then `Rebuild | PermChange`), checked
site by site; `meta_update` is the only builder with a variable
approval id and so the only remaining conditional.
Retention loses the per-template bucket with the enum that keyed it.
The dashboard renders one recent-builds list, so one flat newest-first
cap (`MAX_HISTORY_DAGS`) bounds it. `HISTORY_GRACE_SECS` goes too — it
existed to stop a burst of same-template DAGs evicting each other
inside one poll interval, which is not a failure mode a flat cap has.
That takes `snapshot_capped()` and the `snapshot_no_grace()` test hook
with it.
The queue is runtime-only (empty graph on boot), so the serde changes
carry no migration risk.
The revert hook is dead by construction, so it can only ever be wrong.
DAG state `Cancelled` has exactly one producer: `JobQueue::cancel`, which
refuses unless every work node is still `Pending`. A cancel *cascade*
(some node failed, downstream cancelled) rolls up `Failed` instead —
`dag_rollup` short-circuits on any failed subtree node. So on a DAG that
reaches `Cancelled`, no node ever executed: the `SetWanted` head provably
never ran and `wanted` still reads whatever the operator last set it to.
There is therefore nothing to revert, and `revert_intent` did not revert
anything — it wrote `Wanted::from_running(observed)`, i.e. the agent's
*observed* state, over an intent the DAG never touched. Harmless when
observed already matched, silent corruption otherwise: cancel a queued
start for an agent that is down but `wanted = Up` (crashed, or caught
mid-bounce) and the intent flips to `Offline`, leaving it
deliberately-stopped as far as reconcile and crash-watch are concerned.
The hook made sense when `set_wanted` was a pre-submit side effect
written before the DAG ran; moving it into the DAG as a node left the
hook vestigial.
Drop `HookKind::RevertIntent`, `revert_intent`, and the power-op arm of
`terminal_hook` — start / stop / graceful-stop now settle with no
terminal hook, same as restart always did. The test asserts the general
statement across restart/stop/start x graceful x running: stop and start
carry a `SetWanted` head, and cancelling them still fires no hook.
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).
Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
sibling ordering directly (deps order execution, parent groups a subtree
whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
separate top-level root (AfterAny Prebuild) so it survives the cancel-
cascade of any failed step (recovery-start invariant) and converges to
the persisted `wanted` on a fresh lease. This is the multi-root
correction to the single-root-chain sketch: node0=root broke lease-
exemption (hoisting the lease onto Prebuild) and recovery-reconcile
(root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
same way; per-agent power-op subgraphs stay independent roots so a
multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
DAG terminal node deps AfterAny on every group root and runs once the
whole op rolls up. Drop the old Graph::add_dep terminal wiring.
36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
A per-agent lifecycle lease gates that agent's container globally across
concurrent DAGs, so it should be held for exactly as long as the agent's
work in a DAG is in flight, no longer. settle() previously freed every
lease a DAG held only at whole-DAG terminal, so a multi-agent DAG (a
hive-wide restart) kept agent A's container locked until B and C also
finished, blocking any other DAG wanting A.
Now free each agent's lease the moment its own subgraph within the DAG is
terminal (no live node still targets it), and drop that agent's dashboard
transient pill on the same edge via a new per-agent release channel. A
single-agent DAG is unaffected: its agent's subgraph goes terminal exactly
when the whole DAG does, so behaviour is identical.
append_subgraph is the multi-node/multi-agent generalisation of the
single-node append_node, so the two in-DAG-growth channels collapse to
one: the Reconcile planner now emits its mechanical Start/Stop as a
single-node append_subgraph rooted on the reconcile node (stamping
claim.agent on the NodeSpec, which append_node inherited implicitly).
Removes NodeOutput.append_nodes + its scheduler drain loop and
JobQueue::append_node. No behaviour change — a channel unification.
MetaLock's non-sweep completion now grows one rebuild subgraph per
affected agent into the same DAG (append_subgraph), replacing the
fan-out-child-DAGs + cancel_children dance. Drops NodeOutput.fanout and
scheduler's fanout_specs. meta_update DAG carries Rebuilding transient so
each cascade agent gets crash-watch suppression at Swap (the property the
old child Rebuild DAGs held via their own transient); MetaLock head needs
no lease so the pseudo-agent gets no pill.
append_children/parent_id and child-DAG tests are intentionally left for
the #2453 capstone.
The boot is now ONE DAG, assembled inline in submit_boot_tree — no boot_root
Noop anchor, no per-agent child DAGs, no display-only parent_id grouping: a
sweep MetaLock root (only when something is stale) that grows one rebuild
subgraph per stale agent into the same DAG (via append_subgraph, previous
commit), plus one Reconcile root per drifted agent (independent — a boot
reconcile needs no lock bump).
- submit_boot_tree builds the DagSpec inline; removed the single-use
templates::{boot_root, startup_sweep} builders (inlined per the operator's
"don't force single-use shapes into templates.rs" steer).
- fanout_specs simplified to the meta-update cascade path only — the startup
sweep no longer fans out child DAGs, so its branch was dead.
- test: append_subgraph_roots_on_emitter_and_rebases_local_deps.
Vestigial after this (deliberately left as follow-ups, flagged in the PR):
NodeKind::Noop is now unconstructed (contained to hive-c0re, removable);
Template::StartupSweep is unconstructed but a hive-sh4re wire type
(frontend-coordinated removal).
First half of making the startup sweep one DAG. Adds the runtime
subgraph-append machinery and switches the sweep MetaLock from fanning out
child Rebuild DAGs to growing one rebuild subgraph per stale agent into the
same DAG:
- JobQueue::append_subgraph(dag_id, nodes, dep_on) — the multi-node,
multi-agent generalisation of append_node: rebases a subgraph's local deps
onto the DAG's id space and roots it on the emitting node.
- NodeOutput.append_subgraph: Vec<Vec<NodeSpec>> — the executor→scheduler
channel for it; scheduler drains it before completing the emitting node
(same ordering as append_nodes).
- run_meta_lock sweep branch returns the stale agents' rebuild_nodes
subgraphs via append_subgraph instead of fanout.
Follow-up commit collapses submit_boot_tree (drop boot_root + per-agent
reconcile child DAGs) so the whole boot is one DAG built inline.
Agent was a single field on Dag/DagSpec, making a DAG structurally
one-agent — a multi-agent op could only ever be N separate DAGs. Move it
onto Node/NodeSpec (and the NodeView wire type), drop it from Dag/DagSpec
(and DagView): a DAG can now span agents.
- lifecycle lease keys on the node's agent, still globally exclusive per
agent across all DAGs (Inner.leases unchanged in shape). A DAG holds one
lease per distinct agent it touches; settle() frees each at DAG-terminal
(per-agent-subgraph early release is a follow-up, only observable with
multi-agent DAGs).
- transient guard keyed (dag_id, agent); cancel-revert + Rebuilt events
walk TerminalDag.agents.
- submit-time dedup removed (a multi-agent DAG has no single agent to key
on); every submit enqueues a fresh DAG. Whether dedup needs reintroducing
is tracked in a follow-up sub-issue.
- templates gain a node(agent, kind, deps) helper stamping the agent onto
every node; meta templates stamp "hyperhive".
Templates stay single-agent in this PR — behaviour is unchanged, only the
representation + wire shape. Multi-agent DAG emission (restart/restart-all/
broad stop+start as one DAG) and the SetWanted-as-a-node change are
follow-ups off #2439.
- deploy-window gate (meta::exclusive) + path-limited meta commits:
a perm/lock/topology commit can no longer sweep an ApprovalDeploy's
staged flake.lock and neuter abort_deploy (regression test included)
- cancel surfaces now buffer terminal roll-ups the scheduler drains,
so a queued approval DAG cancelled by the operator resolves its
approval instead of dangling, and cancelled power ops revert their
wanted flip to the observed state
- hivectl restart / restart-all ride the queue (lease serialization,
transient guard) and restart sets wanted=Up like the old kill+start
- exactly one Rebuilt event per rebuild DAG, emitted at terminal
- StopForUpdate pre-seeds a missing agent_power row from the pre-stop
observation so a rebuild can't strand an unknown agent offline
- history trim keeps terminal fan-out parents with live children
- audit_log back on db::open; swarm.js badge for reconcile DAGs
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
