Every template built a `Vec<NodeSpec>` whose edges and parents were
positional indices into that vector, so a shape was expressed as
arithmetic: `base + 1`, `stop_root + 2`, `sfu + 1`, and a
`reconcile_index()` helper that read the emitted vector's length to find
out where its own last node had landed. `concat_subgraphs` existed
solely to rebase one per-agent subgraph's indices onto another's.
Templates now declare into a `hive_jobq::JobBuilder` and hold the
handles they get back, so an edge names the node it waits on. The
arithmetic is gone, and with it:
- `NodeSpec` and the job-queue's own index-based `Dep`.
- `insert_group`'s index resolution — it wraps `Scheduler::insert_job`.
- `concat_subgraphs` — per-agent chains share one builder and each keeps
its own root, so independence is structural rather than computed.
- `reconcile_index` and `dep_index`.
- `templates::validate` and its petgraph toposort. It rejected dangling
deps and cycles; both are now unrepresentable, since a handle only
exists for an already-declared node and every edge therefore points
backwards. (petgraph stays in the tree for `agent_config::topology`.)
`NodeOutput.append_subgraph` becomes `Vec<Job>`: an executor cannot
reach the queue, so it hands back declarations and the scheduler inserts
them under its own lock. That is what the in-DAG growth path always
wanted — a transferable declaration, not a vector of specs.
Resource declaration is unchanged in behaviour: the `templates::node`
helper applies `NodeKind::resource_deps()` at the construction site, so
every node still declares what its kind needs. Moving that declaration
to the call sites is #2818's job; this leaves it one place to delete.
Three tests went with the guard they covered — they hand-built malformed
specs out of indices, which is the representation that made those shapes
possible. Two more now read a DAG's shape off the queue rather than out
of a spec vector, which is where it is observable. The remaining 45
job-queue tests are unchanged and still pass: lease serialization,
roll-up, cancel-cascade, in-DAG growth and per-agent concurrency all
behave as before.
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.
It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.
`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.
`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.
## the safety half, which is deliberately not the display half
`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.
`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.
For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.
## known cost, accepted on the issue
A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.
`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
Second of the `Dag` field removals, and the same shape as the first:
`DagSpec`/`NodeKind::Dag` carried an `inputs: Vec<String>` that exactly
one node ever read. Both reads live inside `run_meta_lock` — the
`meta::lock_update` call and the `meta_update_cascade_agents` fan-out —
so the list now rides `NodeKind::MetaLock` itself.
The executor stops touching `Claim` for this node entirely: its dispatch
arm already destructured `MetaLock { sweep, fanout }`, so `inputs` joins
them and the `claim` parameter, which had no other use, is gone.
Falls out of that:
- `Claim::inputs` and `DagMeta::inputs` delete.
- `dag_view`'s DAG-level projection onto the `MetaLock` node reads the
payload instead. The wire `NodeView::inputs` is unchanged: still
populated on the `meta_lock` node alone.
- the boot sweep names no inputs (it bumps `hyperhive` alone via
`lock_update_hyperhive`), which the construction site now says out loud
rather than leaving implicit in an empty DAG-level field.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (320 passed) and `nix fmt`. No option surface is touched, so
no nix-eval gate.
`DagSpec`/`NodeKind::Dag` carried an `Option<i64>` approval id that four
deploy phases read back out through `Claim`, via a fallible helper whose
error ("approval deploy dag N has no approval_id") described a state the
type system should have forbidden. Two other templates (`spawn`,
`meta_update`) set the field for nothing: their approval is resolved by
the `ResolveApproval` tails, which already carry the id themselves.
So the id moves onto the nodes that actually need it —
`DeployWindow` / `MergeVerify` / `DeployApply` / `FinalizeDeploy` /
`DeployTail` each take an `i64`, the same way `ResolveApproval` always
has. `templates::approval_deploy` builds all of them in one place with
the value in hand, and `deploy_rebuild_nodes` takes it as a parameter so
the `FinalizeDeploy` it appends at runtime is constructed the same way.
Falls out of that:
- `deploy_approval_id` and its runtime error path delete; each executor
takes the id from its own node payload at dispatch.
- `run_deploy_window` had nothing left to do but validate that id, so the
node joins `Dag` on the shared no-op arm.
- `Claim::approval_id` and `DagMeta::approval_id` delete.
- `dag_view`'s DAG-level projection onto `DeployWindow` reads the payload
instead. The wire `NodeView::approval_id` is unchanged: still set on
the deploy root alone, so the dashboard still renders one approval link
per DAG rather than one per phase.
No option surface is touched, so there is no nix-eval gate here; checked
with clippy (`--all-targets -D warnings`), `cargo test -p hive-c0re`
(320 passed) and `nix fmt`.
A host restart brings hive-c0re up and the startup sweep rebuilds every
stale agent. Until now that stop was mechanical: `StopForUpdate` hung
straight off `Prebuild`, so an agent that was mid-turn when the host went
down had its turn cut off rather than finished.
The sweep now builds the same `Signal` -> `Drain` -> `StopForUpdate` chain
a graceful `hivectl restart` already uses, reusing the existing nodes and
`GRACEFUL_STOP_TIMEOUT` unchanged. Cost is bounded: the per-agent drains
overlap, so the sweep waits one timeout in total rather than one per agent.
`Signal` parents the rest of the stop instead of sitting beside it. All
three of `Signal` / `Drain` / `StopForUpdate` declare the agent lease, and
a resource is held across its holder's whole subtree — as siblings each
would take the lease separately, leaving a window between them for another
DAG to claim the agent mid-bounce.
Scope is the boot sweep alone: a manual rebuild, a meta-update cascade
child and a deploy all still stop mechanically, and a test pins that shape.
`rebuild_nodes` takes a `RebuildOpts` struct rather than a second
positional `bool`, which two adjacent flags would have made easy to swap at
a call site. Its callers no longer hard-code the subgraph's length either:
the `EmitRebuilt` tails and `FinalizeDeploy` used literal indices that
silently encoded "this builder emits exactly six nodes with `Reconcile`
last", which a variable-length subgraph turns into a wrong-node edge rather
than a compile error. They read the index off the emitted list now.
Review follow-ups on #2785.
`DepWhen` wraps `BitFlags<TerminalState>` instead of a hand-rolled `u8`,
so the bit manipulation belongs to the library and `TerminalState` gains
its flag value from `#[bitflags]` rather than a `bit()` match anyone
could get wrong. `of`/`accepts`/`is_empty` become one-liners over it.
Serialization is written out by hand rather than derived: clippy's
`unsafe_derive_deserialize` fires on deriving over a type with unsafe
internals, and the honest fix is to say what the wire form is. It is now
the list of accepted outcomes — `["done","failed"]` — which reads better
than a bitmask and survives the bits being renumbered.
Also drops the `pub use` re-export of `DepWhen` / `TerminalState` from
hive-c0re's `model`. It existed so that `use super::model::…` kept
compiling, which is a shim for one consumer's convenience; the sites
import from `hive_jobq` directly now.
And removes comments narrating what the code used to be. Git holds that.
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
`DepWhen` was two named cases, so every new combination wanted a new
variant. It is now a set over the terminal outcomes: a `u8` bitset
newtype, no dependency, with `AFTER_OK` / `AFTER_ANY` kept as the two
constants the templates actually use. "Run regardless" is all outcomes,
"anything that isn't a failure" is `{Done, Cancelled}`, a compensation
branch is `{Failed}` — closed under combination, so it never needs
another variant.
`TerminalState` is its own type rather than a subset of `State`, so an
edge cannot name `Pending` / `Running` / `Finishing`. Those are
meaningless in a dependency and are better unrepresentable than
validated against. The empty set is the one thing that can't be typed
away — nothing satisfies it, so `validate` rejects it next to the cycle
check.
Two consequences worth calling out:
- `cascade_cancel` collapses to one rule: a pending node is doomed once
any edge it names can no longer be satisfied. The hardcoded `AfterOk`
special case is gone, and a weak-edged node survives its dependency's
cancellation because of its own edge rather than by exemption.
- The cascade now runs on **any** terminal outcome, `Done` included.
With sets, success rules dependents out just as failure does — a
`{Failed}` branch is unsatisfiable the moment its dependency succeeds,
and leaving it `Pending` would wedge the subtree non-terminal forever.
That is a hang, not a wrong answer, so it is the load-bearing half of
this commit.
Edges are conjunctive, so "any of these N failed" is not directly
sayable. The composition that works is in the tests: the success branch
depends `AFTER_OK` on every root, so it is itself cancelled the moment
one of them doesn't succeed, and the failure branch hangs off *that*
with `{Cancelled}`. Exactly one of the two runs.
Also deletes hive-c0re's duplicate `DepWhen` enum and the
`to_crate_when` translation beside it. The copy bought nothing and had
to be widened in lockstep with the crate's edge model — it is the
in-between layer #2772 exists to remove, and it is what broke the build
when the crate's spelling changed.
All 34 jobq tests pass, including the four new ones covering both
directions of a failure-only branch, weak-edge survival of a cancelled
dependency, and the aggregator composition.
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
`Template` was a DAG-level enum that three different things read back
out: `terminal_hook()` mapped it to a side effect, the retention pass
bucketed history by it, and a tracing field printed it. None of those
needed a *label* — they needed the two facts the label happened to
encode. So the enum was a lossy stand-in for intent, and every new DAG
shape had to pick the variant whose inferred behaviour matched, whether
or not the name fit (`reparent` rode `MetaUpdate` for exactly this
reason, with a 10-line comment apologising for it).
Replace the inference with a declaration: `DagSpec.hook:
Option<HookKind>`. Only the builder assembling a DAG knows why it did
so, so only the builder can say what should happen when it settles.
`run_terminal_hook` becomes a field read, and `reparent`'s apology
becomes `hook: None`.
Hook assignment is byte-identical to the old precedence rule
(`approval_id.is_some()` wins, then `Rebuild | PermChange`), checked
site by site; `meta_update` is the only builder with a variable
approval id and so the only remaining conditional.
Retention loses the per-template bucket with the enum that keyed it.
The dashboard renders one recent-builds list, so one flat newest-first
cap (`MAX_HISTORY_DAGS`) bounds it. `HISTORY_GRACE_SECS` goes too — it
existed to stop a burst of same-template DAGs evicting each other
inside one poll interval, which is not a failure mode a flat cap has.
That takes `snapshot_capped()` and the `snapshot_no_grace()` test hook
with it.
The queue is runtime-only (empty graph on boot), so the serde changes
carry no migration risk.
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
A config-PR deploy was one opaque node that fetched, verified, merged,
built and compensated. That shape made three things impossible: the
nix-heavy phases could not take the meta window without the cheap ones
holding it too, a crash mid-build left no node to run the rollback, and
the dashboard could only ever show "deploying" for the whole thing.
Replace it with a `DeployWindow` group root over `MergeVerify ->
DeployApply` (AfterOk) plus a `DeployTail` hanging off the apply with
AfterAny, so the tail runs whether the apply succeeded, failed, or was
cancel-cascaded by a failing verify.
The two-phase approval deploy keeps a bumped `flake.lock` staged
uncommitted for the whole container build, so no other meta mutation may
land inside that span — until now enforced by a process-global
`meta::exclusive()` mutex held inside each executor fn.
A `MutexGuard` cannot outlive the fn that takes it, which is what blocks
decomposing the opaque `ApprovalDeploy` node into scheduler-visible
sub-nodes: the window has to span them. Replace the mutex with
`Resource::MetaWindow`, a global capacity-1 queue resource declared by
every meta-mutating node kind (`NodeKind::needs_meta_window`). Resources
are held by a subtree root across its whole subtree, so a later increment
can hang the deploy's phases under one window-holding parent.
Same global serialisation as before, and the scheduler now blocks a node
from being claimed rather than parking a worker on a mutex.
Split the rebuild's meta preamble out of `Prebuild` into a new `MetaSync`
node. `Prebuild` must NOT hold the window: the old mutex was deliberately
scoped to drop before the multi-minute toplevel build, which only reads
the store, and a cap-1 global held across it would serialise every
agent's rebuild behind every other's. `MetaSync` is a sibling root that
`Prebuild` deps `AfterOk` on — not its parent, since a parent's resource
covers its whole subtree and would reintroduce exactly that problem.
Queue tests: shape assertions gain the extra node, which is the point of
the change (phases become nodes). The concurrency invariants are intact
but observed one step later — the `MetaSync` heads take turns on the
window, exactly as the runtime mutex made them, so those tests now
complete the heads before asserting that the prebuilds overlap.
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).
Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
sibling ordering directly (deps order execution, parent groups a subtree
whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
separate top-level root (AfterAny Prebuild) so it survives the cancel-
cascade of any failed step (recovery-start invariant) and converges to
the persisted `wanted` on a fresh lease. This is the multi-root
correction to the single-root-chain sketch: node0=root broke lease-
exemption (hoisting the lease onto Prebuild) and recovery-reconcile
(root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
same way; per-agent power-op subgraphs stay independent roots so a
multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
DAG terminal node deps AfterAny on every group root and runs once the
whole op rolls up. Drop the old Graph::add_dep terminal wiring.
36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
Since the boot sweep (#2450) and meta-update cascade (#2476) became
single DAGs that grow subgraphs in-place, nothing constructs the old
fan-out anchors/parents anymore:
- NodeKind::Noop (the old boot_root grouping anchor) — no constructors.
- Template::StartupSweep / Source::StartupSweep (the old fan-out parent
template + cascade-child source) — replaced by Template::Boot and
Source::AutoUpdate/MetaUpdate respectively.
Drops the three variants + their as_str arms + the Noop executor arm, and
refreshes the stale fan-out/anchor doc comments (Boot/MetaUpdate/Source
docs, coordinator.md, dashboard.md). Frontend: the queue-kind glyph moves
from the dead startup_sweep to boot (which had none), and the dead
rqe-source-startup_sweep style is dropped.
No behaviour change — pure dead-variant removal.
With the meta-update cascade (#2476) and startup sweep (#2450) folded
into single DAGs that grow per-agent subgraphs via append_subgraph,
nothing links parent/child DAGs anymore — parent_id is dead.
hive-c0re: drop parent_id from Dag/DagSpec (+ the DagView copy); delete
append_children and cancel_children (no callers); simplify trim_history
(no more terminal-parent-with-live-children guard — a one-big-DAG is
terminal only when its whole graph settles); drop the rebuild() parent_id
param; QueueDag returns just the polled DAG (no fan-out children to
gather). hive-sh4re: drop the DagView.parent_id wire field.
frontend: a multi-step op is one DAG now, so renderRebuildQueue drops the
childrenOf/orphans cross-DAG grouping and renders each entry flat; its
per-agent subgraphs render as nodes within the one row (split by deps).
Removed the dead rqe-child style + isChild plumbing.
Docs + the child-DAG queue tests updated/removed to match.
MetaLock's non-sweep completion now grows one rebuild subgraph per
affected agent into the same DAG (append_subgraph), replacing the
fan-out-child-DAGs + cancel_children dance. Drops NodeOutput.fanout and
scheduler's fanout_specs. meta_update DAG carries Rebuilding transient so
each cascade agent gets crash-watch suppression at Swap (the property the
old child Rebuild DAGs held via their own transient); MetaLock head needs
no lease so the pseudo-agent gets no pill.
append_children/parent_id and child-DAG tests are intentionally left for
the #2453 capstone.
The boot is now ONE DAG, assembled inline in submit_boot_tree — no boot_root
Noop anchor, no per-agent child DAGs, no display-only parent_id grouping: a
sweep MetaLock root (only when something is stale) that grows one rebuild
subgraph per stale agent into the same DAG (via append_subgraph, previous
commit), plus one Reconcile root per drifted agent (independent — a boot
reconcile needs no lock bump).
- submit_boot_tree builds the DagSpec inline; removed the single-use
templates::{boot_root, startup_sweep} builders (inlined per the operator's
"don't force single-use shapes into templates.rs" steer).
- fanout_specs simplified to the meta-update cascade path only — the startup
sweep no longer fans out child DAGs, so its branch was dead.
- test: append_subgraph_roots_on_emitter_and_rebases_local_deps.
Vestigial after this (deliberately left as follow-ups, flagged in the PR):
NodeKind::Noop is now unconstructed (contained to hive-c0re, removable);
Template::StartupSweep is unconstructed but a hive-sh4re wire type
(frontend-coordinated removal).
Hive-wide `stop` / `start` / `restart` emit ONE DAG with a per-agent
subgraph each (concurrent on their own leases) instead of N DAGs — and each
subgraph is now built dynamically from the agent's live running state rather
than a fixed template shape:
- online agent: the full stop→reconcile (restart: stop-for-update→reconcile)
chain; `graceful` prepends signal→drain.
- offline agent: just `SetWanted → Reconcile` (nothing to quiesce/stop; a
restart of a down agent is really a start).
The head `SetWanted` (intent) and tail `Reconcile` (convergence guarantee)
are always present; only the mechanical `Signal`/`Drain`/`StopForUpdate`
nodes are state-conditional. Keeping `Reconcile` in every shape closes the
TOCTOU window — a race-up between the `is_running` read and node exec is
still converged in-DAG (with `StopForUpdate`-noop as the backstop) — with no
reliance on an external reconcile sweep.
The state-aware assembly needs an async `is_running` read, so it moves out
of the pure/sync `templates.rs` into `submit.rs`, layered as pure
`*_chain(running)` → pure `*_spec(targets)` (the unit-test seam) → async
`*_many` (reads live state + submits). `templates.rs` keeps only the shared
pure primitives (`node`/`after_ok`/`rebuild_nodes`).
Callers await the now-async submit fns (server, dashboard, socket_server).
Tests exercise both the online and offline shapes via the pure `*_spec`
seam. docs/coordinator.md shapes updated.
A hive-wide restart was N separate single-agent DAGs (one submit::restart
per agent). Now that agent is per-node (#2445), make it ONE DAG with a
per-agent restart subgraph each.
- templates::restart takes an agent list: each agent gets an independent
subgraph (a head SetWanted(Up) root, then its restart chain), so the N
subgraphs run concurrently on their own leases. One agent = the ordinary
single-agent restart; unifies the old restart + graceful_restart fns.
- submit::restart / graceful_restart stay as single-agent wrappers over
the new submit::restart_many(agents, graceful).
- server.rs handle_restart_all + handle_restart_scoped submit one
restart_many call instead of looping per agent. Infra containers
unchanged (no lease/DAG, synchronous).
Scope: restart + restart-all only. Broad stop+start is the same pattern
(stop/start templates take agent lists) — a follow-up increment.
The durable 'wanted' power intent was written by submit::{start,stop,
restart,graceful_restart,graceful_stop} as a synchronous pre-submit side
effect, then read by the DAG's tail Reconcile. That's not crash-safe
(a crash between the write and the enqueue loses it) and, with agent now
per-node, can't be per-agent in a DAG that spans agents.
Move it into the DAG as a head SetWanted node:
- NodeKind::SetWanted { up } + run_set_wanted executor (fails the node on
a write error, unlike the old warn-and-continue, so a stale intent
never reaches Reconcile).
- LEASE-NEEDING, not lease-exempt: it takes the agent lease so a power-op
DAG's intent-write + reconcile is atomic per-agent. If it were exempt,
two racing ops (restart vs stop) would run both intent-writes up front
and clobber each other before either reconciled — defeating the point
of moving the write into the DAG. (In stale_start the lease is thus held
across the head Prebuild, but that's a no-op there: the agent is down so
prebuild is skipped.)
- templates: explicit SetWanted node 0 on restart/graceful_restart/
graceful_stop, plus dedicated start/stop templates (SetWanted -> Reconcile)
and stale_start (SetWanted(Up) -> rebuild subgraph, reusing rebuild_nodes).
No compose helper / rebuild variant. reconcile_only is now boot-only.
- submit.rs: drop the set_wanted side effect; the stale-rev shape decision
(start vs stale_start) stays submit-side.
All 33 job_queue tests pass (shape/lease tests updated for the head node).
Agent was a single field on Dag/DagSpec, making a DAG structurally
one-agent — a multi-agent op could only ever be N separate DAGs. Move it
onto Node/NodeSpec (and the NodeView wire type), drop it from Dag/DagSpec
(and DagView): a DAG can now span agents.
- lifecycle lease keys on the node's agent, still globally exclusive per
agent across all DAGs (Inner.leases unchanged in shape). A DAG holds one
lease per distinct agent it touches; settle() frees each at DAG-terminal
(per-agent-subgraph early release is a follow-up, only observable with
multi-agent DAGs).
- transient guard keyed (dag_id, agent); cancel-revert + Rebuilt events
walk TerminalDag.agents.
- submit-time dedup removed (a multi-agent DAG has no single agent to key
on); every submit enqueues a fresh DAG. Whether dedup needs reintroducing
is tracked in a follow-up sub-issue.
- templates gain a node(agent, kind, deps) helper stamping the agent onto
every node; meta templates stamp "hyperhive".
Templates stay single-agent in this PR — behaviour is unchanged, only the
representation + wire shape. Multi-agent DAG emission (restart/restart-all/
broad stop+start as one DAG) and the SetWanted-as-a-node change are
follow-ups off #2439.
mara's review on #2436: no submit-await-submit composition, even
server-side. Adds Template::GracefulRestart (Signal -> Drain ->
StopForUpdate -> Reconcile, wanted=Up) mirroring how Restart already
does StopForUpdate -> Reconcile, plus submit::graceful_restart and
templates::graceful_restart. handle_restart_scoped now submits exactly
one DAG per agent up front for both the graceful and non-graceful
case -- no await_dags in the loop anymore.
- deploy-window gate (meta::exclusive) + path-limited meta commits:
a perm/lock/topology commit can no longer sweep an ApprovalDeploy's
staged flake.lock and neuter abort_deploy (regression test included)
- cancel surfaces now buffer terminal roll-ups the scheduler drains,
so a queued approval DAG cancelled by the operator resolves its
approval instead of dangling, and cancelled power ops revert their
wanted flip to the observed state
- hivectl restart / restart-all ride the queue (lease serialization,
transient guard) and restart sets wanted=Up like the old kill+start
- exactly one Rebuilt event per rebuild DAG, emitted at terminal
- StopForUpdate pre-seeds a missing agent_power row from the pre-stop
observation so a rebuild can't strand an unknown agent offline
- history trim keeps terminal fan-out parents with live children
- audit_log back on db::open; swarm.js badge for reconcile DAGs
jobs are now DAGs of primitive nodes (prebuild, stop-for-update, swap,
reconcile, signal, drain, ...) driven by one scheduler with N build
slots + per-agent lifecycle leases. per-agent power intent (wanted
up/offline) is durable in agent_power.sqlite; Reconcile nodes converge
observed state to it. kills the graceful-stop watcher thread, the
deferred-start follow-up, and the cascade pre-enqueue (fan-out on
MetaLock completion instead). tracker: #2166
