A node no longer hands back a recipe for the scheduler to replay later. It
declares straight onto a builder it was given, and that builder is inserted
as part of completing the node.
Deleted: `pub type Declare`, `struct NodeOutput` (+ its hand-written `Debug`),
`JobQueue::append_subgraph`. Nothing added to `Dag` / `DagView`.
jobq gains `Scheduler::new_job()` (the only way to obtain a `JobBuilder`) and
`complete_growing(id, outcome, grown)`, which inserts under `id` and *then*
completes it, so a DAG cannot roll terminal while grown work is still pending.
`complete()` and `complete_growing()` share a private `finish()` rather than
one redirecting through the other. The DAG-gone guard lives beside the graph
now, where it cannot be skipped, instead of being a caller-side lookup.
The growth executors return data (`run_meta_lock -> (Vec<String>, RebuildOpts)`,
`run_reconcile -> Option<NodeKind>`) rather than taking the builder: a `&Job`
parameter is live for the whole function body, and `&RefCell<T>` is never
`Send`, so an async fn taking one cannot be spawned. `run_node` threads the
builder by value and hands it back.
A node can now declare work and then fail, which was previously inexpressible.
`grown` is dropped in that case — failure cancel-cascades downstream, so
inserting it would only add nodes to immediately cancel — and the log line
carries `grown_nodes` so the drop is visible.
Resources were derived from the node's kind: `templates::node` called
`NodeKind::resource_deps()`, which fanned out to `needs_build_slot` /
`needs_lease` / `needs_meta_window`. That made the requirement a property
of the *kind*, so a kind that happened to run under an ancestor already
holding the resource could get away with declaring nothing.
Three did. `Start`, `Stop` and `PostSwap` appear in none of the three
predicates, and that was only safe because one construction site fans
them out from inside a lease-holding `Reconcile` — a fact about today's
DAG shape, not about the nodes.
Each of the 41 construction sites now says what it holds. `Start` /
`Stop` / `PostSwap` declare the agent lease; per the contract that is a
re-entrant borrow, which a new test pins rather than argues.
`running_transients` reads the node's declared deps instead of
re-deriving from the kind. That closes the blank-pill gap: the pill went
blank during container start, stop and the post-swap tail because the
declaration was missing, not because the filter was wrong.
The deleted predicates carried the only written record of three design
decisions; each moved to the `Resource` variant it constrains rather than
dying with its function.
Three findings from the operator's review, all correct.
1. Two insert_job's. Graph::insert_job had no caller outside hive-jobq's
own tests -- production only ever went through Scheduler::insert_job.
It existed because the graph-level one got written first. Deleted; the
tests moved onto a Scheduler, which is where insertion belongs anyway.
2. insert_job was not atomic, and the previous commit made that worse: a
forward edge or forward parent surfaced mid-loop, leaving the nodes
before it in the graph, and resolve_wanted ran after every insert, so
an unknown handle failed once the whole job was already committed.
The module documented this under "Partial insertion" instead of fixing
it -- prose describing a hole is not a design.
All three are decidable from what the builder holds, so
check_declaration_order now runs before the first insert and the loop
indexes ids directly. A malformed job leaves the graph untouched.
What remains mid-insert is the graph's own rejection (out-of-group
dep, empty DepWhen); closing that needs a dry-run validate on Graph,
which is a separate change.
3. DagSpec no longer boxes its recipe: it is generic over the closure,
which travels from the template that built it straight into submit.
The box bought type inference, and paying for it costs annotations --
`|b: &Job|` at each declaration site (the field needs an HRTB, and an
unannotated closure binds one lifetime) and `+ use<>` on each
returning signature (or the opaque type captures the caller's borrows).
Erasure is still needed where several recipe shapes share one type:
the boxed Declare stays for the executor's append_subgraph, and a test
table uses an erase() helper.
The operator's instruction on the issue was "the closure returns an array
of guids, and enqueue_job returns the node ids in that order". What was
here instead returned a HashMap of everything inserted, and no caller used
the keys: submit dropped the return, insert_group did into_values(), and
the scheduler ignored what append_subgraph handed back. The guid-keyed
lookup was dead weight, and into_values() made that Vec arbitrarily
ordered -- harmless only because nothing read it.
insert_job now takes FnOnce(&JobBuilder) -> Vec<NodeGuid> and returns the
matching ids positionally. A handle from another job is UnknownNode rather
than a silent omission: the return is positional, so a short vector would
misalign every id after it.
c0re's Declare stays FnOnce(&Job) and the wrapper names no handles in one
place, rather than ending seven templates in an empty vector -- a DAG is
addressed by its container node, which submit inserts itself. That frees
insert_group from needing every id, so the node_rt pre-seeding goes too:
NodeRuntime is one Option field and every reader already tolerated a
missing entry (entry().or_default(), get().and_then(), iter().find()).
The tests are the argument for the shape: capturing a handle through a
mutable binding to look it up in the map afterwards collapses into
returning it and destructuring the result.
Follows the jobq change: a builder can no longer be constructed or
inserted outside `hive_jobq`, so `DagSpec` cannot hold one. It carries a
`Declare` — `Box<dyn FnOnce(&Job) + Send>` — and the queue runs it
against a builder jobq owns, at the moment it inserts.
`NodeOutput.append_subgraph` becomes `Vec<Declare>` for the same reason,
and this is where the shape was always heading: that field's doc already
said an executor "cannot reach the queue, so it hands the declaration
back", while its type was a `Vec<Job>` the executor had built itself.
The rejected `build_nodes -> Vec<NodeSpec>` was the first version of that
escape hatch; a recipe is the last one, because there is no job-shaped
value to hand over at all.
Templates and the power-op assemblers move their owned data into the
closure and are otherwise unchanged — `rebuild_nodes`, `node` and the
tail helpers already took `&Job` and returned handles, so only each
template's outermost frame moved.
Two `Debug` impls are hand-written: a closure has nothing to show, and
its nodes do not exist until the queue runs it. `NodeOutput` reports how
many subgraphs were emitted, `DagSpec` its source and reason.
`append_subgraph`'s `is_empty()` early-return is gone — you cannot ask a
recipe whether it will declare anything without running it. It now
inserts and returns an empty id list if nothing was declared, which
takes the queue lock in a case that previously skipped it.
The two in-DAG-growth tests build `Declare`s now, so they exercise the
shape an executor actually produces rather than one only a test could
construct. 45 job-queue tests unchanged and passing.
Every template built a `Vec<NodeSpec>` whose edges and parents were
positional indices into that vector, so a shape was expressed as
arithmetic: `base + 1`, `stop_root + 2`, `sfu + 1`, and a
`reconcile_index()` helper that read the emitted vector's length to find
out where its own last node had landed. `concat_subgraphs` existed
solely to rebase one per-agent subgraph's indices onto another's.
Templates now declare into a `hive_jobq::JobBuilder` and hold the
handles they get back, so an edge names the node it waits on. The
arithmetic is gone, and with it:
- `NodeSpec` and the job-queue's own index-based `Dep`.
- `insert_group`'s index resolution — it wraps `Scheduler::insert_job`.
- `concat_subgraphs` — per-agent chains share one builder and each keeps
its own root, so independence is structural rather than computed.
- `reconcile_index` and `dep_index`.
- `templates::validate` and its petgraph toposort. It rejected dangling
deps and cycles; both are now unrepresentable, since a handle only
exists for an already-declared node and every edge therefore points
backwards. (petgraph stays in the tree for `agent_config::topology`.)
`NodeOutput.append_subgraph` becomes `Vec<Job>`: an executor cannot
reach the queue, so it hands back declarations and the scheduler inserts
them under its own lock. That is what the in-DAG growth path always
wanted — a transferable declaration, not a vector of specs.
Resource declaration is unchanged in behaviour: the `templates::node`
helper applies `NodeKind::resource_deps()` at the construction site, so
every node still declares what its kind needs. Moving that declaration
to the call sites is #2818's job; this leaves it one place to delete.
Three tests went with the guard they covered — they hand-built malformed
specs out of indices, which is the representation that made those shapes
possible. Two more now read a DAG's shape off the queue rather than out
of a spec vector, which is where it is observable. The remaining 45
job-queue tests are unchanged and still pass: lease serialization,
roll-up, cancel-cascade, in-DAG growth and per-agent concurrency all
behave as before.
It never produced a `Dep::Node`, so returning `Vec<Dep<Resource>>` made
every caller match a variant that cannot occur. `running_transients`
paid for it with a two-arm match to pull the agent out of a lease edge.
`Vec<(Resource, u32)>` says the same thing in the type, and is what the
job builder's `.needs_units(name, count)` takes — the insertion path
wraps it back into a `Dep::Resource` at the one place that still speaks
in edges.
mara on !2910: "why is set_transient still a thing if it completely
derives from nodes?"
It was still a thing because the scheduler mirrored the derived set into
a stored map that every consumer read — derived state computed once and
then cached, with the reconciliation loop existing only to keep the cache
honest. `transient_snapshot()` now derives: `running_transients()` off the
live graph, with the handful of entries that have no node behind them
(destroy, migration) overlaid on top. There is no cached copy left to go
stale or disagree with what is running.
`set_transient` / `clear_transient` split by what they actually do:
`set_manual_transient` / `clear_manual_transient` own the stored map for
the no-node callers, and `emit_transient_set` / `emit_transient_cleared`
publish the edges both paths need.
Two things had to survive, and both are edges rather than state:
- The dashboard's `TransientSet` / `TransientCleared` events. The
scheduler carries the previous derived value and emits the diff.
- The crash watcher's grace window. `recent_transient_within` answers
"was a transient cleared just now?", which is what stops a deliberate
stop from reading as a crash on the next 10s poll — a derived read of
current state cannot answer it, so the clear still stamps. The
scheduler keeps `deliberate_stop` alongside the label precisely so it
is available at clear time: the node it came from is, by definition, no
longer running to be asked.
`TransientState::since` becomes wall-clock and, for derived entries, is
the node's own `started_at` — the true start of the operation rather than
the moment a watcher first noticed it, which is what the old
guard-creation timestamp actually measured.
`running_transients` returns a named `RunningTransient` rather than a
4-tuple; two of its fields are strings and one is a bool whose meaning is
not guessable at a call site.
Note for anyone reaching for a timestamp here: chrono is vendored with
`default-features = false`, so there is no `Utc::now()`. The workspace
convention is `wire_time::now_unix()` / `from_secs()`.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
mara on !2910: "rename now, we will see if we can remove it later when
some of the users have been removed or work differently."
Nothing is held. The old name described a transient the DAG declared and
kept for its whole lifetime — precisely the thing this PR replaces — so
it outlived its own meaning the moment the derivation landed. The value
is recomputed from the running set on every call.
Kept as a function rather than inlined at its single call site, per the
above: removing it is a later step that depends on its users changing,
not something this PR should force.
Rename plus its two references (the call in `reconcile_transients` and
the module doc link). No behaviour change; the doc comment records what
the old name meant so the rename doesn't erase the reason for it.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (322 + 41 passed) and `nix fmt`.
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.
It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.
`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.
`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.
## the safety half, which is deliberately not the display half
`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.
`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.
For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.
## known cost, accepted on the issue
A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.
`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
mara on !2909: "shouldnt node_by_id be part of jobq?" — yes. Resolving a
raw value to a `NodeId` is the exact inverse of `NodeId::get`, which
already lives in hive-jobq, and it is only a search because the graph
owns the counter that makes ids unfabricable. Both halves of that
round-trip belong on the same side of the crate boundary.
Placing it in `QueueInner` also put it in a layer slated for removal, so
the c0re-side helper would have had to move later anyway — and it was
private there, leaving any other caller needing the same resolution to
write the same `nodes().find_map(…)` by hand.
`Graph::resolve_id` replaces it, with a unit test covering the
round-trip and the rejection of a value that was never an id.
While re-reading the diff for that question: the doc comment added in
the previous commit landed *between* `container`'s doc comment and its
signature, silently reattaching "The container node of `dag_id`" to the
new helper and leaving `container` undocumented. Restored.
No behaviour change and no wire change — same search, same call site.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-jobq -p hive-c0re` (41 + 322 passed) and `nix fmt`.
`JobQueue::cancel(dag_id)` resolved the id to a `NodeKind::Dag` container
and cancelled that. But the container lookup was the only DAG-specific
part — everything that makes cancel work already lives in the scheduler:
`cancel_node` marks the node `Cancelled` and cascades to its pending
descendants, sparing any node whose edge accepts `Cancelled` (which is
what keeps a dropped approval DAG from dangling its row).
So `cancel` now takes any node id. A group root cancels the whole group,
which is what the dashboard's button does today and why nothing about
its behaviour changes: a DAG id *is* its root node's id. An interior
node cancels just that branch — a capability the DAG-scoped version
could not express, covered by the new test (a hive-wide restart drops
one agent's subgraph while the other keeps running).
`QueueInner::node_by_id` replaces `container()` here: same search, same
cost, without asserting the node is a DAG container. `container()` stays
for `first_error` and the append-subgraph guard, which are genuinely
DAG-scoped.
No wire change. The route is `POST /api/rebuild-queue/{id}/cancel` with
a `u64` path param — same type, same route, and the client keeps sending
the same number. Only the param's documented meaning moves from "DAG id"
to "node id".
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (322 passed) and `nix fmt`. No option surface touched, so no
nix-eval gate.
Second of the `Dag` field removals, and the same shape as the first:
`DagSpec`/`NodeKind::Dag` carried an `inputs: Vec<String>` that exactly
one node ever read. Both reads live inside `run_meta_lock` — the
`meta::lock_update` call and the `meta_update_cascade_agents` fan-out —
so the list now rides `NodeKind::MetaLock` itself.
The executor stops touching `Claim` for this node entirely: its dispatch
arm already destructured `MetaLock { sweep, fanout }`, so `inputs` joins
them and the `claim` parameter, which had no other use, is gone.
Falls out of that:
- `Claim::inputs` and `DagMeta::inputs` delete.
- `dag_view`'s DAG-level projection onto the `MetaLock` node reads the
payload instead. The wire `NodeView::inputs` is unchanged: still
populated on the `meta_lock` node alone.
- the boot sweep names no inputs (it bumps `hyperhive` alone via
`lock_update_hyperhive`), which the construction site now says out loud
rather than leaving implicit in an empty DAG-level field.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (320 passed) and `nix fmt`. No option surface is touched, so
no nix-eval gate.
`DagSpec`/`NodeKind::Dag` carried an `Option<i64>` approval id that four
deploy phases read back out through `Claim`, via a fallible helper whose
error ("approval deploy dag N has no approval_id") described a state the
type system should have forbidden. Two other templates (`spawn`,
`meta_update`) set the field for nothing: their approval is resolved by
the `ResolveApproval` tails, which already carry the id themselves.
So the id moves onto the nodes that actually need it —
`DeployWindow` / `MergeVerify` / `DeployApply` / `FinalizeDeploy` /
`DeployTail` each take an `i64`, the same way `ResolveApproval` always
has. `templates::approval_deploy` builds all of them in one place with
the value in hand, and `deploy_rebuild_nodes` takes it as a parameter so
the `FinalizeDeploy` it appends at runtime is constructed the same way.
Falls out of that:
- `deploy_approval_id` and its runtime error path delete; each executor
takes the id from its own node payload at dispatch.
- `run_deploy_window` had nothing left to do but validate that id, so the
node joins `Dag` on the shared no-op arm.
- `Claim::approval_id` and `DagMeta::approval_id` delete.
- `dag_view`'s DAG-level projection onto `DeployWindow` reads the payload
instead. The wire `NodeView::approval_id` is unchanged: still set on
the deploy root alone, so the dashboard still renders one approval link
per DAG rather than one per phase.
No option surface is touched, so there is no nix-eval gate here; checked
with clippy (`--all-targets -D warnings`), `cargo test -p hive-c0re`
(320 passed) and `nix fmt`.
Per mara's review on #2896: has_log: bool was fully redundant once
build_log_id: Option<i64> existed alongside it (has_log was always
just build_log_id.is_some()). Dropped has_log, threading the single
Option<i64> field through job_queue::mod.rs, the hivectl NodeView
test-helper literal, and the one remaining frontend consumer
(findLiveBuild's live-log-panel gate, which now checks
build_log_id != null instead of the separate bool).
Also fixed a now-stale doc comment on GET /api/build-log/{node_id}
that claimed the dashboard used on-demand node-id fetches "instead
of an inline build_log_id on the wire" -- no longer true after this
PR put one there for the BUILD L0GS deep-link.
cargo build/clippy/test clean across the three touched crates; nix
fmt clean; frontend build verified (0 has_log references, 3
build_log_id references in the built builds.js bundle).
Fixes hyperhive#2895. The rebuild-queue tree's per-node log icon (the
printer-glyph "open" affordance next to each node in the R3BU1LD
QU3U3 tab) linked directly to the raw-text download endpoint
(/api/build-log/<node_id>/raw, which sets Content-Disposition:
attachment server-side) -- surprising, since nothing about that icon
signals "this leaves the app", unlike the other two explicit
"download raw"/"raw" links elsewhere on the page.
Point it at the existing ?id=N#buildlogs deep-link into the BUILD
L0GS tab instead (builds.js's fetchBuild already auto-expands +
scrolls to the matching row there). That deep-link's id is the
build-log history row id -- a different id space than the queue
tree's NodeId, and wasn't exposed to the frontend before (only a
derived has_log bool was). Added NodeView.build_log_id: Option<i64>
to the wire type alongside the existing has_log (kept, since
findLiveBuild's separate live-log-panel gate still needs a plain
bool), threaded through job_queue::mod.rs, updated hivectl's NodeView
test-helper literal.
The raw download is still one click away once on that row's BUILD
L0GS detail (the two already-explicit raw-download links are
untouched). cargo build/clippy/test clean across the three touched
crates (hive-c0re, hive-host-sock, hivectl); nix fmt clean; frontend
build verified (grep for build_log_id in the built builds.js bundle).
`NodeId` has been globally unique across DAGs since #2801, so the dag id
carried no information the node id didn't. The parameter was already
underscore-prefixed as unused, but still populated by `claim_ready`, carried
through the scheduler's mpsc on every `Claim`, and passed at the call site —
three layers of plumbing feeding a dead argument.
Removing it surfaced four more dead things it had been keeping alive:
`settle_approval_tail`, `settle_rebuild_tail` and `drain_meta_syncs` each took
a dag id they only forwarded to `complete_node`, and one `submit` binding was
never read. Those are deleted rather than underscore-prefixed — prefixing is
what let the original argument survive this long.
`Claim.dag_id` stays: it has live consumers in the tracing spans,
`append_subgraph`'s container guard, `Ctx` for the build-log link,
`first_error`, and the approval-deploy context.
`hive_host_sock::jobs::State` was a hand-maintained copy of
`hive_jobq::State` — five variants spelled the same in both, kept in sync
by whoever remembered. Adding `Skipped` last week meant adding it twice.
The wire crate now re-exports the scheduler's enum and `to_wire_state` is
gone.
Two states that were hidden now reach clients. `to_wire_state` renamed
`Pending` to `Queued` and folded `Finishing` into `Running`, so the
dashboard could not distinguish a node waiting on its dependencies from
one whose own work is done while its sub-nodes still run. Both are now
visible, and consumers say which they mean.
Every consumer had to move with it, and only the Rust ones said so: the
exhaustive matches in `hivectl` and `DagView::rollup_state` failed to
compile, while the dashboard's fourteen string comparisons would have
gone quietly wrong — a `finishing` node no longer counting as running,
a `pending` node no longer as queued.
The frontend also builds CSS class names out of the state string
(`rqe-` + state, `rqe-node-` + state) and keys its glyph map on it, all
lowercase. Those go through a `stateSlug` helper now; comparisons use the
wire spelling, presentation lowercases. Without that split every queue
entry and node chip would have silently lost its styling.
Dropping the `State as JobState` alias in hive-c0re falls out of this:
the alias only existed to tell two `State` types apart, and there is one
now.
`dag_of` and `dag_first_error` had shrunk to a single delegating call once
the walks moved into `hive-jobq`; their callers say what they mean without
the hop.
`subtree` was worse than redundant. It collected the descendant ids into a
`Vec` and both callers then looked each node up again by id — `dag_view`
needed a `let … else { continue }` for a lookup that could not fail.
Iterating `descendants()` hands back the node directly, so the round-trip
and the re-lookup both go.
`JobQueue::cancel` decided whether a DAG could be cancelled by reading node
run-state, walked the subtree, judged per node whether that node had asked
to observe cancellation, and re-ran the container's roll-up. Every one of
those is a fact the scheduler owns; core was reaching across the boundary
to compute them.
`Scheduler::cancel_node` now takes the whole subtree: cancelling a node
cancels the work under it, since a group is abandoned by abandoning its
root. The existing method generalises rather than gaining a sibling — it
had one production caller, which this replaces.
The gate runs over the work *under* the node, not the node itself: a group
root's state is its subtree's roll-up rather than a step that ran, so a
container is `Finishing` and never `Pending`, and gating on it would refuse
every cancel. A node with no children is its own work, which keeps the
previous single-node behaviour.
`observes_cancellation` moves in with it — it reads a node's declared edges
and knows nothing about what the payload means.
Core keeps the one genuinely domain-specific step, resolving a wire
`dag_id` to its container node, and is three lines otherwise.
Returning `false` for an unknown id gave the same answer as a node that is
merely still running, so a caller polling a stale id would wait forever for
a state that can never arrive. `Option<bool>` makes the two cases separate,
matching `node()`'s convention that `None` means the id isn't in the graph.
hive-c0re hand-rolled four traversals over a graph it doesn't own, because
`Graph` exposed only `node()` and `nodes()`. They are generic — nothing in
them knows what a hyperhive DAG is — so they move to `hive-jobq` and core
delegates.
`Graph` gains `root_of`, `descendants`, `roots`, `is_settled` and
`first_error`; they reuse the private `is_descendant` the crate already had
for its dep-scope rule. `subtree` gets faster on the way: core walked every
node's whole parent chain to the root for every node in the graph, where
`is_descendant` stops as soon as it sees the ancestor.
`first_error` deliberately looks for the first `Failed` descendant that
*carries* an error rather than the first `Failed` one. A node that rolled
its failure up from a child holds no error of its own and sorts before that
child, so the simpler version reports `None` for the common case and the
dashboard loses the reason. The distinction has its own test.
`dag_is_terminal` is deleted rather than moved: it was already a plain
`state.is_terminal()` read, and its three call sites now ask the graph.
A node ruled out by its own dependency edges settles `Skipped` host-side,
but the wire folded it into `Cancelled` and `dag_view` filtered it out
entirely, so a client never saw which branch a run didn't take. Post-#2785
that is not a rare shape: every approval DAG has two not-taken tails and
every rebuild has one, on the happy path as much as on failure.
`State` gains `Skipped`, and it counts as terminal — the wait loops in
hivectl's progress display and the daemon's dag-settled check decide
"finished" with `all(is_terminal)`, so omitting it would hang them on
essentially every DAG.
`dag_view` now emits skipped nodes but no longer lets them keep a DAG
alive. Serialization and completion were the same expression: a DAG left
the snapshot because its nodes had all been filtered away. Keeping skipped
nodes on the wire under that rule would pin every finished deploy in the
queue view forever, so the completion test is now its own flag.
`rollupState` in the dashboard gains the matching arm. It has no `done`
case — `done` is inferred by falling off the wire — so its trailing
`return 'queued'` catches anything it doesn't recognise, and a green
deploy would have read as permanently queued the moment the backend
started emitting the new state. The Rust and JS roll-ups have silently
disagreed before; they are edited together here and say so.
Review follow-ups on #2785.
`DepWhen` wraps `BitFlags<TerminalState>` instead of a hand-rolled `u8`,
so the bit manipulation belongs to the library and `TerminalState` gains
its flag value from `#[bitflags]` rather than a `bit()` match anyone
could get wrong. `of`/`accepts`/`is_empty` become one-liners over it.
Serialization is written out by hand rather than derived: clippy's
`unsafe_derive_deserialize` fires on deriving over a type with unsafe
internals, and the honest fix is to say what the wire form is. It is now
the list of accepted outcomes — `["done","failed"]` — which reads better
than a bitmask and survives the bits being renumbered.
Also drops the `pub use` re-export of `DepWhen` / `TerminalState` from
hive-c0re's `model`. It existed so that `use super::model::…` kept
compiling, which is a shim for one consumer's convenience; the sites
import from `hive_jobq` directly now.
And removes comments narrating what the code used to be. Git holds that.
Splits what was one `Cancelled` outcome into two, because they were two
different facts wearing one name:
- `Skipped` — the node's own edges ruled it out. Expected; the failure
branch of a run that succeeded is `Skipped`. A parent's roll-up
**ignores** it.
- `Cancelled` — the work was dropped before it could start. Still
not-success for the roll-up, as before.
Without that split, branching on outcome defeats itself: exactly one
branch is always ruled out, `any_child_failed` counted it, and every DAG
containing a branch would have rolled up failed no matter how the run
went. Caught in review before it was written, not after.
`AFTER_ANY` becomes `{Done, Failed, Skipped}` — "anything except the work
being dropped". That is what it always meant; it only swept in
cancellation because cancellation wasn't distinguishable from
elimination. Audited every user rather than assuming, which is how the
one regression in my own proposal surfaced: `{Done, Failed}` would have
refused to run rebuild's recovery `Reconcile` after a failed `MetaSync`
(that eliminates `Prebuild`, so the tail's dep is `Skipped`, not
`Failed`) and left the container down.
With that, the templates stop computing outcomes and let the graph pick:
- `ResolveApproval { approval_id, outcome }` — one tail per outcome, each
edged to accept only its own, so exactly one is ever runnable.
- `EmitRebuilt { agent, ok }` — a pair. `ok` is not derived, it is which
of the two the graph let run.
Edges are conjunctive, so "any of these roots failed" is not directly
sayable. The composition: the success branch is `AFTER_OK` on every root
(so it is itself eliminated the moment one doesn't succeed), and the
failure branch keys off *that* elimination. The failure branch also
waits on every root — without it, a failed `Prebuild` eliminates the
success branch immediately and the failure would be announced while the
recovery `Reconcile` was still running. The tests caught that one.
Deletes, all of them #2770's host-side debt:
- `Claim.deps`, `DepOutcome`, `Claim::deps_state`, `Claim::deps_error`
and the dep-snapshotting loop in `claim_ready`. Executors read their
own variant now; nothing inspects anything.
- `NodeKind::is_tail()` and the `cancel` exemption built on it. Sparing
is derived from the edges: `cancel` keeps a node iff one of its edges
accepts `Cancelled`. An approval tail names it and survives to resolve
the row; `Reconcile` doesn't and is cancelled with the rest. My earlier
claim that this couldn't dissolve was only true while `AFTER_ANY`
accepted cancellation.
`resolve_approval_dag` / `deploy_terminal_tag` now take `TerminalState`
rather than the wire `State`, so both matches are exhaustive instead of
ending in a catch-all.
Skipped nodes are filtered off the wire alongside `Done` ones. That costs
some dashboard detail on a failed rebuild — which steps were skipped —
and the tests say so with a pointer to the follow-up. Surfacing them as
`Cancelled` instead would be worse: the client roll-up ranks `Cancelled`
above `Running`, so a successful DAG with a not-taken branch would read
as cancelled.
`DepWhen` was two named cases, so every new combination wanted a new
variant. It is now a set over the terminal outcomes: a `u8` bitset
newtype, no dependency, with `AFTER_OK` / `AFTER_ANY` kept as the two
constants the templates actually use. "Run regardless" is all outcomes,
"anything that isn't a failure" is `{Done, Cancelled}`, a compensation
branch is `{Failed}` — closed under combination, so it never needs
another variant.
`TerminalState` is its own type rather than a subset of `State`, so an
edge cannot name `Pending` / `Running` / `Finishing`. Those are
meaningless in a dependency and are better unrepresentable than
validated against. The empty set is the one thing that can't be typed
away — nothing satisfies it, so `validate` rejects it next to the cycle
check.
Two consequences worth calling out:
- `cascade_cancel` collapses to one rule: a pending node is doomed once
any edge it names can no longer be satisfied. The hardcoded `AfterOk`
special case is gone, and a weak-edged node survives its dependency's
cancellation because of its own edge rather than by exemption.
- The cascade now runs on **any** terminal outcome, `Done` included.
With sets, success rules dependents out just as failure does — a
`{Failed}` branch is unsatisfiable the moment its dependency succeeds,
and leaving it `Pending` would wedge the subtree non-terminal forever.
That is a hang, not a wrong answer, so it is the load-bearing half of
this commit.
Edges are conjunctive, so "any of these N failed" is not directly
sayable. The composition that works is in the tests: the success branch
depends `AFTER_OK` on every root, so it is itself cancelled the moment
one of them doesn't succeed, and the failure branch hangs off *that*
with `{Cancelled}`. Exactly one of the two runs.
Also deletes hive-c0re's duplicate `DepWhen` enum and the
`to_crate_when` translation beside it. The copy bought nothing and had
to be widened in lockstep with the crate's edge model — it is the
in-between layer #2772 exists to remove, and it is what broke the build
when the crate's spelling changed.
All 34 jobq tests pass, including the four new ones covering both
directions of a failure-only branch, weak-edge survival of a cancelled
dependency, and the aggregator composition.
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
`Template` was a DAG-level enum that three different things read back
out: `terminal_hook()` mapped it to a side effect, the retention pass
bucketed history by it, and a tracing field printed it. None of those
needed a *label* — they needed the two facts the label happened to
encode. So the enum was a lossy stand-in for intent, and every new DAG
shape had to pick the variant whose inferred behaviour matched, whether
or not the name fit (`reparent` rode `MetaUpdate` for exactly this
reason, with a 10-line comment apologising for it).
Replace the inference with a declaration: `DagSpec.hook:
Option<HookKind>`. Only the builder assembling a DAG knows why it did
so, so only the builder can say what should happen when it settles.
`run_terminal_hook` becomes a field read, and `reparent`'s apology
becomes `hook: None`.
Hook assignment is byte-identical to the old precedence rule
(`approval_id.is_some()` wins, then `Rebuild | PermChange`), checked
site by site; `meta_update` is the only builder with a variable
approval id and so the only remaining conditional.
Retention loses the per-template bucket with the enum that keyed it.
The dashboard renders one recent-builds list, so one flat newest-first
cap (`MAX_HISTORY_DAGS`) bounds it. `HISTORY_GRACE_SECS` goes too — it
existed to stop a burst of same-template DAGs evicting each other
inside one poll interval, which is not a failure mode a flat cap has.
That takes `snapshot_capped()` and the `snapshot_no_grace()` test hook
with it.
The queue is runtime-only (empty graph on boot), so the serde changes
carry no migration risk.
The DagView / NodeView / Source / State / PermPayload types only ever
travel on the host admin socket and the dashboard channels hive-c0re
serves off the same snapshot; their whole consumer set is hive-c0re,
hivectl and the socket protocol crate itself. Living in hive-sh4re made
the five other crates that depend on it carry job-queue types they never
name.
Pure move: git mv of the module plus the import sweep, no type changes.
hive-sh4re keeps its own chrono (wire_time still needs it).
The revert hook is dead by construction, so it can only ever be wrong.
DAG state `Cancelled` has exactly one producer: `JobQueue::cancel`, which
refuses unless every work node is still `Pending`. A cancel *cascade*
(some node failed, downstream cancelled) rolls up `Failed` instead —
`dag_rollup` short-circuits on any failed subtree node. So on a DAG that
reaches `Cancelled`, no node ever executed: the `SetWanted` head provably
never ran and `wanted` still reads whatever the operator last set it to.
There is therefore nothing to revert, and `revert_intent` did not revert
anything — it wrote `Wanted::from_running(observed)`, i.e. the agent's
*observed* state, over an intent the DAG never touched. Harmless when
observed already matched, silent corruption otherwise: cancel a queued
start for an agent that is down but `wanted = Up` (crashed, or caught
mid-bounce) and the intent flips to `Offline`, leaving it
deliberately-stopped as far as reconcile and crash-watch are concerned.
The hook made sense when `set_wanted` was a pre-submit side effect
written before the DAG ran; moving it into the DAG as a node left the
hook vestigial.
Drop `HookKind::RevertIntent`, `revert_intent`, and the power-op arm of
`terminal_hook` — start / stop / graceful-stop now settle with no
terminal hook, same as restart always did. The test asserts the general
statement across restart/stop/start x graceful x running: stop and start
carry a `SetWanted` head, and cancelling them still fires no hook.
`terminal_hook` mapped `Restart` / `GracefulRestart` to `RevertIntent`, but
a restart never writes `wanted` — `restart_chain` deliberately has no
`SetWanted` head, so the tail `Reconcile` converges to the agent's existing
intent and a deliberately-stopped agent isn't forced up by a hive-wide
restart.
`revert_intent` writes `Wanted::from_running(observed)` unconditionally on a
cancelled DAG. So for an agent that is `wanted = Up` but currently down
(crashed, or caught behind another queued op), submitting a restart and then
cancelling it writes `wanted = Offline` — reverting an intent the DAG never
touched, to a value nobody asked for. Reconcile and crash-watch both then
read the agent as deliberately-stopped and leave it down.
It's invisible for a running agent, since `from_running(true)` equals the
intent already on file, which is why it went unnoticed. `cancel` only
succeeds while every node is still `Pending`, so the reachable window is
exactly "queued restart + observed != intent" — precisely when someone
restarts and then thinks better of it.
Drop both restart templates from the `RevertIntent` arm; they fall through
to no terminal hook, which is correct for a DAG that writes no intent.
Document the invariant on `HookKind::RevertIntent` and on `revert_intent`
itself: the hook writes *observed* state, so dispatching it for a template
with no `SetWanted` head doesn't restore an intent, it invents one.
Test covers all four restart shapes (graceful x running), asserting both
that the spec carries no `SetWanted` and that a cancelled restart dispatches
no hook, with a contrast arm pinning stop's revert in place.
Fixeshyperhive/hyperhive#2710
The `step` label was taken off the wire in #2661, when each deploy phase
became a first-class DAG node. Since then it has been written but never
read: `NodeRuntime` derives only `Debug, Default, Clone` — no serde — so
the field could not reach any client, and the only reads of it were the
dedup checks inside its own setters. This deletes the machinery.
Removed:
- `NodeRuntime.step`, `set_step`, `set_step_running`, and the
`rt.step = None` clear in `complete_node`. `NodeRuntime` keeps its
remaining `build_log_id` field (deliberately still a struct — collapsing
it to a bare `Option<i64>` would churn every call site for no gain).
- `Ctx::step` and its ~15 call sites in `job_queue/exec.rs`. `Ctx` itself
stays: it is the build-log sink, which `run_prebuild` and `run_swap`
still use.
- `Coordinator::set_queue_step` and its 11 callers in `actions.rs`.
- `JobQueue::running_node_of`, reachable only from `set_queue_step`.
- `swap_update`'s `on_step` parameter and its one body call.
- The `set_step_only_on_running_and_signals_change` test.
Dropping the calls orphaned parameters, which are removed with their call
sites: `ctx` on ten executors that used it only as a step sink, and
`queue_entry_id` on `run_deploy_merge_verify` / `run_deploy_apply` /
`run_finalize_deploy` plus both `coord` and `queue_entry_id` on
`prepare_applied_target`. `run_deploy_tail` KEEPS its `queue_entry_id` —
that one has a genuine surviving use (the build-log link in the failure
comment posted to the PR).
One behavioural change, called out so it is not mistaken for a dropped
dashboard refresh: `Ctx::step` and `set_queue_step` each emitted a
`rebuild_queue_changed` snapshot when the label changed, and those
emissions go away with them. This is safe — the snapshot payload has no
step field, so those pushes carried nothing a client could observe. Real
state transitions still emit from the scheduler's claim and completion
paths, from `submit`, and from the three `actions.rs` sites. Net effect is
strictly fewer redundant SSE pushes.
Docs: `docs/coordinator.md` still listed `step` as a `NodeView` wire field
and `docs/web-ui/dashboard.md` documented a cyan `↳ <step>` sub-line under
each queue row. Neither has existed since #2661 — both corrected here, plus
the `job_queue/model.rs` module doc.
Not touched: `frontend/packages/dashboard/src/system-sections.css` has a
dead `.rqe-step` rule with no JS referencing it. Left for the frontend
owner rather than deleted here.
Closes: #2664
Add NodeView::parent to the wire (hive-sh4re + hive-c0re dag_view), then
render the recursive parent/child tree in the dashboard build queue instead
of the previous flat chain/fan-out layout.
Wire change (hive-sh4re, hive-c0re):
- NodeView gains parent: Option<NodeId> (skip_serializing_if = None)
- dag_view() projects node.parent, filtering out the Dag container id
(top-level work nodes become parent: None on the wire)
Frontend (builds.js):
- Replace nodeComponents + splitFanOut with buildNodeTree (uses parent
edges directly) + topoSort helper (orders siblings by deps)
- renderTreeNode walks the tree depth-first, rendering indented rows
with └─/├─ connectors and agent label per chip
- Flat chains and fan-out heuristics are gone; structure comes straight
from the scheduler's parent axis
Closes: none (parent issue tracked in forge)
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.
Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.
The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.
`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.
The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.
Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
Non-derivable per-node payload rides the node that owns it. Tagging all
four deploy nodes with the approval id would render the same card four
times in `dag_view`.
Also fix `set_queue_step`'s doc comment, which claimed the DAG-id lookup
was exact because approval DAGs are single-node. They are not anymore;
what actually holds is that the chain is strictly sequential with the
root parked in Finishing, so at most one node is ever Running.
The raw-graph wire drops the inline build_log_id; the client fetches a
node's captured output on demand. Two handlers resolve node id -> log-row
id (JobQueue::build_log_id_of, now keyed by the wire u64) then delegate to
the existing get_full / raw handlers: /api/build-log/<node_id> serves the
BuildLogFull JSON ({stdout, stderr} + header), /raw serves text/plain.
404 when the node has no linked log.
Reconcile with mara + argus's review on the frontend PR (#2660):
- DagView regains started_at/finished_at (DateTime<Utc>), computed
host-side as min/max over ALL subtree nodes (including the Done ones
filtered off the wire). The client can't derive these — the
earliest/only-started node is often Done and absent — so the backend
sets them, per mara's call.
- NodeView gains has_log: bool = build_log_id.is_some(), the precise
old 'node has a captured build log' guard so the dashboard only shows
a log link for nodes that actually produce one.
- server::await_dags: a DAG is settled when gone from the snapshot (fully
Done) or present with all nodes terminal; pending only with a non-terminal
node (DagView no longer carries a rolled-up state).
- DagView::rollup_state() added to hive-sh4re — the shared node-set roll-up
derivation every Rust consumer uses.
- JobQueue::build_log_id_of(node_id) — the node_id -> build_logs lookup the
query endpoint will use; tests assert log-id via it now.
- tests: derive roll-up state; drop the off-wire step/build_log_id wire asserts.
hive-c0re side of the raw-graph wire: dag_view now projects the slim
DagView, reading started_at/finished_at/error straight off the
hive_jobq Node (removes the node_rt double-write from #2645), excludes
Done nodes, and rides approval_id/inputs on the owning node. Template
moved into hive-c0re (model.rs) — no longer on the wire. Still WIP:
build-log endpoint + hivectl derive + compile fixes to follow.
Node gains started_at/finished_at (chrono DateTime<Utc>, serialized
RFC 3339 on the wire per hive_sh4re::wire_time) plus error (String).
Graph::set_state self-stamps started_at on the first Running transition
and finished_at on the first terminal one, via an internal now_utc()
clock (keeps settle/complete signatures stable). Outcome::Failed(String)
carries the failure reason, set on the terminal transition.
hive-c0re complete_node builds Outcome::Failed(msg); its node_rt
side-table stays i64 for now (double-write) until #2637 reads the Node.
Toward #2637: the jobq graph becomes the source of truth for per-node
lifecycle so the queue can be sent to the client as-is.
Replace the in-tree scheduler with the domain-agnostic hive-jobq crate
(merged in #2615): parent-axis grouping + borrow/subtree-reservation
resource model + roll-up completion (State::Finishing).
Host adaptation:
- NodeSpec gains an explicit `parent` axis; templates declare grouping +
sibling ordering directly (deps order execution, parent groups a subtree
whose resource the descendants borrow).
- Rebuild is a nested two-root subtree: Prebuild (root, owns the build slot
for the whole subtree, lease-exempt) -> StopForUpdate (child, owns the
agent lease) -> Swap/PostSwap (children, borrow both); Reconcile is a
separate top-level root (AfterAny Prebuild) so it survives the cancel-
cascade of any failed step (recovery-start invariant) and converges to
the persisted `wanted` on a fresh lease. This is the multi-root
correction to the single-root-chain sketch: node0=root broke lease-
exemption (hoisting the lease onto Prebuild) and recovery-reconcile
(root failure cancels all children).
- Spawn / perm-change / power-ops (stop/start/restart) group-rooted the
same way; per-agent power-op subgraphs stay independent roots so a
multi-agent DAG runs them concurrently, each on its own lease.
- insert_group honours the explicit parent axis (no lease hoisting); the
DAG terminal node deps AfterAny on every group root and runs once the
whole op rolls up. Drop the old Graph::add_dep terminal wiring.
36/36 job_queue tests, full hive-c0re suite green, clippy --all-targets.
A per-agent lifecycle lease gates that agent's container globally across
concurrent DAGs, so it should be held for exactly as long as the agent's
work in a DAG is in flight, no longer. settle() previously freed every
lease a DAG held only at whole-DAG terminal, so a multi-agent DAG (a
hive-wide restart) kept agent A's container locked until B and C also
finished, blocking any other DAG wanting A.
Now free each agent's lease the moment its own subgraph within the DAG is
terminal (no live node still targets it), and drop that agent's dashboard
transient pill on the same edge via a new per-agent release channel. A
single-agent DAG is unaffected: its agent's subgraph goes terminal exactly
when the whole DAG does, so behaviour is identical.
append_subgraph is the multi-node/multi-agent generalisation of the
single-node append_node, so the two in-DAG-growth channels collapse to
one: the Reconcile planner now emits its mechanical Start/Stop as a
single-node append_subgraph rooted on the reconcile node (stamping
claim.agent on the NodeSpec, which append_node inherited implicitly).
Removes NodeOutput.append_nodes + its scheduler drain loop and
JobQueue::append_node. No behaviour change — a channel unification.
