feat(#2788): carry Skipped to the wire as its own state

A node ruled out by its own dependency edges settles `Skipped` host-side,
but the wire folded it into `Cancelled` and `dag_view` filtered it out
entirely, so a client never saw which branch a run didn't take. Post-#2785
that is not a rare shape: every approval DAG has two not-taken tails and
every rebuild has one, on the happy path as much as on failure.

`State` gains `Skipped`, and it counts as terminal — the wait loops in
hivectl's progress display and the daemon's dag-settled check decide
"finished" with `all(is_terminal)`, so omitting it would hang them on
essentially every DAG.

`dag_view` now emits skipped nodes but no longer lets them keep a DAG
alive. Serialization and completion were the same expression: a DAG left
the snapshot because its nodes had all been filtered away. Keeping skipped
nodes on the wire under that rule would pin every finished deploy in the
queue view forever, so the completion test is now its own flag.

`rollupState` in the dashboard gains the matching arm. It has no `done`
case — `done` is inferred by falling off the wire — so its trailing
`return 'queued'` catches anything it doesn't recognise, and a green
deploy would have read as permanently queued the moment the backend
started emitting the new state. The Rust and JS roll-ups have silently
disagreed before; they are edited together here and say so.
This commit is contained in:
atlas 2026-07-27 19:24:26 +02:00
commit 657d1b5061
5 changed files with 88 additions and 36 deletions

View file

@ -135,19 +135,14 @@ impl Default for JobQueue {
/// Map a crate node state onto the wire state (`Pending` ↔ `Queued`;
/// `Finishing` — own logic done, sub-nodes still running — reads as `Running`).
///
/// `Skipped` has no wire counterpart and folds into `Cancelled`: to a reader
/// both mean "this never ran". The distinction is a *scheduling* one — it
/// decides whether a parent's roll-up counts the node — and the wire carries no
/// roll-up input, only display state. In practice a client never sees it either:
/// `dag_view` drops skipped nodes from the snapshot along with `Done` ones.
fn to_wire_state(state: JobState) -> State {
match state {
JobState::Pending => State::Queued,
JobState::Running | JobState::Finishing => State::Running,
JobState::Done => State::Done,
JobState::Failed => State::Failed,
JobState::Cancelled | JobState::Skipped => State::Cancelled,
JobState::Cancelled => State::Cancelled,
JobState::Skipped => State::Skipped,
}
}
@ -599,11 +594,17 @@ impl QueueInner {
/// off each `hive_jobq::Node`; the client derives the DAG label, roll-up
/// state, and DAG timestamps from the node set. Non-derivable per-node
/// payload (`approval_id`, meta `inputs`) rides the owning node. Returns
/// `None` when every work node is `Done` — a fully-completed DAG drops
/// out of the snapshot entirely (a `Failed` one lingers until aged out).
/// `None` when every work node is `Done` or `Skipped` — a fully-settled
/// DAG drops out of the snapshot entirely (a `Failed` one lingers until
/// aged out).
fn dag_view(&self, container: NodeId) -> Option<DagView> {
let meta = self.dag_meta(container)?;
let mut nodes = Vec::new();
// Whether anything in this DAG still has an outcome worth showing.
// Kept separate from `nodes` being non-empty: skipped nodes ride the
// wire so the dashboard can mark the branches that weren't taken, but
// they must not by themselves hold a finished DAG in the snapshot.
let mut any_unsettled = false;
// DAG-level timestamps are taken over *all* subtree nodes (including the
// `Done` ones excluded from the wire) — the client can't derive them
// from a `Done`-filtered node set, so the host computes them here.
@ -619,13 +620,13 @@ impl QueueInner {
if let Some(f) = node.finished_at {
finished.push(f);
}
// `Done` nodes drop off the wire (a finished step isn't interesting),
// and so do `Skipped` ones: a branch that was never taken is noise on
// the dashboard, and surfacing it would also drag the client-side
// roll-up toward `Cancelled` for a run that went fine.
if matches!(node.state, JobState::Done | JobState::Skipped) {
// `Done` nodes drop off the wire — a finished step isn't
// interesting. `Skipped` ones stay: which branch a run *didn't*
// take is the readable half of an outcome-branched DAG.
if matches!(node.state, JobState::Done) {
continue;
}
any_unsettled |= !matches!(node.state, JobState::Skipped);
let deps: Vec<u64> = node
.deps
.iter()
@ -671,7 +672,7 @@ impl QueueInner {
parent,
});
}
if nodes.is_empty() {
if !any_unsettled {
return None;
}
let is_terminal = self.dag_is_terminal(container);

View file

@ -83,8 +83,8 @@ fn settle_rebuild_tail(q: &JobQueue, dag_id: u64, agent: &str, expect_ok: bool)
}
fn state_of(q: &JobQueue, dag_id: u64) -> State {
// A fully-`Done` DAG drops out of the snapshot (its nodes are all
// excluded) — absence is the completion signal, so map it to `Done`.
// A DAG whose nodes have all settled `Done` or `Skipped` drops out of the
// snapshot — absence is the completion signal, so map it to `Done`.
// Otherwise derive the roll-up from the node set, exactly as every wire
// consumer does.
q.snapshot()
@ -310,6 +310,27 @@ fn non_graceful_rebuild_has_no_signal_or_drain() {
);
}
/// A cleanly-finished DAG leaves the snapshot even though its not-taken
/// failure branch is still in the graph as `Skipped`. Skipped nodes ride the
/// wire so the dashboard can mark them, which makes "the node list is empty"
/// and "nothing here is still worth showing" two different questions — only
/// the second one may drop the DAG. Conflating them pins every completed
/// deploy in the queue view forever.
#[test]
fn settled_dag_leaves_the_snapshot_despite_its_skipped_branch() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
for _ in 0..6 {
let c = claim_one(&q);
q.complete_node(id, c.node_id, Ok(()));
}
settle_rebuild_tail(&q, id, "agent-a", true);
assert!(
q.snapshot().iter().all(|d| d.id != id),
"a fully settled DAG drops out of the snapshot"
);
}
// ---- build slots ----
#[test]
@ -858,16 +879,14 @@ fn failed_node_cancels_downstream_but_afterany_reconcile_runs() {
};
assert_eq!(by_kind("prebuild"), State::Failed);
// `StopForUpdate` / `Swap` / `PostSwap` were *ruled out* by the failed
// `Prebuild` — `Skipped`, and skipped nodes are filtered off the wire along
// with `Done` ones. The failure itself is still visible (the `prebuild` row
// above, and the roll-up), which is the part an operator acts on.
// Restoring that detail wants a real `Skipped` wire state the client renders
// as "not run" — surfacing them as `Cancelled` instead would make a
// *successful* DAG with a not-taken branch read as cancelled.
// `Prebuild`. They ride the wire as `Skipped` so an operator can see which
// steps the run never reached, without them reading as failures of their
// own — the roll-up ignores `Skipped` entirely.
for ruled_out in ["stop_for_update", "swap", "post_swap"] {
assert!(
dag.nodes.iter().all(|n| n.kind != ruled_out),
"{ruled_out} was ruled out, so it is off the wire"
assert_eq!(
by_kind(ruled_out),
State::Skipped,
"{ruled_out} was ruled out, so it is on the wire as skipped"
);
}
// The AfterAny reconcile ran (claimed + completed Ok above) → it's `Done`,
@ -908,9 +927,14 @@ fn swap_failure_still_runs_reconcile() {
q.complete_node(id, reconcile.node_id, Ok(()));
let all_dags = q.snapshot();
let dag = all_dags.iter().find(|d| d.id == id).expect("dag");
assert!(
dag.nodes.iter().all(|n| n.kind != "post_swap"),
"PostSwap is ruled out by the failed Swap (`Skipped`, so off the wire)"
assert_eq!(
dag.nodes
.iter()
.find(|n| n.kind == "post_swap")
.expect("post_swap node")
.state,
State::Skipped,
"PostSwap is ruled out by the failed Swap, and says so on the wire"
);
assert_eq!(
dag.nodes