refactor(#2756): replace the DAG terminal hook with real tail nodes
The queue carried a per-DAG `HookKind` that fired an inline side effect
from outside the graph when a container rolled up terminal. mara asked
three times why this could not be an ordinary node; the answer in the
code was a doc-comment claiming a node could not work, and it was wrong.
`DepWhen::AfterAny` already existed with two live users, and a weak edge
is satisfied by a `Cancelled` dep, so a tail node runs on success,
failure and cancel alike. What was genuinely missing was smaller than a
hook: a node had no way to learn how the work it followed ended.
So: `Claim` now carries `deps: Vec<DepOutcome>`, snapshotted at claim
time from the graph the scheduler already holds (no `hive-jobq` change).
`Claim::deps_state()` / `deps_error()` roll that up, and two new kinds
consume it — `ResolveApproval { approval_id }` and `EmitRebuilt { agent }`.
Templates append one as a group-root with `AfterAny` edges onto the DAG's
other group roots; a root's state is its subtree's roll-up, so that
covers every node without fanning out to each of them.
Deleted: `HookKind`, `DagSpec.hook`, `NodeKind::Dag.hook`, `DagMeta.hook`,
`TerminalDag`, `terminal_dag()`, `terminal_summary()`, `dag_agents()`,
`dag_rollup()`, `fire_terminal_hook()`, `run_terminal_hook()`,
`emit_rebuilt()`. `complete_node` returns `()`.
Load-bearing details:
- `JobQueue::cancel` spares tail nodes instead of cancelling the whole
subtree, and returns `bool`. Without this a cancelled approval DAG
would dangle its approval forever — the hazard `tests.rs` already
named. The spared tail's deps are `Cancelled`, which satisfies its weak
edge, so the scheduler claims it and it resolves the row as cancelled.
`hive-jobq` anticipated exactly this: `cancel_node`'s doc already says
to settle afterwards so "a weak-edge terminal node observing the
cancellation" can advance.
- The existing `complete(container)` call after cancelling is kept and is
deliberately a no-op when a tail was spared (a non-terminal child parks
the container back in `Finishing`), so power ops still settle
synchronously with no branch.
- `DeployTail` is NOT `is_tail()`: it does real compensating work, and a
cancelled DAG has nothing to compensate.
- `exec::failure_reason` falls back to `first_error(dag_id)` because a
group root that rolled up `Failed` from a child carries no error of its
own — without it every tail-reported failure would lose its reason.
- `EmitRebuilt` is per agent, so a multi-agent DAG reports each agent's
own outcome rather than painting all of them with the DAG roll-up.
- `ResolveApproval` is agentless: the approval row already names its
agent, and that is also what lets one tail close a multi-agent DAG.
Transients-derived-from-running-nodes and the frontend's node-kind
strings stay out of this change; they touch iris's slice and review
better next to their own diff.
This commit is contained in:
parent
896dfc6194
commit
e8e6998ac5
11 changed files with 521 additions and 324 deletions
|
|
@ -48,6 +48,39 @@ fn claim_one(q: &JobQueue) -> Claim {
|
|||
claims.pop().expect("one claim")
|
||||
}
|
||||
|
||||
/// Claim an approval DAG's `ResolveApproval` tail, assert which approval it
|
||||
/// carries and what it will report to that row, then complete it. Replaces the
|
||||
/// old `terminal_summary()` assertions: the outcome is no longer a struct handed
|
||||
/// to a hook, it's what this node reads off its own deps.
|
||||
fn settle_approval_tail(q: &JobQueue, dag_id: u64, approval_id: i64, expect: State) {
|
||||
let tail = claim_one(q);
|
||||
assert!(
|
||||
matches!(tail.kind, NodeKind::ResolveApproval { approval_id: got } if got == approval_id),
|
||||
"expected the ResolveApproval tail for #{approval_id}, got {:?}",
|
||||
tail.kind
|
||||
);
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
expect,
|
||||
"outcome the tail reports to approval #{approval_id}"
|
||||
);
|
||||
q.complete_node(dag_id, tail.node_id, Ok(()));
|
||||
}
|
||||
|
||||
/// The `EmitRebuilt` counterpart of [`settle_approval_tail`] — claim a rebuild /
|
||||
/// perm-change DAG's tail, assert the `Rebuilt` event it will emit, complete it.
|
||||
fn settle_rebuild_tail(q: &JobQueue, dag_id: u64, agent: &str, expect: State) {
|
||||
let tail = claim_one(q);
|
||||
assert_eq!(tail.kind.as_str(), "emit_rebuilt");
|
||||
assert_eq!(tail.agent, agent, "`Rebuilt` is emitted per agent");
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
expect,
|
||||
"ok-ness of the emitted `Rebuilt`"
|
||||
);
|
||||
q.complete_node(dag_id, tail.node_id, Ok(()));
|
||||
}
|
||||
|
||||
fn state_of(q: &JobQueue, dag_id: u64) -> State {
|
||||
// A fully-`Done` DAG drops out of the snapshot (its nodes are all
|
||||
// excluded) — absence is the completion signal, so map it to `Done`.
|
||||
|
|
@ -196,6 +229,7 @@ fn rebuild_chain_claims_in_dep_order() {
|
|||
);
|
||||
q.complete_node(id, c.node_id, Ok(()));
|
||||
}
|
||||
settle_rebuild_tail(&q, id, "agent-a", State::Done);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
@ -315,8 +349,8 @@ fn lease_serializes_two_lifecycle_dags_for_same_agent() {
|
|||
assert_eq!(second.kind.as_str(), "reconcile");
|
||||
q.complete_node(restart, second.node_id, Ok(()));
|
||||
// Restart's work is terminal → its lease releases, so stop's now-unblocked
|
||||
// Reconcile becomes ready (restart's inline hook fired off the returned
|
||||
// summary — no terminal-hook node).
|
||||
// Reconcile becomes ready (a power op has no tail node, so nothing of
|
||||
// restart's remains claimable).
|
||||
let third = claim_one(&q);
|
||||
assert_eq!(third.dag_id, stop);
|
||||
assert_eq!(third.kind.as_str(), "reconcile");
|
||||
|
|
@ -369,8 +403,8 @@ fn lease_exempt_prebuild_overlaps_other_dag_on_same_agent() {
|
|||
.clone();
|
||||
q.complete_node(stop, reconcile.node_id, Ok(()));
|
||||
// stop's Reconcile done → its lease frees, so rebuild's StopForUpdate
|
||||
// unblocks. (stop's DAG rolls up terminal; its inline hook fires off the
|
||||
// returned summary — no terminal-hook node in the claim set.)
|
||||
// unblocks. (stop's DAG rolls up terminal; a power op has no tail node, so
|
||||
// nothing of stop's is left in the claim set.)
|
||||
let after = q.claim_ready();
|
||||
let sfu = after
|
||||
.iter()
|
||||
|
|
@ -591,7 +625,6 @@ fn append_subgraph_roots_on_emitter_and_rebases_local_deps() {
|
|||
// the emitter and its LOCAL 0-based deps are rebased onto the DAG.
|
||||
let q = JobQueue::new(4);
|
||||
let spec = DagSpec {
|
||||
hook: None,
|
||||
source: Source::AutoUpdate,
|
||||
reason: "sweep".to_owned(),
|
||||
approval_id: None,
|
||||
|
|
@ -703,6 +736,73 @@ fn meta_update_carries_rebuilding_transient_and_grows_cascade_in_dag() {
|
|||
);
|
||||
}
|
||||
|
||||
// ---- dep outcomes on the claim ----
|
||||
|
||||
/// Every claimed node reports how each node it depends on finished, and those
|
||||
/// deps are always already terminal — that is what a node's edges being
|
||||
/// satisfied *means*. A tail node reads its `deps` instead of going back to the
|
||||
/// world to find out how the work below it went.
|
||||
#[test]
|
||||
fn claim_carries_terminal_dep_outcomes() {
|
||||
let q = JobQueue::new(1);
|
||||
let id = submit(&q, rebuild("agent-a", "r"));
|
||||
let mut saw_a_dep = false;
|
||||
loop {
|
||||
let mut claims = q.claim_ready();
|
||||
let Some(claim) = claims.pop() else { break };
|
||||
assert!(claims.is_empty(), "one build slot ⇒ one claim at a time");
|
||||
for dep in &claim.deps {
|
||||
saw_a_dep = true;
|
||||
assert!(
|
||||
dep.state.is_terminal(),
|
||||
"{} was claimed with a non-terminal dep ({:?}) — a node's edges \
|
||||
being satisfied is exactly the claim that its deps have finished",
|
||||
claim.kind.as_str(),
|
||||
dep.state
|
||||
);
|
||||
assert_eq!(
|
||||
dep.state,
|
||||
State::Done,
|
||||
"on the happy path every dep of {} finished Done",
|
||||
claim.kind.as_str()
|
||||
);
|
||||
assert_eq!(dep.error, None, "a Done dep carries no error");
|
||||
}
|
||||
q.complete_node(id, claim.node_id, Ok(()));
|
||||
}
|
||||
assert!(saw_a_dep, "the rebuild DAG has at least one dependent node");
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
/// The failure direction, which is the whole point of carrying outcomes at all:
|
||||
/// `Reconcile` hangs off `Prebuild` with `AfterAny`, so a failed prebuild
|
||||
/// cancel-cascades `StopForUpdate`/`Swap`/`PostSwap` and `Reconcile` still runs
|
||||
/// — and its claim hands it the failure, including the reason, rather than
|
||||
/// leaving the executor to go and re-derive it from the world.
|
||||
#[test]
|
||||
fn claim_dep_outcome_reports_a_failed_dep_with_its_error() {
|
||||
let q = JobQueue::new(1);
|
||||
let id = submit(&q, rebuild("agent-a", "r"));
|
||||
let meta_sync = claim_one(&q);
|
||||
q.complete_node(id, meta_sync.node_id, Ok(()));
|
||||
let prebuild = claim_one(&q);
|
||||
assert_eq!(prebuild.kind.as_str(), "prebuild");
|
||||
q.complete_node(id, prebuild.node_id, Err("nix build exploded".to_owned()));
|
||||
let reconcile = claim_one(&q);
|
||||
assert_eq!(reconcile.kind.as_str(), "reconcile");
|
||||
assert_eq!(
|
||||
reconcile
|
||||
.deps
|
||||
.iter()
|
||||
.map(|d| (d.state, d.error.as_deref()))
|
||||
.collect::<Vec<_>>(),
|
||||
vec![(State::Failed, Some("nix build exploded"))],
|
||||
"reconcile's claim carries the failed prebuild and its reason"
|
||||
);
|
||||
assert_eq!(reconcile.deps_state(), State::Failed);
|
||||
assert_eq!(reconcile.deps_error(), Some("nix build exploded"));
|
||||
}
|
||||
|
||||
// ---- failure: cancel-downstream + AfterAny ----
|
||||
|
||||
#[test]
|
||||
|
|
@ -811,6 +911,7 @@ fn swap_ok_runs_post_swap_before_reconcile() {
|
|||
let reconcile = claim_one(&q);
|
||||
assert_eq!(reconcile.kind.as_str(), "reconcile");
|
||||
q.complete_node(id, reconcile.node_id, Ok(()));
|
||||
settle_rebuild_tail(&q, id, "agent-a", State::Done);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
@ -832,10 +933,17 @@ fn failed_reconcile_marks_dag_failed() {
|
|||
fn cancel_clears_queued_dag() {
|
||||
let q = JobQueue::new(1);
|
||||
let id = submit(&q, rebuild("agent-a", "r"));
|
||||
// Cancel returns the terminal summary (state `Cancelled`) — the inline hook
|
||||
// fires off it at the caller; there's no terminal-hook node to claim.
|
||||
let terminal = q.cancel(id).expect("cancelled");
|
||||
assert_eq!(terminal.state, State::Cancelled);
|
||||
assert!(q.cancel(id), "fully-queued dag cancels");
|
||||
// Every work node is `Cancelled`, but the tail is spared so it can still
|
||||
// report the cancellation — so it is the one thing left to claim.
|
||||
let tail = claim_one(&q);
|
||||
assert_eq!(tail.kind.as_str(), "emit_rebuilt");
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
State::Cancelled,
|
||||
"the spared tail sees its deps cancelled, so it emits no `Rebuilt`"
|
||||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
assert_eq!(state_of(&q, id), State::Cancelled);
|
||||
assert!(q.claim_ready().is_empty());
|
||||
}
|
||||
|
|
@ -845,23 +953,24 @@ fn cancel_refuses_running_dag() {
|
|||
let q = JobQueue::new(1);
|
||||
let id = submit(&q, rebuild("agent-a", "r"));
|
||||
let _ = claim_one(&q);
|
||||
assert!(q.cancel(id).is_none());
|
||||
assert!(!q.cancel(id));
|
||||
assert_eq!(state_of(&q, id), State::Running);
|
||||
}
|
||||
|
||||
/// A cancelled power op must fire **no** compensating hook — not even one that
|
||||
/// A cancelled power op must run **no** compensating node — not even one that
|
||||
/// carries a `SetWanted` head.
|
||||
///
|
||||
/// `cancel` refuses unless every work node is still `Pending`
|
||||
/// (`cancel_refuses_running_dag`) and a cancel *cascade* rolls up `Failed`
|
||||
/// rather than `Cancelled`, so a `Cancelled` DAG provably never executed a
|
||||
/// node: its `SetWanted` never ran and the agent's intent still reads whatever
|
||||
/// Now structural rather than a property of a hook enum: a power op emits no
|
||||
/// tail node at all, so once its work nodes cancel there is simply nothing left
|
||||
/// to claim. `cancel` also refuses unless every work node is still `Pending`
|
||||
/// (`cancel_refuses_running_dag`), so a `Cancelled` DAG provably never executed
|
||||
/// a node: its `SetWanted` never ran and the agent's intent still reads whatever
|
||||
/// the operator last set. A "revert" instead writes the agent's *observed*
|
||||
/// state, which for a down-but-`wanted = Up` agent (crashed, or caught
|
||||
/// mid-bounce) flips the intent to `Offline` and leaves it
|
||||
/// deliberately-stopped as far as reconcile and crash-watch are concerned.
|
||||
#[test]
|
||||
fn cancelled_power_op_fires_no_hook() {
|
||||
fn cancelled_power_op_runs_no_compensating_node() {
|
||||
for graceful in [false, true] {
|
||||
for running in [false, true] {
|
||||
let targets = vec![("agent-a".to_owned(), running)];
|
||||
|
|
@ -896,12 +1005,12 @@ fn cancelled_power_op_fires_no_hook() {
|
|||
);
|
||||
let q = JobQueue::new(1);
|
||||
let id = submit(&q, spec);
|
||||
let summary = q.cancel(id).expect("cancelled while queued");
|
||||
assert_eq!(summary.state, State::Cancelled);
|
||||
assert_eq!(
|
||||
summary.hook, None,
|
||||
assert!(q.cancel(id), "cancelled while queued");
|
||||
assert_eq!(state_of(&q, id), State::Cancelled);
|
||||
assert!(
|
||||
q.claim_ready().is_empty(),
|
||||
"cancelled {name} (graceful={graceful}, running={running}) must \
|
||||
fire no hook — no node of it ever ran"
|
||||
leave nothing to run — a power op emits no tail node"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -920,13 +1029,10 @@ fn dag_settles_terminal_and_releases_lease_after_work() {
|
|||
q.complete_node(id, stop.node_id, Ok(()));
|
||||
let rec = claim_one(&q);
|
||||
assert_eq!(rec.kind.as_str(), "reconcile");
|
||||
// Completing the last work node rolls the container up terminal and returns
|
||||
// the summary the inline hook consumes — there is no terminal-hook node.
|
||||
let summary = q
|
||||
.complete_node(id, rec.node_id, Ok(()))
|
||||
.expect("terminal summary");
|
||||
assert_eq!(summary.state, State::Done);
|
||||
assert!(q.claim_ready().is_empty(), "no terminal-hook node to claim");
|
||||
// Completing the last work node rolls the container up terminal. A power op
|
||||
// has no tail node, so nothing is left to claim.
|
||||
q.complete_node(id, rec.node_id, Ok(()));
|
||||
assert!(q.claim_ready().is_empty(), "no tail node to claim");
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
// Lease released when the work chain settled: a new DAG for the agent claims
|
||||
// immediately.
|
||||
|
|
@ -938,31 +1044,44 @@ fn dag_settles_terminal_and_releases_lease_after_work() {
|
|||
assert_eq!(c.dag_id, next);
|
||||
}
|
||||
|
||||
/// A DAG cancelled while fully queued must still surface a terminal
|
||||
/// roll-up for the scheduler's hooks — otherwise a queued approval
|
||||
/// DAG cancelled by the operator would dangle its approval forever.
|
||||
/// A DAG cancelled while fully queued must still **run its tail**, or a queued
|
||||
/// approval DAG cancelled by the operator would dangle its approval forever.
|
||||
///
|
||||
/// This is the load-bearing case for sparing tails in [`JobQueue::cancel`]: the
|
||||
/// work nodes all cancel, but `ResolveApproval` is weak-edged, so a `Cancelled`
|
||||
/// dep satisfies its edge and it becomes claimable instead of being cancelled
|
||||
/// along with everything else. It reads `Cancelled` off its own deps and resolves
|
||||
/// the approval as "cancelled before completion".
|
||||
#[test]
|
||||
fn cancelled_dag_finalizes_with_terminal_rollup() {
|
||||
fn cancelled_dag_still_runs_its_approval_tail() {
|
||||
let q = JobQueue::new(1);
|
||||
let id = submit(
|
||||
&q,
|
||||
templates::approval_deploy("agent-a", 7, "approval #7".to_owned()),
|
||||
);
|
||||
// Cancel rolls the DAG up terminal and returns its summary — the inline hook
|
||||
// (approval resolution) runs off it at the caller. Cancelled + approval id 7.
|
||||
let summary = q.cancel(id).expect("cancelled");
|
||||
assert_eq!(summary.state, State::Cancelled);
|
||||
assert_eq!(summary.approval_id, Some(7));
|
||||
// The cancelled DAG's summary stays available (until history-trimmed) and
|
||||
// unrelated later activity doesn't disturb it.
|
||||
assert!(q.cancel(id), "fully-queued dag cancels");
|
||||
let tail = claim_one(&q);
|
||||
assert_eq!(tail.dag_id, id);
|
||||
assert_eq!(tail.kind.as_str(), "resolve_approval");
|
||||
assert!(
|
||||
matches!(tail.kind, NodeKind::ResolveApproval { approval_id: 7 }),
|
||||
"the tail carries the approval to resolve, got {:?}",
|
||||
tail.kind
|
||||
);
|
||||
assert_eq!(
|
||||
tail.deps_state(),
|
||||
State::Cancelled,
|
||||
"so the executor resolves the approval as cancelled, not as a failure"
|
||||
);
|
||||
assert_eq!(tail.deps_error(), None, "a cancelled dep carries no error");
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
assert_eq!(state_of(&q, id), State::Cancelled);
|
||||
// Unrelated later activity doesn't disturb the settled DAG.
|
||||
let other = submit(&q, rebuild("agent-b", "r"));
|
||||
let c = claim_one(&q);
|
||||
assert_eq!(c.dag_id, other);
|
||||
q.complete_node(other, c.node_id, Err("boom".to_owned()));
|
||||
assert_eq!(
|
||||
q.terminal_summary(id).map(|t| t.state),
|
||||
Some(State::Cancelled)
|
||||
);
|
||||
assert_eq!(state_of(&q, id), State::Cancelled);
|
||||
}
|
||||
|
||||
// ---- approval deploy subtree ----
|
||||
|
|
@ -1006,13 +1125,12 @@ fn deploy_dag_runs_phases_in_order_and_tails_a_failed_apply() {
|
|||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
let summary = q.terminal_summary(id).expect("dag terminal");
|
||||
settle_approval_tail(&q, id, 7, State::Failed);
|
||||
assert_eq!(
|
||||
summary.state,
|
||||
state_of(&q, id),
|
||||
State::Failed,
|
||||
"an Ok tail must not launder a failed deploy into a success"
|
||||
);
|
||||
assert_eq!(summary.approval_id, Some(7));
|
||||
}
|
||||
|
||||
/// The deploy's happy path: `DeployApply` does not build. It grows the ordinary
|
||||
|
|
@ -1080,9 +1198,8 @@ fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
|
|||
assert!(matches!(tail.kind, NodeKind::DeployTail { .. }));
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
let summary = q.terminal_summary(id).expect("dag terminal");
|
||||
assert_eq!(summary.state, State::Done);
|
||||
assert_eq!(summary.approval_id, Some(11));
|
||||
settle_approval_tail(&q, id, 11, State::Done);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
/// A failure *inside* the grafted rebuild is the failure mode the subgraph
|
||||
|
|
@ -1132,12 +1249,14 @@ fn deploy_dag_skips_finalize_but_still_tails_a_failed_graft() {
|
|||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
let summary = q.terminal_summary(id).expect("dag terminal");
|
||||
assert_eq!(summary.state, State::Failed);
|
||||
settle_approval_tail(&q, id, 13, State::Failed);
|
||||
assert_eq!(state_of(&q, id), State::Failed);
|
||||
assert_eq!(
|
||||
q.first_error(id).as_deref(),
|
||||
Some("profile swap failed"),
|
||||
"the tail annotates failed/<id> with this"
|
||||
"the tail annotates failed/<id> with this — and it is also what
|
||||
`exec::failure_reason` falls back to, since the tail's own dep is a
|
||||
group root that rolled up Failed and so carries no error itself"
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -1168,9 +1287,8 @@ fn deploy_dag_skips_apply_but_still_runs_tail_when_verify_fails() {
|
|||
);
|
||||
q.complete_node(id, tail.node_id, Ok(()));
|
||||
|
||||
let summary = q.terminal_summary(id).expect("dag terminal");
|
||||
assert_eq!(summary.state, State::Failed);
|
||||
assert_eq!(summary.approval_id, Some(9));
|
||||
settle_approval_tail(&q, id, 9, State::Failed);
|
||||
assert_eq!(state_of(&q, id), State::Failed);
|
||||
}
|
||||
|
||||
// ---- build logs, history ----
|
||||
|
|
@ -1219,8 +1337,7 @@ fn history_evicts_oldest_terminals_past_flat_cap() {
|
|||
// Fail the single work node so the DAG *lingers*: a fully-`Done` DAG
|
||||
// drops off the wire entirely, but a `Failed` one is retained (+
|
||||
// history-capped) so the operator can still triage it. Completing the
|
||||
// node rolls the container up terminal (its inline hook fires off the
|
||||
// returned summary — no terminal-hook node).
|
||||
// node rolls the container up terminal.
|
||||
q.complete_node(id, c.node_id, Err("boom".to_owned()));
|
||||
ids.push(id);
|
||||
}
|
||||
|
|
@ -1310,8 +1427,8 @@ fn spawn_shape_provision_create_dropin_reconcile() {
|
|||
assert_eq!(c.approval_id, Some(7));
|
||||
q.complete_node(id, c.node_id, Ok(()));
|
||||
}
|
||||
let report_terminal = state_of(&q, id);
|
||||
assert_eq!(report_terminal, State::Done);
|
||||
settle_approval_tail(&q, id, 7, State::Done);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -1342,6 +1459,7 @@ fn perm_change_shape_prefixes_rebuild_chain() {
|
|||
assert_eq!(c.kind.as_str(), expected);
|
||||
q.complete_node(id, c.node_id, Ok(()));
|
||||
}
|
||||
settle_rebuild_tail(&q, id, "agent-a", State::Done);
|
||||
assert_eq!(state_of(&q, id), State::Done);
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue