refactor(#2815): derive the transient pill from the running node
The dashboard pill was declared once per DAG at submit time, so a rebuild reported `rebuilding` for its entire life — through the prebuild, the stop, the swap, the tail and the reconcile. It named the intent of the request, not what was happening. It is now read off the nodes actually running. A node lights a pill when it is `Running` and declares the agent's own resource. Declaring is the test, not targeting: `Prebuild` and `MetaSync` name an agent but are lease-exempt on purpose (the container keeps serving), so they must not light one. It is also not the lease *owner* — `resource_state()` answers "who holds the slot", which is a different question from "what is running", and a descendant that borrows an ancestor's grant never appears in that map. `TransientKind` is gone entirely rather than being re-derived. The label is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary `NodeView.kind` already ships, so a pill and a DAG node name an operation identically and there is no second taxonomy to keep in step. Work with no node behind it (destroy, migration) supplies its own literal. `DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and `NodeKind::Dag`'s `transient` field all go with it. ## the safety half, which is deliberately not the display half `crash_watch::is_deliberate_stop` used to match a `TransientKind` to decide whether a vanished container was intentional or a crash. That made a pill's display vocabulary decide an alerting question, so renaming or adding a label would silently move the alerting boundary. `TransientState` now carries two independent fields: `label` (rendered, nothing branches on it) and `deliberate_stop` (read only by the crash watcher). The producer sets the second, because the producer is the only thing that knows — it is not recoverable from the first. For queue work that value is `NodeKind::takes_container_down()`, and it is emphatically not "holds a lease": `Create` and `Start` hold the agent's lease exactly like `Stop` does, and a container dying *while starting* is a real crash that must keep reporting as one. The default is `false` on purpose — a wrong `false` costs a spurious crash event, a wrong `true` swallows a real crash silently. ## known cost, accepted on the issue A restart no longer reads `restarting`. No `NodeKind` is unique to a restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` / `Reconcile` — because "restart" is a property of the DAG's shape, not of any node. A restart now reads `signal` / `stop_for_update`, then the agent returns. `Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and re-declare nothing, so they light no pill and the agent reads idle for those windows. Closing that is the resources-where-constructed work (#2818), not this change. Checked with clippy (`--all-targets -D warnings`), `cargo test -p hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
This commit is contained in:
parent
0aac20d863
commit
d3d73b5ffb
14 changed files with 295 additions and 261 deletions
|
|
@ -353,7 +353,6 @@ fn submit_boot_tree(
|
|||
// Rebuilding when the sweep will grow rebuild subgraphs (per-agent
|
||||
// crash-watch suppression during their Swap, applied at claim time);
|
||||
// a reconcile-only boot needs no transient.
|
||||
transient: any_stale.then_some(crate::coordinator::TransientKind::Rebuilding),
|
||||
nodes,
|
||||
};
|
||||
if let Err(e) = coord.job_queue.submit(spec) {
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ use std::sync::Arc;
|
|||
use std::time::Duration;
|
||||
|
||||
use crate::container_view::claude_has_session;
|
||||
use crate::coordinator::{Coordinator, TransientKind};
|
||||
use crate::coordinator::Coordinator;
|
||||
use crate::lifecycle::{self, AGENT_PREFIX};
|
||||
|
||||
const POLL_INTERVAL: Duration = Duration::from_secs(10);
|
||||
|
|
@ -87,7 +87,7 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
|
|||
// guard between two crash-watch polls.
|
||||
let recent = coord.recent_transient_within(RECENT_TRANSIENT_GRACE);
|
||||
for stopped in prev.difference(current) {
|
||||
let active = transients.get(stopped).map(|st| st.kind);
|
||||
let active = transients.get(stopped).map(|st| st.deliberate_stop);
|
||||
let recently_cleared = recent.get(stopped).copied();
|
||||
if is_deliberate_stop(active, recently_cleared) {
|
||||
continue;
|
||||
|
|
@ -101,26 +101,19 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
|
|||
}
|
||||
}
|
||||
|
||||
/// Pure classifier: did the operator stop / restart / destroy /
|
||||
/// rebuild this container, or did it crash? Splits the matcher out
|
||||
/// so it has a focused unit test without needing a Coordinator
|
||||
/// fixture. `active` is the currently-set transient (if any),
|
||||
/// `recently_cleared` is one whose RAII guard dropped within the
|
||||
/// grace window.
|
||||
fn is_deliberate_stop(
|
||||
active: Option<TransientKind>,
|
||||
recently_cleared: Option<TransientKind>,
|
||||
) -> bool {
|
||||
let is_op_kind = |kind: TransientKind| {
|
||||
matches!(
|
||||
kind,
|
||||
TransientKind::Stopping
|
||||
| TransientKind::Restarting
|
||||
| TransientKind::Destroying
|
||||
| TransientKind::Rebuilding
|
||||
)
|
||||
};
|
||||
active.is_some_and(is_op_kind) || recently_cleared.is_some_and(is_op_kind)
|
||||
/// Pure classifier: did an operation take this container down on purpose, or
|
||||
/// did it crash? Splits the matcher out so it has a focused unit test without
|
||||
/// needing a Coordinator fixture. `active` is the currently-set transient's
|
||||
/// `deliberate_stop` (if any), `recently_cleared` is one whose RAII guard
|
||||
/// dropped within the grace window.
|
||||
///
|
||||
/// This reads the flag the transient's creator set and nothing else. It used to
|
||||
/// match a `TransientKind`, which meant a pill's *display* vocabulary decided a
|
||||
/// crash-alert question — so renaming or adding a label silently moved the
|
||||
/// alerting boundary. Whoever starts the operation knows whether the container
|
||||
/// is meant to go down; nothing downstream can re-derive it.
|
||||
fn is_deliberate_stop(active: Option<bool>, recently_cleared: Option<bool>) -> bool {
|
||||
active.unwrap_or(false) || recently_cleared.unwrap_or(false)
|
||||
}
|
||||
|
||||
fn emit_login_transitions(
|
||||
|
|
@ -168,29 +161,12 @@ mod tests {
|
|||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn deliberate_when_active_transient_is_operator_kind() {
|
||||
for kind in [
|
||||
TransientKind::Stopping,
|
||||
TransientKind::Restarting,
|
||||
TransientKind::Destroying,
|
||||
TransientKind::Rebuilding,
|
||||
] {
|
||||
assert!(is_deliberate_stop(Some(kind), None), "{kind:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deliberate_when_recent_transient_is_operator_kind() {
|
||||
// Race repros: lifecycle action completes + drops the guard
|
||||
// between two polls. recent_transient catches it.
|
||||
for kind in [
|
||||
TransientKind::Stopping,
|
||||
TransientKind::Restarting,
|
||||
TransientKind::Destroying,
|
||||
TransientKind::Rebuilding,
|
||||
] {
|
||||
assert!(is_deliberate_stop(None, Some(kind)), "{kind:?}");
|
||||
}
|
||||
fn deliberate_when_either_source_says_so() {
|
||||
// Active guard, and the race repro: a lifecycle action completes and
|
||||
// drops its guard between two polls, so only `recent` still carries it.
|
||||
assert!(is_deliberate_stop(Some(true), None));
|
||||
assert!(is_deliberate_stop(None, Some(true)));
|
||||
assert!(is_deliberate_stop(Some(true), Some(true)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -200,13 +176,16 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn not_deliberate_when_only_spawning_starting() {
|
||||
// Spawning/Starting are never paired with a "stopped" transition
|
||||
// — they're starts. If we see one alongside a stop, it's
|
||||
// unrelated (e.g. just-started container died), still a crash.
|
||||
for kind in [TransientKind::Spawning, TransientKind::Starting] {
|
||||
assert!(!is_deliberate_stop(Some(kind), None), "{kind:?} active");
|
||||
assert!(!is_deliberate_stop(None, Some(kind)), "{kind:?} recent");
|
||||
}
|
||||
fn not_deliberate_when_the_operation_was_bringing_the_container_up() {
|
||||
// The case that used to be spelled `Spawning` / `Starting`: an
|
||||
// operation IS in flight, but it is not one that takes the container
|
||||
// down, so a container that vanishes under it really did crash.
|
||||
//
|
||||
// This is why `deliberate_stop` is carried rather than inferred from
|
||||
// the pill — `Create` and `Start` hold the agent's lease exactly like
|
||||
// `Stop` does, so "has a pill" cannot answer this.
|
||||
assert!(!is_deliberate_stop(Some(false), None));
|
||||
assert!(!is_deliberate_stop(None, Some(false)));
|
||||
assert!(!is_deliberate_stop(Some(false), Some(false)));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue