jobq: a transient is any running node naming the agent

Per mara on #2822: status is the only test. The agent comes off the
node's own payload rather than a declared Resource::Agent edge, so the
lease-exempt kinds (Prebuild, MetaSync) that name an agent without
holding its lease now light a pill — they are work on that agent.

Dropping that test breaks the one-pill-per-agent invariant, since
lease-exemption is exactly what lets one DAG build for an agent while
another holds its lease. Everything keyed by agent alone had to follow:

- reconcile_transients keys (agent, label) via TransientSeen, so a
  second pill cannot evict the first — and cannot lose its
  takes_container_down, which the crash watcher reads at clear time.
- transient_snapshot returns a Vec per agent for the same reason. The
  collapse was silent: a Prebuild could evict a StopForUpdate and its
  deliberate_stop, making an intentional stop report as a crash.
- crash_watch asks whether ANY running node expects the container down.
- the dashboard renders one row per node instead of one per agent.

takes_container_down never reached the frontend; no wire change needed.

315 tests pass unchanged.
This commit is contained in:
atlas 2026-08-03 13:03:17 +02:00 committed by mara
commit 8ce265fdf0
6 changed files with 79 additions and 56 deletions

View file

@ -1294,22 +1294,23 @@ impl Coordinator {
/// crash-watch suppression is a separate, narrower thing
/// ([`Coordinator::suppress_crash_watch`]); the pill comes back for free
/// once those become real nodes.
/// ⚠️ **A `Vec` per agent, not one entry.** `running_transients` tests
/// status alone, so a lease-exempt `Prebuild` for `a` and a lease-holding
/// `StopForUpdate` for `a` are both live pills. Collapsing them to one
/// would pick arbitrarily and, for the crash watcher, silently lose a
/// `deliberate_stop = true` behind a `false` — reporting an intentional
/// stop as a crash.
#[must_use]
pub fn transient_snapshot(&self) -> HashMap<String, TransientState> {
self.job_queue
.running_transients()
.into_iter()
.map(|t| {
(
t.agent,
TransientState {
label: t.label,
deliberate_stop: t.takes_container_down,
since: t.since,
},
)
})
.collect()
pub fn transient_snapshot(&self) -> HashMap<String, Vec<TransientState>> {
let mut out: HashMap<String, Vec<TransientState>> = HashMap::new();
for t in self.job_queue.running_transients() {
out.entry(t.agent).or_default().push(TransientState {
label: t.label,
deliberate_stop: t.takes_container_down,
since: t.since,
});
}
out
}
/// Drop a system message into the given agent's inbox. Wakes the

View file

@ -533,11 +533,14 @@ fn build_port_conflicts(containers: &[ContainerView]) -> Vec<PortConflict> {
/// `ContainerView.pending` inline; this list only catches pre-creation.
fn build_transient_views(
containers: &[ContainerView],
transient_snapshot: &std::collections::HashMap<String, crate::coordinator::TransientState>,
transient_snapshot: &std::collections::HashMap<String, Vec<crate::coordinator::TransientState>>,
) -> Vec<TransientView> {
transient_snapshot
.iter()
.filter(|(name, _)| !containers.iter().any(|c| &c.name == *name))
// One row per running node, so an agent with several shows several
// rather than one of them arbitrarily.
.flat_map(|(name, sts)| sts.iter().map(move |st| (name, st)))
.map(|(name, st)| TransientView {
name: name.clone(),
kind: st.label.clone(),

View file

@ -35,7 +35,7 @@ pub struct TombstoneView {
pub(super) fn build_tombstone_views(
coord: &Coordinator,
containers: &[ContainerView],
transient_snapshot: &std::collections::HashMap<String, crate::coordinator::TransientState>,
transient_snapshot: &std::collections::HashMap<String, Vec<crate::coordinator::TransientState>>,
) -> Vec<TombstoneView> {
let _ = coord; // kept_state_names is a free fn but takes &self by future plan
let live: std::collections::HashSet<&str> = containers

View file

@ -306,26 +306,26 @@ impl JobQueue {
/// template declared at submit time. (A rebuild used to report `rebuilding`
/// for its whole life: prebuild, stop, swap, tail and reconcile alike.)
///
/// A node lights a pill when it is `Running` **and declares the agent's
/// resource itself**. Declaring is the test, not targeting — `Prebuild` /
/// `MetaSync` name an agent but are lease-exempt on purpose, since the
/// container keeps serving through them. Nor is it the lease *owner*:
/// `resource_state()` answers "who holds the slot", a different question.
/// **Status is the only test**: every `Running` node that names an agent is
/// in the set. Naming is targeting, not lease-holding — `Prebuild` /
/// `MetaSync` are lease-exempt (the container keeps serving through them)
/// but they *are* work on that agent, and the operator wants to see it.
///
/// ⚠️ **So there can be more than one entry per agent**, which is the whole
/// difference from the older lease-declaration test: lease-exemption is
/// exactly what lets one DAG build for `a` while another holds `a`'s lease,
/// so both are running and both name `a`. Anything keying this set by agent
/// alone will silently drop one — see [`super::scheduler`].
///
/// `label` is the node's own wire tag ([`NodeKind::as_str`]), the vocabulary
/// [`NodeView::kind`] already ships, so a pill and a DAG node name an
/// operation identically. `takes_container_down` is the crash watcher's
/// input, carried rather than inferred from the label — a `Start` pill and a
/// `Stop` pill are both pills; only one means a vanished container is
/// input and does **not** ride the wire to the frontend — a `Start` pill and
/// a `Stop` pill are both pills; only one means a vanished container is
/// expected.
///
/// Read off the node's **declared** resource edges, not off its kind. Those
/// are the same thing now that every construction site states what it holds,
/// and the distinction is the whole point: `Start` / `Stop` / `PostSwap` run
/// inside a lease-holding ancestor, and while the declaration was derived
/// from the kind they re-declared nothing and lit no pill. Asking the node
/// what it holds cannot go stale that way. An agent's lease is cap-1, so at
/// most one entry per agent.
/// Not the lease *owner* either: `resource_state()` answers "who holds the
/// slot", a different question.
#[must_use]
pub fn running_transients(&self) -> Vec<RunningTransient> {
let inner = self.lock();
@ -334,15 +334,20 @@ impl JobQueue {
.nodes()
.filter(|n| matches!(n.state, State::Running))
.filter_map(|n| {
let agent = n.deps.iter().find_map(|dep| match dep {
hive_jobq::Dep::Resource {
name: Resource::Agent(a),
..
} => Some(a.clone()),
_ => None,
})?;
// Status is the only test. The agent comes off the node's own
// payload, not off a declared `Resource::Agent` edge: the
// lease-exempt kinds (`Prebuild` / `MetaSync`) name an agent
// without declaring its lease, and they are work on that agent
// that the operator wants to see.
//
// Empty means an agentless container kind (`MetaLock`, `Dag`),
// which targets no agent and lights nothing.
let agent = n.payload.agent();
if agent.is_empty() {
return None;
}
Some(RunningTransient {
agent,
agent: agent.to_owned(),
label: n.payload.as_str().to_owned(),
takes_container_down: n.payload.takes_container_down(),
// `started_at` is set when a node enters `Running`, and this

View file

@ -28,6 +28,17 @@ use std::sync::Arc;
use super::exec;
use crate::coordinator::Coordinator;
/// Pills published on the previous tick: `(agent, label) -> takes_container_down`.
///
/// Keyed by the **pair**, not by agent. An agent can have several pills at once
/// now that [`super::JobQueue::running_transients`] tests status alone — a
/// lease-exempt `Prebuild` for `a` runs happily while another DAG holds `a`'s
/// lease, and both name `a`. Keying by agent would drop one arbitrarily and,
/// worse, lose its `takes_container_down` — which is the crash watcher's input
/// and is stored as the value precisely so it survives to *clear* time, when the
/// node that carried it is already gone.
type TransientSeen = HashMap<(String, String), bool>;
/// Scheduler loop. Spawned once at hive-c0re startup from `main.rs`.
///
/// Shutdown semantics: subscribes to `coord.shutdown_rx()`. On a true signal
@ -50,12 +61,11 @@ use crate::coordinator::Coordinator;
/// reconverging silently.
pub async fn run_worker(coord: Arc<Coordinator>) {
let mut shutdown = coord.shutdown_rx();
// Last derived pill set we published, keyed by agent (its lease is cap-1,
// so one pill each). Purely the previous value of a *derived* quantity —
// it exists to spot transitions, since the dashboard wants edges
// (`TransientSet` / `TransientCleared`) and the crash watcher wants the
// moment of the clear. Nothing owns a pill; nothing can leak one.
let mut transients: HashMap<String, (String, bool)> = HashMap::new();
// Last derived pill set we published. Purely the previous value of a
// *derived* quantity — it exists to spot transitions, since the dashboard
// wants edges (`TransientSet` / `TransientCleared`) and the crash watcher
// wants the moment of the clear. Nothing owns a pill; nothing can leak one.
let mut transients = TransientSeen::new();
loop {
// Checked every iteration, not just in the `select!` below — a
// continuous stream of ready claims never reaches the `select!`, so
@ -168,15 +178,14 @@ pub async fn run_worker(coord: Arc<Coordinator>) {
/// starting* is a real crash that must keep reporting as one.
///
/// [`NodeKind::takes_container_down`]: super::NodeKind::takes_container_down
fn reconcile_transients(coord: &Arc<Coordinator>, prev: &mut HashMap<String, (String, bool)>) {
fn reconcile_transients(coord: &Arc<Coordinator>, prev: &mut TransientSeen) {
let running = coord.job_queue.running_transients();
// Cleared: in `prev`, gone (or relabelled) now. Emitted before the sets
// below so a same-agent label change reads as clear-then-set rather than
// two overlapping pills. `deliberate_stop` is carried in `prev` precisely
// so it is still available *here* — the node it came from is, by
// definition, no longer running to be asked.
prev.retain(|agent, (label, deliberate)| {
// Cleared: in `prev`, gone now. Emitted before the sets below so a
// replacement reads as clear-then-set rather than two overlapping pills.
// `deliberate_stop` is the value precisely so it is still available *here* —
// the node it came from is, by definition, no longer running to be asked.
prev.retain(|(agent, label), deliberate| {
let still = running
.iter()
.any(|t| &t.agent == agent && &t.label == label);
@ -187,10 +196,11 @@ fn reconcile_transients(coord: &Arc<Coordinator>, prev: &mut HashMap<String, (St
});
for t in running {
if prev.get(&t.agent).map(|(l, _)| l) == Some(&t.label) {
let key = (t.agent, t.label);
if prev.contains_key(&key) {
continue;
}
coord.emit_transient_set(&t.agent, t.label.clone());
prev.insert(t.agent, (t.label, t.takes_container_down));
coord.emit_transient_set(&key.0, key.1.clone());
prev.insert(key, t.takes_container_down);
}
}

View file

@ -91,9 +91,13 @@ fn emit_crash_transitions(coord: &Coordinator, prev: &HashSet<String>, current:
// Two sources, because a container can go down on purpose either way:
// a running queue node that declared it takes the container down, or a
// no-node operation (destroy, migration) holding a suppression guard.
// `any`, not "the" pill: an agent can have several running nodes at
// once (a lease-exempt build alongside a lease-holding stop), and it
// only takes one of them expecting the container down for this to be
// a deliberate stop rather than a crash.
let active = transients
.get(stopped)
.map(|st| st.deliberate_stop)
.map(|sts| sts.iter().any(|st| st.deliberate_stop))
.or_else(|| coord.crash_watch_suppressed(stopped).then_some(true));
let recently_cleared = recent.get(stopped).copied();
if is_deliberate_stop(active, recently_cleared) {