jobq: a transient is any running node naming the agent

Per mara on #2822: status is the only test. The agent comes off the
node's own payload rather than a declared Resource::Agent edge, so the
lease-exempt kinds (Prebuild, MetaSync) that name an agent without
holding its lease now light a pill — they are work on that agent.

Dropping that test breaks the one-pill-per-agent invariant, since
lease-exemption is exactly what lets one DAG build for an agent while
another holds its lease. Everything keyed by agent alone had to follow:

- reconcile_transients keys (agent, label) via TransientSeen, so a
  second pill cannot evict the first — and cannot lose its
  takes_container_down, which the crash watcher reads at clear time.
- transient_snapshot returns a Vec per agent for the same reason. The
  collapse was silent: a Prebuild could evict a StopForUpdate and its
  deliberate_stop, making an intentional stop report as a crash.
- crash_watch asks whether ANY running node expects the container down.
- the dashboard renders one row per node instead of one per agent.

takes_container_down never reached the frontend; no wire change needed.

315 tests pass unchanged.
This commit is contained in:
atlas 2026-08-03 13:03:17 +02:00 committed by mara
commit 8ce265fdf0
6 changed files with 79 additions and 56 deletions

View file

@ -306,26 +306,26 @@ impl JobQueue {
/// template declared at submit time. (A rebuild used to report `rebuilding`
/// for its whole life: prebuild, stop, swap, tail and reconcile alike.)
///
/// A node lights a pill when it is `Running` **and declares the agent's
/// resource itself**. Declaring is the test, not targeting — `Prebuild` /
/// `MetaSync` name an agent but are lease-exempt on purpose, since the
/// container keeps serving through them. Nor is it the lease *owner*:
/// `resource_state()` answers "who holds the slot", a different question.
/// **Status is the only test**: every `Running` node that names an agent is
/// in the set. Naming is targeting, not lease-holding — `Prebuild` /
/// `MetaSync` are lease-exempt (the container keeps serving through them)
/// but they *are* work on that agent, and the operator wants to see it.
///
/// ⚠️ **So there can be more than one entry per agent**, which is the whole
/// difference from the older lease-declaration test: lease-exemption is
/// exactly what lets one DAG build for `a` while another holds `a`'s lease,
/// so both are running and both name `a`. Anything keying this set by agent
/// alone will silently drop one — see [`super::scheduler`].
///
/// `label` is the node's own wire tag ([`NodeKind::as_str`]), the vocabulary
/// [`NodeView::kind`] already ships, so a pill and a DAG node name an
/// operation identically. `takes_container_down` is the crash watcher's
/// input, carried rather than inferred from the label — a `Start` pill and a
/// `Stop` pill are both pills; only one means a vanished container is
/// input and does **not** ride the wire to the frontend — a `Start` pill and
/// a `Stop` pill are both pills; only one means a vanished container is
/// expected.
///
/// Read off the node's **declared** resource edges, not off its kind. Those
/// are the same thing now that every construction site states what it holds,
/// and the distinction is the whole point: `Start` / `Stop` / `PostSwap` run
/// inside a lease-holding ancestor, and while the declaration was derived
/// from the kind they re-declared nothing and lit no pill. Asking the node
/// what it holds cannot go stale that way. An agent's lease is cap-1, so at
/// most one entry per agent.
/// Not the lease *owner* either: `resource_state()` answers "who holds the
/// slot", a different question.
#[must_use]
pub fn running_transients(&self) -> Vec<RunningTransient> {
let inner = self.lock();
@ -334,15 +334,20 @@ impl JobQueue {
.nodes()
.filter(|n| matches!(n.state, State::Running))
.filter_map(|n| {
let agent = n.deps.iter().find_map(|dep| match dep {
hive_jobq::Dep::Resource {
name: Resource::Agent(a),
..
} => Some(a.clone()),
_ => None,
})?;
// Status is the only test. The agent comes off the node's own
// payload, not off a declared `Resource::Agent` edge: the
// lease-exempt kinds (`Prebuild` / `MetaSync`) name an agent
// without declaring its lease, and they are work on that agent
// that the operator wants to see.
//
// Empty means an agentless container kind (`MetaLock`, `Dag`),
// which targets no agent and lights nothing.
let agent = n.payload.agent();
if agent.is_empty() {
return None;
}
Some(RunningTransient {
agent,
agent: agent.to_owned(),
label: n.payload.as_str().to_owned(),
takes_container_down: n.payload.takes_container_down(),
// `started_at` is set when a node enters `Running`, and this