refactor(#2815): derive the transient pill from the running node
The dashboard pill was declared once per DAG at submit time, so a rebuild reported `rebuilding` for its entire life — through the prebuild, the stop, the swap, the tail and the reconcile. It named the intent of the request, not what was happening. It is now read off the nodes actually running. A node lights a pill when it is `Running` and declares the agent's own resource. Declaring is the test, not targeting: `Prebuild` and `MetaSync` name an agent but are lease-exempt on purpose (the container keeps serving), so they must not light one. It is also not the lease *owner* — `resource_state()` answers "who holds the slot", which is a different question from "what is running", and a descendant that borrows an ancestor's grant never appears in that map. `TransientKind` is gone entirely rather than being re-derived. The label is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary `NodeView.kind` already ships, so a pill and a DAG node name an operation identically and there is no second taxonomy to keep in step. Work with no node behind it (destroy, migration) supplies its own literal. `DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and `NodeKind::Dag`'s `transient` field all go with it. ## the safety half, which is deliberately not the display half `crash_watch::is_deliberate_stop` used to match a `TransientKind` to decide whether a vanished container was intentional or a crash. That made a pill's display vocabulary decide an alerting question, so renaming or adding a label would silently move the alerting boundary. `TransientState` now carries two independent fields: `label` (rendered, nothing branches on it) and `deliberate_stop` (read only by the crash watcher). The producer sets the second, because the producer is the only thing that knows — it is not recoverable from the first. For queue work that value is `NodeKind::takes_container_down()`, and it is emphatically not "holds a lease": `Create` and `Start` hold the agent's lease exactly like `Stop` does, and a container dying *while starting* is a real crash that must keep reporting as one. The default is `false` on purpose — a wrong `false` costs a spurious crash event, a wrong `true` swallows a real crash silently. ## known cost, accepted on the issue A restart no longer reads `restarting`. No `NodeKind` is unique to a restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` / `Reconcile` — because "restart" is a property of the DAG's shape, not of any node. A restart now reads `signal` / `stop_for_update`, then the agent returns. `Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and re-declare nothing, so they light no pill and the agent reads idle for those windows. Closing that is the resources-where-constructed work (#2818), not this change. Checked with clippy (`--all-targets -D warnings`), `cargo test -p hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
This commit is contained in:
parent
0aac20d863
commit
d3d73b5ffb
14 changed files with 295 additions and 261 deletions
|
|
@ -3,12 +3,12 @@
|
|||
//! and on any completion re-evaluate. Concurrency comes from the build-slot
|
||||
//! count, not multiple workers.
|
||||
//!
|
||||
//! Owns the per-DAG transient guard (dashboard pill + crash-watch suppression)
|
||||
//! that the sync queue core can't hold itself. The guard set is *reconciled*
|
||||
//! from live lease ownership ([`super::JobQueue::held_transients`]) each loop:
|
||||
//! a `(dag, agent)` pill exists for exactly as long as that agent's lease is
|
||||
//! held, so it appears when the agent's owner node starts and disappears when
|
||||
//! its subgraph settles — one pill per agent a DAG touches.
|
||||
//! Owns the per-agent transient guard (dashboard pill + crash-watch
|
||||
//! suppression) that the sync queue core can't hold itself. The guard set is
|
||||
//! *reconciled* each loop from [`super::JobQueue::held_transients`], which
|
||||
//! reports what is **running right now** under each held agent lease — so the
|
||||
//! label tracks the DAG's progress (signal → swap → reconcile) instead of
|
||||
//! repeating one intent the template declared before any of it started.
|
||||
//!
|
||||
//! Per-DAG terminal work (approval resolution, `Rebuilt`) is not drained here:
|
||||
//! it runs as the DAG's focused terminal node (`ResolveApproval` /
|
||||
|
|
@ -19,7 +19,7 @@
|
|||
//! its `Start`/`Stop`) flows through `NodeOutput.append_subgraph`, applied
|
||||
//! before the emitting node completes — see `handle_completion`.
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use super::Claim;
|
||||
|
|
@ -54,8 +54,10 @@ struct NodeDone {
|
|||
pub async fn run_worker(coord: Arc<Coordinator>) {
|
||||
let mut shutdown = coord.shutdown_rx();
|
||||
let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel::<NodeDone>();
|
||||
// (DAG id, agent) → transient guard held for that agent's lease window.
|
||||
let mut transients: HashMap<(u64, String), crate::coordinator::TransientGuard> = HashMap::new();
|
||||
// Keyed by agent (its lease is cap-1, so one pill each); the label rides
|
||||
// along so a change of label can be detected and the guard swapped.
|
||||
let mut transients: HashMap<String, (String, crate::coordinator::TransientGuard)> =
|
||||
HashMap::new();
|
||||
loop {
|
||||
// Checked every iteration, not just in the `select!` below — a
|
||||
// continuous stream of ready claims never reaches the `select!`, so
|
||||
|
|
@ -146,18 +148,35 @@ fn handle_completion(coord: &Arc<Coordinator>, done: NodeDone) {
|
|||
coord.emit_rebuild_queue_snapshot();
|
||||
}
|
||||
|
||||
/// Reconcile the transient-guard set against live lease ownership: drop pills
|
||||
/// whose lease is no longer held, create one for each newly-held `(dag, agent)`.
|
||||
/// Reconcile the transient-guard set against the live pill set: drop guards for
|
||||
/// pills that are no longer current, create one for each newly-current
|
||||
/// `(agent, label)`. Keyed by agent — an agent's lease is cap-1, so it has at
|
||||
/// most one pill.
|
||||
///
|
||||
/// `deliberate_stop` rides along per node ([`NodeKind::takes_container_down`])
|
||||
/// rather than being blanket-`true` for anything holding a lease: `Create` and
|
||||
/// `Start` hold the agent's lease too, and a container vanishing *while
|
||||
/// starting* is a real crash that must keep reporting as one.
|
||||
///
|
||||
/// [`NodeKind::takes_container_down`]: super::NodeKind::takes_container_down
|
||||
///
|
||||
/// ⚠️ **`retain` must run to completion before anything is created.**
|
||||
/// `TransientGuard::drop` calls `clear_transient(agent)` — keyed by agent alone,
|
||||
/// with no notion of *which* label it was clearing. So when a pill's label
|
||||
/// changes for the same agent (which is now routine: the label follows the
|
||||
/// running node as a DAG advances), creating the new guard first and dropping
|
||||
/// the old second would clear the pill that was just set. Dropping first is what
|
||||
/// makes the swap safe.
|
||||
fn reconcile_transients(
|
||||
coord: &Arc<Coordinator>,
|
||||
transients: &mut HashMap<(u64, String), crate::coordinator::TransientGuard>,
|
||||
transients: &mut HashMap<String, (String, crate::coordinator::TransientGuard)>,
|
||||
) {
|
||||
let held = coord.job_queue.held_transients();
|
||||
let keys: HashSet<(u64, String)> = held.iter().map(|(d, a, _)| (*d, a.clone())).collect();
|
||||
transients.retain(|k, _| keys.contains(k));
|
||||
for (dag_id, agent, kind) in held {
|
||||
transients
|
||||
.entry((dag_id, agent.clone()))
|
||||
.or_insert_with(|| coord.transient_guard(&agent, kind));
|
||||
transients.retain(|agent, (label, _)| held.iter().any(|(a, l, _)| a == agent && l == label));
|
||||
for (agent, label, takes_down) in held {
|
||||
transients.entry(agent.clone()).or_insert_with(|| {
|
||||
let guard = coord.transient_guard(&agent, label.clone(), takes_down);
|
||||
(label, guard)
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue