vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(#2815): derive the transient pill from the running node

Browse source 
The dashboard pill was declared once per DAG at submit time, so a rebuild
reported `rebuilding` for its entire life — through the prebuild, the
stop, the swap, the tail and the reconcile. It named the intent of the
request, not what was happening.

It is now read off the nodes actually running. A node lights a pill when
it is `Running` and declares the agent's own resource. Declaring is the
test, not targeting: `Prebuild` and `MetaSync` name an agent but are
lease-exempt on purpose (the container keeps serving), so they must not
light one. It is also not the lease *owner* — `resource_state()` answers
"who holds the slot", which is a different question from "what is
running", and a descendant that borrows an ancestor's grant never
appears in that map.

`TransientKind` is gone entirely rather than being re-derived. The label
is the node's own wire tag (`NodeKind::as_str`) — the same vocabulary
`NodeView.kind` already ships, so a pill and a DAG node name an operation
identically and there is no second taxonomy to keep in step. Work with no
node behind it (destroy, migration) supplies its own literal.

`DagSpec::transient`, `Claim::transient`, `DagMeta::transient` and
`NodeKind::Dag`'s `transient` field all go with it.

## the safety half, which is deliberately not the display half

`crash_watch::is_deliberate_stop` used to match a `TransientKind` to
decide whether a vanished container was intentional or a crash. That made
a pill's display vocabulary decide an alerting question, so renaming or
adding a label would silently move the alerting boundary.

`TransientState` now carries two independent fields: `label` (rendered,
nothing branches on it) and `deliberate_stop` (read only by the crash
watcher). The producer sets the second, because the producer is the only
thing that knows — it is not recoverable from the first.

For queue work that value is `NodeKind::takes_container_down()`, and it
is emphatically not "holds a lease": `Create` and `Start` hold the
agent's lease exactly like `Stop` does, and a container dying *while
starting* is a real crash that must keep reporting as one. The default is
`false` on purpose — a wrong `false` costs a spurious crash event, a
wrong `true` swallows a real crash silently.

## known cost, accepted on the issue

A restart no longer reads `restarting`. No `NodeKind` is unique to a
restart — `restart_chain` reuses `Signal` / `StopForUpdate` / `Drain` /
`Reconcile` — because "restart" is a property of the DAG's shape, not of
any node. A restart now reads `signal` / `stop_for_update`, then the
agent returns.

`Start` / `Stop` / `PostSwap` run inside a lease-holding ancestor and
re-declare nothing, so they light no pill and the agent reads idle for
those windows. Closing that is the resources-where-constructed work
(#2818), not this change.

Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re -p hive-jobq` (321 + 40 passed) and `nix fmt`.
This commit is contained in:
atlas 2026-08-01 16:06:06 +02:00
commit d3d73b5ffb
14 changed files with 295 additions and 261 deletions
Download patch file Download diff file Expand all files Collapse all files

75
hive-c0re/src/job_queue/mod.rs
View file

@ -47,7 +47,6 @@ use hive_jobq::{Dep, Graph, NodeId};
use hive_sh4re::wire_time::now_unix;
use tokio::sync::Notify;
use crate::coordinator::TransientKind;
pub use hive_jobq::TerminalState;
pub use model::{DagSpec, DagView, NodeKind, NodeSpec, PermPayload, Source, State};
use resource::Resource;
@ -70,10 +69,6 @@ pub struct Claim {
/// The agent this node targets (its own, not a DAG-level field). Empty for
/// the agentless [`NodeKind::MetaLock`] + [`NodeKind::Dag`] container nodes.
pub agent: String,
/// Transient pill kind for the lease window (from the spec). Whether the
/// pill is currently shown is derived from live lease ownership
/// ([`JobQueue::held_transients`]), not a per-claim edge.
pub transient: Option<TransientKind>,
}
/// Per-node runtime metadata the crate graph doesn't carry. Lifecycle
@ -91,7 +86,6 @@ struct NodeRuntime {
struct DagMeta {
source: Source,
reason: String,
transient: Option<TransientKind>,
created_at: i64,
}
@ -215,7 +209,6 @@ impl JobQueue {
NodeKind::Dag {
source: spec.source,
reason: spec.reason,
transient: spec.transient,
created_at: now_unix(),
},
Vec::new(),
@ -293,15 +286,11 @@ impl JobQueue {
let Some(container) = inner.sched.graph().root_of(id) else {
continue;
};
let Some(meta) = inner.dag_meta(container) else {
continue;
};
claims.push(Claim {
dag_id: container.get(),
node_id: id,
kind,
agent,
transient: meta.transient,
});
// `started_at` is stamped on the graph `Node` by the scheduler's
// transition to `Running` — no host-side copy needed.
@ -410,24 +399,58 @@ impl JobQueue {
.map(ToOwned::to_owned)
}
/// The `(dag_id, agent, kind)` triples for every per-agent lease currently
/// held by a DAG that carries a transient pill — the live transient-pill
/// set, a pull query over crate resource ownership (replaces the old
/// lease-release event stream). A DAG with no transient kind is omitted.
/// The `(agent, label)` pairs for the live transient-pill set — derived from
/// the nodes **actually running**, not from an intent a template declared at
/// submit time. A rebuild used to report `rebuilding` for its whole life:
/// through the prebuild, the stop, the swap, the tail and the reconcile.
///
/// A node lights a pill when it is `Running` **and declares the agent's
/// resource itself**. Declaring is the test, not targeting: `Prebuild` and
/// `MetaSync` name an agent but are lease-exempt on purpose — the container
/// keeps serving right through them — so they must not light one. It is also
/// not the *lease owner*: `resource_state()` answers "who holds the slot",
/// a different question from "what is running".
///
/// The label is the node's own wire tag ([`NodeKind::as_str`]) — the same
/// vocabulary [`NodeView::kind`] already ships, so a pill and a DAG node
/// name an operation identically and there is no second taxonomy to keep in
/// step.
///
/// Consequence, by design: `Start` / `Stop` / `PostSwap` run *inside* a
/// lease-holding ancestor and re-declare nothing, so they light no pill and
/// the agent reads idle for those windows. Closing that is the point of the
/// resources-where-constructed work, not of this function.
///
/// An agent's lease is cap-1, so at most one pair per agent.
/// The third element is [`NodeKind::takes_container_down`] — the crash
/// watcher's input, carried alongside the label rather than inferred from
/// it (a `Start` pill and a `Stop` pill are both pills; only one of them
/// means a vanished container is expected).
#[must_use]
pub fn held_transients(&self) -> Vec<(u64, String, TransientKind)> {
pub fn held_transients(&self) -> Vec<(String, String, bool)> {
let inner = self.lock();
inner
.sched
.resource_state()
.into_iter()
.filter_map(|(res, holder)| {
let Resource::Agent(agent) = res else {
return None;
};
let container = inner.sched.graph().root_of(holder)?;
let kind = inner.dag_meta(container)?.transient?;
Some((container.get(), agent, kind))
.graph()
.nodes()
.filter(|n| matches!(n.state, State::Running))
.filter_map(|n| {
let agent = n
.payload
.resource_deps()
.into_iter()
.find_map(|d| match d {
Dep::Resource {
name: Resource::Agent(a),
..
} => Some(a),
_ => None,
})?;
Some((
agent,
n.payload.as_str().to_owned(),
n.payload.takes_container_down(),
))
})
.collect()
}
@ -481,7 +504,6 @@ impl QueueInner {
let NodeKind::Dag {
source,
reason,
transient,
created_at,
} = &self.sched.graph().node(container)?.payload
else {
@ -490,7 +512,6 @@ impl QueueInner {
Some(DagMeta {
source: *source,
reason: reason.clone(),
transient: *transient,
created_at: *created_at,
})
}