job_queue: retire the now-off-wire step sub-step label

The `step` label was taken off the wire in #2661, when each deploy phase
became a first-class DAG node. Since then it has been written but never
read: `NodeRuntime` derives only `Debug, Default, Clone` — no serde — so
the field could not reach any client, and the only reads of it were the
dedup checks inside its own setters. This deletes the machinery.

Removed:

- `NodeRuntime.step`, `set_step`, `set_step_running`, and the
  `rt.step = None` clear in `complete_node`. `NodeRuntime` keeps its
  remaining `build_log_id` field (deliberately still a struct — collapsing
  it to a bare `Option<i64>` would churn every call site for no gain).
- `Ctx::step` and its ~15 call sites in `job_queue/exec.rs`. `Ctx` itself
  stays: it is the build-log sink, which `run_prebuild` and `run_swap`
  still use.
- `Coordinator::set_queue_step` and its 11 callers in `actions.rs`.
- `JobQueue::running_node_of`, reachable only from `set_queue_step`.
- `swap_update`'s `on_step` parameter and its one body call.
- The `set_step_only_on_running_and_signals_change` test.

Dropping the calls orphaned parameters, which are removed with their call
sites: `ctx` on ten executors that used it only as a step sink, and
`queue_entry_id` on `run_deploy_merge_verify` / `run_deploy_apply` /
`run_finalize_deploy` plus both `coord` and `queue_entry_id` on
`prepare_applied_target`. `run_deploy_tail` KEEPS its `queue_entry_id` —
that one has a genuine surviving use (the build-log link in the failure
comment posted to the PR).

One behavioural change, called out so it is not mistaken for a dropped
dashboard refresh: `Ctx::step` and `set_queue_step` each emitted a
`rebuild_queue_changed` snapshot when the label changed, and those
emissions go away with them. This is safe — the snapshot payload has no
step field, so those pushes carried nothing a client could observe. Real
state transitions still emit from the scheduler's claim and completion
paths, from `submit`, and from the three `actions.rs` sites. Net effect is
strictly fewer redundant SSE pushes.

Docs: `docs/coordinator.md` still listed `step` as a `NodeView` wire field
and `docs/web-ui/dashboard.md` documented a cyan `↳ <step>` sub-line under
each queue row. Neither has existed since #2661 — both corrected here, plus
the `job_queue/model.rs` module doc.

Not touched: `frontend/packages/dashboard/src/system-sections.css` has a
dead `.rqe-step` rule with no JS referencing it. Left for the frontend
owner rather than deleted here.

Closes: #2664
This commit is contained in:
atlas 2026-07-26 14:53:19 +02:00 committed by mara
commit 1db3cc32a1
9 changed files with 51 additions and 210 deletions

View file

@ -43,7 +43,7 @@ pub struct NodeOutput {
pub append_subgraph: Vec<Vec<NodeSpec>>,
}
/// Step-label + build-log sink for one claimed node.
/// Build-log sink for one claimed node.
struct Ctx<'a> {
coord: &'a Arc<Coordinator>,
dag_id: u64,
@ -51,16 +51,6 @@ struct Ctx<'a> {
}
impl Ctx<'_> {
fn step(&self, step: &str) {
if self
.coord
.job_queue
.set_step(self.dag_id, self.node_id, step)
{
self.coord.emit_rebuild_queue_snapshot();
}
}
fn build_log(&self, log_id: i64) {
if self
.coord
@ -85,20 +75,20 @@ pub(super) async fn run_node(coord: &Arc<Coordinator>, claim: &Claim) -> Result<
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, claim, *relock).await,
NodeKind::Prebuild { .. } => run_prebuild(claim, &ctx).await,
NodeKind::Swap { .. } => run_swap(coord, claim, &ctx).await,
NodeKind::PostSwap { .. } => run_post_swap(coord, claim, &ctx).await,
NodeKind::Provision { .. } => run_provision(coord, claim, &ctx).await,
NodeKind::Create { .. } => run_create(claim, &ctx).await,
NodeKind::PostSwap { .. } => run_post_swap(coord, claim).await,
NodeKind::Provision { .. } => run_provision(coord, claim).await,
NodeKind::Create { .. } => run_create(claim).await,
NodeKind::MetaLock { sweep, fanout } => {
run_meta_lock(coord, claim, &ctx, *sweep, fanout.clone()).await
run_meta_lock(coord, claim, *sweep, fanout.clone()).await
}
NodeKind::Reconcile { .. } => run_reconcile(coord, claim).await,
NodeKind::Start { .. } => run_start(coord, claim, &ctx).await,
NodeKind::Stop { .. } => run_stop(coord, claim, &ctx).await,
NodeKind::StopForUpdate { .. } => run_stop_for_update(coord, claim, &ctx).await,
NodeKind::Signal { .. } => Ok(run_signal(coord, claim, &ctx)),
NodeKind::Drain { .. } => run_drain(coord, claim, &ctx).await,
NodeKind::Start { .. } => run_start(coord, claim).await,
NodeKind::Stop { .. } => run_stop(coord, claim).await,
NodeKind::StopForUpdate { .. } => run_stop_for_update(coord, claim).await,
NodeKind::Signal { .. } => Ok(run_signal(coord, claim)),
NodeKind::Drain { .. } => run_drain(coord, claim).await,
NodeKind::WriteDropin { .. } => run_write_dropin(coord, claim).await,
NodeKind::WritePermFile { .. } => run_write_perm_file(coord, claim, &ctx).await,
NodeKind::WritePermFile { .. } => run_write_perm_file(coord, claim).await,
NodeKind::DeployWindow { .. } => run_deploy_window(claim),
NodeKind::MergeVerify { .. } => run_merge_verify(coord, claim).await,
NodeKind::DeployApply { .. } => run_deploy_apply(coord, claim).await,
@ -239,12 +229,9 @@ async fn run_prebuild(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
// stopped agent has no uptime to preserve, so skip the (expensive)
// eval and let the downstream `Swap` build inline.
if crate::lifecycle::is_running(name).await {
ctx.step("nix build");
let flake_ref = format!("{}#{name}", crate::paths::meta_root().display());
crate::lifecycle::prebuild_toplevel(name, &flake_ref, &|log_id| ctx.build_log(log_id))
.await?;
} else {
ctx.step("skipped prebuild (container down)");
}
Ok(NodeOutput::default())
}
@ -261,11 +248,10 @@ async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Res
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
let result =
crate::lifecycle::swap_update(name, &hive, &paths, &|step| ctx.step(step), &|log_id| {
ctx.build_log(log_id);
})
.await;
let result = crate::lifecycle::swap_update(name, &hive, &paths, &|log_id| {
ctx.build_log(log_id);
})
.await;
// On success the Ok-only bookkeeping tail (rev marker, forge/matrix
// sync, kick, rescan, snapshot) runs in the sibling `PostSwap` node,
// which deps `AfterOk(Swap)`. On failure `PostSwap` is cancel-cascaded
@ -284,11 +270,7 @@ async fn run_swap(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Res
/// means the profile swap succeeded. Store/forge/matrix work only — no nix
/// build (build-slot-exempt); the agent lease taken at `Swap` is still held
/// (the whole chain up to `Reconcile` is one agent's subgraph).
async fn run_post_swap(
coord: &Arc<Coordinator>,
claim: &Claim,
ctx: &Ctx<'_>,
) -> Result<NodeOutput> {
async fn run_post_swap(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
let name = &claim.agent;
if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake)
&& let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev)
@ -298,7 +280,6 @@ async fn run_post_swap(
// The `Rebuilt` manager event fires exactly once per DAG from the
// terminal hook — emitting ok here and letting a failed tail `Reconcile`
// add a contradictory !ok would double-report the same rebuild.
ctx.step("forge sync");
// Full forge + matrix sync on every successful rebuild so the rebuild
// path is equivalent to the startup sweep: tokens, config-repo mirror,
// meta access all recover without a hive-c0re restart.
@ -317,16 +298,11 @@ async fn run_post_swap(
/// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy window (`NodeKind::needs_meta_window`) so its commit can't
/// land inside another node's staged deploy window.
async fn run_provision(
coord: &Arc<Coordinator>,
claim: &Claim,
ctx: &Ctx<'_>,
) -> Result<NodeOutput> {
async fn run_provision(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
let name = &claim.agent;
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
ctx.step("provisioning");
crate::lifecycle::provision_container(name, &hive, &paths).await?;
Ok(NodeOutput::default())
}
@ -337,8 +313,7 @@ async fn run_provision(
/// dir creation and MCP listener registration are deferred to the tail
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
/// node stays purely "create", not "create + start".
async fn run_create(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
ctx.step("nixos-container create");
async fn run_create(claim: &Claim) -> Result<NodeOutput> {
crate::lifecycle::create_only(&claim.agent).await?;
Ok(NodeOutput::default())
}
@ -350,12 +325,10 @@ async fn run_create(claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
async fn run_meta_lock(
coord: &Arc<Coordinator>,
claim: &Claim,
ctx: &Ctx<'_>,
sweep: bool,
fanout: Option<Vec<String>>,
) -> Result<NodeOutput> {
if sweep {
ctx.step("nix flake update hyperhive");
if let Err(e) = crate::meta::lock_update_hyperhive().await {
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
}
@ -371,7 +344,6 @@ async fn run_meta_lock(
return Ok(NodeOutput { append_subgraph });
}
let _progress = coord.meta_update_guard();
ctx.step("nix flake update");
crate::meta::lock_update(&claim.inputs).await?;
// Lock file changed — meta-inputs panel re-renders.
crate::dashboard::emit_meta_inputs_snapshot(coord);
@ -424,7 +396,7 @@ async fn run_reconcile(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOu
/// Mechanical container start — the sub-step a `Reconcile` planner fans
/// out when it observes `wanted = Up` and the container down.
async fn run_start(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
async fn run_start(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
let name = &claim.agent;
// Node-local transient only when the DAG holds none (the
// boot-reconcile template); a rebuild/spawn/etc. DAG's lease-window
@ -441,7 +413,6 @@ async fn run_start(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Re
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
let token = crate::lifecycle::converge_start_preamble(name, &hive, &paths).await?;
ctx.step("nixos-container start");
crate::lifecycle::start_with_fallback(token).await?;
// Bind the MCP listener immediately after starting the container.
// The preamble created the runtime dir; the container is now coming
@ -456,13 +427,12 @@ async fn run_start(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Re
/// Mechanical container stop — the sub-step a `Reconcile` planner fans
/// out when it observes `wanted = Offline` and the container up.
async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
let name = &claim.agent;
let _guard = claim
.transient
.is_none()
.then(|| coord.transient_guard(name, crate::coordinator::TransientKind::Stopping));
ctx.step("nixos-container stop");
crate::lifecycle::kill(name).await?;
coord.unregister_agent(name);
coord.notify_manager(&hive_sh4re::HelperEvent::Killed {
@ -474,11 +444,7 @@ async fn run_stop(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Res
/// Mechanical stop for the profile swap. Never *changes* `wanted`;
/// noop when already stopped.
async fn run_stop_for_update(
coord: &Arc<Coordinator>,
claim: &Claim,
ctx: &Ctx<'_>,
) -> Result<NodeOutput> {
async fn run_stop_for_update(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
let name = &claim.agent;
if crate::lifecycle::is_running(name).await {
// Seed a missing agent_power row from the PRE-stop observation
@ -488,7 +454,6 @@ async fn run_stop_for_update(
if let Err(e) = coord.power.get_or_seed(name, true) {
tracing::warn!(%name, error = ?e, "agent_power: pre-stop seed failed");
}
ctx.step("nixos-container stop");
crate::lifecycle::kill(name).await?;
coord.rescan_containers_and_emit().await;
}
@ -504,12 +469,10 @@ async fn run_stop_for_update(
/// `GRACEFUL_STOP_TIMEOUT`. Safe because the harness tests the marker at
/// the top of its loop — a paused agent has no turn in flight, so there
/// is nothing to checkpoint.
fn run_signal(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> NodeOutput {
fn run_signal(coord: &Arc<Coordinator>, claim: &Claim) -> NodeOutput {
if hive_types::Ident::parse(&claim.agent).is_ok_and(|a| Coordinator::is_paused(&a)) {
ctx.step("graceful stop: agent paused, nothing to drain");
return NodeOutput::default();
}
ctx.step("graceful stop: signalling agent");
coord.mark_graceful_stop(&claim.agent);
coord.kick_agent(&claim.agent, "graceful stop requested");
NodeOutput::default()
@ -518,9 +481,8 @@ fn run_signal(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> NodeOut
/// Await the harness clearing the fence (`GracefulStopComplete`) or
/// the timeout — either way the downstream `Reconcile` proceeds with
/// the actual stop.
async fn run_drain(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> Result<NodeOutput> {
async fn run_drain(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
let name = &claim.agent;
ctx.step("graceful stop: draining");
let deadline = std::time::Instant::now() + GRACEFUL_STOP_TIMEOUT;
while coord.is_graceful_stop_pending(name) {
if std::time::Instant::now() >= deadline {
@ -549,18 +511,13 @@ async fn run_write_dropin(coord: &Arc<Coordinator>, claim: &Claim) -> Result<Nod
/// Write + commit the perm file(s) (fused under `META_LOCK` so the
/// working tree is never left dirty), then emit the P3RM1SS10NS-tab
/// snapshots so the dashboard reflects the new assignment.
async fn run_write_perm_file(
coord: &Arc<Coordinator>,
claim: &Claim,
ctx: &Ctx<'_>,
) -> Result<NodeOutput> {
async fn run_write_perm_file(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
use super::model::PermPayload;
let name = &claim.agent;
// The perm file payload rides the node itself (the only consumer).
let NodeKind::WritePermFile { payload, .. } = &claim.kind else {
anyhow::bail!("run_write_perm_file on a non-WritePermFile node");
};
ctx.step("writing + committing perm file");
// Runs under the deploy window (`NodeKind::needs_meta_window`): a
// perm commit landing inside another node's staged prepare→finalize
// window would sweep the staged deploy lock into its commit (the
@ -622,7 +579,7 @@ fn run_deploy_window(claim: &Claim) -> Result<NodeOutput> {
/// failure here cancel-cascades the rest of the subtree with the forge and the
/// applied repo exactly as they were.
async fn run_merge_verify(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
crate::actions::run_deploy_merge_verify(coord, Some(claim.dag_id), deploy_approval_id(claim)?)
crate::actions::run_deploy_merge_verify(coord, deploy_approval_id(claim)?)
.await
.map(|()| NodeOutput::default())
}
@ -637,7 +594,7 @@ async fn run_merge_verify(coord: &Arc<Coordinator>, claim: &Claim) -> Result<Nod
/// ancestor already holding it. On failure nothing is appended and the tail
/// compensates, exactly as before.
async fn run_deploy_apply(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
crate::actions::run_deploy_apply(coord, Some(claim.dag_id), deploy_approval_id(claim)?).await?;
crate::actions::run_deploy_apply(coord, deploy_approval_id(claim)?).await?;
Ok(NodeOutput {
append_subgraph: vec![super::templates::deploy_rebuild_nodes(claim.kind.agent())],
})
@ -647,7 +604,7 @@ async fn run_deploy_apply(coord: &Arc<Coordinator>, claim: &Claim) -> Result<Nod
/// come up clean: drop the rollback ref, plant the `deployed/<id>` tag, commit
/// the staged lock.
async fn run_finalize_deploy(coord: &Arc<Coordinator>, claim: &Claim) -> Result<NodeOutput> {
crate::actions::run_finalize_deploy(coord, Some(claim.dag_id), deploy_approval_id(claim)?)
crate::actions::run_finalize_deploy(coord, deploy_approval_id(claim)?)
.await
.map(|()| NodeOutput::default())
}

View file

@ -99,11 +99,10 @@ pub struct TerminalDag {
/// Per-node runtime metadata the crate graph doesn't carry. Lifecycle
/// (`started_at` / `finished_at` / `error`) lives on the `hive_jobq::Node`
/// itself now, so only the two host-side extras remain: the live sub-step
/// label and the build-log row link (the client fetches the log by node id).
/// itself now, so only the build-log row link remains host-side (the
/// client fetches the log by node id).
#[derive(Debug, Default, Clone)]
struct NodeRuntime {
step: Option<String>,
build_log_id: Option<i64>,
}
@ -129,8 +128,8 @@ struct DagMeta {
/// [`QueueInner::dag_meta`]). One shared crate [`Graph`] holds every DAG.
struct QueueInner {
sched: Scheduler<NodeKind, Resource>,
/// Per-node runtime metadata (build-log id, step, timestamps, error) —
/// mutable after insert, so it can't ride the immutable node payload.
/// Per-node runtime metadata (the build-log id) — mutable after
/// insert, so it can't ride the immutable node payload.
node_rt: HashMap<NodeId, NodeRuntime>,
}
@ -405,14 +404,11 @@ impl JobQueue {
let mut inner = self.lock();
// The failure reason + `finished_at` are stamped onto the graph `Node`
// by the scheduler (the reason rides `Outcome::Failed`); no host-side
// copy. We only clear the live sub-step label here.
// copy, so there is nothing to clear here.
let outcome = match result {
Ok(()) => Outcome::Done,
Err(e) => Outcome::Failed(truncate_error(&e)),
};
if let Some(rt) = inner.node_rt.get_mut(&node_id) {
rt.step = None;
}
let container = inner.dag_of(node_id);
inner.sched.complete(node_id, outcome);
// If this completion rolled the DAG's container up to a terminal state,
@ -461,35 +457,6 @@ impl JobQueue {
terminal
}
/// Set the step label on a `Running` node. Returns `true` when it changed.
pub fn set_step(&self, dag_id: u64, node_id: NodeId, step: &str) -> bool {
let mut inner = self.lock();
if inner.dag_of(node_id).map(NodeId::get) != Some(dag_id) || !inner.node_running(node_id) {
return false;
}
let rt = inner.node_rt.entry(node_id).or_default();
if rt.step.as_deref() == Some(step) {
return false;
}
rt.step = Some(step.to_owned());
true
}
/// Set the step label on the DAG's currently-running node — the DAG-id-only
/// compatibility surface for the opaque approval pipeline.
pub fn set_step_running(&self, dag_id: u64, step: &str) -> bool {
let mut inner = self.lock();
let Some(node_id) = inner.running_node_of(dag_id) else {
return false;
};
let rt = inner.node_rt.entry(node_id).or_default();
if rt.step.as_deref() == Some(step) {
return false;
}
rt.step = Some(step.to_owned());
true
}
/// Link a `build_logs` row to a specific `Running` node.
pub fn set_build_log_id(&self, dag_id: u64, node_id: NodeId, log_id: i64) -> bool {
let mut inner = self.lock();
@ -670,15 +637,6 @@ impl QueueInner {
})
}
/// The DAG's currently-running work node, if any (the opaque approval
/// pipeline's single-node DAGs make this exact).
fn running_node_of(&self, dag_id: u64) -> Option<NodeId> {
let container = self.container(dag_id)?;
self.subtree(container)
.into_iter()
.find(|&id| self.node_running(id))
}
/// Roll-up state over a DAG's work nodes: `Failed` if any failed; else
/// `Running` if any running; else `Queued` if any queued; else `Cancelled`
/// if any cancelled; else `Done`. (Kept eager over the subtree — a failed

View file

@ -7,7 +7,7 @@
//!
//! Two levels: the **DAG** is the unit of cancel / approval-resolution
//! and the dashboard group; the **node** is the unit of scheduling /
//! execution / build-log / step label, and carries its own `agent` (a
//! execution / build-log, and carries its own `agent` (a
//! DAG can span agents). See `docs/coordinator.md::Job queue` for the
//! full design.

View file

@ -1134,29 +1134,7 @@ fn deploy_dag_skips_apply_but_still_runs_tail_when_verify_fails() {
assert_eq!(summary.approval_id, Some(9));
}
// ---- steps, build logs, history ----
#[test]
fn set_step_only_on_running_and_signals_change() {
let q = JobQueue::new(1);
let id = submit(&q, rebuild("agent-a", "r"));
assert!(
!q.set_step_running(id, "too early"),
"no running node yet → refused"
);
let c = claim_one(&q);
assert!(q.set_step(id, c.node_id, "nix build"));
assert!(
!q.set_step(id, c.node_id, "nix build"),
"same label → false"
);
assert!(q.set_step(id, c.node_id, "next phase"));
assert!(q.set_step_running(id, "via running lookup"));
// `step` is host-side only now (off the wire); completion clears it
// internally, but there's no wire field to observe — the return-value
// contract above (running-gating + change signalling) is the behaviour.
q.complete_node(id, c.node_id, Ok(()));
}
// ---- build logs, history ----
#[test]
fn set_build_log_id_links_running_node() {