vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2454): drain agents before stopping them in the boot sweep

Browse source 
A host restart brings hive-c0re up and the startup sweep rebuilds every
stale agent. Until now that stop was mechanical: `StopForUpdate` hung
straight off `Prebuild`, so an agent that was mid-turn when the host went
down had its turn cut off rather than finished.

The sweep now builds the same `Signal` -> `Drain` -> `StopForUpdate` chain
a graceful `hivectl restart` already uses, reusing the existing nodes and
`GRACEFUL_STOP_TIMEOUT` unchanged. Cost is bounded: the per-agent drains
overlap, so the sweep waits one timeout in total rather than one per agent.

`Signal` parents the rest of the stop instead of sitting beside it. All
three of `Signal` / `Drain` / `StopForUpdate` declare the agent lease, and
a resource is held across its holder's whole subtree — as siblings each
would take the lease separately, leaving a window between them for another
DAG to claim the agent mid-bounce.

Scope is the boot sweep alone: a manual rebuild, a meta-update cascade
child and a deploy all still stop mechanically, and a test pins that shape.

`rebuild_nodes` takes a `RebuildOpts` struct rather than a second
positional `bool`, which two adjacent flags would have made easy to swap at
a call site. Its callers no longer hard-code the subgraph's length either:
the `EmitRebuilt` tails and `FinalizeDeploy` used literal indices that
silently encoded "this builder emits exactly six nodes with `Reconcile`
last", which a variable-length subgraph turns into a wrong-node edge rather
than a compile error. They read the index off the emitted list now.
This commit is contained in:
atlas 2026-07-27 19:39:44 +02:00 • committed by mara
commit 15a9d5b652
4 changed files with 236 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

11
hive-c0re/src/job_queue/submit.rs
View file

@ -25,7 +25,7 @@
use std::sync::Arc;
use super::model::{DagSpec, Dep, NodeKind, NodeSpec};
use super::templates::{after_ok, child, node, rebuild_nodes};
use super::templates::{RebuildOpts, after_ok, child, node, rebuild_nodes};
use super::{Source, templates};
use crate::coordinator::{Coordinator, TransientKind};
use crate::lifecycle;
@ -98,7 +98,14 @@ fn start_chain(agent: &str, running: bool, stale: bool) -> Vec<NodeSpec> {
// `MetaSync` root deps `after_ok(0)` = the head). `MetaSync`,
// `Prebuild` + `Reconcile` are their own group roots (top-level, per
// `rebuild_nodes`).
n.extend(rebuild_nodes(agent, true, 1));
n.extend(rebuild_nodes(
agent,
RebuildOpts {
relock: true,
graceful: false,
},
1,
));
} else {
n.push(child(
0,