vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2454): drain agents before stopping them in the boot sweep

Browse source 
A host restart brings hive-c0re up and the startup sweep rebuilds every
stale agent. Until now that stop was mechanical: `StopForUpdate` hung
straight off `Prebuild`, so an agent that was mid-turn when the host went
down had its turn cut off rather than finished.

The sweep now builds the same `Signal` -> `Drain` -> `StopForUpdate` chain
a graceful `hivectl restart` already uses, reusing the existing nodes and
`GRACEFUL_STOP_TIMEOUT` unchanged. Cost is bounded: the per-agent drains
overlap, so the sweep waits one timeout in total rather than one per agent.

`Signal` parents the rest of the stop instead of sitting beside it. All
three of `Signal` / `Drain` / `StopForUpdate` declare the agent lease, and
a resource is held across its holder's whole subtree — as siblings each
would take the lease separately, leaving a window between them for another
DAG to claim the agent mid-bounce.

Scope is the boot sweep alone: a manual rebuild, a meta-update cascade
child and a deploy all still stop mechanically, and a test pins that shape.

`rebuild_nodes` takes a `RebuildOpts` struct rather than a second
positional `bool`, which two adjacent flags would have made easy to swap at
a call site. Its callers no longer hard-code the subgraph's length either:
the `EmitRebuilt` tails and `FinalizeDeploy` used literal indices that
silently encoded "this builder emits exactly six nodes with `Reconcile`
last", which a variable-length subgraph turns into a wrong-node edge rather
than a compile error. They read the index off the emitted list now.
This commit is contained in:
atlas 2026-07-27 19:39:44 +02:00 • committed by mara
commit 15a9d5b652
4 changed files with 236 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

28
hive-c0re/src/job_queue/exec.rs
View file

@ -322,10 +322,25 @@ async fn run_meta_lock(
// (rooted on this `MetaLock`, so they build against the post-bump
// lock), rather than fanning out child DAGs. `relock = true` — a
// boot sweep relocks per-agent like a manual rebuild.
//
// `graceful = true` here and nowhere else: a boot sweep stops agents
// that were already mid-turn when the host came up, so they get their
// drain window rather than being cut off. The per-agent drains overlap,
// so the sweep's cost ceiling is one `GRACEFUL_STOP_TIMEOUT` in total,
// not one per agent.
let append_subgraph = fanout
.unwrap_or_default()
.iter()
.map(|agent| super::templates::rebuild_nodes(agent, true, 0))
.map(|agent| {
super::templates::rebuild_nodes(
agent,
super::templates::RebuildOpts {
relock: true,
graceful: true,
},
0,
)
})
.collect();
return Ok(NodeOutput { append_subgraph });
}
@ -345,7 +360,16 @@ async fn run_meta_lock(
// branch encoded).
let append_subgraph = cascade
.iter()
.map(|agent| super::templates::rebuild_nodes(agent, false, 0))
.map(|agent| {
super::templates::rebuild_nodes(
agent,
super::templates::RebuildOpts {
relock: false,
graceful: false,
},
0,
)
})
.collect();
Ok(NodeOutput { append_subgraph })
}