vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

feat(#2446): release a DAG's per-agent lease when that agent's subgraph is terminal

Browse source 
A per-agent lifecycle lease gates that agent's container globally across
concurrent DAGs, so it should be held for exactly as long as the agent's
work in a DAG is in flight, no longer. settle() previously freed every
lease a DAG held only at whole-DAG terminal, so a multi-agent DAG (a
hive-wide restart) kept agent A's container locked until B and C also
finished, blocking any other DAG wanting A.

Now free each agent's lease the moment its own subgraph within the DAG is
terminal (no live node still targets it), and drop that agent's dashboard
transient pill on the same edge via a new per-agent release channel. A
single-agent DAG is unaffected: its agent's subgraph goes terminal exactly
when the whole DAG does, so behaviour is identical.
This commit is contained in:
atlas 2026-07-15 23:36:35 +02:00 • committed by mara
commit 9fdadb99c0
4 changed files with 136 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

86
hive-c0re/src/job_queue/mod.rs
View file

@ -94,6 +94,17 @@ pub struct TerminalDag {
pub error: Option<String>,
}
/// A single per-agent lease release: agent `agent`'s subgraph within DAG
/// `dag_id` just reached terminal, so the scheduler drops that agent's
/// `(dag_id, agent)` transient guard — ahead of (or coinciding with) the
/// whole-DAG [`TerminalDag`]. The lease itself is freed inside `settle`;
/// this only carries the transient-drop signal out to the scheduler.
#[derive(Debug, Clone)]
pub struct AgentRelease {
pub dag_id: u64,
pub agent: String,
}
#[derive(Debug, Default)]
struct Inner {
dags: VecDeque<Dag>,
@ -108,6 +119,12 @@ struct Inner {
/// terminal hooks (approval resolution, intent revert, transient
/// release) fire exactly once per DAG no matter how it ended.
pending_terminal: Vec<TerminalDag>,
/// Per-agent lease releases not yet consumed by the scheduler
/// ([`JobQueue::drain_agent_releases`]). Fed by `settle` the moment
/// an agent's subgraph within a DAG goes terminal — earlier than the
/// whole-DAG `pending_terminal` for a multi-agent DAG. Drives the
/// per-agent transient-guard drop.
pending_agent_release: Vec<AgentRelease>,
}
/// The queue. Lives on `Coordinator` (one per hive-c0re process); a
@ -419,45 +436,70 @@ impl JobQueue {
std::mem::take(&mut inner.pending_terminal)
}
/// Propagate cancellations, release the leases of newly-terminal
/// DAGs, buffer each terminal roll-up exactly once (the
/// `terminal_reported` flag) for [`Self::drain_terminal`], and trim
/// history.
/// Take the per-agent lease releases accumulated since the last
/// drain. The scheduler calls this every wakeup and drops the
/// matching `(dag_id, agent)` transient guard for each — freeing an
/// agent's dashboard pill the moment its subgraph settles, ahead of
/// the whole-DAG terminal for a multi-agent DAG.
pub fn drain_agent_releases(&self) -> Vec<AgentRelease> {
let mut inner = self.inner.lock().expect("job_queue mutex poisoned");
std::mem::take(&mut inner.pending_agent_release)
}
/// Propagate cancellations, release each agent's lease the moment its
/// own subgraph settles (not at whole-DAG terminal), buffer each
/// terminal roll-up exactly once (the `terminal_reported` flag) for
/// [`Self::drain_terminal`], and trim history.
fn settle(inner: &mut Inner) {
Self::propagate_cancellations(inner);
let mut freed: Vec<String> = Vec::new();
// Per-agent early lease release. A lease gates an agent's
// container globally, so it must be held for exactly as long as
// that agent's work in the DAG is in flight — no longer. The
// moment no live node of a DAG still targets an agent, free that
// agent's lease so a concurrent DAG wanting the same agent can
// proceed, even while the rest of this DAG runs on. For a
// single-agent DAG the agent's subgraph goes terminal exactly
// when the whole DAG does, so this reduces to the old behaviour.
// Runs over ALL dags, gated on this dag actually holding the
// lease, so it fires exactly once per (dag, agent).
let mut released: Vec<AgentRelease> = Vec::new();
for dag in &inner.dags {
for agent in dag.agents() {
if inner.leases.get(agent.as_str()) == Some(&dag.id)
&& dag.agent_subgraph_terminal(&agent)
{
released.push(AgentRelease {
dag_id: dag.id,
agent,
});
}
}
}
for rel in &released {
inner.leases.remove(&rel.agent);
}
inner.pending_agent_release.extend(released);
// Whole-DAG terminal roll-up (approval resolution, intent revert,
// `Rebuilt` events) — still fires once per DAG. Leases are already
// freed by the per-agent pass above by the time we get here.
let mut reports: Vec<TerminalDag> = Vec::new();
for dag in &mut inner.dags {
if !dag.is_terminal() || dag.terminal_reported {
continue;
}
dag.terminal_reported = true;
// Free every agent-lease this DAG holds (one per distinct
// agent it touched). Single-agent DAGs release their one lease;
// a future multi-agent DAG releases all of them at terminal.
// (Per-agent early release — freeing an agent's lease the moment
// that agent's subgraph is terminal rather than at whole-DAG
// terminal — is a refinement for the multi-agent-emission
// follow-up, where it actually matters.)
let agents = dag.agents();
for agent in &agents {
if inner.leases.get(agent.as_str()) == Some(&dag.id) {
freed.push(agent.clone());
}
}
reports.push(TerminalDag {
dag_id: dag.id,
template: dag.template,
agents,
agents: dag.agents(),
approval_id: dag.approval_id,
state: dag.rollup(),
error: dag.first_error().map(str::to_owned),
});
}
inner.pending_terminal.append(&mut reports);
for agent in freed {
inner.leases.remove(&agent);
}
Self::trim_history(inner, now_unix() - HISTORY_GRACE_SECS);
}