Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re re-converges wanted state whenever its watch attaches

pull waits (bounded) for the swarm queue connection instead of failing at boot while the client is still connecting. The watch runs one pull after every attach, including re-attaches, and a failed attach or ended stream is retried with a bounded backoff instead of waiting for the next boot. A first deploy is not queued again while one for the agent is queued or running.

An agent whose first deploy was lost to a c0re restart is now deployed on the next boot, or the next watch re-attach, without anyone re-declaring it.
This commit is contained in:
atlas 2026-10-20 21:43:22 +02:00 • committed by mara
commit 784507f554
5 changed files with 395 additions and 70 deletions

View file

@ -190,6 +190,32 @@ impl JobQueue {
Ok(named)
}
/// Queue `agent`'s first deploy unless one is already queued or running.
///
/// `None` means a `Provision` for the agent is still live. The check and
/// the insert share one lock, so two callers racing (the wanted-state pull
/// and the watch, or the deploy event) queue one deploy between them. A
/// deploy that finished or failed does not count: a lost one is retried.
///
/// # Errors
/// Propagates a graph-insert error.
pub fn insert_first_deploy_once(&self, agent: &str) -> anyhow::Result<Option<Vec<NodeId>>> {
let mut inner = self.lock();
let live = inner.graph().nodes().any(|n| {
!n.state.is_terminal()
&& matches!(&n.payload, model::NodeKind::Provision { agent: a } if a == agent)
});
if live {
return Ok(None);
}
let named = inner
.insert_job(None, |b| templates::first_deploy(b, agent))
.map_err(|e| anyhow::anyhow!("job_queue: graph insert failed: {e}"))?;
drop(inner);
self.notify.notify_one();
Ok(Some(named))
}
/// The scheduler itself, for `hive_jobq`'s run-loop seam
/// (`Scheduler::claim_next`), which takes exactly this type.
///

View file

@ -1537,3 +1537,29 @@ fn each_sweep_holds_its_own_resource() {
vec![vec![Resource::KnowledgeTree]]
);
}
/// The pull, the watch and the deploy event can all see one declaration, so a
/// second first deploy for an agent that has one queued must not be inserted.
#[test]
fn a_first_deploy_is_not_queued_twice_for_one_agent() {
let q = JobQueue::new(1);
let first = q.insert_first_deploy_once("newbie").expect("insert");
assert!(first.is_some(), "control: the first deploy is queued");
let kinds = || declared_shape(&q).len();
let queued = kinds();
assert!(
q.insert_first_deploy_once("newbie")
.expect("insert")
.is_none(),
"a live first deploy for the same agent blocks another"
);
assert_eq!(kinds(), queued, "nothing was added");
assert!(
q.insert_first_deploy_once("other")
.expect("insert")
.is_some(),
"control: another agent's deploy is independent"
);
}