vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

build agents only from internal/meta; delete the hive's local meta flake

Browse source 
The hive no longer renders, locks or commits a meta flake. A rebuild's
MetaSync node picks the internal/meta commit and records it with the
hyperhive and config pins it holds; Prebuild, Swap and Create build from it,
and a head that does not pin the agent's current config fails the build
naming the agent. hive-priv's meta_rev is required.

Removed: meta.rs sync/lock/render, the manual-rebuild relock, the boot
sweep's hyperhive bump (MetaLock), the update-meta-inputs form and its
panel, MetaWindow, the hive's hyperhive_rev setting.

Kept: agent_set::refresh holds what sync_agents also did (topology.json, the
gateway's agent-socket map and nginx include), called from spawn, destroy,
rebuild and the migration.

The boot reconcile and the stale-start upgrade compare the agent's marker
with the hyperhive pin of internal/meta; the dashboard's deployed chip reads
the config marker a successful build writes.
This commit is contained in:
atlas 2026-10-25 13:33:15 +01:00
commit 4aa11b3a94
64 changed files with 1042 additions and 3793 deletions
Download patch file Download diff file Expand all files Collapse all files

434
hive-c0re/src/job_queue/exec.rs
View file

@ -1,9 +1,8 @@
//! Node executors — one async fn per [`NodeKind`], each a thin wrapper
//! over existing `lifecycle.rs` / `meta.rs` / `actions.rs` code. Node
//! over existing `lifecycle.rs` / `meta_rev.rs` / `actions.rs` code. Node
//! executors keep their own internal error handling where it exists
//! today (cold-start fallback inside `Reconcile`, non-fatal boot-time
//! lock bump inside the sweep `MetaLock`, warn-only forge sync in the
//! `Swap` tail); DAG-level failure handling is cancel-downstream in
//! today (cold-start fallback inside `Reconcile`, warn-only forge sync in
//! the `Swap` tail); DAG-level failure handling is cancel-downstream in
//! the queue.
use std::sync::{Arc, Mutex, OnceLock};
@ -61,18 +60,18 @@ pub(super) async fn run_node(
kind: &NodeKind,
) -> (super::JobBuilder, Result<()>) {
// The agent this node targets rides the payload — empty for the agentless
// kinds (`MetaLock`), which never read it.
// kinds (the sweeps), which never read it.
let agent = kind.agent();
// Every arm is `Result<()>`; the three that grow work declare into `builder`
// *synchronously*, after their own awaits have finished. Borrowing `&builder`
// Every arm is `Result<()>`; the one that grows work declares into `builder`
// *synchronously*, after its own awaits have finished. Borrowing `&builder`
// inside an `.await` would make this future non-`Send` (see above), so the
// growth executors return what to grow rather than taking the builder.
let result = match kind {
NodeKind::MetaSync { relock, .. } => run_meta_sync(coord, agent, *relock).await,
NodeKind::MetaSync { .. } => run_meta_sync(agent).await,
NodeKind::Prebuild { .. } => run_prebuild(agent, id).await,
NodeKind::Swap { .. } => run_swap(coord, agent, id).await,
NodeKind::RebuildBookkeeping { .. } => run_rebuild_bookkeeping(coord, agent).await,
NodeKind::Provision { .. } => run_provision(coord, agent).await,
NodeKind::Provision { .. } => run_provision(agent).await,
NodeKind::Create { .. } => run_create(agent).await,
NodeKind::DestroyContainer { .. } => run_destroy_container(coord, agent).await,
NodeKind::PurgeState { .. } => {
@ -83,24 +82,7 @@ pub(super) async fn run_node(
run_destroy_bookkeeping(coord, agent).await;
Ok(())
}
NodeKind::MetaLock {
sweep,
fanout,
inputs,
} => run_meta_lock(coord, *sweep, fanout.clone(), inputs)
.await
.map(|agents| {
// `sweep` is the whole difference: it relocks per-agent like a
// manual rebuild, and it drains agents that were mid-turn when
// the host came up. A cascade does neither. Decided here rather
// than returned, since `run_meta_lock` would only be deriving
// it from the `sweep` this call site already holds.
if *sweep {
super::templates::grown_graceful_rebuilds(&builder, &agents, true);
} else {
super::templates::grown_rebuilds(&builder, &agents, false);
}
}),
NodeKind::Reconcile { .. } => run_reconcile(coord, agent).await.map(|sub| {
if let Some(kind) = sub {
super::templates::fanned_out_mechanical(&builder, kind);
@ -258,12 +240,12 @@ async fn run_wanted_pull(coord: &Arc<Coordinator>) -> Result<()> {
crate::workers::wanted::pull(coord).await
}
/// Rerender the meta flake from whatever containers still exist on disk.
/// Idempotent — a no-op when nothing changed. Lives here because the destroy
/// tail is its only caller; it moved with `destroy` when that became a DAG.
async fn sync_meta_after_lifecycle(coord: &Coordinator) -> Result<()> {
/// Refresh the host-side agent-set files from whatever containers still exist
/// on disk. Idempotent. Lives here because the destroy tail is its only
/// caller; it moved with `destroy` when that became a DAG.
async fn refresh_agent_set_after_lifecycle() -> Result<()> {
let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&coord.hive_env(), &agents).await
crate::agent_set::refresh(&agents).await
}
/// `nixos-container destroy`, then drop the agent from the roster and clear
@ -319,15 +301,13 @@ async fn run_purge_state(agent: &str) {
/// Post-destroy bookkeeping. Infallible by construction: every step is
/// warn-and-continue, because the destroy it follows has already succeeded and
/// none of this is undoable — a failed meta sync or power-store write is a
/// none of this is undoable — a failed agent-set refresh or power-store write is a
/// bookkeeping drift to log, not a reason to red a DAG whose container is
/// already gone.
async fn run_destroy_bookkeeping(coord: &Arc<Coordinator>, agent: &str) {
// Meta flake: drop the agent's input + nixosConfiguration so a future spawn
// under the same name re-seeds cleanly, and so the meta lock doesn't
// reference a vanished applied repo.
if let Err(e) = sync_meta_after_lifecycle(coord).await {
tracing::warn!(error = ?e, %agent, "meta sync after destroy failed");
// Drop the agent from the topology and the gateway files.
if let Err(e) = refresh_agent_set_after_lifecycle().await {
tracing::warn!(error = ?e, %agent, "agent_set refresh after destroy failed");
}
// Drop the durable power intent — a future agent of the same name seeds
// fresh from its observed state.
@ -380,21 +360,16 @@ fn run_set_wanted(coord: &Arc<Coordinator>, agent: &str, up: bool) -> Result<()>
Ok(())
}
/// The rebuild's meta preamble: runtime-dir prep, an idempotent meta
/// `sync_agents`, and the optional per-agent relock. Runs under the deploy
/// window (`Resource::MetaWindow`, held by the scheduler for this
/// node) so its commits can never land inside another node's staged
/// prepare→finalize window.
/// The rebuild's preamble: runtime-dir prep, the agent-set refresh, and the
/// `internal/meta` commit `Prebuild` and `Swap` build from.
///
/// Deliberately a separate node from the [`run_prebuild`] it feeds: that
/// build takes minutes and only *reads* the store, so keeping the global
/// window off it is what lets rebuilds of different agents overlap.
async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Result<()> {
/// Fails when `internal/meta` does not pin the agent's current config yet:
/// there is no other flake to build it from.
async fn run_meta_sync(name: &str) -> Result<()> {
// Runs while the agent is still up — the runtime dir and MCP listener
// already exist. Use the pure path accessor; no need to re-register the
// listener (event-driven: registered at start/create).
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir)?;
// A failed read keeps the archetype last stored, so the agent is built as
// it was.
@ -402,28 +377,18 @@ async fn run_meta_sync(coord: &Arc<Coordinator>, name: &str, relock: bool) -> Re
tracing::warn!(%name, error = ?e, "rebuild: reading the swarm's archetype failed; keeping the stored one");
}
crate::lifecycle::prepare_rebuild_dirs(name, &paths).await?;
// Idempotent meta sync so a manual rebuild can also recover from a
// divergent meta repo; then bump just this agent's input. `relock =
// false` only for meta-update cascade children, where re-locking
// would revert the bump the cascade just committed.
let agents = crate::lifecycle::agents_for_meta_listing().await?;
crate::meta::sync_agents(&hive, &agents).await?;
// An archetype agent has no `agent-<name>` input to relock.
if relock && crate::archetypes::of(name)?.is_none() {
crate::meta::lock_update_for_rebuild(name).await?;
}
// The commit of the swarm's `internal/meta` that `Prebuild` and `Swap`
// build this agent from, resolved once here so both build the same flake;
// none leaves them on the hive's own flake rendered above.
let meta_rev = crate::meta_rev::resolve(name).await;
crate::meta_rev::record(name, meta_rev.as_deref())?;
crate::agent_set::refresh(&agents).await?;
// Resolved once here so `Prebuild` and `Swap` build the same flake.
let build = crate::meta_rev::resolve(name).await?;
crate::meta_rev::record(name, &build)?;
Ok(())
}
/// Out-of-band toplevel build while the container keeps serving: warm
/// `system.build.toplevel` so the later `Swap` hits cache and skips
/// straight to the profile-swap, against a meta repo the upstream
/// `MetaSync` node has already synced. The warm build is skipped when the
/// straight to the profile-swap, from the `internal/meta` commit the upstream
/// `MetaSync` node recorded. The warm build is skipped when the
/// container is already down: its only purpose is to shrink the swap's
/// downtime window, so a stopped agent (no uptime to preserve) doesn't
/// pay the double eval — `Swap` builds inline instead.
@ -433,10 +398,8 @@ async fn run_prebuild(name: &str, id: NodeId) -> Result<()> {
// stopped agent has no uptime to preserve, so skip the (expensive)
// eval and let the downstream `Swap` build inline.
if crate::lifecycle::is_running(name).await {
let flake_ref = match crate::meta_rev::recorded(name) {
Some(rev) => crate::meta_rev::flake_ref(name, &rev),
None => format!("{}#{name}", crate::paths::meta_root().display()),
};
let build = crate::meta_rev::recorded_or_err(name)?;
let flake_ref = crate::meta_rev::flake_ref(name, &build.rev);
crate::lifecycle::prebuild_toplevel(name, &flake_ref, Some(id.get())).await?;
}
Ok(())
@ -453,10 +416,9 @@ async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir)?;
let meta_rev = crate::meta_rev::recorded(name);
let build = crate::meta_rev::recorded_or_err(name)?;
let result =
crate::lifecycle::swap_update(name, &hive, &paths, meta_rev.as_deref(), Some(id.get()))
.await;
crate::lifecycle::swap_update(name, &hive, &paths, &build.rev, Some(id.get())).await;
// On success the Ok-only bookkeeping tail (rev marker, forge/matrix
// sync, kick, rescan, snapshot) runs in the sibling `RebuildBookkeeping` node,
// which deps `AfterOk(Swap)`. On failure `RebuildBookkeeping` is cancel-cascaded
@ -476,10 +438,10 @@ async fn run_swap(coord: &Arc<Coordinator>, name: &str, id: NodeId) -> Result<()
/// build (build-slot-exempt); the agent lease taken at `Swap` is still held
/// (the whole chain up to `Reconcile` is one agent's subgraph).
async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
if let Some(rev) = crate::auto_update::current_flake_rev(&coord.hyperhive_flake)
&& let Err(e) = std::fs::write(crate::paths::applied_rev_marker(name), rev)
if let Some(build) = crate::meta_rev::recorded(name)
&& let Err(e) = crate::meta_rev::mark_deployed(name, &build)
{
tracing::warn!(%name, error = ?e, "write rev marker failed");
tracing::warn!(%name, error = ?e, "write rev markers failed");
}
// The `Rebuilt` manager event is emitted exactly once per agent by the DAG's
// `EmitRebuilt` tail — emitting ok here and letting a failed tail `Reconcile`
@ -497,138 +459,33 @@ async fn run_rebuild_bookkeeping(coord: &Arc<Coordinator>, name: &str) -> Result
// alongside `run_start`'s own kick for the same boot. `hive-agent` now
// self-announces once on its own startup instead — one message per
// actual boot, not one per DAG node that happens to run during it.
// Rescan so dashboards drop the "needs update" chip; lock bump →
// meta-inputs re-render.
// Rescan so dashboards drop the "needs update" chip.
coord.rescan_containers_and_emit().await;
crate::dashboard::emit_meta_inputs_snapshot(coord);
Ok(())
}
/// First-spawn pre-create provisioning: proposed/applied repos, state
/// subvolume, and the meta `sync_agents` registration. Runs under the
/// deploy window (it declares `Resource::MetaWindow`) so its commit can't
/// land inside another node's staged deploy window.
async fn run_provision(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
/// subvolume, and the agent-set refresh.
async fn run_provision(name: &str) -> Result<()> {
let agent_dir = crate::paths::agent_runtime_dir(name);
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir)?;
crate::lifecycle::provision_container(name, &hive, &paths).await?;
crate::lifecycle::provision_container(name, &paths).await?;
Ok(())
}
/// `nixos-container create` proper — the upstream `Provision` node
/// already registered the agent in meta, so this only reads the store
/// (no deploy-window gate needed, mirroring `Prebuild`'s build). Runtime
/// already registered the agent, so this only reads the store. Runtime
/// dir creation and MCP listener registration are deferred to the tail
/// `Reconcile` (`converge_start_preamble` + `register_agent`) so this
/// node stays purely "create", not "create + start".
/// node stays purely "create", not "create + start". Marks the agent as
/// running the pins it was created from.
async fn run_create(name: &str) -> Result<()> {
let meta_rev = crate::meta_rev::resolve(name).await;
crate::lifecycle::create_only(name, meta_rev.as_deref()).await?;
let build = crate::meta_rev::resolve(name).await?;
crate::lifecycle::create_only(name, &build.rev).await?;
crate::meta_rev::mark_deployed(name, &build)?;
Ok(())
}
/// Meta flake lock bump. Boot-sweep flavour is non-fatal (a failed
/// bump must not cancel the fan-out rebuilds — they proceed against
/// the current lock, exactly like today's sweep); the meta-update
/// flavour propagates errors, and a failed bump fans out nothing.
/// Returns the agents whose rebuild subgraphs the caller should grow into this
/// node, rather than declaring them here — the declaration has to happen outside
/// any `.await` (see [`run_node`]).
///
/// Only the agent list: *which* rebuild flavour to grow is a pure function of
/// `sweep`, which the caller passed in, so returning it too would be a round
/// trip rather than a decision.
async fn run_meta_lock(
coord: &Arc<Coordinator>,
sweep: bool,
fanout: Option<Vec<String>>,
inputs: &[String],
) -> Result<Vec<String>> {
if sweep {
if let Err(e) = crate::meta::lock_update_hyperhive().await {
tracing::warn!(error = ?e, "startup sweep: meta lock_update_hyperhive failed");
}
// Grow one rebuild subgraph per stale agent into *this* boot DAG
// (rooted on this `MetaLock`, so they build against the post-bump
// lock), rather than fanning out child DAGs. The caller grows them
// with the graceful flavour.
return Ok(fanout.unwrap_or_default());
}
let _progress = coord.meta_update_guard();
// Resolved before the lock bump so a failed container list aborts
// the update before anything is committed.
let cascade = match fanout {
Some(list) => validate_agent_names(list),
None => meta_update_cascade_agents(inputs).await?,
};
crate::meta::lock_update(inputs).await?;
// Lock file changed — meta-inputs panel re-renders.
crate::dashboard::emit_meta_inputs_snapshot(coord);
// Pull each cascade agent's own input too — an agent's config-repo main
// is trusted, so a meta-input bump is a reasonable place to also catch
// it up. Without this, a meta-input bump rebuilds every affected agent
// against whatever `applied/<name>` already happened to be locked to —
// normally current, but silently stale forever if a past deploy failed
// and nothing since retried it.
// An archetype agent has no config repo and no `agent-<name>` input, so
// it is left out here; it is still rebuilt with the cascade.
let archetypes = crate::archetypes::read_at(&crate::archetypes::path())?;
let custom: Vec<String> = cascade
.iter()
.filter(|name| !archetypes.contains_key(*name))
.cloned()
.collect();
catch_up_cascade_agents(
&custom,
crate::forge::fast_forward_applied_main,
|agent_inputs| async move { crate::meta::lock_update(&agent_inputs).await },
)
.await?;
// Grow one rebuild subgraph per affected agent into *this* meta-update
// DAG (rooted on this `MetaLock`, so they build against the post-bump
// lock), rather than fanning out child DAGs. `relock = false` — the
// cascade children must NOT re-lock, which would revert the bump this
// node just committed (the property the old `fanout_specs` meta-update
// branch encoded).
Ok(cascade)
}
/// Fast-forward each cascade agent's `applied/<name>/main` to its forge
/// config main, then relock every `agent-<name>` input in one call.
///
/// The declared `agent-<name>` input is the `applied/<name>` mirror itself,
/// and the cascade's rebuilds (`relock = false`) build whatever this relock
/// pins, so the mirror has to move first. The fast-forward is best-effort
/// per agent: a failed one leaves that agent relocked against whatever its
/// mirror already has. The relock is one combined call, so one broken input
/// fails it for the whole cascade.
///
/// The fast-forward and the relock are passed in so the order is testable
/// without nix or a forge.
async fn catch_up_cascade_agents<'a, FF, Relock>(
cascade: &'a [String],
mut fast_forward: impl FnMut(&'a str) -> FF,
relock: impl FnOnce(Vec<String>) -> Relock,
) -> Result<()>
where
FF: std::future::Future<Output = bool>,
Relock: std::future::Future<Output = Result<()>>,
{
if cascade.is_empty() {
return Ok(());
}
for name in cascade {
if !fast_forward(name).await {
tracing::warn!(
%name,
"meta-update cascade: applied/main fast-forward skipped or failed"
);
}
}
relock(cascade.iter().map(|name| format!("agent-{name}")).collect()).await
}
/// Idempotent power-converge *planner*: compare `wanted` (durable
/// intent) against observed state and, when they diverge, fan the
/// mechanical `Start` / `Stop` out as a first-class node appended to
@ -814,202 +671,3 @@ async fn run_write_dropin(coord: &Arc<Coordinator>, name: &str) -> Result<()> {
crate::lifecycle::write_dropins(name, &hive, &paths).await?;
Ok(())
}
/// Compute which agents a `nix flake update <inputs>` on the meta
/// flake affects — the fan-out set for `MetaUpdate` DAGs. Empty
/// `inputs` or any input under `hyperhive` → every container;
/// otherwise just the agents named by `agent-<name>` inputs.
/// Sorted by name — the parent field the old depth sort keyed on is
/// gone, and with every agent a root that sort already reduced to
/// this.
///
/// `inputs` is the caller-supplied flake-input-name list (the dashboard's
/// `POST /api/meta-update` form field, operator-supplied but not
/// otherwise validated) — the `agent-<name>` branch parses agent names
/// straight out of it, so each is validated through [`hive_types::Ident`]
/// before it ever reaches a filesystem path or a forge URL built from a
/// cascade agent's name. The `touched_hyperhive` branch's names need no
/// such filter: they come from `lifecycle::list()`, already real
/// container names by construction, not parsed out of caller input.
///
/// # Errors
///
/// The every-container branch could not list the containers.
pub async fn meta_update_cascade_agents(inputs: &[String]) -> Result<Vec<String>> {
let touched_hyperhive = inputs
.iter()
.any(|i| i == "hyperhive" || i.starts_with("hyperhive/"));
let touched_agents = parse_agent_input_names(inputs);
let mut names = if touched_hyperhive || inputs.is_empty() {
crate::lifecycle::agent_names(crate::lifecycle::list().await)
.context("listing containers for the meta-update cascade")?
} else {
touched_agents
};
names.sort();
Ok(names)
}
/// Parse `agent-<name>` input strings into validated agent names. Pure and
/// synchronous (no `lifecycle::list()` call) so the validation — the part
/// that actually matters for safety, since these names end up in
/// filesystem paths and forge URLs — is unit-testable without touching
/// live container state. Delegates the actual filtering to
/// [`validate_agent_names`] after stripping the `agent-` prefix.
fn parse_agent_input_names(inputs: &[String]) -> Vec<String> {
let candidates = inputs
.iter()
.filter_map(|i| i.strip_prefix("agent-"))
.map(|rest| rest.split('/').next().unwrap_or(rest).to_owned());
validate_agent_names(candidates)
}
/// Drop any name that isn't a well-formed [`hive_types::Ident`] — the same
/// validating parser every other agent-name wire field in this codebase
/// goes through, so a name that reaches
/// [`fast_forward_applied_main`](crate::forge::fast_forward_applied_main) or
/// an `agent-configs/<name>` forge URL built off it has already
/// passed the same bar as one that arrived over a socket. Not just a
/// helper for [`parse_agent_input_names`]: `run_meta_lock`'s `fanout`
/// parameter carries pre-computed agent names too, and while today's only
/// caller of that branch (the boot sweep, `sweep = true`) returns before
/// ever reaching the filesystem/git code below, nothing in the type
/// enforces that pairing — so this same filter also guards the
/// `Some(list)` arm, closing the gap for good rather than relying on it
/// staying incidental.
fn validate_agent_names(names: impl IntoIterator<Item = String>) -> Vec<String> {
names
.into_iter()
.filter(|name| {
let valid = hive_types::Ident::parse(name).is_ok();
if !valid {
tracing::warn!(%name, "meta-update cascade: dropping malformed agent name");
}
valid
})
.collect()
}
#[cfg(test)]
mod cascade_agent_tests {
use super::{catch_up_cascade_agents, parse_agent_input_names, validate_agent_names};
use std::cell::RefCell;
use std::collections::BTreeMap;
/// Runs [`catch_up_cascade_agents`] over `cascade` against a model of
/// the applied mirrors: every mirror starts at `"old"`, a fast-forward
/// moves it to `"new"` unless the agent is in `failing`. Returns the
/// mirror head the relock saw for each input it was given.
async fn relocked_heads(cascade: &[&str], failing: &[&str]) -> Vec<(String, &'static str)> {
let cascade: Vec<String> = cascade.iter().map(|&n| n.to_owned()).collect();
let heads = RefCell::new(
cascade
.iter()
.map(|n| (n.clone(), "old"))
.collect::<BTreeMap<_, _>>(),
);
let relocked = RefCell::new(Vec::new());
let fast_forward = |name: &str| {
let ok = !failing.contains(&name);
if ok {
heads.borrow_mut().insert(name.to_owned(), "new");
}
async move { ok }
};
let relock = |inputs: Vec<String>| {
for input in inputs {
let name = input.strip_prefix("agent-").expect("agent input");
let head = heads.borrow()[name];
relocked.borrow_mut().push((input, head));
}
async { anyhow::Ok(()) }
};
catch_up_cascade_agents(&cascade, fast_forward, relock)
.await
.expect("catch up");
relocked.into_inner()
}
#[tokio::test]
async fn the_cascade_relock_pins_the_fast_forwarded_head() {
assert_eq!(
relocked_heads(&["iris", "atlas"], &[]).await,
vec![
("agent-iris".to_owned(), "new"),
("agent-atlas".to_owned(), "new"),
]
);
}
#[tokio::test]
async fn a_failed_fast_forward_still_relocks_that_agent() {
assert_eq!(
relocked_heads(&["iris", "atlas"], &["iris"]).await,
vec![
("agent-iris".to_owned(), "old"),
("agent-atlas".to_owned(), "new"),
]
);
}
/// A relock with no inputs is a bare `nix flake update`, which bumps
/// every meta input.
#[tokio::test]
async fn an_empty_cascade_moves_nothing_and_relocks_nothing() {
let fast_forwarded = std::cell::Cell::new(false);
let relocked = std::cell::Cell::new(false);
let outcome = catch_up_cascade_agents(
&[],
|_| {
fast_forwarded.set(true);
async { true }
},
|_| {
relocked.set(true);
async { anyhow::Ok(()) }
},
)
.await;
outcome.expect("an empty cascade is not an error");
assert!(!fast_forwarded.get(), "nothing to fast-forward");
assert!(!relocked.get(), "an empty relock would bump every input");
}
#[test]
fn well_formed_agent_inputs_pass_through() {
let inputs = vec!["agent-iris".to_owned(), "agent-atlas".to_owned()];
assert_eq!(parse_agent_input_names(&inputs), vec!["iris", "atlas"]);
}
#[test]
fn path_traversal_in_an_agent_input_is_dropped() {
let inputs = vec!["agent-../../etc".to_owned(), "agent-iris".to_owned()];
assert_eq!(parse_agent_input_names(&inputs), vec!["iris"]);
}
#[test]
fn non_agent_inputs_are_ignored_not_misparsed() {
let inputs = vec!["hyperhive".to_owned(), "nixpkgs".to_owned()];
assert!(parse_agent_input_names(&inputs).is_empty());
}
#[test]
fn oversized_or_uppercase_names_are_dropped() {
let too_long = format!("agent-{}", "a".repeat(64));
let inputs = vec![too_long, "agent-Iris".to_owned(), "agent-iris".to_owned()];
assert_eq!(parse_agent_input_names(&inputs), vec!["iris"]);
}
#[test]
fn fanout_supplied_names_are_validated_too() {
// `run_meta_lock`'s `Some(list)` arm feeds `fanout` straight into
// this filter — pin that a bogus name in that pre-computed list
// gets dropped exactly like a parsed `agent-<name>` input would.
let names = vec![
"iris".to_owned(),
"../../etc".to_owned(),
"atlas".to_owned(),
];
assert_eq!(validate_agent_names(names), vec!["iris", "atlas"]);
}
}

4
hive-c0re/src/job_queue/mod.rs
View file

@ -1,7 +1,7 @@
//! Generic job-DAG queue + desired-state reconciliation — the host-side
//! wrapper over the domain-agnostic [`hive_jobq`] scheduler. Jobs are nodes in
//! per-request DAGs (see [`templates`]); the special cases (graceful-stop
//! watcher, deferred-start follow-up, meta-update cascade) collapse into DAG
//! watcher, deferred-start follow-up) collapse into DAG
//! *shapes* over the shared node primitives ([`model::NodeKind`]).
//!
//! [`hive_jobq`] owns the graph, the two-class resource pool, and the roll-up
@ -327,7 +327,7 @@ impl JobQueue {
// without declaring its lease, and they are work on that agent
// that the operator wants to see.
//
// Empty means an agentless container kind (`MetaLock`, `Dag`),
// Empty means an agentless kind (the sweeps and pulls),
// which targets no agent and lights nothing.
let agent = n.payload.agent();
if agent.is_empty() {

41
hive-c0re/src/job_queue/model.rs
View file

@ -16,14 +16,10 @@ pub use hive_host_sock::jobs::State;
use serde::Serialize;
/// The primitive operations — each kind maps to one executor fn in
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta.rs`
/// code. Concurrency is gated by two resource classes (see
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta_rev.rs`
/// code. Concurrency is gated by resource classes (see
/// [`Resource`](super::resource::Resource), declared per node where the node
/// is constructed rather than derived from its kind); the
/// meta *repo* is serialized by `meta::META_LOCK` inside the wrapped
/// functions themselves, which is why there is no `GitCommit` node —
/// a standalone commit node would open a dirty-working-tree window
/// between nodes that the fused `meta.rs` ops deliberately close.
/// is constructed rather than derived from its kind).
///
/// What each kind does, the resource/lease rules, and the DAG shapes they
/// compose into live in the node-inventory table and surrounding sections
@ -33,10 +29,9 @@ use serde::Serialize;
#[serde(tag = "kind", rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")]
pub enum NodeKind {
/// The rebuild's meta-repo preamble. `relock = false` only for
/// meta-update cascade rebuilds (re-locking would revert the bump the
/// cascade just committed).
MetaSync { agent: String, relock: bool },
/// The rebuild's preamble: runtime-dir prep and the `internal/meta`
/// commit the build reads.
MetaSync { agent: String },
/// Out-of-band toplevel build while the container keeps serving.
Prebuild { agent: String },
/// `nixos-container update` profile-swap.
@ -46,7 +41,7 @@ pub enum NodeKind {
/// First-spawn pre-create provisioning.
Provision { agent: String },
/// First-spawn `nixos-container create` proper. Assumes the
/// upstream `Provision` node already registered the agent in meta.
/// upstream `Provision` node already registered the agent.
Create { agent: String },
/// `nixos-container destroy` plus the un-registration that follows it.
///
@ -60,15 +55,7 @@ pub enum NodeKind {
PurgeState { agent: String },
/// The post-destroy bookkeeping tail.
DestroyBookkeeping { agent: String },
/// Meta flake lock bump. `sweep = false`: `meta::lock_update` with this
/// node's own `inputs`; `sweep = true`: `meta::lock_update_hyperhive`
/// (boot sweep, `inputs` empty). See _Notable collapses_ for how the
/// scheduler appends the fan-out `Rebuild` DAGs on completion.
MetaLock {
sweep: bool,
fanout: Option<Vec<String>>,
inputs: Vec<String>,
},
/// Idempotent power converge *planner*: fans a child
/// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than
/// doing the mechanical work itself.
@ -140,11 +127,7 @@ impl hive_jobq_wire::WireNode for NodeKind {
if !agent.is_empty() {
data.insert("agent".to_owned(), agent.into());
}
if let NodeKind::MetaLock { inputs, .. } = self
&& !inputs.is_empty()
{
data.insert("inputs".to_owned(), inputs.clone().into());
}
// Not in the payload at all — the build log is keyed on node identity
// in a side table, which is why `data` is handed the id.
if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id)) {
@ -160,8 +143,7 @@ impl hive_jobq_wire::WireNode for NodeKind {
impl NodeKind {
/// The agent this node targets, or `""` for agentless kinds
/// ([`NodeKind::MetaLock`] on the `hyperhive` pseudo-agent, and the
/// swarm-wide sweeps and pulls).
/// (the swarm-wide sweeps and pulls).
#[must_use]
pub fn agent(&self) -> &str {
match self {
@ -186,8 +168,7 @@ impl NodeKind {
| NodeKind::AgentWindow { agent }
| NodeKind::EmitRebuilt { agent, .. }
| NodeKind::SetWanted { agent, .. } => agent,
NodeKind::MetaLock { .. }
| NodeKind::ForgeSweep
NodeKind::ForgeSweep
| NodeKind::MatrixSweep
| NodeKind::KnowledgePull
| NodeKind::WantedPull => "",

8
hive-c0re/src/job_queue/power.rs
View file

@ -103,7 +103,7 @@ fn start_chain(
// Rebuild subtree chained behind the `SetWanted` head. `MetaSync`, the
// `AgentWindow` brace and `Reconcile` are their own group roots
// (top-level, per `rebuild_nodes`) — hence all four names.
let roots = rebuild_nodes(builder, agent, true, Some(wanted));
let roots = rebuild_nodes(builder, agent, Some(wanted));
vec![
wanted.guid(),
roots.meta_sync.guid(),
@ -287,14 +287,12 @@ pub async fn start_many(
coord: &Arc<Coordinator>,
agents: &[String],
) -> anyhow::Result<Vec<NodeId>> {
let current = crate::auto_update::current_flake_rev(&coord.hyperhive_flake);
let current = crate::meta_rev::current_hyperhive_pin().await;
let mut targets = Vec::with_capacity(agents.len());
for agent in agents {
let running = lifecycle::is_running(agent).await;
let stored = read_rev_marker(&crate::paths::applied_rev_marker(agent));
let stale = current
.as_ref()
.is_some_and(|rev| stored.as_deref() != Some(rev.as_str()));
let stale = !crate::meta_rev::marker_is_fresh(current.as_deref(), stored.as_deref());
if !running && stale {
tracing::info!(%agent, "start: rev stale + agent down — rebuild-then-start");
}

16
hive-c0re/src/job_queue/resource.rs
View file

@ -11,15 +11,13 @@
//!
//! All of a node's resource edges are acquired **atomically**
//! (`try_acquire_all`) — a node never holds one resource while waiting on
//! another, so the multi-resource nodes (a `MetaLock` wants a build slot *and*
//! the meta window) cannot deadlock against each other.
//! another, so the multi-resource nodes (the rebuild brace wants a build slot
//! *and* the agent lease) cannot deadlock against each other.
/// The two resource classes the queue gates concurrency on, as the crate's
/// The resource classes the queue gates concurrency on, as the crate's
/// generic resource type `R`. What each holds, who declares it, and the
/// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_
/// (build slots, the per-agent lease) and _Two further layers protect the
/// meta repo_ (the deploy window) sections — this enum stays the one-line
/// summary, not a second copy.
/// section — this enum stays the one-line summary, not a second copy.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum Resource {
/// One of the `buildSlots` permits, held by a nix-heavy node for its
@ -28,9 +26,7 @@ pub enum Resource {
/// The per-agent lifecycle lease — globally exclusive per agent across
/// all DAGs.
Agent(String),
/// The meta-repo mutation window — a global singleton held by any node
/// that mutates the meta repo, so two meta mutations never interleave.
MetaWindow,
/// The `/knowledge` working tree, held by every `KnowledgePull`. Two
/// overlapping `reset` / `clean` / `pull` passes there fail on
/// `.git/index.lock` and the remote-tracking ref lock.
@ -53,7 +49,7 @@ impl hive_jobq_wire::WireResource for Resource {
match self {
Resource::BuildSlot => "build-slot".to_owned(),
Resource::Agent(agent) => format!("agent:{agent}"),
Resource::MetaWindow => "meta-window".to_owned(),
Resource::KnowledgeTree => "knowledge-tree".to_owned(),
Resource::MatrixSweep => "matrix-sweep".to_owned(),
}

2
hive-c0re/src/job_queue/scheduler.rs
View file

@ -14,7 +14,7 @@
//! focused terminal node (`EmitRebuilt`), dispatched through `exec::run_node`
//! like any other node once the DAG settles.
//!
//! In-DAG growth (a `MetaLock` growing rebuild subgraphs, a `Reconcile` fanning
//! In-DAG growth (a `Reconcile` fanning
//! its `Start`/`Stop`) is declared onto the builder each node is handed, and
//! inserted as part of completing that node. Completion itself is not this
//! module's job any more: it happens *inside* the future

91
hive-c0re/src/job_queue/templates.rs
View file

@ -55,31 +55,6 @@ fn emit_rebuilt_tails(builder: &JobBuilder, agent: &str, roots: &[Handle<'_>]) {
);
}
/// Declare one rebuild subgraph per agent onto the builder a running
/// [`NodeKind::MetaLock`] was handed.
///
/// **Into the emitter's own builder, not as new DAGs.** Growing in-DAG is what
/// roots each subgraph on the `MetaLock`, so the whole sweep (or meta-update
/// cascade) stays one unit of work the operator can watch and cancel, and every
/// rebuild builds against the lock the emitter just bumped.
///
/// Same reason as [`fanned_out_mechanical`] for living here: this was the
/// second construction site declaring nodes inline in an executor.
pub(crate) fn grown_rebuilds(builder: &JobBuilder, agents: &[String], relock: bool) {
for agent in agents {
rebuild_nodes(builder, agent, relock, None);
}
}
/// As [`grown_rebuilds`], but each agent gets its `Signal` → `Drain` window
/// before being stopped. The boot sweep's flavour: it stops agents that were
/// mid-turn when the host came up, so they drain rather than being cut off.
pub(crate) fn grown_graceful_rebuilds(builder: &JobBuilder, agents: &[String], relock: bool) {
for agent in agents {
graceful_rebuild_nodes(builder, agent, relock, None);
}
}
/// Declare the mechanical node a [`NodeKind::Reconcile`] planner fans out
/// (`Start` / `Stop`) onto the builder it was handed while running.
///
@ -162,7 +137,7 @@ pub(crate) fn pause_quiesce<'a>(builder: &'a JobBuilder, agent: &str) -> Handle<
/// cover every node in the subgraph without the caller knowing its shape.
#[derive(Debug, Clone, Copy)]
pub(crate) struct RebuildRoots<'a> {
/// The meta-repo preamble.
/// The preamble.
pub meta_sync: Handle<'a>,
/// The brace holding the agent lease and the build slot — its roll-up
/// carries the whole mechanical subtree (`Prebuild`, the quiesce chain,
@ -182,13 +157,9 @@ impl<'a> RebuildRoots<'a> {
/// The rebuild node subtree — three group roots (`MetaSync`, the `AgentWindow`
/// brace, `Reconcile`). `after`, when given, is the node it chains behind. The
/// shape itself is in `docs/scheduler/coordinator.md`; the code below is the source of
/// truth for it, so only the three choices a reader would otherwise undo are
/// truth for it, so only the two choices a reader would otherwise undo are
/// called out here:
///
/// - **`MetaSync` is a sibling root, not the brace's parent** — it owns the
/// *global* `MetaWindow`, and a resource is held across the holder's whole
/// subtree, so parenting the rebuild under it would serialise every agent's
/// nix build behind one hive-wide window.
/// - **`StopForUpdate` waits on `Prebuild` as well as `Drain`** — running the
/// drain early is the win; taking the container *down* early is pure downtime.
/// - **`Reconcile` is a top-level root, not a child** — so it survives the
@ -198,15 +169,12 @@ impl<'a> RebuildRoots<'a> {
fn rebuild_subtree<'a>(
builder: &'a JobBuilder,
agent: &str,
relock: bool,
graceful: bool,
after: Option<Handle<'a>>,
) -> RebuildRoots<'a> {
let a = || agent.to_owned();
let mut meta_sync = builder
.node(NodeKind::MetaSync { agent: a(), relock })
.needs(Resource::MetaWindow);
let mut meta_sync = builder.node(NodeKind::MetaSync { agent: a() });
if let Some(after) = after {
meta_sync = meta_sync.after_ok(after);
}
@ -263,17 +231,15 @@ fn rebuild_subtree<'a>(
}
}
/// The rebuild subtree, stopping the agent outright — the shape five of the six
/// call sites want. `relock` re-locks the meta flake inside `MetaSync`; `after`,
/// when given, is the node this subgraph chains behind. See
/// [`rebuild_subtree`] for the structure.
/// The rebuild subtree, stopping the agent outright — the shape most call
/// sites want. `after`, when given, is the node this subgraph chains behind.
/// See [`rebuild_subtree`] for the structure.
pub(crate) fn rebuild_nodes<'a>(
builder: &'a JobBuilder,
agent: &str,
relock: bool,
after: Option<Handle<'a>>,
) -> RebuildRoots<'a> {
rebuild_subtree(builder, agent, relock, false, after)
rebuild_subtree(builder, agent, false, after)
}
/// As [`rebuild_nodes`], but the agent gets a `Signal` → `Drain` window to
@ -286,17 +252,15 @@ pub(crate) fn rebuild_nodes<'a>(
pub(crate) fn graceful_rebuild_nodes<'a>(
builder: &'a JobBuilder,
agent: &str,
relock: bool,
after: Option<Handle<'a>>,
) -> RebuildRoots<'a> {
rebuild_subtree(builder, agent, relock, true, after)
rebuild_subtree(builder, agent, true, after)
}
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
/// noops when already down; the tail `Reconcile` auto-noops the start
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
/// leaves it stopped). `relock = false` only for meta-update cascade
/// children.
/// leaves it stopped).
///
/// Closed by an [`NodeKind::EmitRebuilt`] tail edged onto all three group-roots
/// (`MetaSync`, the `AgentWindow` brace, `Reconcile`) — the brace's roll-up
@ -307,8 +271,8 @@ pub(crate) fn graceful_rebuild_nodes<'a>(
///
/// Returns those three roots, so a caller that needs to wait on the rebuild can
/// name them.
pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq::NodeGuid> {
let roots = rebuild_nodes(builder, agent, relock, None);
pub fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
let roots = rebuild_nodes(builder, agent, None);
emit_rebuilt_tails(builder, agent, &roots.all());
vec![
roots.meta_sync.guid(),
@ -318,7 +282,7 @@ pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq
}
/// First deploy of an agent this hive has never seen, asked for by the swarm:
/// `Provision` (proposed/applied repos, state subvolume, meta registration)
/// `Provision` (proposed/applied repos, state subvolume, agent-set registration)
/// then `Create` (`nixos-container create`), drop-in write, then `Reconcile`
/// starts the container (`wanted = Up`, seeded by
/// `swarm_status::queue_first_deploy`). All-or-nothing: `Provision`
@ -330,9 +294,7 @@ pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq
/// Returns the group root so a caller can wait on the whole subtree.
pub fn first_deploy(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
let a = || agent.to_owned();
let provision = builder
.node(NodeKind::Provision { agent: a() })
.needs(Resource::MetaWindow);
let provision = builder.node(NodeKind::Provision { agent: a() });
let create = builder
.node(NodeKind::Create { agent: a() })
.needs(Resource::BuildSlot)
@ -403,31 +365,10 @@ pub fn destroy(builder: &JobBuilder, agent: &str, purge: bool) {
.after_ok(last);
}
/// Meta-input lock bump. The `MetaLock` executor grows one rebuild subgraph
/// per affected agent into *this same* DAG on completion (declared onto the
/// builder it was handed) — appended *after* the bump lands so their prebuilds
/// run against the post-bump lock, and a failed bump appends nothing
/// (replacing the old fan-out-child-DAGs dance).
/// `transient = Rebuilding` because those appended subgraphs are rebuilds:
/// it's applied per-agent at claim time (the `MetaLock` head needs no lease,
/// so the "hyperhive" pseudo-agent gets no pill), giving each cascade agent
/// crash-watch suppression during its `Swap` — the property the old child
/// `Rebuild` DAGs carried via their own transient.
pub fn meta_update(builder: &JobBuilder, inputs: Vec<String>) {
let _lock = builder
.node(NodeKind::MetaLock {
sweep: false,
fanout: None,
inputs,
})
.needs(Resource::BuildSlot)
.needs(Resource::MetaWindow);
}
// The boot is assembled inline in `workers/auto_update.rs::submit_boot_tree`
// as ONE `Boot` DAG (a sweep `MetaLock` root that grows rebuild subgraphs
// in-DAG, plus a `Reconcile` root per drifted agent) — no anchor node and no
// per-agent child DAGs.
// as ONE `Boot` DAG (a graceful rebuild subgraph per stale agent, plus a
// `Reconcile` root per drifted agent) — no anchor node and no per-agent child
// DAGs.
/// One matrix user/space sweep. Holding [`Resource::MatrixSweep`] is what keeps
/// two passes from overlapping, whichever caller submitted each.

82
hive-c0re/src/job_queue/tests.rs
View file

@ -51,7 +51,7 @@ fn insert_named(
}
fn rebuild(builder: &JobBuilder, agent: &str) -> Vec<hive_jobq::NodeGuid> {
templates::rebuild(builder, agent, true)
templates::rebuild(builder, agent)
}
/// Restart shape with every agent treated as **running** — the online
@ -468,7 +468,7 @@ fn rebuild_chain_is_declared_serial() {
fn graceful_rebuild_chain_drains_before_stopping() {
let q = JobQueue::new(1);
insert(&q, |builder| {
templates::graceful_rebuild_nodes(builder, "agent-a", true, None);
templates::graceful_rebuild_nodes(builder, "agent-a", None);
});
// Asserted as full rows, not just kinds: the kind list is identical whether
// the quiesce chain runs beside the build or nested under it, so a
@ -515,8 +515,8 @@ fn graceful_rebuild_chain_drains_before_stopping() {
}
/// The non-graceful shape is the default everywhere except the boot sweep:
/// a manual rebuild, a meta-update cascade child and a deploy must NOT spend a
/// drain window, so `StopForUpdate` waits only on `Prebuild`.
/// a manual rebuild and a deploy must NOT spend a drain window, so
/// `StopForUpdate` waits only on `Prebuild`.
#[test]
fn non_graceful_rebuild_has_no_signal_or_drain() {
// Read the shape off the queue rather than out of a node list: a declared
@ -524,7 +524,7 @@ fn non_graceful_rebuild_has_no_signal_or_drain() {
// observable where it matters — in what the scheduler runs.
let q = JobQueue::new(1);
insert(&q, |builder| {
templates::rebuild_nodes(builder, "agent-a", true, None);
templates::rebuild_nodes(builder, "agent-a", None);
});
assert_eq!(
declared_shape(&q)
@ -576,10 +576,10 @@ fn rebuild_chain_declares_its_resources_on_the_brace() {
res("reconcile"),
],
[
// The meta preamble takes the global window and *nothing else* —
// no slot (it does no nix work) and no lease. It stays a sibling
// root so that hive-global window is not held across any build.
vec![Resource::MetaWindow],
// The preamble declares nothing — no slot (it does no nix work) and
// no lease. It stays a sibling root so nothing it holds is held
// across a build.
vec![],
// The brace takes both, atomically, and holds them for its whole
// subtree. Hoisting the slot here is not a widening: it already
// spanned the entire rebuild when `Prebuild` held it, because a unit
@ -782,44 +782,43 @@ fn boot_sweep_nodes_declare_their_own_resources() {
// nothing in this module covered it. When resource derivation moved to the
// construction sites, this path was missed and both kinds silently declared
// nothing — dropping the agent lease a boot `Reconcile` needs to not race
// another DAG's container ops, and letting the sweep `MetaLock` land its
// meta commit inside another node's staged deploy window. Nothing failed to
// compile; only an exhaustive caller list would have caught it.
// another DAG's container ops. Nothing failed to compile; only an
// exhaustive caller list would have caught it.
let q = JobQueue::new(4);
insert(&q, |builder| {
crate::workers::auto_update::boot_nodes(
builder,
true,
vec!["stale-agent".to_owned()],
&["stale-agent".to_owned()],
vec!["drifted-agent".to_owned()],
);
});
let mut lock = declared_resources(&q, node_of(&q, "meta_lock"));
lock.sort_by_key(|r| format!("{r:?}"));
let mut window = declared_resources(&q, node_of(&q, "agent_window"));
window.sort_by_key(|r| format!("{r:?}"));
assert_eq!(
lock,
vec![Resource::BuildSlot, Resource::MetaWindow],
"the sweep MetaLock runs a nix lock bump and commits to meta"
window,
vec![
Resource::Agent("stale-agent".to_owned()),
Resource::BuildSlot
],
"a swept rebuild holds the agent's lease and a build slot"
);
// Two `reconcile` nodes exist: the swept agent's recovery tail and the
// drifted agent's own.
assert_eq!(
declared_resources(&q, node_of(&q, "reconcile")),
vec![Resource::Agent("drifted-agent".to_owned())],
declared_resources_of_kind(&q, "reconcile"),
vec![
vec![Resource::Agent("drifted-agent".to_owned())],
vec![Resource::Agent("stale-agent".to_owned())],
],
"a boot Reconcile touches the container, so it holds that agent's lease"
);
}
/// Crash-watch suppression for a cascade rebuild, which the deleted half of
/// `meta_update_grows_cascade_in_dag` used to assert via a DAG-level
/// `transient` field on the submit-time spec (both the field and the spec type
/// are gone).
///
/// The property is unchanged — a container going down under a rebuild must not
/// read as a crash — but it is no longer a DAG-level declaration: each node
/// answers for itself, so the assertion moves to the nodes a cascade actually
/// runs. Kept as its own test rather than dropped, because it is the *property*
/// that mattered, not the field that used to carry it.
/// Crash-watch suppression for a rebuild: a container going down under a
/// rebuild must not read as a crash. Each node answers for itself, so the
/// assertion is on the nodes a rebuild actually runs.
#[test]
fn rebuild_chain_nodes_suppress_crash_watch() {
for kind in [
@ -990,26 +989,15 @@ fn a_fanned_out_mechanical_node_declares_its_agent_lease() {
);
}
/// A running `MetaLock` grows one rebuild subgraph per agent into **its own
/// DAG**, rooted on itself — not as child DAGs. That is what keeps a boot sweep
/// (or a meta-update cascade) one unit of work, with every rebuild building
/// against the lock the emitter just bumped.
///
/// Replaces `grown_subgraph_roots_on_emitter_and_rebases_local_deps` and
/// `meta_update_grows_cascade_in_dag`, which differed only in which rebuild
/// flavour they grew and each minted a builder by hand to simulate the graft.
/// What they were checking is the `grown_*_rebuilds` templates, so this calls
/// one — the boot sweep's, since that is the caller that grows a graceful one.
///
/// That the grafted work lands under the emitter, and that the emitter parks in
/// `Finishing` until it settles, is `hive_jobq`'s
/// (`a_completing_node_grows_the_work_it_declared`).
/// The boot sweep declares one rebuild subgraph per stale agent into **one
/// DAG**, each an independent root — not as child DAGs. That keeps a boot sweep
/// one unit of work the operator can watch and cancel.
#[test]
fn a_meta_lock_grows_one_rebuild_subgraph_per_agent() {
fn the_boot_sweep_declares_one_rebuild_subgraph_per_stale_agent() {
let q = JobQueue::new(4);
let agents = vec!["alice".to_owned(), "bob".to_owned()];
insert(&q, |builder| {
templates::grown_graceful_rebuilds(builder, &agents, true);
crate::workers::auto_update::boot_nodes(builder, &agents, Vec::new());
});
// One chain per agent, each an independent group root — so the two rebuild