vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(job-queue): build DAGs by naming nodes, not counting them

Browse source 
Every template built a `Vec<NodeSpec>` whose edges and parents were
positional indices into that vector, so a shape was expressed as
arithmetic: `base + 1`, `stop_root + 2`, `sfu + 1`, and a
`reconcile_index()` helper that read the emitted vector's length to find
out where its own last node had landed. `concat_subgraphs` existed
solely to rebase one per-agent subgraph's indices onto another's.

Templates now declare into a `hive_jobq::JobBuilder` and hold the
handles they get back, so an edge names the node it waits on. The
arithmetic is gone, and with it:

- `NodeSpec` and the job-queue's own index-based `Dep`.
- `insert_group`'s index resolution — it wraps `Scheduler::insert_job`.
- `concat_subgraphs` — per-agent chains share one builder and each keeps
  its own root, so independence is structural rather than computed.
- `reconcile_index` and `dep_index`.
- `templates::validate` and its petgraph toposort. It rejected dangling
  deps and cycles; both are now unrepresentable, since a handle only
  exists for an already-declared node and every edge therefore points
  backwards. (petgraph stays in the tree for `agent_config::topology`.)

`NodeOutput.append_subgraph` becomes `Vec<Job>`: an executor cannot
reach the queue, so it hands back declarations and the scheduler inserts
them under its own lock. That is what the in-DAG growth path always
wanted — a transferable declaration, not a vector of specs.

Resource declaration is unchanged in behaviour: the `templates::node`
helper applies `NodeKind::resource_deps()` at the construction site, so
every node still declares what its kind needs. Moving that declaration
to the call sites is #2818's job; this leaves it one place to delete.

Three tests went with the guard they covered — they hand-built malformed
specs out of indices, which is the representation that made those shapes
possible. Two more now read a DAG's shape off the queue rather than out
of a spec vector, which is where it is observable. The remaining 45
job-queue tests are unchanged and still pass: lease serialization,
roll-up, cancel-cascade, in-DAG growth and per-agent concurrency all
behave as before.
This commit is contained in:
atlas 2026-08-02 13:05:17 +02:00 • committed by mara
commit e7c3cf5a3d
9 changed files with 528 additions and 696 deletions
Download patch file Download diff file Expand all files Collapse all files

600
hive-c0re/src/job_queue/templates.rs
View file

@ -1,21 +1,19 @@
//! DAG shape builders — every operation as a template over the shared
//! node primitives — plus submit-time cycle validation (petgraph is
//! confined to this validation; the runtime store stays the plain
//! `Vec<Node>` + `deps`).
//! node primitives.
//!
//! Every node carries its own `agent` (there is no DAG-level agent) — the
//! `node` helper stamps each node's agent. This module holds the *pure*
//! shape builders (no I/O). The hive-wide **power ops** (`stop` / `start` /
//! `restart`) are NOT here: their per-agent shape depends on each agent's
//! live running state (an async `lifecycle::is_running` read), so they are
//! assembled dynamically in `submit.rs` out of the shared pure primitives
//! this module exports (`node`, `after_ok`, `rebuild_nodes`) — one
//! independent per-agent subgraph each, concurrent on its own lease, ONE
//! DAG for the whole hive-wide op. `stop`/`start` write the durable `wanted`
//! intent via a head `SetWanted(w)` node (holding the agent lease, so
//! intent+reconcile is atomic per-agent); `restart` writes no intent — it
//! bounces the container and lets the tail `Reconcile` converge to the
//! agent's existing `wanted`.
//! [`node`] helper stamps each node's agent and declares the resources that
//! node's kind needs. This module holds the *pure* shape builders (no I/O).
//! The hive-wide **power ops** (`stop` / `start` / `restart`) are NOT here:
//! their per-agent shape depends on each agent's live running state (an async
//! `lifecycle::is_running` read), so they are assembled dynamically in
//! `submit.rs` out of the shared pure primitives this module exports
//! ([`node`], [`rebuild_nodes`]) — one independent per-agent subgraph each,
//! concurrent on its own lease, ONE DAG for the whole hive-wide op.
//! `stop`/`start` write the durable `wanted` intent via a head `SetWanted(w)`
//! node (holding the agent lease, so intent+reconcile is atomic per-agent);
//! `restart` writes no intent — it bounces the container and lets the tail
//! `Reconcile` converge to the agent's existing `wanted`.
//!
//! ```text
//! rebuild(a): MetaSync(a) → Prebuild(a) → StopForUpdate(a) → Swap(a) →(ok) PostSwap(a) →(any) Reconcile(a)
@ -26,113 +24,77 @@
//! reparent(moves): Reparent(moves) [no rebuild — topology.json is read live]
//! ```
//!
//! Nodes are **named, not counted**: a template declares a node and holds the
//! handle it gets back, so an edge says which node it waits on instead of
//! computing where that node landed. There is no submit-time cycle validation
//! left to do — `hive_jobq`'s builder inserts in declaration order and rejects
//! a reference to a node declared later, so every edge points backwards and a
//! cycle is unrepresentable rather than merely rejected.
//!
//! For the dynamic power-op shapes (`stop` / `start` / `restart`, built from
//! live online/offline state), see `submit.rs`.
use anyhow::{Result, bail};
use hive_jobq::TerminalState;
use hive_jobq::{DepWhen, TerminalState};
use super::model::{DagSpec, NodeKind, PermPayload, Source};
use super::{Handle, Job};
use super::model::{DagSpec, Dep, NodeKind, NodeSpec, PermPayload, Source};
/// After-ok edge on the previous node — the common chain link. Shared with
/// the async power-op builders in `submit.rs` (which assemble per-agent
/// chains dynamically from live container state).
pub(crate) fn after_ok(on: u64) -> Vec<Dep> {
vec![Dep {
on,
when: DepWhen::AFTER_OK,
}]
}
/// `AfterOk` edges onto every one of a DAG's **group-roots** — the success
/// branch of a per-outcome tail pair, and the aggregator the failure branch
/// keys off.
/// Declare one node carrying `kind`, with the resources that kind needs.
///
/// Group-roots are the right granularity, not "every node": a root's state *is*
/// its subtree's roll-up, so edging the roots covers every descendant while
/// keeping the dep list small and stable as subtrees grow. Because every edge is
/// `AFTER_OK`, this node runs only if *all* of them succeeded — and is ruled out
/// ([`TerminalState::Skipped`]) the moment one doesn't, which is precisely the
/// signal [`on_elimination_of`] waits for.
pub(crate) fn after_ok_all(ons: &[u64]) -> Vec<Dep> {
ons.iter()
.map(|&on| Dep {
on,
when: DepWhen::AFTER_OK,
})
.collect()
}
/// `AFTER_ANY` edges onto every group-root — "wait for all of these to finish,
/// however they went". Ordering only; it accepts any outcome except the DAG
/// being dropped.
pub(crate) fn after_any_all(ons: &[u64]) -> Vec<Dep> {
ons.iter()
.map(|&on| Dep {
on,
when: DepWhen::AFTER_ANY,
})
.collect()
}
/// A single edge satisfied only when `on` was **ruled out** by its own edges.
/// The resource declaration is [`NodeKind::resource_deps`] applied at the
/// construction site — a build slot for nix-heavy kinds, the agent lease for
/// container-affecting ones, the global meta window for meta-mutating ones. A
/// node that declares a resource an ancestor already holds re-enters that
/// grant rather than taking a fresh unit, so declaring costs nothing.
///
/// Dependency edges are conjunctive, so "any one of these several nodes failed"
/// cannot be written directly. This is the composition that expresses it: point
/// the success branch at every root with [`after_ok_all`], then hang the failure
/// branch off *that* node's elimination. Exactly one of the pair ever runs.
///
/// Note it accepts `Skipped` and **not** `Cancelled`: if the whole DAG was
/// dropped before it started, the success branch is marked `Cancelled` directly
/// and this branch is ruled out too — a job nobody ran reports nothing.
pub(crate) fn on_elimination_of(on: u64) -> Vec<Dep> {
vec![Dep {
on,
when: DepWhen::of(&[TerminalState::Skipped]),
}]
}
/// A single edge satisfied only by the listed outcomes of `on` — for the
/// one-tail-per-outcome shape an approval DAG uses.
pub(crate) fn on_outcome(on: u64, outcomes: &[TerminalState]) -> Vec<Dep> {
vec![Dep {
on,
when: DepWhen::of(outcomes),
}]
/// The returned handle is where edges and grouping are declared, and is `Copy`
/// — naming a node as a dependency does not consume the ability to name it
/// again.
pub(crate) fn node(b: &Job, kind: NodeKind) -> Handle<'_> {
// Read the resources off the kind before handing it over — the payload is
// moved into the node, not cloned for it.
let resources = kind.resource_deps();
let mut handle = b.node(kind);
for (name, count) in resources {
handle = handle.needs_units(name, count);
}
handle
}
/// The `Rebuilt`-reporting tail pair for a rebuild-shaped DAG: the success node
/// gated on every group-root in `roots`, and the failure node gated on *its*
/// elimination. `base` is the spec index the pair starts at.
/// elimination.
///
/// Exactly one runs on a DAG that executed, and neither runs on one the operator
/// dropped — see [`on_elimination_of`].
fn emit_rebuilt_tails(agent: &str, roots: &[u64], base: u64) -> Vec<NodeSpec> {
/// dropped — see [`hive_jobq::NodeRef::on_elimination_of`].
fn emit_rebuilt_tails(b: &Job, agent: &str, roots: &[Handle<'_>]) {
let ok = roots.iter().fold(
node(
b,
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: true,
},
),
hive_jobq::NodeRef::after_ok,
);
// The failure branch needs *both*: the ok branch being ruled out (that is the
// "something went wrong" signal) **and** every root actually finished. The
// second half is easy to forget and gets the ordering wrong without it — a
// failed `Prebuild` eliminates the ok branch immediately, while the recovery
// `Reconcile` is still bringing the container back up, so reporting straight
// off the elimination would announce the failure mid-recovery.
let mut on_fail = after_any_all(roots);
on_fail.extend(on_elimination_of(base));
vec![
node(
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: true,
},
after_ok_all(roots),
),
let _failed = roots.iter().fold(
node(
b,
NodeKind::EmitRebuilt {
agent: agent.to_owned(),
ok: false,
},
on_fail,
),
]
)
.on_elimination_of(ok),
hive_jobq::NodeRef::after_any,
);
}
/// The approval-resolving tails for an approval-carrying DAG: one per outcome of
@ -140,48 +102,20 @@ fn emit_rebuilt_tails(agent: &str, roots: &[u64], base: u64) -> Vec<NodeSpec> {
///
/// The `Cancelled` node is what keeps a dropped approval DAG from dangling its
/// row forever — its edge is the only one [`super::JobQueue::cancel`] spares.
fn resolve_approval_tails(approval_id: i64, root: u64) -> Vec<NodeSpec> {
[
fn resolve_approval_tails(b: &Job, approval_id: i64, root: Handle<'_>) {
for outcome in [
TerminalState::Done,
TerminalState::Failed,
TerminalState::Cancelled,
]
.into_iter()
.map(|outcome| {
node(
] {
let _ = node(
b,
NodeKind::ResolveApproval {
approval_id,
outcome,
},
on_outcome(root, &[outcome]),
)
})
.collect()
}
/// Build one **top-level (group-root)** node — `parent = None`. `kind` carries
/// the agent it targets ([`NodeKind`] is the payload directly). Shared with
/// `submit.rs`'s dynamic power-op builders. A root owns whatever resource it
/// declares for its whole subtree; its descendants borrow it (agent-lease /
/// build-slot continuity). Ordering vs other nodes is `deps`; grouping is
/// `parent`.
pub(crate) fn node(kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
NodeSpec {
kind,
deps,
parent: None,
}
}
/// Build a **child** node whose structural parent is spec-index `parent`. The
/// child runs once its parent reaches `Finishing` (the parent gate), so it must
/// NOT `deps` on `parent` (dep-scope validation rejects a dep on one's own
/// parent). `deps` here order the child against its *siblings* only.
pub(crate) fn child(parent: u64, kind: NodeKind, deps: Vec<Dep>) -> NodeSpec {
NodeSpec {
kind,
deps,
parent: Some(parent),
.on_outcome(root, &[outcome]);
}
}
@ -198,28 +132,51 @@ pub(crate) struct RebuildOpts {
pub graceful: bool,
}
/// The rebuild node subtree (nested, three group roots). `base` is the spec
/// index of the first node (`MetaSync`). Structure:
/// - `MetaSync` (base+0, **root**): the meta-repo preamble (dir prep, agent
/// sync, optional relock). Owns the global `MetaWindow` — and *only* for its
/// own short duration, which is why it is a sibling root rather than
/// `Prebuild`'s parent: a resource is held across the holder's whole subtree,
/// so parenting the build under it would extend a hive-global window over
/// every rebuild's nix build.
/// - `Prebuild` (base+1, **root**): `AfterOk` `MetaSync`. Owns the build slot
/// for the whole mechanical subtree below it. Lease-exempt — the nix build
/// overlaps other DAGs on the same agent.
/// - the **stop root** (base+2, child of `Prebuild`): owns the agent lease and
/// runs once `Prebuild` reaches `Finishing` (parent gate). Non-graceful that
/// is `StopForUpdate` itself; graceful it is `Signal`, with `Drain` and then
/// The group-roots a [`rebuild_nodes`] subgraph exposes to its caller: what a
/// tail node edges onto, and what a follow-up node waits for.
///
/// Only the roots — a root's state *is* its subtree's roll-up, so these three
/// cover every node in the subgraph without the caller knowing its shape.
#[derive(Debug, Clone, Copy)]
pub(crate) struct RebuildRoots<'a> {
/// The meta-repo preamble.
pub meta_sync: Handle<'a>,
/// The build root — its roll-up carries the whole
/// `StopForUpdate` → `Swap` → `PostSwap` subtree.
pub prebuild: Handle<'a>,
/// The recovery/convergence tail root.
pub reconcile: Handle<'a>,
}
impl<'a> RebuildRoots<'a> {
/// The three roots as a slice, for edging a tail onto all of them.
fn all(self) -> [Handle<'a>; 3] {
[self.meta_sync, self.prebuild, self.reconcile]
}
}
/// The rebuild node subtree (nested, three group roots). `after`, when given, is
/// the node this subgraph chains behind. Structure:
/// - `MetaSync` (**root**): the meta-repo preamble (dir prep, agent sync,
/// optional relock). Owns the global `MetaWindow` — and *only* for its own
/// short duration, which is why it is a sibling root rather than `Prebuild`'s
/// parent: a resource is held across the holder's whole subtree, so parenting
/// the build under it would extend a hive-global window over every rebuild's
/// nix build.
/// - `Prebuild` (**root**): `AfterOk` `MetaSync`. Owns the build slot for the
/// whole mechanical subtree below it. Lease-exempt — the nix build overlaps
/// other DAGs on the same agent.
/// - the **stop root** (child of `Prebuild`): owns the agent lease and runs once
/// `Prebuild` reaches `Finishing` (parent gate). Non-graceful that is
/// `StopForUpdate` itself; graceful it is `Signal`, with `Drain` and then
/// `StopForUpdate` as its children so the lease stays continuous across the
/// whole stop — siblings would each take the lease separately and leave a gap
/// another DAG could claim the agent in, mid-bounce.
/// - `Swap` (child of `StopForUpdate`): borrows the agent lease from its
/// ancestors and the build slot from `Prebuild` — both continuous.
/// - `PostSwap` (child of `StopForUpdate`): the swap's Ok-only
/// bookkeeping tail (rev marker, forge/matrix sync, kick, rescan), `AfterOk`
/// its sibling `Swap`.
/// - `PostSwap` (child of `StopForUpdate`): the swap's Ok-only bookkeeping tail
/// (rev marker, forge/matrix sync, kick, rescan), `AfterOk` its sibling
/// `Swap`.
/// - `Reconcile` (**last, root**): `AfterAny` `Prebuild`, which rolls up
/// terminal only once its whole mechanical subtree (SFU→Swap→PostSwap) has
/// settled — so `Reconcile` runs after the swap regardless of outcome, and as
@ -228,56 +185,47 @@ pub(crate) struct RebuildOpts {
/// cancel-cascades `Prebuild`, i.e. terminal, so the tail still runs). It
/// takes a fresh lease; the tiny gap is harmless — `Reconcile` converges to
/// the persisted `wanted` idempotently.
pub(crate) fn rebuild_nodes(agent: &str, opts: RebuildOpts, base: u64) -> Vec<NodeSpec> {
pub(crate) fn rebuild_nodes<'a>(
b: &'a Job,
agent: &str,
opts: RebuildOpts,
after: Option<Handle<'a>>,
) -> RebuildRoots<'a> {
let a = || agent.to_owned();
let RebuildOpts { relock, graceful } = opts;
let mut nodes = vec![
node(
NodeKind::MetaSync { agent: a(), relock },
if base == 0 {
Vec::new()
} else {
after_ok(base - 1)
},
),
node(NodeKind::Prebuild { agent: a() }, after_ok(base)),
];
let mut meta_sync = node(b, NodeKind::MetaSync { agent: a(), relock });
if let Some(after) = after {
meta_sync = meta_sync.after_ok(after);
}
let prebuild = node(b, NodeKind::Prebuild { agent: a() }).after_ok(meta_sync);
// The stop root hangs off `Prebuild` and owns the agent lease for
// everything below it.
let stop_root = base + 2;
if graceful {
nodes.push(child(base + 1, NodeKind::Signal { agent: a() }, Vec::new()));
// everything below it. `StopForUpdate` parents the swap pair either way.
let stop_for_update = if graceful {
let signal = node(b, NodeKind::Signal { agent: a() }).part_of(prebuild);
// `Drain` is a *child* of `Signal`, so the parent gate already orders
// it — a child must not dep on its own parent (dep-scope).
nodes.push(child(stop_root, NodeKind::Drain { agent: a() }, Vec::new()));
nodes.push(child(
stop_root,
NodeKind::StopForUpdate { agent: a() },
after_ok(stop_root + 1),
));
let drain = node(b, NodeKind::Drain { agent: a() }).part_of(signal);
node(b, NodeKind::StopForUpdate { agent: a() })
.part_of(signal)
.after_ok(drain)
} else {
nodes.push(child(
base + 1,
NodeKind::StopForUpdate { agent: a() },
Vec::new(),
));
node(b, NodeKind::StopForUpdate { agent: a() }).part_of(prebuild)
};
let swap = node(b, NodeKind::Swap { agent: a() }).part_of(stop_for_update);
let _post_swap = node(b, NodeKind::PostSwap { agent: a() })
.part_of(stop_for_update)
.after_ok(swap);
let reconcile = node(b, NodeKind::Reconcile { agent: a() }).after_any(prebuild);
RebuildRoots {
meta_sync,
prebuild,
reconcile,
}
// Index of `StopForUpdate`, which parents the swap pair either way.
let sfu = if graceful { stop_root + 2 } else { stop_root };
nodes.push(child(sfu, NodeKind::Swap { agent: a() }, Vec::new()));
nodes.push(child(
sfu,
NodeKind::PostSwap { agent: a() },
after_ok(sfu + 1),
));
nodes.push(node(
NodeKind::Reconcile { agent: a() },
vec![Dep {
on: base + 1,
when: DepWhen::AFTER_ANY,
}],
));
nodes
}
/// The rebuild subgraph a [`NodeKind::DeployApply`] grows into its own DAG once
@ -288,7 +236,7 @@ pub(crate) fn rebuild_nodes(agent: &str, opts: RebuildOpts, base: u64) -> Vec<No
/// relocked and staged `flake.lock`, so the appended `MetaSync` must do the dir
/// prep + `sync_agents` *without* re-locking over it.
///
/// `FinalizeDeploy` waits on **two** siblings, which together reproduce the gate
/// `FinalizeDeploy` waits on **two** roots, which together reproduce the gate
/// the old fused node had around its inline `rebuild_no_meta` call:
/// - `AfterOk` `Prebuild` — a parent's state is its roll-up, so this is `Done`
/// only once `StopForUpdate` → `Swap` → `PostSwap` all are (a failed *or*
@ -304,40 +252,27 @@ pub(crate) fn rebuild_nodes(agent: &str, opts: RebuildOpts, base: u64) -> Vec<No
/// inside the `DeployWindow`'s subtree — so the `MetaWindow` this subgraph's
/// `MetaSync` and `FinalizeDeploy` declare is re-entered from the ancestor
/// already holding it rather than deadlocking against it.
pub(crate) fn deploy_rebuild_nodes(agent: &str, approval_id: i64) -> Vec<NodeSpec> {
let mut nodes = rebuild_nodes(
pub(crate) fn deploy_rebuild_nodes(agent: &str, approval_id: i64) -> Job {
let b = Job::new();
let roots = rebuild_nodes(
&b,
agent,
RebuildOpts {
relock: false,
graceful: false,
},
0,
None,
);
let reconcile = reconcile_index(&nodes, 0);
nodes.push(node(
let _finalize = node(
&b,
NodeKind::FinalizeDeploy {
agent: agent.to_owned(),
approval_id,
},
vec![
Dep {
on: 1,
when: DepWhen::AFTER_OK,
},
Dep {
on: reconcile,
when: DepWhen::AFTER_OK,
},
],
));
nodes
}
/// Spec index of the `Reconcile` root a [`rebuild_nodes`] subgraph ends on,
/// for callers that gate a tail on it. Read off the emitted list rather than
/// hard-coded, because the subgraph's length depends on [`RebuildOpts`].
fn reconcile_index(rebuild: &[NodeSpec], base: u64) -> u64 {
base + u64::try_from(rebuild.len()).unwrap_or(0).saturating_sub(1)
)
.after_ok(roots.prebuild)
.after_ok(roots.reconcile);
b
}
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
@ -352,41 +287,41 @@ fn reconcile_index(rebuild: &[NodeSpec], base: u64) -> u64 {
/// node. Edging `Reconcile` alone would not do: it is `AfterAny` `Prebuild`, so
/// it reaches `Done` even after a failed swap and the tail would report success.
pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> DagSpec {
let mut nodes = rebuild_nodes(
let job = Job::new();
let roots = rebuild_nodes(
&job,
agent,
RebuildOpts {
relock,
graceful: false,
},
0,
None,
);
let reconcile = reconcile_index(&nodes, 0);
let tail_base = u64::try_from(nodes.len()).unwrap_or(0);
nodes.extend(emit_rebuilt_tails(agent, &[0, 1, reconcile], tail_base));
emit_rebuilt_tails(&job, agent, &roots.all());
DagSpec {
source,
reason,
nodes,
job,
}
}
/// Approval-driven deploy (`MergeConfigPr`) as a phase subtree rather than the
/// single opaque node it used to be. Structure:
/// - `DeployWindow` (0, **root**): the resource holder — global meta window,
/// agent lease, build slot — held across every child below. No work of its
/// own; it reaches `Finishing` immediately and the children run inside it.
/// - `MergeVerify` (1, child): drift-gate + fetch + eval-verify. Mutates
/// nothing, so a failure here cancel-cascades its siblings with the forge and
/// the applied repo exactly as they were.
/// - `DeployApply` (2, child, `AfterOk` `MergeVerify`): the irreversible half —
/// - `DeployWindow` (**root**): the resource holder — global meta window, agent
/// lease, build slot — held across every child below. No work of its own; it
/// reaches `Finishing` immediately and the children run inside it.
/// - `MergeVerify` (child): drift-gate + fetch + eval-verify. Mutates nothing,
/// so a failure here cancel-cascades its siblings with the forge and the
/// applied repo exactly as they were.
/// - `DeployApply` (child, `AfterOk` `MergeVerify`): the irreversible half —
/// ff-merge + `prepare_deploy`. It doesn't rebuild inline; it grows
/// [`deploy_rebuild_nodes`] into this DAG as its own children, so the build
/// and the closing `FinalizeDeploy` are real nodes under the same window.
/// - `DeployTail` (3, child, `AfterAny` `DeployApply`): the compensation +
/// - `DeployTail` (child, `AfterAny` `DeployApply`): the compensation +
/// bookkeeping tail — rollback when a merge landed unfinalized, forge tag
/// mirror, PR failure comment (see [`NodeKind::DeployTail`]).
///
/// - `ResolveApproval` (4, **root**, `AfterAny` `DeployWindow`): resolves the
/// - `ResolveApproval` (**root**, `AfterAny` `DeployWindow`): resolves the
/// approval row. A root rather than another child, so it isn't inside the
/// window's resource subtree — it runs once the window has released the meta
/// window, lease and build slot. One edge suffices here: `DeployWindow` is the
@ -397,48 +332,47 @@ pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> Dag
/// leaves `flake.lock` staged-uncommitted for the build's whole duration.
pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec {
let a = || agent.to_owned();
let job = Job::new();
let window = node(
&job,
NodeKind::DeployWindow {
agent: a(),
approval_id,
},
);
let verify = node(
&job,
NodeKind::MergeVerify {
agent: a(),
approval_id,
},
)
.part_of(window);
let apply = node(
&job,
NodeKind::DeployApply {
agent: a(),
approval_id,
},
)
.part_of(window)
.after_ok(verify);
let _tail = node(
&job,
NodeKind::DeployTail {
agent: a(),
approval_id,
},
)
.part_of(window)
.after_any(apply);
resolve_approval_tails(&job, approval_id, window);
DagSpec {
source: Source::Approval,
reason,
nodes: vec![
node(
NodeKind::DeployWindow {
agent: a(),
approval_id,
},
Vec::new(),
),
child(
0,
NodeKind::MergeVerify {
agent: a(),
approval_id,
},
Vec::new(),
),
child(
0,
NodeKind::DeployApply {
agent: a(),
approval_id,
},
after_ok(1),
),
child(
0,
NodeKind::DeployTail {
agent: a(),
approval_id,
},
vec![Dep {
on: 2,
when: DepWhen::AFTER_ANY,
}],
),
]
.into_iter()
.chain(resolve_approval_tails(approval_id, 0))
.collect(),
job,
}
}
@ -449,15 +383,17 @@ pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec
/// in the queue tests); production paths no longer emit a bare reconcile.
#[cfg(test)]
pub fn reconcile_only(agent: &str, source: Source, reason: String) -> DagSpec {
let job = Job::new();
let _reconcile = node(
&job,
NodeKind::Reconcile {
agent: agent.to_owned(),
},
);
DagSpec {
source,
reason,
nodes: vec![node(
NodeKind::Reconcile {
agent: agent.to_owned(),
},
Vec::new(),
)],
job,
}
}
@ -473,53 +409,56 @@ pub fn reconcile_only(agent: &str, source: Source, reason: String) -> DagSpec {
/// `AfterAny` onto `Provision` — the DAG's only other group-root, so its roll-up
/// already carries the whole cascade.
pub fn spawn(agent: &str, approval_id: i64, reason: String) -> DagSpec {
let a = || agent.to_owned();
let job = Job::new();
let provision = node(&job, NodeKind::Provision { agent: a() });
let create = node(&job, NodeKind::Create { agent: a() }).part_of(provision);
let dropin = node(&job, NodeKind::WriteDropin { agent: a() }).part_of(create);
let _reconcile = node(&job, NodeKind::Reconcile { agent: a() })
.part_of(create)
.after_ok(dropin);
resolve_approval_tails(&job, approval_id, provision);
DagSpec {
source: Source::Approval,
reason,
nodes: {
let a = || agent.to_owned();
vec![
node(NodeKind::Provision { agent: a() }, Vec::new()),
child(0, NodeKind::Create { agent: a() }, Vec::new()),
child(1, NodeKind::WriteDropin { agent: a() }, Vec::new()),
child(1, NodeKind::Reconcile { agent: a() }, after_ok(2)),
]
.into_iter()
.chain(resolve_approval_tails(approval_id, 0))
.collect()
},
job,
}
}
/// Perm change: commit the JSON file(s), then the rebuild subgraph so
/// the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes
/// effect in the container. Group-roots are `WritePermFile`(0) plus the rebuild
/// subgraph's `MetaSync`(1) / `Prebuild`(2) / `Reconcile`(6), so the
/// `EmitRebuilt` tail edges all four.
/// effect in the container. Group-roots are `WritePermFile` plus the rebuild
/// subgraph's `MetaSync` / `Prebuild` / `Reconcile`, so the `EmitRebuilt` tail
/// edges all four.
pub fn perm_change(agent: &str, source: Source, reason: String, payload: PermPayload) -> DagSpec {
let mut nodes = vec![node(
let job = Job::new();
let write = node(
&job,
NodeKind::WritePermFile {
agent: agent.to_owned(),
payload,
},
Vec::new(),
)];
let rebuild = rebuild_nodes(
);
let roots = rebuild_nodes(
&job,
agent,
RebuildOpts {
relock: true,
graceful: false,
},
1,
Some(write),
);
emit_rebuilt_tails(
&job,
agent,
&[write, roots.meta_sync, roots.prebuild, roots.reconcile],
);
let reconcile = reconcile_index(&rebuild, 1);
nodes.extend(rebuild);
let tail_base = u64::try_from(nodes.len()).unwrap_or(0);
nodes.extend(emit_rebuilt_tails(agent, &[0, 1, 2, reconcile], tail_base));
DagSpec {
source,
reason,
nodes,
job,
}
}
@ -539,25 +478,26 @@ pub fn meta_update(
reason: String,
approval_id: Option<i64>,
) -> DagSpec {
let mut nodes = vec![node(
let job = Job::new();
let lock = node(
&job,
NodeKind::MetaLock {
sweep: false,
fanout: None,
inputs,
},
Vec::new(),
)];
);
// The bump itself has no side effect, so an operator-driven one ends at the
// `MetaLock`; an approval-driven one still has its row to resolve and gets the
// per-outcome tails edged onto that single group-root — whose roll-up covers
// the rebuild subgraphs `MetaLock` grows into itself.
if let Some(approval_id) = approval_id {
nodes.extend(resolve_approval_tails(approval_id, 0));
resolve_approval_tails(&job, approval_id, lock);
}
DagSpec {
source,
reason,
nodes,
job,
}
}
@ -575,10 +515,12 @@ pub fn reparent(
source: Source,
reason: String,
) -> DagSpec {
let job = Job::new();
let _reparent = node(&job, NodeKind::Reparent { moves });
DagSpec {
source,
reason,
nodes: vec![node(NodeKind::Reparent { moves }, Vec::new())],
job,
}
}
@ -586,45 +528,3 @@ pub fn reparent(
// as ONE `Boot` DAG (a sweep `MetaLock` root that grows rebuild subgraphs
// in-DAG, plus a `Reconcile` root per drifted agent) — no anchor node and no
// per-agent child DAGs.
/// Validate a spec before it enters the queue: node ids are dense
/// (index = id), deps + parents reference existing *earlier* nodes, and the
/// dep graph is acyclic (petgraph `toposort`). Rejecting cycles here fixes the
/// old queue's documented "circular dep silently deadlocks forever" caveat.
pub fn validate(spec: &DagSpec) -> Result<()> {
if spec.nodes.is_empty() {
bail!("dag spec {:?} has no nodes", spec.reason);
}
let n = spec.nodes.len();
let mut graph = petgraph::graph::DiGraph::<u32, ()>::new();
let idx: Vec<_> = (0..n)
.map(|i| graph.add_node(u32::try_from(i).unwrap_or(u32::MAX)))
.collect();
for (i, node) in spec.nodes.iter().enumerate() {
// A `parent` must index an earlier node — `insert_group` resolves it to
// an already-inserted `NodeId`, so a forward/out-of-bounds parent would
// otherwise panic there.
if let Some(p) = node.parent
&& usize::try_from(p).is_ok_and(|p| p >= i)
{
bail!(
"dag spec {:?} node {i} has invalid parent {p} (must be an earlier node)",
spec.reason
);
}
for dep in &node.deps {
let Some(&dep_idx) = usize::try_from(dep.on).ok().and_then(|i| idx.get(i)) else {
bail!(
"dag spec {:?} node {i} depends on unknown node {}",
spec.reason,
dep.on
);
};
graph.add_edge(dep_idx, idx[i], ());
}
}
if petgraph::algo::toposort(&graph, None).is_err() {
bail!("dag spec {:?} contains a dependency cycle", spec.reason);
}
Ok(())
}