vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

jobq: a job asks for the ids it wants back

Browse source 
The operator's instruction on the issue was "the closure returns an array
of guids, and enqueue_job returns the node ids in that order". What was
here instead returned a HashMap of everything inserted, and no caller used
the keys: submit dropped the return, insert_group did into_values(), and
the scheduler ignored what append_subgraph handed back. The guid-keyed
lookup was dead weight, and into_values() made that Vec arbitrarily
ordered -- harmless only because nothing read it.

insert_job now takes FnOnce(&JobBuilder) -> Vec<NodeGuid> and returns the
matching ids positionally. A handle from another job is UnknownNode rather
than a silent omission: the return is positional, so a short vector would
misalign every id after it.

c0re's Declare stays FnOnce(&Job) and the wrapper names no handles in one
place, rather than ending seven templates in an empty vector -- a DAG is
addressed by its container node, which submit inserts itself. That frees
insert_group from needing every id, so the node_rt pre-seeding goes too:
NodeRuntime is one Option field and every reader already tolerated a
missing entry (entry().or_default(), get().and_then(), iter().find()).

The tests are the argument for the shape: capturing a handle through a
mutable binding to look it up in the map afterwards collapses into
returning it and destructuring the result.
This commit is contained in:
atlas 2026-08-02 14:03:37 +02:00 • committed by mara
commit bf138ae79a
7 changed files with 147 additions and 76 deletions
Download patch file Download diff file Expand all files Collapse all files

121
hive-jobq/src/builder.rs
View file

@ -74,12 +74,40 @@ pub enum BuildError {
/// The not-yet-declared parent.
parent: NodeGuid,
},
/// A handle named in the closure's return value belongs to a different
/// job. Same cause as a foreign handle on an edge: a [`NodeGuid`] outlives
/// the borrow that tied it to its builder, so one can be carried here.
#[error("handle {node:?} names no node in this job")]
UnknownNode {
/// The handle that resolved to nothing.
node: NodeGuid,
},
/// The graph rejected an otherwise well-formed node — an out-of-group edge,
/// an unsatisfiable [`DepWhen`], and so on.
#[error(transparent)]
Graph(#[from] GraphError),
}
/// Look each handle a job asked for up in what the insert actually minted,
/// preserving the order it asked in — the last step of both insertion entry
/// points.
///
/// # Errors
/// [`BuildError::UnknownNode`] for a handle this job never issued.
pub(crate) fn resolve_wanted(
wanted: &[NodeGuid],
ids: &HashMap<NodeGuid, NodeId>,
) -> Result<Vec<NodeId>, BuildError> {
wanted
.iter()
.map(|g| {
ids.get(g)
.copied()
.ok_or(BuildError::UnknownNode { node: *g })
})
.collect()
}
/// One node as the builder holds it: edges and parent still name *handles*, so
/// nothing here depends on ids the graph has not minted yet.
#[derive(Debug)]
@ -381,39 +409,41 @@ mod tests {
#[test]
fn edges_resolve_to_minted_ids() {
let mut g = graph();
let mut named = None;
let ids = g
.insert_job(None, |b| {
let first = b.node("a");
let second = b.node("b").after_ok(first);
named = Some((first.guid(), second.guid()));
vec![first.guid(), second.guid()]
})
.expect("insert");
let (first, second) = named.expect("declared");
let [first, second] = ids[..] else {
panic!("two ids back, in the order asked for")
};
assert_eq!(
deps_of(&g, ids[&second]),
deps_of(&g, second),
vec![Dep::Node {
id: ids[&first],
id: first,
when: DepWhen::AFTER_OK
}]
);
assert!(deps_of(&g, ids[&first]).is_empty());
assert!(deps_of(&g, first).is_empty());
}
#[test]
fn parent_resolves_to_a_minted_id() {
let mut g = graph();
let mut named = None;
let ids = g
.insert_job(None, |b| {
let root = b.node("a");
let child = b.node("b").part_of(root);
named = Some((root.guid(), child.guid()));
vec![root.guid(), child.guid()]
})
.expect("insert");
let (root, child) = named.expect("declared");
assert_eq!(g.node(ids[&root]).expect("root").parent, None);
assert_eq!(g.node(ids[&child]).expect("child").parent, Some(ids[&root]));
let [root, child] = ids[..] else {
panic!("two ids back")
};
assert_eq!(g.node(root).expect("root").parent, None);
assert_eq!(g.node(child).expect("child").parent, Some(root));
}
/// A handle is `Copy`, so naming the same node as a dependency twice must
@ -421,27 +451,28 @@ mod tests {
#[test]
fn one_handle_can_be_depended_on_twice() {
let mut g = graph();
let mut named = None;
let ids = g
.insert_job(None, |b| {
let shared = b.node("a");
let ok = b.node("b").after_ok(shared);
let any = b.node("c").after_any(shared);
named = Some((shared.guid(), ok.guid(), any.guid()));
vec![shared.guid(), ok.guid(), any.guid()]
})
.expect("insert");
let (shared, ok, any) = named.expect("declared");
let [shared, ok, any] = ids[..] else {
panic!("three ids back")
};
assert_eq!(
deps_of(&g, ids[&ok]),
deps_of(&g, ok),
vec![Dep::Node {
id: ids[&shared],
id: shared,
when: DepWhen::AFTER_OK
}]
);
assert_eq!(
deps_of(&g, ids[&any]),
deps_of(&g, any),
vec![Dep::Node {
id: ids[&shared],
id: shared,
when: DepWhen::AFTER_ANY
}]
);
@ -451,20 +482,21 @@ mod tests {
#[test]
fn resources_become_resource_deps() {
let mut g = graph();
let mut named = None;
let ids = g
.insert_job(None, |b| {
named = Some(
vec![
b.node("a")
.needs("agent/atlas")
.needs_units("build", 2)
.guid(),
);
]
})
.expect("insert");
let only = named.expect("declared");
let [only] = ids[..] else {
panic!("one id back")
};
assert_eq!(
deps_of(&g, ids[&only]),
deps_of(&g, only),
vec![
Dep::Resource {
name: "agent/atlas",
@ -492,6 +524,9 @@ mod tests {
let second = b.node("b");
let _ = first.after_any(second);
named = Some((first.guid(), second.guid()));
// The insert fails, so nothing comes back to ask for — the
// handles under test travel out through `named` instead.
Vec::new()
})
.expect_err("forward edge");
let (first, second) = named.expect("declared");
@ -514,6 +549,7 @@ mod tests {
let parent = b.node("b");
let _ = child.part_of(parent);
named = Some((child.guid(), parent.guid()));
Vec::new()
})
.expect_err("forward parent");
let (child, parent) = named.expect("declared");
@ -539,6 +575,7 @@ mod tests {
let mut foreign = None;
g.insert_job(None, |b| {
foreign = Some(b.node("first job").guid());
Vec::new()
})
.expect("first job inserts");
let foreign = foreign.expect("declared");
@ -546,6 +583,7 @@ mod tests {
let err = g
.insert_job(None, |b| {
let _ = b.node("second job").after_ok(foreign);
Vec::new()
})
.expect_err("foreign handle");
assert!(
@ -565,6 +603,7 @@ mod tests {
// A child may not depend on its own parent: the parent gate
// already orders them, and the edge would deadlock.
let _ = b.node("child").part_of(root).after_ok(root);
Vec::new()
})
.expect_err("out-of-group dep");
assert!(matches!(err, BuildError::Graph(_)), "{err:?}");
@ -578,23 +617,47 @@ mod tests {
let mut g = graph();
let container = g.insert("container", Vec::new(), None).expect("container");
let mut named = None;
let ids = g
.insert_job(Some(container), |b| {
let root = b.node("root");
let child = b.node("child").part_of(root);
named = Some((root.guid(), child.guid()));
vec![root.guid(), child.guid()]
})
.expect("insert");
let (root, child) = named.expect("declared");
assert_eq!(g.node(ids[&root]).expect("root").parent, Some(container));
assert_eq!(g.node(ids[&child]).expect("child").parent, Some(ids[&root]));
let [root, child] = ids[..] else {
panic!("two ids back")
};
assert_eq!(g.node(root).expect("root").parent, Some(container));
assert_eq!(g.node(child).expect("child").parent, Some(root));
}
/// A handle that names no node in *this* job is refused rather than
/// silently dropped from the returned ids — the return is positional, so a
/// short vector would misalign every id after it.
#[test]
fn asking_for_a_foreign_handle_is_an_error() {
let mut g = graph();
let mut foreign = None;
g.insert_job(None, |b| {
foreign = Some(b.node("first job").guid());
Vec::new()
})
.expect("first job inserts");
let foreign = foreign.expect("declared");
let err = g
.insert_job(None, |b| {
let _ = b.node("second job");
vec![foreign]
})
.expect_err("foreign handle asked for");
assert_eq!(err, BuildError::UnknownNode { node: foreign });
}
#[test]
fn an_empty_builder_inserts_nothing() {
let mut g = graph();
let ids = g.insert_job(None, |_| {}).expect("insert");
let ids = g.insert_job(None, |_| Vec::new()).expect("insert");
assert!(ids.is_empty());
assert_eq!(g.nodes().count(), 0);
}

25
hive-jobq/src/lib.rs
View file

@ -458,26 +458,33 @@ impl<N, R> Graph<N, R> {
Ok(id)
}
/// Insert a whole job under `root_parent`, returning the id each handle's
/// node was minted as.
/// Insert a whole job under `root_parent`, returning the ids of the nodes
/// `declare` **asked for**, in the order it named them.
///
/// `declare` receives a fresh [`JobBuilder`] and names the job's nodes on
/// it; the builder never leaves this call, so a job cannot be built in one
/// place and inserted in another. The graph-only counterpart of
/// `declare` receives a fresh [`JobBuilder`], names the job's nodes on it,
/// and returns the handles whose ids it wants back. The builder never
/// leaves this call, so a job cannot be built in one place and inserted in
/// another. The graph-only counterpart of
/// [`crate::scheduler::Scheduler::insert_job`] — prefer that one when a
/// scheduler owns the graph, since it also records what it started.
///
/// Asking is how a caller addresses a node it created: the alternative — a
/// map of everything inserted, or a positional vector — either hands back a
/// lookup nobody performs or reintroduces the counting this API exists to
/// remove.
///
/// # Errors
/// Propagates [`BuildError`] — a forward reference in the job's own
/// declarations, a handle from a different job, or a graph rejection.
pub fn insert_job(
&mut self,
root_parent: Option<NodeId>,
declare: impl FnOnce(&JobBuilder<N, R>),
) -> Result<std::collections::HashMap<NodeGuid, NodeId>, BuildError> {
declare: impl FnOnce(&JobBuilder<N, R>) -> Vec<NodeGuid>,
) -> Result<Vec<NodeId>, BuildError> {
let job = JobBuilder::new();
declare(&job);
job.insert_into(self, root_parent)
let wanted = declare(&job);
let ids = job.insert_into(self, root_parent)?;
crate::builder::resolve_wanted(&wanted, &ids)
}
/// Borrow a node by id.

22
hive-jobq/src/scheduler.rs
View file

@ -103,11 +103,12 @@ impl<N, R: Clone + Eq + Hash> Scheduler<N, R> {
/// Insert a whole job under `root_parent`, returning the id each handle's
/// node was minted as.
///
/// `declare` receives a fresh [`JobBuilder`] and names the job's nodes on
/// it; the builder never leaves this call. That is the whole insertion
/// API — a caller cannot construct a builder, hold one, or insert one
/// itself, so there is no way to end up with a job-shaped value being
/// passed around as a spec.
/// `declare` receives a fresh [`JobBuilder`], names the job's nodes on it,
/// and returns the handles whose ids it wants back — they come back in
/// that order. The builder never leaves this call. That is the whole
/// insertion API — a caller cannot construct a builder, hold one, or
/// insert one itself, so there is no way to end up with a job-shaped value
/// being passed around as a spec.
///
/// The scheduler-side counterpart of [`Graph::insert_job`]: same
/// resolution, but each node goes through [`Scheduler::append`], so a
@ -121,13 +122,14 @@ impl<N, R: Clone + Eq + Hash> Scheduler<N, R> {
pub fn insert_job(
&mut self,
root_parent: Option<NodeId>,
declare: impl FnOnce(&JobBuilder<N, R>),
) -> Result<HashMap<NodeGuid, NodeId>, BuildError> {
declare: impl FnOnce(&JobBuilder<N, R>) -> Vec<NodeGuid>,
) -> Result<Vec<NodeId>, BuildError> {
let job = JobBuilder::new();
declare(&job);
job.insert_with(root_parent, |payload, deps, parent| {
let wanted = declare(&job);
let ids = job.insert_with(root_parent, |payload, deps, parent| {
self.append(payload, deps, parent)
})
})?;
crate::builder::resolve_wanted(&wanted, &ids)
}
/// Claim every currently-runnable pending node and start it: node-deps