vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

jobq-wire: move the generic graph projection into its own crate

Browse source 
The wire types were in hive-host-sock, which is the host *socket* crate — so
anything living there is core-shaped by construction, and the projection had
quietly grown two core dependencies to match: it selected roots by matching
NodeKind::Dag, and rendered payloads through free functions in hive-c0re that
nothing obliged a second host to write.

hive-jobq is the wrong home too. That crate is the scheduler — logic — and
folding presentation in means every consumer of it carries a JSON vocabulary
it may never serve.

So: a new hive-jobq-wire. A host implements WireNode for its payload N and
WireResource for its resource name R; GraphWire::wire_snapshot is
blanket-implemented for Graph<N, R> when both hold, and for nothing else. A
payload that has never said how it displays has no way onto the wire.

wire_snapshot takes the roots to serve rather than reading Graph::roots
itself. Nothing is ever removed from a Graph, so retention is a policy only
the host can hold; hive-c0re passes visible_roots(), which is the existing
MAX_HISTORY_DAGS bound selected structurally (a root is a node with no
parent) instead of by node kind.
This commit is contained in:
atlas 2026-08-02 23:52:32 +02:00 • committed by mara
commit 7966d5eb66
14 changed files with 564 additions and 297 deletions
Download patch file Download diff file Expand all files Collapse all files

145
hive-c0re/src/job_queue/mod.rs
View file

@ -39,11 +39,11 @@ mod tests;
use std::sync::{Arc, Mutex};
use chrono::{DateTime, Utc};
use hive_host_sock::graph::{GraphDep, GraphNode, NodePayload};
use hive_host_sock::jobs::NodeView;
use hive_jobq::resources::ResourceTable;
use hive_jobq::scheduler::{Outcome, Scheduler};
use hive_jobq::{Dep, Graph, NodeId};
use hive_jobq_wire::{GraphNode, GraphWire};
use tokio::sync::Notify;
pub use hive_jobq::TerminalState;
@ -363,26 +363,13 @@ impl JobQueue {
/// finished step is visible rather than vanishing from the payload, which
/// is what makes a fast rebuild render as a single node).
///
/// Retention is the *same* bound as [`Self::snapshot`] — every live group
/// plus the newest terminal ones, via [`visible_dags`]. Serving the raw
/// graph instead would grow without limit: evicted groups' nodes linger
/// until bounded pruning lands.
/// The projection itself is [`hive_jobq_wire`]'s; all this layer supplies
/// is *which* groups to show — see [`visible_roots`] for why the graph
/// can't decide that for itself.
#[must_use]
pub fn graph_snapshot(&self) -> Vec<GraphNode> {
let inner = self.lock();
let mut roots = visible_dags(&inner);
roots.sort_unstable_by_key(|root| root.get());
roots
.into_iter()
.flat_map(|root| {
inner
.graph()
.node(root)
.into_iter()
.chain(inner.graph().descendants(root))
.map(graph_node)
})
.collect()
inner.graph().wire_snapshot(visible_roots(&inner))
}
/// Snapshot every live + retained DAG for `/api/state` + `RebuildQueueChanged`.
@ -519,79 +506,6 @@ fn dag_view(sched: &Sched, container: NodeId) -> Option<DagView> {
})
}
/// Project one graph node onto the generic wire shape.
///
/// Deliberately near-total: everything `hive_jobq` records is carried, and
/// the only editorial decision is what goes in the opaque payload. That is
/// the inverse of [`dag_view`], which decides what a consumer is allowed to
/// see — a viewer that can render *any* graph cannot have that decided for it.
fn graph_node(node: &hive_jobq::Node<NodeKind, Resource>) -> GraphNode {
GraphNode {
id: node.id.get(),
parent: node.parent.map(NodeId::get),
state: node.state,
deps: node.deps.iter().map(graph_dep).collect(),
started_at: node.started_at,
finished_at: node.finished_at,
error: node.error.clone(),
payload: NodePayload {
label: node.payload.as_str().to_owned(),
data: node_data(node.id, &node.payload),
},
}
}
fn graph_dep(dep: &Dep<Resource>) -> GraphDep {
match dep {
Dep::Node { id, when } => GraphDep::Node {
id: id.get(),
accepts: [
TerminalState::Done,
TerminalState::Failed,
TerminalState::Cancelled,
TerminalState::Skipped,
]
.into_iter()
.filter(|outcome| when.accepts(*outcome))
.collect(),
},
Dep::Resource { name, count } => GraphDep::Resource {
name: name.wire_name(),
count: *count,
},
}
}
/// hive-c0re's domain data for one node, as the opaque payload slot.
///
/// Every field here used to be a named column on `NodeView`, meaningful for
/// one node kind and `null` on all the others. As free-form data it costs the
/// wire type nothing and a generic consumer renders it without knowing what
/// any of it means.
fn node_data(id: NodeId, kind: &NodeKind) -> serde_json::Value {
let mut data = serde_json::Map::new();
let agent = kind.agent();
if !agent.is_empty() {
data.insert("agent".to_owned(), agent.into());
}
if let NodeKind::DeployWindow { approval_id, .. } = kind {
data.insert("approval_id".to_owned(), (*approval_id).into());
}
if let NodeKind::MetaLock { inputs, .. } = kind
&& !inputs.is_empty()
{
data.insert("inputs".to_owned(), inputs.clone().into());
}
if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id.get())) {
data.insert("build_log_id".to_owned(), log.into());
}
if data.is_empty() {
serde_json::Value::Null
} else {
serde_json::Value::Object(data)
}
}
/// Which of a DAG's work nodes ride the wire, by index into `states` — or
/// `None` when the DAG has nothing left worth showing and drops out of the
/// snapshot entirely.
@ -665,6 +579,55 @@ fn visible_dags(sched: &Sched) -> Vec<NodeId> {
retain_history(live, terminal, MAX_HISTORY_DAGS)
}
/// The visible **group** set for [`Queue::graph_snapshot`]: every live group
/// root, plus the newest [`MAX_HISTORY_DAGS`] settled ones.
///
/// Same policy as [`visible_dags`], selected *structurally* — a root is a node
/// with no parent. The `DagView` path next door keys on `NodeKind::Dag`
/// instead, which is fine for a projection that already only means anything to
/// hive-c0re, but would make the generic endpoint depend on one node kind that
/// is itself slated for removal.
///
/// **This bound is load-bearing, not tidiness.** Nothing ever removes a node
/// from the graph (bounded pruning is a Stage-C follow-up), so serving
/// `graph.roots()` directly would grow the payload without limit for the whole
/// uptime of the daemon.
fn visible_roots(sched: &Sched) -> Vec<NodeId> {
let roots: Vec<NodeId> = sched.graph().roots().map(|n| n.id).collect();
let mut live: Vec<NodeId> = Vec::new();
let mut terminal: Vec<(NodeId, i64, u64)> = Vec::new();
for root in roots {
if sched.graph().is_settled(root) == Some(true) {
terminal.push((root, group_finished_at(sched, root), root.get()));
} else {
live.push(root);
}
}
retain_history(live, terminal, MAX_HISTORY_DAGS)
}
/// When a whole group last finished: the newest `finished_at` across the root
/// **and** its descendants. Unlike [`dag_finished_at`] the root itself counts,
/// because a generic group root can be an ordinary node with no children at
/// all — reading only descendants would date every such group to the epoch and
/// evict it first.
fn group_finished_at(sched: &Sched, root: NodeId) -> i64 {
sched
.graph()
.node(root)
.and_then(|n| n.finished_at)
.into_iter()
.chain(
sched
.graph()
.descendants(root)
.filter_map(|n| n.finished_at),
)
.map(|t| t.timestamp())
.max()
.unwrap_or(0)
}
/// [`visible_dags`]'s policy, split from the graph it reads: keep every live
/// DAG, plus the newest `cap` terminal ones.
///

38
hive-c0re/src/job_queue/model.rs
View file

@ -283,6 +283,44 @@ pub enum NodeKind {
},
}
/// How a hive-c0re node describes itself to a generic graph viewer.
///
/// Every field in [`WireNode::data`] here used to be a named column on
/// `NodeView`, meaningful for one node kind and `null` on all the others. As
/// free-form data it costs the wire type nothing, and a generic consumer
/// renders it without knowing what any of it means.
impl hive_jobq_wire::WireNode for NodeKind {
fn label(&self) -> String {
self.as_str().to_owned()
}
fn data(&self, id: hive_jobq_wire::WireId) -> serde_json::Value {
let mut data = serde_json::Map::new();
let agent = self.agent();
if !agent.is_empty() {
data.insert("agent".to_owned(), agent.into());
}
if let NodeKind::DeployWindow { approval_id, .. } = self {
data.insert("approval_id".to_owned(), (*approval_id).into());
}
if let NodeKind::MetaLock { inputs, .. } = self
&& !inputs.is_empty()
{
data.insert("inputs".to_owned(), inputs.clone().into());
}
// Not in the payload at all — the build log is keyed on node identity
// in a side table, which is why `data` is handed the id.
if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id)) {
data.insert("build_log_id".to_owned(), log.into());
}
if data.is_empty() {
serde_json::Value::Null
} else {
serde_json::Value::Object(data)
}
}
}
impl NodeKind {
/// Wire string for `NodeView.kind`.
pub fn as_str(&self) -> &'static str {

17
hive-c0re/src/job_queue/resource.rs
View file

@ -58,15 +58,14 @@ pub enum Resource {
MetaWindow,
}
impl Resource {
/// This resource's name on the generic graph wire.
///
/// `hive_jobq` is generic over the resource type, so a viewer that can
/// render any graph gets a string here rather than this enum. The
/// `agent:` prefix keeps the per-agent leases from colliding with a
/// hypothetical global resource that happens to share an agent's name.
#[must_use]
pub fn wire_name(&self) -> String {
/// This resource's name on the generic graph wire.
///
/// `hive_jobq` is generic over the resource type, so a viewer that can render
/// any graph gets a string here rather than this enum. The `agent:` prefix
/// keeps the per-agent leases from colliding with a hypothetical global
/// resource that happens to share an agent's name.
impl hive_jobq_wire::WireResource for Resource {
fn name(&self) -> String {
match self {
Resource::BuildSlot => "build-slot".to_owned(),
Resource::Agent(agent) => format!("agent:{agent}"),