338 lines
17 KiB
Rust
338 lines
17 KiB
Rust
//! Data model for the generic job-DAG queue: the node kinds — the primitive
|
|
//! operations — and what each one carries. The `State` / `PermPayload` wire
|
|
//! enums live in `hive_host_sock::jobs` (they travel on the host admin
|
|
//! socket) and are re-exported here for the queue's internal use. The graph
|
|
//! itself is served through `hive_jobq_wire`'s generic projection — there is
|
|
//! no second, typed view of it any more.
|
|
//!
|
|
//! **One level, not two.** The node is the unit of everything: scheduling,
|
|
//! execution, build-log, cancel, and the dashboard group (a group root's
|
|
//! subtree *is* the group). A DAG used to be a second level above it, with
|
|
//! its own store and its own id; there is no container node any more, so a
|
|
//! job is exactly the nodes it declared. See `docs/scheduler/coordinator.md::Job queue`
|
|
//! for the full design.
|
|
|
|
pub use hive_host_sock::jobs::{PermPayload, State};
|
|
use serde::Serialize;
|
|
|
|
use hive_jobq::TerminalState;
|
|
|
|
/// The primitive operations — each kind maps to one executor fn in
|
|
/// `exec.rs`, a thin wrapper over existing `lifecycle.rs` / `meta.rs`
|
|
/// code. Concurrency is gated by two resource classes (see
|
|
/// [`Resource`](super::resource::Resource), declared per node where the node
|
|
/// is constructed rather than derived from its kind); the
|
|
/// meta *repo* is serialized by `meta::META_LOCK` inside the wrapped
|
|
/// functions themselves, which is why there is no `GitCommit` node —
|
|
/// a standalone commit node would open a dirty-working-tree window
|
|
/// between nodes that the fused `meta.rs` ops deliberately close.
|
|
///
|
|
/// What each kind does, the resource/lease rules, and the DAG shapes they
|
|
/// compose into live in the node-inventory table and surrounding sections
|
|
/// of `docs/scheduler/coordinator.md` — this enum is deliberately not a
|
|
/// second copy of that; each variant below gets a one-line pointer.
|
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
|
#[serde(tag = "kind", rename_all = "snake_case")]
|
|
pub enum NodeKind {
|
|
/// The rebuild's meta-repo preamble. `relock = false` only for
|
|
/// meta-update cascade rebuilds (re-locking would revert the bump the
|
|
/// cascade just committed).
|
|
MetaSync { agent: String, relock: bool },
|
|
/// Out-of-band toplevel build while the container keeps serving.
|
|
Prebuild { agent: String },
|
|
/// `nixos-container update` profile-swap.
|
|
Swap { agent: String },
|
|
/// The post-`Swap` bookkeeping tail as a first-class node.
|
|
RebuildBookkeeping { agent: String },
|
|
/// First-spawn pre-create provisioning.
|
|
Provision { agent: String },
|
|
/// First-spawn `nixos-container create` proper. Assumes the
|
|
/// upstream `Provision` node already registered the agent in meta.
|
|
Create { agent: String },
|
|
/// `nixos-container destroy` plus the un-registration that follows it.
|
|
///
|
|
/// Deliberately excluded from [`NodeKind::takes_container_down`] — see
|
|
/// that method for why a node downstream of a `Stop` must not claim the
|
|
/// container going down as its own.
|
|
DestroyContainer { agent: String },
|
|
/// The `purge = true` half of a destroy: delete the agent's state
|
|
/// subvolume (via hive-priv, since a subvolume root defeats
|
|
/// `remove_dir_all`) plus its state and applied dirs.
|
|
PurgeState { agent: String },
|
|
/// The post-destroy bookkeeping tail.
|
|
DestroyBookkeeping { agent: String, purge: bool },
|
|
/// Meta flake lock bump. `sweep = false`: `meta::lock_update` with this
|
|
/// node's own `inputs`; `sweep = true`: `meta::lock_update_hyperhive`
|
|
/// (boot sweep, `inputs` empty). See _Notable collapses_ for how the
|
|
/// scheduler appends the fan-out `Rebuild` DAGs on completion.
|
|
MetaLock {
|
|
sweep: bool,
|
|
fanout: Option<Vec<String>>,
|
|
inputs: Vec<String>,
|
|
},
|
|
/// Idempotent power converge *planner*: fans a child
|
|
/// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than
|
|
/// doing the mechanical work itself.
|
|
Reconcile { agent: String },
|
|
/// Mechanical container start. Fanned out by a [`NodeKind::Reconcile`]
|
|
/// that observed `wanted = Up` and the container down.
|
|
Start { agent: String },
|
|
/// Mechanical container stop. Fanned out by a [`NodeKind::Reconcile`]
|
|
/// that observed `wanted = Offline` and up.
|
|
Stop { agent: String },
|
|
/// Mechanical `nixos-container stop` for the profile swap. Never
|
|
/// touches `wanted`. Noop if already stopped.
|
|
StopForUpdate { agent: String },
|
|
/// Set the graceful-stop fence + kick the harness so it runs one
|
|
/// stop-checkpoint turn.
|
|
Signal { agent: String },
|
|
/// Await the harness clearing the fence, bounded by
|
|
/// `GRACEFUL_STOP_TIMEOUT`. Resolves ok either way — the
|
|
/// downstream `Reconcile` performs the actual stop.
|
|
Drain { agent: String },
|
|
/// Write the pause marker (`Coordinator::set_paused`) + mark
|
|
/// `pause_pending`.
|
|
PauseSignal { agent: String },
|
|
/// Await the harness reporting `PauseAcknowledged`, bounded by a
|
|
/// timeout. Resolves ok either way — pausing is best-effort from
|
|
/// the queue's perspective, same as `Drain`.
|
|
PauseDrain { agent: String },
|
|
/// `set_nspawn_flags` + `set_resource_limits` + daemon-reload.
|
|
WriteDropin { agent: String },
|
|
/// Commit `tool-groups.json` / `capabilities.json` per its `payload`.
|
|
WritePermFile { agent: String, payload: PermPayload },
|
|
/// Topology move(s) — `set-parent` (len 1) or `set-parent-bulk` (len N) —
|
|
/// as a single queue node. `moves` is typed `(Ident, Option<Ident>)`
|
|
/// pairs, applied in order under one `META_LOCK` acquisition.
|
|
Reparent {
|
|
moves: Vec<(hive_types::Ident, Option<hive_types::Ident>)>,
|
|
},
|
|
/// Group root of the approval-deploy (`MergeConfigPr`) subtree — the
|
|
/// **brace** that owns the deploy window. See _Braces_ and _Approvals_.
|
|
DeployWindow { agent: String, approval_id: i64 },
|
|
/// Group root of a rebuild subtree — the **brace**, scoped to one
|
|
/// agent. Why braces exist and what they cost: _Braces_.
|
|
AgentWindow { agent: String },
|
|
/// Deploy phase 1 — **verify only, mutates nothing.**
|
|
MergeVerify { agent: String, approval_id: i64 },
|
|
/// Deploy phase 2 — the irreversible fast-forward plus the *opening*
|
|
/// half of the two-phase meta deploy. Does **not** build the container
|
|
/// itself; grows the ordinary rebuild subgraph in as its own children
|
|
/// (see the rebuild-path section's approval-deploy paragraph).
|
|
DeployApply { agent: String, approval_id: i64 },
|
|
/// Deploy phase 3 — closes the two-phase meta deploy once
|
|
/// [`NodeKind::DeployApply`]'s rebuild subgraph comes up clean.
|
|
FinalizeDeploy { agent: String, approval_id: i64 },
|
|
/// Deploy compensation **and bookkeeping** tail. See the `DeployTail`
|
|
/// node-inventory row for its three responsibilities and why it isn't
|
|
/// named `AbortDeploy`.
|
|
DeployTail { agent: String, approval_id: i64 },
|
|
/// Tail node of an approval-carrying DAG (spawn / opaque deploy /
|
|
/// config-PR merge): resolve the approval row from how the work ended.
|
|
ResolveApproval {
|
|
approval_id: i64,
|
|
/// Which outcome this node reports. A template emits **one per
|
|
/// outcome**, each edged to accept only that one, so exactly one is
|
|
/// ever runnable. The `Cancelled` one is also the node
|
|
/// [`super::JobQueue::cancel`] spares, since its edge is the only
|
|
/// one that accepts a dropped dependency.
|
|
outcome: TerminalState,
|
|
},
|
|
/// Tail node of a rebuild / perm-change: emit this agent's `Rebuilt`
|
|
/// manager event. One node per agent *and* per outcome; no cancel
|
|
/// variant (a cancelled DAG never ran, so there is no rebuild to
|
|
/// report).
|
|
EmitRebuilt { agent: String, ok: bool },
|
|
/// Write the agent's durable power intent (`wanted = Up` when `up`,
|
|
/// else `Offline`) as the head node of a power-op DAG.
|
|
SetWanted { agent: String, up: bool },
|
|
/// One-shot boot-time forge user/token sweep for every existing
|
|
/// container (`forge::ensure_all`). Agentless.
|
|
ForgeSweep,
|
|
/// One-shot boot-time matrix user/space sweep (`matrix::ensure_all`).
|
|
/// Agentless.
|
|
MatrixSweep,
|
|
/// One-shot boot-time Forgejo webhook registration. Agentless.
|
|
WebhookRegister,
|
|
/// One-shot boot-time `/knowledge` pull (`knowledge::pull`). Agentless.
|
|
KnowledgePull,
|
|
/// One-shot boot-time pull of the agent set the swarm controller
|
|
/// declares for this hive (`wanted::pull`). Agentless.
|
|
WantedPull,
|
|
}
|
|
|
|
/// How a hive-c0re node describes itself to a generic graph viewer.
|
|
///
|
|
/// Every field in the wire node's `data` representation here used to be a named column on
|
|
/// `NodeView`, meaningful for one node kind and `null` on all the others. As
|
|
/// free-form data it costs the wire type nothing, and a generic consumer
|
|
/// renders it without knowing what any of it means.
|
|
impl hive_jobq_wire::WireNode for NodeKind {
|
|
fn label(&self) -> String {
|
|
self.as_str().to_owned()
|
|
}
|
|
|
|
fn data(&self, id: hive_jobq_wire::WireId) -> serde_json::Value {
|
|
let mut data = serde_json::Map::new();
|
|
let agent = self.agent();
|
|
if !agent.is_empty() {
|
|
data.insert("agent".to_owned(), agent.into());
|
|
}
|
|
if let NodeKind::DeployWindow { approval_id, .. } = self {
|
|
data.insert("approval_id".to_owned(), (*approval_id).into());
|
|
}
|
|
if let NodeKind::MetaLock { inputs, .. } = self
|
|
&& !inputs.is_empty()
|
|
{
|
|
data.insert("inputs".to_owned(), inputs.clone().into());
|
|
}
|
|
// Not in the payload at all — the build log is keyed on node identity
|
|
// in a side table, which is why `data` is handed the id.
|
|
if let Some(log) = crate::build_logs::global().and_then(|h| h.id_for_node(id)) {
|
|
data.insert("build_log_id".to_owned(), log.into());
|
|
}
|
|
if data.is_empty() {
|
|
serde_json::Value::Null
|
|
} else {
|
|
serde_json::Value::Object(data)
|
|
}
|
|
}
|
|
}
|
|
|
|
impl NodeKind {
|
|
/// Wire string for the node's label on the graph wire
|
|
/// ([`hive_jobq_wire::WireNode::label`]).
|
|
pub fn as_str(&self) -> &'static str {
|
|
match self {
|
|
NodeKind::MetaSync { .. } => "meta_sync",
|
|
NodeKind::Prebuild { .. } => "prebuild",
|
|
NodeKind::Swap { .. } => "swap",
|
|
NodeKind::RebuildBookkeeping { .. } => "rebuild_bookkeeping",
|
|
NodeKind::Provision { .. } => "provision",
|
|
NodeKind::Create { .. } => "create",
|
|
NodeKind::DestroyContainer { .. } => "destroy_container",
|
|
NodeKind::PurgeState { .. } => "purge_state",
|
|
NodeKind::DestroyBookkeeping { .. } => "destroy_bookkeeping",
|
|
NodeKind::MetaLock { .. } => "meta_lock",
|
|
NodeKind::Reconcile { .. } => "reconcile",
|
|
NodeKind::Start { .. } => "start",
|
|
NodeKind::Stop { .. } => "stop",
|
|
NodeKind::StopForUpdate { .. } => "stop_for_update",
|
|
NodeKind::Signal { .. } => "signal",
|
|
NodeKind::Drain { .. } => "drain",
|
|
NodeKind::PauseSignal { .. } => "pause_signal",
|
|
NodeKind::PauseDrain { .. } => "pause_drain",
|
|
NodeKind::WriteDropin { .. } => "write_dropin",
|
|
NodeKind::WritePermFile { .. } => "write_perm_file",
|
|
NodeKind::Reparent { .. } => "reparent",
|
|
NodeKind::DeployWindow { .. } => "deploy_window",
|
|
NodeKind::AgentWindow { .. } => "agent_window",
|
|
NodeKind::MergeVerify { .. } => "merge_verify",
|
|
NodeKind::DeployApply { .. } => "deploy_apply",
|
|
NodeKind::FinalizeDeploy { .. } => "finalize_deploy",
|
|
NodeKind::DeployTail { .. } => "deploy_tail",
|
|
NodeKind::ResolveApproval { .. } => "resolve_approval",
|
|
NodeKind::EmitRebuilt { .. } => "emit_rebuilt",
|
|
NodeKind::SetWanted { .. } => "set_wanted",
|
|
NodeKind::ForgeSweep => "forge_sweep",
|
|
NodeKind::MatrixSweep => "matrix_sweep",
|
|
NodeKind::WebhookRegister => "webhook_register",
|
|
NodeKind::KnowledgePull => "knowledge_pull",
|
|
NodeKind::WantedPull => "wanted_pull",
|
|
}
|
|
}
|
|
|
|
/// The agent this node targets, or `""` for agentless kinds
|
|
/// ([`NodeKind::MetaLock`] on the `hyperhive` pseudo-agent,
|
|
/// [`NodeKind::Reparent`] which can span multiple agents, and
|
|
/// [`NodeKind::ResolveApproval`] which acts on an approval row).
|
|
#[must_use]
|
|
pub fn agent(&self) -> &str {
|
|
match self {
|
|
NodeKind::MetaSync { agent, .. }
|
|
| NodeKind::Prebuild { agent }
|
|
| NodeKind::Swap { agent }
|
|
| NodeKind::RebuildBookkeeping { agent }
|
|
| NodeKind::Provision { agent }
|
|
| NodeKind::Create { agent }
|
|
| NodeKind::DestroyContainer { agent }
|
|
| NodeKind::PurgeState { agent }
|
|
| NodeKind::DestroyBookkeeping { agent, .. }
|
|
| NodeKind::Reconcile { agent }
|
|
| NodeKind::Start { agent }
|
|
| NodeKind::Stop { agent }
|
|
| NodeKind::StopForUpdate { agent }
|
|
| NodeKind::Signal { agent }
|
|
| NodeKind::Drain { agent }
|
|
| NodeKind::PauseSignal { agent }
|
|
| NodeKind::PauseDrain { agent }
|
|
| NodeKind::WriteDropin { agent }
|
|
| NodeKind::WritePermFile { agent, .. }
|
|
| NodeKind::DeployWindow { agent, .. }
|
|
| NodeKind::AgentWindow { agent }
|
|
| NodeKind::MergeVerify { agent, .. }
|
|
| NodeKind::DeployApply { agent, .. }
|
|
| NodeKind::FinalizeDeploy { agent, .. }
|
|
| NodeKind::DeployTail { agent, .. }
|
|
| NodeKind::EmitRebuilt { agent, .. }
|
|
| NodeKind::SetWanted { agent, .. } => agent,
|
|
NodeKind::MetaLock { .. }
|
|
| NodeKind::Reparent { .. }
|
|
| NodeKind::ResolveApproval { .. }
|
|
| NodeKind::ForgeSweep
|
|
| NodeKind::MatrixSweep
|
|
| NodeKind::WebhookRegister
|
|
| NodeKind::KnowledgePull
|
|
| NodeKind::WantedPull => "",
|
|
}
|
|
}
|
|
|
|
/// Whether running this node is *expected* to take the agent's container
|
|
/// down. Feeds `RunningTransient::takes_container_down`, which the crash watcher
|
|
/// reads to tell an intentional stop from a crash.
|
|
///
|
|
/// This is a **safety** question, not a display one — it decides whether a
|
|
/// vanished container raises an alert. It is deliberately not derived from
|
|
/// the pill label: a label is free to be renamed or added without moving
|
|
/// the alerting boundary, and only the operation itself knows its intent.
|
|
///
|
|
/// Default is `false`, and that asymmetry is the point. A wrong `false`
|
|
/// costs a spurious crash event; a wrong `true` **swallows a real crash**
|
|
/// silently. So a kind earns `true` by being listed here, and anything new
|
|
/// is noisy-but-safe until someone decides otherwise.
|
|
#[must_use]
|
|
pub fn takes_container_down(&self) -> bool {
|
|
matches!(
|
|
self,
|
|
// Explicit stops, and the quiesce steps that precede one.
|
|
NodeKind::Stop { .. }
|
|
| NodeKind::StopForUpdate { .. }
|
|
| NodeKind::Signal { .. }
|
|
| NodeKind::Drain { .. }
|
|
| NodeKind::SetWanted { up: false, .. }
|
|
// The rebuild's own machinery: the container is down across the
|
|
// swap and the drop-in write that reconfigures it.
|
|
| NodeKind::Swap { .. }
|
|
| NodeKind::WriteDropin { .. }
|
|
)
|
|
// Everything else is `false` on purpose, including the ones that would
|
|
// be easy to wave through:
|
|
// - `Create` / `Start` / `SetWanted{up}` bring a container UP. A
|
|
// container disappearing *while starting* is a genuine crash and has
|
|
// to keep reporting as one.
|
|
// - `DestroyContainer` looks like the most obvious `true` on this list
|
|
// and is the one that must stay `false`. It is edged downstream of a
|
|
// `Stop`, so the container is already down when it claims; the stop
|
|
// that the operator asked for is accounted for by the node that
|
|
// performs it. A container found alive at destroy time is a genuine
|
|
// bug, and a `true` here would suppress the alert that says so.
|
|
// - `Reconcile` is a planner; it fans out `Start` / `Stop`, which carry
|
|
// their own answer.
|
|
// - `DeployWindow` brackets a deploy without itself stopping anything.
|
|
// - `AgentWindow` likewise, though it *parents* nodes that answer
|
|
// `true`. This is per-node, not per-subtree, and each of those
|
|
// children reports for itself — so `true` here would only widen
|
|
// suppression over the build and tail, where a vanished container is
|
|
// still a real crash.
|
|
}
|
|
}
|