vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: trim NodeKind/Resource doc comments now that coordinator.md covers them

Browse source
This commit is contained in:
damocles 2026-09-02 02:40:26 +02:00 • committed by mara
commit 223ac257e0
3 changed files with 96 additions and 306 deletions
Download patch file Download diff file Expand all files Collapse all files

29
docs/scheduler/coordinator.md
View file

@ -44,7 +44,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) |
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead |
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta |
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
@ -56,22 +56,23 @@ Cheap — no build slot:
| Node | Behavior |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail — rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good, then mirrors the config repo to the forge. Infallible by construction |
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction |
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
| `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#<name>` ref resolves. Store/meta-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here) |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's own between-turns poll is already responsive enough |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap` |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/<id>`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree |
| `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent |
| `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent *and* per outcome |
@ -220,9 +221,9 @@ resources are free. Resources:
(`SetWanted` is a store write, not a container op, but takes the lease anyway
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
`Prebuild`, `MetaLock`, `WritePermFile`, `Reparent` — they touch the store /
meta, not the running container, which is exactly why a stop can land while
another DAG's prebuild is still building. Also exempt, for a different
`Prebuild`, `Provision`, `MetaLock`, `WritePermFile`, `Reparent` — they
touch the store / meta, not the running container, which is exactly why a
stop can land while another DAG's prebuild is still building. Also exempt, for a different
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
container, but their `AgentWindow` brace holds the lease above them — see
@ -306,6 +307,13 @@ compensation step that survives a hive-c0re restart — the pre-merge
local variable, so `DeployTail` can still undo a half-finished deploy after a
crash.
`DeployWindow` declares all three resources (build slot, lease, meta window)
on itself rather than letting each phase declare its own, because the queue
acquires a node's resources atomically (all-or-nothing): a child that took
the build slot while its parent held the meta window could block waiting for
a resource its own parent already committed to, a lock-ordering hazard that
one multi-resource root avoids by construction.
`Spawn` and `UpdateMetaInputs` approvals map onto the ordinary `spawn` /
`meta-update` shapes. The scheduler fires `actions::resolve_approval_dag`
exactly once when **any** approval-carrying DAG settles terminal — deploys
@ -427,8 +435,9 @@ Sequence for a rebuild DAG (each step is its own queue node):
The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface
here, before the running container is touched. (Runs even for a stopped
container — same total nix work, one uniform DAG shape.)
here, before the running container is touched. Skipped when the
container is already down — a stopped agent has no uptime to preserve,
so `Swap` builds inline instead of paying the double eval.
3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
(near-instant after the prebuild).

328
hive-c0re/src/job_queue/model.rs
View file

@ -26,118 +26,59 @@ use hive_jobq::TerminalState;
/// functions themselves, which is why there is no `GitCommit` node —
/// a standalone commit node would open a dirty-working-tree window
/// between nodes that the fused `meta.rs` ops deliberately close.
///
/// What each kind does, the resource/lease rules, and the DAG shapes they
/// compose into live in the node-inventory table and surrounding sections
/// of `docs/scheduler/coordinator.md` — this enum is deliberately not a
/// second copy of that; each variant below gets a one-line pointer.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum NodeKind {
/// The rebuild's meta-repo preamble: `lifecycle::prepare_rebuild_dirs`,
/// an idempotent meta `sync_agents`, and an optional per-agent relock.
/// `relock = false` only for meta-update cascade rebuilds (re-locking
/// would revert the bump the cascade just committed).
///
/// Its own node — ahead of, and *not* an ancestor of, [`NodeKind::Prebuild`]
/// — precisely because it is the only part of the rebuild that mutates the
/// meta repo and so holds the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow). Fusing
/// it into `Prebuild` (or making it `Prebuild`'s parent, which holds a
/// resource across the whole subtree) would extend that global window over
/// the multi-minute toplevel build and serialize rebuilds hive-wide.
/// Store/meta work only — build-slot- and lease-exempt.
/// The rebuild's meta-repo preamble. `relock = false` only for
/// meta-update cascade rebuilds (re-locking would revert the bump the
/// cascade just committed).
MetaSync { agent: String, relock: bool },
/// Out-of-band toplevel build while the container keeps serving:
/// `lifecycle::prebuild_toplevel`, reading a meta repo the upstream
/// [`NodeKind::MetaSync`] has already synced. The warm build is skipped
/// when the container is already down — it only exists to shrink the
/// swap's downtime, which a stopped agent doesn't need (`Swap` builds
/// inline instead).
/// Out-of-band toplevel build while the container keeps serving.
Prebuild { agent: String },
/// `nixos-container update` profile-swap (requires the container
/// stopped). Re-applies nspawn flags + resource limits first —
/// rebuild is the reconcile verb. The post-rebuild bookkeeping tail
/// lives in the sibling `RebuildBookkeeping` node.
/// `nixos-container update` profile-swap.
Swap { agent: String },
/// The post-`Swap` bookkeeping tail as a first-class node: rev marker,
/// forge + matrix sync, manager kick, container rescan, meta-inputs
/// snapshot. Split out of `Swap` for dashboard visibility + retry
/// granularity. Deps `AfterOk(Swap)`, so it runs only when the profile
/// swap succeeded; the tail `Reconcile` deps `AfterAny(RebuildBookkeeping)`, so on
/// swap failure this node is cancel-cascaded (a terminal state) and
/// recovery still runs. Store/forge/matrix work only — no nix build.
///
/// Declares **no resources**: it is a coordinated child of
/// [`NodeKind::AgentWindow`], which holds the agent lease (and the build
/// slot) for the whole rebuild subtree. See `templates.rs`'s module doc for
/// why the brace is the one place a resource is declared on behalf of
/// others.
/// The post-`Swap` bookkeeping tail as a first-class node.
RebuildBookkeeping { agent: String },
/// First-spawn pre-create provisioning: proposed/applied repos,
/// state subvolume, and meta registration (`sync_agents`). Runs
/// ahead of `Create` so the `nixos-container create --flake
/// meta#<name>` ref resolves. Store/meta-only — no container yet —
/// so it's lease- and build-slot-exempt like `Prebuild`.
/// First-spawn pre-create provisioning.
Provision { agent: String },
/// First-spawn `nixos-container create` proper. Assumes the
/// upstream `Provision` node already registered the agent in meta.
Create { agent: String },
/// `nixos-container destroy` plus the un-registration that follows it:
/// drop the agent from the coordinator's roster and clear its ephemeral
/// runtime dir.
/// `nixos-container destroy` plus the un-registration that follows it.
///
/// **Deliberately not in [`NodeKind::takes_container_down`]**, and that
/// is the design rather than an oversight. This node runs *downstream of
/// a `Stop`*, which already carries the flag honestly, so by the time it
/// claims there is nothing left to take down. A container still live here
/// is a real bug and must page someone — a `true` would absorb exactly
/// that signal, and the flag's whole asymmetry (see that method) is that
/// a wrong `true` silently swallows a crash.
/// Deliberately excluded from [`NodeKind::takes_container_down`] — see
/// that method for why a node downstream of a `Stop` must not claim the
/// container going down as its own.
DestroyContainer { agent: String },
/// The `purge = true` half of a destroy: delete the agent's state
/// subvolume (via hive-priv, since a subvolume root defeats
/// `remove_dir_all`) plus its state and applied dirs. Its own node
/// because it is conditional — a plain destroy never inserts it — and
/// because it is the irreversible step, so it earns a distinct row in
/// the graph rather than hiding inside a bookkeeping tail.
/// `remove_dir_all`) plus its state and applied dirs.
PurgeState { agent: String },
/// The post-destroy bookkeeping tail: meta sync, fail the agent's pending
/// approvals, drop the durable power intent, notify the manager, rescan
/// containers, re-emit the tombstone + schedule snapshots, resync
/// tmpfiles. Split from [`NodeKind::DestroyContainer`] for the same
/// reason [`NodeKind::RebuildBookkeeping`] is split from `Swap`:
/// dashboard visibility and retry granularity for work that is pure
/// store/meta bookkeeping and touches no container.
///
/// `purge` only selects the wording of the approval-failure reason and
/// the manager notification; the destructive work is `PurgeState`'s.
/// The post-destroy bookkeeping tail.
DestroyBookkeeping { agent: String, purge: bool },
/// Meta flake lock bump. `sweep = false`: `meta::lock_update`
/// (commit fused, under `META_LOCK`) with this node's own `inputs`;
/// `sweep = true`: `meta::lock_update_hyperhive`, *non-fatal* (a
/// failed boot-time bump must not cancel the fan-out rebuilds).
/// On success the scheduler appends child `Rebuild` DAGs: the
/// precomputed `fanout` list when present (boot sweep), else the
/// post-bump affected set (`meta_update_cascade_agents`).
///
/// `inputs` are the flake inputs to bump — empty means "all", and the
/// boot sweep leaves them empty since it bumps `hyperhive` alone. They
/// ride this node because it is the only thing that reads them.
/// Meta flake lock bump. `sweep = false`: `meta::lock_update` with this
/// node's own `inputs`; `sweep = true`: `meta::lock_update_hyperhive`
/// (boot sweep, `inputs` empty). See _Notable collapses_ for how the
/// scheduler appends the fan-out `Rebuild` DAGs on completion.
MetaLock {
sweep: bool,
fanout: Option<Vec<String>>,
inputs: Vec<String>,
},
/// Idempotent power converge *planner*: read `wanted` + observed
/// state and decide the action (start if `Up` & down, stop if
/// `Offline` & up, else noop). The mechanical work is not done in
/// this node — it fans a child [`NodeKind::Start`] / [`NodeKind::Stop`]
/// DAG out at runtime so the sub-step is a first-class DAG node.
/// Idempotent power converge *planner*: fans a child
/// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than
/// doing the mechanical work itself.
Reconcile { agent: String },
/// Mechanical container start: the start preamble (runtime dir +
/// drop-ins), `start_with_fallback`, MCP listener registration, and
/// the manager kick. Fanned out by a [`NodeKind::Reconcile`] that
/// observed `wanted = Up` and the container down.
/// Mechanical container start. Fanned out by a [`NodeKind::Reconcile`]
/// that observed `wanted = Up` and the container down.
Start { agent: String },
/// Mechanical container stop: `nixos-container` kill, MCP listener
/// unregister, and the `Killed` manager notify. Fanned out by a
/// [`NodeKind::Reconcile`] that observed `wanted = Offline` and up.
/// Mechanical container stop. Fanned out by a [`NodeKind::Reconcile`]
/// that observed `wanted = Offline` and up.
Stop { agent: String },
/// Mechanical `nixos-container stop` for the profile swap. Never
/// touches `wanted`. Noop if already stopped.
@ -150,10 +91,7 @@ pub enum NodeKind {
/// downstream `Reconcile` performs the actual stop.
Drain { agent: String },
/// Write the pause marker (`Coordinator::set_paused`) + mark
/// `pause_pending`. No kick, unlike `Signal` — the harness's own
/// between-turns poll (`PAUSE_POLL`, 1s default) is already
/// responsive enough, and `Signal`'s kick-message body ("you were
/// just (re)started") would be actively misleading here.
/// `pause_pending`.
PauseSignal { agent: String },
/// Await the harness reporting `PauseAcknowledged`, bounded by a
/// timeout. Resolves ok either way — pausing is best-effort from
@ -161,197 +99,65 @@ pub enum NodeKind {
PauseDrain { agent: String },
/// `set_nspawn_flags` + `set_resource_limits` + daemon-reload.
WriteDropin { agent: String },
/// Commit `tool-groups.json` / `capabilities.json` per its `payload`
/// (commit fused under `META_LOCK`). The payload rides this node — the only
/// consumer — rather than the generic DAG container.
/// Commit `tool-groups.json` / `capabilities.json` per its `payload`.
WritePermFile { agent: String, payload: PermPayload },
/// Topology move(s) — `set-parent` (len 1) or `set-parent-bulk` (len N) —
/// as a single queue node. Agentless like [`NodeKind::MetaLock`]: a
/// reparent touches the meta repo, not any one container, and a bulk
/// move spans multiple agents anyway. Declares the meta window, same
/// precedent as [`NodeKind::WritePermFile`] (also a small
/// git-commit-under-`META_LOCK` op) — a reparent's commit must not land
/// inside another node's staged deploy `prepare_deploy`→`finalize_deploy`
/// window. `(child, new_parent)` pairs, applied in order under one
/// `META_LOCK` acquisition / one git commit (`meta::bulk_commit_topology`
/// handles both the single- and multi-move case uniformly).
/// as a single queue node. `moves` is typed `(Ident, Option<Ident>)`
/// pairs, applied in order under one `META_LOCK` acquisition.
Reparent {
moves: Vec<(hive_types::Ident, Option<hive_types::Ident>)>,
},
/// Group root of the approval-deploy (`MergeConfigPr`) subtree, and the
/// node that **owns the deploy window**. It performs no work of its own —
/// it exists so the resources it declares (the global
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow), the
/// agent lease, a build slot) are held continuously across every child
/// phase, which a per-node acquisition could not guarantee.
///
/// All three resources are declared *here*, on one node, on purpose. The
/// queue acquires a node's resources atomically (all-or-nothing), so a
/// single multi-resource root can never hold one and block on another —
/// whereas letting a child take the build slot while its parent held the
/// meta window would introduce exactly that pattern, and with it a
/// lock-ordering argument that has to be re-verified on every future edit.
/// Cheap, too: the window has to span the container build regardless (see
/// [`NodeKind::DeployApply`]), so nothing is over-serialised by hoisting
/// the slot and the lease up alongside it.
///
/// Carries the approval row every phase below it re-reads, like each of
/// those phases does — the id is the node's own payload, not something a
/// DAG-level catch-all hands down.
/// Group root of the approval-deploy (`MergeConfigPr`) subtree — the
/// **brace** that owns the deploy window. See _Braces_ and _Approvals_.
DeployWindow { agent: String, approval_id: i64 },
/// Group root of a rebuild subtree — the **brace**: declares the agent
/// lease and the build slot, holds both for the whole subtree, and performs
/// no work of its own. Same pure-resource-holder shape as
/// [`NodeKind::DeployWindow`], scoped to one agent.
///
/// Its children declare no resources and borrow these grants, which is what
/// lets `Prebuild` run beside the `Signal` → `Drain` window. Why braces
/// exist and what they cost: `docs/scheduler/coordinator.md`, _Braces_.
/// Group root of a rebuild subtree — the **brace**, scoped to one
/// agent. Why braces exist and what they cost: _Braces_.
AgentWindow { agent: String },
/// Deploy phase 1 — **verify only, mutates nothing.** Drift-gate the
/// approval's PR head, fetch it into the applied repo, and eval-verify the
/// merge head. Any failure here aborts the deploy with the forge state
/// untouched, so it is safely retryable and cancel-safe: nothing downstream
/// has happened yet.
/// Deploy phase 1 — **verify only, mutates nothing.**
MergeVerify { agent: String, approval_id: i64 },
/// Deploy phase 2 — the irreversible fast-forward plus the *opening* half of
/// the two-phase meta deploy: park the rollback ref, ff-merge the reviewed
/// head to `main` via the forge API, ff `applied/main`, and
/// `meta::prepare_deploy` (which stages `flake.lock` uncommitted).
///
/// It does **not** run the container rebuild itself. It grows the ordinary
/// rebuild subgraph into this DAG as its own children
/// ([`super::templates::deploy_rebuild_nodes`], `relock = false` — the lock
/// is already staged), so the multi-minute build renders as the same real
/// nodes every other rebuild does instead of one opaque box. Closing the
/// staged-lock window is likewise its own node
/// ([`NodeKind::FinalizeDeploy`]), and the compensation path is
/// [`NodeKind::DeployTail`].
/// Deploy phase 2 — the irreversible fast-forward plus the *opening*
/// half of the two-phase meta deploy. Does **not** build the container
/// itself; grows the ordinary rebuild subgraph in as its own children
/// (see the rebuild-path section's approval-deploy paragraph).
DeployApply { agent: String, approval_id: i64 },
/// Deploy phase 3 — close the two-phase meta deploy once the rebuild
/// subgraph under [`NodeKind::DeployApply`] has come up clean: drop the
/// rollback ref, plant the `deployed/<id>` tag, commit the staged
/// `flake.lock` (`meta::finalize_deploy`).
///
/// Its two git steps are **fatal**, deliberately. They are the writes that
/// tell [`NodeKind::DeployTail`] a deploy confirmed good, so a node that
/// merely warned on them could report success while leaving the tail
/// looking at the git state of a failure — and the tail would then roll a
/// *good* deploy back. Failing loudly keeps the node's outcome and the
/// repo's state saying the same thing.
///
/// The trailing `meta::finalize_deploy` stays warn-only: by then the
/// container already runs the new config, and an uncommitted staged lock is
/// something the operator can commit by hand.
/// Deploy phase 3 — closes the two-phase meta deploy once
/// [`NodeKind::DeployApply`]'s rebuild subgraph comes up clean.
FinalizeDeploy { agent: String, approval_id: i64 },
/// Deploy compensation **and bookkeeping** tail — `AfterAny`
/// [`NodeKind::DeployApply`], so it runs on success, failure, and cancel
/// alike, in the same spirit as the rebuild template's tail `Reconcile`
/// ("always runs, decides internally"). It:
/// 1. compensates a merge that landed but was never finalized — roll `main`
/// back, reset the tree, `meta::abort_deploy`, plant `failed/<id>`;
/// 2. mirrors whichever deploy tag got planted to the forge config repo
/// (`forge::push_config`), always, best-effort;
/// 3. posts the failing build log back onto the config PR when the deploy
/// failed, so the manager sees the rejection without leaving the forge.
///
/// Steps 2 and 3 are why this is `DeployTail` and not `AbortDeploy`: it has
/// work to do on the success path too, and a node name that claims
/// otherwise would be a lie on the dashboard.
///
/// For (1) it needs no knowledge of how far the deploy got, because that state is
/// parked in the applied repo rather than passed between nodes:
/// `DeployApply` writes the pre-merge `main` sha to
/// `refs/hyperhive/rollback/<approval-id>` before the fast-forward and
/// [`NodeKind::FinalizeDeploy`] deletes it. So the ref existing *is*
/// the "a merge landed and was not finalized" signal, and its absence makes
/// this node a no-op. Parking it in git rather than in a node payload also
/// means it survives a `hive-c0re` restart mid-deploy, which an in-memory
/// queue does not.
/// Deploy compensation **and bookkeeping** tail. See the `DeployTail`
/// node-inventory row for its three responsibilities and why it isn't
/// named `AbortDeploy`.
DeployTail { agent: String, approval_id: i64 },
/// Tail node of an approval-carrying DAG (spawn / opaque deploy / config-PR
/// merge): resolve the approval row from how the work actually ended.
///
/// Weak-edged (`DepWhen::AFTER_ANY`) like [`NodeKind::DeployTail`], so it runs on
/// success, failure **and cancel** alike and decides internally. It reads its
/// dependencies' terminal states from the node's dependency list rather than
/// re-deriving them from the world the way `DeployTail` reads git: a node is
/// *told* how the work it follows ended, it does not go back out and ask.
///
/// Agentless on purpose: the approval row already names its agent, so
/// carrying one here would be a second copy free to drift. Like
/// [`NodeKind::MetaLock`] it reports `""` from [`NodeKind::agent`] and takes
/// no lease — which is also what lets one close a multi-agent DAG.
/// Tail node of an approval-carrying DAG (spawn / opaque deploy /
/// config-PR merge): resolve the approval row from how the work ended.
ResolveApproval {
approval_id: i64,
/// Which outcome this node reports. A template emits **one per outcome**,
/// each edged to accept only that one, so exactly one is ever runnable
/// and the executor has nothing to decide — it resolves the row the way
/// its own variant says. The `Cancelled` one is also the node that
/// [`super::JobQueue::cancel`] spares, since its edge is the only one
/// that accepts a dropped dependency.
/// Which outcome this node reports. A template emits **one per
/// outcome**, each edged to accept only that one, so exactly one is
/// ever runnable. The `Cancelled` one is also the node
/// [`super::JobQueue::cancel`] spares, since its edge is the only
/// one that accepts a dropped dependency.
outcome: TerminalState,
},
/// Tail node of a rebuild / perm-change: emit this agent's `Rebuilt` manager
/// event — `ok` when its deps are `Done`, `!ok` carrying the failure note when
/// they `Failed`, and **nothing at all** when they `Cancelled` (a cancelled DAG
/// never ran, so there is no rebuild to report).
///
/// One node per **agent** — a multi-agent DAG reports each agent's own
/// outcome rather than painting all of them with the whole DAG's roll-up —
/// and one per **outcome**: `ok` isn't computed here, it's which of the pair
/// the graph let run.
///
/// No cancel variant, deliberately: a DAG dropped before it started has no
/// rebuild to report, and neither tail's edge accepts `Cancelled`, so both
/// are cancelled with the rest and nothing is emitted.
/// Tail node of a rebuild / perm-change: emit this agent's `Rebuilt`
/// manager event. One node per agent *and* per outcome; no cancel
/// variant (a cancelled DAG never ran, so there is no rebuild to
/// report).
EmitRebuilt { agent: String, ok: bool },
/// Write the agent's durable power intent (`wanted = Up` when `up`, else
/// `Offline`) as a first-class DAG node, at the head of a power-op
/// template so the downstream `Reconcile` reads it. Replaces the old
/// pre-submit `set_wanted` side effect: the intent write is now part of
/// the atomic DAG (crash-safe, per-agent — a multi-agent DAG carries one
/// `SetWanted` per agent). Build-slot-exempt (a store write), but
/// **lease-needing**: it takes the agent's lifecycle lease so the whole
/// power-op DAG (intent write → reconcile) is atomic per-agent — two
/// racing ops (e.g. restart vs stop) can't clobber each other's intent
/// before either reconciles, which is the point of moving the write into
/// the DAG. (In `stale_start` the lease is thus held across the head
/// `Prebuild`, but that's a no-op there — the agent is down, so prebuild
/// is skipped.)
/// Write the agent's durable power intent (`wanted = Up` when `up`,
/// else `Offline`) as the head node of a power-op DAG.
SetWanted { agent: String, up: bool },
/// One-shot boot-time forge user/token sweep for every existing
/// container (`forge::ensure_all`) — moved out of a bare `tokio::spawn`
/// so it shows as real work on the dashboard instead of running
/// invisibly until it fails. Agentless: it sweeps every container, not
/// one. Store/network-I/O only — build-slot- and lease-exempt, same
/// class as [`NodeKind::MetaSync`]/[`NodeKind::Provision`].
/// container (`forge::ensure_all`). Agentless.
ForgeSweep,
/// One-shot boot-time matrix user/space sweep (`matrix::ensure_all`),
/// for the same dashboard-visibility reason as [`NodeKind::ForgeSweep`].
/// The *periodic* re-sweep (every 30 min, recovering token files
/// `hive-matrix-daemon` deleted) stays a background loop in
/// `main.rs` — only the boot-time instance is a DAG node.
/// One-shot boot-time matrix user/space sweep (`matrix::ensure_all`).
/// Agentless.
MatrixSweep,
/// One-shot boot-time Forgejo webhook registration, for
/// `internal/knowledge` (push → git pull) and the `agent-configs` org
/// (`pull_request` → config-PR approval). No-op when the core token, hive
/// domain, or HMAC secret aren't available yet — mirrors the guard the
/// `tokio::spawn` block it replaced already used. Agentless, build-slot-
/// and lease-exempt.
/// One-shot boot-time Forgejo webhook registration. Agentless.
WebhookRegister,
/// One-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling
/// any commits that landed while `hive-c0re` was down. Same rationale as
/// [`NodeKind::MatrixSweep`]: the periodic hourly re-pull stays a
/// background loop, only the boot-time instance is a DAG node.
/// One-shot boot-time `/knowledge` pull (`knowledge::pull`). Agentless.
KnowledgePull,
/// One-shot boot-time pull of the agent set the swarm controller declares
/// for this hive (`wanted::pull`), converging the agents it names.
///
/// Unlike the three above there is **no** background loop behind this one:
/// boot is the whole cadence. The per-agent fast path is the deploy event
/// (`swarm_status`), and this is what repairs a missed one. Agentless — it
/// reads the whole declaration, not one agent.
/// One-shot boot-time pull of the agent set the swarm controller
/// declares for this hive (`wanted::pull`). Agentless.
WantedPull,
}

45
hive-c0re/src/job_queue/resource.rs
View file

@ -15,46 +15,21 @@
//! the meta window) cannot deadlock against each other.
/// The two resource classes the queue gates concurrency on, as the crate's
/// generic resource type `R`.
/// generic resource type `R`. What each holds, who declares it, and the
/// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_
/// (build slots, the per-agent lease) and _Two further layers protect the
/// meta repo_ (the deploy window) sections — this enum stays the one-line
/// summary, not a second copy.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum Resource {
/// One of the `buildSlots` permits, held by a nix-heavy node for its
/// duration. Capacity is `services.hyperhive.c0re.buildSlots` (default 1),
/// set on the [`hive_jobq::resources::ResourceTable`] at construction.
/// duration.
BuildSlot,
/// The per-agent lifecycle lease — globally exclusive per agent across all
/// DAGs (unconfigured, so the crate's default capacity 1 applies). Held by
/// a DAG's first container-affecting node for that agent and re-entered by
/// the rest of that agent's subtree via the crate's recursive lock, so two
/// DAGs never interleave container ops on one agent.
///
/// Nodes that touch the *store or meta repo* rather than the running
/// container do not declare it — `MetaSync`, `Prebuild`, `Provision`,
/// `MetaLock`, `WritePermFile`, `Reparent`. That exemption is what lets a
/// `Prebuild` overlap another DAG's work on the same agent. `Provision`
/// precedes the container's existence entirely, so the lease is first taken
/// at the `Create` it feeds.
/// The per-agent lifecycle lease — globally exclusive per agent across
/// all DAGs.
Agent(String),
/// The meta-repo mutation window — a global singleton (default capacity 1)
/// held by any node that mutates the meta repo, so two meta mutations never
/// interleave. That exclusion is load-bearing: a commit landing inside
/// another node's staged `prepare_deploy`→`finalize_deploy` window would
/// sweep the staged `flake.lock` into its own commit and neuter
/// `abort_deploy`. Replaces the former runtime `meta::exclusive()` mutex: a
/// `MutexGuard` cannot span scheduler nodes, but a resource held by a
/// subtree root *can* — which is what lets the two-phase deploy
/// (`prepare_deploy` stages `flake.lock` uncommitted across the whole
/// container build, `finalize_deploy`/`abort_deploy` resolve it) be
/// decomposed into sub-nodes instead of one opaque node. Descendants of a
/// holder re-enter it through the crate's recursive lock, exactly like
/// [`Resource::Agent`].
///
/// Deliberately **not** declared by `Prebuild`: the window must stay off the
/// multi-minute toplevel build, which only *reads* the store. Holding a
/// hive-global cap-1 across it would serialize every agent's rebuild behind
/// every other's — which is why the meta preamble is its own `MetaSync`
/// node, and a sibling of `Prebuild` rather than its parent (a resource is
/// held across the holder's whole subtree).
/// The meta-repo mutation window — a global singleton held by any node
/// that mutates the meta repo, so two meta mutations never interleave.
MetaWindow,
}