diff --git a/docs/scheduler/coordinator.md b/docs/scheduler/coordinator.md index 74cdb1b4..8d753aba 100644 --- a/docs/scheduler/coordinator.md +++ b/docs/scheduler/coordinator.md @@ -44,7 +44,7 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration: | Node | Wraps | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) | +| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node). Skipped when the container is already down; `Swap` builds inline instead | | `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node | | `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent in meta | | `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion | @@ -56,22 +56,23 @@ Cheap — no build slot: | Node | Behavior | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation | -| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail — rolls `applied/main` back from the parked `refs/hyperhive/rollback/` and aborts the staged meta lock when the deploy never confirmed good, then mirrors the config repo to the forge. Infallible by construction | +| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail: (1) rolls `applied/main` back from the parked `refs/hyperhive/rollback/` and aborts the staged meta lock when the deploy never confirmed good; (2) mirrors whichever deploy tag landed to the forge config repo, always, best-effort; (3) posts the failing build log back onto the config PR when the deploy failed. Named for (2)/(3), which run on the success path too — not `AbortDeploy`. Infallible by construction | | `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build | | `Provision` | first-spawn pre-create provisioning — proposed/applied repos, state subvolume, meta registration (`sync_agents`); runs ahead of `Create` so the `nixos-container create --flake meta#` ref resolves. Store/meta-only, no container yet | | `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | | `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down | | `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | -| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here) | +| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace | | `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ | | `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn | | `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way | -| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's own between-turns poll is already responsive enough | +| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here | | `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` | | `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims | | `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state/applied dirs. Own node because it's conditional and the irreversible step | -| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap` | +| `DestroyBookkeeping` | the post-destroy tail — meta sync, fail pending approvals, drop the power intent, notify the manager, rescan, re-emit the tombstone, resync tmpfiles. Same split rationale as `RebuildBookkeeping`/`Swap`. Its `purge` flag only selects the wording of the approval-failure reason and the manager notification — the destructive work is `PurgeState`'s | +| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG, replacing the old pre-submit side effect. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles | | `FinalizeDeploy` | deploy phase 3 — drop the rollback ref, plant `deployed/`, commit the staged `flake.lock`. The first two git steps are fatal on purpose, so a confirmed-good deploy's outcome and the repo's state can't disagree | | `ResolveApproval` | tail of an approval-carrying DAG — resolve the approval row from how the work ended (`AfterAny`, one node emitted per outcome). Agentless: the approval row already names its agent | | `EmitRebuilt` | tail of a rebuild/perm-change — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent *and* per outcome | @@ -220,9 +221,9 @@ resources are free. Resources: (`SetWanted` is a store write, not a container op, but takes the lease anyway so a power-op DAG's intent write + reconcile is atomic — two racing ops can't clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`, - `Prebuild`, `MetaLock`, `WritePermFile`, `Reparent` — they touch the store / - meta, not the running container, which is exactly why a stop can land while - another DAG's prebuild is still building. Also exempt, for a different + `Prebuild`, `Provision`, `MetaLock`, `WritePermFile`, `Reparent` — they + touch the store / meta, not the running container, which is exactly why a + stop can land while another DAG's prebuild is still building. Also exempt, for a different reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`, `Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the container, but their `AgentWindow` brace holds the lease above them — see @@ -306,6 +307,13 @@ compensation step that survives a hive-c0re restart — the pre-merge local variable, so `DeployTail` can still undo a half-finished deploy after a crash. +`DeployWindow` declares all three resources (build slot, lease, meta window) +on itself rather than letting each phase declare its own, because the queue +acquires a node's resources atomically (all-or-nothing): a child that took +the build slot while its parent held the meta window could block waiting for +a resource its own parent already committed to, a lock-ordering hazard that +one multi-resource root avoids by construction. + `Spawn` and `UpdateMetaInputs` approvals map onto the ordinary `spawn` / `meta-update` shapes. The scheduler fires `actions::resolve_approval_dag` exactly once when **any** approval-carrying DAG settles terminal — deploys @@ -427,8 +435,9 @@ Sequence for a rebuild DAG (each step is its own queue node): The container keeps serving the previous generation while eval + fetch + build happen out-of-band. `nixos-container update` then finds the result cached and skips straight to the profile-swap. Build failures surface - here, before the running container is touched. (Runs even for a stopped - container — same total nix work, one uniform DAG shape.) + here, before the running container is touched. Skipped when the + container is already down — a stopped agent has no uptime to preserve, + so `Swap` builds inline instead of paying the double eval. 3. `StopForUpdate` — bring the container down (noop when already stopped). 4. `Swap` — `nixos-container update --flake meta#` profile-swap (near-instant after the prebuild). diff --git a/hive-c0re/src/job_queue/model.rs b/hive-c0re/src/job_queue/model.rs index 3f329dfa..6ac92760 100644 --- a/hive-c0re/src/job_queue/model.rs +++ b/hive-c0re/src/job_queue/model.rs @@ -26,118 +26,59 @@ use hive_jobq::TerminalState; /// functions themselves, which is why there is no `GitCommit` node — /// a standalone commit node would open a dirty-working-tree window /// between nodes that the fused `meta.rs` ops deliberately close. +/// +/// What each kind does, the resource/lease rules, and the DAG shapes they +/// compose into live in the node-inventory table and surrounding sections +/// of `docs/scheduler/coordinator.md` — this enum is deliberately not a +/// second copy of that; each variant below gets a one-line pointer. #[derive(Debug, Clone, PartialEq, Eq, Serialize)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum NodeKind { - /// The rebuild's meta-repo preamble: `lifecycle::prepare_rebuild_dirs`, - /// an idempotent meta `sync_agents`, and an optional per-agent relock. - /// `relock = false` only for meta-update cascade rebuilds (re-locking - /// would revert the bump the cascade just committed). - /// - /// Its own node — ahead of, and *not* an ancestor of, [`NodeKind::Prebuild`] - /// — precisely because it is the only part of the rebuild that mutates the - /// meta repo and so holds the global - /// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow). Fusing - /// it into `Prebuild` (or making it `Prebuild`'s parent, which holds a - /// resource across the whole subtree) would extend that global window over - /// the multi-minute toplevel build and serialize rebuilds hive-wide. - /// Store/meta work only — build-slot- and lease-exempt. + /// The rebuild's meta-repo preamble. `relock = false` only for + /// meta-update cascade rebuilds (re-locking would revert the bump the + /// cascade just committed). MetaSync { agent: String, relock: bool }, - /// Out-of-band toplevel build while the container keeps serving: - /// `lifecycle::prebuild_toplevel`, reading a meta repo the upstream - /// [`NodeKind::MetaSync`] has already synced. The warm build is skipped - /// when the container is already down — it only exists to shrink the - /// swap's downtime, which a stopped agent doesn't need (`Swap` builds - /// inline instead). + /// Out-of-band toplevel build while the container keeps serving. Prebuild { agent: String }, - /// `nixos-container update` profile-swap (requires the container - /// stopped). Re-applies nspawn flags + resource limits first — - /// rebuild is the reconcile verb. The post-rebuild bookkeeping tail - /// lives in the sibling `RebuildBookkeeping` node. + /// `nixos-container update` profile-swap. Swap { agent: String }, - /// The post-`Swap` bookkeeping tail as a first-class node: rev marker, - /// forge + matrix sync, manager kick, container rescan, meta-inputs - /// snapshot. Split out of `Swap` for dashboard visibility + retry - /// granularity. Deps `AfterOk(Swap)`, so it runs only when the profile - /// swap succeeded; the tail `Reconcile` deps `AfterAny(RebuildBookkeeping)`, so on - /// swap failure this node is cancel-cascaded (a terminal state) and - /// recovery still runs. Store/forge/matrix work only — no nix build. - /// - /// Declares **no resources**: it is a coordinated child of - /// [`NodeKind::AgentWindow`], which holds the agent lease (and the build - /// slot) for the whole rebuild subtree. See `templates.rs`'s module doc for - /// why the brace is the one place a resource is declared on behalf of - /// others. + /// The post-`Swap` bookkeeping tail as a first-class node. RebuildBookkeeping { agent: String }, - /// First-spawn pre-create provisioning: proposed/applied repos, - /// state subvolume, and meta registration (`sync_agents`). Runs - /// ahead of `Create` so the `nixos-container create --flake - /// meta#` ref resolves. Store/meta-only — no container yet — - /// so it's lease- and build-slot-exempt like `Prebuild`. + /// First-spawn pre-create provisioning. Provision { agent: String }, /// First-spawn `nixos-container create` proper. Assumes the /// upstream `Provision` node already registered the agent in meta. Create { agent: String }, - /// `nixos-container destroy` plus the un-registration that follows it: - /// drop the agent from the coordinator's roster and clear its ephemeral - /// runtime dir. + /// `nixos-container destroy` plus the un-registration that follows it. /// - /// **Deliberately not in [`NodeKind::takes_container_down`]**, and that - /// is the design rather than an oversight. This node runs *downstream of - /// a `Stop`*, which already carries the flag honestly, so by the time it - /// claims there is nothing left to take down. A container still live here - /// is a real bug and must page someone — a `true` would absorb exactly - /// that signal, and the flag's whole asymmetry (see that method) is that - /// a wrong `true` silently swallows a crash. + /// Deliberately excluded from [`NodeKind::takes_container_down`] — see + /// that method for why a node downstream of a `Stop` must not claim the + /// container going down as its own. DestroyContainer { agent: String }, /// The `purge = true` half of a destroy: delete the agent's state /// subvolume (via hive-priv, since a subvolume root defeats - /// `remove_dir_all`) plus its state and applied dirs. Its own node - /// because it is conditional — a plain destroy never inserts it — and - /// because it is the irreversible step, so it earns a distinct row in - /// the graph rather than hiding inside a bookkeeping tail. + /// `remove_dir_all`) plus its state and applied dirs. PurgeState { agent: String }, - /// The post-destroy bookkeeping tail: meta sync, fail the agent's pending - /// approvals, drop the durable power intent, notify the manager, rescan - /// containers, re-emit the tombstone + schedule snapshots, resync - /// tmpfiles. Split from [`NodeKind::DestroyContainer`] for the same - /// reason [`NodeKind::RebuildBookkeeping`] is split from `Swap`: - /// dashboard visibility and retry granularity for work that is pure - /// store/meta bookkeeping and touches no container. - /// - /// `purge` only selects the wording of the approval-failure reason and - /// the manager notification; the destructive work is `PurgeState`'s. + /// The post-destroy bookkeeping tail. DestroyBookkeeping { agent: String, purge: bool }, - /// Meta flake lock bump. `sweep = false`: `meta::lock_update` - /// (commit fused, under `META_LOCK`) with this node's own `inputs`; - /// `sweep = true`: `meta::lock_update_hyperhive`, *non-fatal* (a - /// failed boot-time bump must not cancel the fan-out rebuilds). - /// On success the scheduler appends child `Rebuild` DAGs: the - /// precomputed `fanout` list when present (boot sweep), else the - /// post-bump affected set (`meta_update_cascade_agents`). - /// - /// `inputs` are the flake inputs to bump — empty means "all", and the - /// boot sweep leaves them empty since it bumps `hyperhive` alone. They - /// ride this node because it is the only thing that reads them. + /// Meta flake lock bump. `sweep = false`: `meta::lock_update` with this + /// node's own `inputs`; `sweep = true`: `meta::lock_update_hyperhive` + /// (boot sweep, `inputs` empty). See _Notable collapses_ for how the + /// scheduler appends the fan-out `Rebuild` DAGs on completion. MetaLock { sweep: bool, fanout: Option>, inputs: Vec, }, - /// Idempotent power converge *planner*: read `wanted` + observed - /// state and decide the action (start if `Up` & down, stop if - /// `Offline` & up, else noop). The mechanical work is not done in - /// this node — it fans a child [`NodeKind::Start`] / [`NodeKind::Stop`] - /// DAG out at runtime so the sub-step is a first-class DAG node. + /// Idempotent power converge *planner*: fans a child + /// [`NodeKind::Start`] / [`NodeKind::Stop`] out at runtime rather than + /// doing the mechanical work itself. Reconcile { agent: String }, - /// Mechanical container start: the start preamble (runtime dir + - /// drop-ins), `start_with_fallback`, MCP listener registration, and - /// the manager kick. Fanned out by a [`NodeKind::Reconcile`] that - /// observed `wanted = Up` and the container down. + /// Mechanical container start. Fanned out by a [`NodeKind::Reconcile`] + /// that observed `wanted = Up` and the container down. Start { agent: String }, - /// Mechanical container stop: `nixos-container` kill, MCP listener - /// unregister, and the `Killed` manager notify. Fanned out by a - /// [`NodeKind::Reconcile`] that observed `wanted = Offline` and up. + /// Mechanical container stop. Fanned out by a [`NodeKind::Reconcile`] + /// that observed `wanted = Offline` and up. Stop { agent: String }, /// Mechanical `nixos-container stop` for the profile swap. Never /// touches `wanted`. Noop if already stopped. @@ -150,10 +91,7 @@ pub enum NodeKind { /// downstream `Reconcile` performs the actual stop. Drain { agent: String }, /// Write the pause marker (`Coordinator::set_paused`) + mark - /// `pause_pending`. No kick, unlike `Signal` — the harness's own - /// between-turns poll (`PAUSE_POLL`, 1s default) is already - /// responsive enough, and `Signal`'s kick-message body ("you were - /// just (re)started") would be actively misleading here. + /// `pause_pending`. PauseSignal { agent: String }, /// Await the harness reporting `PauseAcknowledged`, bounded by a /// timeout. Resolves ok either way — pausing is best-effort from @@ -161,197 +99,65 @@ pub enum NodeKind { PauseDrain { agent: String }, /// `set_nspawn_flags` + `set_resource_limits` + daemon-reload. WriteDropin { agent: String }, - /// Commit `tool-groups.json` / `capabilities.json` per its `payload` - /// (commit fused under `META_LOCK`). The payload rides this node — the only - /// consumer — rather than the generic DAG container. + /// Commit `tool-groups.json` / `capabilities.json` per its `payload`. WritePermFile { agent: String, payload: PermPayload }, /// Topology move(s) — `set-parent` (len 1) or `set-parent-bulk` (len N) — - /// as a single queue node. Agentless like [`NodeKind::MetaLock`]: a - /// reparent touches the meta repo, not any one container, and a bulk - /// move spans multiple agents anyway. Declares the meta window, same - /// precedent as [`NodeKind::WritePermFile`] (also a small - /// git-commit-under-`META_LOCK` op) — a reparent's commit must not land - /// inside another node's staged deploy `prepare_deploy`→`finalize_deploy` - /// window. `(child, new_parent)` pairs, applied in order under one - /// `META_LOCK` acquisition / one git commit (`meta::bulk_commit_topology` - /// handles both the single- and multi-move case uniformly). + /// as a single queue node. `moves` is typed `(Ident, Option)` + /// pairs, applied in order under one `META_LOCK` acquisition. Reparent { moves: Vec<(hive_types::Ident, Option)>, }, - /// Group root of the approval-deploy (`MergeConfigPr`) subtree, and the - /// node that **owns the deploy window**. It performs no work of its own — - /// it exists so the resources it declares (the global - /// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow), the - /// agent lease, a build slot) are held continuously across every child - /// phase, which a per-node acquisition could not guarantee. - /// - /// All three resources are declared *here*, on one node, on purpose. The - /// queue acquires a node's resources atomically (all-or-nothing), so a - /// single multi-resource root can never hold one and block on another — - /// whereas letting a child take the build slot while its parent held the - /// meta window would introduce exactly that pattern, and with it a - /// lock-ordering argument that has to be re-verified on every future edit. - /// Cheap, too: the window has to span the container build regardless (see - /// [`NodeKind::DeployApply`]), so nothing is over-serialised by hoisting - /// the slot and the lease up alongside it. - /// - /// Carries the approval row every phase below it re-reads, like each of - /// those phases does — the id is the node's own payload, not something a - /// DAG-level catch-all hands down. + /// Group root of the approval-deploy (`MergeConfigPr`) subtree — the + /// **brace** that owns the deploy window. See _Braces_ and _Approvals_. DeployWindow { agent: String, approval_id: i64 }, - /// Group root of a rebuild subtree — the **brace**: declares the agent - /// lease and the build slot, holds both for the whole subtree, and performs - /// no work of its own. Same pure-resource-holder shape as - /// [`NodeKind::DeployWindow`], scoped to one agent. - /// - /// Its children declare no resources and borrow these grants, which is what - /// lets `Prebuild` run beside the `Signal` → `Drain` window. Why braces - /// exist and what they cost: `docs/scheduler/coordinator.md`, _Braces_. + /// Group root of a rebuild subtree — the **brace**, scoped to one + /// agent. Why braces exist and what they cost: _Braces_. AgentWindow { agent: String }, - /// Deploy phase 1 — **verify only, mutates nothing.** Drift-gate the - /// approval's PR head, fetch it into the applied repo, and eval-verify the - /// merge head. Any failure here aborts the deploy with the forge state - /// untouched, so it is safely retryable and cancel-safe: nothing downstream - /// has happened yet. + /// Deploy phase 1 — **verify only, mutates nothing.** MergeVerify { agent: String, approval_id: i64 }, - /// Deploy phase 2 — the irreversible fast-forward plus the *opening* half of - /// the two-phase meta deploy: park the rollback ref, ff-merge the reviewed - /// head to `main` via the forge API, ff `applied/main`, and - /// `meta::prepare_deploy` (which stages `flake.lock` uncommitted). - /// - /// It does **not** run the container rebuild itself. It grows the ordinary - /// rebuild subgraph into this DAG as its own children - /// ([`super::templates::deploy_rebuild_nodes`], `relock = false` — the lock - /// is already staged), so the multi-minute build renders as the same real - /// nodes every other rebuild does instead of one opaque box. Closing the - /// staged-lock window is likewise its own node - /// ([`NodeKind::FinalizeDeploy`]), and the compensation path is - /// [`NodeKind::DeployTail`]. + /// Deploy phase 2 — the irreversible fast-forward plus the *opening* + /// half of the two-phase meta deploy. Does **not** build the container + /// itself; grows the ordinary rebuild subgraph in as its own children + /// (see the rebuild-path section's approval-deploy paragraph). DeployApply { agent: String, approval_id: i64 }, - /// Deploy phase 3 — close the two-phase meta deploy once the rebuild - /// subgraph under [`NodeKind::DeployApply`] has come up clean: drop the - /// rollback ref, plant the `deployed/` tag, commit the staged - /// `flake.lock` (`meta::finalize_deploy`). - /// - /// Its two git steps are **fatal**, deliberately. They are the writes that - /// tell [`NodeKind::DeployTail`] a deploy confirmed good, so a node that - /// merely warned on them could report success while leaving the tail - /// looking at the git state of a failure — and the tail would then roll a - /// *good* deploy back. Failing loudly keeps the node's outcome and the - /// repo's state saying the same thing. - /// - /// The trailing `meta::finalize_deploy` stays warn-only: by then the - /// container already runs the new config, and an uncommitted staged lock is - /// something the operator can commit by hand. + /// Deploy phase 3 — closes the two-phase meta deploy once + /// [`NodeKind::DeployApply`]'s rebuild subgraph comes up clean. FinalizeDeploy { agent: String, approval_id: i64 }, - /// Deploy compensation **and bookkeeping** tail — `AfterAny` - /// [`NodeKind::DeployApply`], so it runs on success, failure, and cancel - /// alike, in the same spirit as the rebuild template's tail `Reconcile` - /// ("always runs, decides internally"). It: - /// 1. compensates a merge that landed but was never finalized — roll `main` - /// back, reset the tree, `meta::abort_deploy`, plant `failed/`; - /// 2. mirrors whichever deploy tag got planted to the forge config repo - /// (`forge::push_config`), always, best-effort; - /// 3. posts the failing build log back onto the config PR when the deploy - /// failed, so the manager sees the rejection without leaving the forge. - /// - /// Steps 2 and 3 are why this is `DeployTail` and not `AbortDeploy`: it has - /// work to do on the success path too, and a node name that claims - /// otherwise would be a lie on the dashboard. - /// - /// For (1) it needs no knowledge of how far the deploy got, because that state is - /// parked in the applied repo rather than passed between nodes: - /// `DeployApply` writes the pre-merge `main` sha to - /// `refs/hyperhive/rollback/` before the fast-forward and - /// [`NodeKind::FinalizeDeploy`] deletes it. So the ref existing *is* - /// the "a merge landed and was not finalized" signal, and its absence makes - /// this node a no-op. Parking it in git rather than in a node payload also - /// means it survives a `hive-c0re` restart mid-deploy, which an in-memory - /// queue does not. + /// Deploy compensation **and bookkeeping** tail. See the `DeployTail` + /// node-inventory row for its three responsibilities and why it isn't + /// named `AbortDeploy`. DeployTail { agent: String, approval_id: i64 }, - /// Tail node of an approval-carrying DAG (spawn / opaque deploy / config-PR - /// merge): resolve the approval row from how the work actually ended. - /// - /// Weak-edged (`DepWhen::AFTER_ANY`) like [`NodeKind::DeployTail`], so it runs on - /// success, failure **and cancel** alike and decides internally. It reads its - /// dependencies' terminal states from the node's dependency list rather than - /// re-deriving them from the world the way `DeployTail` reads git: a node is - /// *told* how the work it follows ended, it does not go back out and ask. - /// - /// Agentless on purpose: the approval row already names its agent, so - /// carrying one here would be a second copy free to drift. Like - /// [`NodeKind::MetaLock`] it reports `""` from [`NodeKind::agent`] and takes - /// no lease — which is also what lets one close a multi-agent DAG. + /// Tail node of an approval-carrying DAG (spawn / opaque deploy / + /// config-PR merge): resolve the approval row from how the work ended. ResolveApproval { approval_id: i64, - /// Which outcome this node reports. A template emits **one per outcome**, - /// each edged to accept only that one, so exactly one is ever runnable - /// and the executor has nothing to decide — it resolves the row the way - /// its own variant says. The `Cancelled` one is also the node that - /// [`super::JobQueue::cancel`] spares, since its edge is the only one - /// that accepts a dropped dependency. + /// Which outcome this node reports. A template emits **one per + /// outcome**, each edged to accept only that one, so exactly one is + /// ever runnable. The `Cancelled` one is also the node + /// [`super::JobQueue::cancel`] spares, since its edge is the only + /// one that accepts a dropped dependency. outcome: TerminalState, }, - /// Tail node of a rebuild / perm-change: emit this agent's `Rebuilt` manager - /// event — `ok` when its deps are `Done`, `!ok` carrying the failure note when - /// they `Failed`, and **nothing at all** when they `Cancelled` (a cancelled DAG - /// never ran, so there is no rebuild to report). - /// - /// One node per **agent** — a multi-agent DAG reports each agent's own - /// outcome rather than painting all of them with the whole DAG's roll-up — - /// and one per **outcome**: `ok` isn't computed here, it's which of the pair - /// the graph let run. - /// - /// No cancel variant, deliberately: a DAG dropped before it started has no - /// rebuild to report, and neither tail's edge accepts `Cancelled`, so both - /// are cancelled with the rest and nothing is emitted. + /// Tail node of a rebuild / perm-change: emit this agent's `Rebuilt` + /// manager event. One node per agent *and* per outcome; no cancel + /// variant (a cancelled DAG never ran, so there is no rebuild to + /// report). EmitRebuilt { agent: String, ok: bool }, - /// Write the agent's durable power intent (`wanted = Up` when `up`, else - /// `Offline`) as a first-class DAG node, at the head of a power-op - /// template so the downstream `Reconcile` reads it. Replaces the old - /// pre-submit `set_wanted` side effect: the intent write is now part of - /// the atomic DAG (crash-safe, per-agent — a multi-agent DAG carries one - /// `SetWanted` per agent). Build-slot-exempt (a store write), but - /// **lease-needing**: it takes the agent's lifecycle lease so the whole - /// power-op DAG (intent write → reconcile) is atomic per-agent — two - /// racing ops (e.g. restart vs stop) can't clobber each other's intent - /// before either reconciles, which is the point of moving the write into - /// the DAG. (In `stale_start` the lease is thus held across the head - /// `Prebuild`, but that's a no-op there — the agent is down, so prebuild - /// is skipped.) + /// Write the agent's durable power intent (`wanted = Up` when `up`, + /// else `Offline`) as the head node of a power-op DAG. SetWanted { agent: String, up: bool }, /// One-shot boot-time forge user/token sweep for every existing - /// container (`forge::ensure_all`) — moved out of a bare `tokio::spawn` - /// so it shows as real work on the dashboard instead of running - /// invisibly until it fails. Agentless: it sweeps every container, not - /// one. Store/network-I/O only — build-slot- and lease-exempt, same - /// class as [`NodeKind::MetaSync`]/[`NodeKind::Provision`]. + /// container (`forge::ensure_all`). Agentless. ForgeSweep, - /// One-shot boot-time matrix user/space sweep (`matrix::ensure_all`), - /// for the same dashboard-visibility reason as [`NodeKind::ForgeSweep`]. - /// The *periodic* re-sweep (every 30 min, recovering token files - /// `hive-matrix-daemon` deleted) stays a background loop in - /// `main.rs` — only the boot-time instance is a DAG node. + /// One-shot boot-time matrix user/space sweep (`matrix::ensure_all`). + /// Agentless. MatrixSweep, - /// One-shot boot-time Forgejo webhook registration, for - /// `internal/knowledge` (push → git pull) and the `agent-configs` org - /// (`pull_request` → config-PR approval). No-op when the core token, hive - /// domain, or HMAC secret aren't available yet — mirrors the guard the - /// `tokio::spawn` block it replaced already used. Agentless, build-slot- - /// and lease-exempt. + /// One-shot boot-time Forgejo webhook registration. Agentless. WebhookRegister, - /// One-shot boot-time `/knowledge` pull (`knowledge::pull`), reconciling - /// any commits that landed while `hive-c0re` was down. Same rationale as - /// [`NodeKind::MatrixSweep`]: the periodic hourly re-pull stays a - /// background loop, only the boot-time instance is a DAG node. + /// One-shot boot-time `/knowledge` pull (`knowledge::pull`). Agentless. KnowledgePull, - /// One-shot boot-time pull of the agent set the swarm controller declares - /// for this hive (`wanted::pull`), converging the agents it names. - /// - /// Unlike the three above there is **no** background loop behind this one: - /// boot is the whole cadence. The per-agent fast path is the deploy event - /// (`swarm_status`), and this is what repairs a missed one. Agentless — it - /// reads the whole declaration, not one agent. + /// One-shot boot-time pull of the agent set the swarm controller + /// declares for this hive (`wanted::pull`). Agentless. WantedPull, } diff --git a/hive-c0re/src/job_queue/resource.rs b/hive-c0re/src/job_queue/resource.rs index a3c08f07..c4cd0c05 100644 --- a/hive-c0re/src/job_queue/resource.rs +++ b/hive-c0re/src/job_queue/resource.rs @@ -15,46 +15,21 @@ //! the meta window) cannot deadlock against each other. /// The two resource classes the queue gates concurrency on, as the crate's -/// generic resource type `R`. +/// generic resource type `R`. What each holds, who declares it, and the +/// exemptions: see `docs/scheduler/coordinator.md`'s _Scheduler semantics_ +/// (build slots, the per-agent lease) and _Two further layers protect the +/// meta repo_ (the deploy window) sections — this enum stays the one-line +/// summary, not a second copy. #[derive(Debug, Clone, PartialEq, Eq, Hash)] pub enum Resource { /// One of the `buildSlots` permits, held by a nix-heavy node for its - /// duration. Capacity is `services.hyperhive.c0re.buildSlots` (default 1), - /// set on the [`hive_jobq::resources::ResourceTable`] at construction. + /// duration. BuildSlot, - /// The per-agent lifecycle lease — globally exclusive per agent across all - /// DAGs (unconfigured, so the crate's default capacity 1 applies). Held by - /// a DAG's first container-affecting node for that agent and re-entered by - /// the rest of that agent's subtree via the crate's recursive lock, so two - /// DAGs never interleave container ops on one agent. - /// - /// Nodes that touch the *store or meta repo* rather than the running - /// container do not declare it — `MetaSync`, `Prebuild`, `Provision`, - /// `MetaLock`, `WritePermFile`, `Reparent`. That exemption is what lets a - /// `Prebuild` overlap another DAG's work on the same agent. `Provision` - /// precedes the container's existence entirely, so the lease is first taken - /// at the `Create` it feeds. + /// The per-agent lifecycle lease — globally exclusive per agent across + /// all DAGs. Agent(String), - /// The meta-repo mutation window — a global singleton (default capacity 1) - /// held by any node that mutates the meta repo, so two meta mutations never - /// interleave. That exclusion is load-bearing: a commit landing inside - /// another node's staged `prepare_deploy`→`finalize_deploy` window would - /// sweep the staged `flake.lock` into its own commit and neuter - /// `abort_deploy`. Replaces the former runtime `meta::exclusive()` mutex: a - /// `MutexGuard` cannot span scheduler nodes, but a resource held by a - /// subtree root *can* — which is what lets the two-phase deploy - /// (`prepare_deploy` stages `flake.lock` uncommitted across the whole - /// container build, `finalize_deploy`/`abort_deploy` resolve it) be - /// decomposed into sub-nodes instead of one opaque node. Descendants of a - /// holder re-enter it through the crate's recursive lock, exactly like - /// [`Resource::Agent`]. - /// - /// Deliberately **not** declared by `Prebuild`: the window must stay off the - /// multi-minute toplevel build, which only *reads* the store. Holding a - /// hive-global cap-1 across it would serialize every agent's rebuild behind - /// every other's — which is why the meta preamble is its own `MetaSync` - /// node, and a sibling of `Prebuild` rather than its parent (a resource is - /// held across the holder's whole subtree). + /// The meta-repo mutation window — a global singleton held by any node + /// that mutates the meta repo, so two meta mutations never interleave. MetaWindow, }