diff --git a/docs/approvals.md b/docs/approvals.md index ca84dc77..d54b590e 100644 --- a/docs/approvals.md +++ b/docs/approvals.md @@ -38,23 +38,33 @@ request. comments, CI status) and sees a matching card on the dashboard with a "review PR on forge" deep link. They click ◆ APPR0VE (or `hivectl approvals approve ` on the CLI) once satisfied. -4. On approve, `run_merge_config_pr`: - - re-reads the live PR head and **aborts if it drifted** from the - reviewed `fetched_sha` (the submitter must push again, which queues - a fresh approval); - - fetches that head into the applied repo and **eval-verifies** it — - a flake eval on a throwaway checkout. This is the trust gate: it - relies on c0re's own eval, not on any in-repo (agent-forgeable) - signal like a CI status; - - fast-forwards the forge config repo's `main` to the reviewed head - (this IS the merge — a `core`-authenticated ff-push, so a moved PR - head can't substitute bytes) and marks the PR merged (best-effort; - `main` is already there); - - runs the shared deploy tail (`deploy_applied_target`): builds under - the tag sequence below, and on success `applied/main` - fast-forwards to the reviewed commit + the meta lock bumps. On - failure, main stays put and the working tree resets to the previous - deployed commit. +4. On approve, a deploy DAG runs three phases under a resource-holding + `DeployWindow` root (see *Queue templates* below): + - `MergeVerify` re-reads the live PR head and **aborts if it drifted** + from the reviewed `fetched_sha` (the submitter must push again, + which queues a fresh approval); then fetches that head into the + applied repo and **eval-verifies** it — a flake eval on a throwaway + checkout. This is the trust gate: it relies on c0re's own eval, not + on any in-repo (agent-forgeable) signal like a CI status. Nothing is + mutated in this phase, so a rejection here leaves the forge and the + applied repo exactly as they were. + - `DeployApply` parks the pre-merge `applied/main` in + `refs/hyperhive/rollback/`, then fast-forward-merges + the reviewed head to the forge config repo's `main` (this IS the + merge — a `core`-authenticated ff-merge pinned to the reviewed sha, + so a moved PR head can't substitute bytes), and runs the deploy + proper (`deploy_applied_target`): ff `applied/main`, two-phase meta + deploy, container rebuild. On success it drops the rollback ref and + plants `deployed/`. + - `DeployTail` runs on **every** outcome, including a cancel-cascade. + If the rollback ref survived, the deploy never confirmed good: it + rolls `applied/main` back, resyncs the working tree, and aborts the + staged meta lock, so the agent stays on its last-good tree. Then it + mirrors the config repo (and its new deploy tag) to the forge. + + The rollback state lives in a **git ref, not a local variable**, on + purpose: hive-c0re can restart between the apply and the tail, and the + tail still has to know what to undo when it does. 5. `HelperEvent::ApprovalResolved` (and `Rebuilt`) land in the **submitting agent's** inbox via `notify_submitter`, carrying both the canonical sha and the terminal tag (the approval row carries a @@ -100,13 +110,12 @@ kind-specific payload carrier. approval row. No MCP tool call needed — the forge PR IS the request. `commit_ref` stores the **PR number** (decimal), and `fetched_sha` is the PR **head sha at queue time** (the "reviewed" sha). On approve, - `run_merge_config_pr` re-reads the live PR head and aborts if it - drifted from `fetched_sha` (submitter must push again to - re-trigger), then fetches that head into the applied repo, - eval-verifies it, fast-forwards the forge config repo's `main` to - it (the merge), marks the PR merged (best-effort — `main` is - already there), and runs the shared deploy tail - (`deploy_applied_target`). Never a first spawn. + the deploy DAG's `MergeVerify` phase re-reads the live PR head and + aborts if it drifted from `fetched_sha` (submitter must push again to + re-trigger), then fetches that head into the applied repo and + eval-verifies it; `DeployApply` fast-forward-merges the forge config + repo's `main` to it (the merge) and runs `deploy_applied_target`; + `DeployTail` compensates on failure. Never a first spawn. - `Spawn` — direct container creation from the agent's config repo. `commit_ref` is empty. Submitted via `HostRequest::RequestSpawn` (operator-gated, the `◆ R3QU3ST SP4WN` dashboard button + @@ -239,8 +248,9 @@ place (it only re-locks when the declared url itself changes), so the forge-declared / applied-deployed split is stable. Per-deploy lock flow (two-phase, owned by -`actions::run_merge_config_pr` → `deploy_applied_target` → -`meta::{prepare,finalize,abort}_deploy`): +`actions::run_deploy_apply` → `deploy_applied_target` → +`meta::{prepare,finalize,abort}_deploy`, with the abort half moved out +into `actions::run_deploy_tail`): 1. `meta::prepare_deploy(name)` runs `nix flake lock --update-input agent-` without @@ -254,10 +264,13 @@ Per-deploy lock flow (two-phase, owned by ")` stages `flake.lock` and commits with `deploy deployed/ `. Meta's git log gains one entry per successful deploy. -4. On failure — `meta::abort_deploy()` runs - `git restore flake.lock` so the meta history shows only +4. On failure — the `DeployTail` node runs `meta::abort_deploy()` + (`git restore flake.lock`) so the meta history shows only successes; the failure stays as an annotated `failed/` - tag in `applied/`. + tag in `applied/`. The tail runs on every outcome, so this + also covers a hive-c0re restart mid-build: the staged lock is + dropped and `applied/main` rolled back from the parked + `refs/hyperhive/rollback/`. Single-phase variants exist for paths without rollback semantics: `meta::lock_update_for_rebuild(name)` for @@ -346,18 +359,22 @@ the approval handler submits a DAG to the global job queue | `ApprovalKind` | DAG submitted | source | |---|---|---| -| `MergeConfigPr` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` | +| `MergeConfigPr` | `rebuild` (`DeployWindow` root + `MergeVerify → DeployApply` + `DeployTail`) | `approval` | | `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` | | `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` | | `InitConfig` | — runs inline (sub-second git seed) | — | | `SchedulePrompt` | — runs inline (single sqlite insert) | — | -The DAG carries the originating `approval_id`. The `ApprovalDeploy` -node runs `run_approval_merge_config_pr` (the two-phase meta deploy -stays inside `actions.rs`) and fires the matching `HelperEvent::*` via -`finish_approval` itself; `Spawn` and `UpdateMetaInputs` DAGs resolve -through `actions::resolve_approval_dag` when the DAG settles terminal -(a spawn additionally runs the post-spawn forge bookkeeping there). +The DAG carries the originating `approval_id`, surfaced on the node that +owns it — for a deploy that's the `DeployWindow` root, so the dashboard +renders one approval card, not four. **Every** queued kind resolves +through `actions::resolve_approval_dag` when its DAG settles terminal: +the deploy's phases are ordinary queue nodes, so the DAG's own terminal +state is the authoritative outcome. That hook fires the matching +`HelperEvent::*` via `finish_approval`, derives the `Rebuilt` event's +terminal tag (verifying the tag actually resolves in the applied repo — +a pre-merge rejection plants none), posts the failing build log back to +the config PR, and for a spawn runs the post-spawn forge bookkeeping. Two visible consequences: diff --git a/docs/coordinator.md b/docs/coordinator.md index a16f11eb..1ed1fbf1 100644 --- a/docs/coordinator.md +++ b/docs/coordinator.md @@ -37,12 +37,15 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration: | `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node | | `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) | | `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion | -| `ApprovalDeploy` | the opaque apply-commit / merge-config-PR pipeline (see _Approvals_ below) | +| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) | +| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild | Cheap — no build slot: | Node | Behavior | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation | +| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail — rolls `applied/main` back from the parked `refs/hyperhive/rollback/` and aborts the staged meta lock when the deploy never confirmed good, then mirrors the config repo to the forge. Infallible by construction | | `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build | | `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | @@ -64,7 +67,8 @@ container build: - **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue resource declared by every node kind that mutates the meta repo — `MetaSync`, `MetaLock`, `WritePermFile`, `Provision`'s agent registration, and - `ApprovalDeploy` for its whole span (`NodeKind::needs_meta_window`). Two meta + `DeployWindow` — the deploy subtree's root, which holds it across every + phase below it (`NodeKind::needs_meta_window`). Two meta mutations can therefore never interleave, so no commit lands inside another node's staged window. It is a queue resource rather than a runtime mutex because a resource is held by a subtree root across its whole subtree, which @@ -180,7 +184,7 @@ resources are free. Resources: per-node; a DAG can span agents) and globally exclusive per agent across all DAGs: acquired at a container-affecting node (`SetWanted`, `StopForUpdate`, `Swap`, `Signal`, `Drain`, `Reconcile`, `WriteDropin`, - `Create`, `ApprovalDeploy`), held by the owning DAG until it's terminal, + `Create`, `DeployWindow`), held by the owning DAG until it's terminal, so two DAGs never interleave container ops on the same agent. A DAG touching several agents holds one lease per agent. (`SetWanted` is a store write, not a container op, but takes the lease anyway so a power-op DAG's @@ -213,15 +217,30 @@ per template. ### Approvals -`MergeConfigPr` approvals ride as single-node -`ApprovalDeploy` DAGs: the two-phase `prepare_deploy` / `finalize_deploy` / -`abort_deploy` meta orchestration stays inside `actions.rs` in v1 -(deliberately not modeled as scheduler nodes) and resolves the approval -itself. `Spawn` and `UpdateMetaInputs` approvals map onto the ordinary -`spawn` / `meta-update` shapes; the scheduler fires -`actions::resolve_approval_dag` exactly once when such a DAG settles -terminal (including cancelled-while-queued, which fails the approval instead -of dangling it). +`MergeConfigPr` approvals ride as a four-node deploy subtree: + +``` +DeployWindow (root — build slot + lease + meta window, no work of its own) +├── MergeVerify drift gate, fetch, verify_commit +├── DeployApply AfterOk(verify) park rollback ref, ff-merge, deploy +└── DeployTail AfterAny(apply) compensate, mirror to forge +``` + +The root's resources are held across the whole subtree, so the two-phase +`prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock` +protected even though the phases are separate nodes. Splitting them buys +three things a single opaque node couldn't have: per-phase visibility on the +dashboard, a `MergeVerify` failure that provably mutated nothing, and a +compensation step that survives a hive-c0re restart — the pre-merge +`applied/main` is parked in `refs/hyperhive/rollback/`, not in a +local variable, so `DeployTail` can still undo a half-finished deploy after a +crash. + +`Spawn` and `UpdateMetaInputs` approvals map onto the ordinary `spawn` / +`meta-update` shapes. The scheduler fires `actions::resolve_approval_dag` +exactly once when **any** approval-carrying DAG settles terminal — deploys +included, since their outcome is now the DAG's own state (including +cancelled-while-queued, which fails the approval instead of dangling it). ### Wire shape @@ -338,8 +357,9 @@ Sequence for a rebuild DAG (each step is its own queue node): The approval apply-commit pipeline still drives `lifecycle::rebuild_no_meta` (the fused stop/update/start path with an inline start) inside its -`ApprovalDeploy` node, because it verifies the agent comes back up before -finalizing the deploy tag. +`DeployApply` node, because it verifies the agent comes back up before +finalizing the deploy tag. Breaking that fused path apart into the +`Prebuild → Swap → Reconcile` chain above is increment 2b of #2664, not 2a. ### Cold-start fallback diff --git a/hive-c0re/src/forge/pr_merge.rs b/hive-c0re/src/forge/pr_merge.rs index bd259e6e..6b685736 100644 --- a/hive-c0re/src/forge/pr_merge.rs +++ b/hive-c0re/src/forge/pr_merge.rs @@ -1,5 +1,5 @@ //! PR-based config-flow merge primitives — the forge-side mechanics -//! hive-c0re's approve-handler (`run_merge_config_pr`) orchestrates to +//! hive-c0re's deploy apply node (`actions::run_deploy_apply`) orchestrates to //! land an operator-approved config PR. Part of the operator trust //! boundary; moved verbatim from the `forge` module root. @@ -15,7 +15,7 @@ use super::{CONFIG_ORG, api, core_token, forge_git_url}; // The dashboard-approve-driven flow has hive-c0re verify an operator-approved // config PR, then land it: fast-forward-merge the verified sha into the // protected default branch via the forge merge API (= the merge). These fns are -// the forge-side mechanics the c0re approve-handler (`run_merge_config_pr`) +// the forge-side mechanics the c0re deploy apply node (`run_deploy_apply`) // orchestrates; the orchestration fetches the verified sha into the agent's // applied repo (for the eval-verify) before calling `merge_config_pr_ff`. The // core token is sourced internally (`core_token`), never passed in. `repo` is diff --git a/hive-c0re/src/forge/repos.rs b/hive-c0re/src/forge/repos.rs index a9fe91d8..5a6957a8 100644 --- a/hive-c0re/src/forge/repos.rs +++ b/hive-c0re/src/forge/repos.rs @@ -907,7 +907,7 @@ async fn apply_operator_branch_protection(repo: &str, token: &str) -> Result<()> /// - **`main` is never directly pushable** — no push is enabled on the /// protected branch, so neither the agent (a write collaborator) nor /// hive-c0re can `git push` it. It only advances via the config-PR merge -/// handler (`run_merge_config_pr`), which fast-forward-*merges* the reviewed +/// node (`actions::run_deploy_apply`), which fast-forward-*merges* the reviewed /// head through the forge merge API (`Do=fast-forward-only`, /// `head_commit_id` pinned to the reviewed sha). /// - **merge is whitelisted to `core`** — only hive-c0re can merge a config PR; diff --git a/hive-c0re/src/lifecycle/setup.rs b/hive-c0re/src/lifecycle/setup.rs index 39d037db..eef1f6ec 100644 --- a/hive-c0re/src/lifecycle/setup.rs +++ b/hive-c0re/src/lifecycle/setup.rs @@ -71,7 +71,7 @@ async fn ensure_applied_remote(proposed_dir: &Path, name: &str) -> Result<()> { /// proposed's initial commit in via `git fetch`, tag it `deployed/0`. /// This is the *only* time hive-c0re reads from `proposed` for an /// agent — subsequent config changes are fetched from the reviewed -/// forge PR head at merge time (see `actions::run_merge_config_pr`). +/// forge PR head at merge time (see `actions::run_deploy_apply`). /// /// `proposed_dir` is `None` on rebuild paths where the repo already /// exists — we just verify it's the right shape and bail otherwise. diff --git a/hive-c0re/src/socket_server/config_approvals.rs b/hive-c0re/src/socket_server/config_approvals.rs index 95f1caaa..6cf7617b 100644 --- a/hive-c0re/src/socket_server/config_approvals.rs +++ b/hive-c0re/src/socket_server/config_approvals.rs @@ -89,7 +89,7 @@ pub(super) fn handle_request_update_meta_inputs( /// dashboard shows the pending card immediately. /// /// The PR head sha is stored as `fetched_sha` on the approval row — the -/// "reviewed sha" the approve handler (`run_merge_config_pr`) drift-gates +/// "reviewed sha" the deploy's `MergeVerify` node drift-gates /// against before doing anything irreversible. This does NOT fetch the commit /// into the applied repo at submission time (that happens inside the approve /// handler, step 2, after the drift check). No flake pre-flight either —