docs: describe the deploy subtree
Rewrite the approval flow's step 4 as the three phases, with the reason the rollback state is a git ref, and refresh the coordinator's node inventory + dispatch table. Fix four doc comments that still pointed at the deleted `run_merge_config_pr`.
This commit is contained in:
atlas
mara
parent
8499c793fe
commit
ff8ada77b7
6 changed files with 92 additions and 55 deletions
|
|
@ -38,23 +38,33 @@ request.
|
|||
comments, CI status) and sees a matching card on the dashboard with a
|
||||
"review PR on forge" deep link. They click ◆ APPR0VE (or
|
||||
`hivectl approvals approve <id>` on the CLI) once satisfied.
|
||||
4. On approve, `run_merge_config_pr`:
|
||||
- re-reads the live PR head and **aborts if it drifted** from the
|
||||
reviewed `fetched_sha` (the submitter must push again, which queues
|
||||
a fresh approval);
|
||||
- fetches that head into the applied repo and **eval-verifies** it —
|
||||
a flake eval on a throwaway checkout. This is the trust gate: it
|
||||
relies on c0re's own eval, not on any in-repo (agent-forgeable)
|
||||
signal like a CI status;
|
||||
- fast-forwards the forge config repo's `main` to the reviewed head
|
||||
(this IS the merge — a `core`-authenticated ff-push, so a moved PR
|
||||
head can't substitute bytes) and marks the PR merged (best-effort;
|
||||
`main` is already there);
|
||||
- runs the shared deploy tail (`deploy_applied_target`): builds under
|
||||
the tag sequence below, and on success `applied/main`
|
||||
fast-forwards to the reviewed commit + the meta lock bumps. On
|
||||
failure, main stays put and the working tree resets to the previous
|
||||
deployed commit.
|
||||
4. On approve, a deploy DAG runs three phases under a resource-holding
|
||||
`DeployWindow` root (see *Queue templates* below):
|
||||
- `MergeVerify` re-reads the live PR head and **aborts if it drifted**
|
||||
from the reviewed `fetched_sha` (the submitter must push again,
|
||||
which queues a fresh approval); then fetches that head into the
|
||||
applied repo and **eval-verifies** it — a flake eval on a throwaway
|
||||
checkout. This is the trust gate: it relies on c0re's own eval, not
|
||||
on any in-repo (agent-forgeable) signal like a CI status. Nothing is
|
||||
mutated in this phase, so a rejection here leaves the forge and the
|
||||
applied repo exactly as they were.
|
||||
- `DeployApply` parks the pre-merge `applied/main` in
|
||||
`refs/hyperhive/rollback/<approval-id>`, then fast-forward-merges
|
||||
the reviewed head to the forge config repo's `main` (this IS the
|
||||
merge — a `core`-authenticated ff-merge pinned to the reviewed sha,
|
||||
so a moved PR head can't substitute bytes), and runs the deploy
|
||||
proper (`deploy_applied_target`): ff `applied/main`, two-phase meta
|
||||
deploy, container rebuild. On success it drops the rollback ref and
|
||||
plants `deployed/<id>`.
|
||||
- `DeployTail` runs on **every** outcome, including a cancel-cascade.
|
||||
If the rollback ref survived, the deploy never confirmed good: it
|
||||
rolls `applied/main` back, resyncs the working tree, and aborts the
|
||||
staged meta lock, so the agent stays on its last-good tree. Then it
|
||||
mirrors the config repo (and its new deploy tag) to the forge.
|
||||
|
||||
The rollback state lives in a **git ref, not a local variable**, on
|
||||
purpose: hive-c0re can restart between the apply and the tail, and the
|
||||
tail still has to know what to undo when it does.
|
||||
5. `HelperEvent::ApprovalResolved` (and `Rebuilt`) land in the
|
||||
**submitting agent's** inbox via `notify_submitter`, carrying both the
|
||||
canonical sha and the terminal tag (the approval row carries a
|
||||
|
|
@ -100,13 +110,12 @@ kind-specific payload carrier.
|
|||
approval row. No MCP tool call needed — the forge PR IS the request.
|
||||
`commit_ref` stores the **PR number** (decimal), and `fetched_sha` is
|
||||
the PR **head sha at queue time** (the "reviewed" sha). On approve,
|
||||
`run_merge_config_pr` re-reads the live PR head and aborts if it
|
||||
drifted from `fetched_sha` (submitter must push again to
|
||||
re-trigger), then fetches that head into the applied repo,
|
||||
eval-verifies it, fast-forwards the forge config repo's `main` to
|
||||
it (the merge), marks the PR merged (best-effort — `main` is
|
||||
already there), and runs the shared deploy tail
|
||||
(`deploy_applied_target`). Never a first spawn.
|
||||
the deploy DAG's `MergeVerify` phase re-reads the live PR head and
|
||||
aborts if it drifted from `fetched_sha` (submitter must push again to
|
||||
re-trigger), then fetches that head into the applied repo and
|
||||
eval-verifies it; `DeployApply` fast-forward-merges the forge config
|
||||
repo's `main` to it (the merge) and runs `deploy_applied_target`;
|
||||
`DeployTail` compensates on failure. Never a first spawn.
|
||||
- `Spawn` — direct container creation from the agent's config repo.
|
||||
`commit_ref` is empty. Submitted via `HostRequest::RequestSpawn`
|
||||
(operator-gated, the `◆ R3QU3ST SP4WN` dashboard button +
|
||||
|
|
@ -239,8 +248,9 @@ place (it only re-locks when the declared url itself changes), so
|
|||
the forge-declared / applied-deployed split is stable.
|
||||
|
||||
Per-deploy lock flow (two-phase, owned by
|
||||
`actions::run_merge_config_pr` → `deploy_applied_target` →
|
||||
`meta::{prepare,finalize,abort}_deploy`):
|
||||
`actions::run_deploy_apply` → `deploy_applied_target` →
|
||||
`meta::{prepare,finalize,abort}_deploy`, with the abort half moved out
|
||||
into `actions::run_deploy_tail`):
|
||||
|
||||
1. `meta::prepare_deploy(name)` runs
|
||||
`nix flake lock --update-input agent-<n>` without
|
||||
|
|
@ -254,10 +264,13 @@ Per-deploy lock flow (two-phase, owned by
|
|||
<id>")` stages `flake.lock` and commits with
|
||||
`deploy <n> deployed/<id> <sha12>`. Meta's git log gains
|
||||
one entry per successful deploy.
|
||||
4. On failure — `meta::abort_deploy()` runs
|
||||
`git restore flake.lock` so the meta history shows only
|
||||
4. On failure — the `DeployTail` node runs `meta::abort_deploy()`
|
||||
(`git restore flake.lock`) so the meta history shows only
|
||||
successes; the failure stays as an annotated `failed/<id>`
|
||||
tag in `applied/<n>`.
|
||||
tag in `applied/<n>`. The tail runs on every outcome, so this
|
||||
also covers a hive-c0re restart mid-build: the staged lock is
|
||||
dropped and `applied/main` rolled back from the parked
|
||||
`refs/hyperhive/rollback/<id>`.
|
||||
|
||||
Single-phase variants exist for paths without
|
||||
rollback semantics: `meta::lock_update_for_rebuild(name)` for
|
||||
|
|
@ -346,18 +359,22 @@ the approval handler submits a DAG to the global job queue
|
|||
|
||||
| `ApprovalKind` | DAG submitted | source |
|
||||
|---|---|---|
|
||||
| `MergeConfigPr` | `rebuild` (single opaque `ApprovalDeploy` node) | `approval` |
|
||||
| `MergeConfigPr` | `rebuild` (`DeployWindow` root + `MergeVerify → DeployApply` + `DeployTail`) | `approval` |
|
||||
| `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` |
|
||||
| `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` |
|
||||
| `InitConfig` | — runs inline (sub-second git seed) | — |
|
||||
| `SchedulePrompt` | — runs inline (single sqlite insert) | — |
|
||||
|
||||
The DAG carries the originating `approval_id`. The `ApprovalDeploy`
|
||||
node runs `run_approval_merge_config_pr` (the two-phase meta deploy
|
||||
stays inside `actions.rs`) and fires the matching `HelperEvent::*` via
|
||||
`finish_approval` itself; `Spawn` and `UpdateMetaInputs` DAGs resolve
|
||||
through `actions::resolve_approval_dag` when the DAG settles terminal
|
||||
(a spawn additionally runs the post-spawn forge bookkeeping there).
|
||||
The DAG carries the originating `approval_id`, surfaced on the node that
|
||||
owns it — for a deploy that's the `DeployWindow` root, so the dashboard
|
||||
renders one approval card, not four. **Every** queued kind resolves
|
||||
through `actions::resolve_approval_dag` when its DAG settles terminal:
|
||||
the deploy's phases are ordinary queue nodes, so the DAG's own terminal
|
||||
state is the authoritative outcome. That hook fires the matching
|
||||
`HelperEvent::*` via `finish_approval`, derives the `Rebuilt` event's
|
||||
terminal tag (verifying the tag actually resolves in the applied repo —
|
||||
a pre-merge rejection plants none), posts the failing build log back to
|
||||
the config PR, and for a spawn runs the post-spawn forge bookkeeping.
|
||||
|
||||
Two visible consequences:
|
||||
|
||||
|
|
|
|||
|
|
@ -37,12 +37,15 @@ Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
|||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `PostSwap` node |
|
||||
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `ApprovalDeploy` | the opaque apply-commit / merge-config-PR pipeline (see _Approvals_ below) |
|
||||
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
|
||||
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
| Node | Behavior |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
|
||||
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail — rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good, then mirrors the config repo to the forge. Infallible by construction |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
|
|
@ -64,7 +67,8 @@ container build:
|
|||
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
|
||||
resource declared by every node kind that mutates the meta repo — `MetaSync`,
|
||||
`MetaLock`, `WritePermFile`, `Provision`'s agent registration, and
|
||||
`ApprovalDeploy` for its whole span (`NodeKind::needs_meta_window`). Two meta
|
||||
`DeployWindow` — the deploy subtree's root, which holds it across every
|
||||
phase below it (`NodeKind::needs_meta_window`). Two meta
|
||||
mutations can therefore never interleave, so no commit lands inside another
|
||||
node's staged window. It is a queue resource rather than a runtime mutex
|
||||
because a resource is held by a subtree root across its whole subtree, which
|
||||
|
|
@ -180,7 +184,7 @@ resources are free. Resources:
|
|||
per-node; a DAG can span agents) and globally exclusive per agent across
|
||||
all DAGs: acquired at a container-affecting node (`SetWanted`,
|
||||
`StopForUpdate`, `Swap`, `Signal`, `Drain`, `Reconcile`, `WriteDropin`,
|
||||
`Create`, `ApprovalDeploy`), held by the owning DAG until it's terminal,
|
||||
`Create`, `DeployWindow`), held by the owning DAG until it's terminal,
|
||||
so two DAGs never interleave container ops on the same agent. A DAG
|
||||
touching several agents holds one lease per agent. (`SetWanted` is a store
|
||||
write, not a container op, but takes the lease anyway so a power-op DAG's
|
||||
|
|
@ -213,15 +217,30 @@ per template.
|
|||
|
||||
### Approvals
|
||||
|
||||
`MergeConfigPr` approvals ride as single-node
|
||||
`ApprovalDeploy` DAGs: the two-phase `prepare_deploy` / `finalize_deploy` /
|
||||
`abort_deploy` meta orchestration stays inside `actions.rs` in v1
|
||||
(deliberately not modeled as scheduler nodes) and resolves the approval
|
||||
itself. `Spawn` and `UpdateMetaInputs` approvals map onto the ordinary
|
||||
`spawn` / `meta-update` shapes; the scheduler fires
|
||||
`actions::resolve_approval_dag` exactly once when such a DAG settles
|
||||
terminal (including cancelled-while-queued, which fails the approval instead
|
||||
of dangling it).
|
||||
`MergeConfigPr` approvals ride as a four-node deploy subtree:
|
||||
|
||||
```
|
||||
DeployWindow (root — build slot + lease + meta window, no work of its own)
|
||||
├── MergeVerify drift gate, fetch, verify_commit
|
||||
├── DeployApply AfterOk(verify) park rollback ref, ff-merge, deploy
|
||||
└── DeployTail AfterAny(apply) compensate, mirror to forge
|
||||
```
|
||||
|
||||
The root's resources are held across the whole subtree, so the two-phase
|
||||
`prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock`
|
||||
protected even though the phases are separate nodes. Splitting them buys
|
||||
three things a single opaque node couldn't have: per-phase visibility on the
|
||||
dashboard, a `MergeVerify` failure that provably mutated nothing, and a
|
||||
compensation step that survives a hive-c0re restart — the pre-merge
|
||||
`applied/main` is parked in `refs/hyperhive/rollback/<approval-id>`, not in a
|
||||
local variable, so `DeployTail` can still undo a half-finished deploy after a
|
||||
crash.
|
||||
|
||||
`Spawn` and `UpdateMetaInputs` approvals map onto the ordinary `spawn` /
|
||||
`meta-update` shapes. The scheduler fires `actions::resolve_approval_dag`
|
||||
exactly once when **any** approval-carrying DAG settles terminal — deploys
|
||||
included, since their outcome is now the DAG's own state (including
|
||||
cancelled-while-queued, which fails the approval instead of dangling it).
|
||||
|
||||
### Wire shape
|
||||
|
||||
|
|
@ -338,8 +357,9 @@ Sequence for a rebuild DAG (each step is its own queue node):
|
|||
|
||||
The approval apply-commit pipeline still drives `lifecycle::rebuild_no_meta`
|
||||
(the fused stop/update/start path with an inline start) inside its
|
||||
`ApprovalDeploy` node, because it verifies the agent comes back up before
|
||||
finalizing the deploy tag.
|
||||
`DeployApply` node, because it verifies the agent comes back up before
|
||||
finalizing the deploy tag. Breaking that fused path apart into the
|
||||
`Prebuild → Swap → Reconcile` chain above is increment 2b of #2664, not 2a.
|
||||
|
||||
### Cold-start fallback
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
//! PR-based config-flow merge primitives — the forge-side mechanics
|
||||
//! hive-c0re's approve-handler (`run_merge_config_pr`) orchestrates to
|
||||
//! hive-c0re's deploy apply node (`actions::run_deploy_apply`) orchestrates to
|
||||
//! land an operator-approved config PR. Part of the operator trust
|
||||
//! boundary; moved verbatim from the `forge` module root.
|
||||
|
||||
|
|
@ -15,7 +15,7 @@ use super::{CONFIG_ORG, api, core_token, forge_git_url};
|
|||
// The dashboard-approve-driven flow has hive-c0re verify an operator-approved
|
||||
// config PR, then land it: fast-forward-merge the verified sha into the
|
||||
// protected default branch via the forge merge API (= the merge). These fns are
|
||||
// the forge-side mechanics the c0re approve-handler (`run_merge_config_pr`)
|
||||
// the forge-side mechanics the c0re deploy apply node (`run_deploy_apply`)
|
||||
// orchestrates; the orchestration fetches the verified sha into the agent's
|
||||
// applied repo (for the eval-verify) before calling `merge_config_pr_ff`. The
|
||||
// core token is sourced internally (`core_token`), never passed in. `repo` is
|
||||
|
|
|
|||
|
|
@ -907,7 +907,7 @@ async fn apply_operator_branch_protection(repo: &str, token: &str) -> Result<()>
|
|||
/// - **`main` is never directly pushable** — no push is enabled on the
|
||||
/// protected branch, so neither the agent (a write collaborator) nor
|
||||
/// hive-c0re can `git push` it. It only advances via the config-PR merge
|
||||
/// handler (`run_merge_config_pr`), which fast-forward-*merges* the reviewed
|
||||
/// node (`actions::run_deploy_apply`), which fast-forward-*merges* the reviewed
|
||||
/// head through the forge merge API (`Do=fast-forward-only`,
|
||||
/// `head_commit_id` pinned to the reviewed sha).
|
||||
/// - **merge is whitelisted to `core`** — only hive-c0re can merge a config PR;
|
||||
|
|
|
|||
|
|
@ -71,7 +71,7 @@ async fn ensure_applied_remote(proposed_dir: &Path, name: &str) -> Result<()> {
|
|||
/// proposed's initial commit in via `git fetch`, tag it `deployed/0`.
|
||||
/// This is the *only* time hive-c0re reads from `proposed` for an
|
||||
/// agent — subsequent config changes are fetched from the reviewed
|
||||
/// forge PR head at merge time (see `actions::run_merge_config_pr`).
|
||||
/// forge PR head at merge time (see `actions::run_deploy_apply`).
|
||||
///
|
||||
/// `proposed_dir` is `None` on rebuild paths where the repo already
|
||||
/// exists — we just verify it's the right shape and bail otherwise.
|
||||
|
|
|
|||
|
|
@ -89,7 +89,7 @@ pub(super) fn handle_request_update_meta_inputs(
|
|||
/// dashboard shows the pending card immediately.
|
||||
///
|
||||
/// The PR head sha is stored as `fetched_sha` on the approval row — the
|
||||
/// "reviewed sha" the approve handler (`run_merge_config_pr`) drift-gates
|
||||
/// "reviewed sha" the deploy's `MergeVerify` node drift-gates
|
||||
/// against before doing anything irreversible. This does NOT fetch the commit
|
||||
/// into the applied repo at submission time (that happens inside the approve
|
||||
/// handler, step 2, after the drift check). No flake pre-flight either —
|
||||
|
|
|
|||
Loading…
Reference in a new issue