job_queue: split ApprovalDeploy into a four-node deploy subtree
A config-PR deploy was one opaque node that fetched, verified, merged, built and compensated. That shape made three things impossible: the nix-heavy phases could not take the meta window without the cheap ones holding it too, a crash mid-build left no node to run the rollback, and the dashboard could only ever show "deploying" for the whole thing. Replace it with a `DeployWindow` group root over `MergeVerify -> DeployApply` (AfterOk) plus a `DeployTail` hanging off the apply with AfterAny, so the tail runs whether the apply succeeded, failed, or was cancel-cascaded by a failing verify.
This commit is contained in:
atlas
mara
parent
001ea38ea4
commit
bdf15168db
2 changed files with 99 additions and 19 deletions
|
|
@ -186,11 +186,64 @@ pub enum NodeKind {
|
|||
/// (commit fused under `META_LOCK`). The payload rides this node — the only
|
||||
/// consumer — rather than the generic DAG container.
|
||||
WritePermFile { agent: String, payload: PermPayload },
|
||||
/// Opaque approval deploy pipeline (`MergeConfigPr`): the two-phase
|
||||
/// prepare/finalize/abort meta deploy stays inside `actions.rs` in v1 —
|
||||
/// deliberately not
|
||||
/// modeled as scheduler nodes (see the design doc §9).
|
||||
ApprovalDeploy { agent: String },
|
||||
/// Group root of the approval-deploy (`MergeConfigPr`) subtree, and the
|
||||
/// node that **owns the deploy window**. It performs no work of its own —
|
||||
/// it exists so the resources it declares (the global
|
||||
/// [`Resource::MetaWindow`](super::resource::Resource::MetaWindow), the
|
||||
/// agent lease, a build slot) are held continuously across every child
|
||||
/// phase, which a per-node acquisition could not guarantee.
|
||||
///
|
||||
/// All three resources are declared *here*, on one node, on purpose. The
|
||||
/// queue acquires a node's resources atomically (all-or-nothing), so a
|
||||
/// single multi-resource root can never hold one and block on another —
|
||||
/// whereas letting a child take the build slot while its parent held the
|
||||
/// meta window would introduce exactly that pattern, and with it a
|
||||
/// lock-ordering argument that has to be re-verified on every future edit.
|
||||
/// Cheap, too: the window has to span the container build regardless (see
|
||||
/// [`NodeKind::DeployApply`]), so nothing is over-serialised by hoisting
|
||||
/// the slot and the lease up alongside it.
|
||||
DeployWindow { agent: String },
|
||||
/// Deploy phase 1 — **verify only, mutates nothing.** Drift-gate the
|
||||
/// approval's PR head, fetch it into the applied repo, and eval-verify the
|
||||
/// merge head. Any failure here aborts the deploy with the forge state
|
||||
/// untouched, so it is safely retryable and cancel-safe: nothing downstream
|
||||
/// has happened yet.
|
||||
MergeVerify { agent: String },
|
||||
/// Deploy phase 2 — everything from the irreversible fast-forward onward:
|
||||
/// ff-merge the reviewed head to `main` via the forge API, two-phase meta
|
||||
/// `prepare_deploy`, the container rebuild, then on success the
|
||||
/// `deployed/<id>` tag + `finalize_deploy`.
|
||||
///
|
||||
/// Still one node in this increment: splitting the tail into
|
||||
/// `FfMain`/`PrepareDeploy`/rebuild/`FinalizeDeploy` children is the next
|
||||
/// one. What *is* already split out is the compensation path — see
|
||||
/// [`NodeKind::DeployTail`].
|
||||
DeployApply { agent: String },
|
||||
/// Deploy compensation **and bookkeeping** tail — `AfterAny`
|
||||
/// [`NodeKind::DeployApply`], so it runs on success, failure, and cancel
|
||||
/// alike, in the same spirit as the rebuild template's tail `Reconcile`
|
||||
/// ("always runs, decides internally"). It:
|
||||
/// 1. compensates a merge that landed but was never finalized — roll `main`
|
||||
/// back, reset the tree, `meta::abort_deploy`, plant `failed/<id>`;
|
||||
/// 2. mirrors whichever deploy tag got planted to the forge config repo
|
||||
/// (`forge::push_config`), always, best-effort;
|
||||
/// 3. posts the failing build log back onto the config PR when the deploy
|
||||
/// failed, so the manager sees the rejection without leaving the forge.
|
||||
///
|
||||
/// Steps 2 and 3 are why this is `DeployTail` and not `AbortDeploy`: it has
|
||||
/// work to do on the success path too, and a node name that claims
|
||||
/// otherwise would be a lie on the dashboard.
|
||||
///
|
||||
/// For (1) it needs no knowledge of how far the deploy got, because that state is
|
||||
/// parked in the applied repo rather than passed between nodes:
|
||||
/// `DeployApply` writes the pre-merge `main` sha to
|
||||
/// `refs/hyperhive/rollback/<approval-id>` before the fast-forward and
|
||||
/// deletes it once the deploy has been finalized. So the ref existing *is*
|
||||
/// the "a merge landed and was not finalized" signal, and its absence makes
|
||||
/// this node a no-op. Parking it in git rather than in a node payload also
|
||||
/// means it survives a `hive-c0re` restart mid-deploy, which an in-memory
|
||||
/// queue does not.
|
||||
DeployTail { agent: String },
|
||||
/// Write the agent's durable power intent (`wanted = Up` when `up`, else
|
||||
/// `Offline`) as a first-class DAG node, at the head of a power-op
|
||||
/// template so the downstream `Reconcile` reads it. Replaces the old
|
||||
|
|
@ -243,7 +296,10 @@ impl NodeKind {
|
|||
NodeKind::Drain { .. } => "drain",
|
||||
NodeKind::WriteDropin { .. } => "write_dropin",
|
||||
NodeKind::WritePermFile { .. } => "write_perm_file",
|
||||
NodeKind::ApprovalDeploy { .. } => "approval_deploy",
|
||||
NodeKind::DeployWindow { .. } => "deploy_window",
|
||||
NodeKind::MergeVerify { .. } => "merge_verify",
|
||||
NodeKind::DeployApply { .. } => "deploy_apply",
|
||||
NodeKind::DeployTail { .. } => "deploy_tail",
|
||||
NodeKind::SetWanted { .. } => "set_wanted",
|
||||
NodeKind::Dag { .. } => "dag",
|
||||
}
|
||||
|
|
@ -269,7 +325,10 @@ impl NodeKind {
|
|||
| NodeKind::Drain { agent }
|
||||
| NodeKind::WriteDropin { agent }
|
||||
| NodeKind::WritePermFile { agent, .. }
|
||||
| NodeKind::ApprovalDeploy { agent }
|
||||
| NodeKind::DeployWindow { agent }
|
||||
| NodeKind::MergeVerify { agent }
|
||||
| NodeKind::DeployApply { agent }
|
||||
| NodeKind::DeployTail { agent }
|
||||
| NodeKind::SetWanted { agent, .. } => agent,
|
||||
NodeKind::MetaLock { .. } | NodeKind::Dag { .. } => "",
|
||||
}
|
||||
|
|
@ -284,7 +343,7 @@ impl NodeKind {
|
|||
| NodeKind::Swap { .. }
|
||||
| NodeKind::Create { .. }
|
||||
| NodeKind::MetaLock { .. }
|
||||
| NodeKind::ApprovalDeploy { .. }
|
||||
| NodeKind::DeployWindow { .. }
|
||||
)
|
||||
}
|
||||
|
||||
|
|
@ -306,7 +365,7 @@ impl NodeKind {
|
|||
| NodeKind::Signal { .. }
|
||||
| NodeKind::Drain { .. }
|
||||
| NodeKind::WriteDropin { .. }
|
||||
| NodeKind::ApprovalDeploy { .. }
|
||||
| NodeKind::DeployWindow { .. }
|
||||
| NodeKind::SetWanted { .. }
|
||||
)
|
||||
}
|
||||
|
|
@ -337,7 +396,7 @@ impl NodeKind {
|
|||
| NodeKind::Provision { .. }
|
||||
| NodeKind::MetaLock { .. }
|
||||
| NodeKind::WritePermFile { .. }
|
||||
| NodeKind::ApprovalDeploy { .. }
|
||||
| NodeKind::DeployWindow { .. }
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -140,10 +140,24 @@ pub fn rebuild(agent: &str, source: Source, reason: String, relock: bool) -> Dag
|
|||
}
|
||||
}
|
||||
|
||||
/// Approval-driven deploy (`MergeConfigPr`): the whole two-phase pipeline
|
||||
/// stays one opaque node in v1 (design doc §9) — wire-visible as a `rebuild`
|
||||
/// card like today.
|
||||
/// Approval-driven deploy (`MergeConfigPr`) as a phase subtree rather than the
|
||||
/// single opaque node it used to be. Structure:
|
||||
/// - `DeployWindow` (0, **root**): the resource holder — global meta window,
|
||||
/// agent lease, build slot — held across every child below. No work of its
|
||||
/// own; it reaches `Finishing` immediately and the children run inside it.
|
||||
/// - `MergeVerify` (1, child): drift-gate + fetch + eval-verify. Mutates
|
||||
/// nothing, so a failure here cancel-cascades its siblings with the forge and
|
||||
/// the applied repo exactly as they were.
|
||||
/// - `DeployApply` (2, child, `AfterOk` `MergeVerify`): the irreversible half —
|
||||
/// ff-merge, `prepare_deploy`, rebuild, `finalize_deploy`.
|
||||
/// - `DeployTail` (3, child, `AfterAny` `DeployApply`): the compensation +
|
||||
/// bookkeeping tail — rollback when a merge landed unfinalized, forge tag
|
||||
/// mirror, PR failure comment (see [`NodeKind::DeployTail`]).
|
||||
///
|
||||
/// The window still spans the container build, as it must: `prepare_deploy`
|
||||
/// leaves `flake.lock` staged-uncommitted for the build's whole duration.
|
||||
pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec {
|
||||
let a = || agent.to_owned();
|
||||
DagSpec {
|
||||
template: Template::Rebuild,
|
||||
source: Source::Approval,
|
||||
|
|
@ -151,12 +165,19 @@ pub fn approval_deploy(agent: &str, approval_id: i64, reason: String) -> DagSpec
|
|||
approval_id: Some(approval_id),
|
||||
inputs: Vec::new(),
|
||||
transient: Some(TransientKind::Rebuilding),
|
||||
nodes: vec![node(
|
||||
NodeKind::ApprovalDeploy {
|
||||
agent: agent.to_owned(),
|
||||
},
|
||||
Vec::new(),
|
||||
)],
|
||||
nodes: vec![
|
||||
node(NodeKind::DeployWindow { agent: a() }, Vec::new()),
|
||||
child(0, NodeKind::MergeVerify { agent: a() }, Vec::new()),
|
||||
child(0, NodeKind::DeployApply { agent: a() }, after_ok(1)),
|
||||
child(
|
||||
0,
|
||||
NodeKind::DeployTail { agent: a() },
|
||||
vec![Dep {
|
||||
on: 2,
|
||||
when: DepWhen::AfterAny,
|
||||
}],
|
||||
),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue