vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: grow the rebuild subgraph from DeployApply (#2664)

Browse source 
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.

Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.

The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.

`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.

The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.

Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
This commit is contained in:
atlas 2026-07-25 23:24:48 +02:00 • committed by mara
commit 3429a8c5a6
10 changed files with 388 additions and 257 deletions
Download patch file Download diff file Expand all files Collapse all files

46
hive-c0re/src/job_queue/model.rs
View file

@ -209,16 +209,36 @@ pub enum NodeKind {
/// untouched, so it is safely retryable and cancel-safe: nothing downstream
/// has happened yet.
MergeVerify { agent: String },
/// Deploy phase 2 — everything from the irreversible fast-forward onward:
/// ff-merge the reviewed head to `main` via the forge API, two-phase meta
/// `prepare_deploy`, the container rebuild, then on success the
/// `deployed/<id>` tag + `finalize_deploy`.
/// Deploy phase 2 — the irreversible fast-forward plus the *opening* half of
/// the two-phase meta deploy: park the rollback ref, ff-merge the reviewed
/// head to `main` via the forge API, ff `applied/main`, and
/// `meta::prepare_deploy` (which stages `flake.lock` uncommitted).
///
/// Still one node in this increment: splitting the tail into
/// `FfMain`/`PrepareDeploy`/rebuild/`FinalizeDeploy` children is the next
/// one. What *is* already split out is the compensation path — see
/// It does **not** run the container rebuild itself. It grows the ordinary
/// rebuild subgraph into this DAG as its own children
/// ([`super::templates::deploy_rebuild_nodes`], `relock = false` — the lock
/// is already staged), so the multi-minute build renders as the same real
/// nodes every other rebuild does instead of one opaque box. Closing the
/// staged-lock window is likewise its own node
/// ([`NodeKind::FinalizeDeploy`]), and the compensation path is
/// [`NodeKind::DeployTail`].
DeployApply { agent: String },
/// Deploy phase 3 — close the two-phase meta deploy once the rebuild
/// subgraph under [`NodeKind::DeployApply`] has come up clean: drop the
/// rollback ref, plant the `deployed/<id>` tag, commit the staged
/// `flake.lock` (`meta::finalize_deploy`).
///
/// Its two git steps are **fatal**, deliberately. They are the writes that
/// tell [`NodeKind::DeployTail`] a deploy confirmed good, so a node that
/// merely warned on them could report success while leaving the tail
/// looking at the git state of a failure — and the tail would then roll a
/// *good* deploy back. Failing loudly keeps the node's outcome and the
/// repo's state saying the same thing.
///
/// The trailing `meta::finalize_deploy` stays warn-only: by then the
/// container already runs the new config, and an uncommitted staged lock is
/// something the operator can commit by hand.
FinalizeDeploy { agent: String },
/// Deploy compensation **and bookkeeping** tail — `AfterAny`
/// [`NodeKind::DeployApply`], so it runs on success, failure, and cancel
/// alike, in the same spirit as the rebuild template's tail `Reconcile`
@ -238,7 +258,7 @@ pub enum NodeKind {
/// parked in the applied repo rather than passed between nodes:
/// `DeployApply` writes the pre-merge `main` sha to
/// `refs/hyperhive/rollback/<approval-id>` before the fast-forward and
/// deletes it once the deploy has been finalized. So the ref existing *is*
/// [`NodeKind::FinalizeDeploy`] deletes it. So the ref existing *is*
/// the "a merge landed and was not finalized" signal, and its absence makes
/// this node a no-op. Parking it in git rather than in a node payload also
/// means it survives a `hive-c0re` restart mid-deploy, which an in-memory
@ -299,6 +319,7 @@ impl NodeKind {
NodeKind::DeployWindow { .. } => "deploy_window",
NodeKind::MergeVerify { .. } => "merge_verify",
NodeKind::DeployApply { .. } => "deploy_apply",
NodeKind::FinalizeDeploy { .. } => "finalize_deploy",
NodeKind::DeployTail { .. } => "deploy_tail",
NodeKind::SetWanted { .. } => "set_wanted",
NodeKind::Dag { .. } => "dag",
@ -328,6 +349,7 @@ impl NodeKind {
| NodeKind::DeployWindow { agent }
| NodeKind::MergeVerify { agent }
| NodeKind::DeployApply { agent }
| NodeKind::FinalizeDeploy { agent }
| NodeKind::DeployTail { agent }
| NodeKind::SetWanted { agent, .. } => agent,
NodeKind::MetaLock { .. } | NodeKind::Dag { .. } => "",
@ -389,6 +411,13 @@ impl NodeKind {
/// That is why the meta preamble is its own [`NodeKind::MetaSync`] node,
/// and why that node is a sibling rather than `Prebuild`'s parent (a
/// resource held by a parent covers its whole subtree).
///
/// Two of these kinds run *inside* a [`NodeKind::DeployWindow`]'s subtree
/// (the appended rebuild's `MetaSync`, and [`NodeKind::FinalizeDeploy`]).
/// They still declare the window: a descendant re-enters an ancestor's hold
/// through the crate's recursive lock, exactly as `Start` / `Stop` re-enter
/// a `Reconcile`'s agent lease. Declaring it is what keeps the requirement
/// true of the *node* rather than of one particular DAG shape.
pub fn needs_meta_window(&self) -> bool {
matches!(
self,
@ -397,6 +426,7 @@ impl NodeKind {
| NodeKind::MetaLock { .. }
| NodeKind::WritePermFile { .. }
| NodeKind::DeployWindow { .. }
| NodeKind::FinalizeDeploy { .. }
)
}
}