vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

config PRs: remove the hive's config-PR webhook, poll and core merge

Browse source 
An operator's merge on the forge deploys a config PR through
swarm-controller's DeployRequest{rev}. The hive-side path that queued a
MergeConfigPr approval and merged the PR as `core` goes:

- the `/webhook/config-pr` receiver, its HMAC secret, the WebhookRegister
  boot node and the org-hook registration; the hive vhost's `/webhook/`
  location
- the 5-minute config-PR poll
- ApprovalKind::MergeConfigPr, its dashboard card, and the deploy DAG it
  drove (DeployWindow, MergeVerify, DeployApply, FinalizeDeploy,
  DeployTail), with verify_commit, the two-phase meta deploy, rollback
  refs, the PR-failure comment and forge/pr_merge.rs
- `fetched_sha`, `sha_short`/`pr_number` on approval events, and
  `sha`/`tag` on HelperEvent::ApprovalResolved: only the merge path set
  them

`config_repo`, `merged_pr_for_commit` and `post_pr_comment` move to
forge/pr_comment.rs for the merged-rev deploy's refusal comment.
Approvals v5 drops stored `merge_config_pr` rows; a test reopens a v4
database holding them.

Closes #4850
This commit is contained in:
atlas 2026-10-02 22:32:17 +02:00
commit efbfec6d01
39 changed files with 286 additions and 3199 deletions
Download patch file Download diff file Expand all files Collapse all files

106
hive-c0re/src/job_queue/exec.rs
View file

@ -121,24 +121,13 @@ pub(super) async fn run_node(
// takes it directly instead of re-matching the kind behind a `bail!`
// that could never fire.
NodeKind::WritePermFile { payload, .. } => run_write_perm_file(coord, agent, payload).await,
NodeKind::MergeVerify { approval_id, .. } => {
run_merge_verify(coord, *approval_id, id).await
}
NodeKind::DeployApply { approval_id, .. } => {
run_deploy_apply(coord, *approval_id, id).await.map(|()| {
super::templates::deploy_rebuild_nodes(&builder, agent, *approval_id);
})
}
NodeKind::FinalizeDeploy { approval_id, .. } => {
run_finalize_deploy(coord, *approval_id).await
}
NodeKind::DeployTail { approval_id, .. } => {
run_deploy_tail(coord, coord.job_queue.root_of(id), agent, *approval_id).await
}
NodeKind::ResolveApproval {
approval_id,
outcome,
} => run_resolve_approval(coord, coord.job_queue.root_of(id), *approval_id, *outcome).await,
} => {
run_resolve_approval(coord, coord.job_queue.root_of(id), *approval_id, *outcome);
Ok(())
}
NodeKind::EmitRebuilt { ok, .. } => {
run_emit_rebuilt(coord, agent, coord.job_queue.root_of(id), *ok).await;
Ok(())
@ -147,10 +136,9 @@ pub(super) async fn run_node(
// Braces carry no work of their own; completing one lets it reach
// `Finishing` so the nodes under it start. What they declare stays held
// until their whole subtree settles.
NodeKind::DeployWindow { .. } | NodeKind::AgentWindow { .. } => Ok(()),
NodeKind::AgentWindow { .. } => Ok(()),
NodeKind::ForgeSweep => run_forge_sweep().await,
NodeKind::MatrixSweep => run_matrix_sweep().await,
NodeKind::WebhookRegister => run_webhook_register().await,
NodeKind::KnowledgePull => run_knowledge_pull(coord).await,
NodeKind::WantedPull => run_wanted_pull(coord).await,
};
@ -218,34 +206,6 @@ fn matrix_sweep_banner(ctx: sweep_health::SweepFailure) -> String {
)
}
/// Boot-time Forgejo webhook management as a DAG node — see
/// [`NodeKind::WebhookRegister`]. Mirrors the guard chain the
/// `tokio::spawn` block it replaced used: no-op (not an error) when the
/// HMAC secret, core token, or hive domain aren't available yet.
///
/// Registers the config-PR hook only. The knowledge hook is the swarm
/// controller's, which addresses an event to each hive over the queue.
async fn run_webhook_register() -> Result<()> {
let Ok(webhook_secret) = crate::webhook_secret::load_or_generate() else {
tracing::debug!("webhook secret unavailable; skipping hook registration");
return Ok(());
};
let Some(token) = crate::forge::core_token() else {
return Ok(());
};
let domain = std::env::var("HYPERHIVE_HIVE_DOMAIN")
.ok()
.filter(|v| !v.is_empty());
let Some(domain) = domain else {
tracing::debug!("HYPERHIVE_HIVE_DOMAIN unset; skipping webhook registration");
return Ok(());
};
if let Err(e) = crate::forge::ensure_config_pr_webhook(&token, &domain, &webhook_secret).await {
tracing::warn!(error = ?e, "forge: ensure_config_pr_webhook failed");
}
Ok(())
}
/// `/knowledge` pull as a DAG node — see [`NodeKind::KnowledgePull`]. Every
/// pass reports here, whoever submitted it: as the node's own error on the
/// dashboard, and into the debounced banner.
@ -313,17 +273,16 @@ async fn run_wanted_pull(coord: &Arc<Coordinator>) -> Result<()> {
/// answer. Best-effort — a resolution failure is logged inside
/// [`crate::actions::resolve_approval_dag`], never surfaced as a node failure,
/// since the work already happened and failing the tail would only misreport it.
async fn run_resolve_approval(
fn run_resolve_approval(
coord: &Arc<Coordinator>,
dag_id: Option<u64>,
approval_id: i64,
outcome: TerminalState,
) -> Result<()> {
) {
let reason = (outcome == TerminalState::Failed)
.then(|| dag_id.and_then(|dag| coord.job_queue.first_error(dag)))
.flatten();
crate::actions::resolve_approval_dag(coord, approval_id, outcome, reason.as_deref()).await;
Ok(())
crate::actions::resolve_approval_dag(coord, approval_id, outcome, reason.as_deref());
}
/// Rerender the meta flake from whatever containers still exist on disk.
@ -629,9 +588,8 @@ async fn run_meta_lock(
// against whatever `applied/<name>` already happened to be locked to —
// normally current, but silently stale forever if a past deploy failed
// and nothing since retried it. Relocks against each agent's *declared*
// input (the forge URL, not the local `applied/<name>` mirror
// `prepare_deploy` uses for a reviewed deploy) — this path has no PR to
// review, so there's nothing to gate. One combined call, not per-agent:
// input (the forge URL, not the local `applied/<name>` mirror a rebuild
// relocks against) — this path has nothing to gate. One combined call, not per-agent:
// simpler, at the cost of one unreachable/broken agent repo failing the
// whole cascade relock rather than just that agent.
if !cascade.is_empty() {
@ -890,50 +848,6 @@ async fn run_write_perm_file(
Ok(())
}
/// Deploy phase 1 — drift gate, fetch, eval-verify. Mutates nothing, so a
/// failure here cancel-cascades the rest of the subtree with the forge and the
/// applied repo exactly as they were.
async fn run_merge_verify(coord: &Arc<Coordinator>, approval_id: i64, id: NodeId) -> Result<()> {
crate::actions::run_deploy_merge_verify(coord, approval_id, Some(id.get())).await
}
/// Deploy phase 2 — the irreversible half: ff-merge, then phase 1 of the
/// two-phase meta deploy.
///
/// On success it grows the ordinary rebuild subgraph (plus its closing
/// `FinalizeDeploy`) into this DAG rooted on *this* node — which is what puts
/// the appended nodes inside the `DeployWindow`'s subtree, so the `MetaWindow`
/// their `MetaSync` declares is re-entered rather than deadlocked against the
/// ancestor already holding it. On failure nothing is appended and the tail
/// compensates, exactly as before.
async fn run_deploy_apply(coord: &Arc<Coordinator>, approval_id: i64, id: NodeId) -> Result<()> {
crate::actions::run_deploy_apply(coord, approval_id, Some(id.get())).await
}
/// Deploy phase 3 — close the staged-lock window once the appended rebuild has
/// come up clean: drop the rollback ref, plant the `deployed/<id>` tag, commit
/// the staged lock.
async fn run_finalize_deploy(coord: &Arc<Coordinator>, approval_id: i64) -> Result<()> {
crate::actions::run_finalize_deploy(coord, approval_id).await
}
/// Deploy compensation + bookkeeping tail. `AfterAny` the apply node, so it
/// runs on every outcome; it is deliberately infallible (see
/// [`crate::actions::run_deploy_tail`]) — a failing tail must not flip an
/// otherwise-successful deploy's DAG state.
///
/// Takes the agent from the node payload so the tail can still compensate when
/// the approval row is gone (deny race, purge).
async fn run_deploy_tail(
coord: &Arc<Coordinator>,
dag_id: Option<u64>,
agent: &str,
approval_id: i64,
) -> Result<()> {
crate::actions::run_deploy_tail(coord, dag_id, agent, approval_id).await;
Ok(())
}
/// Compute which agents a `nix flake update <inputs>` on the meta
/// flake affects — the fan-out set for `MetaUpdate` DAGs. Empty
/// `inputs` or any input under `hyperhive` → every container;

36
hive-c0re/src/job_queue/model.rs
View file

@ -102,28 +102,11 @@ pub enum NodeKind {
WriteDropin { agent: String },
/// Commit `tool-groups.json` / `capabilities.json` per its `payload`.
WritePermFile { agent: String, payload: PermPayload },
/// Group root of the approval-deploy (`MergeConfigPr`) subtree — the
/// **brace** that owns the deploy window. See _Braces_ and _Approvals_.
DeployWindow { agent: String, approval_id: i64 },
/// Group root of a rebuild subtree — the **brace**, scoped to one
/// agent. Why braces exist and what they cost: _Braces_.
AgentWindow { agent: String },
/// Deploy phase 1 — **verify only, mutates nothing.**
MergeVerify { agent: String, approval_id: i64 },
/// Deploy phase 2 — the irreversible fast-forward plus the *opening*
/// half of the two-phase meta deploy. Does **not** build the container
/// itself; grows the ordinary rebuild subgraph in as its own children
/// (see the rebuild-path section's approval-deploy paragraph).
DeployApply { agent: String, approval_id: i64 },
/// Deploy phase 3 — closes the two-phase meta deploy once
/// [`NodeKind::DeployApply`]'s rebuild subgraph comes up clean.
FinalizeDeploy { agent: String, approval_id: i64 },
/// Deploy compensation **and bookkeeping** tail. See the `DeployTail`
/// node-inventory row for its three responsibilities and why it isn't
/// named `AbortDeploy`.
DeployTail { agent: String, approval_id: i64 },
/// Tail node of an approval-carrying DAG (opaque deploy /
/// config-PR merge): resolve the approval row from how the work ended.
/// Tail node of an approval-carrying DAG (a meta-inputs update): resolve
/// the approval row from how the work ended.
ResolveApproval {
approval_id: i64,
/// Which outcome this node reports. A template emits **one per
@ -147,8 +130,6 @@ pub enum NodeKind {
/// Matrix user/space sweep (`matrix::ensure_all`): at boot and every 30
/// minutes. Agentless.
MatrixSweep,
/// One-shot boot-time Forgejo webhook registration. Agentless.
WebhookRegister,
/// `/knowledge` pull (`knowledge::pull`): at boot, on the swarm
/// knowledge-changed event, and hourly. Agentless.
KnowledgePull,
@ -174,9 +155,6 @@ impl hive_jobq_wire::WireNode for NodeKind {
if !agent.is_empty() {
data.insert("agent".to_owned(), agent.into());
}
if let NodeKind::DeployWindow { approval_id, .. } = self {
data.insert("approval_id".to_owned(), (*approval_id).into());
}
if let NodeKind::MetaLock { inputs, .. } = self
&& !inputs.is_empty()
{
@ -221,19 +199,13 @@ impl NodeKind {
| NodeKind::PauseDrain { agent }
| NodeKind::WriteDropin { agent }
| NodeKind::WritePermFile { agent, .. }
| NodeKind::DeployWindow { agent, .. }
| NodeKind::AgentWindow { agent }
| NodeKind::MergeVerify { agent, .. }
| NodeKind::DeployApply { agent, .. }
| NodeKind::FinalizeDeploy { agent, .. }
| NodeKind::DeployTail { agent, .. }
| NodeKind::EmitRebuilt { agent, .. }
| NodeKind::SetWanted { agent, .. } => agent,
NodeKind::MetaLock { .. }
| NodeKind::ResolveApproval { .. }
| NodeKind::ForgeSweep
| NodeKind::MatrixSweep
| NodeKind::WebhookRegister
| NodeKind::KnowledgePull
| NodeKind::WantedPull => "",
}
@ -280,8 +252,8 @@ impl NodeKind {
// bug, and a `true` here would suppress the alert that says so.
// - `Reconcile` is a planner; it fans out `Start` / `Stop`, which carry
// their own answer.
// - `DeployWindow` brackets a deploy without itself stopping anything.
// - `AgentWindow` likewise, though it *parents* nodes that answer
// - `AgentWindow` brackets a rebuild without itself stopping anything,
// though it *parents* nodes that answer
// `true`. This is per-node, not per-subtree, and each of those
// children reports for itself — so `true` here would only widen
// suppression over the build and tail, where a vanished container is

103
hive-c0re/src/job_queue/templates.rs
View file

@ -6,7 +6,7 @@
//! that already held the resource could get away with declaring nothing.
//!
//! **The one sanctioned exception is a brace** — a pure-resource-holder root
//! ([`NodeKind::AgentWindow`], [`NodeKind::DeployWindow`]) declaring for a
//! ([`NodeKind::AgentWindow`]) declaring for a
//! coordinated subtree whose members then declare nothing. See
//! `docs/scheduler/coordinator.md`, _Braces_ — which also carries the per-operation DAG
//! shapes, so they are not restated here.
@ -314,42 +314,6 @@ pub(crate) fn graceful_rebuild_nodes<'a>(
rebuild_subtree(builder, agent, relock, true, after)
}
/// The rebuild subgraph a [`NodeKind::DeployApply`] grows into its own DAG once
/// the merge has landed and `prepare_deploy` has staged the lock, plus the
/// [`NodeKind::FinalizeDeploy`] that closes the window behind it.
///
/// `relock = false` is the whole reason this composes: `prepare_deploy` already
/// relocked and staged `flake.lock`, so the appended `MetaSync` must do the dir
/// prep + `sync_agents` *without* re-locking over it.
///
/// `FinalizeDeploy` waits on **two** roots, which together reproduce the gate
/// the old fused node had around its inline `rebuild_no_meta` call:
/// - `AfterOk` `Prebuild` — a parent's state is its roll-up, so this is `Done`
/// only once `StopForUpdate` → `Swap` → `RebuildBookkeeping` all are (a failed *or*
/// cancelled child rolls the parent up `Failed`). That's the old
/// `build_result`.
/// - `AfterOk` `Reconcile` — the old call passed `deferred_start = false` on
/// purpose: the container had to come back up *before* the deploy was
/// finalized. `Reconcile` alone would not do, being `AfterAny` — it reaches
/// `Done` even after a failed `Swap`.
///
/// Declared into a **running** `DeployApply`'s own builder, not submitted: the
/// roots below become children of that node, which puts them inside the
/// `DeployWindow`'s subtree — so the `MetaWindow` this subgraph's `MetaSync`
/// and `FinalizeDeploy` declare is re-entered from the ancestor already holding
/// it rather than deadlocking against it.
pub(crate) fn deploy_rebuild_nodes(builder: &JobBuilder, agent: &str, approval_id: i64) {
let roots = rebuild_nodes(builder, agent, false, None);
let _finalize = builder
.node(NodeKind::FinalizeDeploy {
agent: agent.to_owned(),
approval_id,
})
.needs(Resource::MetaWindow)
.after_ok(roots.agent_window)
.after_ok(roots.reconcile);
}
/// One uniform rebuild shape — no `was_running` branch. `StopForUpdate`
/// noops when already down; the tail `Reconcile` auto-noops the start
/// when `wanted = Offline` (a rebuild of a deliberately-stopped agent
@ -375,71 +339,6 @@ pub fn rebuild(builder: &JobBuilder, agent: &str, relock: bool) -> Vec<hive_jobq
]
}
/// Approval-driven deploy (`MergeConfigPr`) as a phase subtree rather than the
/// single opaque node it used to be. Structure:
/// - `DeployWindow` (**root**): the resource holder — global meta window, agent
/// lease, build slot — held across every child below. No work of its own; it
/// reaches `Finishing` immediately and the children run inside it.
/// - `MergeVerify` (child): drift-gate + fetch + eval-verify. Mutates nothing,
/// so a failure here cancel-cascades its siblings with the forge and the
/// applied repo exactly as they were.
/// - `DeployApply` (child, `AfterOk` `MergeVerify`): the irreversible half —
/// ff-merge + `prepare_deploy`. It doesn't rebuild inline; it grows
/// [`deploy_rebuild_nodes`] into this DAG as its own children, so the build
/// and the closing `FinalizeDeploy` are real nodes under the same window.
/// - `DeployTail` (child, `AfterAny` `DeployApply`): the compensation +
/// bookkeeping tail — rollback when a merge landed unfinalized, forge tag
/// mirror, PR failure comment (see [`NodeKind::DeployTail`]).
///
/// - `ResolveApproval` (**root**, `AfterAny` `DeployWindow`): resolves the
/// approval row. A root rather than another child, so it isn't inside the
/// window's resource subtree — it runs once the window has released the meta
/// window, lease and build slot. One edge suffices here: `DeployWindow` is the
/// DAG's only other group-root, so its roll-up already *is* the whole
/// pipeline's outcome.
///
/// The window still spans the container build, as it must: `prepare_deploy`
/// leaves `flake.lock` staged-uncommitted for the build's whole duration.
pub fn approval_deploy(builder: &JobBuilder, agent: &str, approval_id: i64) {
let a = || agent.to_owned();
// The window is the widest holder in the tree: it brackets a nix
// build (`BuildSlot`), takes the container down across the swap
// (`Agent`), and serialises the meta mutation its subtree performs
// (`MetaWindow`). All three are held for its whole subtree, which
// is what lets the appended rebuild's `MetaSync` and the
// `FinalizeDeploy` re-enter rather than contend.
let window = builder
.node(NodeKind::DeployWindow {
agent: a(),
approval_id,
})
.needs(Resource::BuildSlot)
.needs(Resource::Agent(a()))
.needs(Resource::MetaWindow);
let verify = builder
.node(NodeKind::MergeVerify {
agent: a(),
approval_id,
})
.part_of(window);
let apply = builder
.node(NodeKind::DeployApply {
agent: a(),
approval_id,
})
.part_of(window)
.after_ok(verify);
let _tail = builder
.node(NodeKind::DeployTail {
agent: a(),
approval_id,
})
.part_of(window)
.after_any(apply);
resolve_approval_tails(builder, approval_id, window);
}
/// First deploy of an agent this hive has never seen, asked for by the swarm:
/// `Provision` (proposed/applied repos, state subvolume, meta registration)
/// then `Create` (`nixos-container create`), drop-in write, then `Reconcile`

180
hive-c0re/src/job_queue/tests.rs
View file

@ -1203,19 +1203,15 @@ fn cancelled_power_op_runs_no_compensating_node() {
fn cancelled_dag_still_runs_its_approval_tail() {
let q = JobQueue::new(1);
insert(&q, |builder| {
templates::approval_deploy(builder, "agent-a", 7);
templates::meta_update(builder, Vec::new(), Some(7));
});
// Found by kind, not returned: `approval_deploy` deliberately names
// nothing, because nothing polls it — the approval row is how an operator
// follows a deploy, so the template is fire-and-forget in production and a
// test must not make it return an id it would otherwise have no use for.
let window = node_of(&q, "deploy_window").get();
assert!(q.cancel(window), "fully-queued dag cancels");
// Found by kind, not returned: `meta_update` names nothing, because the
// approval row is how an operator follows it.
let lock = node_of(&q, "meta_lock").get();
assert!(q.cancel(lock), "fully-queued dag cancels");
// The `Cancelled` tail is the only node whose edge accepts a dropped
// dependency, so it is the only one `cancel` spares — and *which* tail
// survives is the whole assertion: the template emits one per outcome and
// the spared one names how the approval row is about to be resolved.
// Nothing computes it, so reading the survivor is reading the answer.
// survives names how the approval row is about to be resolved.
let spared = pending_payloads(&q);
assert!(
matches!(
@ -1227,171 +1223,9 @@ fn cancelled_dag_still_runs_its_approval_tail() {
),
"only the cancelled-outcome tail is spared, got {spared:?}"
);
// ⚠️ `Cancelled`, and it reads terminal **while the spared tail is still
// pending** — the one place this differs from the container era, where the
// cancel landed on a node *above* the window and the window rolled up
// `Finishing`. Here the operator cancels the window itself, so its own
// state is `Cancelled` however its subtree is doing. Deliberately asserted
// rather than routed around: the group's card goes terminal while a
// bookkeeping node runs on. That is acceptable for the tail this test
// protects (it resolves the approval row and nothing waits on it), and it
// would not be for work an operator expects to still be watching.
assert_eq!(state_of(&q, window), State::Cancelled);
// An unrelated group landing in the same graph doesn't disturb this one's
// state — a root rolls up its own subtree, not the graph.
let _other = insert_named(&q, |builder| rebuild(builder, "agent-b"));
assert_eq!(state_of(&q, window), State::Cancelled);
assert_eq!(state_of(&q, lock), State::Cancelled);
}
// ---- approval deploy subtree ----
/// The config-PR deploy is a subtree, not one opaque node. The
/// resource-holding root completes immediately (its `Finishing` state is the
/// parent gate that releases the children), then the phases run strictly in
/// order — and the `AfterAny` tail still runs when the irreversible half fails,
/// because it's the node that compensates for it.
#[test]
fn deploy_dag_runs_phases_in_order_and_tails_a_failed_apply() {
let q = JobQueue::new(1);
insert(&q, |builder| {
templates::approval_deploy(builder, "agent-a", 7);
});
assert_eq!(
declared_shape(&q),
vec![
// The window is the group root and holds the meta window for the
// whole subtree; the three phases are its sub-nodes.
row("deploy_window", None, &[]),
row("merge_verify", Some("deploy_window"), &[]),
// Apply only on a clean verify — a failed verify cancel-cascades
// it, which is what leaves the forge and the applied repo untouched.
row(
"deploy_apply",
Some("deploy_window"),
&[("merge_verify", "done")]
),
// The compensation tail accepts every terminal outcome of apply,
// *including `skipped`* — which is the state apply lands in when
// verify failed and it never ran. That one edge is the entire
// "still tails a failed apply / a failed verify" behaviour, and it
// is why two separate DAG-driving tests collapsed into this table.
row(
"deploy_tail",
Some("deploy_window"),
&[("deploy_apply", "done|failed|skipped")]
),
// One approval tail per outcome, gated on the window's roll-up.
row("resolve_approval", None, &[("deploy_window", "done")]),
row("resolve_approval", None, &[("deploy_window", "failed")]),
row("resolve_approval", None, &[("deploy_window", "cancelled")]),
]
);
}
/// The deploy's happy path: `DeployApply` does not build. It grows the ordinary
/// rebuild chain into the live DAG under itself, and `FinalizeDeploy` — gated on
/// that graft finishing — plants the deploy tag last.
///
/// The queue is built with **one** build slot on purpose. `DeployWindow` already
/// holds that slot (and the meta window) for the whole subtree, so the grafted
/// `Prebuild` can only ever claim by *re-entering* its ancestor's hold. If the
/// graft were rooted anywhere outside `DeployWindow`'s subtree it would block on
/// a resource its own DAG owns and deadlock — this test is what pins that down.
#[test]
fn deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it() {
// What `DeployApply` grows is `deploy_rebuild_nodes`' output, and that is a
// pure declaration — so it is declared here directly rather than by running
// a deploy far enough to graft it. **Reproducing the runtime path is not
// needed to test what the runtime path declares.**
//
// The grafting mechanism itself is hive_jobq's and tested there: the work
// lands under the emitter *before* it settles, and the emitter parks in
// `Finishing` so a downstream `AfterAny` gate stays shut while the new
// children run (`a_completing_node_grows_the_work_it_declared`,
// `parent_parks_in_finishing_until_children_roll_up`).
let q = JobQueue::new(1);
insert(&q, |builder| {
templates::deploy_rebuild_nodes(builder, "agent-a", 11);
});
assert_eq!(
declared_shape(&q),
vec![
row("meta_sync", None, &[]),
row("agent_window", None, &[("meta_sync", "done")]),
row("prebuild", Some("agent_window"), &[]),
row(
"stop_for_update",
Some("agent_window"),
&[("prebuild", "done")]
),
row("swap", Some("stop_for_update"), &[]),
row(
"rebuild_bookkeeping",
Some("stop_for_update"),
&[("swap", "done")]
),
row(
"reconcile",
None,
&[("agent_window", "done|failed|skipped")]
),
// The deploy tag is planted only after the rebuild came up clean:
// `AfterOk` on **both** roots, so either one failing skips it. That
// pair of edges is the whole "skips finalize on a failed graft"
// behaviour — no run needed to see it.
row(
"finalize_deploy",
None,
&[("agent_window", "done"), ("reconcile", "done")]
),
]
);
}
// `deploy_dag_skips_finalize_but_still_tails_a_failed_graft` lived here.
//
// A failure *inside* the grafted rebuild is the failure mode subgraph growth
// introduces: the deploy is already merged and the container half-swapped, so
// `FinalizeDeploy` must be cancel-cascaded (no `deployed/<id>` tag planted)
// while the tail still runs to compensate, and `Reconcile` is deliberately
// still reached — it boots the container back up.
//
// Every declared half of that is asserted by
// `deploy_apply_grows_rebuild_subgraph_and_finalizes_after_it`, which reads
// `deploy_rebuild_nodes`' shape directly:
//
// - "a failed swap still reaches Reconcile" is the `reconcile` row's
// `AfterAny` edge on `prebuild` (`done|failed|skipped`);
// - "finalize is cancel-cascaded" is `finalize_deploy`'s `AfterOk` pair —
// either root failing skips it;
// - "the tail still runs" is `deploy_tail`'s own `done|failed|skipped` edge
// on apply, asserted in the `approval_deploy` table above.
//
// The runtime halves are hive_jobq's: cascade on failure, roll-up, and
// `first_error` digging past a group root that rolled up `Failed` while
// carrying no error of its own (`first_error_skips_a_rolled_up_failure_
// carrying_no_error`). That last one is why the DAG reports "profile swap
// failed" rather than nothing — the mechanism, not this shape.
// `deploy_dag_skips_apply_but_still_runs_tail_when_verify_fails` lived here.
//
// A pre-merge rejection (drift gate, eval failure) cancel-cascades the
// irreversible half via its `AfterOk` edge, while the tail is still reached —
// it owns the forge mirror, not just compensation. That test and
// `deploy_dag_runs_phases_in_order_and_tails_a_failed_apply` differed only in
// *where* they injected the failure, and each drove the whole DAG to watch the
// tail run anyway.
//
// Both outcomes follow from one declared edge, which the surviving test now
// asserts directly: the tail accepts `done|failed|skipped` on apply, and
// `skipped` is exactly the state apply lands in when verify failed and it never
// ran. The runtime halves are hive_jobq's and tested there —
// `failed_after_ok_dep_cancels_dependents_but_after_any_still_runs` and
// `failed_child_rolls_parent_up_to_failed` (an Ok tail cannot launder a failed
// deploy into a success).
// ---- history ----
//
// The node → build-log link is no longer queue state: the log row carries