vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

job_queue: grow the rebuild subgraph from DeployApply (#2664)

Browse source 
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.

Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.

The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.

`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.

The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.

Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
This commit is contained in:
atlas 2026-07-25 23:24:48 +02:00 • committed by mara
commit 3429a8c5a6
10 changed files with 388 additions and 257 deletions
Download patch file Download diff file Expand all files Collapse all files

189
hive-c0re/src/actions.rs
View file

@ -19,7 +19,8 @@ use crate::lifecycle;
///
/// Dispatch:
/// - `MergeConfigPr` → a `DeployWindow` DAG (`MergeVerify → DeployApply →
/// DeployTail` under a resource-holding root; ~30-90s)
/// FinalizeDeploy`, plus an `AfterAny` `DeployTail`, under a
/// resource-holding root; ~30-90s)
/// - `UpdateMetaInputs` → a `MetaUpdate` DAG (fan-out on completion)
/// - `Spawn` → a `Spawn` DAG (`Create → WriteDropin → Reconcile`)
/// - `InitConfig` → inline (<1s; queue card would be noise)
@ -105,8 +106,9 @@ pub async fn approve(coord: Arc<Coordinator>, id: i64) -> Result<()> {
// The work ends in a container rebuild, so route it through the
// rebuild queue. The queue worker dispatches the deploy DAG's
// nodes to `run_deploy_merge_verify` (drift gate + eval),
// `run_deploy_apply` (ff-merge + rebuild) and `run_deploy_tail`
// (compensation + forge mirror).
// `run_deploy_apply` (ff-merge, then grows the rebuild subgraph),
// `run_finalize_deploy` (deploy tag + lock commit) and
// `run_deploy_tail` (compensation + forge mirror).
enqueue_approval_rebuild(
&coord,
approval.agent.as_str(),
@ -148,8 +150,8 @@ fn enqueue_approval_rebuild(
/// handed between nodes: hive-c0re can restart between the apply and the tail,
/// and the whole point of splitting the deploy is that the tail still knows
/// what to undo when it does. The ref's existence IS the "a merge landed but
/// hasn't been confirmed good yet" flag — [`run_deploy_apply`] drops it the
/// moment the rebuild succeeds.
/// hasn't been confirmed good yet" flag — [`run_finalize_deploy`] drops it the
/// moment the rebuild has come up clean.
fn rollback_ref(approval_id: i64) -> String {
format!("refs/hyperhive/rollback/{approval_id}")
}
@ -164,7 +166,6 @@ struct DeployCtx {
pr: u64,
/// The PR head sha the operator reviewed (`approval.fetched_sha`).
reviewed: String,
agent_dir: std::path::PathBuf,
applied_dir: std::path::PathBuf,
/// The agent's forge config repo (`<owner>/<name>`).
repo: String,
@ -184,7 +185,6 @@ fn deploy_ctx(coord: &Coordinator, approval_id: i64) -> Result<DeployCtx> {
Ok(DeployCtx {
pr,
reviewed,
agent_dir: crate::paths::agent_runtime_dir(approval.agent.as_str()),
applied_dir: crate::paths::applied_dir(approval.agent.as_str()),
repo: crate::forge::config_repo(approval.agent.as_str()),
approval,
@ -246,20 +246,24 @@ pub async fn run_deploy_merge_verify(
}
/// `DeployApply` node body — the irreversible half. Parks the rollback ref,
/// fast-forward-merges the PR (THE merge), then runs the deploy proper.
/// fast-forward-merges the PR (THE merge), then opens the deploy.
///
/// The ref is parked *before* the merge, so a hive-c0re crash anywhere from
/// here on still leaves [`run_deploy_tail`] enough to undo. If the merge itself
/// fails, `main` never moved and the tail's compensation is a no-op against the
/// same sha — harmless, and cheaper than trying to be clever about it.
///
/// Returning `Ok` is the signal for the caller to grow the rebuild subgraph into
/// this DAG under this node; the build itself does not happen here.
///
/// # Errors
///
/// Returns an error if the approval can't be loaded, if reading or parking the
/// pre-merge `main` sha fails, if the forge refuses the fast-forward merge
/// (including a head that drifted between verify and merge), or if the deploy
/// of the merged target fails. From the merge onward a failure is *not*
/// retryable on its own — [`run_deploy_tail`] runs `AfterAny` to compensate.
/// (including a head that drifted between verify and merge), or if
/// fast-forwarding `applied/main` / `meta::prepare_deploy` fails. From the merge
/// onward a failure is *not* retryable on its own — [`run_deploy_tail`] runs
/// `AfterAny` to compensate.
pub async fn run_deploy_apply(
coord: &Arc<Coordinator>,
queue_entry_id: Option<u64>,
@ -290,13 +294,11 @@ pub async fn run_deploy_apply(
Err(e) => bail!("ff-merge PR #{pr}: {e}"),
}
deploy_applied_target(
prepare_applied_target(
coord,
agent,
&ctx.agent_dir,
&ctx.applied_dir,
&ctx.reviewed,
approval_id,
queue_entry_id,
)
.await
@ -307,9 +309,11 @@ pub async fn run_deploy_apply(
/// construction: it is the recovery step, so it has nothing to hand a failure
/// to. Every fallible call inside warns and continues.
///
/// 1. If the rollback ref survived, the deploy did not confirm good: roll
/// `applied/main` back to the parked sha, resync the working tree, and drop
/// the staged meta lock so the deploy log only ever shows successes.
/// 1. If the rollback ref survived, the deploy did not confirm good: tag the
/// merged commit `failed/<id>` (annotated with the DAG's first error, while
/// that sha is still reachable), then roll `applied/main` back to the parked
/// sha, resync the working tree, and drop the staged meta lock so the deploy
/// log only ever shows successes.
/// 2. Mirror the agent's config repo to the forge. `main` is already ff'd by
/// the merge, so only the `deployed/<id>` / `failed/<id>` tag refspec
/// actually lands — that's what gives the merged commit a forge-visible
@ -339,6 +343,34 @@ pub async fn run_deploy_tail(
"deploy tail: rollback ref outlived a successful deploy; dropping it without compensating"
);
} else {
// Mark the commit that failed to deploy, before undoing the merge
// that put it on `main`. After the rollback below, that sha is only
// reachable through this tag.
//
// Gated on `main` having actually moved: the rollback ref is parked
// *before* the merge, so its existence alone doesn't mean a merge
// happened. A pre-merge rejection (drift gate, eval failure, or the
// ff-merge itself failing) has no deployed commit to blame, and
// tagging the previous — innocent — head would point the operator at
// a commit that never got near a container.
//
// The annotation is read off the DAG rather than passed down from
// the node that failed: this node runs `AfterAny` its subject, so by
// now that node has settled `Failed` with its error recorded.
if let Ok(merged) = lifecycle::git_rev_parse(&applied_dir, "refs/heads/main").await
&& merged != prev_main
{
let tag = format!("failed/{approval_id}");
let body = queue_entry_id
.and_then(|dag_id| coord.job_queue.first_error(dag_id))
.unwrap_or_else(|| "deploy failed".to_owned());
if let Err(e) =
lifecycle::git_tag_annotated(&applied_dir, &tag, &merged, &body).await
{
tracing::warn!(%agent, approval_id, error = ?e, "deploy tail: annotate failed tag failed");
}
}
coord.set_queue_step(queue_entry_id, "roll back applied/main");
if let Err(e) =
lifecycle::git_update_ref(&applied_dir, "refs/heads/main", &prev_main).await
@ -740,9 +772,11 @@ fn finish_approval(
result
}
/// Post-merge deploy for the config-PR flow. Fast-forwards `applied/main` to
/// `target`, syncs the working tree, runs the meta two-phase deploy + container
/// rebuild, and plants the `deployed/<id>` / `failed/<id>` bookkeeping tag.
/// Open the deploy for the config-PR flow: fast-forward `applied/main` to
/// `target`, sync the working tree, and run phase 1 of the meta two-phase
/// deploy. The container rebuild that used to run inline here is now the
/// subgraph [`run_deploy_apply`]'s node grows into the DAG, and the closing half
/// is [`run_finalize_deploy`].
///
/// **Undo is not this function's job.** Every early return here leaves the
/// applied repo dirty on purpose — [`run_deploy_tail`] owns compensation, and
@ -751,18 +785,14 @@ fn finish_approval(
/// pre-merge sha in a git ref instead of a local variable.
///
/// Caller-specific bits stay OUT of here: fetching the PR head, the
/// `verify_commit` gate, and the ff-merge. `target` is both what `applied/main`
/// fast-forwards to and the sha `meta::finalize_deploy` records — for a merge
/// they are always the same reviewed head. The agent always already exists here
/// `verify_commit` gate, and the ff-merge. The agent always already exists here
/// (a merge is never a first spawn), so there's no `sync_agents` step — the
/// operator `Spawn` flow owns first-time meta registration.
async fn deploy_applied_target(
async fn prepare_applied_target(
coord: &Arc<Coordinator>,
agent: &str,
agent_dir: &std::path::Path,
applied_dir: &std::path::Path,
target: &str,
id: i64,
queue_entry_id: Option<u64>,
) -> Result<()> {
coord.set_queue_step(queue_entry_id, "fast-forward applied/main");
@ -777,70 +807,59 @@ async fn deploy_applied_target(
.map_err(|e| anyhow::anyhow!("read-tree to main: {e:#}"))?;
coord.set_queue_step(queue_entry_id, "meta prepare_deploy");
// Phase 1 of the meta two-phase deploy: relock without committing.
// Phase 1 of the meta two-phase deploy: relock without committing. The
// staged lock then stays uncommitted across the whole appended rebuild —
// which is why the `MetaWindow` is held by the deploy root, not by a node.
crate::meta::prepare_deploy(agent)
.await
.map_err(|e| anyhow::anyhow!("meta prepare_deploy: {e:#}"))?;
.map_err(|e| anyhow::anyhow!("meta prepare_deploy: {e:#}"))
}
// Container-level rebuild (or first-time create) against meta#<name>.
// Step labels are emitted inside rebuild_no_meta via the callback so
// the dashboard reflects actual phase progress rather than a static
// "nixos-container update" label for the whole multi-minute window.
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(agent, agent_dir.to_path_buf());
let build_result = lifecycle::rebuild_no_meta(
agent,
&hive,
&paths,
// Inline start: the apply-commit flow verifies the agent comes
// back up before finalizing the deploy tag, so the start stays
// part of this entry rather than a deferred fast-lane follow-up.
false,
&|step| coord.set_queue_step(queue_entry_id, step),
&|log_id| coord.set_queue_build_log(queue_entry_id, log_id),
)
.await;
/// `FinalizeDeploy` node body — phase 2 of the meta two-phase deploy, run once
/// the appended rebuild subgraph has built, swapped, and brought the container
/// back up.
///
/// Drops the rollback ref *first*: from here the deploy is good and
/// [`run_deploy_tail`] must not roll `main` back. Ordering that ahead of the tag
/// plant is what makes the tail's `deployed/<id>` cross-check a second line of
/// defence rather than the only one. No agent kick — the rebuild's own
/// `PostSwap` already did it.
///
/// # Errors
///
/// Returns an error if the approval can't be loaded, if dropping the rollback
/// ref fails, or if planting the `deployed/<id>` tag fails. Those two git writes
/// *are* the deploy's "confirmed good" signal, so warning past them would let
/// this node report success while leaving the tail looking at the git state of a
/// failure — and the tail would then compensate a good deploy. A failing
/// `meta::finalize_deploy` is deliberately *not* an error: the container is
/// already running the new config by then, and the staged `flake.lock` it
/// couldn't commit is something the operator can land by hand.
pub async fn run_finalize_deploy(
coord: &Arc<Coordinator>,
queue_entry_id: Option<u64>,
approval_id: i64,
) -> Result<()> {
let ctx = deploy_ctx(coord, approval_id)?;
let agent = ctx.approval.agent.as_str();
let target = ctx.reviewed.as_str();
match build_result {
Ok(_) => {
coord.set_queue_step(queue_entry_id, "finalize deploy");
// Drop the compensation ref FIRST: from here the deploy is good and
// the tail must not roll `main` back. Ordering it ahead of the tag
// plant is what makes the tail's `deployed/<id>` cross-check a
// second line of defence rather than the only one.
if let Err(e) = lifecycle::git_delete_ref(applied_dir, &rollback_ref(id)).await {
tracing::warn!(%agent, %id, error = ?e, "drop rollback ref after successful deploy failed");
}
let tag = format!("deployed/{id}");
if let Err(e) = lifecycle::git_tag(applied_dir, &tag, target).await {
tracing::warn!(%agent, %id, error = ?e, "plant deployed tag failed");
}
if let Err(e) = crate::meta::finalize_deploy(agent, target, &tag).await {
// The build itself succeeded — meta lock landed but
// couldn't be committed. Surface as a soft warn so the
// operator can git-commit by hand if they care.
tracing::warn!(%agent, %id, error = ?e, "meta finalize_deploy failed");
}
// Wake the agent on its next turn so claude sees the
// config change took effect. Same hint pattern as
// auto_update::rebuild_agent — manager approved a
// proposal, agent picks up where it left off with the
// new env / packages.
coord.kick_agent(agent, "config update applied");
Ok(())
}
Err(e) => {
// Plant the failure marker here rather than in the tail: this is
// the only place that holds the build error to annotate it with.
// The repo-state rollback is the tail's, via the parked ref.
let tag = format!("failed/{id}");
let body = format!("{e:#}");
if let Err(te) = lifecycle::git_tag_annotated(applied_dir, &tag, target, &body).await {
tracing::warn!(%agent, %id, error = ?te, "annotate failed tag failed");
}
Err(e)
}
coord.set_queue_step(queue_entry_id, "drop rollback ref");
lifecycle::git_delete_ref(&ctx.applied_dir, &rollback_ref(approval_id))
.await
.map_err(|e| anyhow::anyhow!("drop rollback ref for approval {approval_id}: {e:#}"))?;
coord.set_queue_step(queue_entry_id, "plant deployed tag");
let tag = format!("deployed/{approval_id}");
lifecycle::git_tag(&ctx.applied_dir, &tag, target)
.await
.map_err(|e| anyhow::anyhow!("plant {tag}: {e:#}"))?;
coord.set_queue_step(queue_entry_id, "meta finalize_deploy");
if let Err(e) = crate::meta::finalize_deploy(agent, target, &tag).await {
tracing::warn!(%agent, approval_id, error = ?e, "meta finalize_deploy failed");
}
Ok(())
}
/// Tear down a sub-agent container. By default this is non-destructive to