vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

job_queue: grow the rebuild subgraph from DeployApply (#2664)

Browse source 
The config-PR deploy's apply node still did the whole container rebuild
inline, through the last surviving `lifecycle::rebuild_no_meta` call. It
now merges, opens the two-phase meta deploy, and returns the ordinary
rebuild chain as a subgraph the scheduler grafts into the live DAG under
it. A new `FinalizeDeploy` node, gated on that graft, plants the deploy
tag and commits the staged lock.

Net effect: "did the agent come back up?" is answered by `Reconcile`
succeeding, the same way it is for every other rebuild, instead of by a
fused inline start — and each deploy phase is its own queue node, so the
dashboard shows which one is running.

The grafted nodes root on the apply node, so they land inside
`DeployWindow`'s subtree and re-enter the meta window and build slot it
already holds rather than deadlocking against them. The new happy-path
test runs on a one-slot queue specifically to pin that down.

`FinalizeDeploy`'s two git writes are fatal, deliberately: they are what
tells `DeployTail` a deploy confirmed good, so a node that merely warned
on them could report success while leaving the tail looking at the git
state of a failure — and the tail would then roll a good deploy back.
The trailing `meta::finalize_deploy` stays warn-only, since by then the
container already runs the new config.

The `failed/<id>` annotated tag moves into the tail, which is now the
only place holding a failed deploy. It reads the reason off the DAG via
a new `JobQueue::first_error`, and is gated on `main` having actually
moved — the rollback ref is parked *before* the merge, so its existence
alone does not mean a merge happened, and a pre-merge rejection must not
tag the previous, innocent head.

Removing the last inline rebuild orphaned a chain of now-dead code:
`rebuild_no_meta`, `container_exists`, `Coordinator::set_queue_build_log`
and `JobQueue::set_build_log_id_running`, all deleted here.
This commit is contained in:
atlas 2026-07-25 23:24:48 +02:00 • committed by mara
commit 3429a8c5a6
10 changed files with 388 additions and 257 deletions
Download patch file Download diff file Expand all files Collapse all files

122
hive-c0re/src/lifecycle/mod.rs
View file

@ -246,9 +246,9 @@ pub async fn create_container(name: &str, hive: &HiveEnv, paths: &AgentPaths) ->
create_only(name).await
}
/// Rebuild-path preamble shared by the job queue's `Prebuild` node and
/// `rebuild_no_meta`: fail fast on a port collision, then make sure
/// the applied repo + state dirs exist. Container untouched.
/// Rebuild-path preamble, run by the job queue's `Prebuild` node: fail fast on
/// a port collision, then make sure the applied repo + state dirs exist.
/// Container untouched.
pub async fn prepare_rebuild_dirs(name: &str, paths: &AgentPaths) -> Result<()> {
validate(name)?;
if let Some(other) = port_collision(name).await {
@ -331,19 +331,6 @@ pub async fn agents_for_meta_listing() -> Result<Vec<crate::meta::AgentSpec>> {
agents_for_meta(None).await
}
/// True when the named container already exists (appears in
/// `nixos-container list`). Used by the apply-commit path to decide
/// between first-spawn (`nixos-container create`) and normal rebuild
/// (`nixos-container update`).
pub async fn container_exists(name: &str) -> bool {
let container = container_name(name);
list()
.await
.unwrap_or_default()
.iter()
.any(|c| c == &container)
}
pub async fn kill(name: &str) -> Result<()> {
validate(name)?;
priv_run("stop", name).await
@ -453,9 +440,9 @@ async fn wait_until_running(name: &str, timeout: std::time::Duration) -> bool {
}
}
/// Internal implementation of the cold-start fallback. Used by
/// [`start_with_fallback`] (public, token-gated) and by
/// [`rebuild_no_meta`] where the preamble is already enforced structurally.
/// Internal implementation of the cold-start fallback, behind
/// [`start_with_fallback`] (public, token-gated) — the inner form exists for
/// callers that have already run the drop-in preamble themselves.
///
/// [`start`] already treats unit-active (not the exit code) as success and
/// waits out a slow boot, so this only layers the activation-error recovery on
@ -539,101 +526,6 @@ pub async fn destroy(name: &str) -> Result<()> {
Ok(())
}
/// Container-level rebuild without touching the meta repo. The one
/// remaining fused stop/update/start pipeline: the approval deploy
/// (`actions::deploy_applied_target`) drives meta through the
/// two-phase prepare/finalize/abort flow itself and needs the inline
/// start to verify the agent comes back up before finalizing. Every
/// other rebuild is a job-queue DAG (`Prebuild → StopForUpdate → Swap
/// → Reconcile`) whose `Prebuild` executor owns the meta sync +
/// relock this path's deleted `rebuild` wrapper used to do.
///
/// `on_step` is called at each phase boundary with a short human-readable
/// label so callers can surface progress (e.g. update the rebuild-queue
/// step shown in the dashboard). Pass `&|_| ()` when progress reporting
/// is not needed.
///
/// `on_build_log_id` is called with the build-log row id immediately after
/// the `nixos-container update` log row opens, before the actual update
/// command starts. Callers can use this to link the queue entry to the log
/// for live streaming. Pass `&|_| ()` when not needed.
///
/// `defer_start` skips the start-after-update for a previously-running
/// container and returns `true` instead, so a queue-side caller can hand
/// the (potentially slow) container boot to the fast lane rather than
/// holding the serialized build lane through it. With `defer_start =
/// false` the start (with cold-start fallback) runs inline as before and
/// the return value is always `false`. The spawn path always starts
/// inline — a freshly-created container boots as part of provisioning.
pub async fn rebuild_no_meta(
name: &str,
hive: &HiveEnv,
paths: &AgentPaths,
defer_start: bool,
on_step: &(dyn Fn(&str) + Send + Sync),
on_build_log_id: &(dyn Fn(i64) + Send + Sync),
) -> Result<bool> {
prepare_rebuild_dirs(name, paths).await?;
let flake_ref = format!("{}#{name}", crate::paths::meta_root().display());
if container_exists(name).await {
// Rebuild strategy: stop-before-update + pre-build.
// See `docs/coordinator.md::Container lifecycle`.
let was_running = is_running(name).await;
write_dropins(name, hive, paths).await?;
if was_running {
on_step("nix build");
prebuild_toplevel(name, &flake_ref, &|_| ()).await?;
on_step("nixos-container stop");
priv_run("stop", name).await?;
}
on_step("nixos-container update");
let update_result = priv_run_inner("update", name, Some(on_build_log_id)).await;
if let Err(ref update_err) = update_result {
// The update failed (e.g. nix build error). If the agent was
// running before we stopped it, try to bring it back up on the
// previous successful configuration so it doesn't stay dead.
// The start failure is logged but not promoted to an error —
// we always propagate the original update error (below).
if was_running {
tracing::warn!(
%name,
error = %update_err,
"nixos-container update failed; attempting restart on old config"
);
on_step("nixos-container start (recovery)");
if let Err(e) = priv_run("start", name).await {
tracing::warn!(%name, error = %e, "recovery start after failed update also failed");
}
}
}
update_result?;
if was_running {
if defer_start {
// The caller re-queues the start on the fast lane so the
// build lane is freed for the next entry instead of
// waiting out the container boot here.
return Ok(true);
}
on_step("nixos-container start");
// write_dropins was called above; use the inner fn directly
// since the preamble is enforced structurally in this path.
start_with_fallback_inner(name).await?;
}
Ok(false)
} else {
// Spawn path: create is atomic, no prebuild needed.
// See `docs/coordinator.md::Spawn path`.
on_step("nixos-container create");
priv_run("create", name).await?;
// Runtime dir must exist before nixos-container start.
ensure_agent_runtime_dir(name)?;
write_dropins(name, hive, paths).await?;
on_step("nixos-container start");
priv_run("start", name).await?;
Ok(false)
}
}
/// Pre-build `system.build.toplevel` against `meta#<name>` so the
/// subsequent `nixos-container update` finds the result cached and
/// skips straight to the profile-swap. Store-warming only — container
@ -652,7 +544,7 @@ pub async fn prebuild_toplevel(
use tokio::io::{AsyncBufReadExt, BufReader};
// Split `<root>#<name>` so we can re-emit with the explicit
// `nixosConfigurations.<name>` segment. The flake_ref shape is
// constructed by `rebuild_no_meta` and always contains exactly one
// constructed by the caller and always contains exactly one
// `#`; `split_once` returning None here would be a programmer
// error we'd want to surface loudly rather than paper over.
let (flake_root, fragment) = flake_ref