vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

defer start-after-rebuild to the fast lane so the build lane frees up (closes #2094)

Browse source
This commit is contained in:
damocles 2026-07-01 23:37:13 +02:00
commit b191858366
6 changed files with 162 additions and 61 deletions
Download patch file Download diff file Expand all files Collapse all files

22
docs/coordinator.md
View file

@ -34,7 +34,7 @@ somewhere."
| Kind | Description |
|------|-------------|
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. |
| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see *Deferred start* under the rebuild path below. |
| `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. |
| `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. |
| `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. |
@ -42,7 +42,7 @@ somewhere."
| `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. |
| `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/<agent>`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. |
**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill`. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.)
**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill` via the direct API paths. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard.
### Dedup
@ -185,6 +185,19 @@ Sequence for a running container:
If the container is already stopped, step 1 is skipped (no downtime to shave — no
point evaluating the flake twice).
**Deferred start (queue-dispatched rebuilds):** step 4 can take a while
(container boot), and holding the serialized build lane through it delays the
next queued rebuild's nix build for no reason. Queue-dispatched rebuilds
therefore pass `defer_start` — `rebuild_no_meta` skips the start and returns
`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead
(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under
it — the same split the graceful-stop path uses for its container stop). The
build-lane entry completes when the profile-swap finishes; a start failure
surfaces on the `Start` entry, which runs with the cold-start fallback. Direct
callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy
flow which verifies the agent comes back up before finalizing) keep the start
inline.
### Cold-start fallback
`start` after `update` can exit non-zero when packages are **removed** between
@ -194,7 +207,10 @@ half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
Both errors are preserved and surfaced if the cold-start also fails.
Both errors are preserved and surfaced if the cold-start also fails. The fallback
lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild
path and the queue's fast-lane `Start` handler (which the deferred
start-after-rebuild rides).
### Spawn path (new container)

6
hive-c0re/src/actions.rs
View file

@ -862,6 +862,10 @@ async fn deploy_applied_target(
agent,
&hive,
&paths,
// Inline start: the apply-commit flow verifies the agent comes
// back up before finalizing the deploy tag, so the start stays
// part of this entry rather than a deferred fast-lane follow-up.
false,
&|step| coord.set_queue_step(queue_entry_id, step),
&|log_id| {
if let Some(qid) = queue_entry_id
@ -874,7 +878,7 @@ async fn deploy_applied_target(
.await;
match build_result {
Ok(()) => {
Ok(_) => {
coord.set_queue_step(queue_entry_id, "finalize deploy");
let tag = format!("deployed/{id}");
if let Err(e) = lifecycle::git_tag(applied_dir, &tag, target_ref).await {

38
hive-c0re/src/auto_update.rs
View file

@ -73,6 +73,13 @@ pub fn agent_config_pending(name: &str, deployed_sha: Option<&str>) -> bool {
/// rebuilds, where re-locking would revert the bump the cascade just
/// committed (see `lifecycle::rebuild`).
///
/// `defer_start_source` is `Some(source)` for queue-dispatched rebuilds:
/// instead of holding the serialized build lane through the container
/// boot, the start-after-rebuild is enqueued as a fast-lane `Start`
/// entry (grouped under this rebuild via `parent_id`, same split as the
/// graceful-stop follow-up). Pass `None` for direct callers to keep the
/// start inline.
///
/// # Errors
///
/// Propagates errors from `coord.ensure_runtime` and `lifecycle::rebuild`.
@ -82,6 +89,7 @@ pub async fn rebuild_agent(
current_rev: &str,
queue_entry_id: Option<u64>,
relock: bool,
defer_start_source: Option<crate::rebuild_queue::QueueSource>,
) -> Result<()> {
tracing::info!(%name, rev = %current_rev, "rebuild agent");
let agent_dir = coord
@ -99,6 +107,7 @@ pub async fn rebuild_agent(
&hive,
&paths,
relock,
defer_start_source.is_some(),
&|step| coord.set_queue_step(queue_entry_id, step),
&|log_id| {
if let Some(qid) = queue_entry_id
@ -111,10 +120,33 @@ pub async fn rebuild_agent(
.await;
drop(guard);
match &result {
Ok(()) => {
Ok(needs_start) => {
if let Err(e) = std::fs::write(rev_marker_path(name), current_rev) {
tracing::warn!(%name, error = ?e, "write rev marker failed");
}
// Deferred start: hand the container boot to the fast lane so
// this build-lane entry completes now and the next queued
// rebuild's nix build overlaps with the boot. `parent_id`
// groups the follow-up under this rebuild on the dashboard —
// same split the graceful-stop path uses for its container
// stop. A start failure surfaces on the Start entry (with
// the cold-start fallback) instead of failing the rebuild.
if *needs_start && let Some(source) = defer_start_source {
coord
.rebuild_queue
.enqueue_full(crate::rebuild_queue::FullEnqueue {
kind: crate::rebuild_queue::QueueKind::Start,
agent: name.to_owned(),
source,
reason: format!("start after rebuild of {name}"),
parent_id: queue_entry_id,
inputs: Vec::new(),
approval_id: None,
perm_payload: None,
depends_on: Vec::new(),
});
coord.emit_rebuild_queue_snapshot();
}
coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: name.to_owned(),
ok: true,
@ -161,7 +193,7 @@ pub async fn rebuild_agent(
coord.rescan_containers_and_emit().await;
}
}
result
result.map(|_| ())
}
/// Whether this hive is "ruthless" — running with no root/manager agent at
@ -210,7 +242,7 @@ pub async fn ensure_root_agent(coord: &Arc<Coordinator>) -> Result<()> {
);
let coord_clone = coord.clone();
if let Err(e) =
rebuild_agent(&coord_clone, MANAGER_NAME, rev.as_str(), None, true).await
rebuild_agent(&coord_clone, MANAGER_NAME, rev.as_str(), None, true, None).await
{
tracing::warn!(error = ?e, "manager migration rebuild failed");
}

109
hive-c0re/src/lifecycle.rs
View file

@ -372,6 +372,50 @@ pub async fn start(name: &str) -> Result<()> {
priv_run("start", name).await
}
/// Start with the cold-start fallback: when a plain start fails (the
/// activation-error shape), retry once via stop + kill + start before
/// giving up. Used by the queue's fast-lane `Start` handler and the
/// inline start-after-rebuild path.
/// See `docs/coordinator.md::Cold-start fallback`.
///
/// # Errors
///
/// Propagates the retry's start error (annotated with the original
/// failure) when the fallback also fails.
pub async fn start_with_fallback(name: &str) -> Result<()> {
validate(name)?;
if let Err(start_err) = priv_run("start", name).await {
let container = container_name(name);
tracing::warn!(
container = %container,
error = %start_err,
"start failed (possible activation error); retrying via stop + kill + start"
);
priv_run("stop", name).await.unwrap_or_else(|e| {
tracing::warn!(
container = %container,
error = %e,
"stop before cold-start retry failed (ignored)"
);
});
priv_run("kill", name).await.unwrap_or_else(|e| {
tracing::warn!(
container = %container,
error = %e,
"kill before cold-start retry failed (ignored)"
);
});
priv_run("start", name).await.map_err(|e| {
anyhow::anyhow!(
"cold-start fallback also failed: {e:#} \
(original start error: {start_err:#})"
)
})
} else {
Ok(())
}
}
/// Stop + start without regenerating any config. For "kick the container"
/// without touching the flake or nspawn flags.
pub async fn restart(name: &str) -> Result<()> {
@ -421,14 +465,19 @@ pub async fn destroy(name: &str) -> Result<()> {
///
/// Propagates errors from meta-flake sync / lock-update and the
/// `nixos-container` apply + restart shellouts.
///
/// Returns `true` when `defer_start` suppressed the start-after-update —
/// the caller owns bringing the container back up (see
/// [`rebuild_no_meta`]).
pub async fn rebuild(
name: &str,
hive: &HiveEnv,
paths: &AgentPaths,
relock: bool,
defer_start: bool,
on_step: &(dyn Fn(&str) + Send + Sync),
on_build_log_id: &(dyn Fn(i64) + Send + Sync),
) -> Result<()> {
) -> Result<bool> {
// Sync the meta flake (idempotent — no-op when the rendered
// flake matches disk) so a manual rebuild from the dashboard
// can also recover from a divergent meta repo (e.g. an agent
@ -449,7 +498,7 @@ pub async fn rebuild(
if relock {
crate::meta::lock_update_for_rebuild(name).await?;
}
rebuild_no_meta(name, hive, paths, on_step, on_build_log_id).await
rebuild_no_meta(name, hive, paths, defer_start, on_step, on_build_log_id).await
}
/// Container-level rebuild without touching the meta repo. Callers
@ -467,13 +516,22 @@ pub async fn rebuild(
/// the `nixos-container update` log row opens, before the actual update
/// command starts. Callers can use this to link the queue entry to the log
/// for live streaming. Pass `&|_| ()` when not needed.
///
/// `defer_start` skips the start-after-update for a previously-running
/// container and returns `true` instead, so a queue-side caller can hand
/// the (potentially slow) container boot to the fast lane rather than
/// holding the serialized build lane through it. With `defer_start =
/// false` the start (with cold-start fallback) runs inline as before and
/// the return value is always `false`. The spawn path always starts
/// inline — a freshly-created container boots as part of provisioning.
pub async fn rebuild_no_meta(
name: &str,
hive: &HiveEnv,
paths: &AgentPaths,
defer_start: bool,
on_step: &(dyn Fn(&str) + Send + Sync),
on_build_log_id: &(dyn Fn(i64) + Send + Sync),
) -> Result<()> {
) -> Result<bool> {
validate(name)?;
if let Some(other) = port_collision(name).await {
bail!(
@ -528,42 +586,16 @@ pub async fn rebuild_no_meta(
}
update_result?;
if was_running {
// Cold-start fallback on activation errors.
// See `docs/coordinator.md::Cold-start fallback`.
on_step("nixos-container start");
if let Err(start_err) = priv_run("start", name).await {
tracing::warn!(
container = %container,
error = %start_err,
"start after rebuild failed (possible activation error); \
retrying via stop + kill + start"
);
priv_run("stop", name).await.unwrap_or_else(|e| {
tracing::warn!(
container = %container,
error = %e,
"stop before cold-start retry failed (ignored)"
);
});
priv_run("kill", name).await.unwrap_or_else(|e| {
tracing::warn!(
container = %container,
error = %e,
"kill before cold-start retry failed (ignored)"
);
});
priv_run("start", name).await.map_err(|e| {
anyhow::anyhow!(
"cold-start fallback also failed: {e:#} \
(original start error: {start_err:#})"
)
})
} else {
Ok(())
if defer_start {
// The caller re-queues the start on the fast lane so the
// build lane is freed for the next entry instead of
// waiting out the container boot here.
return Ok(true);
}
} else {
Ok(())
on_step("nixos-container start");
start_with_fallback(name).await?;
}
Ok(false)
} else {
// Spawn path: create is atomic, no prebuild needed.
// See `docs/coordinator.md::Spawn path`.
@ -579,7 +611,8 @@ pub async fn rebuild_no_meta(
set_resource_limits(&container, &hive.agent_cpu_quota, &hive.agent_memory_max).await?;
systemd_daemon_reload().await?;
on_step("nixos-container start");
priv_run("start", name).await
priv_run("start", name).await?;
Ok(false)
}
}

44
hive-c0re/src/rebuild_queue.rs
View file

@ -890,21 +890,12 @@ async fn dispatch(
dispatch_rebuild_approval(coord, entry, approval_id).await
}
(QueueKind::Rebuild, None) => {
let current_rev =
crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default();
// A meta-update cascade has just set the meta lock; re-locking
// in the per-agent rebuild would revert it (the agent's own
// flake.lock wins). Every other source wants the relock so it
// advances to applied/<n>/main.
let relock = entry.source != QueueSource::MetaUpdate;
crate::auto_update::rebuild_agent(
coord,
&entry.agent,
&current_rev,
Some(entry.id),
relock,
)
.await
rebuild_for_entry(coord, entry, relock).await
}
(QueueKind::MetaUpdate, Some(approval_id)) => {
crate::actions::run_approval_update_meta_inputs(coord, Some(entry.id), approval_id)
@ -1002,9 +993,7 @@ async fn dispatch(
}
// Now rebuild so the updated HIVE_TOOL_GROUPS / HIVE_CAPABILITIES
// env var takes effect in the container.
let current_rev =
crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default();
crate::auto_update::rebuild_agent(coord, name, &current_rev, Some(entry.id), true).await
rebuild_for_entry(coord, entry, true).await
}
(QueueKind::GracefulStop, _) => {
run_graceful_stop(coord, entry);
@ -1015,9 +1004,36 @@ async fn dispatch(
}
}
/// Queue-side container rebuild for `entry.agent`: resolves the current
/// flake rev and hands off to `rebuild_agent` with the entry's id +
/// source. Passing the source defers the start-after-rebuild to a
/// fast-lane `Start` follow-up (grouped under this entry via
/// `parent_id`), so the build lane is freed for the next entry instead
/// of waiting out the container boot.
async fn rebuild_for_entry(
coord: &std::sync::Arc<crate::coordinator::Coordinator>,
entry: &QueueEntry,
relock: bool,
) -> anyhow::Result<()> {
let current_rev =
crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default();
crate::auto_update::rebuild_agent(
coord,
&entry.agent,
&current_rev,
Some(entry.id),
relock,
Some(entry.source),
)
.await
}
/// Start a stopped container off the queue (`QueueKind::Start`), with a
/// `Starting` transient so the dashboard shows a visible queued→running
/// progression rather than the sub-second flash of a direct start.
/// Uses the cold-start fallback (stop + kill + start retry) so the
/// deferred start-after-rebuild keeps the same activation-error recovery
/// it had when it ran inline on the build lane.
async fn run_start(
coord: &std::sync::Arc<crate::coordinator::Coordinator>,
entry: &QueueEntry,
@ -1025,7 +1041,7 @@ async fn run_start(
let name = &entry.agent;
let _guard = coord.transient_guard(name, crate::coordinator::TransientKind::Starting);
coord.set_queue_step(Some(entry.id), "nixos-container start");
crate::lifecycle::start(name).await?;
crate::lifecycle::start_with_fallback(name).await?;
coord.kick_agent(name, "container started");
coord.rescan_containers_and_emit().await;
Ok(())

4
hive-c0re/src/server.rs
View file

@ -458,14 +458,14 @@ async fn handle_rebuild(coord: &Arc<Coordinator>, name: &str) -> Result<HostResp
let agent_dir = coord.ensure_runtime(name)?;
let hive = coord.hive_env();
let paths = Coordinator::agent_paths(name, agent_dir);
let result = lifecycle::rebuild(name, &hive, &paths, true, &|_| (), &|_| ()).await;
let result = lifecycle::rebuild(name, &hive, &paths, true, false, &|_| (), &|_| ()).await;
// Mirror auto_update::rebuild_agent — the manager wants to know
// about every rebuild attempt regardless of which surface triggered
// it, especially failures (build error → manager can adjust the
// agent's agent.nix). Without this the admin-socket CLI was a
// notify-gap.
match &result {
Ok(()) => {
Ok(_) => {
coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
agent: name.to_owned(),
ok: true,