diff --git a/docs/coordinator.md b/docs/coordinator.md index bfd55093..7a90920f 100644 --- a/docs/coordinator.md +++ b/docs/coordinator.md @@ -34,7 +34,7 @@ somewhere." | Kind | Description | |------|-------------| -| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. | +| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see *Deferred start* under the rebuild path below. | | `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. | | `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. | | `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. | @@ -42,7 +42,7 @@ somewhere." | `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. | | `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. | -**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill`. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) +**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill` via the direct API paths. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard. ### Dedup @@ -185,6 +185,19 @@ Sequence for a running container: If the container is already stopped, step 1 is skipped (no downtime to shave — no point evaluating the flake twice). +**Deferred start (queue-dispatched rebuilds):** step 4 can take a while +(container boot), and holding the serialized build lane through it delays the +next queued rebuild's nix build for no reason. Queue-dispatched rebuilds +therefore pass `defer_start` — `rebuild_no_meta` skips the start and returns +`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead +(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under +it — the same split the graceful-stop path uses for its container stop). The +build-lane entry completes when the profile-swap finishes; a start failure +surfaces on the `Start` entry, which runs with the cold-start fallback. Direct +callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy +flow which verifies the agent comes back up before finalizing) keep the start +inline. + ### Cold-start fallback `start` after `update` can exit non-zero when packages are **removed** between @@ -194,7 +207,10 @@ half-started at that point. Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes) → `start` (clean cold-start, no generation transition, new activation runs cleanly). -Both errors are preserved and surfaced if the cold-start also fails. +Both errors are preserved and surfaced if the cold-start also fails. The fallback +lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild +path and the queue's fast-lane `Start` handler (which the deferred +start-after-rebuild rides). ### Spawn path (new container) diff --git a/hive-c0re/src/actions.rs b/hive-c0re/src/actions.rs index 041f9d9e..5fc88933 100644 --- a/hive-c0re/src/actions.rs +++ b/hive-c0re/src/actions.rs @@ -862,6 +862,10 @@ async fn deploy_applied_target( agent, &hive, &paths, + // Inline start: the apply-commit flow verifies the agent comes + // back up before finalizing the deploy tag, so the start stays + // part of this entry rather than a deferred fast-lane follow-up. + false, &|step| coord.set_queue_step(queue_entry_id, step), &|log_id| { if let Some(qid) = queue_entry_id @@ -874,7 +878,7 @@ async fn deploy_applied_target( .await; match build_result { - Ok(()) => { + Ok(_) => { coord.set_queue_step(queue_entry_id, "finalize deploy"); let tag = format!("deployed/{id}"); if let Err(e) = lifecycle::git_tag(applied_dir, &tag, target_ref).await { diff --git a/hive-c0re/src/auto_update.rs b/hive-c0re/src/auto_update.rs index db81b452..b2d9ef0a 100644 --- a/hive-c0re/src/auto_update.rs +++ b/hive-c0re/src/auto_update.rs @@ -73,6 +73,13 @@ pub fn agent_config_pending(name: &str, deployed_sha: Option<&str>) -> bool { /// rebuilds, where re-locking would revert the bump the cascade just /// committed (see `lifecycle::rebuild`). /// +/// `defer_start_source` is `Some(source)` for queue-dispatched rebuilds: +/// instead of holding the serialized build lane through the container +/// boot, the start-after-rebuild is enqueued as a fast-lane `Start` +/// entry (grouped under this rebuild via `parent_id`, same split as the +/// graceful-stop follow-up). Pass `None` for direct callers to keep the +/// start inline. +/// /// # Errors /// /// Propagates errors from `coord.ensure_runtime` and `lifecycle::rebuild`. @@ -82,6 +89,7 @@ pub async fn rebuild_agent( current_rev: &str, queue_entry_id: Option, relock: bool, + defer_start_source: Option, ) -> Result<()> { tracing::info!(%name, rev = %current_rev, "rebuild agent"); let agent_dir = coord @@ -99,6 +107,7 @@ pub async fn rebuild_agent( &hive, &paths, relock, + defer_start_source.is_some(), &|step| coord.set_queue_step(queue_entry_id, step), &|log_id| { if let Some(qid) = queue_entry_id @@ -111,10 +120,33 @@ pub async fn rebuild_agent( .await; drop(guard); match &result { - Ok(()) => { + Ok(needs_start) => { if let Err(e) = std::fs::write(rev_marker_path(name), current_rev) { tracing::warn!(%name, error = ?e, "write rev marker failed"); } + // Deferred start: hand the container boot to the fast lane so + // this build-lane entry completes now and the next queued + // rebuild's nix build overlaps with the boot. `parent_id` + // groups the follow-up under this rebuild on the dashboard — + // same split the graceful-stop path uses for its container + // stop. A start failure surfaces on the Start entry (with + // the cold-start fallback) instead of failing the rebuild. + if *needs_start && let Some(source) = defer_start_source { + coord + .rebuild_queue + .enqueue_full(crate::rebuild_queue::FullEnqueue { + kind: crate::rebuild_queue::QueueKind::Start, + agent: name.to_owned(), + source, + reason: format!("start after rebuild of {name}"), + parent_id: queue_entry_id, + inputs: Vec::new(), + approval_id: None, + perm_payload: None, + depends_on: Vec::new(), + }); + coord.emit_rebuild_queue_snapshot(); + } coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt { agent: name.to_owned(), ok: true, @@ -161,7 +193,7 @@ pub async fn rebuild_agent( coord.rescan_containers_and_emit().await; } } - result + result.map(|_| ()) } /// Whether this hive is "ruthless" — running with no root/manager agent at @@ -210,7 +242,7 @@ pub async fn ensure_root_agent(coord: &Arc) -> Result<()> { ); let coord_clone = coord.clone(); if let Err(e) = - rebuild_agent(&coord_clone, MANAGER_NAME, rev.as_str(), None, true).await + rebuild_agent(&coord_clone, MANAGER_NAME, rev.as_str(), None, true, None).await { tracing::warn!(error = ?e, "manager migration rebuild failed"); } diff --git a/hive-c0re/src/lifecycle.rs b/hive-c0re/src/lifecycle.rs index 12d614b0..3296866a 100644 --- a/hive-c0re/src/lifecycle.rs +++ b/hive-c0re/src/lifecycle.rs @@ -372,6 +372,50 @@ pub async fn start(name: &str) -> Result<()> { priv_run("start", name).await } +/// Start with the cold-start fallback: when a plain start fails (the +/// activation-error shape), retry once via stop + kill + start before +/// giving up. Used by the queue's fast-lane `Start` handler and the +/// inline start-after-rebuild path. +/// See `docs/coordinator.md::Cold-start fallback`. +/// +/// # Errors +/// +/// Propagates the retry's start error (annotated with the original +/// failure) when the fallback also fails. +pub async fn start_with_fallback(name: &str) -> Result<()> { + validate(name)?; + if let Err(start_err) = priv_run("start", name).await { + let container = container_name(name); + tracing::warn!( + container = %container, + error = %start_err, + "start failed (possible activation error); retrying via stop + kill + start" + ); + priv_run("stop", name).await.unwrap_or_else(|e| { + tracing::warn!( + container = %container, + error = %e, + "stop before cold-start retry failed (ignored)" + ); + }); + priv_run("kill", name).await.unwrap_or_else(|e| { + tracing::warn!( + container = %container, + error = %e, + "kill before cold-start retry failed (ignored)" + ); + }); + priv_run("start", name).await.map_err(|e| { + anyhow::anyhow!( + "cold-start fallback also failed: {e:#} \ + (original start error: {start_err:#})" + ) + }) + } else { + Ok(()) + } +} + /// Stop + start without regenerating any config. For "kick the container" /// without touching the flake or nspawn flags. pub async fn restart(name: &str) -> Result<()> { @@ -421,14 +465,19 @@ pub async fn destroy(name: &str) -> Result<()> { /// /// Propagates errors from meta-flake sync / lock-update and the /// `nixos-container` apply + restart shellouts. +/// +/// Returns `true` when `defer_start` suppressed the start-after-update — +/// the caller owns bringing the container back up (see +/// [`rebuild_no_meta`]). pub async fn rebuild( name: &str, hive: &HiveEnv, paths: &AgentPaths, relock: bool, + defer_start: bool, on_step: &(dyn Fn(&str) + Send + Sync), on_build_log_id: &(dyn Fn(i64) + Send + Sync), -) -> Result<()> { +) -> Result { // Sync the meta flake (idempotent — no-op when the rendered // flake matches disk) so a manual rebuild from the dashboard // can also recover from a divergent meta repo (e.g. an agent @@ -449,7 +498,7 @@ pub async fn rebuild( if relock { crate::meta::lock_update_for_rebuild(name).await?; } - rebuild_no_meta(name, hive, paths, on_step, on_build_log_id).await + rebuild_no_meta(name, hive, paths, defer_start, on_step, on_build_log_id).await } /// Container-level rebuild without touching the meta repo. Callers @@ -467,13 +516,22 @@ pub async fn rebuild( /// the `nixos-container update` log row opens, before the actual update /// command starts. Callers can use this to link the queue entry to the log /// for live streaming. Pass `&|_| ()` when not needed. +/// +/// `defer_start` skips the start-after-update for a previously-running +/// container and returns `true` instead, so a queue-side caller can hand +/// the (potentially slow) container boot to the fast lane rather than +/// holding the serialized build lane through it. With `defer_start = +/// false` the start (with cold-start fallback) runs inline as before and +/// the return value is always `false`. The spawn path always starts +/// inline — a freshly-created container boots as part of provisioning. pub async fn rebuild_no_meta( name: &str, hive: &HiveEnv, paths: &AgentPaths, + defer_start: bool, on_step: &(dyn Fn(&str) + Send + Sync), on_build_log_id: &(dyn Fn(i64) + Send + Sync), -) -> Result<()> { +) -> Result { validate(name)?; if let Some(other) = port_collision(name).await { bail!( @@ -528,42 +586,16 @@ pub async fn rebuild_no_meta( } update_result?; if was_running { - // Cold-start fallback on activation errors. - // See `docs/coordinator.md::Cold-start fallback`. - on_step("nixos-container start"); - if let Err(start_err) = priv_run("start", name).await { - tracing::warn!( - container = %container, - error = %start_err, - "start after rebuild failed (possible activation error); \ - retrying via stop + kill + start" - ); - priv_run("stop", name).await.unwrap_or_else(|e| { - tracing::warn!( - container = %container, - error = %e, - "stop before cold-start retry failed (ignored)" - ); - }); - priv_run("kill", name).await.unwrap_or_else(|e| { - tracing::warn!( - container = %container, - error = %e, - "kill before cold-start retry failed (ignored)" - ); - }); - priv_run("start", name).await.map_err(|e| { - anyhow::anyhow!( - "cold-start fallback also failed: {e:#} \ - (original start error: {start_err:#})" - ) - }) - } else { - Ok(()) + if defer_start { + // The caller re-queues the start on the fast lane so the + // build lane is freed for the next entry instead of + // waiting out the container boot here. + return Ok(true); } - } else { - Ok(()) + on_step("nixos-container start"); + start_with_fallback(name).await?; } + Ok(false) } else { // Spawn path: create is atomic, no prebuild needed. // See `docs/coordinator.md::Spawn path`. @@ -579,7 +611,8 @@ pub async fn rebuild_no_meta( set_resource_limits(&container, &hive.agent_cpu_quota, &hive.agent_memory_max).await?; systemd_daemon_reload().await?; on_step("nixos-container start"); - priv_run("start", name).await + priv_run("start", name).await?; + Ok(false) } } diff --git a/hive-c0re/src/rebuild_queue.rs b/hive-c0re/src/rebuild_queue.rs index 957ebbc7..8482a838 100644 --- a/hive-c0re/src/rebuild_queue.rs +++ b/hive-c0re/src/rebuild_queue.rs @@ -890,21 +890,12 @@ async fn dispatch( dispatch_rebuild_approval(coord, entry, approval_id).await } (QueueKind::Rebuild, None) => { - let current_rev = - crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default(); // A meta-update cascade has just set the meta lock; re-locking // in the per-agent rebuild would revert it (the agent's own // flake.lock wins). Every other source wants the relock so it // advances to applied//main. let relock = entry.source != QueueSource::MetaUpdate; - crate::auto_update::rebuild_agent( - coord, - &entry.agent, - ¤t_rev, - Some(entry.id), - relock, - ) - .await + rebuild_for_entry(coord, entry, relock).await } (QueueKind::MetaUpdate, Some(approval_id)) => { crate::actions::run_approval_update_meta_inputs(coord, Some(entry.id), approval_id) @@ -1002,9 +993,7 @@ async fn dispatch( } // Now rebuild so the updated HIVE_TOOL_GROUPS / HIVE_CAPABILITIES // env var takes effect in the container. - let current_rev = - crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default(); - crate::auto_update::rebuild_agent(coord, name, ¤t_rev, Some(entry.id), true).await + rebuild_for_entry(coord, entry, true).await } (QueueKind::GracefulStop, _) => { run_graceful_stop(coord, entry); @@ -1015,9 +1004,36 @@ async fn dispatch( } } +/// Queue-side container rebuild for `entry.agent`: resolves the current +/// flake rev and hands off to `rebuild_agent` with the entry's id + +/// source. Passing the source defers the start-after-rebuild to a +/// fast-lane `Start` follow-up (grouped under this entry via +/// `parent_id`), so the build lane is freed for the next entry instead +/// of waiting out the container boot. +async fn rebuild_for_entry( + coord: &std::sync::Arc, + entry: &QueueEntry, + relock: bool, +) -> anyhow::Result<()> { + let current_rev = + crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default(); + crate::auto_update::rebuild_agent( + coord, + &entry.agent, + ¤t_rev, + Some(entry.id), + relock, + Some(entry.source), + ) + .await +} + /// Start a stopped container off the queue (`QueueKind::Start`), with a /// `Starting` transient so the dashboard shows a visible queued→running /// progression rather than the sub-second flash of a direct start. +/// Uses the cold-start fallback (stop + kill + start retry) so the +/// deferred start-after-rebuild keeps the same activation-error recovery +/// it had when it ran inline on the build lane. async fn run_start( coord: &std::sync::Arc, entry: &QueueEntry, @@ -1025,7 +1041,7 @@ async fn run_start( let name = &entry.agent; let _guard = coord.transient_guard(name, crate::coordinator::TransientKind::Starting); coord.set_queue_step(Some(entry.id), "nixos-container start"); - crate::lifecycle::start(name).await?; + crate::lifecycle::start_with_fallback(name).await?; coord.kick_agent(name, "container started"); coord.rescan_containers_and_emit().await; Ok(()) diff --git a/hive-c0re/src/server.rs b/hive-c0re/src/server.rs index 88817f6b..a2c63a82 100644 --- a/hive-c0re/src/server.rs +++ b/hive-c0re/src/server.rs @@ -458,14 +458,14 @@ async fn handle_rebuild(coord: &Arc, name: &str) -> Result { + Ok(_) => { coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt { agent: name.to_owned(), ok: true,