defer start-after-rebuild to the fast lane so the build lane frees up (closes #2094)

docs/coordinator.md

@@ -34,7 +34,7 @@ somewhere."
 
 | Kind | Description |
 |------|-------------|
-| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. |
+| `Rebuild` | Single-agent rebuild. Covers manual, approval-driven, auto-update, and meta-update cascade variants — all funnel through the same path. The start-after-rebuild is **deferred to a fast-lane `Start` follow-up** (`parent_id` = this entry) so the build lane is freed as soon as the profile-swap finishes instead of waiting out the container boot — see *Deferred start* under the rebuild path below. |
 | `MetaUpdate` | `nix flake update` on the meta flake. The worker runs the lock bump itself, then enqueues a cascade of `Rebuild` entries with `parent_id` set to the meta-update's id. |
 | `Spawn` | First-deploy of a new agent (approval-driven). Same serialisation as `Rebuild` from the operator's POV. |
 | `Destroy` | For future use (`destroy --purge` does real I/O). Variant exists so the wire shape doesn't change later; not currently routed through the queue. |
@@ -42,7 +42,7 @@ somewhere."
 | `PermChange` | Write a tool-group or capability change to the shared JSON file (`tool-groups.json` / `capabilities.json`), then rebuild the agent so the updated `HIVE_TOOL_GROUPS` / `HIVE_CAPABILITIES` env var takes effect. Serialising the file write through the queue prevents concurrent dashboard batch-apply actions from racing on the shared file. After a successful file write, emits `CapabilitiesChanged` or `ToolGroupsChanged` SSE snapshot so the P3RM1SS10NS tab updates live. |
 | `GracefulStop` | Quiesce then stop a container (the `?graceful=true` path on `/api/kill/<agent>`). Signals the harness (its next `Recv` returns `GracefulStop` — the inbound fence — so it runs a stop-checkpoint turn that flushes durable `/state`, then takes the normal post-turn compaction path if it crossed the watermark, then exits) and **immediately releases the build lane**, spawning a detached watcher that holds the `Stopping` transient across the drain (bounded by a 3-min timeout → hard-stop fallback) and then enqueues a fast-lane `Stop` (`parent_id` = this entry) for the actual `nixos-container stop`. Net: a whole-hive graceful stop signals every agent up front, drains overlap, and only the container teardowns serialise (on the fast lane). Queued so the signal can't race an in-flight rebuild for the same agent. |
 
-**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill`. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.)
+**Intentionally not queued** (sub-second ops): the *hard* `start`, `stop`, `kill` via the direct API paths. (A *graceful* stop is the `GracefulStop` kind above — it takes a checkpoint turn, so it rides the queue.) The queue's fast lane does carry `Start` / `Stop` kinds, but only as **follow-ups** other entries enqueue for themselves — the graceful-stop teardown and the deferred start-after-rebuild — so the container op groups under its parent entry on the dashboard.
 
 ### Dedup
 
@@ -185,6 +185,19 @@ Sequence for a running container:
 If the container is already stopped, step 1 is skipped (no downtime to shave — no
 point evaluating the flake twice).
 
+**Deferred start (queue-dispatched rebuilds):** step 4 can take a while
+(container boot), and holding the serialized build lane through it delays the
+next queued rebuild's nix build for no reason. Queue-dispatched rebuilds
+therefore pass `defer_start` — `rebuild_no_meta` skips the start and returns
+`true`, and `rebuild_agent` enqueues a fast-lane `Start` entry instead
+(`parent_id` = the rebuild entry, so the dashboard groups the follow-up under
+it — the same split the graceful-stop path uses for its container stop). The
+build-lane entry completes when the profile-swap finishes; a start failure
+surfaces on the `Start` entry, which runs with the cold-start fallback. Direct
+callers (admin-socket CLI, root-agent migration nudge, the apply-commit deploy
+flow which verifies the agent comes back up before finalizing) keep the start
+inline.
+
 ### Cold-start fallback
 
 `start` after `update` can exit non-zero when packages are **removed** between
@@ -194,7 +207,10 @@ half-started at that point.
 
 Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
 → `start` (clean cold-start, no generation transition, new activation runs cleanly).
-Both errors are preserved and surfaced if the cold-start also fails.
+Both errors are preserved and surfaced if the cold-start also fails. The fallback
+lives in `lifecycle::start_with_fallback`, shared by the inline start-after-rebuild
+path and the queue's fast-lane `Start` handler (which the deferred
+start-after-rebuild rides).
 
 ### Spawn path (new container)
 

hive-c0re/src/actions.rs

@@ -862,6 +862,10 @@ async fn deploy_applied_target(
         agent,
         &hive,
         &paths,
+        // Inline start: the apply-commit flow verifies the agent comes
+        // back up before finalizing the deploy tag, so the start stays
+        // part of this entry rather than a deferred fast-lane follow-up.
+        false,
         &|step| coord.set_queue_step(queue_entry_id, step),
         &|log_id| {
             if let Some(qid) = queue_entry_id
@@ -874,7 +878,7 @@ async fn deploy_applied_target(
     .await;
 
     match build_result {
-        Ok(()) => {
+        Ok(_) => {
             coord.set_queue_step(queue_entry_id, "finalize deploy");
             let tag = format!("deployed/{id}");
             if let Err(e) = lifecycle::git_tag(applied_dir, &tag, target_ref).await {

hive-c0re/src/auto_update.rs

@@ -73,6 +73,13 @@ pub fn agent_config_pending(name: &str, deployed_sha: Option<&str>) -> bool {
 /// rebuilds, where re-locking would revert the bump the cascade just
 /// committed (see `lifecycle::rebuild`).
 ///
+/// `defer_start_source` is `Some(source)` for queue-dispatched rebuilds:
+/// instead of holding the serialized build lane through the container
+/// boot, the start-after-rebuild is enqueued as a fast-lane `Start`
+/// entry (grouped under this rebuild via `parent_id`, same split as the
+/// graceful-stop follow-up). Pass `None` for direct callers to keep the
+/// start inline.
+///
 /// # Errors
 ///
 /// Propagates errors from `coord.ensure_runtime` and `lifecycle::rebuild`.
@@ -82,6 +89,7 @@ pub async fn rebuild_agent(
     current_rev: &str,
     queue_entry_id: Option<u64>,
     relock: bool,
+    defer_start_source: Option<crate::rebuild_queue::QueueSource>,
 ) -> Result<()> {
     tracing::info!(%name, rev = %current_rev, "rebuild agent");
     let agent_dir = coord
@@ -99,6 +107,7 @@ pub async fn rebuild_agent(
         &hive,
         &paths,
         relock,
+        defer_start_source.is_some(),
         &|step| coord.set_queue_step(queue_entry_id, step),
         &|log_id| {
             if let Some(qid) = queue_entry_id
@@ -111,10 +120,33 @@ pub async fn rebuild_agent(
     .await;
     drop(guard);
     match &result {
-        Ok(()) => {
+        Ok(needs_start) => {
             if let Err(e) = std::fs::write(rev_marker_path(name), current_rev) {
                 tracing::warn!(%name, error = ?e, "write rev marker failed");
             }
+            // Deferred start: hand the container boot to the fast lane so
+            // this build-lane entry completes now and the next queued
+            // rebuild's nix build overlaps with the boot. `parent_id`
+            // groups the follow-up under this rebuild on the dashboard —
+            // same split the graceful-stop path uses for its container
+            // stop. A start failure surfaces on the Start entry (with
+            // the cold-start fallback) instead of failing the rebuild.
+            if *needs_start && let Some(source) = defer_start_source {
+                coord
+                    .rebuild_queue
+                    .enqueue_full(crate::rebuild_queue::FullEnqueue {
+                        kind: crate::rebuild_queue::QueueKind::Start,
+                        agent: name.to_owned(),
+                        source,
+                        reason: format!("start after rebuild of {name}"),
+                        parent_id: queue_entry_id,
+                        inputs: Vec::new(),
+                        approval_id: None,
+                        perm_payload: None,
+                        depends_on: Vec::new(),
+                    });
+                coord.emit_rebuild_queue_snapshot();
+            }
             coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
                 agent: name.to_owned(),
                 ok: true,
@@ -161,7 +193,7 @@ pub async fn rebuild_agent(
             coord.rescan_containers_and_emit().await;
         }
     }
-    result
+    result.map(|_| ())
 }
 
 /// Whether this hive is "ruthless" — running with no root/manager agent at
@@ -210,7 +242,7 @@ pub async fn ensure_root_agent(coord: &Arc<Coordinator>) -> Result<()> {
             );
             let coord_clone = coord.clone();
             if let Err(e) =
-                rebuild_agent(&coord_clone, MANAGER_NAME, rev.as_str(), None, true).await
+                rebuild_agent(&coord_clone, MANAGER_NAME, rev.as_str(), None, true, None).await
             {
                 tracing::warn!(error = ?e, "manager migration rebuild failed");
             }

hive-c0re/src/lifecycle.rs

@@ -372,6 +372,50 @@ pub async fn start(name: &str) -> Result<()> {
     priv_run("start", name).await
 }
 
+/// Start with the cold-start fallback: when a plain start fails (the
+/// activation-error shape), retry once via stop + kill + start before
+/// giving up. Used by the queue's fast-lane `Start` handler and the
+/// inline start-after-rebuild path.
+/// See `docs/coordinator.md::Cold-start fallback`.
+///
+/// # Errors
+///
+/// Propagates the retry's start error (annotated with the original
+/// failure) when the fallback also fails.
+pub async fn start_with_fallback(name: &str) -> Result<()> {
+    validate(name)?;
+    if let Err(start_err) = priv_run("start", name).await {
+        let container = container_name(name);
+        tracing::warn!(
+            container = %container,
+            error = %start_err,
+            "start failed (possible activation error); retrying via stop + kill + start"
+        );
+        priv_run("stop", name).await.unwrap_or_else(|e| {
+            tracing::warn!(
+                container = %container,
+                error = %e,
+                "stop before cold-start retry failed (ignored)"
+            );
+        });
+        priv_run("kill", name).await.unwrap_or_else(|e| {
+            tracing::warn!(
+                container = %container,
+                error = %e,
+                "kill before cold-start retry failed (ignored)"
+            );
+        });
+        priv_run("start", name).await.map_err(|e| {
+            anyhow::anyhow!(
+                "cold-start fallback also failed: {e:#} \
+                 (original start error: {start_err:#})"
+            )
+        })
+    } else {
+        Ok(())
+    }
+}
+
 /// Stop + start without regenerating any config. For "kick the container"
 /// without touching the flake or nspawn flags.
 pub async fn restart(name: &str) -> Result<()> {
@@ -421,14 +465,19 @@ pub async fn destroy(name: &str) -> Result<()> {
 ///
 /// Propagates errors from meta-flake sync / lock-update and the
 /// `nixos-container` apply + restart shellouts.
+///
+/// Returns `true` when `defer_start` suppressed the start-after-update —
+/// the caller owns bringing the container back up (see
+/// [`rebuild_no_meta`]).
 pub async fn rebuild(
     name: &str,
     hive: &HiveEnv,
     paths: &AgentPaths,
     relock: bool,
+    defer_start: bool,
     on_step: &(dyn Fn(&str) + Send + Sync),
     on_build_log_id: &(dyn Fn(i64) + Send + Sync),
-) -> Result<()> {
+) -> Result<bool> {
     // Sync the meta flake (idempotent — no-op when the rendered
     // flake matches disk) so a manual rebuild from the dashboard
     // can also recover from a divergent meta repo (e.g. an agent
@@ -449,7 +498,7 @@ pub async fn rebuild(
     if relock {
         crate::meta::lock_update_for_rebuild(name).await?;
     }
-    rebuild_no_meta(name, hive, paths, on_step, on_build_log_id).await
+    rebuild_no_meta(name, hive, paths, defer_start, on_step, on_build_log_id).await
 }
 
 /// Container-level rebuild without touching the meta repo. Callers
@@ -467,13 +516,22 @@ pub async fn rebuild(
 /// the `nixos-container update` log row opens, before the actual update
 /// command starts. Callers can use this to link the queue entry to the log
 /// for live streaming. Pass `&|_| ()` when not needed.
+///
+/// `defer_start` skips the start-after-update for a previously-running
+/// container and returns `true` instead, so a queue-side caller can hand
+/// the (potentially slow) container boot to the fast lane rather than
+/// holding the serialized build lane through it. With `defer_start =
+/// false` the start (with cold-start fallback) runs inline as before and
+/// the return value is always `false`. The spawn path always starts
+/// inline — a freshly-created container boots as part of provisioning.
 pub async fn rebuild_no_meta(
     name: &str,
     hive: &HiveEnv,
     paths: &AgentPaths,
+    defer_start: bool,
     on_step: &(dyn Fn(&str) + Send + Sync),
     on_build_log_id: &(dyn Fn(i64) + Send + Sync),
-) -> Result<()> {
+) -> Result<bool> {
     validate(name)?;
     if let Some(other) = port_collision(name).await {
         bail!(
@@ -528,42 +586,16 @@ pub async fn rebuild_no_meta(
         }
         update_result?;
         if was_running {
-            // Cold-start fallback on activation errors.
+            if defer_start {
-            // See `docs/coordinator.md::Cold-start fallback`.
+                // The caller re-queues the start on the fast lane so the
-            on_step("nixos-container start");
+                // build lane is freed for the next entry instead of
-            if let Err(start_err) = priv_run("start", name).await {
+                // waiting out the container boot here.
-                tracing::warn!(
+                return Ok(true);
-                    container = %container,
-                    error = %start_err,
-                    "start after rebuild failed (possible activation error); \
-                     retrying via stop + kill + start"
-                );
-                priv_run("stop", name).await.unwrap_or_else(|e| {
-                    tracing::warn!(
-                        container = %container,
-                        error = %e,
-                        "stop before cold-start retry failed (ignored)"
-                    );
-                });
-                priv_run("kill", name).await.unwrap_or_else(|e| {
-                    tracing::warn!(
-                        container = %container,
-                        error = %e,
-                        "kill before cold-start retry failed (ignored)"
-                    );
-                });
-                priv_run("start", name).await.map_err(|e| {
-                    anyhow::anyhow!(
-                        "cold-start fallback also failed: {e:#} \
-                         (original start error: {start_err:#})"
-                    )
-                })
-            } else {
-                Ok(())
             }
-        } else {
+            on_step("nixos-container start");
-            Ok(())
+            start_with_fallback(name).await?;
         }
+        Ok(false)
     } else {
         // Spawn path: create is atomic, no prebuild needed.
         // See `docs/coordinator.md::Spawn path`.
@@ -579,7 +611,8 @@ pub async fn rebuild_no_meta(
         set_resource_limits(&container, &hive.agent_cpu_quota, &hive.agent_memory_max).await?;
         systemd_daemon_reload().await?;
         on_step("nixos-container start");
-        priv_run("start", name).await
+        priv_run("start", name).await?;
+        Ok(false)
     }
 }
 

hive-c0re/src/rebuild_queue.rs

@@ -890,21 +890,12 @@ async fn dispatch(
             dispatch_rebuild_approval(coord, entry, approval_id).await
         }
         (QueueKind::Rebuild, None) => {
-            let current_rev =
-                crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default();
             // A meta-update cascade has just set the meta lock; re-locking
             // in the per-agent rebuild would revert it (the agent's own
             // flake.lock wins). Every other source wants the relock so it
             // advances to applied/<n>/main.
             let relock = entry.source != QueueSource::MetaUpdate;
-            crate::auto_update::rebuild_agent(
+            rebuild_for_entry(coord, entry, relock).await
-                coord,
-                &entry.agent,
-                &current_rev,
-                Some(entry.id),
-                relock,
-            )
-            .await
         }
         (QueueKind::MetaUpdate, Some(approval_id)) => {
             crate::actions::run_approval_update_meta_inputs(coord, Some(entry.id), approval_id)
@@ -1002,9 +993,7 @@ async fn dispatch(
             }
             // Now rebuild so the updated HIVE_TOOL_GROUPS / HIVE_CAPABILITIES
             // env var takes effect in the container.
-            let current_rev =
+            rebuild_for_entry(coord, entry, true).await
-                crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default();
-            crate::auto_update::rebuild_agent(coord, name, &current_rev, Some(entry.id), true).await
         }
         (QueueKind::GracefulStop, _) => {
             run_graceful_stop(coord, entry);
@@ -1015,9 +1004,36 @@ async fn dispatch(
     }
 }
 
+/// Queue-side container rebuild for `entry.agent`: resolves the current
+/// flake rev and hands off to `rebuild_agent` with the entry's id +
+/// source. Passing the source defers the start-after-rebuild to a
+/// fast-lane `Start` follow-up (grouped under this entry via
+/// `parent_id`), so the build lane is freed for the next entry instead
+/// of waiting out the container boot.
+async fn rebuild_for_entry(
+    coord: &std::sync::Arc<crate::coordinator::Coordinator>,
+    entry: &QueueEntry,
+    relock: bool,
+) -> anyhow::Result<()> {
+    let current_rev =
+        crate::auto_update::current_flake_rev(&coord.hyperhive_flake).unwrap_or_default();
+    crate::auto_update::rebuild_agent(
+        coord,
+        &entry.agent,
+        &current_rev,
+        Some(entry.id),
+        relock,
+        Some(entry.source),
+    )
+    .await
+}
+
 /// Start a stopped container off the queue (`QueueKind::Start`), with a
 /// `Starting` transient so the dashboard shows a visible queued→running
 /// progression rather than the sub-second flash of a direct start.
+/// Uses the cold-start fallback (stop + kill + start retry) so the
+/// deferred start-after-rebuild keeps the same activation-error recovery
+/// it had when it ran inline on the build lane.
 async fn run_start(
     coord: &std::sync::Arc<crate::coordinator::Coordinator>,
     entry: &QueueEntry,
@@ -1025,7 +1041,7 @@ async fn run_start(
     let name = &entry.agent;
     let _guard = coord.transient_guard(name, crate::coordinator::TransientKind::Starting);
     coord.set_queue_step(Some(entry.id), "nixos-container start");
-    crate::lifecycle::start(name).await?;
+    crate::lifecycle::start_with_fallback(name).await?;
     coord.kick_agent(name, "container started");
     coord.rescan_containers_and_emit().await;
     Ok(())

hive-c0re/src/server.rs

@@ -458,14 +458,14 @@ async fn handle_rebuild(coord: &Arc<Coordinator>, name: &str) -> Result<HostResp
     let agent_dir = coord.ensure_runtime(name)?;
     let hive = coord.hive_env();
     let paths = Coordinator::agent_paths(name, agent_dir);
-    let result = lifecycle::rebuild(name, &hive, &paths, true, &|_| (), &|_| ()).await;
+    let result = lifecycle::rebuild(name, &hive, &paths, true, false, &|_| (), &|_| ()).await;
     // Mirror auto_update::rebuild_agent — the manager wants to know
     // about every rebuild attempt regardless of which surface triggered
     // it, especially failures (build error → manager can adjust the
     // agent's agent.nix). Without this the admin-socket CLI was a
     // notify-gap.
     match &result {
-        Ok(()) => {
+        Ok(_) => {
             coord.notify_manager(&hive_sh4re::HelperEvent::Rebuilt {
                 agent: name.to_owned(),
                 ok: true,