feat: pause an agent's turn loop without stopping its container
A paused agent keeps its container, its claude session and its dashboard/todo servers up, but stops driving turns. Messages queue unacked and are drained on resume. The whole protocol is a single marker file, `<harness>/paused`. That directory is already a bind-mount shared between host and container, so both sides just stat the same path: the harness reads it to decide whether to drive a turn, hive-c0re reads it to render the badge and writes/removes it for `hivectl pause|resume`. No new wire protocol, no container round-trip, and it is sticky across restarts by construction. Not calling `recv_next` while paused *is* the queueing semantic, so there is no fencing to get wrong: reminders buffer in their unbounded channel, the todo `Notify` permit coalesces, and a `request_next_turn` that raced the pause survives because the gate sits above `self_continue.take()`. Graceful stop is handled host-side rather than in the harness: a paused agent provably has no turn in flight, so `run_signal` skips the fence entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for a checkpoint turn that will never run. `paused` is reported on `ContainerView` / `AgentStatusRow` for the dashboard, orthogonal to `running` and reported for stopped containers too. Closes: hyperhive/hyperhive issue 2271
This commit is contained in:
parent
1356a1f049
commit
31008c83df
14 changed files with 305 additions and 1 deletions
|
|
@ -13,6 +13,15 @@ use serde::Serialize;
|
|||
use crate::coordinator::Coordinator;
|
||||
use crate::lifecycle::{self, AGENT_PREFIX};
|
||||
|
||||
// Independent per-agent flags, each its own badge on the dashboard card
|
||||
// and each diffed separately by `rescan_containers_and_emit`. Grouping
|
||||
// them into nested structs would need `serde(flatten)` to keep the JSON
|
||||
// the frontend already consumes, for no readability gain — same
|
||||
// rationale as `LifecycleScope` in hive-host-sock.
|
||||
#[allow(
|
||||
clippy::struct_excessive_bools,
|
||||
reason = "flat wire projection of independent per-agent flags"
|
||||
)]
|
||||
#[derive(Serialize, Clone, PartialEq, Eq, Debug)]
|
||||
pub struct ContainerView {
|
||||
/// Logical agent name (no `h-` prefix). Used in action URLs.
|
||||
|
|
@ -44,6 +53,15 @@ pub struct ContainerView {
|
|||
/// for stopped containers.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub active_model: Option<String>,
|
||||
/// The agent's turn loop is parked: the container is up and still
|
||||
/// serving its web UI / MCP daemons, but it drives no turns and its
|
||||
/// inbox messages queue unacked until it resumes. Sourced from the
|
||||
/// pause marker in the harness dir (`Coordinator::is_paused`), so it
|
||||
/// stays true across a container restart — and, unlike `needs_login`,
|
||||
/// is reported for stopped containers too: pausing a stopped agent is
|
||||
/// a legitimate way to keep it idle when it next boots.
|
||||
#[serde(default)]
|
||||
pub paused: bool,
|
||||
}
|
||||
|
||||
/// Build the full container list. Wraps `lifecycle::list()` and
|
||||
|
|
@ -89,6 +107,7 @@ pub async fn build_all() -> Vec<ContainerView> {
|
|||
} else {
|
||||
None
|
||||
};
|
||||
let paused = Coordinator::is_paused(&logical);
|
||||
out.push(ContainerView {
|
||||
port: lifecycle::agent_web_port(logical.as_str()),
|
||||
running,
|
||||
|
|
@ -99,6 +118,7 @@ pub async fn build_all() -> Vec<ContainerView> {
|
|||
deployed_sha,
|
||||
parent,
|
||||
active_model,
|
||||
paused,
|
||||
});
|
||||
}
|
||||
out
|
||||
|
|
|
|||
|
|
@ -1444,6 +1444,50 @@ impl Coordinator {
|
|||
crate::paths::agent_state_dir(name).join("harness")
|
||||
}
|
||||
|
||||
/// Host-side path of the pause marker — the same file the harness
|
||||
/// resolves in-container via `hive_sh4re::paths::paused_marker`,
|
||||
/// reached through the harness bind-mount. Its presence means the
|
||||
/// agent's turn loop is parked: the harness still serves its web UI
|
||||
/// and MCP daemons, but drives no turns, so inbox messages queue up
|
||||
/// unacked until the marker is removed.
|
||||
///
|
||||
/// Both sides only ever *stat* or create/remove this file, so there
|
||||
/// is no protocol between them and pause survives a container
|
||||
/// restart (and can be set on a stopped container).
|
||||
pub fn agent_paused_marker(name: &hive_types::Ident) -> PathBuf {
|
||||
Self::agent_harness_dir(name).join(hive_sh4re::paths::PAUSED_MARKER_FILE)
|
||||
}
|
||||
|
||||
/// Whether `name` is currently paused. A stat error (missing agent
|
||||
/// dir, permissions) reads as "not paused" — the pause indicator is
|
||||
/// advisory on the host side, and the harness is the component that
|
||||
/// actually enforces it.
|
||||
#[must_use]
|
||||
pub fn is_paused(name: &hive_types::Ident) -> bool {
|
||||
Self::agent_paused_marker(name).exists()
|
||||
}
|
||||
|
||||
/// Create or remove the pause marker. Idempotent in both
|
||||
/// directions: pausing an already-paused agent (or resuming a
|
||||
/// running one) is a no-op rather than an error, so the dashboard
|
||||
/// toggle and `hivectl pause|resume` don't have to read-then-write.
|
||||
pub fn set_paused(name: &hive_types::Ident, paused: bool) -> std::io::Result<()> {
|
||||
let marker = Self::agent_paused_marker(name);
|
||||
if paused {
|
||||
if let Some(parent) = marker.parent() {
|
||||
std::fs::create_dir_all(parent)?;
|
||||
}
|
||||
// `create_new` would fail on the second pause; truncating an
|
||||
// existing empty marker is the idempotent equivalent.
|
||||
std::fs::write(&marker, b"")
|
||||
} else {
|
||||
match std::fs::remove_file(&marker) {
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Enumerate names that have a persistent state dir under
|
||||
/// `/var/lib/hyperhive/agents/` (i.e. config / claude creds /
|
||||
/// notes survive). Includes both currently-existing containers and
|
||||
|
|
|
|||
|
|
@ -497,7 +497,18 @@ async fn run_stop_for_update(
|
|||
|
||||
/// Set the graceful fence + kick so the harness sees it promptly and
|
||||
/// runs its one stop-checkpoint turn.
|
||||
///
|
||||
/// Skipped entirely for a paused agent: its loop parks on the pause
|
||||
/// marker without polling the broker, so it would never observe the
|
||||
/// fence and the downstream drain would just burn
|
||||
/// `GRACEFUL_STOP_TIMEOUT`. Safe because the harness tests the marker at
|
||||
/// the top of its loop — a paused agent has no turn in flight, so there
|
||||
/// is nothing to checkpoint.
|
||||
fn run_signal(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> NodeOutput {
|
||||
if hive_types::Ident::parse(&claim.agent).is_ok_and(|a| Coordinator::is_paused(&a)) {
|
||||
ctx.step("graceful stop: agent paused, nothing to drain");
|
||||
return NodeOutput::default();
|
||||
}
|
||||
ctx.step("graceful stop: signalling agent");
|
||||
coord.mark_graceful_stop(&claim.agent);
|
||||
coord.kick_agent(&claim.agent, "graceful stop requested");
|
||||
|
|
|
|||
|
|
@ -101,6 +101,9 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
|
|||
HostRequest::Restart { name } => {
|
||||
submit_single(&coord, name.as_str(), Verb::Restart).await
|
||||
}
|
||||
HostRequest::SetPaused { name, paused } => {
|
||||
handle_set_paused(&coord, name, *paused).await
|
||||
}
|
||||
HostRequest::RestartAll => handle_restart_all(&coord).await?,
|
||||
HostRequest::RestartScoped { scope, graceful } => {
|
||||
handle_restart_scoped(&coord, scope, *graceful).await?
|
||||
|
|
@ -297,6 +300,29 @@ async fn handle_spawn(coord: &Arc<Coordinator>, name: &str) -> Result<HostRespon
|
|||
Ok(HostResponse::success())
|
||||
}
|
||||
|
||||
/// `hivectl pause|resume` / the dashboard toggle: write or remove the
|
||||
/// agent's pause marker.
|
||||
///
|
||||
/// Deliberately not a lifecycle DAG. There's no container operation to
|
||||
/// sequence — it's one marker file, and the harness picks it up on its
|
||||
/// next poll — so queueing it would only add latency and a lease. That
|
||||
/// also means it works on a stopped agent: the marker is sticky, so the
|
||||
/// agent comes up paused.
|
||||
async fn handle_set_paused(
|
||||
coord: &std::sync::Arc<Coordinator>,
|
||||
name: &hive_types::Ident,
|
||||
paused: bool,
|
||||
) -> HostResponse {
|
||||
if let Err(e) = Coordinator::set_paused(name, paused) {
|
||||
return HostResponse::error(format!("set paused={paused} for {name}: {e}"));
|
||||
}
|
||||
tracing::info!(%name, paused, "agent pause marker updated");
|
||||
// Refresh the dashboard's view so the paused badge flips without
|
||||
// waiting for the next periodic rescan.
|
||||
coord.rescan_containers_and_emit().await;
|
||||
HostResponse::success()
|
||||
}
|
||||
|
||||
/// Collect per-agent status rows for `hivectl status` and the dashboard.
|
||||
async fn handle_agent_status() -> HostResponse {
|
||||
let rows = crate::container_view::build_all()
|
||||
|
|
@ -316,6 +342,7 @@ async fn handle_agent_status() -> HostResponse {
|
|||
// whether to drop the column entirely.
|
||||
pending_reminders: 0,
|
||||
parent: v.parent,
|
||||
paused: v.paused,
|
||||
})
|
||||
.collect();
|
||||
HostResponse::agent_statuses(rows)
|
||||
|
|
|
|||
|
|
@ -252,6 +252,7 @@ mod tests {
|
|||
deployed_sha: None,
|
||||
parent: None,
|
||||
active_model: None,
|
||||
paused: false,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue