vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: pause an agent's turn loop without stopping its container

Browse source 
A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.

The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.

Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.

Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.

`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.

Closes: hyperhive/hyperhive issue 2271
This commit is contained in:
atlas 2026-07-26 02:31:54 +02:00 • committed by mara
commit 31008c83df
14 changed files with 305 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

24
docs/persistence.md
View file

@ -200,6 +200,30 @@ selected for this agent (default `haiku` when absent). Written by
`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
Survives destroy/recreate, gone on `--purge`.
### `/harness/paused` (per agent)
Empty marker file. Its presence parks the agent's turn loop: the
harness keeps serving its web UI and MCP daemons but drives no turns,
and inbox messages queue unacked until it's removed (see
[turn loop](turn-loop.md#the-loop)).
Unusually, it's read and written from **both** sides of the harness
bind-mount, and that's the whole design: the harness stats it
in-container via `hive_sh4re::paths::paused_marker`, while hive-c0re
stats it on the host (`Coordinator::is_paused`) to populate the
`paused` field on the agent card, and creates/removes it
(`Coordinator::set_paused`) for `hivectl agents pause|resume` and the
dashboard toggle. Because the file itself is the only shared state
there's no protocol between them, no round-trip into the container, and
pause keeps working when the harness is wedged or the container is
stopped.
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
is the agent's own space to fill, and this is harness control state.
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
back paused after a restart, which is the intended behaviour rather
than an accident of storage.
## State dirs (per agent)
Under `/var/lib/hyperhive/agents/<name>/`:

25
docs/tools/hivectl.md
View file

@ -139,11 +139,14 @@ hivectl agents list # roster: every agent's status + technical st
hivectl agents list --json # same data as raw JSON rows (for scripting)
hivectl agents restart iris # stop + start the `iris` container (no rebuild)
hivectl agents restart-all # stop + start every managed agent container in sequence
hivectl agents pause iris # park iris's turn loop, leave the container running
hivectl agents resume iris # let it drive turns again, draining what queued up
```
`list` prints a padded table with one row per managed agent —
`NAME STATUS REV PARENT REMIND`. STATUS collapses the health flags
(`running` / `stopped`, plus ` needs-login` / ` needs-update` when set);
(`running` / `stopped`, plus ` paused` / ` needs-login` / ` needs-update`
when set — `paused` is orthogonal to running, see below);
REV is the first 12 chars of the agent's locked config sha; PARENT is its
place in the topology tree (`-` for a root agent); REMIND is the count of
pending reminders. It reuses the same per-agent aggregation the dashboard
@ -155,6 +158,26 @@ when you need to kick a container from the host without going through
the agent hierarchy. Failures on `restart-all` are collected and
reported at the end rather than aborting mid-run.
`pause` / `resume` are the "stop burning tokens without losing the
container" pair. Pausing writes a marker file into the agent's harness
dir (`<state>/<name>/harness/paused`) which the harness re-stats every
5 s at the top of its serve loop; while it's there the agent drives no
turns, but the container, its mounts, its warm caches, its web UI and
its MCP daemons all stay up. Inbox messages queue **unacked**, so a
resume drains the backlog rather than dropping it. Points worth knowing:
- **Sticky.** The marker lives on the persistent harness mount, so a
paused agent stays paused across a container restart — and pausing a
*stopped* agent makes it come up parked.
- **Not a DAG.** Unlike `restart`/`stop`, there's no container operation
to sequence, so it applies immediately with nothing to wait on.
- **Stopping a paused agent is still fast.** The graceful-stop
handshake is skipped for a paused agent (it would never answer), which
is safe precisely because the pause check sits at the top of the loop:
a paused agent has no turn in flight to checkpoint.
- Visible as ` paused` in `agents list`'s STATUS column, as a `paused`
field on the JSON rows, and as a badge on the dashboard card.
## Choom
Drop into an interactive Claude session inside an agent container.

9
docs/turn-loop.md
View file

@ -8,6 +8,15 @@ claude has access to in return.
Each agent harness (`hive-agent` — one serve-loop binary for all
agents) runs:
0. Check the pause marker (`<harness>/paused`). While it exists the
loop does nothing but re-stat it every 5 s — no broker poll, no
claude process. Because step 1 is never reached, messages stay
queued and unacked, so a resume drains the backlog instead of
losing it; reminders and todo wakes buffer in their channels. The
check runs before the self-continue slot is consumed, so a pending
`request_next_turn` survives the pause. Set it with
`hivectl agents pause <name>` or the dashboard toggle; see
[persistence](persistence.md#-harnesspaused-per-agent).
1. Long-poll `Recv` on its socket. The host-side broker
(`broker.rs::recv_blocking_batch`) returns immediately if there's
a pending message, otherwise waits up to 30 s for a broker `Sent`

44
hive-agent/src/main.rs
View file

@ -40,6 +40,12 @@ const DEFAULT_SOCKET: &str = "/run/hive/mcp.sock";
/// Default web UI port — used when `HIVE_PORT` env is unset.
const DEFAULT_WEB_PORT: u16 = 8042;
/// How often the serve loop re-stats the pause marker while parked.
/// Only paid while an agent is actually paused, and only against the
/// local harness dir, so a tight-ish interval is cheap and keeps
/// `hivectl resume` feeling immediate.
const PAUSE_POLL: Duration = Duration::from_secs(5);
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use std::time::Duration;
@ -659,7 +665,45 @@ async fn serve_loop<S: Surface>(
// in-process instead of long-polling the broker. Never
// persisted: it lives entirely in this loop's stack.
let mut self_continue: Option<hive_sh4re::DeliveredMessage> = None;
// Tracks the last observed pause state so the transitions get logged
// once each instead of twelve lines a minute while parked.
let mut was_paused = false;
loop {
// Pause gate. While the marker is present this loop drives no
// turns at all — deliberately *before* the `self_continue.take()`
// below, so a `request_next_turn` that raced the pause is still
// waiting when the agent resumes rather than being consumed by
// a turn that never runs.
//
// Nothing here touches the broker: not calling `S::recv_next` is
// exactly the "messages queue unacked, resume drains the
// backlog" semantic, with no fencing and nothing to requeue.
// Reminders (unbounded channel) and todo wakes (a `Notify`
// permit) buffer on their own. The web UI and MCP daemons run as
// separate tasks, so the agent stays inspectable while parked.
//
// A `GracefulStop` can't be observed while parked, and doesn't
// need to be: hive-c0re skips the stop-checkpoint handshake for
// a paused agent, because this check sits at the top of the loop
// and so a paused agent provably has no turn in flight.
if hive_sh4re::paths::paused_marker().exists() {
if !was_paused {
tracing::info!("pause marker present — parking the turn loop");
bus.emit(LiveEvent::Note {
text: "paused: turn loop parked, messages will queue".into(),
});
was_paused = true;
}
tokio::time::sleep(PAUSE_POLL).await;
continue;
}
if was_paused {
tracing::info!("pause marker cleared — resuming the turn loop");
bus.emit(LiveEvent::Note {
text: "resumed: draining whatever queued while paused".into(),
});
was_paused = false;
}
let next = match self_continue.take() {
Some(msg) => msg,
None => match {

20
hive-c0re/src/container_view.rs
View file

@ -13,6 +13,15 @@ use serde::Serialize;
use crate::coordinator::Coordinator;
use crate::lifecycle::{self, AGENT_PREFIX};
// Independent per-agent flags, each its own badge on the dashboard card
// and each diffed separately by `rescan_containers_and_emit`. Grouping
// them into nested structs would need `serde(flatten)` to keep the JSON
// the frontend already consumes, for no readability gain — same
// rationale as `LifecycleScope` in hive-host-sock.
#[allow(
clippy::struct_excessive_bools,
reason = "flat wire projection of independent per-agent flags"
)]
#[derive(Serialize, Clone, PartialEq, Eq, Debug)]
pub struct ContainerView {
/// Logical agent name (no `h-` prefix). Used in action URLs.
@ -44,6 +53,15 @@ pub struct ContainerView {
/// for stopped containers.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub active_model: Option<String>,
/// The agent's turn loop is parked: the container is up and still
/// serving its web UI / MCP daemons, but it drives no turns and its
/// inbox messages queue unacked until it resumes. Sourced from the
/// pause marker in the harness dir (`Coordinator::is_paused`), so it
/// stays true across a container restart — and, unlike `needs_login`,
/// is reported for stopped containers too: pausing a stopped agent is
/// a legitimate way to keep it idle when it next boots.
#[serde(default)]
pub paused: bool,
}
/// Build the full container list. Wraps `lifecycle::list()` and
@ -89,6 +107,7 @@ pub async fn build_all() -> Vec<ContainerView> {
} else {
None
};
let paused = Coordinator::is_paused(&logical);
out.push(ContainerView {
port: lifecycle::agent_web_port(logical.as_str()),
running,
@ -99,6 +118,7 @@ pub async fn build_all() -> Vec<ContainerView> {
deployed_sha,
parent,
active_model,
paused,
});
}
out

44
hive-c0re/src/coordinator.rs
View file

@ -1444,6 +1444,50 @@ impl Coordinator {
crate::paths::agent_state_dir(name).join("harness")
}
/// Host-side path of the pause marker — the same file the harness
/// resolves in-container via `hive_sh4re::paths::paused_marker`,
/// reached through the harness bind-mount. Its presence means the
/// agent's turn loop is parked: the harness still serves its web UI
/// and MCP daemons, but drives no turns, so inbox messages queue up
/// unacked until the marker is removed.
///
/// Both sides only ever *stat* or create/remove this file, so there
/// is no protocol between them and pause survives a container
/// restart (and can be set on a stopped container).
pub fn agent_paused_marker(name: &hive_types::Ident) -> PathBuf {
Self::agent_harness_dir(name).join(hive_sh4re::paths::PAUSED_MARKER_FILE)
}
/// Whether `name` is currently paused. A stat error (missing agent
/// dir, permissions) reads as "not paused" — the pause indicator is
/// advisory on the host side, and the harness is the component that
/// actually enforces it.
#[must_use]
pub fn is_paused(name: &hive_types::Ident) -> bool {
Self::agent_paused_marker(name).exists()
}
/// Create or remove the pause marker. Idempotent in both
/// directions: pausing an already-paused agent (or resuming a
/// running one) is a no-op rather than an error, so the dashboard
/// toggle and `hivectl pause|resume` don't have to read-then-write.
pub fn set_paused(name: &hive_types::Ident, paused: bool) -> std::io::Result<()> {
let marker = Self::agent_paused_marker(name);
if paused {
if let Some(parent) = marker.parent() {
std::fs::create_dir_all(parent)?;
}
// `create_new` would fail on the second pause; truncating an
// existing empty marker is the idempotent equivalent.
std::fs::write(&marker, b"")
} else {
match std::fs::remove_file(&marker) {
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
other => other,
}
}
}
/// Enumerate names that have a persistent state dir under
/// `/var/lib/hyperhive/agents/` (i.e. config / claude creds /
/// notes survive). Includes both currently-existing containers and

11
hive-c0re/src/job_queue/exec.rs
View file

@ -497,7 +497,18 @@ async fn run_stop_for_update(
/// Set the graceful fence + kick so the harness sees it promptly and
/// runs its one stop-checkpoint turn.
///
/// Skipped entirely for a paused agent: its loop parks on the pause
/// marker without polling the broker, so it would never observe the
/// fence and the downstream drain would just burn
/// `GRACEFUL_STOP_TIMEOUT`. Safe because the harness tests the marker at
/// the top of its loop — a paused agent has no turn in flight, so there
/// is nothing to checkpoint.
fn run_signal(coord: &Arc<Coordinator>, claim: &Claim, ctx: &Ctx<'_>) -> NodeOutput {
if hive_types::Ident::parse(&claim.agent).is_ok_and(|a| Coordinator::is_paused(&a)) {
ctx.step("graceful stop: agent paused, nothing to drain");
return NodeOutput::default();
}
ctx.step("graceful stop: signalling agent");
coord.mark_graceful_stop(&claim.agent);
coord.kick_agent(&claim.agent, "graceful stop requested");

27
hive-c0re/src/server.rs
View file

@ -101,6 +101,9 @@ async fn dispatch(req: &HostRequest, coord: Arc<Coordinator>) -> HostResponse {
HostRequest::Restart { name } => {
submit_single(&coord, name.as_str(), Verb::Restart).await
}
HostRequest::SetPaused { name, paused } => {
handle_set_paused(&coord, name, *paused).await
}
HostRequest::RestartAll => handle_restart_all(&coord).await?,
HostRequest::RestartScoped { scope, graceful } => {
handle_restart_scoped(&coord, scope, *graceful).await?
@ -297,6 +300,29 @@ async fn handle_spawn(coord: &Arc<Coordinator>, name: &str) -> Result<HostRespon
Ok(HostResponse::success())
}
/// `hivectl pause|resume` / the dashboard toggle: write or remove the
/// agent's pause marker.
///
/// Deliberately not a lifecycle DAG. There's no container operation to
/// sequence — it's one marker file, and the harness picks it up on its
/// next poll — so queueing it would only add latency and a lease. That
/// also means it works on a stopped agent: the marker is sticky, so the
/// agent comes up paused.
async fn handle_set_paused(
coord: &std::sync::Arc<Coordinator>,
name: &hive_types::Ident,
paused: bool,
) -> HostResponse {
if let Err(e) = Coordinator::set_paused(name, paused) {
return HostResponse::error(format!("set paused={paused} for {name}: {e}"));
}
tracing::info!(%name, paused, "agent pause marker updated");
// Refresh the dashboard's view so the paused badge flips without
// waiting for the next periodic rescan.
coord.rescan_containers_and_emit().await;
HostResponse::success()
}
/// Collect per-agent status rows for `hivectl status` and the dashboard.
async fn handle_agent_status() -> HostResponse {
let rows = crate::container_view::build_all()
@ -316,6 +342,7 @@ async fn handle_agent_status() -> HostResponse {
// whether to drop the column entirely.
pending_reminders: 0,
parent: v.parent,
paused: v.paused,
})
.collect();
HostResponse::agent_statuses(rows)

1
hive-c0re/src/stats/host_stats.rs
View file

@ -252,6 +252,7 @@ mod tests {
deployed_sha: None,
parent: None,
active_model: None,
paused: false,
}
}

13
hive-host-sock/src/lib.rs
View file

@ -93,6 +93,19 @@ pub enum HostRequest {
/// For "kick the container" operations that don't touch the flake or
/// nspawn flags. Mirrors `lifecycle::restart` (kill + start).
Restart { name: Ident },
/// Park (or un-park) an agent's turn loop without touching its
/// container: `hivectl pause|resume <name>` and the dashboard
/// toggle. Writes/removes the marker file the harness polls, so the
/// container stays up and keeps serving its web UI and MCP daemons
/// while burning no tokens. Inbox messages queue unacked and the
/// backlog drains on resume. Not a lifecycle DAG — it's a single
/// marker write, so it applies immediately and works on a stopped
/// container too (the pause is sticky and takes effect at next boot).
SetPaused {
name: Ident,
/// `true` pauses, `false` resumes. Idempotent either way.
paused: bool,
},
/// Stop and restart all managed containers in sequence. Convenience
/// wrapper for `hivectl agents restart-all`; iterates the live
/// container list and restarts each one.

18
hive-sh4re/src/lib.rs
View file

@ -366,6 +366,18 @@ pub struct ContainerInfo {
/// projection of the dashboard's per-agent `ContainerView`. Carries the
/// agent's running/health flags plus the technical state an operator
/// wants in a roster overview (`hivectl agents list`).
//
// Four orthogonal, independently-observed facts about one agent, each
// rendered as its own column/token by `hivectl agents list` and read
// individually by `--json` consumers. Any combination is meaningful
// (a stopped agent can be paused and need an update), so folding them
// into a state machine or nested flag structs would only add
// `serde(flatten)` indirection to preserve the same flat JSON. Same
// rationale as `LifecycleScope` in hive-host-sock.
#[allow(
clippy::struct_excessive_bools,
reason = "flat wire projection of independent per-agent flags"
)]
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AgentStatusRow {
/// Logical agent name (no `h-` prefix).
@ -385,6 +397,12 @@ pub struct AgentStatusRow {
/// Count of this agent's pending reminders.
#[serde(default)]
pub pending_reminders: u64,
/// The agent's turn loop is parked (pause marker present in its
/// harness dir): the container may well be up and serving, it just
/// drives no turns. Orthogonal to `running` — an agent can be
/// paused while stopped, and pause survives a restart.
#[serde(default)]
pub paused: bool,
/// Parent in the topology tree. `None` marks a root-level agent.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub parent: Option<String>,

23
hive-sh4re/src/paths.rs
View file

@ -28,3 +28,26 @@ pub fn harness_dir() -> PathBuf {
let label = std::env::var("HIVE_LABEL").unwrap_or_default();
PathBuf::from(format!("/agents/{label}/harness"))
}
/// File name of the pause marker inside the harness dir. Shared so the
/// in-container resolver below and hive-c0re's host-side one (which
/// builds the same path from `/var/lib/hyperhive/agents/{name}/harness`)
/// cannot drift apart.
pub const PAUSED_MARKER_FILE: &str = "paused";
/// Marker file whose presence means "this agent is paused": the harness
/// keeps serving its web UI and MCP daemons but drives no turns, so
/// inbox messages queue up unacked until it's removed.
///
/// It lives in the harness dir rather than `state/` because `state/` is
/// the agent's own scratch space — this is harness control state. The
/// harness dir is bind-mounted from the host, so the marker is the
/// single source of truth for both sides: the harness stats it to gate
/// the turn loop, and hive-c0re stats it to render the paused
/// indicator and creates/removes it for `hivectl pause|resume`. Being a
/// plain file, it survives container restarts — pause is sticky by
/// construction, and works even when the harness isn't running.
#[must_use]
pub fn paused_marker() -> PathBuf {
harness_dir().join(PAUSED_MARKER_FILE)
}

31
hivectl/src/agents.rs
View file

@ -57,6 +57,9 @@ async fn agents_list(socket: &Path, json: bool) -> Result<()> {
// the common case (`running`) stays short and anomalies stand out.
let status_of = |r: &hive_sh4re::AgentStatusRow| -> String {
let mut s = if r.running { "running" } else { "stopped" }.to_owned();
if r.paused {
s.push_str(" paused");
}
if r.needs_login {
s.push_str(" needs-login");
}
@ -127,6 +130,32 @@ async fn agents_restart_all(socket: &Path, no_wait: bool) -> Result<()> {
wait_for_dags(socket, resp.queued_dags.unwrap_or_default(), no_wait).await
}
/// `hivectl agents pause|resume` — flip the agent's pause marker. Not a
/// DAG, so there's nothing to wait on: the daemon writes the marker and
/// the harness picks it up on its next poll.
async fn set_paused(socket: &Path, name: &str, paused: bool) -> Result<()> {
let resp = crate::client::request(
socket,
HostRequest::SetPaused {
name: crate::util::parse_ident(name)?,
paused,
},
)
.await
.with_context(|| format!("connect to daemon socket {}", socket.display()))?;
if resp.ok {
let verb = if paused { "paused" } else { "resumed" };
println!("{verb}: {name}");
Ok(())
} else {
bail!(
"{} {name}: {}",
if paused { "pause" } else { "resume" },
resp.error.as_deref().unwrap_or("unknown error")
)
}
}
/// Dispatch `hivectl agents <verb>` — container lifecycle over the host
/// admin socket.
pub(crate) async fn run_agents(socket: &Path, cmd: AgentsCmd) -> Result<()> {
@ -134,6 +163,8 @@ pub(crate) async fn run_agents(socket: &Path, cmd: AgentsCmd) -> Result<()> {
AgentsCmd::List { json } => agents_list(socket, json).await,
AgentsCmd::Restart { name, no_wait } => agents_restart(socket, &name, no_wait).await,
AgentsCmd::RestartAll { no_wait } => agents_restart_all(socket, no_wait).await,
AgentsCmd::Pause { name } => set_paused(socket, &name, true).await,
AgentsCmd::Resume { name } => set_paused(socket, &name, false).await,
AgentsCmd::Spawn { name } => {
let name = crate::util::parse_ident(&name)?;
render(crate::client::request(socket, HostRequest::Spawn { name }).await?)

16
hivectl/src/cli.rs
View file

@ -500,6 +500,22 @@ pub enum AgentsCmd {
#[arg(long)]
no_wait: bool,
},
/// Park an agent's turn loop, leaving the container running.
///
/// The harness stops driving turns but keeps serving its web UI and
/// MCP daemons, so the container, its mounts and its warm caches stay
/// up while it burns no tokens. Inbox messages queue unacked and the
/// backlog drains on `resume`. Sticky: it survives a restart, and
/// pausing a stopped agent makes it come up paused.
Pause {
/// Agent name.
name: String,
},
/// Resume a paused agent — it drains whatever queued up while parked.
Resume {
/// Agent name.
name: String,
},
/// Spawn a new agent container directly, bypassing the approval queue.
///
/// Operator-on-the-host only; use `request-spawn` for an approval-gated