feat: pause an agent's turn loop without stopping its container

A paused agent keeps its container, its claude session and its
dashboard/todo servers up, but stops driving turns. Messages queue
unacked and are drained on resume.

The whole protocol is a single marker file, `<harness>/paused`. That
directory is already a bind-mount shared between host and container, so
both sides just stat the same path: the harness reads it to decide
whether to drive a turn, hive-c0re reads it to render the badge and
writes/removes it for `hivectl pause|resume`. No new wire protocol, no
container round-trip, and it is sticky across restarts by construction.

Not calling `recv_next` while paused *is* the queueing semantic, so
there is no fencing to get wrong: reminders buffer in their unbounded
channel, the todo `Notify` permit coalesces, and a `request_next_turn`
that raced the pause survives because the gate sits above
`self_continue.take()`.

Graceful stop is handled host-side rather than in the harness: a paused
agent provably has no turn in flight, so `run_signal` skips the fence
entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for
a checkpoint turn that will never run.

`paused` is reported on `ContainerView` / `AgentStatusRow` for the
dashboard, orthogonal to `running` and reported for stopped containers
too.

Closes: hyperhive/hyperhive issue 2271
This commit is contained in:
atlas 2026-07-26 02:31:54 +02:00 committed by mara
commit 31008c83df
14 changed files with 305 additions and 1 deletions

View file

@ -200,6 +200,30 @@ selected for this agent (default `haiku` when absent). Written by
`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
Survives destroy/recreate, gone on `--purge`.
### `/harness/paused` (per agent)
Empty marker file. Its presence parks the agent's turn loop: the
harness keeps serving its web UI and MCP daemons but drives no turns,
and inbox messages queue unacked until it's removed (see
[turn loop](turn-loop.md#the-loop)).
Unusually, it's read and written from **both** sides of the harness
bind-mount, and that's the whole design: the harness stats it
in-container via `hive_sh4re::paths::paused_marker`, while hive-c0re
stats it on the host (`Coordinator::is_paused`) to populate the
`paused` field on the agent card, and creates/removes it
(`Coordinator::set_paused`) for `hivectl agents pause|resume` and the
dashboard toggle. Because the file itself is the only shared state
there's no protocol between them, no round-trip into the container, and
pause keeps working when the harness is wedged or the container is
stopped.
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
is the agent's own space to fill, and this is harness control state.
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
back paused after a restart, which is the intended behaviour rather
than an accident of storage.
## State dirs (per agent)
Under `/var/lib/hyperhive/agents/<name>/`:

View file

@ -139,11 +139,14 @@ hivectl agents list # roster: every agent's status + technical st
hivectl agents list --json # same data as raw JSON rows (for scripting)
hivectl agents restart iris # stop + start the `iris` container (no rebuild)
hivectl agents restart-all # stop + start every managed agent container in sequence
hivectl agents pause iris # park iris's turn loop, leave the container running
hivectl agents resume iris # let it drive turns again, draining what queued up
```
`list` prints a padded table with one row per managed agent —
`NAME STATUS REV PARENT REMIND`. STATUS collapses the health flags
(`running` / `stopped`, plus ` needs-login` / ` needs-update` when set);
(`running` / `stopped`, plus ` paused` / ` needs-login` / ` needs-update`
when set — `paused` is orthogonal to running, see below);
REV is the first 12 chars of the agent's locked config sha; PARENT is its
place in the topology tree (`-` for a root agent); REMIND is the count of
pending reminders. It reuses the same per-agent aggregation the dashboard
@ -155,6 +158,26 @@ when you need to kick a container from the host without going through
the agent hierarchy. Failures on `restart-all` are collected and
reported at the end rather than aborting mid-run.
`pause` / `resume` are the "stop burning tokens without losing the
container" pair. Pausing writes a marker file into the agent's harness
dir (`<state>/<name>/harness/paused`) which the harness re-stats every
5 s at the top of its serve loop; while it's there the agent drives no
turns, but the container, its mounts, its warm caches, its web UI and
its MCP daemons all stay up. Inbox messages queue **unacked**, so a
resume drains the backlog rather than dropping it. Points worth knowing:
- **Sticky.** The marker lives on the persistent harness mount, so a
paused agent stays paused across a container restart — and pausing a
*stopped* agent makes it come up parked.
- **Not a DAG.** Unlike `restart`/`stop`, there's no container operation
to sequence, so it applies immediately with nothing to wait on.
- **Stopping a paused agent is still fast.** The graceful-stop
handshake is skipped for a paused agent (it would never answer), which
is safe precisely because the pause check sits at the top of the loop:
a paused agent has no turn in flight to checkpoint.
- Visible as ` paused` in `agents list`'s STATUS column, as a `paused`
field on the JSON rows, and as a badge on the dashboard card.
## Choom
Drop into an interactive Claude session inside an agent container.

View file

@ -8,6 +8,15 @@ claude has access to in return.
Each agent harness (`hive-agent` — one serve-loop binary for all
agents) runs:
0. Check the pause marker (`<harness>/paused`). While it exists the
loop does nothing but re-stat it every 5 s — no broker poll, no
claude process. Because step 1 is never reached, messages stay
queued and unacked, so a resume drains the backlog instead of
losing it; reminders and todo wakes buffer in their channels. The
check runs before the self-continue slot is consumed, so a pending
`request_next_turn` survives the pause. Set it with
`hivectl agents pause <name>` or the dashboard toggle; see
[persistence](persistence.md#-harnesspaused-per-agent).
1. Long-poll `Recv` on its socket. The host-side broker
(`broker.rs::recv_blocking_batch`) returns immediately if there's
a pending message, otherwise waits up to 30 s for a broker `Sent`