feat: pause an agent's turn loop without stopping its container
A paused agent keeps its container, its claude session and its dashboard/todo servers up, but stops driving turns. Messages queue unacked and are drained on resume. The whole protocol is a single marker file, `<harness>/paused`. That directory is already a bind-mount shared between host and container, so both sides just stat the same path: the harness reads it to decide whether to drive a turn, hive-c0re reads it to render the badge and writes/removes it for `hivectl pause|resume`. No new wire protocol, no container round-trip, and it is sticky across restarts by construction. Not calling `recv_next` while paused *is* the queueing semantic, so there is no fencing to get wrong: reminders buffer in their unbounded channel, the todo `Notify` permit coalesces, and a `request_next_turn` that raced the pause survives because the gate sits above `self_continue.take()`. Graceful stop is handled host-side rather than in the harness: a paused agent provably has no turn in flight, so `run_signal` skips the fence entirely instead of eating the full `GRACEFUL_STOP_TIMEOUT` waiting for a checkpoint turn that will never run. `paused` is reported on `ContainerView` / `AgentStatusRow` for the dashboard, orthogonal to `running` and reported for stopped containers too. Closes: hyperhive/hyperhive issue 2271
This commit is contained in:
parent
1356a1f049
commit
31008c83df
14 changed files with 305 additions and 1 deletions
|
|
@ -200,6 +200,30 @@ selected for this agent (default `haiku` when absent). Written by
|
|||
`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
|
||||
Survives destroy/recreate, gone on `--purge`.
|
||||
|
||||
### `/harness/paused` (per agent)
|
||||
|
||||
Empty marker file. Its presence parks the agent's turn loop: the
|
||||
harness keeps serving its web UI and MCP daemons but drives no turns,
|
||||
and inbox messages queue unacked until it's removed (see
|
||||
[turn loop](turn-loop.md#the-loop)).
|
||||
|
||||
Unusually, it's read and written from **both** sides of the harness
|
||||
bind-mount, and that's the whole design: the harness stats it
|
||||
in-container via `hive_sh4re::paths::paused_marker`, while hive-c0re
|
||||
stats it on the host (`Coordinator::is_paused`) to populate the
|
||||
`paused` field on the agent card, and creates/removes it
|
||||
(`Coordinator::set_paused`) for `hivectl agents pause|resume` and the
|
||||
dashboard toggle. Because the file itself is the only shared state
|
||||
there's no protocol between them, no round-trip into the container, and
|
||||
pause keeps working when the harness is wedged or the container is
|
||||
stopped.
|
||||
|
||||
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
||||
is the agent's own space to fill, and this is harness control state.
|
||||
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
||||
back paused after a restart, which is the intended behaviour rather
|
||||
than an accident of storage.
|
||||
|
||||
## State dirs (per agent)
|
||||
|
||||
Under `/var/lib/hyperhive/agents/<name>/`:
|
||||
|
|
|
|||
|
|
@ -139,11 +139,14 @@ hivectl agents list # roster: every agent's status + technical st
|
|||
hivectl agents list --json # same data as raw JSON rows (for scripting)
|
||||
hivectl agents restart iris # stop + start the `iris` container (no rebuild)
|
||||
hivectl agents restart-all # stop + start every managed agent container in sequence
|
||||
hivectl agents pause iris # park iris's turn loop, leave the container running
|
||||
hivectl agents resume iris # let it drive turns again, draining what queued up
|
||||
```
|
||||
|
||||
`list` prints a padded table with one row per managed agent —
|
||||
`NAME STATUS REV PARENT REMIND`. STATUS collapses the health flags
|
||||
(`running` / `stopped`, plus ` needs-login` / ` needs-update` when set);
|
||||
(`running` / `stopped`, plus ` paused` / ` needs-login` / ` needs-update`
|
||||
when set — `paused` is orthogonal to running, see below);
|
||||
REV is the first 12 chars of the agent's locked config sha; PARENT is its
|
||||
place in the topology tree (`-` for a root agent); REMIND is the count of
|
||||
pending reminders. It reuses the same per-agent aggregation the dashboard
|
||||
|
|
@ -155,6 +158,26 @@ when you need to kick a container from the host without going through
|
|||
the agent hierarchy. Failures on `restart-all` are collected and
|
||||
reported at the end rather than aborting mid-run.
|
||||
|
||||
`pause` / `resume` are the "stop burning tokens without losing the
|
||||
container" pair. Pausing writes a marker file into the agent's harness
|
||||
dir (`<state>/<name>/harness/paused`) which the harness re-stats every
|
||||
5 s at the top of its serve loop; while it's there the agent drives no
|
||||
turns, but the container, its mounts, its warm caches, its web UI and
|
||||
its MCP daemons all stay up. Inbox messages queue **unacked**, so a
|
||||
resume drains the backlog rather than dropping it. Points worth knowing:
|
||||
|
||||
- **Sticky.** The marker lives on the persistent harness mount, so a
|
||||
paused agent stays paused across a container restart — and pausing a
|
||||
*stopped* agent makes it come up parked.
|
||||
- **Not a DAG.** Unlike `restart`/`stop`, there's no container operation
|
||||
to sequence, so it applies immediately with nothing to wait on.
|
||||
- **Stopping a paused agent is still fast.** The graceful-stop
|
||||
handshake is skipped for a paused agent (it would never answer), which
|
||||
is safe precisely because the pause check sits at the top of the loop:
|
||||
a paused agent has no turn in flight to checkpoint.
|
||||
- Visible as ` paused` in `agents list`'s STATUS column, as a `paused`
|
||||
field on the JSON rows, and as a badge on the dashboard card.
|
||||
|
||||
## Choom
|
||||
|
||||
Drop into an interactive Claude session inside an agent container.
|
||||
|
|
|
|||
|
|
@ -8,6 +8,15 @@ claude has access to in return.
|
|||
Each agent harness (`hive-agent` — one serve-loop binary for all
|
||||
agents) runs:
|
||||
|
||||
0. Check the pause marker (`<harness>/paused`). While it exists the
|
||||
loop does nothing but re-stat it every 5 s — no broker poll, no
|
||||
claude process. Because step 1 is never reached, messages stay
|
||||
queued and unacked, so a resume drains the backlog instead of
|
||||
losing it; reminders and todo wakes buffer in their channels. The
|
||||
check runs before the self-continue slot is consumed, so a pending
|
||||
`request_next_turn` survives the pause. Set it with
|
||||
`hivectl agents pause <name>` or the dashboard toggle; see
|
||||
[persistence](persistence.md#-harnesspaused-per-agent).
|
||||
1. Long-poll `Recv` on its socket. The host-side broker
|
||||
(`broker.rs::recv_blocking_batch`) returns immediately if there's
|
||||
a pending message, otherwise waits up to 30 s for a broker `Sent`
|
||||
|
|
|
|||
Loading…
Reference in a new issue