docs: sort the top-level directories into five sections by what the reader wants
Every docs directory moves under 1-getting-started, 2-handbook, 4-implementation or 5-project; the options reference (section 3) comes from the website and has no directory here. trust-boundary stays whole under the handbook for now. web-ui, swarm and tools split between handbook and implementation, and the command-line references move to 2-handbook/reference. Links, path references in code and nix, and the docs index follow the moves; no page content changes.
This commit is contained in:
iris
mara
parent
53ccbcb4b3
commit
6739bb903d
1 changed files with 736 additions and 701 deletions
259
docs/4-implementation/turn-loop/README.md
Normal file
259
docs/4-implementation/turn-loop/README.md
Normal file
|
|
@ -0,0 +1,259 @@
|
|||
# Turn loop + MCP
|
||||
|
||||
How the harness wakes up, what it asks the agent's runtime to do, and what
|
||||
tools the agent has access to in return.
|
||||
|
||||
## Runtimes
|
||||
|
||||
Every turn runs through `hive-runtime`, on the runtime
|
||||
`services.hyperhive.agent.runtime` selects:
|
||||
|
||||
- **`claude`** (default) — one `claude --print` process per turn.
|
||||
→ [claude-invocation.md](claude-invocation.md)
|
||||
- **`acp`** — any Agent Client Protocol agent, run as one long-lived child
|
||||
of the harness. → [`hive-runtime/README.md`](../../../hive-runtime/README.md)
|
||||
|
||||
Both report the turn in claude's `stream-json` shape, so the loop below is
|
||||
the same for either.
|
||||
|
||||
## The loop
|
||||
|
||||
Each agent harness (`hive-agent` — one serve-loop binary for all
|
||||
agents) runs:
|
||||
|
||||
0. Check the pause marker (`<harness>/paused`). While it exists the
|
||||
loop does nothing but re-stat it every 5 s — no broker poll, no
|
||||
claude process. Because step 1 is never reached, messages stay
|
||||
queued and unacked, so a resume drains the backlog instead of
|
||||
losing it; reminders and todo wakes buffer in their channels. Set
|
||||
it with `hivectl agent <name> pause` or the dashboard toggle; see
|
||||
[persistence](../../2-handbook/agent-lifecycle/persistence.md#-harnesspaused-per-agent).
|
||||
1. Long-poll `Recv` on its socket. The host-side broker
|
||||
(`broker.rs::recv_blocking_batch`) returns immediately if there's
|
||||
a pending message, otherwise waits up to 30 s for a broker `Sent`
|
||||
event for this recipient. After a failed `Recv` (a socket error, or
|
||||
a broker error, which comes back at once) the loop waits before the
|
||||
next one: 250 ms, doubling up to 30 s, and no wait again once a
|
||||
`Recv` succeeds. Todo wakes and local messages don't wait for it.
|
||||
2. Pop one message. Peek the remaining inbox depth with `Status`.
|
||||
3. Emit `LiveEvent::TurnStart { from, body, unread }` onto the SSE
|
||||
bus.
|
||||
4. Hand the wake prompt to the runtime: on `claude`, spawn
|
||||
`claude --print` and pipe it over stdin; on `acp`, send it as a
|
||||
prompt on the agent's session.
|
||||
5. Stream the turn's `stream-json` lines into the bus as
|
||||
`LiveEvent::Stream(value)`. Pump stderr as `Note`.
|
||||
6. Wait for the turn to end and classify its outcome from the
|
||||
stream + result — success, compaction, rate-limit, auth-failure, stall,
|
||||
or hard failure. The outcome drives the post-turn action (see
|
||||
[Turn outcomes](#turn-outcomes)); the session handles compaction
|
||||
internally (see
|
||||
[Compaction](claude-invocation.md#compaction)). This page describes
|
||||
rate-limit and auth-failure detection [below](#failure-detection-and-login-claude).
|
||||
7. Emit `LiveEvent::TurnEnd { ok, note }`. Sleep `poll_ms` to avoid
|
||||
tight loops on transient failures.
|
||||
|
||||
### Failure detection and login (claude)
|
||||
|
||||
- **Rate limit** — a `429` / `rate_limit` marker on stderr, or a parsed
|
||||
`{"type":"error"}` rate-limit event on stdout (conversation-text
|
||||
mentions don't count), sets the `rate_limited` sentinel, parks for
|
||||
`HIVE_RATE_LIMIT_SLEEP_SECS` (default 300), then retries. The UI shows
|
||||
a `⊘ rate limited` badge while parked.
|
||||
- **Auth failure (401)** — `drive_turn` retries once (transient
|
||||
token-refresh races clear on retry); a second `AuthFailed` writes
|
||||
`{state_dir}/hyperhive-needs-login`, requeues the message, and parks in
|
||||
`wait_for_login` — the same path as a cold boot with no session. The
|
||||
operator re-auths via the per-agent web UI; the queued message then
|
||||
drives the next turn.
|
||||
- **Login detection** — both boot (`login::has_session`, Online vs
|
||||
NeedsLogin) and `wait_for_login`'s resume check key off the credential
|
||||
files in `login::CRED_FILE_NAMES` (the set `/logout` deletes).
|
||||
`wait_for_login` takes a `since: SystemTime` baseline (the instant of
|
||||
the 401 that parked it, or `login::NO_PRIOR_FAILURE` at cold boot) and
|
||||
resumes only once a credential file's mtime postdates it — so stale
|
||||
credentials already on disk at the 401 don't trigger an instant
|
||||
false-resume, and a login that lands *before* `wait_for_login` even
|
||||
starts polling still resumes correctly (baselining on a fixed instant
|
||||
rather than an entry-time directory snapshot is what closes that race).
|
||||
Leftover session-history files still don't read as a live session after
|
||||
a logout + container recreate (`has_session`/`is_cred_file` scope to
|
||||
`CRED_FILE_NAMES` either way).
|
||||
|
||||
## Harness binary shape
|
||||
|
||||
Two sibling binaries over one runtime library, all role-agnostic (there
|
||||
is one role: agent — the privilege boundary lives server-side at the
|
||||
broker socket (`/run/hive/mcp.sock`), which refuses privileged `Request`
|
||||
variants regardless of who sends them):
|
||||
|
||||
- `hive-agent` — long-running harness loop (the inbox poll +
|
||||
runtime turn + ack/requeue cycle described above).
|
||||
- `hive-runtime` — the library every turn runs through, one `Runtime`
|
||||
impl per runtime (`ClaudeRuntime`, `AcpRuntime`); `hive-subagent-mcp`
|
||||
drives its nested sessions through it too.
|
||||
- `hive-agent-mcp` — MCP server for the built-in `hyperhive` surface.
|
||||
Run with `--http <addr>` as a persistent streamable-HTTP daemon (the
|
||||
`hive-mcp-http` systemd unit, on `services.hyperhive.agent.mcp.httpPort`, default
|
||||
8790); claude connects to its URL via `--mcp-config`. HTTP is the sole
|
||||
transport — no per-turn stdio child (eliminates the re-registration
|
||||
race).
|
||||
|
||||
A small `Surface` trait, with one zero-sized impl, factors
|
||||
`hive-agent`'s wire types (`hive_core_agent_sock::{Request, Response}` —
|
||||
one unified enum shared by every agent socket) and its turn
|
||||
loop, so the loop itself has no per-role branches. See
|
||||
`hive-agent/src/main.rs`'s module
|
||||
doc for the trait shape.
|
||||
|
||||
### Boot wiring
|
||||
|
||||
`serve_main` reads `HIVE_PORT` (default `DEFAULT_WEB_PORT`) +
|
||||
`HIVE_LABEL` (default `"hive"` for standalone runs; the meta
|
||||
flake sets it unconditionally for any container-deployed agent;
|
||||
see `docs/5-project/process/conventions.md::Hive identity` for the env stack),
|
||||
opens turn-stats sqlite, prepares the on-boot files (see
|
||||
[claude-invocation](claude-invocation.md#on-boot-files)),
|
||||
installs claude plugins, spawns `web_ui::serve` + `vacuum::run`,
|
||||
and either drops into `serve_loop` directly (`Online`) or parks on
|
||||
the login flow first (`NeedsLogin`). `hive-forge-notify` polls forge
|
||||
notifications on its own, not this loop — see
|
||||
[`forge.md`](../../2-handbook/integrations/forge.md).
|
||||
|
||||
Boot also opens the todos store and the socket in-container producers
|
||||
dial. Matrix / bash / forge-notify daemons and the in-process
|
||||
`disk_watch` todo producer (low state-disk space) are the built-in
|
||||
producers, but the socket accepts any `subsystem` marker — a
|
||||
user-configured MCP server can push its own todos the same way. See
|
||||
[`docs/2-handbook/agent-lifecycle/persistence.md`](../../2-handbook/agent-lifecycle/persistence.md#state-dirs-per-agent) for
|
||||
what each built-in todo producer watches and how the store + `get_loose_ends`
|
||||
merge work.
|
||||
|
||||
Plugin install failures aren't fatal: the harness logs each at `warn` and skips it.
|
||||
|
||||
### Turn outcomes
|
||||
|
||||
`turn::TurnOutcome` (`Result<bool, TurnError>` — `Ok(compacted)` on success,
|
||||
else a `TurnError`) drives the post-claude branch:
|
||||
|
||||
| Outcome | Action |
|
||||
| --- | --- |
|
||||
| `Ok(_)` (`false` normal / `true` compacted) | `ack_turn` |
|
||||
| `Err(PromptTooLong)` | `drive_turn` archived the session (the lib already compacted + retried and it still overflowed); requeue inflight so the message redelivers into a fresh session that fits — no status park |
|
||||
| `Err(RateLimited)` | sleep `HIVE_RATE_LIMIT_SLEEP_SECS` (default 300), requeue inflight, status back to `online` |
|
||||
| `Err(AuthFailed)` | emit `needs_login_idle` sentinel, requeue inflight, park in `wait_for_login` |
|
||||
| `Err(SessionNotFound)` | resume + create self-heal both missed ("shouldn't happen"); requeue inflight so the next turn creates fresh — no status park, message not dropped |
|
||||
| `Err(ApiStall)` | idle watchdog sent claude SIGINT after `HIVE_TURN_IDLE_SECS` (default 600) of output silence and claude exited; sleep `HIVE_STALL_SLEEP_SECS` (default 60), requeue inflight, status back to `online` |
|
||||
| `Err(AgentStall(note))` | the same, for an ACP agent: after `HIVE_TURN_IDLE_SECS` with no `session/update` the harness sent the agent `session/cancel`, and SIGINT if it ignored that; `note` says which |
|
||||
| `Err(Failed(err))` | the harness logs it at `warn` as `turn failed`, the line the Grafana alert rule `hh-agent-turn-failed` (`nix/host-modules/swarm-grafana.nix`) fires on |
|
||||
|
||||
`ApiStall` catches an Anthropic API stall — a multi-retry connection storm where
|
||||
the stream goes silent for minutes. On `claude` the idle watchdog lives in
|
||||
`hive-claude`'s driver (`Config::idle_timeout`, enforced around `child.wait()`): the timer
|
||||
resets on every stdout line, so a large but still-streaming turn is never cut —
|
||||
only complete output silence for the window trips it. The harness sets the
|
||||
window from `HIVE_TURN_IDLE_SECS` (`0` disables) and maps the driver's
|
||||
`Error::IdleTimeout` onto `TurnError::ApiStall`. The watchdog sends claude's
|
||||
process group SIGINT, so claude stops the commands it started, and then
|
||||
waits for claude to exit. Nothing escalates to SIGKILL: a claude that
|
||||
ignores the SIGINT keeps the turn open until someone kills it or the unit
|
||||
stops.
|
||||
|
||||
`hive-runtime` applies the same window to an ACP agent's turns. That's
|
||||
also how a provider HTTP 429 ends an ACP turn when the agent retries it without
|
||||
reporting it: the turn goes silent until the watchdog stops it.
|
||||
|
||||
After the outcome handler, the stats sink records a row. `handle_turn`
|
||||
reports the result to `serve_loop` via `TurnControl { auth_failed }` —
|
||||
on auth failure the loop parks in `wait_for_login`; otherwise it loops
|
||||
straight back to the idle wait (step 1). No same-turn
|
||||
self-continue mechanism exists: every multi-step continuation rides an
|
||||
external wake instead — a new inbox message, a `remind`, or an
|
||||
in-container todo wake (bash-task completion, forge notification,
|
||||
matrix activity). Ending the turn and letting one of
|
||||
those drive the next one is strictly better than parking in-process:
|
||||
it checkpoints the session and observes wakes that only reach the
|
||||
harness between turns.
|
||||
|
||||
## Harness systemd unit shape
|
||||
|
||||
One harness serve binary (`hive-agent`, with its `hive-agent-mcp`
|
||||
sibling), one shared `nix/agent-modules/` tree, one service unit
|
||||
(`systemd.services.hive-agent`) for all agents. Privilege
|
||||
differences live in the agent's swarm permissions and the broker socket (which tools
|
||||
and manager-surface calls each agent receives).
|
||||
|
||||
`nix/templates/agent.nix` is the one entry point every agent builds from, and
|
||||
it imports the shared `nix/agent-modules/`.
|
||||
`services.hyperhive.agent.docs.enable` defaults to `false` for every agent,
|
||||
ruth included; an agent opts in from its own `agent.nix` (see
|
||||
[config.md](config.md), "Reference docs").
|
||||
|
||||
### Environment variables set on the unit
|
||||
|
||||
- `HOME = /home/<userName>` — systemd defaults `HOME` to `/` for
|
||||
services without `User=` set; with the per-agent user the harness
|
||||
needs the right home so claude finds its bind-mounted `~/.claude/`
|
||||
session dir.
|
||||
- `HIVE_STATIC_DIR = <mergedDist>` — `tower_http::ServeDir` root for
|
||||
the per-agent web UI; merged dist = agent default + every
|
||||
`services.hyperhive.agent.frontend.extraFiles` overlay.
|
||||
- `HIVE_ASSETS_DIR = "${config.services.hyperhive.agent.packages.assets}/share/hyperhive"`
|
||||
(`nix/agent-modules/agent-service.nix:480`; `packages.assets` resolves
|
||||
to the `hyperhive-assets` derivation) — set directly on the unit,
|
||||
**not** via `environment.variables`, because the latter only
|
||||
populates `/etc/profile` which systemd services don't inherit.
|
||||
|
||||
### `PATH` setup (the wrapper-dir trick)
|
||||
|
||||
```nix
|
||||
path = [ "/run/wrappers" "/run/current-system/sw" ];
|
||||
```
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
`/run/wrappers` (not `/run/wrappers/bin`) comes first so setuid
|
||||
wrappers — notably `sudo` — resolve before bare nix-store binaries; see
|
||||
[`docs/5-project/process/gotchas.md`](../../5-project/process/gotchas.md) ("`systemd.services.*.path` appends
|
||||
`/bin` to every entry") for why the trailing `/bin` matters in
|
||||
general. It's load-bearing here because the harness runs as the
|
||||
per-agent user: without the wrapper dir on `PATH`, `sudo` resolves to
|
||||
the non-setuid nix-store binary and every
|
||||
`services.hyperhive.agent.user.passwordlessSudo` grant fails with "must be owned by
|
||||
uid 0 and have the setuid bit set."
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
### `serviceConfig` highlights
|
||||
|
||||
- `ExecStart = "${config.services.hyperhive.agent.packages.hive-agent}/bin/${binary}"`
|
||||
(`nix/agent-modules/agent-service.nix:535`); `packages.hive-agent` is
|
||||
wired to the flake's `hyperhive.packages.<system>.hive-agent` output
|
||||
(`nix/agent-modules/packages.nix:8-22`) — same binary for every agent.
|
||||
- `RuntimeDirectory = "hive-config"` → `/run/hive-config/` owned by
|
||||
`User=`, autocleared on stop. The harness writes regenerated
|
||||
`claude-{mcp-config,settings,system-prompt}` files there
|
||||
(`paths::config_dir`). Deliberately separate from `/run/hive`, which
|
||||
the host bind-mounts in root-owned and which holds hive-c0re's
|
||||
`mcp.sock`.
|
||||
- `User = Group = userName` — drops root inside the container; sudo is
|
||||
the explicit escalation surface (`services.hyperhive.agent.user.passwordlessSudo`).
|
||||
|
||||
## Sub-pages
|
||||
|
||||
The rest lives alongside this page, in three topic files:
|
||||
|
||||
- **[claude-invocation.md](claude-invocation.md)** — how the harness
|
||||
spawns `claude --print` each turn, the two-pronged compaction (reactive +
|
||||
proactive), and the on-boot files it materialises (`--mcp-config`,
|
||||
`--system-prompt-file`).
|
||||
- **[config.md](config.md)** — the optional per-agent knobs the meta
|
||||
flake wires in (reference docs, icon, passwordless sudo, dashboard links,
|
||||
custom static files, connectivity overrides, claude plugins, cargo message
|
||||
filtering).
|
||||
- **[mcp.md](mcp.md)** — the MCP tool surface claude sees: core tools,
|
||||
permission-gated tools, self-wake, authoritative state, the tool envelope,
|
||||
and the built-in tool allowlist.
|
||||
|
||||
Per-subsystem impl detail lives in each module's `//!` doc-comment; these pages
|
||||
describe present-state behaviour + wiring, not line-level mechanics.
|
||||
Loading…
Reference in a new issue