hyperhive/docs/turn-loop.md

Turn loop + MCP

How the harness wakes up, what it asks claude to do, and what tools claude has access to in return.

The loop

Each agent harness (hive serve, with role picked from $HIVE_ROLE — "agent" for sub-agents, "manager" for the manager; post-#598 this is one binary not two) runs:

1. Long-poll Recv on its socket. The host-side broker (broker.rs::recv_blocking_batch) returns immediately if there's a pending message, otherwise waits up to 30 s for a broker Sent event for this recipient.
2. Pop one message. Peek the remaining inbox depth with Status.
3. Emit LiveEvent::TurnStart { from, body, unread } onto the SSE bus.
4. Spawn claude (one process per turn) and pipe the wake prompt over stdin.
5. Stream stdout (JSON lines) into the bus as LiveEvent::Stream(value). Pump stderr as Note.
6. Wait for claude to exit. Compaction is two-pronged — reactive on Prompt is too long and proactive on a context watermark (see Compaction below). Rate-limit detection: on stderr the harness does a raw-line match for 429 / rate_limit markers; on stdout it only fires on parsed {"type":"error"} JSON events (avoiding false positives when agents discuss rate_limit_error in conversation text). On detection the harness sets the rate_limited sentinel (Bus::emit_status("rate_limited")), sleeps HIVE_RATE_LIMIT_SLEEP_SECS (default 300), then retries. The dashboard and per-agent page show a ⊘ rate limited badge while the harness is parked. Auth-failed detection (closes #419): both stdout and stderr pumps also match AUTH_FAIL_MARKERS ("authentication_failed", 401, etc.). On match the harness writes {state_dir}/hyperhive-needs-login, emits needs_login_idle status, requeues the inflight message (so it replays after re-auth), and parks in wait_for_login — the same path used at boot. The operator re-authenticates via the per-agent web UI login flow; on success the sentinel is cleared and the queued message drives the next turn normally. Mtime-snapshot resumption (closes #542): wait_for_login snapshots the ~/.claude/ dir (newest file mtime + file count) at entry and only resumes when that snapshot advances — not just when credentials exist on disk. This prevents a silent infinite-401 loop: stale credentials already on disk at the time of the 401 no longer cause an immediate false-resume. The DirSnapshot struct tracks both axes; either a mtime advance OR a file-count change triggers resume (the count axis handles filesystems where modified() errors on every file).
7. Emit LiveEvent::TurnEnd { ok, note }. Sleep poll_ms to avoid tight loops on transient failures.

The claude invocation

claude --print --verbose --output-format stream-json --model <name> \
  --continue --settings /run/hive/claude-settings.json \
  --system-prompt-file /run/hive/claude-system-prompt.md \
  --mcp-config /run/hive/claude-mcp-config.json --strict-mcp-config \
  --tools <builtins> --allowedTools <builtins+mcp>
# wake prompt piped over stdin

<name> is read from Bus::model() on each turn. The initial default is set by hyperhive.model in the agent's agent.nix (NixOS option; propagates via HIVE_DEFAULT_MODEL env var; falls back to "haiku" if unset). The operator can flip it at runtime with /model <name> in the web terminal — the next turn picks it up. The choice is persisted to /state/hyperhive-model so it survives restart; override path: HYPERHIVE_MODEL_FILE env var for tests.

Context-window size is looked up per-model via events::context_window_tokens(model). Resolution order (first match wins):

1. HIVE_CONTEXT_WINDOW_TOKENS_<KEY> env var, where KEY (lowercased) is a substring of the active model name. Injected by the meta flake from services.hyperhive.c0re.contextWindowTokens (host-level NixOS option, defaults: haiku=200k, sonnet=1M, opus=1M). Override these for all agents at once without a per-agent config change.
2. HIVE_CONTEXT_WINDOW_TOKENS — single global override for any model (useful in dev / test).
3. Hard fallback: 200_000 (conservative; only reached outside NixOS where the env vars aren't set).

The effective window drives watermarks and is exposed at runtime via /api/state.context_window_tokens so the UI can show a percentage-of-window ctx badge.

--continue keeps a persistent session per agent (claude stores sessions in ~/.claude/projects/, which is bind-mounted persistently). Auto-compact and auto-memory are disabled via --settings because hyperhive owns compaction — see Compaction below. A one-shot --continue suppression is available via POST /api/new-session (or /new-session slash command in the per-agent terminal) — Bus::take_skip_continue() flips an AtomicBool once per turn, the next claude invocation drops --continue, every subsequent turn resumes normal behaviour.

Compaction

claude's own in-session auto-compact is off (--settings); hyperhive owns it explicitly in turn::drive_turn. There are two triggers:

• Reactive — claude-code prints Prompt is too long (the PROMPT_TOO_LONG_MARKER). The session is already past the context window, so no turn can run on it — drive_turn runs /compact straight away and retries the same wake-up prompt once. No notes-checkpoint turn is possible here: the detail is gone.
• Proactive — a turn finishes cleanly but the last inference's context size (Bus::last_ctx_usage().context_tokens()) is at or above a watermark. While the session is still healthy, drive_turn injects one synthetic notes-checkpoint turn (CHECKPOINT_PROMPT — "context is filling up, flush durable state into /state now") and then runs /compact. This gives the agent a chance to persist in-flight task state, decisions, and file paths before the conversation detail collapses into a summary.

The compact watermark defaults to 75% of context_window_tokens(model) (dynamically derived — 150k for haiku, 750k for sonnet/opus). Override with HIVE_COMPACT_WATERMARK_TOKENS (absolute token count); set to 0 to disable proactive compaction entirely (the reactive path always applies). The proactive path is best-effort — a failed checkpoint turn or /compact is surfaced as a Note but never fails the turn that already succeeded. The operator can also force a compaction any time via /api/compact.

• Auto session-reset — a third path that fires when both conditions hold: context is ≥ a watermark (HIVE_AUTO_RESET_WATERMARK_TOKENS, default 50% of context_window_tokens(model)) AND the time since the last turn exceeds the assumed prompt-cache TTL (HIVE_CACHE_TTL_SECS, default 3600). Claude's prompt cache lives ~5 minutes; if the cache is already cold, resuming with --continue pays the full re-upload cost of the current context with no benefit over starting fresh. So: drive_turn injects one AUTO_RESET_CHECKPOINT_PROMPT notes turn ("flush state to files, cache is cold") then arms Bus::take_skip_continue() for the real turn — the next turn runs without --continue, starting a fresh session. Unlike proactive compaction the session is dropped entirely, not compacted. Set HIVE_AUTO_RESET_WATERMARK_TOKENS=0 to disable.

The child runs with cwd = /state (when the bind exists; falls back to the parent's cwd in dev), so any relative path in a tool call (Read foo.md, Bash ls, Write notes.md) lands in the agent's durable bind-mounted dir. CLAUDE.md auto-load walks upward from /state — drop a per-agent CLAUDE.md there if you want long-term hints that survive destroy/recreate.

The wake prompt is intentionally minimal: just the popped message's from/body, plus an inline ({unread} more pending — drain via …) hint when unread > 0. Claude drives any further recv/send itself via the embedded MCP server.

Whenever hive-c0re starts / restarts / rebuilds a container, it also drops a system message into the agent's inbox via Coordinator::kick_agent — a one-line "you were just (re)started, check /state/ for your notes, --continue session is intact". The next turn picks it up like any other inbox message.

On-boot files

hive_ag3nt::turn::write_* writes three files next to the per-agent socket at /run/hive/ once at startup:

• claude-mcp-config.json — re-invokes the running binary as mcp child (so the same binary serves as harness + as claude's MCP child process).
• claude-settings.json — the --settings blob (auto-compact and auto-memory off, effortLevel medium).
• claude-system-prompt.md — rendered from hive-ag3nt/prompts/system.md by hive_ag3nt::prompt::render: HTML-comment markers (<!-- role:agent -->...<!-- /role:agent -->, same for role:manager) gate the role-specific blocks (closes #519); everything else is shared. Five placeholders are then substituted: {label} (short agent name), {qualified_label} (hive-qualified name@domain form), {operator_pronouns}, {hive_identity} (e.g. on hive `pr1ma`; empty when hyperhive.hiveName is unset), and {swarm_identity} (same shape for the swarm). Pronouns come from HIVE_OPERATOR_PRONOUNS env (set by the meta flake from services.hyperhive.c0re.operatorPronouns, default she/her). Passed via --system-prompt-file.

The shared per-turn plumbing lives in hive_ag3nt::turn::{write_mcp_config, write_settings, write_system_prompt, run_turn, drive_turn, emit_turn_end, wait_for_login, compact_session} so the two binaries can't drift.

MCP surface

The harness ships an embedded MCP server (rmcp 1.7). Claude launches it as a stdio child via --mcp-config. The hyperhive socket name is hyperhive, so the tools land in claude as mcp__hyperhive__<tool>.

Sub-agent tools

• send(to, body, in_reply_to?) — message a peer (logical agent name), another agent, or the operator (recipient operator, surfaces in the dashboard inbox). Use to: "<parent>" to address the agent's topology parent without knowing its label; the broker resolves the sentinel at delivery time (falls back to operator for root agents). Optional in_reply_to: i64 links this message to a prior message id for thread rendering in the dashboard message flow and the per-agent inbox.
• recv(wait_seconds?, max?) — drain inbox messages. Without wait_seconds (or with 0) returns immediately, a cheap "anything pending?" peek. Positive value parks the turn up to that many seconds (cap 180) — incoming messages wake instantly, otherwise returns empty at the timeout. max (default 1, server-side cap 32) drains up to N popped rows in one round-trip; wait_seconds applies to the first message, then the call drains up to max total.
• ask(question, options?, multi?, ttl_seconds?, to?) — surface a structured question. Same shape as the manager's; recipient defaults to the operator (dashboard) but can be set to a peer agent name via to: "<agent>". Answer routes back to the asker's own inbox as HelperEvent::QuestionAnswered via coord.notify_agent. For peer questions the recipient sees a HelperEvent::QuestionAsked event and replies with answer(id, answer).
• answer(id, answer) — respond to a question_asked event routed to this agent. Authorisation is strict: only the declared target (or the operator via the dashboard) can answer.
• get_loose_ends() — list everything still pending against this agent: unanswered questions it asked / was asked, plus reminders it scheduled. Each row carries an id + kind for cancel_loose_end.
• cancel_loose_end(kind, id) — withdraw a question (posts [cancelled by <self>] to unblock the asker), a reminder (hard-delete before fire), or (manager only) an approval (transitions to Cancelled; sub-agents refused with a clear error). Sub-agents may only cancel rows they own.
• remind(message, due) — schedule a reminder that lands in this agent's own inbox at a future time (sender shows as reminder). Large payloads spill to /agents/<self>/state/reminders/ with the inbox message a short pointer. Each agent's pending-reminder count is capped (default 50, override via HIVE_REMIND_MAX_PENDING_PER_AGENT); scheduling a new one fails if the cap is already hit.
• set_status(text) — set a free-text status string visible on the operator dashboard. Persisted to {state_dir}/hyperhive-status; survives harness restarts. Pass an empty string to clear. Validated: must be a single line and ≤ 200 Unicode characters; the server rejects multi-line or over-length text with a clear error.
• get_agent_meta(name?) — fetch identity + status metadata for an agent: { name, role, hyperhive_rev, running, status_text, status_set_at }. Pass name to query a peer (e.g. check whether a sub-agent is idle before sending it work). Omit name to get your own identity stamp — replaces the previous whoami tool. running is true when the container is up. When running is false the host clears status_text / status_set_at (they would be stale snapshots from before the container stopped) before serving the response. Status fields are also None when the target has never called set_status or has cleared it.
• request_next_turn() — ask the harness to start another turn immediately after this one ends, even if the inbox is empty. Use for multi-turn tasks (long builds, sequential steps) where you want to continue without waiting for an external message. The next turn starts with from: "self" and body: "continue". No-op if new inbox messages arrive before this turn ends. No args.

Waking the agent from inside the container

External MCP servers (and any other in-container process) can inject a wake-up event into the agent's inbox via the per-agent socket at /run/hive/mcp.sock. Two equivalent paths:

• Shell out to hive wake --from <label> --body <text> (use --body - to read body from stdin). Already on the container's PATH since the harness binary is in systemPackages. Convenient for shell-script integrations. Works for both agent and manager roles.

• Speak the wire protocol directly — JSON-line over the unix socket: {"cmd":"wake","from":"matrix","body":"new dm from @alice"}\n. Same shape any other AgentRequest uses; see hive-sh4re::AgentRequest::Wake.

The wake event lands in the broker as {from:<label>, to:<agent>, body}, which wakes whatever recv call the harness is currently blocked on. Next turn fires with the wake prompt formed from that message — claude sees "from: matrix" (or whatever label) and reacts.

Identity = socket: anything that can connect to /run/hive/mcp.sock is implicitly trusted to inject these, which is fine because the bind-mount is the agent's own container only.

Built-in matrix MCP (mcp__matrix__*)

When hyperhive.matrix.enable = true (default) and the host-level matrix tuwunel is configured, the harness auto-injects hive-matrix-mcp as a second stdio MCP server. Tools land as mcp__matrix__<name>:

• send_message(room, body) — send a markdown message; room accepts !id:server or #alias:server, daemon resolves either
• send_dm(user_id, body) — send a direct message to a matrix user
• send_reaction(room, event_id, key) — react to a message
• send_reply(room, event_id, body) — threaded reply
• mark_read(room, event_id) — advance the read receipt
• list_rooms() — enumerate joined rooms ({ id, canonical_alias, name, member_count } per room)
• list_room_members(room) — members of a room
• read_room(room, limit?) — recent timeline events

The daemon (hive-matrix-daemon) holds the long-running matrix-sdk Client + sync loop; the stdio bridge (hive-matrix-mcp) is spawned per turn and forwards tool calls over /run/hive-matrix.sock. Both silently exit when <state>/matrix-token is absent (account not yet provisioned). Incoming room events wake the agent via AgentRequest::Wake with from: "matrix" and a teaser body ([matrix] <sender> in <room>: <first-100c>… where <sender> is the matrix user id and <room> is the alias or id as reported by the SDK).

Extra MCP servers (per-agent)

Each agent's NixOS config can declare additional MCP servers via hyperhive.extraMcpServers.<key> = { command, args, env, allowedTools }. The module writes the map to /etc/hyperhive/extra-mcp.json; the harness reads it at boot and merges every entry into --mcp-config (under mcpServers.<key>) and --allowedTools (as mcp__<key>__<pattern>). The agent's flake.nix forwards every flake input to agent.nix as the flakeInputs module arg, so external MCP-server flakes are pulled in by adding them to inputs.* and referenced as flakeInputs.<name>.packages.${pkgs.system}.default — the resolved sha lands in the agent's own flake.lock and rolls up to meta's.

Manager tools (in addition to send/recv)

• request_init_config(name, description?) — first step of a two-step spawn. Queues an InitConfig approval (≤9 char name); on operator approve, hive-c0re seeds the proposed config repo with a default agent.nix template and sends the manager a HelperEvent::ConfigReady { agent }. The manager then edits agent.nix, commits the changes, and calls request_apply_commit with the commit sha — the first ApplyCommit on a freshly-init'd config creates the container. Fails if a proposed repo for this name already exists. (The pre-#442 path through a separate manager-side request_spawn was removed; operator can still direct-spawn an empty agent from the dashboard's ◆ R3QU3ST SP4WN button which routes via HostRequest::RequestSpawn.)
• kill(name) — graceful stop. No approval required.
• start(name) — start a stopped sub-agent. No approval.
• restart(name) — stop + start. No approval.
• update(name) — rebuild (re-applies the current hyperhive flake
  • agent.nix, restarts). No approval, idempotent. Manager calls this on receipt of a needs_update system event.
• request_apply_commit(agent, commit_ref) — submit a config change for any agent (hm1nd for the manager's own config) for operator approval.
• request_update_meta_inputs(inputs?, description?) — queue an approval to run nix flake update [inputs...] on the meta flake. Pass specific input names (e.g. ["bitburner-agent"]) or omit for all. Returns immediately; lock update runs on operator approval. Does NOT trigger rebuilds — call update(name) on affected agents after approval resolves.
• ask(question, options?, multi?, ttl_seconds?, to?) — surface a structured question to the operator (default) or a sub-agent (to: "<agent>"). Non-blocking — returns the queued question id; the answer arrives later as HelperEvent::QuestionAnswered { id, question, answer, answerer } in the asker's inbox. Options always render alongside a free-text fallback; multi=true renders options as checkboxes. ttl_seconds auto-cancels with answer [expired] (and answerer: "ttl-watchdog") after the deadline (useful for time-sensitive decisions that become moot if no one has responded). The operator can also manually cancel with [cancelled] via the dashboard.
• answer(id, answer) — respond to a question_asked event that was routed to the manager (a sub-agent did ask(to: "manager", ...)). Surfaces in the asker's inbox as the same question_answered event.
• get_logs(agent, lines?) — fetch recent journal lines for a sub-agent container (diagnose MCP-registration failures, startup crashes, etc.). Pass the plain logical agent name; hive-c0re resolves the machine name (h-<name>, manager hm1nd). lines defaults to 50, host-capped at 500.
• request_schedule_prompt(targets, body, first_fire_at_unix, interval_seconds?, description?) — queue an operator-approval for a scheduled prompt. On approve, body is fanned out to each targets agent at first_fire_at_unix; recurring if interval_seconds is set, one-shot otherwise. Even self-targeted schedules go through approval (use remind for unapproved self-wake). Long downtime fires once per recurring row on resume (catch-up clamp).
• edit_schedule(id, body?, description?, interval_seconds?, next_fire_at_unix?, targets_add?, targets_remove?) — partial-update a schedule (#474/#478). Pass only the fields to change; absent fields are left alone. targets_add / targets_remove mutate the recipient list in the same transaction; re-adding a previously-cancelled target drops its tombstone + history (fresh start). Clearing a scalar (e.g. interval_seconds: null) flips recurring→one-shot. Refuses cancelled rows. Same authorization as cancel_schedule.
• cancel_schedule(id, targets?) — cancel a schedule. Omit targets / pass empty to cancel the whole schedule; pass a list to cancel just those recipients (auto-cancels when every target is removed). Authorization: manager can cancel schedules it owns or any owned by a sub-agent in its topology subtree.
• fire_schedule_now(id) — fire a scheduled prompt out of band. Runs the per-target fan-out once immediately. Recurring schedules keep their cadence (the manual fire is additive); one-shot schedules are consumed by the fire and cancelled afterwards. Same authorization rules as cancel_schedule.
• list_schedules() — snapshot every schedule (active + cancelled-but-not-reaped): id, owner, body, per-target last_fired_at + last_result, next_fire_at_unix, interval_seconds. Use to look up an id before cancelling or to audit upcoming wake-ups across the swarm.
• remind / get_loose_ends / cancel_loose_end / set_status / get_agent_meta — same as the sub-agent tools above. get_loose_ends scopes to the manager's own items by default; pass agent: "*" for a hive-wide view, or agent: "<name>" to inspect one agent. cancel_loose_end may cancel any agent's row.

The boundary: lifecycle ops on existing sub-agents (kill/start/restart) are at the manager's discretion — no operator approval. Creating a new agent (request_init_config → request_apply_commit for the first sha) and changing any agent's config (request_apply_commit) still go through the approval queue.

Authoritative state

hive_ag3nt::events::Bus carries the current turn-loop state in addition to the broadcast channel and the events history. Variants:

• Idle — sitting on Recv waiting for mail.
• Thinking — claude --print is running for a turn.
• Compacting — operator-triggered /compact is in flight.

The harness flips state at the relevant transitions (set_state(Thinking) before drive_turn, set_state(Idle) after; set_state(Compacting) around compact_session). Exposed via /api/state.turn_state + turn_state_since (unix seconds); the agent page renders this rather than deriving from SSE events.

Tool envelope

mcp::run_tool_envelope: every MCP tool handler logs the request, runs the body, logs the result. Pre-/post-log only — the inbox status hint moved to the wake prompt + UI header.

Tool whitelist (mcp::ALLOWED_BUILTIN_TOOLS)

• Allowed built-ins: Bash, Edit, Glob, Grep, Read, Write.
• Denied by omission: WebFetch, WebSearch, Task, NotebookEdit, TodoWrite.
• Allowed MCP tools: as listed above per flavor.

By default Bash is approved wholesale — any shell command runs without confirmation. To restrict an agent to specific command families, set hyperhive.allowedBashPatterns in its agent.nix:

hyperhive.allowedBashPatterns = [ "git *" "ls *" ];

The harness reads /etc/hyperhive/bash-allow.json and replaces Bash in --allowedTools with Bash(git *) + Bash(ls *) etc. Commands outside the pattern list require confirmation — which in --print mode means they will not run. An empty list (default) keeps the current wholesale Bash entry.