docs: sort the top-level directories into five sections by what the reader wants
Every docs directory moves under 1-getting-started, 2-handbook, 4-implementation or 5-project; the options reference (section 3) comes from the website and has no directory here. trust-boundary stays whole under the handbook for now. web-ui, swarm and tools split between handbook and implementation, and the command-line references move to 2-handbook/reference. Links, path references in code and nix, and the docs index follow the moves; no page content changes.
This commit is contained in:
iris
mara
parent
53ccbcb4b3
commit
6739bb903d
1 changed files with 736 additions and 701 deletions
|
|
@ -1,317 +0,0 @@
|
|||
# The claude invocation
|
||||
|
||||
How a turn runs on the `claude` runtime, the default
|
||||
(`services.hyperhive.agent.runtime`). An `acp` agent's turns:
|
||||
→ [`hive-runtime/README.md`](../../hive-runtime/README.md).
|
||||
|
||||
```
|
||||
claude --print --verbose --output-format stream-json --model <name> \
|
||||
--effort <level> --resume <title> # or --name <title> on first use \
|
||||
--system-prompt-file /run/hive-config/claude-system-prompt.md \
|
||||
--mcp-config /run/hive-config/claude-mcp-config.json --strict-mcp-config \
|
||||
--tools <builtins> --allowedTools <builtins+mcp>
|
||||
# wake prompt piped over stdin
|
||||
```
|
||||
|
||||
**Crate split.** `hive-agent` runs every turn through `hive-runtime`,
|
||||
whose `ClaudeRuntime` is a pass-through to the external **`hive-claude`**
|
||||
crate (a crates.io dependency, not in this repo). `hive-claude` holds the
|
||||
generic subprocess mechanics — spawning `claude --print`, streaming +
|
||||
classifying stream-json, session lookup/archive, and the durable-session
|
||||
compaction loop (`hive_claude::{Claude, InfiniteSession, Attach,
|
||||
CompactionPolicy, PercentPolicy, Telemetry, Sink, SessionStore}`).
|
||||
`hive-agent`'s `turn` module is the
|
||||
hyperhive **policy layer** on top: it builds the per-turn config from
|
||||
the bus, bridges the output stream onto the event bus (`BusSink`), and
|
||||
owns the compaction / autoreset / retry decisions in `drive_turn`. The
|
||||
lib returns everything it parsed from a turn (usage, cost, context
|
||||
window, resolved model) as `Telemetry`, which the policy layer applies
|
||||
to the bus.
|
||||
|
||||
**Which `claude` binary.** The bare name `claude`, resolved off the
|
||||
harness unit's PATH. By default that's the `claude-code` in the agent's
|
||||
own nixpkgs (the meta flake's `nixpkgs` input) via
|
||||
`environment.systemPackages`. Since that's typically a release channel and
|
||||
this package moves fast, the operator can pin one per hive with
|
||||
`services.hyperhive.deploy.hive-controller.claudeCodePackage`. `hive-priv`
|
||||
writes its store path into the container's `/etc/hyperhive-hive-facts` as
|
||||
`HYPERHIVE_CLAUDE_CODE_PATH` before every start. `claude` on PATH is a wrapper
|
||||
that runs that build when the path exists, and the container's own
|
||||
`claude-code` when the variable is unset or the path no longer exists, saying
|
||||
so on stderr in the second case. A bump reaches an agent at its next container
|
||||
start, not at a rebuild. See docs/process/gotchas.md::`claude-code` is unfree.
|
||||
|
||||
Hive-enforced settings ship at `/etc/claude-code/managed-settings.json`
|
||||
(claude-code's canonical managed-settings path — precedence #1,
|
||||
read-only, un-overridable), wired in `nix/agent-modules/claude-settings.nix`
|
||||
from the `prompts/claude-settings.json` asset. `effortLevel` is
|
||||
deliberately not in that file — the `--effort` flag
|
||||
(`HIVE_DEFAULT_EFFORT` / the swarm's effort setting) controls effort
|
||||
live, which managed scope would otherwise lock.
|
||||
|
||||
The harness reads `<name>` from `Bus::model()` on each turn.
|
||||
`services.hyperhive.agent.model` in the agent's `agent.nix` sets the initial
|
||||
default (NixOS option; propagates via `HIVE_DEFAULT_MODEL` env var; falls
|
||||
back to an existing `/harness/hyperhive-model` file, then `"haiku"`, if
|
||||
unset). The swarm changes the model at runtime, not the web
|
||||
terminal: a model or effort the swarm sets for the agent replaces the
|
||||
`Bus` value from the next turn, and the harness doesn't persist it;
|
||||
clearing it goes back to the local value →
|
||||
[`swarm/inference.md`](../swarm/inference.md).
|
||||
|
||||
`harness_state::context_window_tokens(model)` looks up
|
||||
context-window size per-model. Resolution order (first
|
||||
match wins):
|
||||
|
||||
1. The harness's per-family defaults, matched when the family name
|
||||
is a case-insensitive substring of the active model name:
|
||||
haiku=200k, sonnet=1M, opus=1M.
|
||||
2. `HIVE_CONTEXT_WINDOW_TOKENS` — single global override for any
|
||||
other model (useful in dev / test).
|
||||
3. Hard fallback: `200_000`.
|
||||
|
||||
Once a turn completes, the window the model reports
|
||||
(`modelUsage.*.contextWindow`) replaces this value.
|
||||
|
||||
The effective window drives watermarks; the harness exposes it at
|
||||
runtime via `/api/state.context_window_tokens` so the UI can show a
|
||||
percentage-of-window ctx badge.
|
||||
|
||||
**Session identity — a constant title.** Every turn keys on one fixed,
|
||||
harness-owned session title (`turn::session_title()`, default
|
||||
`hive-session`, override `HIVE_SESSION_TITLE`). The durable
|
||||
`hive_claude::InfiniteSession` (built once by the serve loop via
|
||||
`turn::make_session`, then reused) `--resume <title>`s it; the _first_ use
|
||||
(bootstrap, post-archive, post-purge) misses and the session re-runs the
|
||||
same prompt once with `--name <title>` to mint it. That single self-heal
|
||||
rule is the whole identity system — there is **no** scraped session-id
|
||||
file. Because the
|
||||
title is constant, `/compact` and its post-compact retry provably target
|
||||
the same session (killing the old "compact ran on a different/empty
|
||||
session" bug), and a `choom` invocation in the same cwd can't hijack the
|
||||
context (it won't carry that title). claude stores sessions in
|
||||
`~/.claude/projects/<cwd-slug>/<uuid>.jsonl` (bind-mounted persistently);
|
||||
`--name` writes the title into the file as a `custom-title` event, which
|
||||
is what `--resume <title>` resolves against. The harness never passes
|
||||
bare `--continue` (it resumes the _latest_ session in the cwd — the hijack
|
||||
vector). The managed settings at `/etc/claude-code/managed-settings.json`
|
||||
disable automemory and dynamic workflows (the `/workflows` feature):
|
||||
`disableWorkflows` keeps the `/workflows` machinery from spawning sub-runs
|
||||
that burn usage on the harness's autonomous turns. Claude's own native
|
||||
autocompact is **on**, at the widest window claude-code allows — see
|
||||
[Compaction](#compaction) below for why (it's a safety net, not the
|
||||
primary mechanism; hyperhive still owns that). `outputStyle` is fleet-wide
|
||||
**Concise** — output tokens are real spend on an always-on agent, and the
|
||||
preamble/recap Concise drops carries no information a turn-loop consumer
|
||||
needs.
|
||||
|
||||
**Session reset** is available via `POST /api/new-session` (the state
|
||||
badge's `new session` menu row). It does _not_ touch the session inline —
|
||||
that would race a mid-write claude process. Instead `Bus::request_session_reset()`
|
||||
sets a one-shot flag consumed at the next turn boundary by `drive_turn`,
|
||||
which **archives** the current session: it renames the backing
|
||||
`<uuid>.jsonl` to `<uuid>.jsonl.archived` (dropped out of claude's `*.jsonl`
|
||||
resolution glob, history preserved on disk, only the file carrying _our_
|
||||
title — it leaves any `choom` session sharing the cwd alone). The next
|
||||
turn's `--resume <title>` then misses and self-heals into a fresh session.
|
||||
|
||||
## Compaction
|
||||
|
||||
hyperhive owns compaction as policy: the `hive_claude::InfiniteSession`
|
||||
keeps the session alive across the context window with two triggers baked
|
||||
into its `run`:
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
- **Reactive** — claude-code prints `Prompt is too long`. The session is
|
||||
_already_ past the window, so no turn can run on it — the session
|
||||
`/compact`s straight away and retries the same wake-up prompt once. No
|
||||
notes-checkpoint turn is possible here: the detail is gone. If the retry
|
||||
_still_ overflows, `run` surfaces `Error::PromptTooLong`; `drive_turn`
|
||||
then archives the session (session lifecycle stays hive-side) and the
|
||||
serve loop requeues the message so it redelivers into a fresh session
|
||||
(see [Turn outcomes](README.md#turn-outcomes) — the wake prompt itself is small, so
|
||||
the overflow was the accumulated context the archive clears).
|
||||
- **Proactive** — a turn finishes cleanly but the last inference's context
|
||||
size crossed the policy watermark. While the session is still healthy it
|
||||
runs one synthetic _notes-checkpoint_ turn (`CHECKPOINT_PROMPT` —
|
||||
"context is filling up, flush durable state into `/state` now") and
|
||||
_then_ `/compact`s, so the agent can persist in-flight state before the
|
||||
detail collapses into a summary.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
**hive-agent checks both triggers above between its own turns** —
|
||||
`PercentPolicy::should_compact` reads the percent watermark from the
|
||||
last completed turn's telemetry (checked after `attempt()` returns), and
|
||||
the reactive path only fires once claude-code has already refused a whole
|
||||
turn as too long. Neither can see context growth happening _inside_ a
|
||||
single long turn's own tool-calling before it returns — a turn whose tool
|
||||
calls alone blow past the window hits the reactive path with no preceding
|
||||
checkpoint, losing whatever wasn't already persisted (a known, root-caused gap).
|
||||
|
||||
**Safety net for that gap: claude-code's own native autocompact is on**,
|
||||
via the managed settings (`"autoCompactEnabled": true`,
|
||||
`"autoCompactWindow": 1000000` — the CLI's documented max, so it fires only
|
||||
right before the real hard limit, not as an earlier watermark). This is
|
||||
deliberately _not_ the primary mechanism — it runs with no checkpoint and
|
||||
none of the `CHECKPOINT_PROMPT` notes-flush machinery above, so a session
|
||||
saved by it still loses whatever wasn't already durable — but it can act
|
||||
_mid-turn_, underneath `InfiniteSession::run`, which the two triggers above
|
||||
structurally can't. Don't confuse this with the `services.hyperhive.agent.autoCompact`
|
||||
nix option below: that one is a per-agent on/off for hyperhive's _own_
|
||||
proactive watermark and has no effect on claude's native mechanism, which
|
||||
is fleet-wide and set once in `hive-agent/prompts/claude-settings.json`.
|
||||
|
||||
The **when** for hyperhive's own policy is a `hive_claude::CompactionPolicy` injected by the harness:
|
||||
`turn::make_session` builds a `PercentPolicy` that compacts once the
|
||||
model-reported context fill reaches `HIVE_COMPACT_WATERMARK_PERCENT`
|
||||
(default **75%**), falling back to `harness_state::context_window_tokens(model)`
|
||||
for the window on turns the model didn't report one. `0` disables proactive
|
||||
compaction (the reactive path always applies). The proactive path is
|
||||
best-effort — a failed checkpoint or `/compact` never fails the turn that
|
||||
already succeeded. An ACP agent gets the same policy; how it compacts is in
|
||||
[`hive-runtime/README.md`](../../hive-runtime/README.md#acp-backend-compaction).
|
||||
|
||||
The operator can force a compaction any time via `POST /api/compact`. It's
|
||||
**deferred**: the handler sets `Bus::request_compact()` and returns
|
||||
immediately; the harness runs the `/compact` at the next turn boundary
|
||||
(end of the in-flight turn in `drive_turn`, or — when the agent is idle —
|
||||
in `turn::run_pending_compact` on the serve loop's next empty poll). This
|
||||
lets `/api/compact` work mid-turn instead of only when idle, without racing
|
||||
a live claude process.
|
||||
|
||||
To disable proactive compaction for a specific agent, use the nix option:
|
||||
|
||||
```nix
|
||||
services.hyperhive.agent.autoCompact = false; # default true
|
||||
```
|
||||
|
||||
Setting `autoCompact = false` sets `HIVE_COMPACT_WATERMARK_TOKENS=0`, which
|
||||
the percent resolver still honours as a disable. Useful for large-context
|
||||
models (sonnet/opus) where the 75% heuristic fires before the session is
|
||||
actually full — the reactive path (compact-on-overflow at the hard limit)
|
||||
still applies.
|
||||
|
||||
- **Auto session-reset** — a third path (`turn::maybe_auto_reset`,
|
||||
pre-turn) that fires when both conditions hold: context is ≥ a watermark
|
||||
(`HIVE_AUTO_RESET_WATERMARK_TOKENS`, default **50% of
|
||||
`context_window_tokens(model)`**) AND the time since the last turn
|
||||
exceeds the assumed prompt-cache TTL (`HIVE_CACHE_TTL_SECS`, default
|
||||
`3600`). Claude's prompt cache goes cold after a while; once it's cold,
|
||||
`--resume`-ing a large session pays the full re-upload cost with no
|
||||
benefit over starting fresh. `drive_turn` **archives** the current
|
||||
session (same mechanism as the operator reset — rename `<uuid>.jsonl` →
|
||||
`.archived`) so the next turn's `--resume <title>` misses and starts
|
||||
fresh. Unlike proactive compaction, `drive_turn` drops the session
|
||||
entirely rather than compacting it — and _no_ preceding checkpoint turn runs, because any turn
|
||||
before the reset would just re-warm the cache and defeat the purpose.
|
||||
Set `HIVE_AUTO_RESET_WATERMARK_TOKENS=0` to disable. Autoreset and the
|
||||
operator reset are mutually exclusive per turn (both archive → fresh
|
||||
turn), so an explicit operator reset short-circuits the heuristic.
|
||||
|
||||
The child runs with `cwd = /state` (when the bind exists; falls
|
||||
back to the parent's cwd in dev), so any relative path in a tool
|
||||
call (`Read foo.md`, `Bash ls`, `Write notes.md`) lands in the
|
||||
agent's durable bind-mounted dir. AGENTS.md autoload walks
|
||||
upward from `/state` — drop a per-agent AGENTS.md there if you
|
||||
want long-term hints that survive destroy/recreate. A `CLAUDE.md`
|
||||
in `/state` or above it shadows `AGENTS.md`: claude then loads only
|
||||
the `CLAUDE.md` (see `docs/agent-lifecycle/persistence.md` for the
|
||||
boot-time rename).
|
||||
|
||||
The wake prompt is intentionally minimal: the popped message's
|
||||
`from`/`body`, prefixed with a `[msg #<id>]` broker-row-id marker
|
||||
(so the agent knows what id to pass to `ack_until` for bulk triage;
|
||||
transient pings show no marker because their sentinel id 0 has nothing
|
||||
to ack), plus an inline `({unread} more pending — drain via …)` hint
|
||||
when `unread > 0`. Claude drives any further `recv`/`send` itself via
|
||||
the embedded MCP server.
|
||||
|
||||
`hive-agent` self-announces once on every boot:
|
||||
`boot_announce::announce` pushes a one-line "you were just
|
||||
(re)started, check /state/ for your notes, your session is intact"
|
||||
todo the moment the in-agent todo store opens, before the serve loop
|
||||
starts. This replaced `hive-c0re`'s old `Coordinator::kick_agent`
|
||||
calls at the tail of the rebuild and start DAG nodes — those fired a
|
||||
broker message per DAG node, so one rebuild (always a real
|
||||
`StopForUpdate` → `Swap` → `RebuildBookkeeping` → `Reconcile` →
|
||||
`Start` cycle) queued two separate restart messages for a single
|
||||
actual boot. The harness itself is the one side that knows for
|
||||
certain whether a boot happened, so it dedups for free: exactly one
|
||||
todo per real process start, surfaced the normal todo-wake way
|
||||
rather than as an inbox message.
|
||||
|
||||
## On-boot files
|
||||
|
||||
`hive-agent`'s `turn` module writes two files into its own per-service
|
||||
runtime dir, `/run/hive-config/` (`RuntimeDirectory = "hive-config"`,
|
||||
deliberately separate from `/run/hive/`, the host-owned dir holding the
|
||||
per-agent socket — so the harness owns this write surface and nothing
|
||||
needs to `chown` a bind mount), once at startup:
|
||||
|
||||
- `claude-mcp-config.json` — points claude at the persistent
|
||||
`hive-mcp-http` daemon (`http://127.0.0.1:{port}/mcp`, port from
|
||||
`services.hyperhive.agent.mcp.httpPort`, default 8790) for the built-in `hyperhive`
|
||||
surface. HTTP is the sole transport for it — no per-turn stdio child,
|
||||
so the URL survives the per-turn claude re-spawn (no re-registration
|
||||
race), trading that for a hard dependency on the daemon's uptime
|
||||
(`Restart=always`, no stdio fallback). Extra servers stay stdio.
|
||||
- `claude-system-prompt.md` — rendered from
|
||||
`hive-agent/prompts/system.md` by `hive-agent`'s `prompt::render`:
|
||||
HTML-comment markers (`<!-- role:agent -->...<!-- /role:agent -->`,
|
||||
same for `role:manager`) gate the role-specific blocks; both flavors
|
||||
share everything else. `<!-- config:repo -->` and
|
||||
`<!-- config:archetype -->` blocks gate the config-repo paragraphs: an
|
||||
agent with `HIVE_ARCHETYPE` set (the meta flake sets it on an agent built
|
||||
from an [archetype](../swarm/archetypes.md)) gets the archetype block and
|
||||
every other agent the repo block, with `{archetype}` filled in with the
|
||||
archetype's name. Five placeholders are then
|
||||
substituted: `{label}` (short agent name), `{qualified_label}`
|
||||
(hive-qualified `name@domain` form),
|
||||
`{hive_identity}` (for example `` on hive `pr1ma` ``; empty when
|
||||
`HYPERHIVE_HIVE_NAME` is unset), `{swarm_identity}` (same
|
||||
shape for the swarm), and `{operators}`: a paragraph listing the
|
||||
swarm's operators with the pronouns from each one's forge profile,
|
||||
for example `Operators: mara (she/her), sol.`. The harness reads
|
||||
them with one direct get on the one key of the `operator-profiles`
|
||||
KV bucket, which `swarm-controller` writes. The harness waits up to five seconds for
|
||||
them. When the bucket is empty or doesn't answer in time,
|
||||
`{operators}` is empty and the prompt names no operator. Either way
|
||||
the template tells agents to say "the operator" in third person. A
|
||||
pronoun change reaches an agent at its next harness start, since
|
||||
the prompt renders once per start.
|
||||
When you enable `services.hyperhive.agent.docs.enable`, `HIVE_DOCS_DIR` is present
|
||||
in the environment and `render()` appends a one-sentence pointer
|
||||
telling the agent where the docs live.
|
||||
Passed via `--system-prompt-file`.
|
||||
|
||||
**Marker grammar.** `<!-- KIND:X -->` opens a block; any
|
||||
`<!-- /KIND:X -->` closes the current block of that kind. The renderer
|
||||
filters the two kinds one after the other. The renderer always uses
|
||||
role `agent`, so it elides blocks with other role tags, and config `repo`
|
||||
or `archetype` as above. Nesting **isn't**
|
||||
supported — a stray opener with no closer runs until end of file.
|
||||
The renderer tolerates whitespace inside markers (`<!--role:foo-->` parses the
|
||||
same as `<!-- role:foo -->`). Content outside any marker is always
|
||||
included. Today's `system.md` carries no role markers (single agent
|
||||
role), and the role grammar stays wired for a future manager / multi-role
|
||||
prompt; it carries one pair of config blocks.
|
||||
|
||||
**`hive_identity` / `swarm_identity` shape.** Each carries a
|
||||
leading space + backticked name (` on hive \`pr1ma\``,
|
||||
` in swarm \`constellat1on\``) when the corresponding env var
|
||||
is set, otherwise empty string. The independence lets the
|
||||
template drop one or both into the opener prose without
|
||||
breaking single-hive deployments that never set the option;
|
||||
the renderer also treats `Some("")`from a caller as`None` so
|
||||
empty-string env vars and missing env vars round-trip the
|
||||
same way.
|
||||
|
||||
The per-turn plumbing described on this page — on-boot files, session
|
||||
identity, the reset/autoreset/retry state machine, and the
|
||||
telemetry-to-bus bridge — lives in `hive-agent`'s `turn` module; see its
|
||||
`//!` doc comment (`hive-agent/src/turn.rs`) for the exact call shape.
|
||||
The actual claude spawn, stream classification, and the
|
||||
reactive/proactive compaction loop are in the `hive-claude` crate, reached
|
||||
through `hive-runtime`'s `ClaudeRuntime`.
|
||||
Login-wait lives in `hive-agent`'s `login` module.
|
||||
Loading…
Reference in a new issue