hyperhive/CLAUDE.md

hyperhive — developer reference

Operator + dev notes: conventions, gotchas, per-subsystem design.

• High-level project intro: README.md.
• Open work + backlog: TODO.md.

File map

hive-c0re/         host daemon + CLI (one binary, subcommand-dispatched)
  src/main.rs           clap setup; serve / spawn / kill / rebuild / list /
                         pending / approve / deny / destroy [--purge] /
                         request-spawn; periodic broker vacuum task
  src/server.rs         host admin socket (HostRequest → dispatch)
  src/client.rs         admin-socket client
  src/manager_server.rs manager-privileged socket (ManagerRequest)
  src/agent_server.rs   per-sub-agent socket listener (long-poll Recv)
  src/broker.rs         sqlite Message store + broadcast channel for SSE +
                         hourly vacuum of delivered>30d
  src/approvals.rs      sqlite Approval queue + kinds
  src/operator_questions.rs  sqlite question queue backing `ask_operator`
  src/coordinator.rs    shared state (broker/approvals/questions/transient/
                         sockets) + tombstone enumeration
  src/actions.rs        approve/deny/destroy (transient-aware)
  src/auto_update.rs    startup rebuild scan + ensure_manager
  src/lifecycle.rs      `nixos-container` shellouts, per-agent flake generator
  src/dashboard.rs      axum HTTP: static shell + /api/state JSON + actions
  assets/               index.html, dashboard.css, app.js (include_str!)

hive-ag3nt/        in-container harness crate; produces TWO binaries
  src/lib.rs            re-exports + DEFAULT_SOCKET, DEFAULT_WEB_PORT
  src/client.rs         generic JSON-line request/response over unix socket
  src/web_ui.rs         per-container axum HTTP page (incl /api/cancel,
                         /api/compact, /events/history)
  src/events.rs         LiveEvent + broadcast Bus + sqlite-backed history
                         (/state/hyperhive-events.sqlite) + hourly vacuum
  src/turn.rs           claude --print + stream-json pump; --compact retry
  src/mcp.rs            embedded MCP server (rmcp): AgentServer + ManagerServer
  src/login.rs          probe /root/.claude/ for a valid session
  src/login_session.rs  drives `claude auth login` over stdio pipes
  src/bin/hive-ag3nt.rs sub-agent main (Serve + Mcp subcommands)
  src/bin/hive-m1nd.rs  manager main (Serve + Mcp subcommands)
  assets/               index.html, agent.css, app.js (include_str!)
  prompts/              static role/tools/settings for claude (include_str!):
                          agent.md  — sub-agent system prompt
                          manager.md — manager system prompt
                          claude-settings.json — --settings JSON

hive-sh4re/        wire types (HostRequest/Response, AgentRequest/Response,
                   ManagerRequest/Response, Message, Approval, HelperEvent)

nix/
  modules/hive-c0re.nix         systemd service + firewall + git wiring
  templates/harness-base.nix    shared scaffolding for sub-agents + manager
  templates/agent-base.nix      sub-agent nixosConfiguration
  templates/manager.nix         manager nixosConfiguration

Conventions

• Naming. Containers are length-bounded (nixos-container ≤ 11 chars). Sub-agents are h-<name> with <name> ≤ 9 chars; the manager is hm1nd. MAX_AGENT_NAME enforces the cap in lifecycle.rs. Per-agent web UI port = WEB_PORT_BASE + FNV1a(name) % WEB_PORT_RANGE (8100..8999); manager fixed at 8000; dashboard cfg.dashboardPort (default 7000).
• Identity = socket. No auth/tokens on the per-agent sockets. The socket path identifies the principal; perms come from "who has the bind-mount."
• Wire protocol. JSON line-delimited over unix sockets in both directions (host admin / manager / agent). /messages/stream is text/event-stream.
• Commit messages. Short, lowercase, no Co-Authored-By trailer.
• Commit before test. Stage and commit when work looks ready, then run validation (cargo check, nix flake check, real lpt2 deploy). Failures get a follow-up commit rather than an amend.
• rebuild is the reconcile verb. Idempotently rewrites /etc/nixos-containers/<C>.conf (PRIVATE_NETWORK=0, clears HOST_ADDRESS/LOCAL_ADDRESS, sets EXTRA_NSPAWN_FLAGS), regenerates applied/<name>/flake.nix, writes the systemd limits drop-in, then nixos-container update + stop + start. Anything that changes per-container state on the host should be re-applied here.
• Actions are factored. approve / deny / destroy live in actions.rs; the admin socket and the dashboard POST handlers both call into them so the two surfaces never drift.
• Async forms. Dashboard + per-agent mutating forms carry data-async; a delegated submit listener in assets/app.js intercepts, shows a spinner, POSTs with application/x-www-form-urlencoded (axum Form extractor rejects multipart), calls refreshState() on success. New mutating forms should add data-async and optionally data-confirm.

Gotchas / lessons learned

• nixos-container doesn't expose --bind on the CLI. Path is via EXTRA_NSPAWN_FLAGS in /etc/nixos-containers/<NAME>.conf — the start script (/nix/store/.../container_-start) expands it unquoted into the systemd-nspawn invocation. We rewrite this line in set_nspawn_flags().
• /run/systemd/nspawn/*.nspawn overrides are ignored by nixos-container's start script (it builds the nspawn cmd line directly).
• boot.isNspawnContainer = true, not boot.isContainer = true. Renamed in nixos-25.11+.
• nixos-container create auto-assigns HOST_ADDRESS/LOCAL_ADDRESS in the .conf. The start script's if HOST_ADDRESS set → --network-veth branch then forces a private netns — which is silently fatal for our web UIs (the bind is invisible from the host). We force-clear those vars (and HOST_ADDRESS6 / LOCAL_ADDRESS6 / HOST_BRIDGE) plus set PRIVATE_NETWORK=0.
• systemd service PATH ≠ host PATH. The hive-c0re service sets path = [ pkgs.git "/run/current-system/sw" ]. In-container harness services do the same so anything an agent adds to its own agent.nix (environment.systemPackages) is visible to claude's Bash tool without editing the service definition. environment.HYPERHIVE_GIT bakes git's absolute path in (read by lifecycle::git_command()) for the host.
• RuntimeDirectoryPreserve = "yes" keeps /run/hyperhive/ (and the per-agent sub-dirs) across hive-c0re restarts. Without it, every restart wipes bind sources and existing containers can't be started.
• register_agent is idempotent — drops any prior socket task before rebinding. Required so a hive-c0re restart followed by rebuild alice recreates the agent's socket without needing a clean reinstall.
• claude-code is unfree. harness-base.nix allow-list's it specifically. The flake pins it to nixpkgs-unstable via overlays.claude-unstable (stable lags too far). The overlay imports unstable with its own allowUnfreePredicate so the access inside the overlay doesn't itself trip.
• Claude credentials are per-agent. /var/lib/hyperhive/agents/<name>/claude/ bind-mounts to /root/.claude (RW). Sharing one dir across agents is NOT viable — OAuth refresh tokens rotate, so any sibling refresh invalidates all the others. Login flow runs from the per-agent web UI; creds persist across destroy/recreate.
• Persistent notes dir per agent. /var/lib/hyperhive/agents/<name>/state/ bind-mounts to /state (RW). System prompts tell agents to keep durable knowledge here (/state/notes.md, anything else under /state/). Survives destroy/recreate alongside the claude dir.
• Orphan approvals. If state dirs are wiped out from under a pending approval (test scripts, manual rm -rf), the dashboard's next render marks them failed with note "agent state dir missing" so they fall out of pending. They stay in sqlite for audit.

Web UI shape

Both the dashboard (port 7000) and the per-agent web UIs (8000 / 8100-8999) are SPAs with the same skeleton:

• GET / → static assets/index.html (placeholders for state-driven sections).
• GET /static/*.css + GET /static/*.js → static assets shipped via include_str! so there's no runtime file dependency.
• GET /api/state → JSON snapshot the JS app renders into the DOM.
• GET /events/stream (per-agent) and GET /messages/stream (dashboard) are text/event-stream SSE for live updates.

Per-agent endpoints: POST /send, POST /login/{start,code,cancel}, POST /api/cancel, POST /api/compact, GET /events/history.

Dashboard endpoints: POST /{approve,deny}/{id}, POST /{rebuild,kill,restart,start,destroy}/{name}, POST /purge-tombstone/{name}, POST /answer-question/{id}, POST /request-spawn, POST /update-all.

The JS app handles all form[data-async] submissions via a delegated listener: read data-confirm, swap the button to a spinner, POST application/x-www-form-urlencoded (axum's Form extractor rejects multipart), then on success re-enable the button (refreshState often keeps the form mounted) and call refreshState() (re-fetch /api/state and re-render). No full-page reloads.

Per-agent + dashboard state shapes live in dashboard.rs::StateSnapshot and web_ui.rs::StateSnapshot. When adding new state fields, plumb through the snapshot struct and the relevant assets/app.js render function — never reach for server-side HTML rendering again.

Agent MCP surface + turn loop

The harness ships an embedded MCP server (rmcp 1.7) that claude launches as a stdio child via --mcp-config. Subcommand: hive-ag3nt mcp (or hive-m1nd mcp for the manager surface).

Sub-agent tools:

• mcp__hyperhive__send(to, body) — message a peer or the operator.
• mcp__hyperhive__recv() — drain one inbox message.

Manager additionally:

• mcp__hyperhive__request_spawn(name) — queue Spawn approval.
• mcp__hyperhive__kill(name) — graceful stop. No approval.
• mcp__hyperhive__start(name) — start a stopped sub-agent. No approval.
• mcp__hyperhive__restart(name) — stop + start. No approval.
• mcp__hyperhive__request_apply_commit(agent, commit_ref) — submit a config change for any agent (including hm1nd for self-mods).
• mcp__hyperhive__ask_operator(question, options?) — non-blocking; queues a question on the dashboard, returns the question id. Operator's answer arrives later as a HelperEvent::OperatorAnswered in the manager inbox.

The shared per-turn plumbing lives in hive_ag3nt::turn::{write_mcp_config, write_settings, write_system_prompt, run_turn, drive_turn, emit_turn_end, wait_for_login} so the two binaries can't drift.

On harness boot, three files get dropped next to the mcp socket at /run/hive/:

• claude-mcp-config.json — re-invokes the running binary as mcp child.
• claude-settings.json — --settings blob (auto-compact/auto-memory off, effortLevel medium).
• claude-system-prompt.md — rendered from prompts/{agent,manager}.md with {label} substituted. Passed via --system-prompt-file.

Each turn:

claude --print --verbose --output-format stream-json --model haiku \
  --continue --settings /run/hive/claude-settings.json \
  --system-prompt-file /run/hive/claude-system-prompt.md \
  --mcp-config /run/hive/claude-mcp-config.json --strict-mcp-config \
  --tools <builtins> --allowedTools <builtins+mcp>
# wake prompt piped over stdin — minimal, just from/body + optional unread hint

--continue keeps a persistent session per agent (claude stores sessions in ~/.claude/projects/, which is bind-mounted persistently). Auto-compact and auto-memory are disabled because hyperhive owns compaction (/compact on overflow, retry once).

Loop control. The harness pops one inbox message per cycle (the wake signal — Recv long-polls server-side for up to 30s waking instantly on a new broker Sent event for this agent), peeks the remaining inbox depth with Status, and emits TurnStart { from, body, unread }. The wake prompt piped to claude includes a one-line ({unread} more pending — drain via …) hint when unread > 0. Claude drives any further recv/send itself.

Tool envelope (mcp::run_tool_envelope): every MCP tool handler logs the request, runs the body, logs the result. Pre-/post-log only; the old [status] N unread message(s) appendage was removed once unread moved into the wake prompt + UI header. New tools call this helper.

Tool whitelist (mcp::ALLOWED_BUILTIN_TOOLS):

• Allowed built-ins: Bash, Edit, Glob, Grep, Read, TodoWrite, Write.
• Denied by omission: WebFetch, WebSearch, Task, NotebookEdit.
• Allowed MCP tools: as listed above per flavor.

Bash is on the allow-list pending a finer-grained pattern allow-list (Bash(git *)-style) — see TODO.md.

Live view. Each agent runs an events::Bus (broadcast channel + sqlite-backed history at /state/hyperhive-events.sqlite). The harness emits TurnStart { from, body, unread }, Stream(value) (one per parsed stream-json line), Note, TurnEnd { ok, note }. The web UI:

• fetches GET /events/history on page load and replays the last 2000 events (oldest first, .no-anim so they don't stagger),
• then subscribes to GET /events/stream (SSE) for live tail,
• shows a granular state badge above the terminal (💤 idle / 🧠 thinking / ○ offline · <age>) driven from turn_start/turn_end, with a flash animation on transition,
• sticky-bottom auto-scroll: scrolling up parks the view; new rows surface a "↓ N new" pill instead of yanking. Scrolling back to bottom clears the counter,
• terminal-themed: phosphor mauve glow, Crust bg, backdrop-filter blur, row fade-in slide-up, banner gradient shimmer while state=thinking.

Per-tool rendering:

• TurnStart → ◆ TURN ← <from> · N unread header + indented body.
• Stream tool_use → → Read /path / → Bash $ cmd / → send → operator: "..." etc., per-tool pretty rather than raw JSON.
• Stream tool_result short → flat ← ...; long → collapsed <details> ▸ ← Nl · headline (click to expand full body).
• Stream thinking → shows the thinking text if claude provided one, otherwise the bare · thinking … indicator.
• Stream system init, result, rate_limit_event are dropped — too noisy and TurnEnd already says the turn finished.
• Note → · text.
• TurnEnd → ✓ turn ok / ✗ turn fail — note and triggers a refreshState() so the page form view reflects state transitions (e.g. login just landed).

The operator input lives inside the terminal-wrap as a prompt-style textarea below the live tail: multi-line (Enter sends, Shift+Enter newlines), tab-completes slash commands. Available slash commands:

• /help — list commands locally
• /clear — wipe the visible terminal (server history kept)
• /cancel — POST /api/cancel (host shellouts pkill -INT claude, emits a Note); also surfaces as a ■ cancel turn button in the state row while state=thinking
• /compact — POST /api/compact (host spawns turn::compact_session in the background; output streams into the live panel)

Unknown /foo shows an error row instead of being silently sent.

Manager (hm1nd) is hive-c0re-managed

The manager container runs through the same lifecycle as sub-agents. On hive-c0re serve startup, if hm1nd is missing, hive-c0re creates it. The manager's flake lives at /var/lib/hyperhive/applied/hm1nd/; its proposed config at /var/lib/hyperhive/agents/hm1nd/config/. Manager can edit its own agent.nix (visible inside the container at /agents/hm1nd/config/) and submit request-apply-commit hm1nd <sha> for operator approval.

Differences from sub-agents:

• flake.nix extends hyperhive.nixosConfigurations.manager (vs agent-base).
• Container name is hm1nd (no h- prefix).
• Fixed web UI port (MANAGER_PORT = 8000).
• set_nspawn_flags adds an extra bind: /var/lib/hyperhive/agents → /agents (RW), so the manager can edit per-agent proposed repos.
• First-deploy spawn bypasses the approval queue (manager is required infrastructure).
• Per-agent socket lives at /run/hyperhive/manager/, owned by manager_server::start.

Migration note (for older hosts): drop any containers.hm1nd = { ... } block from your host NixOS config. hyperhive creates and updates the manager itself now.

Manager policy (from prompts/manager.md): the manager does NOT rubber-stamp sub-agent config requests. It verifies (role match, package legitimacy, cheaper alternative, blast radius) before committing + calling request_apply_commit. For ambiguous cases or anything that needs human signal, the manager calls ask_operator(question, options?) which queues the question on the dashboard and returns the id immediately; the operator's answer arrives later as HelperEvent::OperatorAnswered in the manager inbox. Store at hive-c0re::operator_questions (sqlite); answer flow: POST /answer-question/{id} → OperatorQuestions::answer → notify_manager(OperatorAnswered { ... }).

Helper events to the manager

Coordinator::notify_manager(&HelperEvent) enqueues an inbox message from sender system with the event JSON in the body. The manager harness no longer short-circuits these — they drive a regular claude turn so the manager can react. Variants (hive_sh4re::HelperEvent):

• ApprovalResolved { id, agent, commit_ref, status, note } — fired by actions::approve + actions::deny whenever an approval transitions to its terminal state.
• Spawned { agent, ok, note } — actions::approve (Spawn-kind) + admin HostRequest::Spawn.
• Rebuilt { agent, ok, note } — auto_update::rebuild_agent (covers startup scan + manual /rebuild from dashboard) + actions::approve (ApplyCommit).
• Killed { agent } — admin HostRequest::Kill + dashboard /kill.
• Destroyed { agent } — actions::destroy.
• OperatorAnswered { id, question, answer } — dashboard::post_answer_question fires this after the operator submits the answer form for a question the manager queued via ask_operator.

To add a new event: new HelperEvent variant + call sites + update prompts/manager.md so the manager knows the new shape.

Auto-update on startup

hive-c0re serve runs auto_update::run in a background task right after opening the coordinator. It enumerates managed containers and rebuilds any whose recorded hyperhive rev differs from the current one — sub-agents and manager go through the same lifecycle::rebuild path. "Rev" = canonical filesystem path of cfg.hyperhiveFlake. Marker file: /var/lib/hyperhive/applied/.<name>.hyperhive-rev.

If the flake input has no canonical path (e.g. a github: URL), auto-update is a no-op — rebuild manually.

The dashboard surfaces pending updates per agent: a clickable "needs update ↻" badge appears whenever the marker differs from current rev. The badge POSTs /rebuild/<name>, calling the same auto_update::rebuild_agent path so manual triggers and the startup scan can't drift. When at least one container is stale, a top-level ↻ UPD4TE 4LL button appears that loops over every stale container.

Dashboard action surface

Page sections (top to bottom):

1. C0NTAINERS — live containers with their action surface (below).
2. K3PT ST4T3 — destroyed-but-state-kept tombstones (size + age + claude-creds badge). Two actions: ⊕ R3V1V3 (queues a Spawn approval; existing state is reused), PURG3 (wipes state + applied dirs; POST /purge-tombstone/{name}).
3. M1ND H4S QU3STI0NS — pending ask_operator questions (amber pulsing border). Always renders a free-text fallback alongside any option list; multi=true renders options as checkboxes; submit merges selections + free text comma-joined.
4. 0PER4T0R 1NB0X — recent messages addressed to operator (last 50, from the broker).
5. P3NDING APPR0VALS — the queue. The R3QU3ST SP4WN form lives at the top of this section since submitting it immediately queues an approval that lands directly below.
6. MESS4GE FL0W — live broker SSE tail.

Container row (two-line layout, assets/app.js::renderContainers):

• Line 1: agent name (link → new tab), m1nd/ag3nt chip, needs login / needs update warning badges, in-flight ◐ pending-state… pill (replaces buttons during start/stop/restart/rebuild/destroy), container name + port.
• Line 2: action buttons — ↻ R3BU1LD always, DESTR0Y + PURG3 on sub-agents, ↺ R3ST4RT + (sub-agents) ■ ST0P when running, ▶ ST4RT when stopped. Buttons dim + disable while a transient lifecycle action is in flight.

↻ UPD4TE 4LL button appears above the containers list when any agent is stale.

Banner pulses on each broker SSE event (pulseBanner with a 4s grace timer).

Approval flow

End-to-end: manager edits per-agent proposed repo → commits → submits commit sha → user approves on host CLI or dashboard button → hive-c0re reads the file at that sha from proposed, applies into applied, commits there, runs nixos-container update. Helper-event JSON (ApprovalResolved) lands in the manager's inbox.

Two separate git repos per agent:

/var/lib/hyperhive/agents/<name>/config/    # proposed — manager edits, hive-c0re reads only
└── agent.nix                               # the only file the manager can change
                                            # (initial commit by hive-c0re on first spawn,
                                            # never touched by hive-c0re again)

/var/lib/hyperhive/applied/<name>/          # applied — hive-c0re-only; container builds here
├── flake.nix                               # auto-generated; references hyperhive_flake
└── agent.nix                               # overwritten by approve from the proposed commit

The container's --flake ref is <applied_dir>#default. The flake extends hyperhive.nixosConfigurations.{agent-base|manager} with ./agent.nix plus an inline module setting programs.git.config.user (committer identity = the agent's name) and systemd.services.<harness>.environment (HIVE_PORT, HIVE_LABEL, HIVE_DASHBOARD_PORT).

Persistence + retention

Two sqlite files; both autovacuum on a 1h tokio task:

• /var/lib/hyperhive/broker.sqlite (host) — messages + approvals + operator_questions tables. Broker::vacuum_delivered drops delivered messages older than 30 days; undelivered rows are always kept. Approvals + questions are kept indefinitely (auditable).
• /state/hyperhive-events.sqlite (per-container, bind-mounted from /var/lib/hyperhive/agents/<name>/state/) — every LiveEvent emitted on the per-agent Bus. Hourly vacuum drops rows older than 7 days, then trims to the most recent 2000. Path overridable via HYPERHIVE_EVENTS_DB (for dev / no-/state setups; on open failure the Bus falls back to no-store mode rather than crashing the harness). Survives destroy/recreate; gone on --purge.

State dirs (per agent, under /var/lib/hyperhive/agents/<name>/): config/ (proposed nix repo), claude/ (creds, bind-mounted RW to /root/.claude), state/ (durable notes + events db, bind-mounted to /state). Wiped only on explicit --purge. Tombstones (state dir without a live container) surface in the dashboard's K3PT ST4T3 section so the operator can either revive or purge.