hyperhive/CLAUDE.md

hyperhive

Multi-Claude-Code-agent orchestration on nixos-containers. A host-side Rust daemon (hive-c0re) spawns nspawn-isolated agent containers and brokers messages between them. A manager agent (hm1nd) coordinates the swarm and gates lifecycle changes on user approval via git commits, surfaced through a vibec0re-styled HTTP dashboard with live SSE message-flow.

PLAN.md is the living design doc. Read it for the why and the phase roadmap; this file is the operator/developer reference for the how.

Architecture

host (NixOS, hive-c0re.service)
│
├── hive-c0re  (Rust daemon — coordinator + dashboard + CLI)
│   ├── lifecycle      — nixos-container CRUD (spawn/kill/rebuild/list)
│   ├── broker         — sqlite message store + broadcast channel
│   ├── approvals      — sqlite approval queue
│   ├── coordinator    — shared state (broker/approvals/agent sockets)
│   ├── actions        — approve/deny (shared between admin socket & dashboard)
│   ├── server         — host admin socket (JSON line protocol)
│   ├── manager_server — manager-only privileged socket
│   ├── agent_server   — per-sub-agent sockets
│   ├── dashboard      — axum HTTP UI + SSE message-flow + approve/deny + T4LK
│   └── client         — admin-socket client (powers `hive-c0re spawn|kill|…`)
│
├── /run/hyperhive/
│   ├── host.sock                — admin CLI ↔ daemon
│   ├── manager.sock             → hm1nd container at /run/hive/mcp.sock
│   └── agents/<name>/mcp.sock   → h-<name> container at /run/hive/mcp.sock
│
├── /var/lib/hyperhive/
│   ├── broker.sqlite            — messages + approvals tables
│   ├── agents/<name>/config/    — proposed repo (manager-editable, RO to hive-c0re)
│   └── applied/<name>/          — applied repo (hive-c0re-only, container builds here)
│
└── nixos-containers
    ├── h-<name>  (sub-agents, hive-ag3nt binary)
    └── hm1nd     (manager, hive-m1nd binary)

Crates / file map

hive-c0re/         host daemon + CLI (one binary, subcommand-dispatched)
  src/main.rs           clap setup; serve / spawn / kill / rebuild / list /
                         pending / approve / deny
  src/server.rs         host admin socket (HostRequest → dispatch)
  src/client.rs         admin-socket client
  src/manager_server.rs manager-privileged socket (ManagerRequest)
  src/agent_server.rs   per-sub-agent socket listener
  src/broker.rs         sqlite Message store + broadcast channel for SSE
  src/approvals.rs      sqlite Approval queue
  src/coordinator.rs    shared state (broker/approvals/agent_flake/sockets)
  src/actions.rs        approve/deny (admin socket + dashboard both call in)
  src/lifecycle.rs      `nixos-container` shellouts, per-agent flake generator,
                         systemd drop-ins, git helpers, agent_web_port hash
  src/dashboard.rs      axum HTTP UI: containers list, T4LK form, approvals
                         (diff + Approve/Deny buttons), SSE message flow

hive-ag3nt/        in-container harness crate; produces TWO binaries
  src/lib.rs            DEFAULT_SOCKET, DEFAULT_WEB_PORT, re-exports
  src/client.rs         generic JSON-line request/response over unix socket
  src/web_ui.rs         per-container axum HTTP page (label + placeholder)
  src/bin/hive-ag3nt.rs sub-agent CLI (serve/send/recv); turn loop + web UI
  src/bin/hive-m1nd.rs  manager CLI (serve/send/recv/spawn/kill/
                         request-apply-commit); recognises HelperEvent

hive-sh4re/        wire types (HostRequest/Response, AgentRequest/Response,
                   ManagerRequest/Response, Message, Approval, HelperEvent)

nix/
  modules/hive-c0re.nix       systemd service + firewall + git path wiring
  templates/agent-base.nix    sub-agent nixos-container template
  templates/manager.nix       manager nixos-container template

tests/roundtrip.sh   Phase 3 messaging round-trip
tests/approval.sh    Phase 5 end-to-end approval flow
tests/dashboard.sh   Phase 6+7 HTTP dashboard + SSE + orphan GC

docs/damocles-migration.md   options for moving damocles onto hyperhive

Conventions

• Naming. Containers are length-bounded (nixos-container ≤ 11 chars). Sub-agents are h-<name> with <name> ≤ 9 chars; the manager is hm1nd. MAX_AGENT_NAME enforces the cap in lifecycle.rs. Per-agent web UI port = WEB_PORT_BASE + FNV1a(name) % WEB_PORT_RANGE (8100..8999); manager fixed at 8000; dashboard cfg.dashboardPort (default 7000).
• Identity = socket. No auth/tokens on the per-agent sockets. The socket path identifies the principal; perms come from "who has the bind-mount."
• Wire protocol. JSON line-delimited over unix sockets in both directions (host admin / manager / agent). /messages/stream is text/event-stream.
• Commit messages. Short, lowercase, no Co-Authored-By trailer.
• Commit before test. Stage and commit when work looks ready, then run validation (cargo check, nix flake check, real lpt2 deploy). Failures get a follow-up commit rather than an amend.
• rebuild is the reconcile verb. Idempotently rewrites /etc/nixos-containers/<C>.conf (PRIVATE_NETWORK=0, clears HOST_ADDRESS/LOCAL_ADDRESS, sets EXTRA_NSPAWN_FLAGS), regenerates applied/<name>/flake.nix, writes the systemd limits drop-in, then nixos-container update + stop + start. Anything that changes per-container state on the host should be re-applied here.
• Actions are factored. approve / deny live in actions.rs; the admin socket and the dashboard POST handlers both call into them, so the two surfaces never drift.

Gotchas / lessons learned

• nixos-container doesn't expose --bind on the CLI. Path is via EXTRA_NSPAWN_FLAGS in /etc/nixos-containers/<NAME>.conf — the start script (/nix/store/.../container_-start) expands it unquoted into the systemd-nspawn invocation. We rewrite this line in set_nspawn_flags().
• /run/systemd/nspawn/*.nspawn overrides are ignored by nixos-container's start script (it builds the nspawn cmd line directly).
• boot.isNspawnContainer = true, not boot.isContainer = true. Renamed in nixos-25.11+.
• nixos-container create auto-assigns HOST_ADDRESS/LOCAL_ADDRESS in the .conf. The start script's if HOST_ADDRESS set → --network-veth branch then forces a private netns — which is silently fatal for our web UIs (the bind is invisible from the host). We force-clear those vars (and HOST_ADDRESS6 / LOCAL_ADDRESS6 / HOST_BRIDGE) plus set PRIVATE_NETWORK=0.
• systemd service PATH ≠ host PATH. Our service explicitly sets path = [ pkgs.git "/run/current-system/sw" ]. Additionally, environment.HYPERHIVE_GIT = "${pkgs.git}/bin/git" bakes the absolute path in (read by lifecycle::git_command()) so git resolution doesn't depend on PATH plumbing at all.
• RuntimeDirectoryPreserve = "yes" keeps /run/hyperhive/ (and the per-agent sub-dirs) across hive-c0re restarts. Without it, every restart wipes bind sources and existing containers can't be started.
• register_agent is idempotent — drops any prior socket task before rebinding. Required so a hive-c0re restart followed by rebuild alice recreates the agent's socket without needing a clean reinstall.
• claude-code is unfree. agent-base.nix allow-list's it specifically. The flake pins it to nixpkgs-unstable via overlays.claude-unstable (stable lags too far). The overlay imports unstable with its own allowUnfreePredicate so the access inside the overlay doesn't itself trip.
• Claude credentials are stateful and per-container. No ANTHROPIC_API_KEY env var path. Today's stopgap: nixos-container root-login h-<name> → claude (interactive) → log in once. The harness falls back to echo replies when claude --print fails. Phase 8 moves this to a per-agent persistent dir at /var/lib/hyperhive/agents/<name>/claude/ bind-mounted into the container, with the interactive login driven from the agent's web UI. Sharing one ~/.claude across agents is NOT viable — OAuth refresh tokens rotate, so any sibling refresh invalidates all the others.
• Echo guard. hive-ag3nt serve skips auto-reply when the incoming body starts with "echo: ". Prevents ping-pong loops when both sides fall back to echo. Real conversations between claude-backed agents will runaway — bounding them is the manager's job.
• Orphan approvals. If state dirs are wiped out from under a pending approval (test scripts, manual rm -rf), the dashboard's next render marks them failed with note "agent state dir missing" so they fall out of pending. They stay in sqlite for audit.

Agent MCP surface + turn loop

The harness ships an embedded MCP server (rmcp 1.7) that claude launches as a stdio child via --mcp-config. Subcommand: hive-ag3nt mcp. Tools:

• mcp__hyperhive__send(to, body) — message a peer or the operator.
• mcp__hyperhive__recv() — drain one inbox message.

Both translate to AgentRequest::Send/Recv against the agent's own /run/hive/mcp.sock (the existing hyperhive socket). The MCP surface is just claude's view of that socket — same authority, friendlier protocol.

The turn loop in hive-ag3nt serve writes /run/hive/claude-mcp-config.json at boot pointing at /proc/self/exe mcp (the running hive-ag3nt binary's nix store path). Each turn invokes:

claude --print --mcp-config <path> --tools <builtins> --allowedTools <builtins+mcp> <prompt>

Tool whitelist (see ALLOWED_BUILTIN_TOOLS in hive-ag3nt::mcp):

• Allowed built-ins: Bash, Edit, Glob, Grep, NotebookEdit, Read, TodoWrite, Write.
• Denied by omission: WebFetch, WebSearch, Task — no external egress or nested-agent spawning until we have a real policy story.
• Allowed MCP tools: mcp__hyperhive__send, mcp__hyperhive__recv.

Bash is on the allow-list "for now" — pending a finer-grained allow-list system for command patterns (Bash(git *)-style). When that lands, the builtin_tools_arg shape will probably change to a setting / hooks combo per claude-code's permissions plumbing.

Manager will get its own subcommand later with request_spawn, kill, request_apply_commit added to the TOOLS list.

Manager (hm1nd) is hive-c0re-managed

The manager container runs through the same lifecycle as sub-agents — no separate code path. On hive-c0re serve startup, if nixos-container list doesn't include hm1nd, hive-c0re creates it. The manager's flake lives at /var/lib/hyperhive/applied/hm1nd/; its proposed (manager-editable) config at /var/lib/hyperhive/agents/hm1nd/config/. Manager can edit its own agent.nix (visible inside the container at /agents/hm1nd/config/), commit, and submit request-apply-commit hm1nd <sha> for operator approval — same flow as for sub-agents.

Differences from sub-agents:

• flake.nix extends hyperhive.nixosConfigurations.manager (vs agent-base).
• Container name is hm1nd (no h- prefix).
• Fixed web UI port (MANAGER_PORT = 8000).
• set_nspawn_flags adds an extra bind: /var/lib/hyperhive/agents → /agents (RW), so the manager can edit per-agent proposed repos.
• First-deploy spawn bypasses the approval queue (manager is required infrastructure).
• Per-agent socket is the manager socket at /run/hyperhive/manager/, owned by manager_server::start. coordinator::ensure_runtime returns that path for manager and the usual /run/hyperhive/agents/<name>/ for the rest.

Migration note: drop any containers.hm1nd = { ... } block from your host NixOS config. hyperhive creates and updates the manager itself now.

Auto-update on startup

hive-c0re serve runs auto_update::run in a background task right after opening the coordinator. It enumerates managed containers and rebuilds any whose recorded hyperhive rev differs from the current one:

• Sub-agents rebuild via lifecycle::rebuild (regenerates applied/<name>/flake.nix, sets nspawn flags, nixos-container update --flake).
• Manager runs nixos-container update hm1nd (no --flake). The manager's config lives in the host's NixOS module; this is belt-and-braces on top of NixOS's own container activation. Idempotent when nothing has actually changed.

"Rev" = canonical filesystem path of cfg.hyperhiveFlake (so /etc/hyperhive resolving to a new /nix/store/...-source triggers a rebuild). Marker file: /var/lib/hyperhive/applied/.<name>.hyperhive-rev. If the flake input has no canonical path (e.g. a github: URL), auto-update is a no-op — rebuild manually. The task is async and never blocks the admin socket; failures are logged and don't take the daemon down.

The dashboard surfaces pending updates per agent: a clickable "needs update ↻" badge appears whenever the marker differs from current rev. The badge POSTs /rebuild/<name>, calling the same auto_update::rebuild_agent / rebuild_manager path so manual triggers and the startup scan can't drift.

Build / deploy / test

# inside the repo (devshell first; no global cargo)
nix develop -c cargo check
nix develop -c cargo clippy --workspace --all-targets -- -D warnings
nix develop -c cargo build

# evaluate everything (incl. rust+nix+toml fmt + clippy)
nix flake check

# build only the workspace package
nix build .#default
./result/bin/{hive-c0re,hive-ag3nt,hive-m1nd}

# deploy to an existing host that imports hyperhive.nixosModules.hive-c0re
cd ~/Repos/<nixos-config-repo>
nix flake update --update-input hyperhive
sudo nixos-rebuild switch --flake .#<host>
sudo systemctl restart hive-c0re   # if only env/options changed

# end-to-end tests (each idempotent; runs as root)
sudo bash tests/roundtrip.sh    # alice ↔ bob echo round-trip
sudo bash tests/approval.sh     # manager edit → request → user approve → rebuilt
sudo bash tests/dashboard.sh    # HTTP UI, approve POST, SSE, orphan GC

The host config also needs hyperhive.overlays.default applied — the module's default package = pkgs.hyperhive requires the overlay to bring the package in. The claude-unstable overlay is applied internally to per-agent flakes already.

Phase status

• ✅ Phase 0 — repo + Cargo workspace + flake + agent-base template
• ✅ Phase 1 — container lifecycle; nixos-container update hot-reload works under the patch stack (validated on muede-lpt2)
• ✅ Phase 2 — per-agent sockets, in-memory broker, agent harness round-trips
• ✅ Phase 3 — sqlite broker (durable) + claude-or-echo turn loop
• ✅ Phase 4 — hm1nd manager binary + manager socket + declarative containers.hm1nd
• ✅ Phase 5 — git-commit approval flow
  • 5a — sqlite approval queue (request_apply_commit/pending/approve/deny)
  • 5b — per-agent config flakes
  • 5c — manager edits proposed, hive-c0re writes-only applied; container builds from applied. Approve = read agent.nix at the approved commit from proposed, copy into applied, commit + rebuild. Manager cannot move applied/main on its own.
• ✅ Phase 6 — per-container web UIs (HIVE_PORT deterministic-hash) + hive-c0re dashboard (default 7000, vibec0re aesthetic, deep-linked)
• ✅ Phase 7 — polish:
  • 7a — dashboard Approve/Deny buttons + unified diff (similar crate)
  • 7b — broker broadcast + /messages/stream SSE + live message-flow panel
  • 7c — ApprovalResolved helper events into manager inbox
  • 7d — MemoryMax=2G + CPUQuota=50% systemd drop-in per container
  • 7e — damocles migration plan (docs/damocles-migration.md)
• ✅ Phase 7 follow-ups:
  • Dashboard T4LK form — operator can send messages from the browser (POST /send, becomes from: "operator" broker message)
  • Orphan-approval GC on dashboard render (stale entries auto-failed)
  • PRIVATE_NETWORK=0 + HOST_ADDRESS=/LOCAL_ADDRESS= cleared in set_nspawn_flags so sub-agent web UI ports are reachable on the host
  • HYPERHIVE_GIT env var (absolute path) bypasses PATH ambiguity

Phase 8 — real claude in containers + login UX (in progress)

See PLAN.md → "Phase 8" for the full design. Summary:

• Per-agent persistent creds dir. Bind /var/lib/hyperhive/agents/<name>/claude/ → /root/.claude (RW) in set_nspawn_flags. One OAuth lineage per agent; refresh rotations stay contained to that agent.
• State dirs persist by default. destroy keeps /var/lib/hyperhive/agents/<name>/ unless the operator passes an explicit wipe flag. Recreating an agent of the same name reuses prior creds.
• First spawn is approval-gated. New agent names go through the same approval queue as config edits. Manager calls RequestSpawn (CLI: hive-m1nd request-spawn <name>); operator can also queue from the dashboard or hive-c0re request-spawn <name>. The host's direct hive-c0re spawn <name> still works as a privileged bypass for tests. Approve runs lifecycle::spawn in a background task; the dashboard polls via <meta refresh> and renders a spinner row while nixos-container create + update + start is in flight.
• "needs login" partial-run state. No valid session in ~/.claude/ → harness binds the web UI but does NOT start the turn loop. The harness polls the dir; as soon as a login lands it transitions into the turn loop without a restart. Dashboard surfaces the state per-agent via a needs login badge in the container list. "Valid session" today is a heuristic (any regular file inside /root/.claude/); we may refine once the filename layout claude writes is locked in.
• Login from the per-agent web UI. Spawn claude auth login with plain stdio pipes (no PTY initially), surface the OAuth URL from stdout on the page, accept the resulting code via a paste field, write it to the process stdin. Once ~/.claude/ populates, the existing needs-login polling loop flips state to Online and starts the turn loop — no separate signaling needed. The exact command is overridable via HYPERHIVE_LOGIN_CMD so we can adjust without rebuilding. If pipes turn out to be insufficient (claude refuses without a TTY, raw-mode input, ANSI-only output) we redo the backend with a PTY (e.g. portable-pty).

Implementation order: bind-mount/dir creation → approval-gated spawn + spinner → "needs login" partial run → PTY login endpoint. The login UI has nowhere to live until the partial-run mode exists, so don't ship it earlier.

Approval flow

End-to-end: manager edits per-agent proposed repo → commits → submits commit sha → user approves on host CLI or dashboard button → hive-c0re reads the file at that sha from proposed, applies into applied, commits there, runs nixos-container update. Helper-event JSON lands in the manager's inbox.

# Inside the hm1nd container (manager has /agents bind-mounted RW):
cd /agents/alice/config
$EDITOR agent.nix              # e.g. environment.systemPackages = [ pkgs.htop ];
git commit -am "add htop"
SHA=$(git rev-parse HEAD)
hive-m1nd request-apply-commit alice $SHA
exit

# On the host (CLI):
sudo hive-c0re pending          # shows queued approval with id N
sudo hive-c0re approve N        # validates, applies, rebuilds
sudo nixos-container run h-alice -- which htop

# Or on the dashboard (browser):
http://<host>:7000/             # ◆ APPR0VE button next to the diff

Per-agent layout — two separate git repos:

/var/lib/hyperhive/agents/<name>/config/    # proposed — manager edits, hive-c0re reads only
├── .git/
└── agent.nix                               # the only file the manager can change
                                            # (initial commit by hive-c0re on first spawn,
                                            # never touched by hive-c0re again)

/var/lib/hyperhive/applied/<name>/          # applied — hive-c0re-only; container builds here
├── .git/
├── flake.nix                               # hive-c0re-managed; references hyperhive_flake
└── agent.nix                               # overwritten by approve from the proposed commit

The container's --flake ref is <applied_dir>#default. The flake's nixosConfigurations.default extends hyperhive.nixosConfigurations.agent-base with ./agent.nix plus an inline module that sets environment.etc."gitconfig".text (committer identity = the agent's name) and systemd.services.hive-ag3nt.environment.HIVE_PORT/HIVE_LABEL.

Security backlog

• Unprivileged containers (userns mapping). Today the nspawn container runs as a fully privileged root. Goal: PrivateUsersChown=yes (or the nixos-container equivalent) so uid 0 inside maps to an unprivileged uid on the host, and a container-root compromise lands the attacker on an ordinary user account, not the host's root. Requires per-agent state dirs to be chown'd to that uid on the host side.
• Bash command allow-list. Replace the blanket Bash allow with a pattern allow-list (Bash(git *), Bash(nix build .*), etc.) per claude-code's --allowedTools extended grammar. Likely lives in agent.nix so each agent can scope its own shell surface.

Per-agent settings backlog

• Model override. Hard-coded to claude-haiku-4-5-20251001 in the turn loop right now. Surface it as a per-agent override: operator via dashboard, manager via request_apply_commit setting an attr on the agent's flake (most natural place since the flake already carries per-agent env/identity).

Polish backlog

Not phased — pick when relevant:

• Operator inbox view — drain replies addressed to operator and show in the dashboard (today they accumulate in sqlite unread).
• Per-agent UI substance — show last N inbox messages, last turn timing, link back to dashboard.
• xterm.js terminal — embed in each per-container UI, attach to a PTY exposed by the harness.
• destroy verb — currently nixos-container destroy + manual rm -rf. Should be one hive-c0re verb that also purges approvals + state dirs.
• Bounded broker — cap rows per recipient or auto-vacuum delivered messages older than a threshold.
• Container crash events — watch container@*.service via D-Bus, push HelperEvent::ContainerCrash to the manager.

Inspirations

• ~/Repos/bitburner-agent — sibling project, drives Claude Code in a turn loop against a Bitburner CDP session. Patterns to steal as we grow: per-cycle prompt diffing (vs full state), notes compaction as a separate short-lived Claude session, MCP server registering tools from a single TOOLS array, dashboard with SSE + xterm.js + sqlite stats sampler, opaque "terminal event" stream that unifies tool-call / sleep / op-notice / etc.