vinzenz/hyperhive
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
hyperhive/CLAUDE.md

21 KiB
Raw Blame History

hyperhive

Multi-Claude-Code-agent orchestration on nixos-containers. A host-side Rust daemon (hive-c0re) spawns nspawn-isolated agent containers and brokers messages between them. A manager agent (hm1nd) coordinates the swarm and gates lifecycle changes on user approval via git commits, surfaced through a vibec0re-styled HTTP dashboard with live SSE message-flow.

PLAN.md is the living design doc. Read it for the why and the phase roadmap; this file is the operator/developer reference for the how.

Architecture

host (NixOS, hive-c0re.service)
│
├── hive-c0re  (Rust daemon — coordinator + dashboard + CLI)
│   ├── lifecycle      — nixos-container CRUD (spawn/kill/rebuild/list)
│   ├── broker         — sqlite message store + broadcast channel
│   ├── approvals      — sqlite approval queue
│   ├── coordinator    — shared state (broker/approvals/agent sockets)
│   ├── actions        — approve/deny (shared between admin socket & dashboard)
│   ├── server         — host admin socket (JSON line protocol)
│   ├── manager_server — manager-only privileged socket
│   ├── agent_server   — per-sub-agent sockets
│   ├── dashboard      — axum HTTP UI + SSE message-flow + approve/deny + T4LK
│   └── client         — admin-socket client (powers `hive-c0re spawn|kill|…`)
│
├── /run/hyperhive/
│   ├── host.sock                — admin CLI ↔ daemon
│   ├── manager.sock             → hm1nd container at /run/hive/mcp.sock
│   └── agents/<name>/mcp.sock   → h-<name> container at /run/hive/mcp.sock
│
├── /var/lib/hyperhive/
│   ├── broker.sqlite            — messages + approvals tables
│   ├── agents/<name>/config/    — proposed repo (manager-editable, RO to hive-c0re)
│   └── applied/<name>/          — applied repo (hive-c0re-only, container builds here)
│
└── nixos-containers
    ├── h-<name>  (sub-agents, hive-ag3nt binary)
    └── hm1nd     (manager, hive-m1nd binary)

Crates / file map

hive-c0re/         host daemon + CLI (one binary, subcommand-dispatched)
  src/main.rs           clap setup; serve / spawn / kill / rebuild / list /
                         pending / approve / deny
  src/server.rs         host admin socket (HostRequest → dispatch)
  src/client.rs         admin-socket client
  src/manager_server.rs manager-privileged socket (ManagerRequest)
  src/agent_server.rs   per-sub-agent socket listener
  src/broker.rs         sqlite Message store + broadcast channel for SSE
  src/approvals.rs      sqlite Approval queue
  src/coordinator.rs    shared state (broker/approvals/agent_flake/sockets)
  src/actions.rs        approve/deny (admin socket + dashboard both call in)
  src/lifecycle.rs      `nixos-container` shellouts, per-agent flake generator,
                         systemd drop-ins, git helpers, agent_web_port hash
  src/dashboard.rs      axum HTTP UI: containers list, T4LK form, approvals
                         (diff + Approve/Deny buttons), SSE message flow

hive-ag3nt/        in-container harness crate; produces TWO binaries
  src/lib.rs            DEFAULT_SOCKET, DEFAULT_WEB_PORT, re-exports
  src/client.rs         generic JSON-line request/response over unix socket
  src/web_ui.rs         per-container axum HTTP page (label + placeholder)
  src/bin/hive-ag3nt.rs sub-agent CLI (serve/send/recv); turn loop + web UI
  src/bin/hive-m1nd.rs  manager CLI (serve/send/recv/spawn/kill/
                         request-apply-commit); recognises HelperEvent

hive-sh4re/        wire types (HostRequest/Response, AgentRequest/Response,
                   ManagerRequest/Response, Message, Approval, HelperEvent)

nix/
  modules/hive-c0re.nix       systemd service + firewall + git path wiring
  templates/agent-base.nix    sub-agent nixos-container template
  templates/manager.nix       manager nixos-container template

tests/roundtrip.sh   Phase 3 messaging round-trip
tests/approval.sh    Phase 5 end-to-end approval flow
tests/dashboard.sh   Phase 6+7 HTTP dashboard + SSE + orphan GC

docs/damocles-migration.md   options for moving damocles onto hyperhive

Conventions

  • Naming. Containers are length-bounded (nixos-container ≤ 11 chars). Sub-agents are h-<name> with <name> ≤ 9 chars; the manager is hm1nd. MAX_AGENT_NAME enforces the cap in lifecycle.rs. Per-agent web UI port = WEB_PORT_BASE + FNV1a(name) % WEB_PORT_RANGE (8100..8999); manager fixed at 8000; dashboard cfg.dashboardPort (default 7000).
  • Identity = socket. No auth/tokens on the per-agent sockets. The socket path identifies the principal; perms come from "who has the bind-mount."
  • Wire protocol. JSON line-delimited over unix sockets in both directions (host admin / manager / agent). /messages/stream is text/event-stream.
  • Commit messages. Short, lowercase, no Co-Authored-By trailer.
  • Commit before test. Stage and commit when work looks ready, then run validation (cargo check, nix flake check, real lpt2 deploy). Failures get a follow-up commit rather than an amend.
  • rebuild is the reconcile verb. Idempotently rewrites /etc/nixos-containers/<C>.conf (PRIVATE_NETWORK=0, clears HOST_ADDRESS/LOCAL_ADDRESS, sets EXTRA_NSPAWN_FLAGS), regenerates applied/<name>/flake.nix, writes the systemd limits drop-in, then nixos-container update + stop + start. Anything that changes per-container state on the host should be re-applied here.
  • Actions are factored. approve / deny live in actions.rs; the admin socket and the dashboard POST handlers both call into them, so the two surfaces never drift.

Gotchas / lessons learned

  • nixos-container doesn't expose --bind on the CLI. Path is via EXTRA_NSPAWN_FLAGS in /etc/nixos-containers/<NAME>.conf — the start script (/nix/store/.../container_-start) expands it unquoted into the systemd-nspawn invocation. We rewrite this line in set_nspawn_flags().
  • /run/systemd/nspawn/*.nspawn overrides are ignored by nixos-container's start script (it builds the nspawn cmd line directly).
  • boot.isNspawnContainer = true, not boot.isContainer = true. Renamed in nixos-25.11+.
  • nixos-container create auto-assigns HOST_ADDRESS/LOCAL_ADDRESS in the .conf. The start script's if HOST_ADDRESS set → --network-veth branch then forces a private netns — which is silently fatal for our web UIs (the bind is invisible from the host). We force-clear those vars (and HOST_ADDRESS6 / LOCAL_ADDRESS6 / HOST_BRIDGE) plus set PRIVATE_NETWORK=0.
  • systemd service PATH ≠ host PATH. Our service explicitly sets path = [ pkgs.git "/run/current-system/sw" ]. Additionally, environment.HYPERHIVE_GIT = "${pkgs.git}/bin/git" bakes the absolute path in (read by lifecycle::git_command()) so git resolution doesn't depend on PATH plumbing at all.
  • RuntimeDirectoryPreserve = "yes" keeps /run/hyperhive/ (and the per-agent sub-dirs) across hive-c0re restarts. Without it, every restart wipes bind sources and existing containers can't be started.
  • register_agent is idempotent — drops any prior socket task before rebinding. Required so a hive-c0re restart followed by rebuild alice recreates the agent's socket without needing a clean reinstall.
  • claude-code is unfree. agent-base.nix allow-list's it specifically. The flake pins it to nixpkgs-unstable via overlays.claude-unstable (stable lags too far). The overlay imports unstable with its own allowUnfreePredicate so the access inside the overlay doesn't itself trip.
  • Claude credentials are stateful and per-container. No ANTHROPIC_API_KEY env var path. Today's stopgap: nixos-container root-login h-<name> → claude (interactive) → log in once. The harness falls back to echo replies when claude --print fails. Phase 8 moves this to a per-agent persistent dir at /var/lib/hyperhive/agents/<name>/claude/ bind-mounted into the container, with the interactive login driven from the agent's web UI. Sharing one ~/.claude across agents is NOT viable — OAuth refresh tokens rotate, so any sibling refresh invalidates all the others.
  • Echo guard. hive-ag3nt serve skips auto-reply when the incoming body starts with "echo: ". Prevents ping-pong loops when both sides fall back to echo. Real conversations between claude-backed agents will runaway — bounding them is the manager's job.
  • Orphan approvals. If state dirs are wiped out from under a pending approval (test scripts, manual rm -rf), the dashboard's next render marks them failed with note "agent state dir missing" so they fall out of pending. They stay in sqlite for audit.

Agent MCP surface

The harness ships an embedded MCP server (rmcp 1.7) that claude can launch via --mcp-config. Subcommand: hive-ag3nt mcp. Tools:

  • send(to, body) — message a peer or the operator.
  • recv() — drain one inbox message.

Both translate to AgentRequest::Send/Recv against the agent's own /run/hive/mcp.sock (the existing hyperhive socket). The MCP surface is just claude's view of that socket — same authority, friendlier protocol.

Manager will get its own subcommand later with request_spawn, kill, request_apply_commit added to the TOOLS list.

Manager (hm1nd) is hive-c0re-managed

The manager container runs through the same lifecycle as sub-agents — no separate code path. On hive-c0re serve startup, if nixos-container list doesn't include hm1nd, hive-c0re creates it. The manager's flake lives at /var/lib/hyperhive/applied/hm1nd/; its proposed (manager-editable) config at /var/lib/hyperhive/agents/hm1nd/config/. Manager can edit its own agent.nix (visible inside the container at /agents/hm1nd/config/), commit, and submit request-apply-commit hm1nd <sha> for operator approval — same flow as for sub-agents.

Differences from sub-agents:

  • flake.nix extends hyperhive.nixosConfigurations.manager (vs agent-base).
  • Container name is hm1nd (no h- prefix).
  • Fixed web UI port (MANAGER_PORT = 8000).
  • set_nspawn_flags adds an extra bind: /var/lib/hyperhive/agents → /agents (RW), so the manager can edit per-agent proposed repos.
  • First-deploy spawn bypasses the approval queue (manager is required infrastructure).
  • Per-agent socket is the manager socket at /run/hyperhive/manager/, owned by manager_server::start. coordinator::ensure_runtime returns that path for manager and the usual /run/hyperhive/agents/<name>/ for the rest.

Migration note: drop any containers.hm1nd = { ... } block from your host NixOS config. hyperhive creates and updates the manager itself now.

Auto-update on startup

hive-c0re serve runs auto_update::run in a background task right after opening the coordinator. It enumerates managed containers and rebuilds any whose recorded hyperhive rev differs from the current one:

  • Sub-agents rebuild via lifecycle::rebuild (regenerates applied/<name>/flake.nix, sets nspawn flags, nixos-container update --flake).
  • Manager runs nixos-container update hm1nd (no --flake). The manager's config lives in the host's NixOS module; this is belt-and-braces on top of NixOS's own container activation. Idempotent when nothing has actually changed.

"Rev" = canonical filesystem path of cfg.hyperhiveFlake (so /etc/hyperhive resolving to a new /nix/store/...-source triggers a rebuild). Marker file: /var/lib/hyperhive/applied/.<name>.hyperhive-rev. If the flake input has no canonical path (e.g. a github: URL), auto-update is a no-op — rebuild manually. The task is async and never blocks the admin socket; failures are logged and don't take the daemon down.

The dashboard surfaces pending updates per agent: a clickable "needs update ↻" badge appears whenever the marker differs from current rev. The badge POSTs /rebuild/<name>, calling the same auto_update::rebuild_agent / rebuild_manager path so manual triggers and the startup scan can't drift.

Build / deploy / test

# inside the repo (devshell first; no global cargo)
nix develop -c cargo check
nix develop -c cargo clippy --workspace --all-targets -- -D warnings
nix develop -c cargo build

# evaluate everything (incl. rust+nix+toml fmt + clippy)
nix flake check

# build only the workspace package
nix build .#default
./result/bin/{hive-c0re,hive-ag3nt,hive-m1nd}

# deploy to an existing host that imports hyperhive.nixosModules.hive-c0re
cd ~/Repos/<nixos-config-repo>
nix flake update --update-input hyperhive
sudo nixos-rebuild switch --flake .#<host>
sudo systemctl restart hive-c0re   # if only env/options changed

# end-to-end tests (each idempotent; runs as root)
sudo bash tests/roundtrip.sh    # alice ↔ bob echo round-trip
sudo bash tests/approval.sh     # manager edit → request → user approve → rebuilt
sudo bash tests/dashboard.sh    # HTTP UI, approve POST, SSE, orphan GC

The host config also needs hyperhive.overlays.default applied — the module's default package = pkgs.hyperhive requires the overlay to bring the package in. The claude-unstable overlay is applied internally to per-agent flakes already.

Phase status

  • ✅ Phase 0 — repo + Cargo workspace + flake + agent-base template
  • ✅ Phase 1 — container lifecycle; nixos-container update hot-reload works under the patch stack (validated on muede-lpt2)
  • ✅ Phase 2 — per-agent sockets, in-memory broker, agent harness round-trips
  • ✅ Phase 3 — sqlite broker (durable) + claude-or-echo turn loop
  • ✅ Phase 4 — hm1nd manager binary + manager socket + declarative containers.hm1nd
  • ✅ Phase 5 — git-commit approval flow
    • 5a — sqlite approval queue (request_apply_commit/pending/approve/deny)
    • 5b — per-agent config flakes
    • 5c — manager edits proposed, hive-c0re writes-only applied; container builds from applied. Approve = read agent.nix at the approved commit from proposed, copy into applied, commit + rebuild. Manager cannot move applied/main on its own.
  • ✅ Phase 6 — per-container web UIs (HIVE_PORT deterministic-hash) + hive-c0re dashboard (default 7000, vibec0re aesthetic, deep-linked)
  • ✅ Phase 7 — polish:
    • 7a — dashboard Approve/Deny buttons + unified diff (similar crate)
    • 7b — broker broadcast + /messages/stream SSE + live message-flow panel
    • 7c — ApprovalResolved helper events into manager inbox
    • 7d — MemoryMax=2G + CPUQuota=50% systemd drop-in per container
    • 7e — damocles migration plan (docs/damocles-migration.md)
  • ✅ Phase 7 follow-ups:
    • Dashboard T4LK form — operator can send messages from the browser (POST /send, becomes from: "operator" broker message)
    • Orphan-approval GC on dashboard render (stale entries auto-failed)
    • PRIVATE_NETWORK=0 + HOST_ADDRESS=/LOCAL_ADDRESS= cleared in set_nspawn_flags so sub-agent web UI ports are reachable on the host
    • HYPERHIVE_GIT env var (absolute path) bypasses PATH ambiguity

Phase 8 — real claude in containers + login UX (in progress)

See PLAN.md → "Phase 8" for the full design. Summary:

  • Per-agent persistent creds dir. Bind /var/lib/hyperhive/agents/<name>/claude/ → /root/.claude (RW) in set_nspawn_flags. One OAuth lineage per agent; refresh rotations stay contained to that agent.
  • State dirs persist by default. destroy keeps /var/lib/hyperhive/agents/<name>/ unless the operator passes an explicit wipe flag. Recreating an agent of the same name reuses prior creds.
  • First spawn is approval-gated. New agent names go through the same approval queue as config edits. Manager calls RequestSpawn (CLI: hive-m1nd request-spawn <name>); operator can also queue from the dashboard or hive-c0re request-spawn <name>. The host's direct hive-c0re spawn <name> still works as a privileged bypass for tests. Approve runs lifecycle::spawn in a background task; the dashboard polls via <meta refresh> and renders a spinner row while nixos-container create + update + start is in flight.
  • "needs login" partial-run state. No valid session in ~/.claude/ → harness binds the web UI but does NOT start the turn loop. The harness polls the dir; as soon as a login lands it transitions into the turn loop without a restart. Dashboard surfaces the state per-agent via a needs login badge in the container list. "Valid session" today is a heuristic (any regular file inside /root/.claude/); we may refine once the filename layout claude writes is locked in.
  • Login from the per-agent web UI. Spawn claude auth login with plain stdio pipes (no PTY initially), surface the OAuth URL from stdout on the page, accept the resulting code via a paste field, write it to the process stdin. Once ~/.claude/ populates, the existing needs-login polling loop flips state to Online and starts the turn loop — no separate signaling needed. The exact command is overridable via HYPERHIVE_LOGIN_CMD so we can adjust without rebuilding. If pipes turn out to be insufficient (claude refuses without a TTY, raw-mode input, ANSI-only output) we redo the backend with a PTY (e.g. portable-pty).

Implementation order: bind-mount/dir creation → approval-gated spawn + spinner → "needs login" partial run → PTY login endpoint. The login UI has nowhere to live until the partial-run mode exists, so don't ship it earlier.

Approval flow

End-to-end: manager edits per-agent proposed repo → commits → submits commit sha → user approves on host CLI or dashboard button → hive-c0re reads the file at that sha from proposed, applies into applied, commits there, runs nixos-container update. Helper-event JSON lands in the manager's inbox.

# Inside the hm1nd container (manager has /agents bind-mounted RW):
cd /agents/alice/config
$EDITOR agent.nix              # e.g. environment.systemPackages = [ pkgs.htop ];
git commit -am "add htop"
SHA=$(git rev-parse HEAD)
hive-m1nd request-apply-commit alice $SHA
exit

# On the host (CLI):
sudo hive-c0re pending          # shows queued approval with id N
sudo hive-c0re approve N        # validates, applies, rebuilds
sudo nixos-container run h-alice -- which htop

# Or on the dashboard (browser):
http://<host>:7000/             # ◆ APPR0VE button next to the diff

Per-agent layout — two separate git repos:

/var/lib/hyperhive/agents/<name>/config/    # proposed — manager edits, hive-c0re reads only
├── .git/
└── agent.nix                               # the only file the manager can change
                                            # (initial commit by hive-c0re on first spawn,
                                            # never touched by hive-c0re again)

/var/lib/hyperhive/applied/<name>/          # applied — hive-c0re-only; container builds here
├── .git/
├── flake.nix                               # hive-c0re-managed; references hyperhive_flake
└── agent.nix                               # overwritten by approve from the proposed commit

The container's --flake ref is <applied_dir>#default. The flake's nixosConfigurations.default extends hyperhive.nixosConfigurations.agent-base with ./agent.nix plus an inline module that sets environment.etc."gitconfig".text (committer identity = the agent's name) and systemd.services.hive-ag3nt.environment.HIVE_PORT/HIVE_LABEL.

Polish backlog

Not phased — pick when relevant:

  • Operator inbox view — drain replies addressed to operator and show in the dashboard (today they accumulate in sqlite unread).
  • Per-agent UI substance — show last N inbox messages, last turn timing, link back to dashboard.
  • xterm.js terminal — embed in each per-container UI, attach to a PTY exposed by the harness.
  • destroy verb — currently nixos-container destroy + manual rm -rf. Should be one hive-c0re verb that also purges approvals + state dirs.
  • Bounded broker — cap rows per recipient or auto-vacuum delivered messages older than a threshold.
  • Container crash events — watch container@*.service via D-Bus, push HelperEvent::ContainerCrash to the manager.

Inspirations

  • ~/Repos/bitburner-agent — sibling project, drives Claude Code in a turn loop against a Bitburner CDP session. Patterns to steal as we grow: per-cycle prompt diffing (vs full state), notes compaction as a separate short-lived Claude session, MCP server registering tools from a single TOOLS array, dashboard with SSE + xterm.js + sqlite stats sampler, opaque "terminal event" stream that unifies tool-call / sleep / op-notice / etc.