vinzenz/hyperhive
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
hyperhive/PLAN.md

21 KiB
Raw Blame History

hyperhive — Plan

Status. All phases 0–7 have shipped. This file is the original design doc; CLAUDE.md is the current source of truth for what's actually built, the file map, gotchas, and operator runbook. Keep this file for the why and the original phase rationale; CLAUDE.md for how things are today.

Names.

  • Repo: hyperhive
  • Host-side daemon (helper): hive-c0re — its own crate (daemon + CLI in one binary).
  • In-container harness: hive-ag3nt crate, with two [[bin]] targets:
    • hive-ag3nt — runs in each sub-agent container.
    • hive-m1nd — runs in the manager container (same crate, second main.rs, wires the manager tool surface).
  • Shared crate between hive-c0re and hive-ag3nt (wire protocol, MCP verb types, message shapes): hive-sh4re.

Relationship to damocles. Damocles is a separate, currently-running setup. hyperhive is a new, independent system in its own repo. Damocles' existing claude-container.nix informs the agent-base template but is not a dependency. Migration options laid out in docs/damocles-migration.md; recommendation is to keep them separate for now.

What we're building

A multi-Claude-Code-agent setup on a single host:

  • Each agent runs in its own nixos-container (the NixOS wrapper around systemd-nspawn — gives us nixos-container update, declarative configs, etc.).
  • A manager agent (itself a nixos-container) coordinates: spawns/kills agents, routes inter-agent messages, filters relevance, summarises, gates lifecycle changes on user approval.
  • Host-side Rust daemon hive-c0re owns container lifecycle, hosts the MCP transport, brokers messages, runs the dashboard, and orchestrates approval-via-git-commit.
  • Inside each agent container, hive-ag3nt runs the agent turn loop (drives claude as a subprocess), provides the per-agent web UI, and is the MCP client for that agent.
  • Inside the manager container, hive-m1nd runs the analogous loop with the manager's tool surface and /agents/** RW access. It's a second binary in the hive-ag3nt crate — same lib code, different main.rs.
  • Wire protocol and types shared between hive-c0re and the harness live in hive-sh4re.

Architecture

┌────────────────── host ────────────────────────────────────┐
│                                                             │
│  hive-c0re (Rust daemon, NixOS service)                     │
│  ├── lifecycle  : nixos-container CRUD + nixos-container    │
│  │                update on approved config commits         │
│  ├── broker     : sqlite message store + routing            │
│  ├── mcp        : per-socket MCP servers (see Sockets)      │
│  ├── approvals  : git-commit-based change requests          │
│  └── dashboard  : HTMX/SSE web UI                           │
│                                                             │
│  /var/lib/hyperhive/                                        │
│  ├── state-repo/                  (world: who exists, etc.) │
│  ├── shared-instructions/         (RO into every agent)     │
│  └── agents/                                                │
│      ├── manager/                                           │
│      │   ├── config/   (flake repo, manager's own setup)    │
│      │   ├── prompts/  (manager's role/CLAUDE.md)           │
│      │   └── state/    (RW for manager)                     │
│      └── <name>/                                            │
│          ├── config/   (flake repo, RW for manager, RO ag)  │
│          ├── prompts/  (RO inside agent)                    │
│          └── state/    (RW for agent + manager)             │
│                                                             │
│  Sockets (one per principal — perms by mount location):     │
│  ├── /run/hyperhive/host.sock          admin/CLI on host    │
│  ├── /run/hyperhive/manager.sock       → manager container  │
│  └── /run/hyperhive/agents/<name>.sock → that agent         │
│                                                             │
│  ┌─ nixos-container: hm1nd ──────────────────────┐          │
│  │  hive-m1nd (Rust, hive-ag3nt crate)           │          │
│  │   ├ MCP client → /run/hyperhive/manager.sock  │          │
│  │   ├ turn loop driving `claude`                │          │
│  │   └ per-container web UI                      │          │
│  │  /var/lib/hyperhive/agents/** bind RW         │          │
│  │  MCP tools (manager surface):                 │          │
│  │   send(to, body, wait_for_reply: bool),       │          │
│  │   recv / next_event,                          │          │
│  │   request_spawn, request_kill,                │          │
│  │   request_apply_commit, inject_peer_info,     │          │
│  │   update_shared_instructions                  │          │
│  └────────────────────────────────────────────────┘         │
│                                                             │
│  ┌─ nixos-container: h-<name> ───────────────────┐          │
│  │  hive-ag3nt (Rust)                            │          │
│  │   MCP client → /run/hyperhive/agents/<name>.sock         │
│  │  state/ RW, config/ + prompts/ + shared RO    │          │
│  │  MCP tools (agent surface):                   │          │
│  │   send, recv, request_install                 │          │
│  └────────────────────────────────────────────────┘         │
└─────────────────────────────────────────────────────────────┘

Key model decisions

Sockets, not identity gating. One unix socket per principal, bind-mounted into the right place. The socket is the principal — no SO_PEERCRED lookups, no token plumbing. Perms are filesystem perms on the host side, plus the fact that only the matching container has the bind-mount. Each socket runs the MCP tool surface appropriate to its principal.

Container naming. nixos-container caps the total container name at 11 chars (it gets encoded into network interface names). The manager runs as hm1nd (a compressed form of hive-m1nd). Sub-agents run as h-<name> with <name> capped to 9 chars (MAX_AGENT_NAME). The two namespaces don't collide. On-disk paths (/var/lib/hyperhive/agents/foo/) and socket paths (/run/hyperhive/agents/foo/mcp.sock) use the bare logical name; the prefixing lives only at the nixos-container layer.

Approvals are git commits. hive-c0re maintains a state-repo on host that records the world (which agents exist, their roles, etc.). Per-agent flake configs (agents/<name>/config/) are themselves git repos. The manager edits clones with plain git CLI inside its container and asks hive-c0re to apply a commit via request_apply_commit(agent, sha). hive-c0re queues it; once approved, fast-forwards main and reconciles state (rebuild containers, etc.). No abstract "approval token" — the commit hash is the token.

Approval UX evolves. Early phases approve by CLI (hive-c0re approve <sha>) or even direct git merge on the host. The dashboard's commit-view UI is one of the last features built (so the system runs end-to-end on CLI long before the browser does anything useful).

Manager is a second binary in the same crate. hive-m1nd lives in the hive-ag3nt crate as a second [[bin]], sharing the crate's lib code. The two binaries differ in:

  • Tool surface (manager's main.rs wires request_spawn/request_kill/request_apply_commit/inject_peer_info/update_shared_instructions).
  • Bind-mounts (manager gets agents/** RW; agents get only their own state/).
  • Auto-restart (manager is auto-restarted by hive-c0re; agents stay down once killed).
  • The manager container is declared in the host NixOS module as a known container.

Manager concurrency = event loop. hive-m1nd pulls from a heterogeneous next_event stream: inbound agent messages, replies to sync sends, lifecycle events from hive-c0re (crash, OOM, approval-resolved), and dashboard signals. One queue, claude turn per event.

Anthropic credentials. Shared key on host — revised in Phase 8. Per-agent persistent ~/.claude/ dir bind-mounted from /var/lib/hyperhive/agents/<name>/claude/. OAuth refresh tokens rotate, so sharing across agents is a non-starter (any sibling refresh invalidates all the others). One interactive login per agent, ever; creds survive destroy/recreate by default. Login flow runs from the per-agent web UI (see Phase 8).

Workdir bootstrap. Each agent's state/ starts empty. Initial-task message tells the agent what to clone/set up. Manager can drop big artefacts into state/ directly (it has RW) and pass the path as a message reference.

Riskiest assumptions (test in this order)

  1. nixos-container update hot-reloads under the in-flight nsresourced / mountfsd / privateUsers patch stack without nuking the running harness (and thus the claude session). Validated in Phase 1.
  2. The harness driving claude as a long-running turn loop is stable — claude as a subprocess streaming output back; harness deciding when a turn ends; injecting inbox messages as new user turns. Validated in Phase 3.
  3. hive-m1nd (as a claude session) sensibly drives spawn / route / peer-injection. Behavioural; only knowable by running it. Phase 4.
  4. hive-c0re throughput under multiple agents — bursty messaging, MCP relay, sqlite writes. Likely fine; flag for measurement.

Phased path

All phases ✅ shipped. Each section below is the original design with notes on what actually landed and what deviated. See CLAUDE.md → "Phase status" for the canonical summary.

✅ Phase 0 — repo bootstrap

  • Create ~/Repos/hyperhive/, init flake.
  • Cargo workspace: hive-c0re/, hive-sh4re/, hive-ag3nt/ (the last with two [[bin]] targets — hive-ag3nt and hive-m1nd). All compile, all do nothing useful.
  • NixOS module skeleton (nix/modules/hive-c0re.nix) that runs the daemon as a systemd service on the host.
  • Agent base template (nix/templates/agent-base.nix) that builds a nixos-container including the hive-ag3nt binary.
  • Exit: nixos-container create test-agent --flake .#agent-base && nixos-container start test-agent brings up a container whose hive-ag3nt prints "hello" and exits.

✅ Phase 1 — container lifecycle + Risk 1

  • hive-c0re: open host admin socket (/run/hyperhive/host.sock); verbs spawn(name), kill(name), rebuild(name), list(). Uses nixos-container underneath; container name on the host is h-<name> (sub-agents) or hm1nd (manager).
  • CLI tool talking to the admin socket (same hive-c0re binary, subcommand-driven).
  • Manually mutate an agent's config flake, call rebuild, observe whether hive-ag3nt survives.
  • Decision: if hot-reload doesn't preserve the harness, that becomes a hard requirement of hive-ag3nt's design (resume from disk state). Document the outcome.
  • Exit: spawn / rebuild / kill via CLI is reliable; known behaviour for in-flight rebuilds.

✅ Phase 2 — sockets + minimal MCP

  • hive-c0re opens manager.sock and agents/<name>.sock (one per spawned agent). Per-socket MCP server with the right tool surface baked in. Types from hive-sh4re.
  • hive-ag3nt: MCP client (types from hive-sh4re), connects to its socket on startup, exchanges hello.
  • Tools: agent gets send(to, body), recv(). No persistence yet (in-memory).
  • Exit: two test agents exchange messages through hive-c0re manually-driven.

✅ Phase 3 — broker + turn loop

  • hive-c0re: sqlite-backed message store (messages table; id, sender, recipient, body, sent_at, delivered_at). Survives hive-c0re restart.
  • hive-ag3nt (lib): real turn loop. Reads from recv; feeds new messages as user turns to claude; captures output; calls send for outbound. Long-running.
  • Exit: two hive-ag3nt-driven agents have a back-and-forth conversation through hive-c0re.

✅ Phase 4 — hive-m1nd + privileged surface

  • hive-m1nd binary (second [[bin]] in hive-ag3nt) wires the manager tool surface.
  • Manager container (hm1nd) declared in host NixOS module (auto-restart). Bind-mount agents/** RW.
  • Manager socket gets the privileged tool surface: request_spawn/request_kill, request_apply_commit, inject_peer_info, send(..., wait_for_reply=true).
  • Smoke: attach a terminal to the manager container (nixos-container root-login); ask hive-m1nd to spawn an agent and route a message to it.
  • Exit: manager spawns, routes, kills a child agent end-to-end; lifecycle still gated by manual CLI approval (no GUI yet).

✅ Phase 5 — git-commit approval flow

  • state-repo on host tracks world (agents directory listing, allow-lists, etc.).
  • Per-agent config/ flake repos created at spawn time.
  • Manager's container: bind-mounted clones; uses plain git CLI to edit/commit.
  • request_apply_commit(name, sha) queues a change in hive-c0re. Approval = CLI hive-c0re approve <sha>; on approve, hive-c0re fast-forwards main and reconciles (rebuild if config changed, run nixos-container update).
  • Per-agent allow-list for request_install: in-list installs become auto-applied commits; novel pkgs become pending commits.
  • Exit: manager adds a package to an agent → user approves on CLI → agent picks it up.

✅ Phase 6 — per-agent web UI + dashboard MVP

  • hive-ag3nt web UI module (in the crate's lib): HTTP on a per-container host port (host network): status, last messages, embedded terminal (xterm.js over WebSocket). Both hive-ag3nt and hive-m1nd binaries expose it.
  • Dashboard served by hive-c0re: agent list, per-agent status, links to each agent's UI, link to manager's UI.
  • No approval UI yet; users still approve via CLI.
  • Exit: browser is a usable navigation layer over the whole system.

✅ Phase 7 — dashboard commit view + polish

  • Pending-commits view in the dashboard with diff rendering and Approve/Deny buttons (replaces the CLI approve step).
  • Live message-flow view (hive-c0re sees all MCP relay traffic).
  • hive-c0re event push into hive-m1nd's next_event (crashes, OOM, approval resolved).
  • Resource caps in nixos-container units (MemoryMax, CPUQuota).
  • Migration plan for damocles' existing containers (separate doc).

Repo layout (target)

~/Repos/hyperhive/
├── hive-c0re/                        # host-side Rust crate (daemon + CLI)
│   ├── src/
│   │   ├── main.rs
│   │   ├── lifecycle.rs              # nixos-container CRUD
│   │   ├── broker.rs                 # sqlite message store
│   │   ├── mcp/                      # MCP servers (per-socket)
│   │   ├── approvals.rs              # git-commit change requests
│   │   ├── state_repo.rs             # world-state git repo
│   │   └── dashboard/                # HTMX/SSE web UI
│   └── Cargo.toml
├── hive-sh4re/                       # shared crate (hive-c0re ↔ hive-ag3nt)
│   ├── src/
│   │   ├── lib.rs
│   │   ├── protocol.rs               # MCP wire types, verb shapes
│   │   └── message.rs                # message types
│   └── Cargo.toml
├── hive-ag3nt/                       # in-container harness crate; two binaries
│   ├── src/
│   │   ├── lib.rs                    # shared harness code
│   │   ├── mcp_client.rs             # MCP client over unix socket
│   │   ├── turn_loop.rs              # claude subprocess driver
│   │   ├── web_ui.rs                 # per-container UI scaffolding
│   │   └── bin/
│   │       ├── hive-ag3nt.rs         # agent tool surface wiring
│   │       └── hive-m1nd.rs          # manager tool surface wiring
│   └── Cargo.toml                    # [[bin]] hive-ag3nt + [[bin]] hive-m1nd
├── nix/
│   ├── modules/hive-c0re.nix         # NixOS module for the host
│   ├── templates/
│   │   ├── agent-base.nix            # base agent nixos-container (pulls hive-ag3nt)
│   │   └── manager.nix               # manager nixos-container (pulls hive-m1nd)
│   └── overlay.nix                   # exposes the three crates + both bins as pkgs
├── web/                              # static assets, HTMX templates
├── flake.nix
├── Cargo.toml                        # workspace
└── PLAN.md                           # this file

Resolved implementation decisions

The original open-decisions list, with what we picked:

  • Wire format. Custom JSON-line over unix sockets (host admin / manager / per-agent), not real MCP stdio. Simpler and good enough for now; can swap to MCP later. SSE for the dashboard message-flow.
  • Per-agent web UI. axum HTTP server inside each container at a port hashed from the agent name (8100–8999); manager at fixed 8000; dashboard at 7000. Plain HTML, no HTMX, no xterm.js yet.
  • state-repo schema. Per-agent dir with files; not a single TOML. Realised as two parallel git repos per agent: proposed (manager-editable) and applied (hive-c0re-only). Container builds from applied.
  • Manager access to applied state. Not RW-mounted. Manager only has proposed/ bind-mounted; applied/ is hive-c0re-only.
  • One binary or two. One: hive-c0re is daemon + CLI dispatched by subcommand (serve / spawn / kill / rebuild / list / pending / approve / deny).

⏳ Phase 8 — real claude in containers + login UX

Until this lands the harness falls back to the echo path; we've never run an end-to-end turn with a real model in a real container.

Credential model. Per-agent persistent dir at /var/lib/hyperhive/agents/<name>/claude/ bind-mounted RW to /root/.claude inside the container. Not shared across agents: OAuth refresh tokens rotate, and sharing one dir means the first refresh by any sibling invalidates all the others. Each agent owns its own token lineage from first login onward.

State-dir persistence. Agent state dirs (including the claude creds dir) persist across destroy/recreate by default. The destroy verb only purges state when given an explicit "wipe" flag from the operator — recreating an agent of the same name reuses prior creds with no re-login.

First-deploy approval. Spawning a brand-new agent name goes through the existing approval queue (same path as config edits). The dashboard shows a spinner while nixos-container create + update + start run.

"needs login" agent state. If the bound ~/.claude/ has no valid session, the harness boots in a partial mode: per-agent web UI is up, but the turn loop does NOT start. Dashboard surfaces the state per-agent so the operator knows where to click.

Login over the per-agent web UI. No more nixos-container root-login for the common case. The agent's web UI exposes a "log in" action that:

  1. Spawns claude /login (or equivalent) inside the container with plain stdio pipes — no PTY unless we discover we need one.
  2. Reads the OAuth URL from the process stdout and shows it on the page.
  3. Provides a paste field for the resulting code; writes it to the process stdin.
  4. On success, transitions out of "needs login" and starts the turn loop.

If claude turns out to require a TTY (refuses on !isatty(), uses raw-mode input, or only renders the URL with ANSI styling), redo the backend with a PTY (e.g. portable-pty). Don't pre-build for that — start simple.

Sequence. Ship in this order — don't do (4) before (3) or there's nowhere for the login UI to live: (1) bind-mount + per-agent dir creation in lifecycle::set_nspawn_flags, (2) approval-gated first spawn + dashboard spinner, (3) harness "needs login" partial-run mode, (4) PTY-backed login endpoint on the per-agent UI.

Exit: spawn a new agent from the dashboard → approve → wait for spinner → click "log in" on the agent's page → complete OAuth in the browser → paste code → agent enters the turn loop and replies to a T4LK message via real claude --print.

Polish backlog (not phased)

See CLAUDE.md → "Polish backlog" for the live list. Highlights: operator inbox drain, per-agent UI substance, xterm.js terminal embed, destroy verb, bounded broker, container-crash events via D-Bus.

Explicitly deferred / out of v1 scope

  • Per-agent API keys, cost attribution.
  • Pooled / pre-warmed containers.
  • Destroy verb on the hive-c0re API (use rm on host).
  • Backup / replication of agents/ state.
  • Migration of existing damocles containers (docs/damocles-migration.md).
  • Anything about multiple hosts.