From e9e4408af1d5f72eace78e45353aea8649a80d76 Mon Sep 17 00:00:00 2001 From: iris Date: Sat, 15 Aug 2026 11:51:18 +0200 Subject: [PATCH 01/64] docs(gotchas): group entries by area, fix misplaced nix-fmt section --- docs/gotchas.md | 148 +++++++++++++++++++++++++++--------------------- 1 file changed, 84 insertions(+), 64 deletions(-) diff --git a/docs/gotchas.md b/docs/gotchas.md index 6cbc7f9c..b5bf6782 100644 --- a/docs/gotchas.md +++ b/docs/gotchas.md @@ -2,8 +2,12 @@ NixOS + nspawn quirks and lessons we hit the hard way. If something here looks unmotivated in the code, there's usually a story underneath. +Grouped by area — jump to the section that matches what you're +touching. -## `nixos-container` doesn't expose `--bind` on the CLI +## NixOS / nspawn containers + +### `nixos-container` doesn't expose `--bind` on the CLI The CLI doesn't accept `--bind`. Path is via `EXTRA_NSPAWN_FLAGS` in `/etc/nixos-containers/.conf` — the start script @@ -11,18 +15,18 @@ The CLI doesn't accept `--bind`. Path is via `EXTRA_NSPAWN_FLAGS` in `systemd-nspawn` invocation. `lifecycle::set_nspawn_flags()` rewrites this line. -## `/run/systemd/nspawn/*.nspawn` overrides are ignored +### `/run/systemd/nspawn/*.nspawn` overrides are ignored `nixos-container`'s start script builds the nspawn command line directly. Dropping a `.nspawn` file under `/run/systemd/nspawn/` looks like the obvious extension point and does nothing. Use `EXTRA_NSPAWN_FLAGS` (above). -## `boot.isNspawnContainer = true` +### `boot.isNspawnContainer = true` Not `boot.isContainer = true`. Renamed in nixos-25.11+. -## `nixos-container create` auto-assigns `HOST_ADDRESS` / `LOCAL_ADDRESS` +### `nixos-container create` auto-assigns `HOST_ADDRESS` / `LOCAL_ADDRESS` …in the `.conf`. The start script's `if HOST_ADDRESS set → --network-veth` branch then forces a private netns — silently fatal @@ -30,7 +34,7 @@ for our web UIs (the bind is invisible from the host). We force-clear `HOST_ADDRESS` / `LOCAL_ADDRESS` / `HOST_ADDRESS6` / `LOCAL_ADDRESS6` / `HOST_BRIDGE` and set `PRIVATE_NETWORK=0`. -## systemd service PATH ≠ host PATH +### systemd service PATH ≠ host PATH The hive-c0re service sets `path = [ pkgs.git "/run/current-system/sw" ]`. In-container harness services do the same so anything an agent adds @@ -40,7 +44,7 @@ editing the service definition. `environment.HYPERHIVE_GIT` bakes git's absolute path in (read by `lifecycle::git_command()`) for the host. -## `systemd.services.*.path` appends `/bin` to every entry +### `systemd.services.*.path` appends `/bin` to every entry NixOS's `systemd.services..path` list feeds every entry through `lib.makeBinPath`, which **appends `/bin` unconditionally**. That's @@ -62,19 +66,21 @@ contains a non-existent directory. The first symptom is usually the setuid sudo wrapper lives at `/run/wrappers/bin/sudo` and the path entry resolves to `/run/wrappers/bin/bin` instead. -## `RuntimeDirectoryPreserve = "yes"` +### `RuntimeDirectoryPreserve = "yes"` …keeps `/run/hyperhive/` (and the per-agent sub-dirs) across hive-c0re restarts. Without it, every restart wipes bind sources and existing containers can't be started. -## `register_agent` is idempotent +### `register_agent` is idempotent Drops any prior socket task before rebinding. Required so a hive-c0re restart followed by `rebuild alice` recreates the agent's socket without needing a clean reinstall. -## `claude-code` is unfree +## Claude Code packaging & credentials + +### `claude-code` is unfree `claude-code` comes from the flake's main `nixpkgs` (nixos-26.05). It's unfree, so the agent modules set `config.allowUnfreePredicate` @@ -125,7 +131,7 @@ the hive's `claude` out from under it. The price of the root is that an old `claude-code` can't be reclaimed until every agent has rebuilt past it and the old generations are gone. -## Claude credentials are per-agent +### Claude credentials are per-agent `/var/lib/hyperhive/agents//claude/` bind-mounts to `/home//.claude` (RW). Sharing one dir across agents is NOT viable — @@ -133,7 +139,7 @@ OAuth refresh tokens rotate, so any sibling refresh invalidates all the others. Login flow runs from the per-agent web UI; creds persist across `destroy`/recreate (`--purge` wipes them). -## Persistent notes dir per agent +### Persistent notes dir per agent `/var/lib/hyperhive/agents//state/` bind-mounts to `/agents//state` (RW; uniform for all agents). @@ -143,7 +149,9 @@ durable knowledge here (`notes.md`, anything else). The harness also writes its events log here (`hyperhive-events.sqlite`). Survives `destroy`/recreate alongside the claude dir. -## Web UI ports collide on hash +## Networking & ports + +### Web UI ports collide on hash Sub-agent web UI ports are deterministic FNV-1a of the agent name modulo 900 (range 8100..8999). With ~30 agents the birthday-paradox @@ -155,7 +163,7 @@ reproducible from just the name. Every agent hashes into 8100..8999 via the same FNV-1a; dashboard at `cfg.dashboardPort` (default 7000). -## Restart races on TCP bind +### Restart races on TCP bind Both the dashboard and per-agent web UI use `tokio::net::TcpSocket` with `SO_REUSEADDR` plus a retry-on-`AddrInUse` loop (12 tries, @@ -166,22 +174,18 @@ overlap" case. REUSEADDR does **not** allow two simultaneous `LISTEN` sockets on the same port (that would be `SO_REUSEPORT`, which we don't use) — exclusivity is preserved. -## Orphan approvals +## Approvals + +### Orphan approvals If state dirs are wiped out from under a pending approval (test scripts, manual `rm -rf`), the dashboard's next render marks them `failed` with note `"agent state dir missing"` so they fall out of `pending`. They stay in sqlite for audit. -## Nix store `cp -r` preserves read-only bits +## Gateway / SPA serving -Copying a nix store path with `cp -r src/. $out/` inside a -`pkgs.runCommand` derivation preserves the read-only permissions of -store files. Any subsequent write into the copied tree (adding new -files in subdirectories) fails with `EPERM`. Fix: pass -`--no-preserve=mode,ownership` so the output tree is writable. - -## SPA fallback: use `Accept` header map, not `try_files ... /index.html` +### SPA fallback: use `Accept` header map, not `try_files ... /index.html` The naive nginx pattern for a path-prefix SPA (`try_files $uri $uri/ /matrix/index.html`) silently swallows asset 404s — a missing JS file @@ -210,7 +214,17 @@ firefox / safari are consistent). Asset fetches (`image/*`, fall through to the trailing `=404`. No extension list to maintain; no named-location indirection needed. -## `nix build flake#name` does not walk into `nixosConfigurations` +## Build & dev workflow + +### Nix store `cp -r` preserves read-only bits + +Copying a nix store path with `cp -r src/. $out/` inside a +`pkgs.runCommand` derivation preserves the read-only permissions of +store files. Any subsequent write into the copied tree (adding new +files in subdirectories) fails with `EPERM`. Fix: pass +`--no-preserve=mode,ownership` so the output tree is writable. + +### `nix build flake#name` does not walk into `nixosConfigurations` `nix build` resolves the fragment (`#name`) against the flake's **top-level output attrs** — not against `nixosConfigurations` @@ -233,12 +247,7 @@ instead of `meta#nixosConfigurations.argus.config…`. The fix: `split_once('#')` to separate flake path from name, then template `{path}#nixosConfigurations.{name}.config.system.build.toplevel`. -## `hive-forge`: prefer over raw curl pipelines - -Full CLI reference: [`docs/tools/forge.md`](tools/forge.md). -Never use raw `curl` for forge access. - -## Containerized nix-daemon needs `sandbox-fallback = true` +### Containerized nix-daemon needs `sandbox-fallback = true` Agent containers bind-mount the host's nix-daemon socket. nspawn containers don't get user-namespaces by default, so `nix build` @@ -249,7 +258,7 @@ and fail outright if the host daemon's fall back to unsandboxed local builds rather than failing. Security implications: `docs/security.md`. -## Linking workspace binaries locally needs `nix develop` +### Linking workspace binaries locally needs `nix develop` The Rust workspace links `libsqlite3-sys` (rusqlite) against the system `libsqlite3`. Agent containers carry no system libsqlite3 on @@ -271,7 +280,7 @@ e.g. `docs/tools/hivectl-cli.md` via the `hivectl markdown-docs` subcommand (its `hivectl-docs` flake check otherwise only fails in CI on drift). -## Split asset derivations away from the rust workspace +### Split asset derivations away from the rust workspace `nix/packages/assets.nix` builds the branding SVG/PNG family + claude system-prompt template + claude-settings JSON as its own derivation, @@ -285,7 +294,46 @@ The agent-configs PNG is rendered from the SVG via `rsvg-convert` at build time; librsvg dependency lives here, not in the rust derivation's `nativeBuildInputs`. -## Weston VNC compositor (per-agent `hyperhive.gui.enable`) +### `nix fmt` fails in a git worktree with "object not found" + +`nix fmt` (and any `nix` command that fetches a `git+file://` flake +URL) uses libgit2 internally to compute `revCount` — the number of +commits reachable from HEAD. This walk fails with: + +``` +error: getting Git object '': object not found (libgit2 error code = 9) +``` + +when a commit that was reachable at some earlier evaluation is now gone +(GC'd, rebased away, or pruned). The failure is persistent: clearing +`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` does not +help because the missing object is a structural gap in the git object +graph itself, not in nix's caches. + +**Workaround: use a plain clone, not a git worktree.** + +```bash +git clone http:///hyperhive/hyperhive.git ~/hh-work +cd ~/hh-work && nix fmt +``` + +The root cause is specific to worktrees: a worktree shares the object +store with its parent repo. If the parent repo's history was rewritten +(rebase, force-push, `git gc --prune`) while the worktree was checked +out at a branch tip that references the pruned commits via its reflog or +history, libgit2's rev-walk encounters the gap. A plain clone has its +own self-consistent object store and is immune to the issue. + +## Tooling + +### `hive-forge`: prefer over raw curl pipelines + +Full CLI reference: [`docs/tools/forge.md`](tools/forge.md). +Never use raw `curl` for forge access. + +## GUI (weston/VNC) + +### Weston VNC compositor (per-agent `hyperhive.gui.enable`) `nix/agent-modules/weston-vnc.nix` adds an optional Weston Wayland compositor with the VNC backend, surfaced as @@ -361,7 +409,9 @@ connects to the compositor at `127.0.0.1:`. `wl_event_source_timer_update` treats as "disarm", so the compositor never goes idle and never locks. -## Nix options reference (`nix/docs/default.nix`) +## Nix docs pipeline + +### Nix options reference (`nix/docs/default.nix`) `pkgs.nixosOptionsDoc` over two evaluated module trees: `hostEval` (a stub NixOS system loading the `nix/host-modules/` aggregator with every @@ -397,7 +447,7 @@ options tree picks up everything under that root — picking against stray roots produces an empty tree and renders the host page as template chrome with no `

` headers. -### Docs drv stability: `nixSrc` +#### Docs drv stability: `nixSrc` Naively, the docs evaluation depends on `self` (the flake's store path), so every commit — even Rust-only or frontend-only changes — produces new @@ -426,33 +476,3 @@ Why `builtins.unsafeDiscardStringContext`? The path string make `builtins.path` include `self` as a build dependency even after content-addressing the directory. Discarding the context makes the resulting `nixSrc` truly independent of `self`'s store path. - -### `nix fmt` fails in a git worktree with "object not found" - -`nix fmt` (and any `nix` command that fetches a `git+file://` flake -URL) uses libgit2 internally to compute `revCount` — the number of -commits reachable from HEAD. This walk fails with: - -``` -error: getting Git object '': object not found (libgit2 error code = 9) -``` - -when a commit that was reachable at some earlier evaluation is now gone -(GC'd, rebased away, or pruned). The failure is persistent: clearing -`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` does not -help because the missing object is a structural gap in the git object -graph itself, not in nix's caches. - -**Workaround: use a plain clone, not a git worktree.** - -```bash -git clone http:///hyperhive/hyperhive.git ~/hh-work -cd ~/hh-work && nix fmt -``` - -The root cause is specific to worktrees: a worktree shares the object -store with its parent repo. If the parent repo's history was rewritten -(rebase, force-push, `git gc --prune`) while the worktree was checked -out at a branch tip that references the pruned commits via its reflog or -history, libgit2's rev-walk encounters the gap. A plain clone has its -own self-consistent object store and is immune to the issue. From f500863f2f03673dc471396fae03da4e1d3e89f7 Mon Sep 17 00:00:00 2001 From: iris Date: Sat, 15 Aug 2026 11:51:22 +0200 Subject: [PATCH 02/64] docs(turn-loop): fix crate-count claim, trim historical framing --- docs/turn-loop/README.md | 36 ++++++++++++++---------------------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/docs/turn-loop/README.md b/docs/turn-loop/README.md index ee382050..71baa55c 100644 --- a/docs/turn-loop/README.md +++ b/docs/turn-loop/README.md @@ -65,11 +65,10 @@ agents) runs: ## Harness binary shape -Two sibling binaries out of the one `hive-ag3nt` crate, all -role-agnostic. (The earlier split into `hive-ag3nt` + `hive-m1nd` -was collapsed because the privilege boundary lives server-side at -the broker socket (`/run/hive/mcp.sock`): `ManagerRequest` calls are -refused by the standard agent socket regardless of who sends them.) +Two sibling crates, both role-agnostic (there is one role: agent — +the privilege boundary lives server-side at the broker socket +(`/run/hive/mcp.sock`), which refuses `ManagerRequest` calls regardless +of who sends them): - `hive-agent` — long-running harness loop (the inbox poll + claude-pump + ack/requeue cycle described above). @@ -80,18 +79,11 @@ refused by the standard agent socket regardless of who sends them.) transport — no per-turn stdio child (eliminates the re-registration race). -### `Surface` trait + zero-sized type tags - -`AgentRequest` / `AgentResponse` (= `ManagerRequest` / `ManagerResponse` — -type aliases) are the wire types. There is one role: agent. -`bin/hive-agent.rs` factors the turn loop through a `Surface` trait -with one zero-sized impl (`AgentSurface`) wrapping: - -- One async method per wire op: `ack_turn`, `requeue_inflight`, - `inbox_unread`, `post_turn_counts`, `send_to_parent`, `recv_next`. - -`main()` calls `serve_main::` for all roles. The turn -loop (`serve_loop` / `handle_turn`) has no per-role branches. +`hive-agent`'s wire types (`AgentRequest` / `AgentResponse`, aliased as +`ManagerRequest` / `ManagerResponse`) and its turn loop are factored +through a small `Surface` trait with one zero-sized impl, so the loop +itself has no per-role branches. See `hive-agent/src/main.rs`'s module +doc for the trait shape. ### Boot wiring @@ -103,12 +95,12 @@ opens turn-stats sqlite, prepares the on-boot files (see [claude-invocation](claude-invocation.md#on-boot-files)), installs claude plugins, spawns `web_ui::serve` + `vacuum::run`, and either drops into `serve_loop` directly (`Online`) or parks on -the login flow first (`NeedsLogin`). (The forge notification poller -used to be spawned here too; it is its own process now — -`hive-forge-notify`, see [`forge.md`](../forge.md).) +the login flow first (`NeedsLogin`). Forge notifications are polled by +their own process, not this loop — see `hive-forge-notify` in +[`forge.md`](../forge.md). -`spawn_todo_socket` opens the todos store and the socket the in-container -producers dial. Matrix / bash / forge-notify daemons and the in-process +Boot also opens the todos store and the socket in-container producers +dial. Matrix / bash / forge-notify daemons and the in-process `disk_watch` todo producer (low state-disk space) are the built-in producers, but the socket accepts any `subsystem` marker — a user-configured MCP server can push its own todos the same way. See From 5df263b2022e626de0771991aa1aeab0f786f409 Mon Sep 17 00:00:00 2001 From: iris Date: Sat, 15 Aug 2026 11:51:28 +0200 Subject: [PATCH 03/64] docs(turn-loop): drop historical framing, point to turn.rs module doc --- docs/turn-loop/claude-invocation.md | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/docs/turn-loop/claude-invocation.md b/docs/turn-loop/claude-invocation.md index f5b3df9a..0d92d3d4 100644 --- a/docs/turn-loop/claude-invocation.md +++ b/docs/turn-loop/claude-invocation.md @@ -218,9 +218,7 @@ socket at `/run/hive/` once at startup: `services.hyperhive.c0re.operatorPronouns`, default `she/her`). When `hyperhive.docs.enable` is set, `HIVE_DOCS_DIR` is present in the environment and `render()` appends a one-sentence pointer - telling the agent the docs are mounted at that path (in lieu of - the old CLAUDE.md-in-docs-dir approach, which was dropped in - favour of this direct injection). + telling the agent the docs are mounted at that path. Passed via `--system-prompt-file`. **Marker grammar.** `` opens a block; any @@ -242,14 +240,11 @@ socket at `/run/hive/` once at startup: empty-string env vars and missing env vars round-trip the same way. -The per-turn plumbing lives in `hive_ag3nt::turn`: `write_mcp_config` / -`write_system_prompt` (on-boot files), `make_session` (builds the durable -`InfiniteSession`, once), `drive_turn` (the policy state machine — -reset/auto-reset, the turn, 401-retry, deferred-compact-at-turn-end), -`run_pending_compact` (idle operator compact), `BusSink` (stream → bus + -`Telemetry` applied via `apply_telemetry`), `emit_turn_end`, `session_title` -/ `session_store` / `archive_session` (identity + turn-boundary reset). The -actual claude spawn, stream classification, and the reactive/proactive -compaction loop are in the `hive-claude` crate. Login-wait -(`wait_for_login`) lives in `hive_ag3nt::login`. +The per-turn plumbing described on this page — on-boot files, session +identity, the reset/auto-reset/retry state machine, and the +telemetry-to-bus bridge — lives in `hive-agent`'s `turn` module; see its +`//!` doc comment (`hive-agent/src/turn.rs`) for the exact call shape. +The actual claude spawn, stream classification, and the +reactive/proactive compaction loop are in the `hive-claude` crate. +Login-wait lives in `hive-agent`'s `login` module. From d14d3b25afae1bffe7afc5d55e750f91c141141f Mon Sep 17 00:00:00 2001 From: iris Date: Sat, 15 Aug 2026 11:51:33 +0200 Subject: [PATCH 04/64] docs(turn-loop): dedupe self-continue note, drop wake-CLI history --- docs/turn-loop/mcp.md | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/docs/turn-loop/mcp.md b/docs/turn-loop/mcp.md index 40fcc11b..86f6b121 100644 --- a/docs/turn-loop/mcp.md +++ b/docs/turn-loop/mcp.md @@ -95,12 +95,11 @@ at_unix_timestamp?)`. payloads spill to `/agents//state/reminders/`. Pending count capped at 50 per agent (`HIVE_REMIND_MAX_PENDING_PER_AGENT`). -There is no same-turn self-continue tool: ending the turn and letting -an external wake drive the next one is always the right move — it -checkpoints the session and observes wakes that only reach the harness -between turns. Multi-step work rides `remind` for a durable self-wake, -or an in-container todo wake (bash-task completion, forge notification, -matrix activity) for work already in flight. +There is no same-turn self-continue tool — see +[Turn outcomes](README.md#turn-outcomes) for why. Multi-step work rides +`remind` for a durable self-wake, or an in-container todo wake +(bash-task completion, forge notification, matrix activity) for work +already in flight. **Meta** (`meta` group): `set_status(text)`, `get_agent_meta(name?)`. @@ -164,11 +163,9 @@ inject a wake-up event into the agent's inbox via the per-agent socket at `/run/hive/mcp.sock`. Speak the wire protocol directly — JSON-line over the unix socket: `{"cmd":"wake","from":"matrix","body": "new dm from @alice"}\n`. Same shape as any other `AgentRequest`; see -`hive-sh4re::AgentRequest::Wake`. (An earlier `hive-agent-wake` CLI -wrapper existed for this but was removed — no shipped co-process -daemon actually shelled out to it; every one that wakes the harness -(matrix, bash) dials the socket directly, so the raw protocol is the -only path now.) +`hive-sh4re::AgentRequest::Wake`. Every built-in producer that wakes +the harness (matrix, bash) dials the socket directly — there is no +CLI wrapper, just the raw protocol. The wake event lands in the broker as `{from: