hyperhive/docs/5-project/process/gotchas.md

Gotchas

NixOS + nspawn quirks and hard-won lessons. If something here looks unmotivated in the code, there's probably a story underneath. Grouped by area — jump to the section that matches what you're touching.

NixOS / nspawn containers

nixos-container doesn't expose --bind on the CLI

The CLI doesn't accept --bind. Path is via EXTRA_NSPAWN_FLAGS in /etc/nixos-containers/<NAME>.conf — the start script (/nix/store/.../container_-start) expands it unquoted into the systemd-nspawn invocation. lifecycle::host_config::set_nspawn_flags() rewrites this line.

nixos-container ignores /run/systemd/nspawn/*.nspawn overrides

nixos-container's start script builds the nspawn command line directly. Dropping a .nspawn file under /run/systemd/nspawn/ looks like a natural extension point and does nothing. Use EXTRA_NSPAWN_FLAGS (above).

boot.isNspawnContainer = true

Not boot.isContainer = true. Renamed in nixos-25.11+.

nixos-container create autoassigns HOST_ADDRESS / LOCAL_ADDRESS

…in the .conf. The start script's if HOST_ADDRESS set → --network-veth branch then forces a private netns — silently fatal for the web UIs (the bind is invisible from the host). Every agent container runs isolated: hive-priv's write_nspawn_flags writes PRIVATE_NETWORK=1 plus a veth pair onto the host bridge, HOST_ADDRESS set to the bridge gateway IP (so nixos-container's in-container init installs a default route before the DHCP lease arrives), rest left for DHCP.

systemd service PATH ≠ host PATH

The hive-c0re service sets path = [ pkgs.git "/run/current-system/sw" ]. In-container harness services do the same so anything an agent adds to its own agent.nix (environment.systemPackages) is visible to the mcp__bash__run MCP tool (and any other in-container process) without editing the service definition. environment.HYPERHIVE_GIT bakes git's absolute path in (read by lifecycle::git_command()) for the host.

systemd.services.*.path appends /bin to every entry

NixOS's systemd.services.<unit>.path list feeds every entry through lib.makeBinPath, which appends /bin unconditionally. That's the right thing for Nix packages (their outPath is the store root, not the bin/ subdir), but it bites when you pass a string that already ends with /bin:

# ❌ /run/wrappers/bin → /run/wrappers/bin/bin (does not exist)
path = [ "/run/wrappers/bin" "/run/current-system/sw" ];

# ✅ /run/wrappers → /run/wrappers/bin  (the real wrappers dir)
path = [ "/run/wrappers" "/run/current-system/sw" ];

The bug is silent: nix eval succeeds, the unit starts, but PATH contains a non-existent directory. The first symptom is typically sudo: must be owned by uid 0 and have the setuid bit set because the setuid sudo wrapper lives at /run/wrappers/bin/sudo and the path entry resolves to /run/wrappers/bin/bin instead.

RuntimeDirectoryPreserve = "yes"

…keeps /run/hyperhive/ (and the per-agent sub-dirs) across hive-c0re restarts. Without it, every restart wipes bind sources and existing containers fail to start.

DynamicUser + RuntimeDirectoryPreserve hides the socket from nginx

With both set, systemd creates the runtime directory under /run/private/ (0700 root:root) and symlinks /run/<name> to it (systemd.exec(5), RuntimeDirectoryPreserve=). No unprivileged user can traverse that, so nginx's connect() to a socket in it fails with EACCES whatever the directory and socket modes are. A unit serving a socket to another user runs as a static user instead (swarm-controller, swarm-logs-auth, swarm-metrics-auth, swarm-metrics-agent-auth). module-eval-core-toggle fails on any unix: upstream of nginx owned by such a unit.

RestrictAddressFamilies fails as "Address family not supported by protocol"

A unit whose RestrictAddressFamilies omits a family gets EAFNOSUPPORT (errno 97) back from socket(). Clients surface that as "tcp open error: Address family not supported by protocol" — the message names the protocol and never the sandbox, so it reads like a dead network, a missing route, or an IPv6 problem.

⇒ On that error, read the unit before you touch the network.

Two things to get right when a daemon needs outbound TCP:

• list AF_INET and AF_INET6 — omitting one leaves a client that works until DNS hands back the other family;
• list AF_NETLINK too. glibc's getaddrinfo opens a netlink socket to enumerate local addresses before it returns any, so name resolution fails without it even when the unit allows AF_INET.

The directive is a claim about what the program does, and nothing re-checks it when the program changes. A unit that only served a unix socket when it was written is correct at [ "AF_UNIX" ] and silently wrong the day someone adds an HTTP client. Check the unit in the same commit as the client — and when narrowing it, prefer a test that derives the required families from the code (which fails on the next client too) over one that asserts today's list.

register_agent is idempotent

Drops any prior socket task before rebinding. Required so a hive-c0re restart followed by rebuild alice recreates the agent's socket without needing a clean reinstall.

Claude Code packaging & credentials

claude-code is unfree

claude-code comes from the flake's main nixpkgs (nixos-26.05). It's unfree, so the agent modules set config.allowUnfreePredicate at the container level to allowlist claude-code specifically — scoped, only this one package, because each per-agent nixosConfiguration evaluates its own nixpkgs instance and the operator's host-level allowUnfree does not propagate in. Operators don't need to set anything on their side. Only runtime = "claude" agents get claude and the allowance; an ACP agent whose acp.command needs claude adds pkgs.claude-code and nixpkgs.config.allowUnfreePackages = [ "claude-code" ] itself.

That same isolation is why an agent can't pick a claude out of a different nixpkgs by itself: a container only ever sees the one nixpkgs the meta flake injects, so an agent.nix naming the host's nixpkgs-unstable has nothing to name. A release channel can trail unstable by weeks on this package, which is what services.hyperhive.deploy.hive-controller.claudeCodePackage is for — set it host-side and every agent runs that build.

What crosses is the store path, not the derivation. Containers share the host's /nix/store, so the binary is already reachable inside them with its whole closure. hive-priv writes the path into each container's /etc/hyperhive-hive-facts at start, and the agent's claude wrapper on PATH runs its bin/claude. Two things rule out the natural-seeming alternatives: a path:/nix/store/<pkg> flake input is re-copied into the store as a reference-less -source (so the runtime closure never arrives), and lib.types.package fed a bare path runs builtins.storePath, which pure evaluation rejects. services.hyperhive.agent.docs.source gets away with being an input only because a docs tree has no runtime dependencies.

The storePath pitfall is worth spelling out, because it's not confined to options the operator writes: any option of type package fed a store-path string coerces through lib.toDerivation, i.e. builtins.storePath. environment.systemPackages and systemd.services.<name>.path both do it (the latter takes plain strings like /run/wrappers happily, but Nix treats anything under builtins.storeDir as a package). Wrap a path handed to the container as text in a real derivation — a symlink farm built from the interpolated string — before it can go anywhere a package is expected.

The catch is that a path handed to the container at start is text, not a reference — the container's closure doesn't keep the binary alive. The host does: it interpolates the package into /etc/hyperhive/serve.json, so it lands in the host's system closure and is gc-rooted by the running generation. builtins.toJSON preserves string context, which is the load-bearing detail; discard the context anywhere on that path and nix-collect-garbage will eventually take the hive's claude out from under it. The price of the root is that nix-collect-garbage can't reclaim an old claude-code until every agent has restarted past it and the old generations are gone.

Claude credentials are per-agent

/var/lib/hyperhive/agents/<name>/claude/ bind-mounts to /home/<name>/.claude (RW). Sharing one dir across agents isn't viable — OAuth refresh tokens rotate, so any sibling refresh invalidates all the others. Login flow runs from the per-agent web UI; creds persist across destroy/recreate (--purge wipes them).

Persistent notes dir per agent

/var/lib/hyperhive/agents/<name>/state/ bind-mounts to /agents/<name>/state (RW; uniform for all agents). The harness exposes the same path via $HYPERHIVE_STATE_DIR. System prompts tell agents to keep durable knowledge here (notes.md, anything else) — the harness's own internal files (hyperhive-events.sqlite, hyperhive-turn-stats.sqlite, hyperhive-model) live in the separate harness dir instead, so they don't clutter what claude sees as "my notes dir" (see docs/2-handbook/agent-lifecycle/persistence.md). Survives destroy/recreate alongside the claude dir.

Persistent work dir per agent

/var/lib/hyperhive/agents/<name>/work/ bind-mounts to /agents/<name>/work (RW, owned by the agent user). It's the place for git worktrees and build dirs: a claude started under state/ walks up into state/AGENTS.md and takes on the agent's identity, and one started under work/ doesn't. The container rootfs is ephemeral (nspawn --ephemeral), so /home/<name> is empty after every restart: anything the agent keeps goes in work/ or state/. work/ survives destroy/recreate and goes on --purge. The per-agent disk metric counts it; the dashboard doesn't serve it.

Networking & ports

Web UI ports collide on hash

Sub-agent web UI ports are deterministic FNV-1a of the agent name modulo 900 (range 8100..8999). With ~30 agents the birthday-paradox collision rate gets meaningful; at 2–3 agents you can still get unlucky. Operator resolves a collision by renaming the offending agent (different hash → different port) and rebuilding. No state file, no probing, no port-allocation drift — the value is reproducible from just the name. Every agent hashes into 8100..8999 via the same FNV-1a; dashboard at services.hyperhive.deploy.hive-controller.dashboardPort (default 7000).

Restart races on TCP bind

Both the dashboard and per-agent web UI use tokio::net::TcpSocket with SO_REUSEADDR plus a retry-on-AddrInUse loop (12 tries, exponential backoff capped at 2s, ~22s total). REUSEADDR handles the TIME_WAIT case from a clean previous exit; retry covers the genuine "previous process is still alive during a systemd restart overlap" case. REUSEADDR does not allow two simultaneous LISTEN sockets on the same port (that would be SO_REUSEPORT, not used here) — it preserves exclusivity.

Gateway / SPA serving

SPA fallback: use Accept header map, not try_files ... /index.html

The naive nginx pattern for an SPA (try_files $uri $uri/ /index.html) silently swallows asset 404s — a missing JS file returns index.html with a 200, so the JS runtime never loads and the page renders blank with no visible error. Extension allowlists (tried as an alternative) have the same maintenance problem: any new file extension the SPA ships breaks silently.

The pattern that works (nix/host-modules/hive-matrix.nix, serving fluffychat at the matrix gateway vhost's root) keys the fallback on the HTTP Accept header:

# Outside the server block (appendHttpConfig):
map $http_accept $matrix_spa_target {
  default          "/__matrix_spa_no_html_fallback";
  "~*text/html"    "/index.html";
}

# Inside the location:
try_files $uri $uri/ $matrix_spa_target =404;

Top-frame navigations always send Accept: text/html,... (chrome / firefox / safari are consistent). Asset fetches (image/*, application/javascript, */*) don't carry text/html, so they fall through to the trailing =404. No extension list to maintain; no named-location indirection needed.

Build & dev workflow

Nix store cp -r preserves read-only bits

Copying a nix store path with cp -r src/. $out/ inside a pkgs.runCommand derivation preserves the read-only permissions of store files. Any subsequent write into the copied tree (adding new files in subdirectories) fails with EPERM. Fix: pass --no-preserve=mode,ownership so the output tree is writable.

nix build flake#name doesn't walk into nixosConfigurations

nix build resolves the fragment (#name) against the flake's top-level output attrs — not against nixosConfigurations specifically. nixos-container and nixos-rebuild use their own internal convention that routes an agent name to nixosConfigurations.<name>.config.system.build.toplevel, but nix build has no such convention.

# ❌ silently builds the wrong thing (or errors if attr doesn't exist)
nix build "git+<forge>/internal/meta.git?rev=<rev>#argus.config.system.build.toplevel"

# ✅ explicit path nix build actually resolves
nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.config.system.build.toplevel"

Containerized nix-daemon needs sandbox-fallback = true

nixos-containers' start script bind-mounts the host's nix-daemon socket directory into every container (hyperhive adds no bind for it; an agent with services.hyperhive.agent.nixBuilds = "none" gets it shadowed, see docs/4-implementation/trust-boundary/nix-builds.md). nspawn containers don't get user-namespaces by default, so nix build invocations inside the container can't set up the build sandbox and fail outright if the host daemon's nix.settings.sandbox-fallback is false (nixpkgs default). nix/agent-modules/default.nix does lib.mkForce true so builds fall back to unsandboxed local builds rather than failing. Security implications: docs/4-implementation/trust-boundary/nix-builds.md.

Linking workspace binaries locally needs nix develop

The Rust workspace links libsqlite3-sys (rusqlite) against the system libsqlite3. Agent containers carry no system libsqlite3 on the linker path, so a plain cargo build of any binary dies with cannot find -lsqlite3 (deps and ring compile fine — only the final link fails). cargo check / cargo clippy still work in the ambient shell since they never link.

Build + run binaries through the dev shell, which carries sqlite on NIX_LDFLAGS:

nix develop -c cargo build -p hive-c0re --bin hivectl
nix develop -c cargo run -p hive-c0re --bin hivectl -- <args>

This is also how you regenerate committed generated docs locally — for example docs/2-handbook/reference/hivectl-cli.md via the hivectl markdown-docs subcommand (its hivectl-docs flake check otherwise only fails in CI on drift).

Split asset derivations away from the rust workspace

nix/packages/assets.nix builds the branding SVG/PNG family + claude system-prompt template + claude-settings JSON as its own derivation, separate from the hive-ag3nt / hive-c0re crates. Reason: when the rust build's src was the whole repo tree, any tweak to branding/agent-configs.svg or hive-agent/prompts/system.md invalidated the cargo cache and forced a full rebuild. crane (and naersk before it) couldn't see "these inputs are unused by rust" on its own — the split breaks the coupling at the derivation boundary. rsvg-convert renders the agent-configs PNG from the SVG at build time; librsvg dependency lives here, not in the rust derivation's nativeBuildInputs.

nix fmt fails in a git worktree with "object not found"

nix fmt (and any nix command that fetches a git+file:// flake URL) uses libgit2 internally to compute revCount — the number of commits reachable from HEAD. This walk fails with:

error: getting Git object '<hash>': object not found (libgit2 error code = 9)

when a commit that was reachable at some earlier evaluation is now gone (GC'd, rebased away, or pruned). The failure is persistent: clearing ~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite} doesn't help because the missing object is a structural gap in the git object graph itself, not in nix's caches.

Workaround: use a plain clone, not a git worktree.

git clone http://<forge>/hyperhive/hyperhive.git ~/hh-work
cd ~/hh-work && nix fmt

The root cause is specific to worktrees: a worktree shares the object store with its parent repo. If the parent repo's history was rewritten (rebase, force-push, git gc --prune) while the worktree remained at a branch tip that references the pruned commits via its reflog or history, libgit2's rev-walk encounters the gap. A plain clone has its own self-consistent object store and is immune to the issue.

Tooling

hive-forge: prefer over raw curl pipelines

Full CLI reference: docs/2-handbook/tools/forge.md. Never use raw curl for forge access.

GUI (weston/VNC)

Weston VNC compositor (per-agent services.hyperhive.agent.gui.enable)

nix/agent-modules/weston-vnc.nix adds an optional Weston Wayland compositor with the vnc-backend plugin, surfaced as services.hyperhive.agent.gui.enable = true per-agent. The harness's /screen/ws WebSocket relay (docs/2-handbook/web-ui/agent.md::Per-agent endpoints) connects to the compositor at 127.0.0.1:<vnc_port>.

• Port allocation: a fixed port (services.hyperhive.agent.gui.vncPort, default 5900). No per-agent hashing: network isolation is unconditional (each agent has its own netns — see docs/4-implementation/networking/network.md#container-isolation), so the VNC port is container-local and can't collide across agents. The harness learns the port from the HIVE_GUI_VNC_PORT env var (set on the harness service when gui.enable) — no marker file, no runtime hash. (Unlike the agent web-UI port, which is still an FNV-1a hash because those listen on the shared host stack — see Web UI ports collide on hash.)
• Non-root, shared user session: weston runs as the agent's own user (services.hyperhive.agent.user.name, the same user hive-ag3nt runs as), not root, so the GUI and the agent share one session. The runtime dir is a fixed /run/gui (systemd RuntimeDirectory=gui, 0700, RuntimeDirectoryPreserve=yes so it survives weston restarts for the wayland client sharing the /run/gui/wayland-0 socket). Wayland clients in the agent's config (for example bitburner electron) must run as the same user with XDG_RUNTIME_DIR=/run/gui.
• One shared D-Bus session bus (gui-dbus.service): a single persistent dbus-daemon --session bound at /run/gui/bus, run as the agent user, ordered before weston.service (it shares the same RuntimeDirectory=gui, creating the dir first). Chromium/electron via ozone refuse to map an xdg_toplevel without a reachable session bus ("Failed to connect to the bus" → binds xdg_wm_base then destroys it = invisible window even though CDP works). The fix is not to wrap each client in its own dbus-run-session (a private throwaway bus per process — that's a separate session, defeating the one-session model); it's this one shared bus; systemd.globalEnvironment exports its address as DBUS_SESSION_BUS_ADDRESS=unix:path=/run/gui/bus so weston, the harness and every GUI client inherit it.
• Fixed Wayland socket name (--socket=wayland-0): weston is launched with --socket=wayland-0 so the socket path is deterministic. nix/agent-modules/weston-vnc.nix exports WAYLAND_DISPLAY=wayland-0 and XDG_RUNTIME_DIR=/run/gui as global system environment variables (gated on services.hyperhive.agent.gui.enable) so every systemd service in the container inherits them. Without this, services starting Wayland clients couldn't find the compositor — libwayland falls back to a headless display or errors out, the app "works" on a second invisible display, and the VNC session shows a blank weston desktop (known double-screen issue).
• VNC bind address: Weston's vnc-backend has no CLI bind-address flag (unlike the rdp-backend's --address), so the listener binds 0.0.0.0. The harness relay only connects via 127.0.0.1; the host firewall blocks the per-agent VNC port range from external access. A future weston.ini [vnc] address= would restrict the bind directly, once upstream supports it.
• PAM service name: literal weston-remote-access — that's the string libweston passes to pam_start() in libweston/auth.c. Using weston falls back to the system default PAM stack and rejects auth. weston-vnc.nix configures the service to pam_permit.so for all three module types (auth / account / session) so the browser's empty Apple-DH credentials (type 30) always pass — neatvnc ≥ 0.9 calls the PAM auth callback regardless of weston.ini auth-method=none, so the permit fallback is what actually lets the empty-cred client through.
• Type = "simple" (not notify): switch-to-configuration must never block on weston signalling readiness. A misconfigured weston degrades to a Restart=on-failure loop visible in journalctl, it doesn't abort the nixos-container update. Same reasoning as the best-effort oneshots in docs/5-project/process/conventions.md.
• [core] idle-time=0: disables weston's 300-second idle timeout. Without it the VNC desktop fades to black and desktop-shell shows its unlock screen — useless for an agent desktop viewed over /screen. idle-time=0 updates the idle timer with a 0ms delay, which wl_event_source_timer_update treats as "disarm," so the compositor never goes idle and never locks.

Nix docs pipeline

Nix options reference (nix/docs/default.nix)

pkgs.nixosOptionsDoc over two evaluated module trees: hostEval (a stub NixOS system loading the nix/host-modules/ aggregator with every hyperhive subsystem mkForce false so heavy build inputs stay out of the eval) and agentEval (evaluates agent.nix fresh for the per-agent options tree).

Five output trees consumed by flake.nix, all markdown:

• docs-host — operator-facing host module options (services.hyperhive.* minus the swarm/deploy subtrees below)
• docs-swarm — swarm-wide facts, identical on every host in the swarm (services.hyperhive.swarm.*)
• docs-deploy — this host's own deployment decisions, necessarily different per host (services.hyperhive.deploy.* — added by nix/host-modules/deploy.nix, split out of swarm.* for exactly this reason: see that file's own comment)
• docs-agent — per-agent harness options (hyperhive.* declared in nix/agent-modules/)
• docs — bundle of index.md + host.md + swarm.md + deploy.md + agent.md

Pipeline:

• CommonMark from nixosOptionsDoc.optionsCommonMark is the only output — the source of truth, emitted as .md.
• The website repo renders HTML + CSS downstream (its own options.nix), consuming this bundle's host.md / agent.md, rendering them with cmark-gfm, and sharing one stylesheet (docs.css) across /options/ and the prose /docs/ tree. Keeping rendering in the website means the theme has a single home, sharing colours across both trees.
• transformOptions strips the nix-store prefix from option declaration paths and rewrites them as forge URLs, so the rendered docs link back to the source.

Host options live entirely under services.hyperhive.*. pickSubtrees takes ["services" "hyperhive"] as its root, so the options tree picks up everything under that root — picking against stray roots produces an empty tree and renders the host page as template chrome with no <h2> headers. docs-host then drops the swarm/deploy subtrees from that picked tree (removeAttrs, not a second pickSubtrees root — the one exclusion this file needs doesn't earn a general-purpose helper) since docs-swarm/docs-deploy each pick their own root instead.

Docs drv stability: nixSrc

Naively, the docs evaluation depends on self (the flake's store path), so every commit — even Rust-only or frontend-only changes — produces new docs drv hashes. The remote builder must rebuild docs from scratch for every PR branch, and if its store is full the build fails with a cached failure that blocks CI for the whole branch.

The fix (nix/docs/default.nix):

1. nixSrc — builtins.path on the nix/ directory, wrapped in builtins.unsafeDiscardStringContext to strip self's store-path context. The resulting store path is content-addressed from the nix/ file contents only. Docs drvs only change when a .nix file changes.

2. The package options the modules consume (services.hyperhive.agent.packages.*, services.hyperhive.deploy.hive-controller.*) carry no in-module defaults and every default that references them has a defaultText, so the doc walk never forces a package — no stubs needed, and the Rust/frontend build closure stays out of the eval.

3. Nix evaluates both hostEval and agentEval from nixSrc paths (not self), so the docs drv dependency chain ends at nixSrc.

Why builtins.unsafeDiscardStringContext? The path string toString self + "/nix" carries self's string context, which would make builtins.path include self as a build dependency even after content-addressing the directory. Discarding the context makes the resulting nixSrc truly independent of self's store path.