309879dba0
iris's #718 scope: move substantive design context from `#` comment blocks in `nix/` to corresponding `docs/` files, leave short references in code. iris handed it back to me on #10114 since nix/ is my lane + #775 established the pattern. First pass — three highest-density blocks in harness-base.nix: 1. **First-boot agent-user migration** (~70 lines → `~20 lines code + short ref` in the activation script). Substantive prose moves to new `docs/persistence.md::First-boot agent-user migration (post-#658)` section explaining the 4 steps the script performs + the eventual removability of the marker-guarded body. 2. **nix-daemon `sandbox-fallback = true`** (10-line block → 5-line ref). New `docs/gotchas.md::Containerized nix-daemon needs sandbox-fallback = true` section covers the user-namespaces rationale + nixpkgs-default override. 3. **Matrix daemon + token-arrival trigger** (~50 lines across two systemd units → ~10 lines code + short refs). New `docs/persistence.md::Matrix per-agent daemon + token-arrival trigger` covers the socket-path rationale, the runtime-dir ownership story, and the first-boot ordering pattern. Net: harness-base.nix -84 lines, docs +74 lines. Substantive design context moves to durable docs; in-code refs follow iris's pattern from her #712 batches (`see docs/<file>::<section>`). Follow-ups: hive-c0re.nix, hive-forge.nix, hive-matrix.nix (already trimmed via #775 but a couple of remaining blocks could go), and the smaller files in #718's scope table. Shipping this first to get the pattern reviewed before larger batches. Verified: `nix eval` on agent-base toplevel still resolves.
240 lines
11 KiB
Markdown
240 lines
11 KiB
Markdown
# Gotchas
|
||
|
||
NixOS + nspawn quirks and lessons we hit the hard way. If something
|
||
here looks unmotivated in the code, there's usually a story underneath.
|
||
|
||
## `nixos-container` doesn't expose `--bind` on the CLI
|
||
|
||
The CLI doesn't accept `--bind`. Path is via `EXTRA_NSPAWN_FLAGS` in
|
||
`/etc/nixos-containers/<NAME>.conf` — the start script
|
||
(`/nix/store/.../container_-start`) expands it unquoted into the
|
||
`systemd-nspawn` invocation. `lifecycle::set_nspawn_flags()` rewrites
|
||
this line.
|
||
|
||
## `/run/systemd/nspawn/*.nspawn` overrides are ignored
|
||
|
||
`nixos-container`'s start script builds the nspawn command line
|
||
directly. Dropping a `.nspawn` file under `/run/systemd/nspawn/`
|
||
looks like the obvious extension point and does nothing. Use
|
||
`EXTRA_NSPAWN_FLAGS` (above).
|
||
|
||
## `boot.isNspawnContainer = true`
|
||
|
||
Not `boot.isContainer = true`. Renamed in nixos-25.11+.
|
||
|
||
## `nixos-container create` auto-assigns `HOST_ADDRESS` / `LOCAL_ADDRESS`
|
||
|
||
…in the `.conf`. The start script's `if HOST_ADDRESS set →
|
||
--network-veth` branch then forces a private netns — silently fatal
|
||
for our web UIs (the bind is invisible from the host). We
|
||
force-clear `HOST_ADDRESS` / `LOCAL_ADDRESS` / `HOST_ADDRESS6` /
|
||
`LOCAL_ADDRESS6` / `HOST_BRIDGE` and set `PRIVATE_NETWORK=0`.
|
||
|
||
## systemd service PATH ≠ host PATH
|
||
|
||
The hive-c0re service sets `path = [ pkgs.git "/run/current-system/sw" ]`.
|
||
In-container harness services do the same so anything an agent adds
|
||
to its own `agent.nix` (`environment.systemPackages`) is visible to
|
||
claude's Bash tool without editing the service definition.
|
||
`environment.HYPERHIVE_GIT` bakes git's absolute path in (read by
|
||
`lifecycle::git_command()`) for the host.
|
||
|
||
## `systemd.services.*.path` appends `/bin` to every entry
|
||
|
||
NixOS's `systemd.services.<unit>.path` list feeds every entry through
|
||
`lib.makeBinPath`, which **appends `/bin` unconditionally**. That's
|
||
the right thing for Nix packages (their `outPath` is the store root,
|
||
not the `bin/` subdir), but it bites when you pass a string that
|
||
already ends with `/bin`:
|
||
|
||
```nix
|
||
# ❌ /run/wrappers/bin → /run/wrappers/bin/bin (does not exist)
|
||
path = [ "/run/wrappers/bin" "/run/current-system/sw" ];
|
||
|
||
# ✅ /run/wrappers → /run/wrappers/bin (the real wrappers dir)
|
||
path = [ "/run/wrappers" "/run/current-system/sw" ];
|
||
```
|
||
|
||
The bug is silent: `nix eval` succeeds, the unit starts, but PATH
|
||
contains a non-existent directory. The first symptom is usually
|
||
`sudo: must be owned by uid 0 and have the setuid bit set` because
|
||
the setuid sudo wrapper lives at `/run/wrappers/bin/sudo` and
|
||
the path entry resolves to `/run/wrappers/bin/bin` instead.
|
||
|
||
## `RuntimeDirectoryPreserve = "yes"`
|
||
|
||
…keeps `/run/hyperhive/` (and the per-agent sub-dirs) across
|
||
hive-c0re restarts. Without it, every restart wipes bind sources and
|
||
existing containers can't be started.
|
||
|
||
## `register_agent` is idempotent
|
||
|
||
Drops any prior socket task before rebinding. Required so a
|
||
hive-c0re restart followed by `rebuild alice` recreates the agent's
|
||
socket without needing a clean reinstall.
|
||
|
||
## `claude-code` is unfree
|
||
|
||
The flake pins it to **nixpkgs-unstable** via
|
||
`overlays.claude-unstable` (stable lags too far). The overlay sets
|
||
`config.allowUnfreePredicate` on its unstable import to whitelist
|
||
`claude-code` specifically — scoped, only this one package.
|
||
`harness-base.nix` does the same at the container level because
|
||
each per-agent `nixosConfiguration` evaluates its own nixpkgs
|
||
instance and the operator's host-level `allowUnfree` does **not**
|
||
propagate in. Operators don't need to set anything on their side.
|
||
|
||
## Claude credentials are per-agent
|
||
|
||
`/var/lib/hyperhive/agents/<name>/claude/` bind-mounts to
|
||
`/home/<name>/.claude` (RW; was `/root/.claude` pre-#658 when every
|
||
harness ran as root). Sharing one dir across agents is NOT viable —
|
||
OAuth refresh tokens rotate, so any sibling refresh invalidates all
|
||
the others. Login flow runs from the per-agent web UI; creds persist
|
||
across `destroy`/recreate (`--purge` wipes them).
|
||
|
||
## Persistent notes dir per agent
|
||
|
||
`/var/lib/hyperhive/agents/<name>/state/` bind-mounts to
|
||
`/agents/<name>/state` (RW; uniform for sub-agents + manager
|
||
post-#604, was `/state` pre-#604). The harness exposes the same path
|
||
via `$HYPERHIVE_STATE_DIR`. System prompts tell agents to keep
|
||
durable knowledge here (`notes.md`, anything else). The harness also
|
||
writes its events log here (`hyperhive-events.sqlite`).
|
||
Survives `destroy`/recreate alongside the claude dir.
|
||
|
||
## Web UI ports collide on hash
|
||
|
||
Sub-agent web UI ports are deterministic FNV-1a of the agent name
|
||
modulo 900 (range 8100..8999). With ~30 agents the birthday-paradox
|
||
collision rate gets meaningful; at 2–3 agents you can still get
|
||
unlucky. Operator resolves a collision by renaming the offending
|
||
agent (different hash → different port) and rebuilding. No state
|
||
file, no probing, no port-allocation drift — the value is
|
||
reproducible from just the name. Every agent — including the
|
||
manager — hashes into 8100..8999 via the same FNV-1a since #753;
|
||
dashboard at `cfg.dashboardPort` (default 7000).
|
||
|
||
## Restart races on TCP bind
|
||
|
||
Both the dashboard and per-agent web UI use `tokio::net::TcpSocket`
|
||
with `SO_REUSEADDR` plus a retry-on-`AddrInUse` loop (12 tries,
|
||
exponential backoff capped at 2s, ~22s total). REUSEADDR handles
|
||
the `TIME_WAIT` case from a clean previous exit; retry covers the
|
||
genuine "previous process is still alive during a systemd restart
|
||
overlap" case. REUSEADDR does **not** allow two simultaneous
|
||
`LISTEN` sockets on the same port (that would be `SO_REUSEPORT`,
|
||
which we don't use) — exclusivity is preserved.
|
||
|
||
## Orphan approvals
|
||
|
||
If state dirs are wiped out from under a pending approval (test
|
||
scripts, manual `rm -rf`), the dashboard's next render marks them
|
||
`failed` with note `"agent state dir missing"` so they fall out of
|
||
`pending`. They stay in sqlite for audit.
|
||
|
||
## Nix store `cp -r` preserves read-only bits
|
||
|
||
Copying a nix store path with `cp -r src/. $out/` inside a
|
||
`pkgs.runCommand` derivation preserves the read-only permissions of
|
||
store files. Any subsequent write into the copied tree (adding new
|
||
files in subdirectories) fails with `EPERM`. Fix: pass
|
||
`--no-preserve=mode,ownership` so the output tree is writable.
|
||
|
||
## SPA fallback: use `Accept` header map, not `try_files ... /index.html`
|
||
|
||
The naive nginx pattern for a path-prefix SPA (`try_files $uri $uri/
|
||
/matrix/index.html`) silently swallows asset 404s — a missing JS file
|
||
returns `index.html` with a 200, so the JS runtime never loads and the
|
||
page renders blank with no visible error (#685; fixed in PR #684;
|
||
#686 filed the follow-up edge-case, addressed in PR #729).
|
||
Extension allowlists (tried in #686 → PR #729) have the same maintenance
|
||
problem: any new file extension the SPA ships breaks silently.
|
||
|
||
The correct pattern (landed in PR #729, `hive-gateway.nix`) keys the
|
||
fallback on the HTTP `Accept` header:
|
||
|
||
```nginx
|
||
# Outside the server block (appendHttpConfig):
|
||
map $http_accept $matrix_spa_target {
|
||
default "/__matrix_spa_no_html_fallback";
|
||
"~*text/html" "/matrix/index.html";
|
||
}
|
||
|
||
# Inside the location:
|
||
try_files $uri $uri/ $matrix_spa_target =404;
|
||
```
|
||
|
||
Top-frame navigations always send `Accept: text/html,...` (chrome /
|
||
firefox / safari are consistent). Asset fetches (`image/*`,
|
||
`application/javascript`, `*/*`) don't carry `text/html`, so they
|
||
fall through to the trailing `=404`. No extension list to maintain;
|
||
no named-location indirection needed.
|
||
|
||
## `nix build flake#name` does not walk into `nixosConfigurations`
|
||
|
||
`nix build` resolves the fragment (`#name`) against the flake's
|
||
**top-level output attrs** — not against `nixosConfigurations`
|
||
specifically. `nixos-container` and `nixos-rebuild` use their own
|
||
internal convention that routes an agent name to
|
||
`nixosConfigurations.<name>.config.system.build.toplevel`, but
|
||
`nix build` has no such convention.
|
||
|
||
```
|
||
# ❌ silently builds the wrong thing (or errors if attr doesn't exist)
|
||
nix build /var/lib/hyperhive/meta#argus.config.system.build.toplevel
|
||
|
||
# ✅ explicit path nix build actually resolves
|
||
nix build /var/lib/hyperhive/meta#nixosConfigurations.argus.config.system.build.toplevel
|
||
```
|
||
|
||
`lifecycle::prebuild_toplevel` hit this in #721 (fixed in #738) by
|
||
constructing the attr path as `{flake_ref}.config…` — which produced
|
||
`meta#argus.config…` instead of `meta#nixosConfigurations.argus.config…`.
|
||
The fix: `split_once('#')` to separate flake path from name, then
|
||
template `{path}#nixosConfigurations.{name}.config.system.build.toplevel`.
|
||
|
||
## `hive-forge`: prefer over raw curl pipelines
|
||
|
||
Every agent container has `hive-forge` in PATH (installed via
|
||
`harness-base.nix`; lives in `/hive-forge` as a proper Rust binary
|
||
since #280). Use it instead of ad-hoc curl pipelines:
|
||
|
||
```bash
|
||
hive-forge view 42 # title + body + comments
|
||
hive-forge comments 42 # list all comments (human-readable)
|
||
hive-forge --json comments 42 # same as above, JSON array (global flag, closes #421)
|
||
hive-forge comment 42 --body "..." # post comment (inline body)
|
||
hive-forge comment 42 --body-file - <<EOF # ...or pipe a HEREDOC
|
||
multi-line body
|
||
EOF
|
||
hive-forge assign 42 damocles
|
||
hive-forge close 42
|
||
hive-forge labels 42 add feature
|
||
hive-forge pr 42 # PR metadata as JSON
|
||
hive-forge pr-create --title "..." --head my-branch --push # also `git push forge my-branch`, suppressing the post-push "Create a pull request" hint (#222)
|
||
hive-forge diff 42 # unified diff (lockfile hunks collapsed by default)
|
||
hive-forge diff 42 --full # include unfiltered lockfile hunks
|
||
hive-forge branches deployed/ # filter branches by pattern
|
||
hive-forge -r other-org/other-repo pr 7 # target a different repo
|
||
hive-forge lint unassigned # open issues/PRs with no assignee
|
||
hive-forge lint no-reviewer --reviewer argus # PRs missing a reviewer comment from argus
|
||
hive-forge lint stale-branches --days 14 # branches with no recent activity
|
||
hive-forge lint assignments # per-assignee open item count
|
||
```
|
||
|
||
`hive-forge <verb> --help` prints the full signature for any verb.
|
||
Credentials come from `$HYPERHIVE_STATE_DIR/forge-token`; default
|
||
repo from `$HIVE_FORGE_REPO`, overridden per-invocation by the
|
||
global `-r/--repo` flag.
|
||
|
||
## Containerized nix-daemon needs `sandbox-fallback = true`
|
||
|
||
Agent containers bind-mount the host's nix-daemon socket. nspawn
|
||
containers don't get user-namespaces by default, so `nix build`
|
||
invocations *inside* the container can't set up the build sandbox
|
||
and fail outright if the host daemon's
|
||
`nix.settings.sandbox-fallback` is `false` (nixpkgs default).
|
||
`nix/templates/harness-base.nix` does `lib.mkForce true` so builds
|
||
fall back to unsandboxed local builds rather than failing. Security
|
||
implications: `docs/security.md`.
|