docs: sort the top-level directories into five sections by what the reader wants
Every docs directory moves under 1-getting-started, 2-handbook, 4-implementation or 5-project; the options reference (section 3) comes from the website and has no directory here. trust-boundary stays whole under the handbook for now. web-ui, swarm and tools split between handbook and implementation, and the command-line references move to 2-handbook/reference. Links, path references in code and nix, and the docs index follow the moves; no page content changes.
This commit is contained in:
iris
mara
parent
53ccbcb4b3
commit
6739bb903d
1 changed files with 736 additions and 701 deletions
|
|
@ -1,552 +0,0 @@
|
|||
# Gotchas
|
||||
|
||||
NixOS + nspawn quirks and hard-won lessons. If something
|
||||
here looks unmotivated in the code, there's probably a story underneath.
|
||||
Grouped by area — jump to the section that matches what you're
|
||||
touching.
|
||||
|
||||
## NixOS / nspawn containers
|
||||
|
||||
### `nixos-container` doesn't expose `--bind` on the CLI
|
||||
|
||||
The CLI doesn't accept `--bind`. Path is via `EXTRA_NSPAWN_FLAGS` in
|
||||
`/etc/nixos-containers/<NAME>.conf` — the start script
|
||||
(`/nix/store/.../container_-start`) expands it unquoted into the
|
||||
`systemd-nspawn` invocation. `lifecycle::host_config::set_nspawn_flags()`
|
||||
rewrites this line.
|
||||
|
||||
### `nixos-container` ignores `/run/systemd/nspawn/*.nspawn` overrides
|
||||
|
||||
`nixos-container`'s start script builds the nspawn command line
|
||||
directly. Dropping a `.nspawn` file under `/run/systemd/nspawn/`
|
||||
looks like a natural extension point and does nothing. Use
|
||||
`EXTRA_NSPAWN_FLAGS` (above).
|
||||
|
||||
### `boot.isNspawnContainer = true`
|
||||
|
||||
Not `boot.isContainer = true`. Renamed in nixos-25.11+.
|
||||
|
||||
### `nixos-container create` autoassigns `HOST_ADDRESS` / `LOCAL_ADDRESS`
|
||||
|
||||
…in the `.conf`. The start script's `if HOST_ADDRESS set →
|
||||
--network-veth` branch then forces a private netns — silently fatal
|
||||
for the web UIs (the bind is invisible from the host). Every agent
|
||||
container runs isolated: `hive-priv`'s `write_nspawn_flags` writes
|
||||
`PRIVATE_NETWORK=1` plus a veth pair onto the host bridge, `HOST_ADDRESS`
|
||||
set to the bridge gateway IP (so `nixos-container`'s in-container init
|
||||
installs a default route before the DHCP lease arrives), rest left for
|
||||
DHCP.
|
||||
|
||||
### systemd service PATH ≠ host PATH
|
||||
|
||||
The hive-c0re service sets `path = [ pkgs.git "/run/current-system/sw" ]`.
|
||||
In-container harness services do the same so anything an agent adds
|
||||
to its own `agent.nix` (`environment.systemPackages`) is visible to
|
||||
the `mcp__bash__run` MCP tool (and any other in-container process) without
|
||||
editing the service definition.
|
||||
`environment.HYPERHIVE_GIT` bakes git's absolute path in (read by
|
||||
`lifecycle::git_command()`) for the host.
|
||||
|
||||
### `systemd.services.*.path` appends `/bin` to every entry
|
||||
|
||||
NixOS's `systemd.services.<unit>.path` list feeds every entry through
|
||||
`lib.makeBinPath`, which **appends `/bin` unconditionally**. That's
|
||||
the right thing for Nix packages (their `outPath` is the store root,
|
||||
not the `bin/` subdir), but it bites when you pass a string that
|
||||
already ends with `/bin`:
|
||||
|
||||
```nix
|
||||
# ❌ /run/wrappers/bin → /run/wrappers/bin/bin (does not exist)
|
||||
path = [ "/run/wrappers/bin" "/run/current-system/sw" ];
|
||||
|
||||
# ✅ /run/wrappers → /run/wrappers/bin (the real wrappers dir)
|
||||
path = [ "/run/wrappers" "/run/current-system/sw" ];
|
||||
```
|
||||
|
||||
The bug is silent: `nix eval` succeeds, the unit starts, but PATH
|
||||
contains a non-existent directory. The first symptom is typically
|
||||
`sudo: must be owned by uid 0 and have the setuid bit set` because
|
||||
the setuid sudo wrapper lives at `/run/wrappers/bin/sudo` and
|
||||
the path entry resolves to `/run/wrappers/bin/bin` instead.
|
||||
|
||||
### `RuntimeDirectoryPreserve = "yes"`
|
||||
|
||||
…keeps `/run/hyperhive/` (and the per-agent sub-dirs) across
|
||||
hive-c0re restarts. Without it, every restart wipes bind sources and
|
||||
existing containers fail to start.
|
||||
|
||||
### `DynamicUser` + `RuntimeDirectoryPreserve` hides the socket from nginx
|
||||
|
||||
With both set, systemd creates the runtime directory under `/run/private/`
|
||||
(`0700 root:root`) and symlinks `/run/<name>` to it (`systemd.exec(5)`,
|
||||
`RuntimeDirectoryPreserve=`). No unprivileged user can traverse that, so
|
||||
nginx's `connect()` to a socket in it fails with `EACCES` whatever the
|
||||
directory and socket modes are. A unit serving a socket to another user
|
||||
runs as a static user instead (`swarm-controller`, `swarm-logs-auth`,
|
||||
`swarm-metrics-auth`, `swarm-metrics-agent-auth`).
|
||||
`module-eval-core-toggle` fails on any `unix:` upstream of nginx owned by
|
||||
such a unit.
|
||||
|
||||
### `RestrictAddressFamilies` fails as "Address family not supported by protocol"
|
||||
|
||||
A unit whose `RestrictAddressFamilies` omits a family gets `EAFNOSUPPORT`
|
||||
(errno 97) back from `socket()`. Clients surface that as _"tcp open error:
|
||||
Address family not supported by protocol"_ — the message names the
|
||||
**protocol** and never the **sandbox**, so it reads like a dead network, a
|
||||
missing route, or an IPv6 problem.
|
||||
|
||||
⇒ On that error, read the unit before you touch the network.
|
||||
|
||||
Two things to get right when a daemon needs outbound TCP:
|
||||
|
||||
- list `AF_INET` **and** `AF_INET6` — omitting one leaves a client that
|
||||
works until DNS hands back the other family;
|
||||
- list `AF_NETLINK` too. glibc's `getaddrinfo` opens a netlink socket to
|
||||
enumerate local addresses before it returns any, so name resolution
|
||||
fails without it even when the unit allows `AF_INET`.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
**The directive is a claim about what the program does, and nothing
|
||||
re-checks it when the program changes.** A unit that only served a unix
|
||||
socket when it was written is correct at `[ "AF_UNIX" ]` and silently wrong
|
||||
the day someone adds an HTTP client. Check the unit in the same commit as
|
||||
the client — and when narrowing it, prefer a test that derives the required
|
||||
families from the code (which fails on the _next_ client too) over one that
|
||||
asserts today's list.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
### `register_agent` is idempotent
|
||||
|
||||
Drops any prior socket task before rebinding. Required so a
|
||||
hive-c0re restart followed by `rebuild alice` recreates the agent's
|
||||
socket without needing a clean reinstall.
|
||||
|
||||
## Claude Code packaging & credentials
|
||||
|
||||
### `claude-code` is unfree
|
||||
|
||||
`claude-code` comes from the flake's main `nixpkgs` (nixos-26.05).
|
||||
It's unfree, so the agent modules set `config.allowUnfreePredicate`
|
||||
at the container level to allowlist `claude-code` specifically —
|
||||
scoped, only this one package, because each per-agent
|
||||
`nixosConfiguration` evaluates its own nixpkgs instance and the
|
||||
operator's host-level `allowUnfree` does **not** propagate in.
|
||||
Operators don't need to set anything on their side. Only `runtime = "claude"`
|
||||
agents get `claude` and the allowance; an ACP agent whose `acp.command`
|
||||
needs `claude` adds `pkgs.claude-code` and
|
||||
`nixpkgs.config.allowUnfreePackages = [ "claude-code" ]` itself.
|
||||
|
||||
That same isolation is why an agent can't pick a claude out of a
|
||||
_different_ nixpkgs by itself: a container only ever sees the one
|
||||
nixpkgs the meta flake injects, so an `agent.nix` naming the host's
|
||||
`nixpkgs-unstable` has nothing to name. A release channel can trail
|
||||
unstable by weeks on this package, which is what
|
||||
`services.hyperhive.deploy.hive-controller.claudeCodePackage` is for — set it host-side
|
||||
and every agent runs that build.
|
||||
|
||||
What crosses is the **store path**, not the derivation. Containers
|
||||
share the host's `/nix/store`, so the binary is already reachable
|
||||
inside them with its whole closure. `hive-priv` writes the path into each
|
||||
container's `/etc/hyperhive-hive-facts` at start, and the agent's `claude`
|
||||
wrapper on PATH runs its `bin/claude`. Two things rule out the natural-seeming alternatives: a
|
||||
`path:/nix/store/<pkg>` flake input is re-copied into the store as a
|
||||
reference-less `-source` (so the runtime closure never arrives), and
|
||||
`lib.types.package` fed a bare path runs `builtins.storePath`, which
|
||||
pure evaluation rejects. `services.hyperhive.agent.docs.source` gets away with being
|
||||
an input only because a docs tree has no runtime dependencies.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
The `storePath` pitfall is worth spelling out, because it's not confined
|
||||
to options the operator writes: **any** option of type `package` fed a
|
||||
store-path _string_ coerces through `lib.toDerivation`, i.e.
|
||||
`builtins.storePath`. `environment.systemPackages` and
|
||||
`systemd.services.<name>.path` both do it (the latter takes plain
|
||||
strings like `/run/wrappers` happily, but Nix treats anything under
|
||||
`builtins.storeDir` as a package). Wrap a path handed to the
|
||||
container as text in a real derivation — a symlink farm built from
|
||||
the interpolated string — before it can go anywhere a package is
|
||||
expected.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
The catch is that a path handed to the container at start is text, not a
|
||||
reference — the container's closure doesn't keep the binary alive.
|
||||
The **host** does: it interpolates the package into
|
||||
`/etc/hyperhive/serve.json`, so it lands in the host's system closure
|
||||
and is gc-rooted by the running generation. `builtins.toJSON` preserves
|
||||
string context, which is the load-bearing detail; discard the context
|
||||
anywhere on that path and `nix-collect-garbage` will eventually take
|
||||
the hive's `claude` out from under it. The price of the root is that
|
||||
`nix-collect-garbage` can't reclaim an old `claude-code` until every
|
||||
agent has restarted past it and the old generations are gone.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
### Claude credentials are per-agent
|
||||
|
||||
`/var/lib/hyperhive/agents/<name>/claude/` bind-mounts to
|
||||
`/home/<name>/.claude` (RW). Sharing one dir across agents **isn't** viable —
|
||||
OAuth refresh tokens rotate, so any sibling refresh invalidates all
|
||||
the others. Login flow runs from the per-agent web UI; creds persist
|
||||
across `destroy`/recreate (`--purge` wipes them).
|
||||
|
||||
### Persistent notes dir per agent
|
||||
|
||||
`/var/lib/hyperhive/agents/<name>/state/` bind-mounts to
|
||||
`/agents/<name>/state` (RW; uniform for all agents).
|
||||
The harness exposes the same path
|
||||
via `$HYPERHIVE_STATE_DIR`. System prompts tell agents to keep
|
||||
durable knowledge here (`notes.md`, anything else) — the harness's own
|
||||
internal files (`hyperhive-events.sqlite`, `hyperhive-turn-stats.sqlite`,
|
||||
`hyperhive-model`) live in the separate `harness` dir instead, so they
|
||||
don't clutter what claude sees as "my notes dir" (see
|
||||
[`docs/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)). Survives `destroy`/recreate
|
||||
alongside the claude dir.
|
||||
|
||||
### Persistent work dir per agent
|
||||
|
||||
`/var/lib/hyperhive/agents/<name>/work/` bind-mounts to
|
||||
`/agents/<name>/work` (RW, owned by the agent user). It's the place for
|
||||
git worktrees and build dirs: a claude started under `state/` walks up
|
||||
into `state/AGENTS.md` and takes on the agent's identity, and one
|
||||
started under `work/` doesn't. The container rootfs is ephemeral
|
||||
(nspawn `--ephemeral`), so `/home/<name>` is empty after every restart:
|
||||
anything the agent keeps goes in `work/` or `state/`. `work/` survives
|
||||
`destroy`/recreate and goes on `--purge`. The per-agent disk metric counts it; the
|
||||
dashboard doesn't serve it.
|
||||
|
||||
## Networking & ports
|
||||
|
||||
### Web UI ports collide on hash
|
||||
|
||||
Sub-agent web UI ports are deterministic FNV-1a of the agent name
|
||||
modulo 900 (range 8100..8999). With ~30 agents the birthday-paradox
|
||||
collision rate gets meaningful; at 2–3 agents you can still get
|
||||
unlucky. Operator resolves a collision by renaming the offending
|
||||
agent (different hash → different port) and rebuilding. No state
|
||||
file, no probing, no port-allocation drift — the value is
|
||||
reproducible from just the name. Every agent hashes into
|
||||
8100..8999 via the same FNV-1a; dashboard
|
||||
at `services.hyperhive.deploy.hive-controller.dashboardPort` (default 7000).
|
||||
|
||||
### Restart races on TCP bind
|
||||
|
||||
Both the dashboard and per-agent web UI use `tokio::net::TcpSocket`
|
||||
with `SO_REUSEADDR` plus a retry-on-`AddrInUse` loop (12 tries,
|
||||
exponential backoff capped at 2s, ~22s total). REUSEADDR handles
|
||||
the `TIME_WAIT` case from a clean previous exit; retry covers the
|
||||
genuine "previous process is still alive during a systemd restart
|
||||
overlap" case. REUSEADDR does **not** allow two simultaneous
|
||||
`LISTEN` sockets on the same port (that would be `SO_REUSEPORT`,
|
||||
not used here) — it preserves exclusivity.
|
||||
|
||||
## Gateway / SPA serving
|
||||
|
||||
### SPA fallback: use `Accept` header map, not `try_files ... /index.html`
|
||||
|
||||
The naive nginx pattern for an SPA (`try_files $uri $uri/
|
||||
/index.html`) silently swallows asset 404s — a missing JS file
|
||||
returns `index.html` with a 200, so the JS runtime never loads and the
|
||||
page renders blank with no visible error. Extension allowlists (tried
|
||||
as an alternative) have the same maintenance problem: any new file
|
||||
extension the SPA ships breaks silently.
|
||||
|
||||
The pattern that works (`nix/host-modules/hive-matrix.nix`, serving
|
||||
fluffychat at the matrix gateway vhost's root) keys the fallback on the
|
||||
HTTP `Accept` header:
|
||||
|
||||
```nginx
|
||||
# Outside the server block (appendHttpConfig):
|
||||
map $http_accept $matrix_spa_target {
|
||||
default "/__matrix_spa_no_html_fallback";
|
||||
"~*text/html" "/index.html";
|
||||
}
|
||||
|
||||
# Inside the location:
|
||||
try_files $uri $uri/ $matrix_spa_target =404;
|
||||
```
|
||||
|
||||
Top-frame navigations always send `Accept: text/html,...` (chrome /
|
||||
firefox / safari are consistent). Asset fetches (`image/*`,
|
||||
`application/javascript`, `*/*`) don't carry `text/html`, so they
|
||||
fall through to the trailing `=404`. No extension list to maintain;
|
||||
no named-location indirection needed.
|
||||
|
||||
## Build & dev workflow
|
||||
|
||||
### Nix store `cp -r` preserves read-only bits
|
||||
|
||||
Copying a nix store path with `cp -r src/. $out/` inside a
|
||||
`pkgs.runCommand` derivation preserves the read-only permissions of
|
||||
store files. Any subsequent write into the copied tree (adding new
|
||||
files in subdirectories) fails with `EPERM`. Fix: pass
|
||||
`--no-preserve=mode,ownership` so the output tree is writable.
|
||||
|
||||
### `nix build flake#name` doesn't walk into `nixosConfigurations`
|
||||
|
||||
`nix build` resolves the fragment (`#name`) against the flake's
|
||||
**top-level output attrs** — not against `nixosConfigurations`
|
||||
specifically. `nixos-container` and `nixos-rebuild` use their own
|
||||
internal convention that routes an agent name to
|
||||
`nixosConfigurations.<name>.config.system.build.toplevel`, but
|
||||
`nix build` has no such convention.
|
||||
|
||||
```
|
||||
# ❌ silently builds the wrong thing (or errors if attr doesn't exist)
|
||||
nix build "git+<forge>/internal/meta.git?rev=<rev>#argus.config.system.build.toplevel"
|
||||
|
||||
# ✅ explicit path nix build actually resolves
|
||||
nix build "git+<forge>/internal/meta.git?rev=<rev>#nixosConfigurations.argus.config.system.build.toplevel"
|
||||
```
|
||||
|
||||
### Containerized nix-daemon needs `sandbox-fallback = true`
|
||||
|
||||
nixos-containers' start script bind-mounts the host's nix-daemon socket
|
||||
directory into every container (hyperhive adds no bind for it; an agent
|
||||
with `services.hyperhive.agent.nixBuilds = "none"` gets it shadowed, see
|
||||
`docs/trust-boundary/security.md`). nspawn
|
||||
containers don't get user-namespaces by default, so `nix build`
|
||||
invocations _inside_ the container can't set up the build sandbox
|
||||
and fail outright if the host daemon's
|
||||
`nix.settings.sandbox-fallback` is `false` (nixpkgs default).
|
||||
`nix/agent-modules/default.nix` does `lib.mkForce true` so builds
|
||||
fall back to unsandboxed local builds rather than failing. Security
|
||||
implications: `docs/trust-boundary/security.md`.
|
||||
|
||||
### Linking workspace binaries locally needs `nix develop`
|
||||
|
||||
The Rust workspace links `libsqlite3-sys` (rusqlite) against the
|
||||
system `libsqlite3`. Agent containers carry no system libsqlite3 on
|
||||
the linker path, so a plain `cargo build` of any binary dies with
|
||||
`cannot find -lsqlite3` (deps and `ring` compile fine — only the
|
||||
final link fails). `cargo check` / `cargo clippy` still work in the
|
||||
ambient shell since they never link.
|
||||
|
||||
Build + run binaries through the dev shell, which carries `sqlite`
|
||||
on `NIX_LDFLAGS`:
|
||||
|
||||
```bash
|
||||
nix develop -c cargo build -p hive-c0re --bin hivectl
|
||||
nix develop -c cargo run -p hive-c0re --bin hivectl -- <args>
|
||||
```
|
||||
|
||||
This is also how you regenerate committed generated docs locally —
|
||||
for example `docs/tools/hivectl-cli.md` via the `hivectl markdown-docs`
|
||||
subcommand (its `hivectl-docs` flake check otherwise only fails in
|
||||
CI on drift).
|
||||
|
||||
### Split asset derivations away from the rust workspace
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
`nix/packages/assets.nix` builds the branding SVG/PNG family + claude
|
||||
system-prompt template + claude-settings JSON as its own derivation,
|
||||
separate from the hive-ag3nt / hive-c0re crates. Reason: when the
|
||||
rust build's `src` was the whole repo tree, any tweak to
|
||||
`branding/agent-configs.svg` or `hive-agent/prompts/system.md`
|
||||
invalidated the cargo cache and forced a full rebuild. crane (and
|
||||
naersk before it) couldn't see "these inputs are unused by rust" on
|
||||
its own — the split breaks the coupling at the derivation boundary.
|
||||
`rsvg-convert` renders the agent-configs PNG from the SVG at
|
||||
build time; librsvg dependency lives here, not in the rust
|
||||
derivation's `nativeBuildInputs`.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
### `nix fmt` fails in a git worktree with "object not found"
|
||||
|
||||
`nix fmt` (and any `nix` command that fetches a `git+file://` flake
|
||||
URL) uses libgit2 internally to compute `revCount` — the number of
|
||||
commits reachable from HEAD. This walk fails with:
|
||||
|
||||
```
|
||||
error: getting Git object '<hash>': object not found (libgit2 error code = 9)
|
||||
```
|
||||
|
||||
when a commit that was reachable at some earlier evaluation is now gone
|
||||
(GC'd, rebased away, or pruned). The failure is persistent: clearing
|
||||
`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` doesn't
|
||||
help because the missing object is a structural gap in the git object
|
||||
graph itself, not in nix's caches.
|
||||
|
||||
**Workaround: use a plain clone, not a git worktree.**
|
||||
|
||||
```bash
|
||||
git clone http://<forge>/hyperhive/hyperhive.git ~/hh-work
|
||||
cd ~/hh-work && nix fmt
|
||||
```
|
||||
|
||||
The root cause is specific to worktrees: a worktree shares the object
|
||||
store with its parent repo. If the parent repo's history was rewritten
|
||||
(rebase, force-push, `git gc --prune`) while the worktree remained at a
|
||||
branch tip that references the pruned commits via its reflog or
|
||||
history, libgit2's rev-walk encounters the gap. A plain clone has its
|
||||
own self-consistent object store and is immune to the issue.
|
||||
|
||||
## Tooling
|
||||
|
||||
### `hive-forge`: prefer over raw curl pipelines
|
||||
|
||||
Full CLI reference: [`docs/tools/forge.md`](../tools/forge.md).
|
||||
Never use raw `curl` for forge access.
|
||||
|
||||
## GUI (weston/VNC)
|
||||
|
||||
### Weston VNC compositor (per-agent `services.hyperhive.agent.gui.enable`)
|
||||
|
||||
`nix/agent-modules/weston-vnc.nix` adds an optional Weston Wayland
|
||||
compositor with the `vnc-backend` plugin, surfaced as
|
||||
`services.hyperhive.agent.gui.enable = true` per-agent. The harness's
|
||||
`/screen/ws` WebSocket relay (`docs/web-ui/agent.md::Per-agent endpoints`)
|
||||
connects to the compositor at `127.0.0.1:<vnc_port>`.
|
||||
|
||||
- **Port allocation**: a **fixed** port (`services.hyperhive.agent.gui.vncPort`,
|
||||
default 5900). No per-agent hashing: network isolation is
|
||||
unconditional (each agent has its own netns — see
|
||||
`docs/networking/network.md#container-isolation`), so the VNC port is
|
||||
container-local and can't collide across agents. The harness learns
|
||||
the port from the `HIVE_GUI_VNC_PORT` env var (set on the harness
|
||||
service when `gui.enable`) — no marker file, no runtime hash. (Unlike
|
||||
the agent **web-UI** port, which is still an FNV-1a hash because those
|
||||
listen on the shared host stack — see `Web UI ports collide on hash`.)
|
||||
- **Non-root, shared user session**: weston runs as the agent's own
|
||||
user (`services.hyperhive.agent.user.name`, the same user hive-ag3nt runs as), not
|
||||
root, so the GUI and the agent share one session. The runtime dir is a
|
||||
fixed `/run/gui` (systemd `RuntimeDirectory=gui`, `0700`,
|
||||
`RuntimeDirectoryPreserve=yes` so it survives weston restarts for the
|
||||
wayland client sharing the `/run/gui/wayland-0` socket). Wayland
|
||||
clients in the agent's config (for example bitburner electron) must run as the
|
||||
same user with `XDG_RUNTIME_DIR=/run/gui`.
|
||||
- **One shared D-Bus session bus (`gui-dbus.service`)**: a single
|
||||
persistent `dbus-daemon --session` bound at `/run/gui/bus`, run as the
|
||||
agent user, ordered `before weston.service` (it shares the same
|
||||
`RuntimeDirectory=gui`, creating the dir first). Chromium/electron via
|
||||
ozone refuse to map an `xdg_toplevel` without a reachable session bus
|
||||
("Failed to connect to the bus" → binds `xdg_wm_base` then destroys it
|
||||
= invisible window even though CDP works). The fix is **not** to wrap
|
||||
each client in its own `dbus-run-session` (a private throwaway bus per
|
||||
process — that's a _separate_ session, defeating the one-session
|
||||
model); it's this one shared bus; `systemd.globalEnvironment` exports
|
||||
its address as `DBUS_SESSION_BUS_ADDRESS=unix:path=/run/gui/bus` so
|
||||
weston, the harness and every GUI client inherit it.
|
||||
- **Fixed Wayland socket name (`--socket=wayland-0`)**: weston is
|
||||
launched with `--socket=wayland-0` so the socket path is
|
||||
deterministic. `nix/agent-modules/weston-vnc.nix` exports `WAYLAND_DISPLAY=wayland-0`
|
||||
and `XDG_RUNTIME_DIR=/run/gui` as global system environment
|
||||
variables (gated on `services.hyperhive.agent.gui.enable`) so every systemd service
|
||||
in the container inherits them. Without this, services starting
|
||||
Wayland clients couldn't find the compositor — libwayland falls
|
||||
back to a headless display or errors out, the app "works" on a
|
||||
second invisible display, and the VNC session shows a blank weston
|
||||
desktop (known double-screen issue).
|
||||
- **VNC bind address**: Weston's `vnc-backend` has no CLI
|
||||
bind-address flag (unlike the `rdp-backend`'s `--address`), so the
|
||||
listener binds `0.0.0.0`. The harness relay only connects via
|
||||
`127.0.0.1`; the host firewall blocks the per-agent VNC port range
|
||||
from external access. A future weston.ini `[vnc] address=` would
|
||||
restrict the bind directly, once upstream supports it.
|
||||
- **PAM service name**: literal `weston-remote-access` — that's the
|
||||
string libweston passes to `pam_start()` in `libweston/auth.c`.
|
||||
Using `weston` falls back to the system default PAM stack and
|
||||
rejects auth. `weston-vnc.nix` configures the service to
|
||||
`pam_permit.so` for all three module types (auth / account / session) so the
|
||||
browser's empty Apple-DH credentials (type 30) always pass —
|
||||
neatvnc ≥ 0.9 calls the PAM auth callback regardless of
|
||||
`weston.ini` `auth-method=none`, so the permit fallback is what
|
||||
actually lets the empty-cred client through.
|
||||
- **`Type = "simple"` (not `notify`)**: `switch-to-configuration`
|
||||
must never block on weston signalling readiness. A misconfigured
|
||||
weston degrades to a `Restart=on-failure` loop visible in
|
||||
`journalctl`, it doesn't abort the `nixos-container update`.
|
||||
Same reasoning as the best-effort oneshots in `docs/process/conventions.md`.
|
||||
- **`[core] idle-time=0`**: disables weston's 300-second idle
|
||||
timeout. Without it the VNC desktop fades to black and
|
||||
desktop-shell shows its unlock screen — useless for an
|
||||
agent desktop viewed over `/screen`. `idle-time=0` updates the
|
||||
idle timer with a 0ms delay, which
|
||||
`wl_event_source_timer_update` treats as "disarm," so the
|
||||
compositor never goes idle and never locks.
|
||||
|
||||
## Nix docs pipeline
|
||||
|
||||
### Nix options reference (`nix/docs/default.nix`)
|
||||
|
||||
`pkgs.nixosOptionsDoc` over two evaluated module trees:
|
||||
`hostEval` (a stub NixOS system loading the `nix/host-modules/` aggregator with every
|
||||
hyperhive subsystem `mkForce false` so heavy build inputs stay out of
|
||||
the eval) and `agentEval` (evaluates `agent.nix` fresh for the
|
||||
per-agent options tree).
|
||||
|
||||
Five output trees consumed by `flake.nix`, all **markdown**:
|
||||
|
||||
- `docs-host` — operator-facing host module options
|
||||
(`services.hyperhive.*` minus the `swarm`/`deploy` subtrees below)
|
||||
- `docs-swarm` — swarm-wide facts, identical on every host in the swarm
|
||||
(`services.hyperhive.swarm.*`)
|
||||
- `docs-deploy` — this host's own deployment decisions, necessarily
|
||||
different per host (`services.hyperhive.deploy.*` — added by
|
||||
`nix/host-modules/deploy.nix`, split out of `swarm.*` for exactly this
|
||||
reason: see that file's own comment)
|
||||
- `docs-agent` — per-agent harness options (`hyperhive.*`
|
||||
declared in `nix/agent-modules/`)
|
||||
- `docs` — bundle of `index.md` + `host.md` + `swarm.md` +
|
||||
`deploy.md` + `agent.md`
|
||||
|
||||
Pipeline:
|
||||
|
||||
- CommonMark from `nixosOptionsDoc.optionsCommonMark` is the only
|
||||
output — the source of truth, emitted as `.md`.
|
||||
- **The website repo renders HTML + CSS downstream**
|
||||
(its own `options.nix`), consuming this bundle's `host.md` /
|
||||
`agent.md`, rendering them with `cmark-gfm`, and sharing one
|
||||
stylesheet (`docs.css`) across `/options/` and the prose `/docs/`
|
||||
tree. Keeping rendering in the website means the theme has a single
|
||||
home, sharing colours across both trees.
|
||||
- `transformOptions` strips the nix-store prefix from option
|
||||
declaration paths and rewrites them as forge URLs, so the
|
||||
rendered docs link back to the source.
|
||||
|
||||
Host options live entirely under `services.hyperhive.*`. `pickSubtrees`
|
||||
takes `["services" "hyperhive"]` as its root, so the options tree picks
|
||||
up everything under that root — picking against
|
||||
stray roots produces an empty tree and renders the host page as
|
||||
template chrome with no `<h2>` headers. `docs-host` then drops the
|
||||
`swarm`/`deploy` subtrees from that picked tree (`removeAttrs`, not a
|
||||
second `pickSubtrees` root — the one exclusion this file needs doesn't
|
||||
earn a general-purpose helper) since `docs-swarm`/`docs-deploy` each
|
||||
pick their own root instead.
|
||||
|
||||
#### Docs drv stability: `nixSrc`
|
||||
|
||||
Naively, the docs evaluation depends on `self` (the flake's store path),
|
||||
so every commit — even Rust-only or frontend-only changes — produces new
|
||||
docs drv hashes. The remote builder must rebuild docs from scratch for
|
||||
every PR branch, and if its store is full the build fails with a cached
|
||||
failure that blocks CI for the whole branch.
|
||||
|
||||
The fix (`nix/docs/default.nix`):
|
||||
|
||||
1. **`nixSrc`** — `builtins.path` on the `nix/` directory, wrapped in
|
||||
`builtins.unsafeDiscardStringContext` to strip `self`'s store-path
|
||||
context. The resulting store path is content-addressed from the nix/
|
||||
file contents only. Docs drvs only change when a `.nix` file changes.
|
||||
|
||||
2. The package options the modules consume (`services.hyperhive.agent.packages.*`,
|
||||
`services.hyperhive.deploy.hive-controller.*`) carry no in-module defaults and every
|
||||
default that references them has a `defaultText`, so the doc walk
|
||||
never forces a package — no stubs needed, and the Rust/frontend
|
||||
build closure stays out of the eval.
|
||||
|
||||
3. Nix evaluates both `hostEval` and `agentEval` from `nixSrc` paths
|
||||
(not `self`), so the docs drv dependency chain ends at `nixSrc`.
|
||||
|
||||
Why `builtins.unsafeDiscardStringContext`? The path string
|
||||
`toString self + "/nix"` carries `self`'s string context, which would
|
||||
make `builtins.path` include `self` as a build dependency even after
|
||||
content-addressing the directory. Discarding the context makes the
|
||||
resulting `nixSrc` truly independent of `self`'s store path.
|
||||
Loading…
Reference in a new issue