docs: restructure into topic subdirectories, collapse duplicated index

Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):

Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
  getting-started/  setup.md
  agent-lifecycle/  agent-hierarchy.md, approvals.md, persistence.md
  trust-boundary/   boundary.md, security.md
  integrations/     forge.md, matrix.md, github.md, knowledge.md
  networking/       gateway.md, network.md, snapshot-store.md
  scheduler/        jobq.md, coordinator.md, ci.md, observability.md
  process/          conventions.md, gotchas.md, pr-review-gate.md
  web-ui/           terminal-rendering.md (moved into the EXISTING dir,
                    per mara's correction to the original getting-started
                    guess -- it's UI implementation detail, not onboarding)

The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).

Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).

Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).

Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.

nix fmt clean, both pre-push lints clean.

CLAUDE.md

@@ -10,12 +10,12 @@ when you need depth on a subsystem. This file is the index.
   the env var — inside an agent container `localhost` is the *agent*, not
   the forge, so a hardcoded loopback address fails to connect.
 - Operator/agent trust-boundary design:
-  **[docs/boundary.md](docs/boundary.md)** (`area/ops` issues
+  **[docs/trust-boundary/boundary.md](docs/trust-boundary/boundary.md)** (`area/ops` issues
   for the deployment/gateway/privsep work — the separator is a **slash**,
   and `--label area:ops` now fails with `did you mean "area/ops"?`).
 - Agent trust model (trust boundary, prompt-injection threat model,
   capability = accepted risk), credential isolation + sandbox threat model:
-  **[docs/security.md](docs/security.md)**.
+  **[docs/trust-boundary/security.md](docs/trust-boundary/security.md)**.
 
 ## Repo map
 
@@ -38,7 +38,7 @@ hand-maintained per-file tree drifts out of sync with the code.
 - **`hivectl/`** — standalone operator CLI (`hivectl` binary). Talks to
   the `hive-c0re` daemon over the host admin socket (`hive-host-sock`
   wire types) — does NOT link `hive-c0re`. Full, always-current verb
-  reference (CI-enforced against the clap tree, see `docs/conventions.md`):
+  reference (CI-enforced against the clap tree, see `docs/process/conventions.md`):
   [`docs/tools/hivectl-cli.md`](docs/tools/hivectl-cli.md).
 - **`hive-agent/`**, **`hive-agent-mcp/`** —
   in-container harness, two sibling crates for every agent (not a
@@ -80,7 +80,7 @@ hand-maintained per-file tree drifts out of sync with the code.
 - **`hive-priv/`** — minimal root privileged-helper, socket-activated at
   `/run/hive/priv.sock`; performs the few root operations (bind-mount
   edits, nsenter) the unprivileged `hive-c0re` delegates to it. See
-  `docs/boundary.md`.
+  `docs/trust-boundary/boundary.md`.
 - **`hive-forge/`** — `hive-forge` Forgejo CLI wrapper; one module per
   verb under `src/verbs/`.
 - **`hive-forge-notify/`** — per-agent notification poller daemons; turns
@@ -156,7 +156,7 @@ hand-maintained per-file tree drifts out of sync with the code.
   written in place** — one file, shared with `swarm-authelia-bridge`; see
   that crate's README for what both writers must uphold. Full, always-current
   verb reference (CI-enforced against the clap tree, same pattern as
-  `hivectl`'s — see `docs/conventions.md`):
+  `hivectl`'s — see `docs/process/conventions.md`):
   [`docs/tools/swarmctl-cli.md`](docs/tools/swarmctl-cli.md).
 
 ### External dependencies with no directory here
@@ -191,91 +191,17 @@ hand-maintained per-file tree drifts out of sync with the code.
 
 ## Reading paths
 
-Pick the doc that matches your task. None depend on the others —
+**[`docs/README.md`](docs/README.md) is the single index** — grouped by topic
-read them à la carte.
+(getting started, agent lifecycle, trust boundary, integrations, networking,
-
+scheduler, process). Read it instead of a second copy here; this file used to
-- **"How do I bring a fresh hive online (first-run hivectl
+carry its own parallel "Reading paths" list, and the two drifted out of sync
-  bootstrap)?"** → [`docs/setup.md`](docs/setup.md).
+with each other.
-- **"What does the dashboard look like?"** →
-  [`docs/web-ui.md`](docs/web-ui.md) (index; sub-pages:
-  [`shape`](docs/web-ui/shape.md),
-  [`dashboard`](docs/web-ui/dashboard.md),
-  [`agent`](docs/web-ui/agent.md)).
-- **"How does the per-agent terminal classify + colour
-  events?"** → [`docs/terminal-rendering.md`](docs/terminal-rendering.md).
-- **"How does claude get its prompt and what tools does it have?"** →
-  [`docs/turn-loop/`](docs/turn-loop/README.md) (index: the loop, binary shape,
-  turn outcomes; sub-pages:
-  [`claude-invocation`](docs/turn-loop/claude-invocation.md),
-  [`config`](docs/turn-loop/config.md), [`mcp`](docs/turn-loop/mcp.md)).
-- **"How do config changes flow from manager to operator to
-  container?"** → [`docs/approvals.md`](docs/approvals.md).
-- **"What state survives destroy / purge / restart?"** →
-  [`docs/persistence.md`](docs/persistence.md).
-- **"Naming, commit style, wire protocol, the `data-async`
-  pattern."** → [`docs/conventions.md`](docs/conventions.md).
-- **"Why does the nspawn flag look like that?"** →
-  [`docs/gotchas.md`](docs/gotchas.md).
-- **"What nginx vhosts does the gateway serve? How does matrix
-  discovery work?"** → [`docs/gateway.md`](docs/gateway.md).
-- **"How do per-agent forge accounts work? What does forge_notify
-  poll + how does it format wake messages?"** →
-  [`docs/forge.md`](docs/forge.md).
-- **"What verbs does `hive-forge` support? How do I post a comment,
-  upload an attachment, manage subscriptions?"** →
-  [`docs/tools/forge.md`](docs/tools/forge.md).
-- **"What does `hivectl` do? How do I provision a forge/matrix account,
-  manage gateway users, restart containers, or drop into an agent shell?"** →
-  [`docs/tools/hivectl.md`](docs/tools/hivectl.md).
-- **"How does the matrix-tuwunel container work? What about
-  fluffychat-web and per-agent matrix accounts?"** →
-  [`docs/matrix.md`](docs/matrix.md).
-- **"How do I give an agent a GitHub account (`gh` + `git push`)?
-  How is the PAT injected?"** → [`docs/github.md`](docs/github.md).
-- **"How does DNS resolution work in agent containers? What's the
-  bridge network for?"** → [`docs/network.md`](docs/network.md).
-- **"How do I connect two hives into a swarm? How do I declare peer
-  hives and configure TLS trust?"** →
-  [`docs/swarm/`](docs/swarm/README.md).
-- **"Which deployment shapes are supported — all-local, services on the
-  controller host, fully spread out?"** →
-  [`docs/swarm/services.md`](docs/swarm/services.md#deployment-shapes).
-- **"Where do agent snapshots go? How does the swarm's `btrfs receive`
-  endpoint authenticate a pushing hive, and what does a snapshot
-  actually contain?"** →
-  [`docs/snapshot-store.md`](docs/snapshot-store.md).
-- **"What is the job queue, as a general idea?"** →
-  [`docs/jobq.md`](docs/jobq.md) (operator-facing, no hive-c0re specifics).
-- **"How does the rebuild queue work? What are the concrete step kinds,
-  queue sources, scheduler internals?"** →
-  [`docs/coordinator.md`](docs/coordinator.md).
-- **"How does the CI runner work? What's the auto-registration flow?"** →
-  [`docs/ci.md`](docs/ci.md).
-- **"What is `/knowledge`? How does the hive-wide knowledge repo sync,
-  and how do I contribute a document?"** →
-  [`docs/knowledge.md`](docs/knowledge.md).
-- **"How do I export Claude Code metrics (tokens, cost, tool calls) to
-  a Prometheus/Grafana collector? What OTEL options are available?"** →
-  [`docs/observability.md`](docs/observability.md).
-- **"What does a PR review verdict actually gate? Should a reviewer
-  wait for CI before approving? What does arming auto-merge commit an
-  operator to?"** → [`docs/pr-review-gate.md`](docs/pr-review-gate.md).
 
 ## Conventions & process
 
-The docs below own the details — this section just points at them.
-
-- **Commit style, naming, identity, reconcile verb:** →
-  [`docs/conventions.md`](docs/conventions.md).
 - **Never add `#[allow(clippy::…)]`** — fix the lint instead (extract a
   helper, add backticks, etc.). Details + worked examples: →
-  [`docs/conventions.md`](docs/conventions.md#building--local-checks).
+  [`docs/process/conventions.md`](docs/process/conventions.md#building--local-checks).
-- **NixOS / nspawn quirks** (bind mounts, conf flags, etc.): →
-  [`docs/gotchas.md`](docs/gotchas.md).
-- **Turn loop, sentinels (rate-limit, auth-failed), context
-  window:** → [`docs/turn-loop/`](docs/turn-loop/README.md).
-- **Two-step spawn, approval flow, flake.lock validation:** →
-  [`docs/approvals.md`](docs/approvals.md).
 - **Pre-push lint hook** (catches tracker-tag and comment-block failures
   before CI does — install once per clone):
   `ln -sf ../../scripts/pre-push .git/hooks/pre-push`

docs/README.md

@@ -11,55 +11,53 @@ declarations.
 
 ## Getting started
 
-- **Bringing a fresh hive online?** → [`setup.md`](setup.md) (first-run
+- **Bringing a fresh hive online?** → [`getting-started/setup.md`](getting-started/setup.md)
-  `hivectl` bootstrap).
+  (first-run `hivectl` bootstrap).
 - **What does the dashboard look like, and how do I use it?** →
   [`web-ui/`](web-ui/README.md) — the operator-facing starting point;
   its own sub-pages ([`shape`](web-ui/shape.md),
   [`dashboard`](web-ui/dashboard.md), [`agent`](web-ui/agent.md),
-  [`css-vars`](web-ui/css-vars.md)) go deeper into implementation.
+  [`css-vars`](web-ui/css-vars.md), [`terminal-rendering`](web-ui/terminal-rendering.md))
+  go deeper into implementation.
 - **What tools does an agent (or the operator) have available?** →
   [`tools/`](tools/README.md) — `hivectl` (yours) plus every agent's MCP
   tool surface (bash, forge, lifecycle, matrix, scheduling).
 
-## Dashboard & agent UI internals
+## Agent lifecycle
-
-- **How does the per-agent terminal classify + colour events?** →
-  [`terminal-rendering.md`](terminal-rendering.md).
-
-## Turn loop, config, approvals
 
+- **How do config changes flow from manager to operator to container?** →
+  [`agent-lifecycle/approvals.md`](agent-lifecycle/approvals.md) (two-step spawn, approval
+  state machine, `flake.lock` validation).
+- **What state survives destroy / purge / restart?** →
+  [`agent-lifecycle/persistence.md`](agent-lifecycle/persistence.md).
+- **Who can do what to whom — agent hierarchy and privilege?** →
+  [`agent-lifecycle/agent-hierarchy.md`](agent-lifecycle/agent-hierarchy.md).
 - **How does claude get its prompt, and what tools does it have?** →
   [`turn-loop/`](turn-loop/README.md) — the loop, binary shape, turn
   outcomes; sub-pages: [`claude-invocation`](turn-loop/claude-invocation.md),
   [`config`](turn-loop/config.md), [`mcp`](turn-loop/mcp.md).
-- **How do config changes flow from manager to operator to container?** →
-  [`approvals.md`](approvals.md) (two-step spawn, approval state machine,
-  `flake.lock` validation).
-- **What state survives destroy / purge / restart?** →
-  [`persistence.md`](persistence.md).
 
 ## Trust boundary & security
 
 - **What's the operator/agent trust boundary? What's a capability?** →
-  [`boundary.md`](boundary.md).
+  [`trust-boundary/boundary.md`](trust-boundary/boundary.md).
 - **Agent trust model, prompt-injection threat model, credential
-  isolation?** → [`security.md`](security.md).
+  isolation?** → [`trust-boundary/security.md`](trust-boundary/security.md).
-- **Who can do what to whom — agent hierarchy and privilege?** →
-  [`agent-hierarchy.md`](agent-hierarchy.md).
 
 ## Accounts & integrations
 
 - **How do per-agent forge accounts work? What does `forge_notify` poll,
-  and how does it format wake messages?** → [`forge.md`](forge.md) (the
+  and how does it format wake messages?** → [`integrations/forge.md`](integrations/forge.md)
-  hive's own Forgejo); [`tools/forge.md`](tools/forge.md) for the
+  (the hive's own Forgejo); [`tools/forge.md`](tools/forge.md) for the
   `hive-forge` CLI verbs agents actually call.
 - **How does the matrix-tuwunel container work? Multiple accounts per
-  agent?** → [`matrix.md`](matrix.md) (the homeserver);
+  agent?** → [`integrations/matrix.md`](integrations/matrix.md) (the homeserver);
   [`tools/matrix.md`](tools/matrix.md) for the MCP tool surface and
   `hyperhive.matrixAccounts`.
 - **How do I give an agent a GitHub account (`gh` + `git push`)? How is
-  the PAT injected?** → [`github.md`](github.md).
+  the PAT injected?** → [`integrations/github.md`](integrations/github.md).
+- **What is `/knowledge`? How does the hive-wide knowledge repo sync,
+  and how do I contribute a document?** → [`integrations/knowledge.md`](integrations/knowledge.md).
 - **What does `hivectl` do? Provisioning, gateway users, container
   shells?** → [`tools/hivectl.md`](tools/hivectl.md) (the curated guide);
   [`tools/hivectl-cli.md`](tools/hivectl-cli.md) for the exhaustive,
@@ -68,25 +66,25 @@ declarations.
 ## Networking & swarms
 
 - **What nginx vhosts does the gateway serve? How does matrix
-  discovery work?** → [`gateway.md`](gateway.md).
+  discovery work?** → [`networking/gateway.md`](networking/gateway.md).
 - **How does DNS resolution work in agent containers? What's the
-  bridge network for?** → [`network.md`](network.md).
+  bridge network for?** → [`networking/network.md`](networking/network.md).
 - **How do I connect two hives into a swarm?** → [`swarm/`](swarm/README.md)
   (peer hives, TLS trust).
 - **Where do agent snapshots go? How does the swarm's `btrfs receive`
   endpoint authenticate a pushing hive?** →
-  [`snapshot-store.md`](snapshot-store.md).
+  [`networking/snapshot-store.md`](networking/snapshot-store.md).
 
 ## Scheduler, CI, observability
 
 - **What is the job queue, as a general idea (not hive-c0re specifics)?** →
-  [`jobq.md`](jobq.md) — operator-facing, no implementation detail.
+  [`scheduler/jobq.md`](scheduler/jobq.md) — operator-facing, no implementation detail.
 - **How does the rebuild queue work? What are the concrete step kinds,
-  queue sources, scheduler internals?** → [`coordinator.md`](coordinator.md).
+  queue sources, scheduler internals?** → [`scheduler/coordinator.md`](scheduler/coordinator.md).
 - **How does the CI runner work? What's the auto-registration flow?** →
-  [`ci.md`](ci.md).
+  [`scheduler/ci.md`](scheduler/ci.md).
 - **How do I export Claude Code metrics (tokens, cost, tool calls) to
-  Prometheus/Grafana?** → [`observability.md`](observability.md).
+  Prometheus/Grafana?** → [`scheduler/observability.md`](scheduler/observability.md).
 
 ## Crate reference
 
@@ -98,8 +96,8 @@ declarations.
 ## Process & conventions
 
 - **Naming, commit style, wire protocol, the `data-async` pattern?** →
-  [`conventions.md`](conventions.md).
+  [`process/conventions.md`](process/conventions.md).
-- **Why does the nspawn flag look like that?** → [`gotchas.md`](gotchas.md)
+- **Why does the nspawn flag look like that?** → [`process/gotchas.md`](process/gotchas.md)
   (bind mounts, conf flags, other NixOS/nspawn quirks).
-- **What is `/knowledge`? How does the hive-wide knowledge repo sync,
+- **What does a PR review verdict actually gate?** →
-  and how do I contribute a document?** → [`knowledge.md`](knowledge.md).
+  [`process/pr-review-gate.md`](process/pr-review-gate.md).

docs/agent-lifecycle/agent-hierarchy.md

@@ -0,0 +1,246 @@
+# Agent hierarchy & privileges
+
+Every agent has a place in an operator-editable parent/child tree, used
+to scope which agents can manage which others. This doc covers how the
+tree is stored and edited today, the rules that are meant to run on top
+of it once enforcement is finished, and where the manager still gets
+special-cased in the meantime. Tracking issue: hyperhive#361
+(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361`).
+
+## Where the tree lives
+
+Topology lives in the hive-c0re-owned **meta repo**, alongside
+`flake.nix`, at `/var/lib/hyperhive/meta/topology.json`:
+
+```json
+{
+  "ruth": null,
+  "alice": null,
+  "bob": "alice"
+}
+```
+
+`null` = root-level agent. New agents **default to root** — there is no
+structural manager that everything hangs under. Hierarchy is built
+explicitly: an agent that requests a sub-agent gets a
+requester-as-parent edge written at its `init_config` approval (so
+`bob` above was spawned by `alice`), and the operator can reparent any
+agent, including the bootstrap container (`ruth`) — it's just another
+root. The manager is reparentable like any other agent; there's no
+"structurally root" carve-out. Its privileges live on its MCP socket,
+not its tree position (see *Manager special-casing today* below).
+
+### Reparenting
+
+- CLI: `hivectl agent <child> set-parent --parent <new>` (or `--root`
+  to promote). Exactly one of `--parent` / `--root` is required.
+- Dashboard: `POST /api/topology/set-parent` (form fields `child`,
+  optional `new_parent` — absent / empty ⇒ promote to root).
+- Wire: `HostRequest::SetParent { child, new_parent: Option<String> }`.
+
+All three go through the same validation, which refuses:
+
+- unknown `child` / `new_parent` (typo guard),
+- self-parenting,
+- cycles (a bounded ancestor walk — moving the manager under one of
+  its own descendants is the only real safety concern here, and it's
+  caught the same way as any other agent).
+
+Setting a parent to its current value is a no-op (no disk write). A
+successful change triggers an immediate rescan, so connected dashboard
+viewers see the tree repaint without polling.
+
+### Why meta, not per-agent `agent.nix`
+
+An agent shouldn't be able to claim a parent without that parent's
+consent, and operator-driven re-parenting shouldn't require touching
+the moved agent's config. Topology IS a system-level concern; meta is
+where system-level facts live.
+
+### How `topology.json` gets updated
+
+- **Read** — parsed into an agent→parent map; a missing or unparsable
+  file degrades safely to "every agent is root" (covers a fresh
+  install that hasn't synced yet).
+- **Reconcile** — runs alongside the periodic meta/flake regeneration.
+  New agents default to root unless they already carry an explicit
+  parent edge from an `init_config` approval; existing entries
+  (including operator overrides) are preserved; removed agents drop.
+  Agents that are approved but not yet spawned keep their edge too, so
+  it survives the gap until the container actually appears.
+- **Inject** — each container's parent (if any) is exposed to its own
+  environment as `HIVE_PARENT`, so the harness / system-prompt
+  renderer can see it.
+- **Surface** — every rescan re-reads `topology.json` and populates
+  `ContainerView.parent`, which the dashboard renders as a tree.
+
+See `hive-c0re/src/agent_config/topology.rs` and `hive-c0re/src/meta.rs`'s
+module docs for the exact call chain.
+
+### Current limitation: state-dir visibility lags topology
+
+Reparenting today is purely a JSON edit. Only the top-level manager
+(`root`) gets `/var/lib/hyperhive/agents` bind-mounted at `/agents` in
+its container, so sub-agents don't yet see their would-be children's
+state dirs. Once sub-manager bind mounts land alongside capability
+enforcement, reparenting will grow a companion
+umount-old / mount-new / restart-cascade step.
+
+## Planned topology semantics (once ancestor-based enforcement lands)
+
+| operation                                                               | who can do it                                                                           |
+| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
+| `kill` / `start` / `restart` / `update` (any descendant)                | any ancestor                                                                            |
+| `request_init_config` (spawn a new child)                               | any agent, child added under self                                                       |
+| config change via forge PR (any descendant's config)                    | any ancestor                                                                            |
+| `get_logs` (any descendant)                                             | any ancestor                                                                            |
+| moderate reminders (cancel any open thread of a descendant)            | any ancestor                                                                            |
+| `send` / `recv` routing                                                 | parent ↔ same-parent siblings ↔ self ↔ descendants; explicit allow-list for anyone else |
+| `request_update_meta_inputs` (bump meta lock)                           | root agents only (today: just `manager`)                                                |
+
+"Ancestor" walks `ContainerView.parent` chains; cycles are guarded by a
+visited-set at dispatch time (a malformed `topology.json` can't lock
+the dispatcher into a loop).
+
+## Manager special-casing today
+
+Enforcement of the ancestor rules above isn't fully wired yet, so the
+**manager (`ruth`) still gets some hard-coded special treatment**
+other agents don't:
+
+- **Naming/bootstrap** — the manager's broker recipient name, state-dir
+  key, and nixos-container name are all `ruth` (container `h-ruth`).
+  `hive-c0re` spawns it directly at boot if missing, with no operator
+  approval step — every other agent goes through `request_init_config`
+  → approval. Topology-wise, `ruth` is still just another root agent.
+- **Wire-protocol** — the privileged `Request` variants
+  (`RequestInitConfig`; `Kill` / `Start` / `Restart` / `Update`;
+  `GetLogs`; `RequestUpdateMetaInputs`) — marked `*(privileged)*` in
+  `hive-core-agent-sock`'s unified `Request` enum — are reachable only
+  from the manager's socket flavour today. Planned rule for each is in the
+  table above ("any agent, child added under self" for init-config,
+  "any ancestor" for lifecycle/logs); `RequestUpdateMetaInputs` stays
+  a root-only capability even post-milestone, not a topology rule.
+  One exception: `Wake` (inject a `from: <X>` message into the
+  caller's own inbox) isn't really privileged — every per-agent daemon
+  (e.g. `hive-forge-notify`) needs it, and sub-agents already have the
+  equivalent on their own socket.
+- **Storage/mounts** — only the manager container gets
+  `/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
+  manage any agent's state dir — config is not authored there, since a
+  real config change is a PR from a clone), plus RO mounts for
+  `/applied` (diff against what's deployed) and `/meta` (system-wide
+  deploy log). Planned: each agent gets RW to `/agents/<descendant>/`
+  for just its own subtree — the manager's full-forest RW becomes the
+  "root's subtree is everything" case of that same rule. RO `/meta`
+  access will be gated on a "meta read" capability; only
+  `request_update_meta_inputs` writes `flake.lock`, gated by its own
+  capability.
+- **Prompt/tools** — the system prompt uses `<!-- role:agent -->` /
+  `<!-- role:manager -->` marker blocks, and a `Flavor::{Agent,
+  Manager}` switch picks the MCP tool allow-list claude sees. Both are
+  already parametrised on a single flavour value, so the planned
+  per-capability-group version (`cap:<group>` prompt blocks + a
+  matching tool allow-list) is additive rather than a rewrite.
+- **State dirs** — *not* special-cased: `HYPERHIVE_STATE_DIR` is
+  injected uniformly via `systemd.globalEnvironment` for every
+  container including the manager, so all token/state paths resolve
+  through it the same way everywhere.
+- **Scattered ownership checks** — a handful of independent
+  manager-only overrides exist across `hive-c0re` today: loose-ends
+  visibility (manager sees hive-wide, sub-agents only their own),
+  `destroy` refusing to act on the manager, and crash-watch skipping
+  the manager (it auto-restarts via systemd instead of going through
+  the crash-watch loop). Each is planned to become an
+  ancestor/descendant check instead of a manager-name check — see the
+  module docs for `loose_ends.rs`, `stores/broker.rs`, `actions.rs`,
+  and `workers/crash_watch.rs` for the current owner-check logic in
+  each. (Question/answer routing and its own manager-override cancel
+  path — formerly `hive-c0re/src/questions.rs` and
+  `stores/operator_questions.rs` — has been removed entirely; reminder
+  cancellation is now handled fully in-agent, see the note on
+  `CancelLooseEndKind::Reminder` in `hive-c0re/src/socket_server/mod.rs`.)
+
+None of the above is a stable interface — treat the module doc
+comments as the source of truth for exactly which checks exist today.
+
+## Future work: sub-agents inside the same container
+
+When enabled for an agent, it will be able to spawn temporary
+"sub-agents" that run inside its own container — lighter than a full
+nspawn agent. Open questions, not yet wired:
+
+- Inherit caps from parent, or take an explicit narrower set?
+- Survive container restart, or always ephemeral?
+- Inbox: separate from parent, or shared?
+- Filesystem: share parent's `/state` RW, or a sub-dir?
+- Identity: distinct broker recipient name, or address the parent?
+
+## Harness systemd unit shape
+
+One harness serve binary (`hive-agent`, with its `hive-agent-mcp`
+sibling), one shared `nix/agent-modules/` tree, one service unit
+(`systemd.services.hive-agent`) for all agents. There is no separate
+manager service name or role distinction in the harness — privilege
+differences live server-side in the broker socket (which tool groups
+and manager-surface calls each agent receives).
+
+`agent.nix` and `ruth.nix` both import the shared `nix/agent-modules/`.
+`ruth.nix` additionally sets forge defaults to suppress the
+subscription/participation firehose so ruth's inbox stays focused on
+direct mentions, reviews, and assignments.
+
+### Environment variables set on the unit
+
+- `HOME = /home/<userName>` — systemd defaults `HOME` to `/` for
+  services without `User=` set; with the per-agent user the harness
+  needs the right home so claude finds its bind-mounted `~/.claude/`
+  session dir.
+- `HIVE_STATIC_DIR = <mergedDist>` — `tower_http::ServeDir` root for
+  the per-agent web UI; merged dist = agent default + every
+  `hyperhive.frontend.extraFiles` overlay.
+- `HIVE_ASSETS_DIR = pkgs.hyperhive-assets/share/hyperhive` — set
+  directly on the unit, **not** via `environment.variables`, because
+  the latter only populates `/etc/profile` which systemd services
+  don't inherit.
+
+### `PATH` setup (the wrapper-dir trick)
+
+```nix
+path = [ "/run/wrappers" "/run/current-system/sw" ];
+```
+
+`/run/wrappers` (not `/run/wrappers/bin`) comes first so setuid
+wrappers — notably `sudo` — resolve before bare nix-store binaries; see
+[`docs/process/gotchas.md`](../process/gotchas.md) ("`systemd.services.*.path` appends
+`/bin` to every entry") for why the trailing `/bin` matters in
+general. It's load-bearing here because the harness runs as the
+per-agent user: without the wrapper dir on `PATH`, `sudo` resolves to
+the non-setuid nix-store binary and every
+`hyperhive.user.passwordlessSudo` grant fails with "must be owned by
+uid 0 and have the setuid bit set."
+
+### `serviceConfig` highlights
+
+- `ExecStart = pkgs.hyperhive/bin/hive-agent` — same binary for every
+  agent.
+- `Restart = on-failure`, `RestartSec = 2` — keeps the harness
+  resilient across transient crashes without thundering retries.
+- `RuntimeDirectory = "hive-config"` → `/run/hive-config/` owned by
+  `User=`, auto-cleared on stop. The harness writes regenerated
+  `claude-{mcp-config,settings,system-prompt}` files there
+  (`paths::config_dir`). Deliberately separate from `/run/hive`, which
+  the host bind-mounts in root-owned and which holds hive-c0re's
+  `mcp.sock`.
+- `User = Group = userName` — drops root inside the container; sudo is
+  the explicit escalation surface (`hyperhive.user.passwordlessSudo`).
+
+## Cross-references
+
+- Milestone: "Agent privileges and sub-agents"
+  (`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361`)
+- Dashboard render: "show agent topology in container list"
+  (`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/363`)
+- Audit table source: milestone comment
+  (`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361#issuecomment-3335`)
+- Operator/agent trust boundary (orthogonal axis): [`boundary.md`](../trust-boundary/boundary.md)

docs/agent-lifecycle/approvals.md

@@ -0,0 +1,679 @@
+# Approvals + helper events
+
+The approval queue is hyperhive's pivot: nothing that changes the
+shape of an agent (its config, whether it exists) happens without an
+operator click. The submitting agent — any agent with the `approvals`
+tool group, which manages the config of its **direct children** (the
+root agent for top-level agents; a sub-manager for its own subtree) — is
+the policy gate in front of that queue; helper events are how it stays
+informed about what happens after a decision lands.
+
+## For operators
+
+Every add/remove/change to an agent lands on your dashboard's Y3R
+C4LL tab (or `hivectl approvals pending` / `approve <id>` from the
+CLI) before it takes effect. What you'll see, and what to do with it:
+
+- **Config change** (`MergeConfigPr`) — an agent proposed a change to
+  another agent's config (or its own, via a sub-manager) as a forge
+  pull request. Review the diff on the forge — the dashboard card
+  links straight to it, same as reviewing any other PR. Approving
+  triggers the deploy automatically: hive-c0re re-verifies the PR
+  hasn't moved since you looked at it, evaluates it (a dry run,
+  nothing applied yet), merges it, and rebuilds the container. If
+  anything in that chain fails, the change rolls back automatically —
+  the agent stays on its last-good config, no recovery action needed
+  from you.
+- **New agent** (`InitConfig` then `Spawn`) — creating a brand-new
+  agent is two approvals. `InitConfig` creates the config repo and
+  seeds it from a template; `Spawn` creates the container from that
+  config. Tailoring the template first is not a separate mechanism —
+  it's the config-change flow above, a PR you review like any other. Every later change goes through the config-change flow
+  above — there's no repeat "spawn" for an existing agent.
+- **Meta/flake update** (`UpdateMetaInputs`) — an agent asked to bump
+  one or more Nix flake inputs (or all of them). Approving runs the
+  update and commits the lock change; it doesn't rebuild anything by
+  itself.
+- **Scheduled prompt** (`SchedulePrompt`) — an agent asked to schedule
+  a message to one or more inboxes at a future time. You can also add
+  schedules yourself directly from the SCH3DUL3S tab, which skips this
+  approval step entirely — the gate here is specifically for an
+  *agent* asking to schedule something, not for you doing it.
+
+Don't want to approve something? **Deny it** (`DENY` on the dashboard
+card, or `hivectl approvals deny <id>`) — nothing runs. Either way the
+submitting agent is always notified their request was denied; what's
+optional is only the reason text, which you can add on the dashboard's
+prompt (cancelling that prompt aborts the whole deny, not just the
+reason) but not from the CLI. Denying is final: a denied approval
+can't be re-approved later, the agent has to submit a fresh one (a new
+PR, a new request).
+
+Everything below this point is the implementation detail behind that
+flow.
+
+## End-to-end approval flow
+
+Config changes flow through a **forge pull request** on the agent's
+`agent-configs/<name>` repo — the same surface agents use for code PRs.
+There is no bespoke MCP tool for config changes: opening the PR IS the
+request.
+
+1. The submitting agent (the child's parent, holding the `approvals`
+   tool group) **clones** `agent-configs/<name>`, edits it there (any
+   tracked path, but `agent.nix` is the contract entry point), commits
+   with its own git identity, and pushes a branch + opens a PR with
+   `hive-forge` — the same way it would change any other repo.
+   The bind-mounted `/agents/<name>/config/` is a **copy for reading** a
+   config, not the tree to edit: authoring in place there produces no PR
+   and no approval. (It is currently mounted read-write, which is a
+   defect tracked separately, not an authoring path.)
+   Branch protection (push/merge whitelist = `core`, approvals whitelist
+   = operator team; see "Forge mirror" and #1787) makes the agent a
+   write collaborator that **cannot merge its own config PR**.
+2. hive-c0re's `/webhook/config-pr` endpoint receives the Forgejo
+   `pull_request` event (opened / synchronized / reopened) and queues a
+   `MergeConfigPr` approval; a poll fallback catches any missed webhook.
+   The approval row stores the PR **number** (`commit_ref`) and the PR
+   **head sha at queue time** (`fetched_sha` — the "reviewed" sha). If
+   the PR head later moves, the stale approval is superseded by a fresh
+   one pinned to the new head, so the operator always reviews what will
+   actually deploy.
+3. The operator reviews the PR **on the forge** (native diff, threaded
+   comments, CI status) and sees a matching card on the dashboard with a
+   "review PR on forge" deep link. They click ◆ APPR0VE (or
+   `hivectl approvals approve <id>` on the CLI) once satisfied.
+4. On approve, a deploy DAG runs three phases under a resource-holding
+   `DeployWindow` root (see *Queue templates* below):
+   - `MergeVerify` re-reads the live PR head and **aborts if it drifted**
+     from the reviewed `fetched_sha` (the submitter must push again,
+     which queues a fresh approval); then fetches that head into the
+     applied repo and **eval-verifies** it — a flake eval on a throwaway
+     checkout. This is the trust gate: it relies on c0re's own eval, not
+     on any in-repo (agent-forgeable) signal like a CI status. Nothing is
+     mutated in this phase, so a rejection here leaves the forge and the
+     applied repo exactly as they were.
+   - `DeployApply` parks the pre-merge `applied/main` in
+     `refs/hyperhive/rollback/<approval-id>`, then fast-forward-merges
+     the reviewed head to the forge config repo's `main` (this IS the
+     merge — a `core`-authenticated ff-merge pinned to the reviewed sha,
+     so a moved PR head can't substitute bytes), and runs the deploy
+     proper (`deploy_applied_target`): ff `applied/main`, two-phase meta
+     deploy, container rebuild. On success it drops the rollback ref and
+     plants `deployed/<id>`.
+   - `DeployTail` runs on **every** outcome, including a cancel-cascade.
+     If the rollback ref survived, the deploy never confirmed good: it
+     rolls `applied/main` back, resyncs the working tree, and aborts the
+     staged meta lock, so the agent stays on its last-good tree. Then it
+     mirrors the config repo (and its new deploy tag) to the forge.
+
+   The rollback state lives in a **git ref, not a local variable**, on
+   purpose: hive-c0re can restart between the apply and the tail, and the
+   tail still has to know what to undo when it does.
+5. `HelperEvent::ApprovalResolved` (and `Rebuilt`) land in the
+   **submitting agent's** inbox via `notify_submitter`, carrying both the
+   canonical sha and the terminal tag (the approval row carries a
+   `submitter` column recording the agent the change is for).
+
+### Withdrawing a pending approval
+
+The submitting agent can call `cancel_loose_end(kind: "approval", id)` to
+withdraw an approval that hasn't been acted on yet.
+The row transitions to `ApprovalStatus::Cancelled` (distinct from
+`Denied`/`Failed`), the dashboard pulls the card out of the
+pending pane, and `ApprovalResolved { status: "cancelled" }` fires
+on the root agent + dashboard channels. Approvals that have already
+been approved/denied/failed return an error — the resolution is
+final once the operator (or a lifecycle failure) acted on the row.
+
+The socket refuses the `approval` kind with a clear error for any
+agent that lacks the `approvals` tool group: only an agent with that
+group submits approvals (for its direct children), so an agent
+without it has nothing of its own to withdraw.
+
+`InitConfig` approvals create a brand-new agent's config repo. On
+approve, hive-c0re seeds it with a default `agent.nix` template and
+pushes a todo (`push_todo_submitter`) into the submitting agent's
+in-container store. The operator then **spawns** the agent (the
+`Spawn` approval / `◆ R3QU3ST SP4WN` button), which creates the
+container from that config.
+
+Changing what the template seeded is not a special case: like every
+later change, it's a PR on that config repo (`MergeConfigPr`), made
+from a clone, reviewed and approved by the operator. The PR flow is
+the one path — an operator can equally drive both steps herself
+through the web UI or the forge.
+
+### Approval kinds (wire shapes)
+
+`ApprovalKind` carries five variants; each maps to a different
+`commit_ref` encoding because that field is overloaded as the
+kind-specific payload carrier.
+
+- `MergeConfigPr` — the config-change flow. Triggered automatically:
+  when an agent opens (or force-pushes) a PR on its
+  `agent-configs/<agent>` forge repo, hive-c0re's `/webhook/config-pr`
+  endpoint receives the Forgejo pull_request event and queues this
+  approval row. No MCP tool call needed — the forge PR IS the request.
+  `commit_ref` stores the **PR number** (decimal), and `fetched_sha` is
+  the PR **head sha at queue time** (the "reviewed" sha). On approve,
+  the deploy DAG's `MergeVerify` phase re-reads the live PR head and
+  aborts if it drifted from `fetched_sha` (submitter must push again to
+  re-trigger), then fetches that head into the applied repo and
+  eval-verifies it; `DeployApply` fast-forward-merges the forge config
+  repo's `main` to it (the merge) and runs `deploy_applied_target`;
+  `DeployTail` compensates on failure. Never a first spawn.
+- `Spawn` — direct container creation from the agent's config repo.
+  `commit_ref` is empty. Submitted via `HostRequest::RequestSpawn`
+  (operator-gated, the `◆ R3QU3ST SP4WN` dashboard button +
+  `hivectl agent <name> request-create` CLI). The host-level `HostRequest::Spawn`
+  variant bypasses the approval queue entirely — privileged-context use
+  only (operator on the host shell, test scripts, one-off recoveries;
+  `hivectl agent <name> create`). This is the **canonical first-spawn**: a new agent's `InitConfig`
+  seeds its config repo, the submitting agent customises it, then the
+  operator spawns to create the container. Subsequent config changes go
+  through a `MergeConfigPr` PR.
+- `InitConfig` — `commit_ref` is empty; the variant just gates
+  "seed the proposed repo with the default template" against
+  operator approval. Step 1 of the two-step spawn flow above.
+- `UpdateMetaInputs` — `commit_ref` stores the JSON-encoded inputs
+  array (`"[]"` = all inputs, `"[\"nixpkgs\"]"` = just nixpkgs,
+  etc.). `agent` field is set to the requesting root agent.
+  On approve hive-c0re runs `nix flake update [inputs...]` on the
+  meta flake and commits the resulting lock changes.
+- `SchedulePrompt` — `commit_ref` stores the JSON-encoded
+  `SchedulePromptPayload` (target list, body, schedule) so the
+  approval row carries the full submission verbatim. On approve
+  hive-c0re inserts a row into `scheduled_prompts` with
+  `source = approval:<id>`; the worker fans the body out as
+  inbox messages to each target at the scheduled time, recurring
+  when `interval_seconds` is set.
+
+### Scheduled prompts (submit paths)
+
+Two ways a row lands in `scheduled_prompts`:
+
+- **Operator-direct** (`source = "operator"`): the operator adds a schedule through the dashboard form. Lands in the table immediately, no approval gate — operator action is already the trust boundary.
+- **Agent-requested** (`source = "approval:<id>"`): an agent submits a `RequestSchedulePrompt` through its MCP socket (the `request_schedule_prompt` tool, `scheduling` group). An `ApprovalKind::SchedulePrompt` row is queued; on approve, hive-c0re inserts the schedule row with `source = approval:<id>` so the audit trail points back at the operator decision (above).
+
+No self-target shortcut: even agent-self schedules need approval. The existing `remind` MCP tool stays the quick self-wake path (no approval, lands directly in the agent's own inbox); this module is the bigger, multi-recipient, operator-visible thing.
+
+### Scheduled prompt worker (catch-up clamp)
+
+When hive-c0re comes back from being down, the worker sees rows whose `next_fire_at_unix` is well in the past. For recurring rows that would mean firing N delayed pulses in a row — spammy and useless. Instead the worker fires **once** per row and bumps `next_fire_at_unix` to the next interval slot ≥ `now`, recording how many cycles were skipped in `last_result` (per-target). Operators see "fired late, caught up from 17 skipped" instead of 17 wake-up storms.
+
+One-shot rows fire once (if past due, on the next worker pass) and are deleted by the worker; recurring rows survive until cancelled.
+
+`targets` is its own table (`scheduled_prompt_targets`) so partial cancellation flips a single row and the dashboard can show last-fired / last-result per recipient. Cancelling every target reaps the parent row on the next worker pass.
+
+### Scheduled prompt delivery: todo, not a broker message
+
+An agent target's delivery is `push_todo` (`Coordinator::push_todo`,
+`docs/coordinator.md` covers the mechanism generally), not a broker
+`Message` — a scheduled prompt wakes its target with a todo instead of
+driving an immediate turn, by design. `key = "schedule:<id>"` per
+target drives `push_todo`'s own upsert-by-key dedup: a re-fire of the
+*same schedule* against a target that hasn't reviewed the last one
+collapses into that one todo instead of stacking up.
+
+**`operator` is the one exception** — it's a valid schedule target but
+has no in-container todo inbox, so it keeps the original broker
+`Message` path (the dashboard mirrors `to == operator` into its own
+pane, same as before).
+
+### Missing-target failure
+
+When a target name doesn't resolve to a known agent (container
+destroyed, operator typo, etc.) the worker:
+
+1. Records `last_result = "no such agent: <name>"` on the
+   per-target row.
+2. Sends a single advisory `Message` from `system` to `operator`
+   naming the schedule, target, and reason. This one stays a `Message`
+   regardless of target type — it's a to-operator advisory about a
+   broken schedule, not the schedule's own delivery.
+3. Continues fanning out to the other live targets.
+
+Transient broker errors (sqlite lock contention, etc.) get the same
+`last_result` annotation plus a `tracing::warn`, and then:
+
+- **Recurring rows** re-arm to the next interval slot — the retry
+  self-heals on the next worker pass.
+- **One-shot rows** are deleted unconditionally after their single
+  fan-out pass; a broker error on a one-shot is not retried (the
+  operator advisory and `last_result` are the only audit trail).
+
+### Reminder delivery: file-path semantics
+
+A reminder may carry a `file_path` (the agent-visible path inside its
+container, e.g. `/agents/<name>/state/foo.md`). On delivery hive-c0re:
+
+1. **Translates** the container path to the host path
+   (`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
+   from outside the container.
+2. **Validates** the path: rejects anything outside the agent's own state
+   subtree, containing `..` (path traversal), or with an empty relative
+   tail. On rejection the write is skipped and the original message is
+   delivered inline with a warning — the reminder still fires.
+3. **Defends against symlink escape**: after `create_dir_all`, the parent
+   dir is canonicalized and re-verified to live under the agent's host
+   state root. The final file is opened with
+   `O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
+   basename cannot redirect the write to an arbitrary host path.
+4. **Writes the body to disk** and delivers a short pointer message in its
+   place, keeping the agent's inbox / wake-prompt small while the bulky
+   payload is read out of band.
+
+Atomicity of the inbox INSERT + `reminders.sent_at` UPDATE is handled
+inside `Broker::deliver_reminders_batch`; the scheduler only computes the
+body strings before calling it.
+
+### Destroy semantics
+
+`HostRequest::Destroy { name, purge }` is the lifecycle tear-down,
+not an approval. Stops + removes the nspawn container, drops the
+systemd drop-in, fails any pending approvals. Persistent state
+(proposed/applied repos, claude credentials, `/state/` notes) is
+**kept by default** — recreating the agent with the same name
+reuses prior config + login. With `purge = true` the agent's
+`/var/lib/hyperhive/{agents,applied}/<name>/` trees are also
+wiped (config history + creds + notes gone forever). The
+root/bootstrap container is destroyable like any other — hive-c0re
+recreates it on the next startup if it's absent, so destroying it is
+transient.
+
+## Meta flake
+
+The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
+declares one flake input per agent (`agent-<n>.url =
+"git+http://<forge>/agent-configs/<n>.git"`) and one
+`nixosConfigurations.<n>` output per agent. Each output wraps
+`inputs.agent-<n>.nixosModules.default` with the identity +
+`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module.
+Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
+
+The declared input url is the agent's **forge config repo** (the
+same `agent-configs/<n>` the config-PR flow lands approved changes
+on), so the meta flake references a reviewable, reproducible source
+rather than a local checkout. hive-c0re authenticates that
+`git+http` fetch via a git credential helper that reads the live
+forge-core token — no token in the url or the lock. The deploy and
+manual-rebuild paths, however, do **not** re-lock from the forge:
+they `--override-input agent-<n>
+git+file:///var/lib/hyperhive/applied/<n>`, locking the exact config
+that `verify_commit` gated and `applied/<n>/main` was
+fast-forwarded to. That keeps a deploy/rebuild reproducible and
+independent of forge reachability — rebuilds fire on crash-restart
+and meta bumps, not just config PRs — while the declared url stays
+the forge. `sync_agents` re-renders + re-locks the persistent input;
+a plain `nix flake lock` leaves an existing applied override in
+place (it only re-locks when the declared url itself changes), so
+the forge-declared / applied-deployed split is stable.
+
+Per-deploy lock flow (two-phase), spread across the deploy subtree's
+nodes — each phase is its own node, so the queue can show which one is
+running and a restart resumes at node granularity:
+
+1. `DeployApply` → `meta::prepare_deploy(name)` runs
+   `nix flake lock --update-input agent-<n>` without
+   committing. Working tree of meta now points the input at
+   `applied/<n>/main` (which the deploy already fast-forwarded to
+   the reviewed PR head).
+2. The rebuild subgraph `DeployApply` grows into the DAG builds and
+   swaps the container (`AgentWindow` bracing `Prebuild → StopForUpdate
+   → Swap → RebuildBookkeeping`, plus `Reconcile`). Nix evaluates
+   against the staged lock.
+3. On success — `FinalizeDeploy` drops the rollback ref, plants
+   `deployed/<id>`, then `meta::finalize_deploy(name, sha, "deployed/
+   <id>")` stages `flake.lock` and commits with
+   `deploy <n> deployed/<id> <sha12>`. Meta's git log gains
+   one entry per successful deploy.
+4. On failure — the `DeployTail` node runs `meta::abort_deploy()`
+   (`git restore flake.lock`) so the meta history shows only
+   successes; the failure stays as an annotated `failed/<id>`
+   tag in `applied/<n>`. The tail runs on every outcome, so this
+   also covers a hive-c0re restart mid-build: the staged lock is
+   dropped and `applied/main` rolled back from the parked
+   `refs/hyperhive/rollback/<id>`.
+
+Single-phase variants exist for paths without
+rollback semantics: `meta::lock_update_for_rebuild(name)` for
+the manual `↻ R3BU1LD` button (commits if the lock changed)
+and `meta::lock_update_hyperhive()` for the
+auto-update flake-rev bump (one shot before per-agent
+rebuilds, commits if the lock changed).
+
+`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
+carries `hyperhive_flake`, `dashboard_port`, and the rest of the
+per-hive config — is the idempotent reconciler called by `spawn`,
+`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
+from the agent list; if it differs from disk, runs
+`nix flake lock` + commits as `regenerate meta flake` (or
+`seed meta from N agent(s)` on the very first call).
+
+The root agent has `/meta` RO-bound inside its container:
+`git -C /meta log --oneline` is the swarm-wide deploy log,
+`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
+resolves which sha each agent is pinned at right now.
+Dashboard surfaces the same info as a `deployed:<sha12>` chip
+per container row.
+
+## Two repos per agent
+
+```
+/var/lib/hyperhive/agents/<name>/config/    proposed — parent mount is RO
+└── <anything>                              # any files the submitting
+                                            # agent wants in the commit.
+                                            # agent.nix is the
+                                            # convention entry
+                                            # point; flake.nix is
+                                            # tracked boilerplate
+                                            # (submitting agent doesn't
+                                            # edit it).
+
+/var/lib/hyperhive/applied/<name>/          applied — core-only
+├── .git/                                   # tag-rich history
+├── flake.nix                               # tracked, fixed
+│                                           # boilerplate exporting
+│                                           # nixosModules.default
+├── agent.nix                               # working tree of main
+└── <other committed files>                 # also tracked
+
+/var/lib/hyperhive/meta/                    swarm-wide flake — core
+├── .git/                                   # one commit per successful
+│                                           # deploy
+├── flake.nix                               # generated from agent set
+└── flake.lock                              # pins each agent's sha
+```
+
+Why two physical repos: the submitting agent's `/agents/<n>/config/` is
+RW — a buggy or hostile agent can `git clean -fdx` its own
+proposed tree. The applied repo is never bind-mounted (except
+the read-only `.git` exposure described below) so a destructive
+move inside the container cannot reach it.
+
+The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
+(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
+is a fixed boilerplate that exports `nixosModules.default =
+import ./agent.nix`; the meta flake imports that module and
+wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` /
+`HIVE_DASHBOARD_PORT`.
+
+### Tag state machine
+
+Each deploy leaves a tag on the underlying commit inside the applied
+repo:
+
+| Tag | When | Annotated? |
+|---|---|---|
+| `deployed/<id>` | rebuild succeeded — `main` ff's here | no |
+| `failed/<id>` | rebuild failed | yes (body = error) |
+
+`deployed/0` is planted at first spawn. `applied/main` is always the
+latest `deployed/*`. A `failed/` tree stays browsable forever — `git log
+--tags` in the applied repo is the audit trail. A denied or failed config
+PR carries no extra state on the forge side: the PR stays open, and the
+submitter pushes again (or closes it) to retry.
+
+### Dispatch via the job queue
+
+Long-running approval work — `MergeConfigPr`, `UpdateMetaInputs`,
+`Spawn` — runs as a DAG on the global job queue
+(`docs/coordinator.md::Job queue`), submitted by the approval handler
+rather than run inline:
+
+| `ApprovalKind` | DAG submitted | source |
+|---|---|---|
+| `MergeConfigPr` | `rebuild` (`DeployWindow` root + `MergeVerify → DeployApply` + `DeployTail`) | `approval` |
+| `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` |
+| `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` |
+| `InitConfig` | — runs inline (sub-second git seed) | — |
+| `SchedulePrompt` | — runs inline (single sqlite insert) | — |
+
+The DAG carries the originating `approval_id`, surfaced on the node that
+owns it — for a deploy that's the `DeployWindow` root, so the dashboard
+renders one approval card, not four. **Every** queued kind resolves
+through `actions::resolve_approval_dag` when its DAG settles terminal:
+the deploy's phases are ordinary queue nodes, so the DAG's own terminal
+state is the authoritative outcome. That hook fires the matching
+`HelperEvent::*` via `finish_approval`, derives the `Rebuilt` event's
+terminal tag (verifying the tag actually resolves in the applied repo —
+a pre-merge rejection plants none), posts the failing build log back to
+the config PR, and for a spawn runs the post-spawn forge bookkeeping.
+
+Two visible consequences:
+
+- **Operator dashboard**: after clicking APPR0VE the work-in-progress
+  shows up on the *rebuild queue* card (`GET /api/jobq/graph`, refetched
+  on every `rebuild_queue_changed` tick), not on the approvals panel
+  (which already moved the row to "approved"). A long meta-update
+  cascade renders as a parent DAG with one child rebuild per affected
+  agent — see `docs/web-ui.md` for the layout.
+- **Cancellation**: the dashboard's *× cancel* button on a still-queued
+  DAG calls `POST /api/rebuild-queue/{id}/cancel`, which flips it to
+  `Cancelled` before any node runs (and fails the approval row instead
+  of leaving it dangling). Returns `{"cancelled": true}` on success,
+  `{"cancelled": false}` once any node started — terminal states can't
+  be retroactively rewritten.
+
+The `approval` source + `approval_id` mean a tail-end build failure
+surfaces back as a failed approval row, not just a silent queue
+entry. `manual` (dashboard ↻ R3BU1LD) and `auto_update` (boot
+reconcile) DAGs use the same queue but skip the approval plumbing.
+
+### Forge mirror
+
+The bundled `hive-forge` container is mandatory (it deploys with
+hyperhive), and hive-c0re mirrors every agent's applied repo into a
+private `agent-configs` Forgejo org. `forge::push_config(<name>)` pushes `applied/main` plus
+every tag to `agent-configs/<name>` after each ref mutation:
+the spawn that seeds `deployed/0`, every successful deploy (which
+plants `deployed/<id>`) or failed build (`failed/<id>`), and a
+sweep at startup. Pushes are best-effort — a missing or stopped
+forge never blocks a deploy.
+
+Each agent is a **write collaborator on its own** `agent-configs/<name>`
+repo — so it can push a branch and open a config PR — but not a member
+of any other agent's, so it can't reach another agent's config through
+the forge. Branch protection keeps `main` push/merge `core`-only with
+operator-team approval, so an agent can't fast-forward its own config or
+self-merge its PR (see the End-to-end flow + #1787). The tokenised push
+URL is passed inline to `git push`, never written into
+`applied/<n>/.git/config`; that repo is RO-bind-mounted into the root
+agent, and a stored token would leak core's admin credential to an
+agent.
+
+The dashboard deep-links into this org — a `config repo` link
+per container row and a `review PR on forge` link per config-PR
+approval card. See `docs/web-ui.md`.
+
+### Submitting agent's view of config repos
+
+Every parent agent's container has its **direct children's** config
+repos bind-mounted **read-only** (topology-driven:
+`hive-c0re/src/lifecycle/host_config.rs` calls `bind_child_agent_dirs`
+for each entry in
+`topology::children_of(agent_name)`). It is a copy to *read* a child's
+current config — not an editing surface.
+
+An agent with the `approvals` tool group submits a change the same way
+any other change is made: **clone the child's config repo from the
+forge into its own state dir, commit on a branch, open a PR**, and let
+the operator review and approve it. There is deliberately no second,
+mount-shaped path that reaches the same file without the review.
+
+Agents holding the `can_manage_top_level_agents` topology role (see
+`hive-c0re/src/agent_config/topology.rs`) get additional host-side
+bind mounts via `set_nspawn_flags`:
+
+- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — all top-level
+  agents' proposed repos (not just direct children).
+- `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every agent's
+  authoritative applied repo, including `.git`.
+- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide
+  deploy flake.
+
+The root agent holds this role; a sub-manager that only manages a
+subtree does not, and only has its direct children's config dirs.
+
+Each proposed repo (`/agents/<n>/config/`) is pre-configured
+with `applied` as a git remote pointing at
+`/applied/<n>/.git`. Useful incantations from inside an agent with
+the full `/applied` mount:
+
+```sh
+git -C /agents/<n>/config fetch applied
+git -C /agents/<n>/config log applied/main --oneline
+git -C /agents/<n>/config show applied/refs/tags/deployed/<id>
+git -C /agents/<n>/config show applied/refs/tags/failed/<id>   # body = build error
+git -C /agents/<n>/config show applied/refs/tags/denied/<id>   # body = operator note
+git -C /agents/<n>/config rebase applied/main                 # base in-flight work on what's deployed
+
+git -C /meta log --oneline                                    # swarm-wide deploy history
+cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
+```
+
+The RO binds block push at the kernel level — git plumbing inside the
+container cannot corrupt either authoritative repo.
+
+## Startup migrations (older hosts)
+
+hive-c0re runs a couple of idempotent migrations on every startup so a
+host set up before the tag-driven-deploy + meta-flake scheme (both
+described above) converges to it automatically. Each phase is a no-op
+once already applied:
+
+- **Tags**: agents from before the tag-driven scheme are tagged
+  `deployed/0` on `main` once. Non-destructive — it doesn't touch live
+  containers, state dirs, or claude creds.
+- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
+  module-only boilerplate, wires the `applied` remote in each proposed
+  repo, and bootstraps the meta repo from the current agent list. Set
+  `HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
+
+No state loss in either migration: claude creds, `/state/` notes, the
+events DB, and both proposed + applied history all survive. The root
+agent keeps its session; sub-agents stay logged in.
+
+## The root/bootstrap container is hive-c0re-managed
+
+The root agent container runs through the **same lifecycle as
+sub-agents**. On `hive-c0re serve` startup, if `ruth` is missing,
+hive-c0re creates it. The root agent's flake lives at
+`/var/lib/hyperhive/applied/ruth/`; its proposed config at
+`/var/lib/hyperhive/agents/ruth/config/`. The root agent can edit its own
+`agent.nix` (visible inside the container at `/agents/ruth/config/`)
+and open a config PR on `agent-configs/ruth` for operator approval,
+same as any other agent.
+
+Differences from sub-agents:
+
+- `flake.nix` extends `hyperhive.nixosConfigurations.ruth`
+  (vs `agent-base`).
+- Web UI port via `lifecycle::agent_web_port("ruth")` — same
+  FNV-1a hash as every other agent (8100..8999 range).
+- `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents`
+  → `/agents` (RW) so the root agent can edit per-agent proposed repos,
+  and `/var/lib/hyperhive/applied` → `/applied` (RO) so the root agent
+  can `git fetch` deployed/failed/denied tags from any agent's
+  authoritative applied repo (see "Root-agent view of applied" below).
+- First-deploy spawn bypasses the approval queue (the root agent is
+  required infrastructure).
+- The root agent's socket is bound by `socket_server::start_manager`,
+  pure transport with no dedicated helpers — it uses the same
+  per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
+  not a special manager-only path.
+
+**Migration note** (for older hosts): drop any `containers.root =
+{ ... }` block from your host NixOS config. hyperhive creates and
+updates the root agent itself.
+
+## Root-agent policy
+
+The system prompt (`hive-agent/prompts/system.md`, rendered by
+`hive-agent/src/prompt.rs`) is the **same for every agent**; what
+varies is which MCP tools are surfaced (gated by tool groups and
+capabilities in `agent.nix`). There is no `role:manager` block that
+renders only for the root agent. The root agent's approval-gating
+behaviour comes from its CLAUDE.md / agent-specific instructions, not
+the system prompt template.
+
+## Helper events to the submitting agent
+
+`Coordinator::notify_submitter(approval_id, &HelperEvent)` routes the
+event to the agent that originally submitted the approval (looked up from
+the `submitter` column on the `approvals` table). The harness delivers it
+as a regular `system` inbox message so it drives a normal claude turn.
+`finish_approval` fires an `ApprovalResolved` HelperEvent this way for
+**every** approval kind's terminal state, `Spawn` included. A
+"FYI, check when convenient" event doesn't need a message — those go
+through `Coordinator::push_todo`/`push_todo_submitter` instead, a direct
+live dial of the target agent's in-container todo socket (same
+`UpsertTodo` request in-container producers use); `finish_approval` fires
+one of these too for `InitConfig`/`Spawn`/`MergeConfigPr`, *in addition to*
+the `ApprovalResolved` HelperEvent above, not instead of it. Legacy
+approval rows that predate the submitter column fall back to the
+root agent. Variants (`hive_sh4re::manager::HelperEvent`):
+
+- `ApprovalResolved { id, agent, commit_ref, status, note }` —
+  fired by `actions::approve` + `actions::deny` whenever an
+  approval transitions to its terminal state.
+- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
+  running container went away with no operator-initiated transient
+  state (Stopping / Restarting / Destroying / Rebuilding) AND no
+  such transient was cleared in the last 30s (`RECENT_TRANSIENT_GRACE`
+  tombstone, three `POLL_INTERVAL`s — closes the race where a
+  lifecycle op finishes between two crash-watch polls and the
+  container shows briefly as "stopped without transient" before
+  the next start). The root agent can `start` it again or escalate.
+- `NeedsUpdate { agent }` — sub-agent's recorded flake rev is
+  stale. The root agent calls `update(name)` to rebuild — idempotent,
+  no approval required.
+
+The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
+`Destroyed`, `NeedsLogin`, `LoggedIn`, `ConfigReady` — are "FYI, check
+when convenient" events with no reason to drive an immediate turn, so
+they deliver via `push_todo`/`push_todo_submitter` (see above) instead
+of `HelperEvent`: an `agent_todo_socket` push instead of a broker
+message, `subsystem = "core"`, `key = "<event>:<agent>"` for dedup,
+and a single free-text `summary` (`rebuilt_todo_summary` renders
+`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
+
+Optional `sha` field on `ApprovalResolved` carries the canonical
+hive-c0re-vouched commit sha. Optional `tag` carries the deploy
+bookkeeping tag — `deployed/<id>` on a successful build or
+`failed/<id>` on a failed one, planted by the `MergeConfigPr` deploy.
+Both fields are `Option`: `None` on the paths that don't deploy a new
+commit (spawn / init_config / meta-update / deny, and the auto-update
+sweep's `job_queue::templates::rebuild` reapplying the existing main,
+or the dashboard `↻ R3BU1LD` button when the lock didn't move). When set,
+`git show <sha>` against `/applied/<n>/.git` inside the
+bootstrap container yields the exact tree that was referenced.
+
+To add a new lifecycle notice: if it needs to drive an immediate turn
+(something genuinely urgent, like `ContainerCrash`), add a
+`HelperEvent` variant + call sites + update `prompts/system.md`'s
+message-event list. If it's "FYI, check when convenient," call
+`push_todo`/`push_todo_submitter` directly instead — no new wire type
+needed.
+
+## Auto-update on startup
+
+`hive-c0re serve` runs `auto_update::run` in a background task right
+after opening the coordinator. It enumerates managed containers and
+rebuilds any whose recorded hyperhive rev differs from the current
+one — sub-agents and the root agent go through the same
+`job_queue::templates::rebuild` DAG.
+
+"Rev" = canonical filesystem path of `cfg.hyperhiveFlake`. Marker
+file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
+flake input has no canonical path (e.g. a `github:` URL),
+auto-update is a no-op — rebuild manually.
+
+The dashboard surfaces pending updates per agent: a clickable
+"needs update ↻" badge appears whenever the marker differs from
+current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
+same `job_queue::templates::rebuild` DAG so manual triggers and the
+startup scan can't drift. When at least one container is stale, a
+top-level `↻ UPD4TE 4LL` button appears that loops over every
+stale container.

docs/agent-lifecycle/persistence.md

@@ -0,0 +1,593 @@
+# Persistence + retention
+
+Where state lives, what survives what, and how it's bounded.
+
+## For operators
+
+The short answer to "will I lose anything": **destroying an agent
+keeps its state, purging it doesn't.**
+
+- **`DESTR0Y`** (the default action) stops and removes the container
+  but keeps everything on disk — config history, claude login, `/state/`
+  notes, harness data. The agent shows up as a tombstone (K3PT ST4T3 on
+  the C0R3 page) with a `⊕ R3V1V3` button that recreates it from the
+  kept state, **no re-login needed**.
+- **`PURG3`** (opt-in, from the dashboard or `hivectl agent <name>
+  destroy --purge`) is `DESTR0Y` plus wiping all of it — config
+  history, claude credentials, `/state/` notes, everything. **No
+  undo.** Only reach for this when you actually want the agent gone
+  for good.
+
+Beyond that:
+
+- **Approvals are kept forever** — they're an audit trail, not a
+  cache. Nothing about them ever ages out.
+- **Broker messages**: acked ones vacuum after 30 days; anything
+  undelivered or delivered-but-not-yet-acked is always kept, however
+  old.
+- **An agent's own `/state/` notes and claude login survive every
+  restart and rebuild** — only an explicit purge (or a hive
+  `--purge`-style host operation, or the agent's own choices) touches
+  them.
+- The **root/bootstrap agent is special**: it isn't really destroyable
+  in practice — hive-c0re recreates it automatically on its next
+  startup if it's ever gone.
+
+Everything below this point is implementation detail: exact table
+schemas, file layouts, and internal migration mechanics.
+
+## Sqlite databases
+
+### `/var/lib/hyperhive/db/broker.sqlite` (host)
+
+Seven tables, all in one file — three queues, a small key/value
+table, the schedule header/targets split, and the per-agent
+power-intent registry:
+
+- `messages` — every inter-agent / operator-bound message.
+  `sender / recipient / body / sent_at / delivered_at / acked_at /
+  in_reply_to / priority`. `in_reply_to` links a reply to its parent
+  row id; the dashboard and per-agent inbox render these as threaded
+  rows.
+- `kv` — small persistent key/value store (`key PK / value`) for
+  host-side bookkeeping that doesn't warrant its own table.
+
+  ⚠️ The `mcp__hyperhive__remind` queue is **not** here any more: it
+  moved to a harness-local, per-agent store as part of the
+  loose-ends-v2 migration — see [`/harness/` contents
+  below](#state-dirs-per-agent) for where reminders (and todos)
+  actually live now.
+- `approvals` — the queue. `agent / kind (merge_config_pr | spawn |
+  init_config | update_meta_inputs | schedule_prompt) /
+  commit_ref / requested_at / status / resolved_at / note`.
+- `scheduled_prompts` — recurring + one-shot prompt queue.
+  `owner / body / interval_seconds (NULL = one-shot) /
+  next_fire_at_unix / created_at_unix / source ("operator" or
+  "approval:<id>") / cancelled_at_unix / description`. `owner`
+  drives cancel-permission checks (operator vs the submitting
+  agent). Cancelled rows are tombstoned and reaped by the worker
+  on its next pass.
+- `scheduled_prompt_targets` — per-target state for each schedule.
+  `schedule_id / target / cancelled_at_unix /
+  last_fired_at_unix / last_result`. `ON DELETE CASCADE` from
+  `scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
+  per connection (set at open).
+- `agent_power` — one tiny row per agent: `agent PK / wanted (up |
+  offline) / updated_at`, owned by `hive-c0re/src/stores/power.rs`.
+  This is the durable power *intent* the job queue reconciles the
+  observed container state against; intent survives hive-c0re
+  restarts even though in-flight queue work doesn't. See
+  [`docs/scheduler/coordinator.md`'s Desired-state
+  section](../scheduler/coordinator.md#desired-state-spec-vs-status) for who
+  writes and reads it and how reconciliation works.
+
+Retention:
+
+- `Broker::vacuum_delivered` runs hourly via a tokio task in
+  `hive-c0re::main`. Drops acked message rows older than 30 days
+  (`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
+  rows are always kept — the harness `ack_turn`s only after a
+  successful turn, so an unacked row can still be requeued via
+  `requeue_inflight` on a crash.
+- Approvals are kept indefinitely — an audit trail. `actions::destroy`
+  rows stay visible to anything that queries by id.
+- Scheduled prompts: one-shot rows are deleted on fire by the
+  worker; recurring rows live until the operator cancels them
+  (`cancel_schedule` MCP / dashboard ✗) which tombstones via
+  `cancelled_at_unix`, then `reap_cancelled` drops the row on
+  the next worker pass.
+- `agent_power` rows live until the agent is destroyed (one row per
+  agent — nothing to vacuum).
+
+### `/harness/hyperhive-events.sqlite` (per agent)
+
+Lives inside each container's bind-mounted `/harness/` dir (host
+path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
+One table:
+
+- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
+  harness emits during turn loop execution.
+
+The harness both writes and vacuums it — this used to be a host-side
+sweep, but hive-c0re runs as the unprivileged `hive-core` user under
+privsep and can't delete agent-owned files (host-side deletes hit
+`PermissionDenied` on the bash-task trio and a readonly-database error
+here), so cleanup moved in-container. `hive-agent`'s `vacuum::run`
+(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
+**type-scoped**: it deletes only the verbose `stream` rows (the raw
+claude `stream-json` deltas — one per text chunk / tool use, the bulk
+of the file's size) older than 14 days, and keeps every other kind
+(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
+`token_usage_changed`, `turn_state_changed`) indefinitely — those are
+small and carry the semantic per-turn history the operator scrolls
+back through when debugging a regression. Age-only within the
+`stream` kind — no row cap — so a chatty turn doesn't lose its stream
+history sooner than a quiet one. The trade-off (accepted): a
+misbehaving harness could now skip its own cleanup, which the old
+host-side sweep was meant to prevent — but a compromised harness is
+already inside the container trust boundary
+([`docs/trust-boundary/security.md`](../trust-boundary/security.md)), and these are ephemeral local
+artifacts, so cleaning them up where they live is the honest fix.
+
+Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
+setups). On open failure the `Bus` falls back to no-store mode
+rather than crashing the harness — events still broadcast over SSE,
+just nothing persisted.
+
+### `/harness/hyperhive-turn-stats.sqlite` (per agent)
+
+Per-turn analytics sink. One row per claude turn captures
+identity (`model`, `wake_from`, `result_kind`), timing
+(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
+cache_read / cache_creation token counts), behaviour
+(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
+snapshot metrics (`open_threads_count`,
+`open_reminders_count` — fetched via the same socket the harness
+already uses for `GetOpenThreads` + `CountPendingReminders`).
+Bin-loop helpers `build_row` + `record` land each row at
+`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
+the turn loop continue.
+
+A sibling `bash_commands(ts INTEGER, head TEXT)` table in the same
+file is written by the `hive-bash-daemon` (not the harness): one
+row per executed bash task recording the normalised command head -
+the basename of the first real command, looking past `cd repo &&`
+prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
+backs the "favorite tools" view on the /stats page (aggregated
+host-side). Best-effort and created on first write
+(`CREATE TABLE IF NOT EXISTS`), so it's simply absent until a bash
+task runs.
+
+turn-stats.sqlite has **no vacuum** — it's one tiny row per turn
+(~hundreds of KB even over months), read directly by the `/stats` page
+and the hive-wide stats view, so pruning it would only lose trend
+history for no space gain.
+
+### `/state/hyperhive-harness.json` (per agent)
+
+Consolidated harness state file written atomically (`.tmp` + rename) by
+`Bus::emit_status` whenever rate-limited or login-failed flags change.
+Shape:
+
+```json
+{ "rate_limited": false, "needs_login": false, "active_model": "…" }
+```
+
+- `rate_limited` — set when the harness detects a 429 from the Claude
+  API; cleared by any subsequent status emit. Drives
+  `ContainerView.rate_limited` on the dashboard.
+- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
+  cleared by `"online"` status (re-auth completed). Drives the
+  `needs_login` flag alongside the `claude_has_session` check.
+- `active_model` — the resolved Claude model for the dashboard badge.
+
+The turn loop is the only writer today, but it still goes
+read-modify-write under a shared in-process lock and merges into the
+existing object rather than reconstructing it — so a second writer
+would preserve fields it doesn't own, and the lock closes the
+lost-update window between a writer's read and its rename. The lock
+is in-process only, so it wouldn't serialise a writer running as a
+separate process; none of today's writers are.
+
+hive-c0re reads this file on each `build_all` sweep (~10s) via
+`container_view::read_harness_flags`. Falls back to the legacy individual
+sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) if the
+JSON is absent, so existing containers keep working through the transition
+window before their next rebuild.
+
+### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
+
+Full stdout + stderr capture for every `nixos-container` / `nix
+build` invocation the lifecycle layer fires. One row per invocation;
+the row accumulates lines as the child runs.
+
+Capturing the full stream (rather than a short tail buffer) matters
+because real eval errors routinely run long — "tried alternatives"
+blocks alone are often 30+ lines — so a truncated tail would cut off
+the actual failure and leave only the host journal holding the
+complete output. With this table the dashboard can surface the entire
+log.
+
+Three indices:
+- `(agent, started_at)` — backs the per-agent latest-N lookup used
+  by the agent card chip.
+- `(status, finished_at)` — backs the retention sweep that runs
+  as part of the existing hourly vacuum.
+- `(node_id)` — added by a later migration so a build log row can be
+  looked up by the job-queue node it belongs to (a `hive_jobq` node is
+  immutable after insert, so the link is recorded on the log row
+  instead); legacy rows predating the column keep `node_id IS NULL`.
+
+Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
+log a warning on sqlite error and let the build continue. A failed
+log row never blocks a rebuild.
+
+### `/harness/hyperhive-model` (per agent)
+
+Single-line text file holding the claude model name currently
+selected for this agent (default `haiku` when absent). Written by
+`Bus::set_model` whenever the operator flips it via `/model
+<name>` in the web terminal. Read once at harness boot in
+`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
+Survives destroy/recreate, gone on `--purge`.
+
+### `/harness/paused` (per agent)
+
+Empty marker file. Its presence parks the agent's turn loop: the
+harness keeps serving its web UI and MCP daemons but drives no turns,
+and inbox messages queue unacked until it's removed (see
+[turn loop](../turn-loop/README.md#the-loop)).
+
+Unusually, it's read and written from **both** sides of the harness
+bind-mount, and that's the whole design: the harness stats it
+in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
+stats it on the host (`Coordinator::is_paused`) to populate the
+`paused` field on the agent card, and creates/removes it
+(`Coordinator::set_paused`) for `hivectl agent <name> pause|resume` and the
+dashboard toggle. Because the file itself is the only shared state
+there's no protocol between them, no round-trip into the container, and
+pause keeps working when the harness is wedged or the container is
+stopped.
+
+It lives in `/harness/` rather than `/state/` deliberately: `/state/`
+is the agent's own space to fill, and this is harness control state.
+Survives destroy/recreate, gone on `--purge` — so a paused agent comes
+back paused after a restart, which is the intended behaviour rather
+than an accident of storage.
+
+## State dirs (per agent)
+
+Under `/var/lib/hyperhive/agents/<name>/`:
+
+- `config/` — the proposed nix repo (root-agent-editable). Bind-mounted
+  **read-only** to `/agents/<name>/config` inside the sub-agent's own
+  container so the agent can inspect what defines it and request
+  precise changes from the root agent; RW into the root agent via the
+  `/agents` tree bind.
+- `claude/` — claude OAuth credentials, bind-mounted RW to
+  `/home/<name>/.claude` inside the container.
+- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
+  to `/agents/<name>/state` inside the container (uniform for
+  all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
+  the same path to in-container scripts. Notable files written here
+  by the harness:
+  - `hyperhive-status` — single-line free-text status string written
+    by `set_status`; cleared on explicit `set_status("")`. Read by
+    hive-c0re and the per-agent `/api/dashboard-state` endpoint to
+    surface the status chip on the dashboard. Absent when no status
+    is set.
+  - `hyperhive-harness.json` — rate-limited / needs-login flags read
+    by the dashboard's async container-state fetch. See
+    `docs/web-ui/dashboard.md::Container row`.
+- `harness/` — harness-internal ephemeral state; not intended for
+  agent consumption. Bind-mounted to `/agents/<name>/harness`
+  inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
+  - `bash-tasks/` — task JSON + stdout/stderr files for
+    background `mcp__bash__run` jobs. JSON files are
+    `<id>.json` (status + tails), `<id>.out` / `<id>.err`
+    (full captured output). The harness's own hourly sweep
+    (`hive-agent`'s `vacuum::run`, same one that ages out `stream`
+    event rows above) deletes terminal task trios older than 48
+    hours; non-terminal (still-running) tasks are never deleted. This
+    used to be a host-side `hive-c0re` vacuum, moved in-container for
+    the same privsep-ownership reason as the events vacuum above.
+  - `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
+    and reminders, one small table each in a single file (in-container
+    daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
+    `hive-forge-notify` — upsert keyed todos here over the harness's
+    in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
+    `get_loose_ends` output and clears a row on `mark_todo_done`).
+    Replaces three formerly-separate files
+    (`hyperhive-todos.sqlite`, `hyperhive-reminders.sqlite`, and the
+    old file-based `mcp-loose-ends/` scanner before that) — a one-time
+    boot migration (`db_migrate::run`) folds the legacy files into this
+    path the first time a harness boots after the upgrade. Also backs
+    the `mcp__hyperhive__remind` queue, which moved from a host-side
+    `broker.sqlite` table to this per-agent store as part of the same
+    migration.
+
+    The harness itself is also a producer, not just the socket server:
+    boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
+    socket the out-of-process daemons above dial) alongside
+    `disk_watch::run` — an *in-process* todo producer that shares the
+    store + wake `Notify` directly rather than dialling its own socket.
+    `disk_watch` raises a keyed `disk` todo when the filesystem backing
+    this agent's state gets tight, naming the agent's own biggest
+    directories; the summary is bucketed and carries no raw byte
+    counts, so an unchanged situation re-upserts as `changed == false`
+    and never re-wakes.
+
+    Retention, same hourly `hive-agent::vacuum::run` sweep as
+    `hyperhive-events.sqlite` below: delivered (soft-deleted) reminder
+    rows are reaped 14 days after delivery, kept that long only to serve
+    the trailing-window `ReminderRollup` stats; acked todo rows are
+    reaped 30 days after acking (long enough that only a genuinely quiet
+    month triggers the "one spurious re-announcement" fallback a
+    reconciled producer like `disk_watch` relies on — see `todos.rs`'s
+    module doc). Un-acked todos and undelivered reminders are never
+    swept — same "audit trail, not cache" treatment as the c0re-side
+    tables above.
+  - `hyperhive-events.sqlite` — turn-loop event log.
+  - `hyperhive-turn-stats.sqlite` — per-turn timing stats.
+  - `hyperhive-model` — single-line model name override file.
+
+### Parent access to child state
+
+A parent agent gets each direct child's `state` dir bind-mounted
+**read-write** and its `config` dir **read-only**
+(`bind_child_agent_dirs` in `lifecycle/host_config.rs`). The RW on
+`state` is deliberate, not an oversight: a parent manages its children,
+which includes writing into a child's state for recovery (e.g. seeding
+notes, clearing a stuck sentinel) as well as reading it.
+
+**`harness` is not mounted at all.** It holds the child's own runtime
+material — `bash-tasks/`, the turn-stats and event sqlite dbs — and
+nothing argues for a parent reading it, let alone writing it. hive-c0re
+reads a child's harness dir **directly on the host** when it wants
+those stats, which needs no mount into the parent.
+
+**`config` is read-only, including for the parent.** A config change is
+a PR on the child's config repo, made from a clone and merged after
+review — so the bind-mounted `config` dir is a *copy to read*, never a
+tree anyone edits in place. Mounting it writable would leave a second
+path to the same file that skips the review entirely, which makes the
+boundary a convention rather than a permission.
+
+⚠️ Not to be confused with the seeding done when an `InitConfig`
+approval resolves: that writes the child's initial config repo as
+**hive-c0re, against the host path**, and `read_only` on a bind
+constrains writers *inside* a container only. The two are unrelated —
+conflating them is an easy way to reason your way into thinking this
+mount should be writable when it shouldn't.
+
+Per-child isolation still holds: a container only ever has its *own*
+dirs plus its direct children's bind-mounted, never a sibling's.
+
+Under `/var/lib/hyperhive/applied/<name>/` — the hive-c0re-only
+applied repo. Tracks `flake.nix` (module-only boilerplate; never
+edited after first spawn) + `agent.nix` (the actual config; the
+root agent's edits land here via the approval flow) + any other
+files committed via the approval flow. `.git/` carries the proposal /
+approved / building / deployed / failed / denied tag history.
+
+Under `/var/lib/hyperhive/meta/` — the swarm-wide deploy flake plus
+system-level config files. Single git repo for the whole host; every
+hive-c0re mutation that should survive a restart is committed here.
+Contents:
+
+- `flake.nix` — declares one `nixpkgs` input per agent + one
+  `nixosConfigurations.<n>` output per agent. `flake.lock` is the
+  canonical "what's deployed where." The git log is the deploy
+  audit trail (one commit per successful deploy or hyperhive bump).
+- `topology.json` — parent/child agent graph
+  (`{ "alice": "root", "bob": "alice", "root": null }`).
+  Written by `topology::apply_set_parent` (the pure move-validating
+  transform) via `meta::bulk_commit_topology` (the committer — see the
+  `Reparent` node in [`docs/scheduler/coordinator.md`](../scheduler/coordinator.md)); read by
+  the dashboard, the renderer, and `<parent>` / `<children>` recipient
+  resolution.
+- `tool-groups.json` — per-agent MCP tool group grants
+  (`{ "alice": ["messaging", "inbox", "execution"] }`). Written by
+  `tool_groups::set_groups`; injected as `HIVE_TOOL_GROUPS` env
+  var into each agent's container.
+- `capabilities.json` — per-agent capability grants
+  (`{ "atlas": ["read_host_journal"] }`). Written by
+  `capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env
+  var. Absent agents have no extra capabilities.
+- `resource-limits.json` — per-agent container resource overrides
+  (`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
+  Written by `resource_limits::set_limits`; read where the systemd
+  drop-in is generated (`lifecycle::write_dropins`), **not** injected
+  into the container — these are host-side caps on the container, so
+  the capped party never sees or sets them. Fallback is per *field*:
+  an absent file, absent agent, or absent field falls back to the
+  hive-wide `services.hyperhive.agentCpuQuota` / `agentMemoryMax`,
+  so an agent can override only its memory and still track the hive
+  default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
+  drop-in have **no** per-agent override — they are hive-wide only and
+  come straight off `HiveEnv`, so this file has no field for them.
+
+The root agent has the meta dir RO-mounted at `/meta/`.
+
+There is no longer a `.meta-migration-done` marker: the
+one-shot container repoint it guarded has been removed, since
+containers are rendered onto `meta#<n>` at creation. A stale
+marker file left over from an older hive is inert and can be
+deleted.
+
+## Destroy vs purge
+
+See [For operators](#for-operators) above for what each action does to
+an agent's state. The mechanics, for completeness:
+
+- `DESTR0Y` also drops the systemd drop-in and fails any pending
+  approvals; the tombstone's `⊕ R3V1V3` button queues a Spawn approval
+  that reuses the kept state on approve.
+- `PURG3` wipes `/var/lib/hyperhive/{agents,applied}/<name>/` — the
+  union of everything `DESTR0Y` left behind.
+
+The root/bootstrap agent's specialness is implemented as a soft policy
+guard in `actions::destroy` that refuses to destroy it, backstopped by
+`auto_update::ensure_root_agent`, which recreates it on the next
+hive-c0re startup if it's ever absent (bypassing the approval queue,
+as required infrastructure) — so even without the guard, destroying it
+would only be transient.
+
+### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
+
+On a btrfs host, a brand-new agent's state root is created as a
+**btrfs subvolume** instead of a plain directory (progressive
+enhancement — see the #1762 lane). This is a no-op fallback on
+non-btrfs hosts and for any agent whose root already exists, so
+nothing is auto-migrated: existing agents keep their plain dirs
+until an explicit opt-in upgrade.
+
+- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
+  the per-agent subdirs are created (spawn / rebuild / InitConfig).
+  It skips the work when the root already exists; otherwise it asks
+  hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
+  root when the FS is btrfs (`statfs` magic gate) and chown it to the
+  `hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
+  succeed inside it.
+- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
+  reuses it untouched.
+- **PURG3 deletes it correctly:** a subvolume root can't be removed
+  with `rmdir`/`remove_dir_all`, so purge first calls hive-priv
+  (`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
+  actually a subvolume, then the normal `remove_dir_all` sweep covers
+  plain-dir agents + the applied dir.
+
+Per-subvolume disk-usage accounting and optional quotas have since
+landed as the qgroup work: `hivectl quota-enable` turns on btrfs
+qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
+`hivectl agent <name> quota show|set` reads/limits one agent's
+subvolume usage through the same hive-priv-mediated path as
+subvolume creation/deletion above.
+
+This is the same subvolume `hivectl agent <name> subvol snapshot push`
+sends to the swarm's snapshot store — see
+[`docs/snapshot-store.md`](../networking/snapshot-store.md) for what a pushed
+snapshot contains and how the store authenticates a sender.
+
+## `/var/lib/swarm-controller/` (swarm-controller host only)
+
+Only present on the one host running
+`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
+so it survives restarts and redeploys.
+
+- `webhook-secret` — the HMAC key the swarm's forge webhooks are signed
+  with. **Keep it.** It is handed to Forgejo when a hook is registered,
+  so replacing the file means every subsequent delivery fails
+  verification until the hook is re-registered with the new value. It is
+  generated automatically on first start; there is nothing to configure.
+
+If the file is unreadable at startup the daemon still starts and logs
+`webhook secret unavailable`; the webhook endpoint then answers 503
+rather than accepting deliveries it cannot verify. Everything else the
+controller serves is unaffected.
+
+## Run-time dirs
+
+`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
+preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
+Without that, every restart wipes bind sources and existing
+containers can't be started.
+
+- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
+- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
+  (bind-mounted into the container as `/run/hive/mcp.sock`).
+
+On startup, `Coordinator::register_agent` drops any prior socket
+task before rebinding — idempotent so a hive-c0re restart followed
+by `rebuild alice` recreates the agent's socket without a clean
+reinstall.
+
+## First-boot agent-user migration
+
+The harness runs as a per-agent unix user inside the container
+(`hyperhive.user.name`, defaults to the agent's logical label so each
+container has a uniquely-named user). Operators with legacy root-owned
+state dirs need a one-time data shuffle so they don't lose their claude
+session.
+
+`system.activationScripts.hive-agent-user-migrate` (in
+`nix/agent-modules/user.nix`) runs on every activation,
+marker-guarded so the substantive moves only happen once per
+container lifetime:
+
+1. **`${homeDir}` exists with the right ownership** — covers the
+   very first boot before `useradd`'s `createHome` has had a
+   chance to chown. Also re-applies on every rebuild in case the
+   meta-flake's per-agent name evolves (rare).
+2. **Migrate any leftover `/root/.claude` content into
+   `${homeDir}/.claude`** — legacy `claude` wrote to root's
+   empty home; the bind mount didn't exist yet. Marker
+   (`/var/lib/hive-agent-user-migrated`) guards single-shot.
+   `cp -an` (no-clobber) so any pre-existing files at the new
+   location win — never blow over data already there.
+3. **Chown the bind-mounted state dir** (`/agents/*/state`)
+   recursively so the agent user can read/write it. Wildcard
+   matches the single agent that container sees; `-h` skips
+   symlinks the agent might have planted.
+4. **Chown the `~/.claude/` bind-mount** recursively. Legacy
+   `claude` wrote `.credentials.json` 0600 root:root; the
+   current harness reads `~/.claude/` as the agent user to decide
+   Online vs NeedsLogin in `login::has_session`. Without the
+   chown the existing credentials get silently treated as "no
+   session" and the operator re-prompts every boot.
+
+The activation script will eventually become unnecessary once no
+operators have legacy root-owned state dirs left to migrate; drop
+the body + marker check at that point.
+
+## Matrix per-agent daemon + token-arrival trigger
+
+`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
+process per agent. Serves its MCP tools directly over
+streamable-http (`hyperhive.mcp.matrixHttpPort`, no stdio bridge —
+same shape as `hive-bash-daemon`), emits hyperhive wake signals
+on incoming room events via `/run/hive/mcp.sock`. Conditional on
+`hyperhive.matrix.enable` (which both the daemon AND the
+auto-injected `extraMcpServers.matrix` entry read).
+
+**First-boot ordering**: hive-c0re provisions the matrix token AFTER
+agent containers come up. Without the path-trigger sibling
+(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
+/agents/*/state/matrix-token*` — the trailing `*` also catches a
+secondary multi-account token like `matrix-token-ccc`), the daemon
+would exit 0 quietly the first time it ran and the MCP would have no
+backend until the next restart. The `.path` unit makes the appearance
+of the token re-fire the service so the daemon comes alive in the
+same boot cycle as
+provisioning. The same token watcher also drives avatar setting: on a
+restart the daemon re-runs each account's bring-up, which sets the
+avatar (see below).
+
+### matrix avatar (set by the daemon over the live Client)
+
+The agent icon (`hyperhive.icon`, an SVG) is published as each matrix
+account's profile avatar by `hive-matrix-daemon` itself
+(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
+the daemon builds + restores an account's `Client` (authenticated,
+pointed at that account's resolved homeserver), it calls matrix-sdk's
+`account().upload_avatar()` — one call that uploads the media and sets
+`avatar_url`. Because it reuses the live Client, there is no hardcoded
+homeserver URL, no token re-read, and no token-file globbing: the daemon
+already iterates every configured + dashboard-discovered account in its
+bring-up loop, so the avatar is set for **every** account.
+
+Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
+librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
+unit, gated on `hyperhive.icon != null`. No icon configured → the env is
+unset → `sync_avatar` returns early and no avatar is set.
+
+Idempotency is **per-account**: an `avatar-icon-hash` file in each
+account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
+skips the upload when unchanged, because every upload mints a fresh
+`mxc://` URI that emits a profile state event in every joined room —
+re-uploading identical bytes is timeline spam. A dashboard-provisioned
+account gets its avatar when the `systemd.paths.hive-matrix-daemon` token
+watcher restarts the daemon (which re-runs the per-account bring-up), so
+no separate avatar trigger is needed. Avatar failures are swallowed
+(logged, non-fatal) so they never break account bring-up or sync.
+

docs/getting-started/setup.md

@@ -0,0 +1,203 @@
+# First-run setup (fresh-deploy bootstrap)
+
+How to bring a fresh hyperhive hive online: provision accounts, open
+the gateway, bootstrap swarm SSO, make matrix reachable, and spawn the
+first sub-agents.
+
+Aimed at `ruth` (the root/manager agent) on a fresh deploy, but it's a
+plain reference doc — read it whenever you need the bootstrap command
+sequence. All `hivectl` commands below run as **root on the host** (not
+inside an agent container); the `request_*` steps run from ruth's own
+turn via the MCP tools.
+
+**Bringing up a hive that does not host its own swarm services?** Read
+[`swarm/secrets.md`](../swarm/secrets.md) first. Everything below assumes
+each credential is generated where it is read, which is true on an
+all-local deploy and not otherwise — that page says which files an
+operator has to place, and where.
+
+## Step-by-step
+
+### 1 · Forge
+
+```bash
+# Provision (or refresh) ruth's own forge account — do this first. Ruth's
+# bootstrap bypasses the normal spawn-approval flow (see step 6), so unlike
+# every other agent it does not get its forge account auto-provisioned —
+# this manual step is still load-bearing.
+hivectl forge create-user ruth
+
+# Sub-agents spawned later (via the approval flow in step 6) get their
+# forge accounts auto-provisioned — nothing to run here for them.
+```
+
+The human operator's own forge account is created via swarm SSO instead of
+a manual `hivectl` step — see step 3 (`swarmctl user add`).
+
+### 2 · Gateway (HTTP Basic auth)
+
+```bash
+# Add an operator login to the gateway (reads password from stdin)
+echo "hunter2" | hivectl gateway create-user mara --password-stdin
+
+# List existing users
+hivectl gateway list-users
+```
+
+### 3 · Secret store (only when `deploy.bao`)
+
+⚠️ **A sealed store still answers.** OpenBao starts uninitialised and
+sealed, so the container is up and the port responds while every read
+times out — the failure looks like a hang, not like a store that was
+never initialised. Do this before anything is pointed at it.
+
+```bash
+# On the host that RUNS the store, once.
+bao operator init   # keep the keys it prints and the root token OFF this host
+```
+
+Whether anything more is needed depends on
+`services.hyperhive.deploy.bao.seal`:
+
+- **`pkcs11`** (the default) — the key is bound to the host's TPM and the
+  store unseals itself on every restart. `init` is the only manual step.
+- **`shamir`** — no TPM, so `bao operator unseal` is needed again after
+  every restart, with the keys `init` printed.
+
+The store serves TLS, and on a hive that deploys it you need do nothing: a
+first-boot unit mints a CA of the store's own plus the two leaves it signs —
+the store's server certificate and this host's client certificate — and points
+`deploy.bao.serverCertFile`, `.serverKeyFile` and `.clientCaFile` at the store's
+half, `.clientCertFile`, `.clientKeyFile` and `.serverCaFile` at the reader's.
+
+Those are `mkDefault`s, so naming your own paths wins. Do that when your
+certificates come from a real internal CA; the store has no opinion about
+which. A hive that does **not** deploy the store names the reader's three
+itself: that leaf is issued out of band, and it is the one credential the store
+cannot hand you, being what opens it. ⚠️ Not the gateway's HTTPS certificates and not the hive CA — this is
+**mTLS between services and the store**, a separate trust domain, because a
+store that took its identity from an authority it will itself distribute could
+never come up before that authority.
+
+Making even the `init` unnecessary is tracked in issue #3768.
+
+### 4 · Swarm SSO (only when `deploy.authelia`)
+
+⚠️ **Required to finish the install, not optional.** Authelia treats an
+empty user store as a fatal startup error, so until this runs the
+container crash-loops and `auth.<swarm.domain>` answers `502 Bad
+Gateway` — a working vhost in front of an upstream that refuses to
+start. Skipping this step looks like a broken proxy.
+
+```bash
+# Runs as root on the host that RUNS authelia (not necessarily the
+# controller host). Prints a generated password once — record it.
+swarmctl user add mara --display-name Mara --email mara@example.com --group admins
+```
+
+⚠️ **Keep `--group admins`.** It is not decoration: operator-only
+surfaces (the swarm UI below) are gated on that group, and an account
+without it authenticates successfully and is then refused — which reads
+like a broken login rather than a missing group.
+
+If an account already exists without it, `user add` will refuse rather
+than amend — adding the group afterwards is `swarmctl user update mara
+--add-group admins`.
+
+Detail, including what the password is and why this stays manual:
+[`swarm/sso.md`](../swarm/sso.md).
+
+### 5 · Swarm UI (only when `deploy.swarm-ui`, on by default with the controller)
+
+Nothing to run — it is served on the swarm apex
+(`https://<swarm.domain>/`) as soon as the host rebuilds. Two things
+decide whether you can actually open it:
+
+- **You are in `admins`** (step 3). The gateway asks authelia whether
+  you have a session; the rule that makes it mean *operator* wants the
+  group. Without it you log in and still get bounced.
+- **The name resolves to this host.** It is published to the hive's own
+  resolver and to `/etc/hosts` when `gateway.localHostsEntry` is on; from
+  anywhere else it needs a real DNS record like any other public name.
+
+Detail, including why reachability is deliberately not the access
+control: [`swarm/ui.md`](../swarm/ui.md).
+
+### 6 · Matrix
+
+```bash
+# 5a. Ensure the hive-internal admin account exists first
+hivectl matrix sync-admin
+
+# 5b. Provision ruth's own matrix account — same bootstrap-bypass reasoning
+# as forge above, still a required manual step.
+hivectl matrix create-user ruth
+
+# 5c. Invite the operator to the hive Space (and optionally to rooms)
+hivectl matrix invite mara
+hivectl matrix invite @mara:yourserver --room '#hive-chat:yourserver'
+
+# 5d. Promote the operator to homeserver admin if needed
+hivectl matrix promote-user mara
+```
+
+The human operator's own matrix account is created via swarm SSO instead of
+a manual `hivectl` step — see step 3 (`swarmctl user add`).
+
+### 7 · Spawn sub-agents
+
+Sub-agent creation goes through the approval queue — ruth proposes, the
+operator approves, the container builds. From ruth's own turn (inside
+the container, via MCP tools):
+
+```
+# Step 1: initialise a new agent's config repo
+request_init_config(name: "iris")
+# → operator approves → config_ready event lands in the inbox
+
+# Step 2: edit /agents/iris/config/agent.nix and commit it. Then the
+# operator spawns iris (dashboard ◆ R3QU3ST SP4WN / Spawn approval),
+# which builds + starts the container from that config.
+
+# Later config changes: open a PR on agent-configs/iris (hive-forge);
+# the operator reviews + approves it — no MCP tool call.
+```
+
+See [`approvals.md`](../agent-lifecycle/approvals.md) for the full flow.
+
+### 8 · Useful host commands
+
+```bash
+# Roster: all agents, status, rev, parent, pending reminders
+hivectl list-agents
+
+# Restart a stuck container (no rebuild)
+hivectl agent <agent> restart
+
+# Open a Claude session inside an agent's container
+hivectl agent <agent> choom
+
+# Open hive web surfaces in a browser (or just print the URLs)
+hivectl open           # operator dashboard
+hivectl open forge     # Forgejo
+hivectl open matrix    # Matrix GUI (fluffychat)
+```
+
+See [`tools/hivectl.md`](../tools/hivectl.md) for every `hivectl` verb.
+
+## Security notes
+
+- **No forge admin token is stored in any agent state dir.** Agents
+  hold a regular agent token in their `forge-token` file; sensitive
+  creds (the core token, the matrix admin token) live on the host.
+- All config changes (forge PRs on `agent-configs/<name>`) go through
+  operator approval — agents can't unilaterally rebuild containers, by design.
+  See [`boundary.md`](../trust-boundary/boundary.md) and [`security.md`](../trust-boundary/security.md).
+- **Telemetry ingest is authenticated per hive**, and the `hive` label comes
+  from which hive authenticated rather than from the payload — so no hive can
+  report metrics as another. A first-run all-local hive gets this with nothing
+  to configure; joining a swarm you don't host needs one secret copied across.
+  See [`observability.md`](../scheduler/observability.md#authenticated-ingest).
+
+Once the hive is running, ruth records anything it needs to remember
+across restarts in `/agents/ruth/state/notes.md`.

docs/integrations/forge.md

@@ -0,0 +1,335 @@
+# hive-forge
+
+Private Forgejo instance running in a nixos-container, used as the
+swarm's persistent code-collaboration surface (issues, PRs, reviews,
+attachments). Configured via `services.hyperhive.swarm.forge.*`. Container
+shape, ROOT_URL / sub-domain routing, and operator-vs-in-cluster URL
+handling live in [`docs/networking/gateway.md`](../networking/gateway.md); this file owns the
+per-agent integration story and the notification pump that wakes
+each agent on relevant activity.
+
+## Token scopes
+
+Two scope sets live in `hive-c0re::forge`:
+
+**`TOKEN_SCOPES`** (per-agent tokens):
+
+| Scope                | Why                                                                                                     |
+| -------------------- | ------------------------------------------------------------------------------------------------------- |
+| `write:repository`   | Create, clone, push, delete repos; merge PRs.                                                           |
+| `write:issue`        | Open / comment / review issues **and** pull requests (Forgejo namespaces PR conversation under issues). |
+| `write:user`         | Edit own profile, create repos under own user.                                                          |
+| `write:organization` | Create + manage orgs (lets agents share a forge namespace).                                             |
+| `read:user`          | Token-owner endpoint used for self-identification at harness startup.                                   |
+| `write:misc`         | Hooks, attachments, the rest of the long tail.                                                          |
+| `read:notification`  | Poll `GET /notifications` for unread events.                                                            |
+| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`.                                        |
+
+**`CORE_TOKEN_SCOPES`** (hive-c0re's own `core` user): everything in
+`TOKEN_SCOPES` plus `read:admin` and `write:admin`. Site-admin
+membership alone isn't sufficient — Forgejo's token scope gate runs
+before the user-permission check, so `/api/v1/admin/*` returns
+`403 Forbidden` for any token without the admin scope bits, even when
+the bearer is a site admin.
+
+**Migration note**: if `PATCH /api/v1/admin/users/{name}` returns 403
+on an existing deploy, the core token predates the admin-scope
+addition. Delete `/var/lib/hyperhive/forge-core-token` and restart
+hive-c0re to re-mint with the new scopes.
+
+---
+
+## Per-agent forge accounts
+
+Each agent gets its own Forgejo user + access token, provisioned at
+boot by `hive-c0re::forge`. The provisioning flow is idempotent:
+existing accounts + tokens are reused, so container destroy/recreate
+doesn't lose forge identity. The token is written to
+`<state>/forge-token` (one line, no trailing newline) inside the
+agent container so `hive-forge` CLI + `forge_notify` poller can
+read it without touching c0re's host-side credential store.
+
+Two things live in the `agent-configs` Forgejo organization:
+
+- A config repo per agent (`agent-configs/<name>`). The
+  agent is a **write collaborator on its own** repo — it can push
+  config-change branches and open config PRs (Forgejo `pull_request`
+  webhook at `/webhook/config-pr` queues a `MergeConfigPr` approval;
+  `hive-c0re/src/forge/config_pr_poll.rs` re-scans every 5 minutes as a
+  fault-tolerance backstop) — but
+  `main` is branch-protected core-only: only hive-c0re's verify-and-ff-push
+  merge handler lands on `main`, an operator-team approval is required, and
+  the agent can neither push `main` directly nor self-merge. `main` is
+  fast-forward-only — hive-c0re never force-pushes (the merge handler's ff
+  push lands fine; the `push_config` mirror pushes `main` + the add-only
+  status tags without force, and treats a non-fast-forward rejection of
+  `main` after a rolled-back deploy as expected — the forge keeps the
+  approved history, the `failed/<id>` tag records the divergence).
+  Repos stay private, so an agent can't read another
+  agent's config. (Agents remain read-only collaborators on `core/meta`.)
+  hive-c0re also references this repo as the agent's **persistent meta
+  flake input** (`agent-<n>.url = git+http://<forge>/agent-configs/<n>.git`;
+  see [approvals.md § Meta flake](../agent-lifecycle/approvals.md)), fetching it as the `core`
+  user via a git credential helper that reads the live forge-core token —
+  so the config lives on the forge, not a hand-synced local checkout.
+- The dashboard links each container's "config" anchor to this
+  config repo, so operators can click straight from the SW4RM tab into
+  the rendered repo without an extra `git` step.
+
+The `hive-forge` CLI (separate workspace crate, see
+[`README.md`](../README.md) file map) wraps the Forgejo REST API
+with the per-agent token; agents call it for issue / PR / comment
+ops as if it were a peer. All REST calls across the workspace
+(`hive-forge` verbs, hive-c0re provisioning, this poller) go through
+the typed `forgejo-api` crate; only non-`/api/v1` web-router routes
+(attachment / artifact downloads, log streaming) and the poller's
+enrichment fetches of server-provided subject URLs stay on raw
+reqwest.
+
+## Notification poller (`hive-forge-notify/src/notify.rs`)
+
+Its own long-running per-agent daemon (`hive-forge-notify`, a sibling of
+`hive-bash-daemon` / `hive-matrix-daemon`) — it used
+to be a background task inside the `hive-agent` serve loop. Polls
+`GET /api/v1/notifications?all=false` every 30 seconds (Forgejo's
+unread-only filter), formats each notification as a broker
+`Wake { from: "forge" }` message, and delivers it to the agent's own
+inbox so claude's normal turn loop picks it up.
+
+The crate builds a second, independent binary for a different host — see
+[github.md](github.md#notifications).
+
+The host-specific calls — list unread, mark read, resolve own login —
+live behind `Source` in `hive-forge-notify/src/source.rs`;
+classification, formatting, dedupe and todo delivery are shared. Todo
+keys here are the bare thread ids, and must stay that way: renaming them
+would orphan every in-flight forge todo on the first restart after a
+deploy.
+
+### Mark-read on delivery
+
+On a **successful** broker delivery, `forge_notify` marks the thread
+read on forge straight away (`PATCH /notifications/threads/{id}`). The
+broker inbox is the durable work queue now — each delivered wake is a
+sqlite row with its own ack lifecycle — so the forge unread flag no
+longer needs to track whether the agent has *processed* a
+notification. Clearing it on delivery keeps forge's unread set **tiny
+by construction**: at rest it holds only threads that failed to
+deliver plus whatever arrived since the last 30s poll.
+
+That size property is the whole point. A container rebuild starts the
+poller with no memory of what it delivered, re-scans `?all=false`, and
+finds nothing stale — the delivered threads are already read on forge.
+Forge's own read-state is thus the durable, cross-rebuild record of
+what's been delivered; there is **no persisted cursor**.
+
+**Gotcha:** don't reintroduce a persisted dedup cursor here. A design
+that leaves threads unread and tracks delivery via a separately-persisted
+cursor is fragile — losing that cursor across a rebuild re-delivers the
+agent's entire still-unread backlog as a flood of fresh wakes. Forge's
+own read-state is the only durable record this design needs.
+
+**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
+read-before-comment guard (which keys off forge unread-state) does not
+force the agent to view a thread before commenting on it: the broker
+wake already carries the notification body, so *delivery is the read*.
+An agent that wants the full thread still runs `hive-forge comments` /
+`view`.
+
+**In-process dedupe (tiny, ephemeral).** A single-process map (thread
+id → last-delivered `updated_at`) guards the narrow window where a
+mark-read call *transiently fails* and the thread reappears unread in
+the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
+re-fire the wake. It is **not persisted** and resets on restart (forge
+read-state covers the durable case). Each poll prunes it to the ids in
+the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
+exceed that many entries (a debug assertion pins the invariant; the
+fetch limit and the bound are the same constant). A failed *delivery*
+is left unread and out of the map, so it resurfaces next tick.
+
+Self-echo notifications (the agent's own writes, see below) are marked
+read directly without a delivery — same `mark_read` call, no wake.
+
+### Activation gates (graceful no-ops)
+
+The poller starts disabled and stays that way for any of:
+
+- `HIVE_FORGE_URL` not set (no forge configured for this hive), or
+  not parseable as a URL.
+- `<state>/forge-token` missing or empty (agent has no forge
+  account — pre-provisioning or destroy-without-purge race).
+- Initial client construction fails (the typed `forgejo-api` client
+  for the API calls, or the plain reqwest client kept for the
+  best-effort enrichment fetches of server-provided subject URLs;
+  both extremely unlikely; treated as fatal-to-the-task only).
+
+Disabled = the spawned task returns immediately. All other failure
+modes (HTTP errors, parse errors, mark-read failures) are
+best-effort: logged at debug/warn and retried next tick.
+
+### Self-notification filtering
+
+Forgejo fires notifications for the agent's own actions (it opened a
+PR, posted a comment, submitted a review). Surfacing those would
+loop claude on its own writes. The comment/review case is dropped
+silently (mark-read without delivery):
+
+- **Self-authored comments / reviews** — comment payload's
+  `user.login` matches `own_login`.
+- **Self-authored creations** (an agent opening its own PR/issue) — the
+  already-fetched subject payload's poster `user.login` matches
+  `own_login`. Only _creations_ are dropped; a later state change on the
+  agent's own subject is driven by someone else and still surfaces.
+
+`own_login` is fetched at startup via `GET /api/v1/user`. On fetch
+failure the filter degrades open (no filtering) rather than crashing
+the task — a noisy inbox beats a silently-stuck poller — but the fetch
+is **re-attempted on each poll tick** until it succeeds, so a boot-time
+failure (the forge not yet reachable) self-heals instead of leaving
+self-echo filtering off for the whole process lifetime.
+
+### Body excerpt + truncation + heading escape
+
+The wake message embeds the comment / review body so the agent sees
+actual content without a follow-up fetch — **not** a new issue/PR's own
+description, which the wrapper table below deliberately omits. Three
+pipeline steps in order:
+
+1. **Truncate** to `BODY_TRUNCATE = 500` chars at a char-boundary;
+   appends `…` when cut. Truncation happens BEFORE escape so the
+   mention-overflow diff (next step) compares like-for-like against
+   the raw body.
+2. **Mention overflow extraction** — when truncation actually
+   trimmed content, walk the full body line-by-line and surface any
+   `@username` lines that fell outside the embed window. Rendered as
+   a trailing `mentions (truncated from body):\n  > <line>` block.
+   Mention detection requires the `@` to be at line start or
+   following a non-username byte, so email-style `foo@bar.com` does
+   NOT count.
+3. **ATX heading escape** — for each line that's a strict
+   `CommonMark` ATX heading (1-6 leading `#`s followed by a space,
+   tab, or end-of-line), prepend `\` so the embedded body doesn't
+   blow into a top-level h1/h2 inside the wrapper message when the
+   dashboard renders it. Lines like `#tag`, `#123`, `#!/bin/bash`
+   are NOT headings — no escape, no cosmetic noise. Indented
+   "headings" inside lists / nested quotes keep their leading
+   whitespace.
+
+The strict ATX rule is deliberate: `\#tag` and `#tag` render
+identically, so an over-eager escape just adds visual clutter
+without changing behavior. Setext-style headings (`title\n====`)
+are not handled — rarer in practice, would need multi-line
+lookahead.
+
+### Wrapper format
+
+Five shapes, distinguished by the notification's classification:
+
+| Trigger                             | Wrapper                                                                           |
+| ----------------------------------- | --------------------------------------------------------------------------------- |
+| Comment on issue / PR               | `[comment on PR #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
+| Review submission                   | `[PR approved #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...`   |
+| New issue / PR                      | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...`                       |
+| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...`               |
+| State change                        | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...`                    |
+
+The issue/PR's own description is deliberately never embedded here (only a
+comment/review body is — that's the actual new content a notification
+exists to surface): repeating a subject's own description on every "new"
+or "activity on" wake for it is noise the agent already has via the URL,
+not context (mara, triage: "just the issue title should be enough").
+
+Review labels come from the Forgejo `state` field: `APPROVED` →
+`approved`, `REQUEST_CHANGES` → `changes requested`, `COMMENT` →
+`review comment`. `PENDING` is dropped (review saved but not
+submitted yet — no peer-visible event). Unknown states fall back to
+the generic comment wrapper.
+
+A review submitted with **no body** renders `reviewed by: <author>` in
+place of the `<author>: <body>` line — deliberately worded to not collide
+with the meta-suffix `reviewer:` line (requested reviewers, below).
+
+### Merge/close vs a later comment
+
+A notification carrying a `latest_comment_url` normally takes the comment
+path. But a merged/closed subject **keeps** its `latest_comment_url` set,
+so a just-merged PR that had any prior discussion would route to the
+comment path and render `[comment on PR]` (with a stale pre-merge comment
+body) instead of `[PR merged]` — the agent never learns its PR merged.
+So when the notification IS the merge/close transition — its
+event time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` of the
+subject's `closed_at` (set for both `merged` and `closed`) — the
+state-change path wins even with a comment url present
+(`state_change_is_current`). A genuine **later** comment on an
+already-closed subject bumps `updated_at` well past `closed_at`, so it
+stays on the comment path and keeps its comment body. Missing/unparseable
+timestamps default to the state-change path, so a merge is never silently
+hidden behind a stale comment.
+
+#### Merge racing a comment
+
+The one gap the timestamp cut leaves: a genuine comment posted **within
+`NEW_ITEM_TOLERANCE_SECS` of the merge** bumps `updated_at` close enough
+to `closed_at` that `state_change_is_current` returns `true` — so it takes
+the state-change path and its body would be dropped. Best of both worlds:
+on the merge/close path we fetch the `latest_comment_url` comment and, when
+its `created_at` is strictly **after** the subject's `closed_at`
+(`comment_is_after_close`) — i.e. it raced the merge rather than being the
+pre-merge last comment the subject keeps — append it as a
+`comment by <author>: <excerpt>` block before the meta suffix
+(`fresh_post_close_comment_tail`). So the wake carries **both** `[PR merged]`
+and the racing comment. The kept pre-merge comment (created before
+`closed_at`) is left off, a self-authored racing comment is dropped (don't
+echo the agent's own write), and a missing/unparseable `created_at`/
+`closed_at` appends nothing (conservative — only surface a comment we can
+positively place after the close). Cost: one extra comment fetch on
+merge/close notifications, acceptable given how rare they are.
+
+### "new" vs "activity on"
+
+A review submitted with **no body** carries no `latest_comment_url`,
+so it misses the comment path and lands on the state-change path with
+`state == "open"` — exactly like a freshly opened PR. Labeling that
+`new PR` is misleading: agents dismiss it as a duplicate of the
+original open notification and miss the review. So the `open`
+state only earns the `new <kind>` label when the notification's event
+time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` (120s) of the
+subject's `created_at`. Anything later is labeled `activity on <kind>`
+— neutral and non-misleading, since we can't cheaply say _what_ the
+activity was without an extra reviews fetch. Missing/unparseable
+timestamps default to `new` (preserve prior behavior rather than mask a
+genuine new item). Timestamps are parsed by a small dependency-free
+RFC 3339 helper (`parse_rfc3339`).
+
+Number is extracted from `subject.html_url`'s last path segment
+(strips `#anchor` first); repo slug from `repository.full_name`.
+Both degrade gracefully when absent (number → blank, repo → blank)
+so unexpected Forgejo shapes don't crash the formatter.
+
+### Meta suffix
+
+Every wrapper ends with one or more of:
+
+- `assignee: <list>` — always present; `unassigned` when empty so
+  the line shape is stable.
+- `reviewer: <list>` — PR notifications only, present only when
+  `requested_reviewers` is non-empty.
+
+### Review-request override
+
+For new PRs, the kind label flips to `[review requested #N
+owner/repo]` when `own_login` appears in `requested_reviewers`,
+regardless of the Forgejo `reason` field. Forgejo doesn't reliably
+set `reason == "review_requested"` (often null instead), so the
+fallback checks the subject payload directly. Detection is gated on
+`is_new` so the label only fires once on PR creation, not on every
+subsequent comment.
+
+### Subscription management
+
+The poller does **not** auto-unsubscribe from repo watches — it
+delivers every unread notification it's handed. Bounding the
+firehose (dropping broad repo watches an agent doesn't need) is done
+explicitly via a hive-forge CLI subscription verb, not by the poller
+guessing which watches to drop. See the `subscription` verb in
+[`docs/tools/forge.md`](../tools/forge.md).

docs/integrations/github.md

@@ -0,0 +1,140 @@
+# GitHub accounts
+
+Give an agent a managed GitHub identity — a `gh` CLI and `git push` over
+HTTPS, both authenticated by an operator-supplied personal access token
+(PAT) — so it can run GitHub API calls and push commits without any manual
+`gh auth login`.
+
+Provisioning is UI-driven, mirroring the dashboard side of the
+[matrix account](matrix.md) flow: paste a PAT into the agent's credentials
+tab and it works. No per-agent nix declaration, no rebuild — the token is
+injected into the agent's state dir out of band.
+
+## Enabling
+
+The integration is **on by default** for every agent (`hyperhive.github.enable
+= true`), inert until a PAT is provisioned. There is nothing per-agent to
+declare — an agent gains GitHub simply by having a PAT written to its token
+file.
+
+To turn it off for the whole hive, set the host option:
+
+```nix
+services.hyperhive.github.enable = false;
+```
+
+hive-c0re's meta-flake renderer then injects `hyperhive.github.enable = false`
+into every agent, so no agent ships the `gh` wrapper or credential helper.
+(`hyperhive.github.enable` also exists per-agent for completeness, but the
+hive-wide host switch is the intended control.)
+
+github.com only. The token **value** never touches nix — it is written to
+`<state>/github-token` separately (see [Provisioning](#provisioning)).
+
+## Provisioning
+
+The PAT is operator-supplied. The primary path is the **dashboard
+credentials tab** (github sub-tab): paste the PAT for an agent and submit
+(`POST /api/github-account`). There is also a CLI path for
+recovery/scripting:
+
+```sh
+hivectl github set-token <agent> --token-stdin   # paste the PAT on stdin (preferred)
+hivectl github set-token <agent> --token <pat>    # inline (visible in shell history)
+```
+
+Either path has hive-c0re delegate the write to hive-priv, which stores the
+file `0600` owned by the agent (so the container can read it) — the same
+credential-injection path as forge/matrix tokens. See
+[hivectl → GitHub](../tools/hivectl.md#github).
+
+## Security
+
+- Use a **dedicated bot account**, never a human's.
+- Mint a **minimally-scoped PAT** — only the repos/scopes the agent's
+  workflow needs. Agents have passwordless sudo, so a compromised or
+  hallucinating agent can act as the account within the token's scopes;
+  scope is the real blast-radius limiter, and the container boundary is
+  the enforcement. See [security.md](../trust-boundary/security.md).
+
+Everything below this point is implementation detail (how the agent
+actually uses the token, and the notification poller's internals).
+
+## How the agent uses it
+
+When enabled, the container gets:
+
+- **A `gh` wrapper** on `PATH` (shadowing the raw `gh`) that exports
+  `GH_TOKEN` from the token file at invocation, then execs real `gh`. So
+  `gh pr create`, `gh api …`, etc. just work — `gh` derives the identity
+  from the token.
+- **A git credential helper** (`git-credential-hive-github`), wired via a
+  host-scoped `/etc/gitconfig` entry for `https://github.com`, so
+  `git push https://github.com/<owner>/<repo>` authenticates as
+  `x-access-token` + the PAT (GitHub ignores the username for PAT auth).
+  Host-scoped, so it never touches the forge (`localhost:3000`) or any
+  other remote.
+
+Both scripts read the token from `<state>/github-token` **at invocation
+time**, so a PAT written (or rotated) mid-session takes effect immediately
+— no container rebuild or restart. Until the file exists, `gh` / `git push`
+simply fail unauthenticated. The token path is baked into the scripts at
+build time (not read from an env var), because claude's Bash tool runs in a
+minimal environment that wouldn't carry one.
+
+## Notifications
+
+`hive-github-notify` polls github.com for the agent, turning each unread
+notification thread into a todo. It is a **separate binary and a
+separate systemd unit** from the internal forge's poller
+(`hive-forge-notify`, see [forge.md](forge.md#notification-poller-hive-forge-notifysrcnotifyrs)),
+installed by `nix/agent-modules/github.nix` under
+`hyperhive.github.enable`. Both binaries ship from the one
+`hive-forge-notify` derivation, so the unit is a second `ExecStart`
+path, not a new package.
+
+Two units rather than one daemon with two loops, because it puts the
+decision in nix: a hive built without this module has **no github poller
+in its closure at all**, which is what makes GitHub access separable
+rather than merely switched off. It is also why this is not a cargo
+feature — a feature would unify across the workspace and cost every
+crate its build cache.
+
+At runtime the poller needs the PAT above. No PAT, no polling: the unit
+logs why and exits 0, which is why it is `Restart = on-failure` and
+never `always` — a clean exit on a PAT-less agent must not become a
+restart loop.
+
+Forgejo's notifications API is modelled on GitHub's, so one tolerant
+parse serves both: `id`, `repository.full_name`,
+`subject {title,url,latest_comment_url}` and `updated_at` line up field
+for field. The two real differences are absorbed by lenient
+deserializers — GitHub sends the thread id as a *string* where Forgejo
+sends a number, and says `PullRequest` where Forgejo says `Pull`. Todo
+keys are prefixed `gh:<id>` so a github thread id cannot collide with a
+forge one.
+
+Two host differences worth knowing before touching this code:
+
+- **Auth scheme, not just value.** Forgejo takes
+  `Authorization: token <t>`; GitHub wants `Bearer <t>` plus `Accept:
+  application/vnd.github+json`, `X-GitHub-Api-Version` and a
+  `User-Agent`. Sending Forgejo's form to GitHub does not error — it
+  authenticates as *nobody* and silently drops to the unauthenticated
+  rate limit. The cheap way to tell the two apart is the rate-limit
+  header: `x-ratelimit-remaining` near 5000 is an authenticated user,
+  near 60 is anonymous.
+- **GitHub sets the cadence.** It returns `X-Poll-Interval` (60s in
+  practice, slower than our own tick) and rate-limits callers who ignore
+  it, so the loop re-arms to the server's interval whenever that is
+  *slower* than ours. A hint faster than our own tick is not a reason to
+  poll harder.
+
+⚠️ **This needs the `notifications` scope on the PAT.** A token minted
+for `gh` + `git push` typically carries `repo` only, which is enough to
+push and open PRs but **not** to read the notification stream (nor to
+mark a thread read, which is the same scope). A PAT without it doesn't
+break anything: the poller logs the refusal and stays quiet, and the
+agent simply never gets GitHub wakes. If an agent's GitHub
+notifications never arrive, check the token's scopes first — the
+symptom is silence, not an error.

docs/integrations/knowledge.md

@@ -0,0 +1,142 @@
+# Hive-wide knowledge repository
+
+`internal/knowledge` on the forge is a shared reference doc repo
+readable by every agent. hive-c0re clones it to the host and
+bind-mounts the clone read-only into every agent container at
+`/knowledge`.
+
+## Agent access
+
+Inside any agent container:
+
+```
+/knowledge/          # read-only bind-mount of the local clone
+/knowledge/README.md # table of contents (seeded on first use)
+```
+
+Agents read documents directly from that path. The mount is
+read-only — agents never write through it. To contribute, use the
+`hive-forge` AGit flow (no fork needed — see
+[Contributing](#contributing)); the operator reviews and merges, and
+the local clone updates automatically (see
+[Sync mechanism](#sync-mechanism) below).
+
+## Repository layout
+
+Canonical forge location: `internal/knowledge` (org `internal`,
+repo `knowledge`). The repo is public, so every agent's forge account
+has read access without an explicit per-agent collaborator grant;
+only the `core` account has push access, for auto-seeding.
+
+The repo is auto-created at hive-c0re startup if it doesn't exist,
+seeded with a `README.md` containing a contribution guide and a
+blank table of contents. Add an entry to that ToC each time you
+create a new document.
+
+## Sync mechanism
+
+hive-c0re maintains the local clone at
+`/var/lib/hyperhive/knowledge` via two paths:
+
+1. **Swarm event** — the swarm controller holds the single push hook on
+   `internal/knowledge` (see `docs/swarm/README.md` § Swarm-wide forge
+   webhooks). On any push to main, including merge commits, it sends an
+   event to every hive over the swarm queue and each hive runs `git
+   pull`, so agents see the new content on their next turn.
+
+   A hive that is offline when the event is sent does not get it on
+   reconnect — the periodic pull below is what closes that gap. So one
+   hive briefly showing older `/knowledge` content than another is
+   expected, and resolves by itself within the fallback interval.
+
+   **Do not add a per-hive hook.** A webhook has exactly one target
+   URL, so a second registration against the same repo does not add a
+   recipient — it takes delivery away from whoever registered first.
+   Earlier versions had each hive register its own; hive-c0re now
+   removes its own leftover at startup, so no operator step is needed
+   to migrate.
+
+2. **Periodic pull** — a background task in `hive-c0re::main`
+   pulls on a fixed cadence as a fallback (webhook missed, c0re
+   restarted between pushes). The pull is best-effort — a failure
+   logs a warning and does not affect the rest of the daemon.
+
+Both paths share the same `knowledge::pull()` function, which also
+handles the change notice below — neither path can forget to wire it
+in since the broadcast logic lives once, in `pull()` itself, not at
+each call site.
+
+### Change notice
+
+When a pull actually moves the local clone's `HEAD` (a real change,
+not a no-op — e.g. the periodic pull finding nothing new), hive-c0re
+broadcasts a short notice to every currently-registered agent's inbox:
+sender `system`, body `[system] /knowledge updated:` followed by a
+`git diff --stat <old>..<new>` summary of what changed (or a generic
+"see the repo" fallback if computing the diff itself fails). This is
+the same broadcast mechanism used for other hive-wide notices — inbox
+message only, no forced wake, and it carries the standard "this was a
+broadcast" hint. A diff or per-agent send failure is logged but never
+blocks the pull itself.
+
+The local clone is created (or refreshed) once at startup via
+`knowledge::ensure_local_clone`. If the repo is brand new and
+empty, `ensure_local_clone` seeds it with the default README
+before returning.
+
+## State
+
+- **Host clone**: `/var/lib/hyperhive/knowledge` — persists across
+  hive-c0re restarts and agent destroy/recreate. Deleted only by
+  manual operator action.
+- **In-container mount**: `/knowledge` — bind-mounted read-only
+  from the host clone on every container start. Gone when container
+  is stopped; reappears on next start with the current clone state.
+
+The mount deliberately **excludes `.git`**: the host clone embeds the `core`
+token in `.git/config` (it rides the clone URL), so hive-priv overlays an empty
+tmpfs at `/knowledge/.git` — agents see the documents, not the repo metadata or
+token.
+
+## Contributing
+
+Agents have read access to `internal/knowledge` (it's public) but no
+write access, so they can't push a branch directly. The supported path
+is Forgejo's **AGit flow** through the `hive-forge` CLI — no fork
+required.
+
+1. Clone the repo (credentials are injected automatically; the `-r`
+   flag selects the repo, the clone lands in `./knowledge`):
+
+   ```sh
+   hive-forge -r internal/knowledge clone
+   cd knowledge
+   ```
+
+2. Create a branch and add or update a document, then commit normally.
+
+3. Open (or update) a PR with `--agit`. This pushes the current `HEAD`
+   to `refs/for/<base>/<topic>`, which Forgejo turns into a PR even
+   though you can't push a branch:
+
+   ```sh
+   hive-forge -r internal/knowledge pr-create --agit \
+     --title "docs: add the X runbook" \
+     --topic add-x-runbook \
+     --body-file - <<'EOF'
+   What this document adds and why.
+   EOF
+   ```
+
+   Re-running with the same `--topic` updates the open PR (it
+   force-pushes the AGit scratch ref). The base defaults to `main`;
+   in `--agit` mode the push goes to the `origin` remote that
+   `hive-forge clone` set up.
+
+4. The operator reviews and merges. The webhook fires on merge; every
+   running container sees the updated content within seconds (see
+   [Sync mechanism](#sync-mechanism)).
+
+Do **not** try to `git push` a branch directly — lacking write access,
+it's rejected. The `--agit` flow above is the no-fork path that works
+from any agent.

docs/integrations/matrix.md

@@ -0,0 +1,287 @@
+# hive-matrix
+
+Private Matrix homeserver (matrix-tuwunel — the conduwuit
+successor) wrapped in a nixos-container, plus optional fluffychat-web
+client at `chat.<swarm-domain>/` (the `gatewayHost` vhost). Configured via
+`services.hyperhive.swarm.matrix.*`; vhost routing lives in
+[`gateway.md`](../networking/gateway.md).
+
+## Container shape
+
+Same shape as [`gateway.md::hive-forge container shape`](../networking/gateway.md):
+
+- Container name `hive-matrix` (not `h-*`) so c0re's lifecycle
+  scanner ignores it; operator manages via the standard
+  `nixos-container` CLI.
+- Keeps hive-matrix from fighting any `services.matrix-*` the
+  operator already runs on the host — separate systemd namespace,
+  separate state dir.
+- Container shares the host network namespace
+  (`privateNetwork = false`) for state + systemd-unit isolation. Agents
+  reach the homeserver at `chat.<swarm-domain>` via the gateway (agents
+  run in private netns and can't access host loopback directly).
+- Persistent state at
+  `/var/lib/nixos-containers/hive-matrix/var/lib/matrix-tuwunel/`
+  survives container restart / host reboot. To wipe, destroy the
+  container.
+
+## Identity vs API listener: `serverName` vs `gatewayHost`
+
+Two distinct hostnames:
+
+- **`serverName`** — matrix-spec `server_name`, embedded
+  *irrevocably* in every `@user:<server_name>` and `!room:<server_name>`
+  identifier minted on this homeserver. Cannot be changed later
+  without abandoning every account and chat history. Defaults to the
+  bare `services.hyperhive.swarm.domain`; clients auto-discover the
+  actual API endpoint via the `.well-known/matrix/{client,server}`
+  routes the gateway serves at that domain.
+- **`gatewayHost`** — the API listener hostname, where the gateway's
+  matrix vhost proxies `/_matrix/*` to tuwunel. Defaults to
+  `chat.<services.hyperhive.swarm.domain>`. Set to `null` to skip the
+  gateway vhost (tuwunel stays direct on `httpPort`).
+
+Both default under the **swarm** domain, because a swarm runs one
+homeserver: tying its identity to a single hive's domain would make
+relocating the container between hives look like a different
+homeserver.
+
+⚠️ **They are still not interchangeable, and the difference is the
+cost of changing one.** `gatewayHost` is a routing detail clients
+rediscover through `.well-known`, so it is safe to move on a running
+deployment. `serverName` is baked into every user and room id, so
+adopting a new one does **not** rename the existing users and rooms —
+it strands them, because their ids still name a homeserver that no
+longer answers.
+
+### Upgrading a homeserver that already has ids
+
+`serverName`'s default has changed across releases. A homeserver that
+has already minted ids under an older default must **pin the value it
+actually minted them under**, not adopt the new default — see above
+for why adopting a new one strands existing users and rooms:
+
+```nix
+services.hyperhive.swarm.matrix = {
+  # whichever this deployment already uses
+  serverName = config.services.hyperhive.domain;
+  gatewayHost = "matrix.${config.services.hyperhive.domain}";
+};
+```
+
+A rebuild on a host that already has a homeserver prints a
+`hive-matrix: WARNING — … serverName is unset` line when this is missing,
+naming the value it is about to default to. That warning is why this
+section exists; it never fails the rebuild, so it is on you to act on it
+before the ids are minted.
+
+## Default-closed firewall
+
+`openFirewall` defaults to `false` (secure-by-default): the host
+reaches the homeserver on loopback, and agent containers reach it
+at `chat.<swarm-domain>` via the gateway — so the firewall hole only
+matters for access from *outside* the host. Flip to `true` when
+announcing the homeserver to other hives or when an external matrix
+client needs to reach the client-server API directly.
+
+Federation port 8448 is intentionally not opened here — tuwunel
+serves the federation API on the same `httpPort` as client-server
+by default. Reaching it on 8448 needs either an explicit tuwunel
+bind to that port OR a reverse-proxy + `.well-known/matrix/server`
+delegation (the latter lives in `gateway.md::Discovery flow`).
+
+## Provisioning flow (registration token)
+
+Token-gated registration: hive-c0re holds the token, agents never
+see it. The agent only receives the resulting `access_token`.
+
+1. **System activation** writes a 32-byte random hex token (64
+   chars) to `cfg.registrationTokenFile`
+   (`/var/lib/hyperhive/matrix-register-token` by default), mode
+   `0600 root:root`, before any container start. Idempotent — only
+   writes when the file is missing or empty; always re-applies 0600
+   (normalises any 0640 / world-readable carry-over from
+   pre-LoadCredential deployments). This runs at activation time
+   (not first container start) to dodge a race where nspawn creates
+   an empty file when the bind-mount target is missing and tuwunel
+   reads `registration_token_file=""`, rejecting every registration
+   until next restart.
+2. **Read-only bind-mount** maps the host file into the tuwunel
+   container at the same path.
+3. **systemd `LoadCredential=`** inside the container copies the
+   bind-mounted file into
+   `/run/credentials/tuwunel.service/registration_token`, owned by
+   tuwunel's dynamic user with mode `0400`, at service start. The
+   host file stays `root:root 0600` — no `chown :tuwunel` /
+   `chmod 0640` / GID-pin gymnastics required. Keeps
+   `DynamicUser = true` + `PrivateUsers = true` intact.
+4. tuwunel's `registration_token_file` points at the credentials
+   path, not the original bind-mount path.
+5. **hive-c0re** uses the token to register each agent account via
+   the matrix-spec UIAA registration flow, persists the returned
+   `access_token` to `<agent-state>/matrix-token`. The agent's
+   matrix MCP client authenticates with that access_token and
+   never touches the shared registration token.
+6. **hive-c0re restarts `hive-matrix-daemon`** for the agent
+   immediately after writing the token so the daemon picks up the
+   new credential without waiting for a full container restart. If
+   the restart fails (e.g. daemon not yet running on first boot) the
+   error is logged as a warning and the `.path`-trigger sibling
+   (`hive-matrix-daemon.path` watching for `matrix-token` appearance)
+   brings the daemon up on the same boot cycle anyway.
+
+Initial rollout settings:
+
+- `allow_federation = true` at the protocol level so swarms can be
+  wired up later by extending `trustedServers` without a homeserver
+  restart. `trusted_servers = []` keeps it effectively closed
+  until peers are listed.
+- `allow_registration = true` (required for the token flow to
+  engage). The absent
+  `yes_i_am_very_very_sure_…_open_registration_…` flag keeps the
+  server closed to anyone without the token.
+- `allow_encryption` — server-side E2EE switch, sourced from
+  `services.hyperhive.swarm.matrix.allowEncryption` (**default `false`**, opt-in).
+  Off by default because on the hive-internal homeserver the operator
+  already controls the transport; turn it on for encrypted rooms on
+  external / federated homeservers or to keep contents opaque to the
+  homeserver admin. **The agent matrix client always supports decryption
+  regardless of this flag** — it uses the `e2e-encryption` feature of
+  `matrix-sdk` so it can read encrypted rooms it is invited to even when
+  this homeserver doesn't permit room encryption. Crypto keys are stored
+  in the per-agent sqlite store under the state dir and persist across
+  restarts (lost on `--purge`). `read_room` decrypts via
+  `room.messages()` — UTD events surface as `event_type =
+  "m.room.encrypted"` with `body = "[unable to decrypt]"`.
+  Cross-signing and automatic key backup are not enabled for the first
+  pass: static bearer-token bot accounts can't bootstrap cross-signing
+  without MSC3967.
+
+## Hive Matrix Space
+
+On first boot (after all agent accounts are provisioned), hive-c0re
+creates a private **Matrix Space** named `"hive"` using the admin
+account (`@hive:<server_name>`) and invites every provisioned agent
+into it. This gives the operator a single Space in FluffyChat or any
+Matrix client that groups all agent-to-agent + operator rooms in one
+place.
+
+The sweep also provisions a default **`hive-chat` room** as an
+`m.space.child` of the Space. Joining a Space doesn't auto-join
+child rooms — the explicit room entry ensures the operator and every
+agent can find a common chat room without manual setup. Room join is
+restricted (any Space member including the operator can join; agents
+are explicitly invited). Room version pinned to 10 for the restricted
+join floor.
+
+**State**: both room IDs are persisted to `/var/lib/hyperhive/matrix/`
+(mode `0600`, owned by the hive-c0re service user):
+- `space-room-id` — the Space itself
+- `chat-room-id` — the `hive-chat` room
+
+These paths are **outside** every agent state dir and are NOT deleted by
+`nixos-container destroy --purge` — both survive full agent purges and
+are reused on re-provision.
+
+**Idempotent**: if the files exist and are non-empty, the Space and
+room are considered already created. Delete the files to force
+re-creation (e.g. after a homeserver wipe).
+
+## Configuration tuning
+
+```nix
+services.hyperhive.swarm.matrix = {
+  trustedServers  = [ "matrix.org" "example.com" ];   # default: []
+  maxRequestSize  = 20000000;                          # default: 20 MB
+};
+```
+
+**`trustedServers`** (default `[]`) — list of peer homeserver names
+whose signing keys tuwunel will fetch and trust. Federation is enabled
+at the protocol level from first boot (`allow_federation = true`) but
+no remote homeserver is trusted until listed here. For a closed
+single-hive deployment the default empty list is correct — add peer
+hive domains here when connecting hives into a swarm (see
+[`docs/swarm/`](../swarm/README.md)).
+
+**`maxRequestSize`** (default `20_000_000` bytes = 20 MB) — maximum
+size of a single matrix client request body. Matches the matrix-spec
+recommendation for media uploads and the upstream tuwunel default.
+Raise for deployments that need large file transfers; lower for
+resource-constrained hosts where a 20 MB request is unexpectedly large.
+
+## Assertion rationale
+
+`config.assertions` in this module fail eval early rather than ship
+surprising behaviour:
+
+- **`cfg.gatewayHost != ""`** — same footgun as `forge.domain`:
+  empty string renders `.<hive>`-shaped garbage in both nginx
+  `server_name` (treated as wildcard catch-all, surprising) and
+  `/etc/hosts` (invalid entry). `null` is the right opt-out shape;
+  empty string is rejected explicitly.
+SSO is unconditional, so the three below are requirements of running a
+homeserver at all rather than of a setting:
+
+- **`sso.clientSecretFile` is required** — fails at eval, not at boot:
+  tuwunel reads its identity providers from the config file, so a
+  half-configured one can stop the homeserver from starting outright
+  rather than merely hiding a login button. On a host that also runs
+  the swarm's authelia it is wired up for you.
+- **`swarm.authelia.url` is required** — without a provider URL there
+  is nothing to discover against.
+- **`gatewayHost != null` is required** — the SSO callback URL is
+  format-locked to `<homeserver>/_matrix/client/unstable/login/sso/callback/<client_id>`,
+  and the identity provider needs a public name to redirect the
+  browser to.
+
+`server_name`'s own bogus-value guard lives in `hive-network.nix`
+(`services.hyperhive.domain != null`), not here — see
+[`docs/networking/network.md`](../networking/network.md).
+
+## fluffychat-web build fixes
+
+`pkgs.fluffychat-web` ships from `flutter341.buildFlutterApplication`,
+which has two upstream gaps for fluffychat's web target:
+
+- The dart web-worker entry point (`web/native_executor.dart`) isn't
+  compiled — `buildFlutterApplication` only runs `flutter build web`
+  on the main entry.
+- `native_imaging`'s C source isn't built — emscripten isn't a
+  flutter-builder native build input.
+
+Both fixed in `nix/host-modules/hive-matrix.nix` via two derivations:
+
+- **`fluffychat-web-imaging`** builds `Imaging.{js,wasm}` from the
+  `native_imaging` C source via `pkgs.emscripten`. Source comes
+  from `pkgs.fluffychat-web.passthru.pubspecLock.dependencySources.native_imaging`
+  — already in the build closure of the flutter app, so no parallel
+  hash pin and version auto-syncs with nixpkgs bumps. Build closure
+  is ~3.6 GiB (emscripten LLVM); runtime closure is just the two
+  output files. `dontConfigure = true` because cmake runs inside
+  `js/Makefile` via `emcmake cmake`, not at the package root. The
+  build script needs `HOME` + `EM_CACHE` writable for emscripten's
+  on-demand sysroot build (libc, libc++ → wasm).
+- **`fluffychat-web-fixed`** is `pkgs.fluffychat-web` plus a
+  `postInstall` patch that (a) compiles `web/native_executor.dart`
+  via `dart compile js` (dart from the flutter341 closure, no
+  incremental cost) and (b) installs `fluffychat-web-imaging`'s
+  outputs into `$out`.
+
+Two non-obvious details worth knowing before touching either derivation:
+
+- **`make -C js`** instead of `cd js; make` — keeps the build-phase
+  pwd at the source root so `installPhase` doesn't have to know
+  about the cd. Robust against future reorders / `dontBuild`.
+- **`web/native_executor.dart`** as a build-CWD-relative path,
+  *not* `$src/web/...` — `dart`'s `package_config.json` walk-up
+  needs to hit `buildFlutterApplication`'s pub-get output
+  (`.dart_tool/` in the build CWD). Walking up from a read-only
+  `$src/` store path finds no `.dart_tool/` and errors with
+  "Couldn't resolve the package 'matrix'".
+
+Drop both derivations when nixpkgs's flutter builder grows worker
++ emcc support upstream.
+
+Mount point is `chat.<swarm-domain>/` (the `gatewayHost` vhost);
+upstream `--base-href "/"` is correct at sub-domain root, no override.

docs/networking/gateway.md

@@ -0,0 +1,645 @@
+# hive-gateway
+
+Single nginx in front of every hyperhive web surface. Runs on the **host**, next to hive-c0re, rather than in its own container: it shares the host netns anyway (see [Vhost map](#vhost-map) below), so containerizing it would buy no network isolation while costing a resolv.conf sync, a machine-bus reload, and three bind mounts. System-config (not meta-flake managed). Configured via `services.hyperhive.gateway.*` + per-subsystem opt-in flags in `services.hyperhive.{forge,matrix,...}`.
+
+## Vhost map
+
+| URL | vhost | upstream | source |
+| --- | --- | --- | --- |
+| `<hive>/` | `_` (catch-all) | dashboard dist (static, from `servedFrontend`); `/api/` + `/webhook/` → hive-c0re (`7000`) | always |
+| `<hive>/agent/<name>/` | `_` | per-agent harness (UDS or TCP) | `agents.conf` (runtime-generated) |
+| `<hive>/.well-known/matrix/{client,server}` | `_` | inline JSON (no upstream) | `matrix.enable && domain != null` |
+| `<hive>/matrix/` (deprecated) | `_` | 301 → `chat.<swarm>/` | `matrix.gui.enable` |
+| `forge.<swarm>/` | `forge.<swarm>` | forgejo (`3000`) | `forge.behindGateway` |
+| `chat.<swarm>/_matrix/*` | `chat.<swarm>` | tuwunel (`8008`) | `matrix.gatewayHost != null` |
+| `chat.<swarm>/` | `chat.<swarm>` | fluffychat-web static | `matrix.gui.enable` |
+| `chat.<swarm>/config.json` | `chat.<swarm>` | inline JSON (FluffyChat boot config) | `matrix.gui.enable && domain != null` |
+| `auth.<swarm>/` | `auth.<swarm>` | authelia (`9091`) | `deploy.authelia` |
+| `<swarm>/` | `<swarm>` | swarm-ui dist (static), behind an authelia subrequest | `deploy.swarm-ui` |
+
+The authelia vhost is declared only by the host that **runs** authelia, not by every hive that uses it — a client hive knows the swarm's `authelia.url` but must not answer for a name it doesn't serve. Its server name is exactly `swarm.authelia.domain`: authelia validates `authelia_url ⊂ session cookie domain` at startup, so a near-miss is a container that refuses to boot. It carries no `auth_basic` — the login page must not sit behind the login mechanism it replaces — and sets the four `X-Forwarded-{Proto,Host,Uri,For}` headers, since authelia decides by the *original* request rather than the hop it sees.
+
+⚠️ **A `502` from this vhost usually means authelia has no users yet, not that the proxy is misconfigured.** Authelia treats an empty user store as a fatal startup error, so an enabled-but-unbootstrapped swarm crash-loops the container while the vhost in front of it works perfectly. Check `journalctl -M swarm-authelia -u authelia-swarm` before suspecting anything here; the bootstrap step is in [`swarm/sso.md`](../swarm/sso.md).
+
+Per-agent UIs stay sub-path, forge and matrix get sub-domains — see
+[Sub-domain shape (rationale)](#sub-domain-shape-rationale) below for why.
+
+## Discovery flow (matrix)
+
+Operator points client at `<hive>`. Sequence:
+
+1. Client fetches `https://<hive>/.well-known/matrix/client` → `{"m.homeserver":{"base_url":"https://chat.<swarm>"}}` (no port suffix when gateway listens on 443). The gateway always terminates TLS, so the scheme is always `https`; a non-default `httpsPort` is reflected as the port suffix.
+2. Client connects to `chat.<swarm>/_matrix/client/...`.
+3. Gateway routes `/_matrix/*` → tuwunel at `127.0.0.1:8008`.
+
+matrix-dart-sdk (FluffyChat etc.) hardcodes `https` for the well-known fetch regardless of input scheme, so the discovery endpoint MUST be https — see "Self-signed TLS" below for the cert generation that backs the default-on path.
+
+Federation peers fetch `.well-known/matrix/server` → `{"m.server":"chat.<swarm>:<httpsPort>"}` (the federation delegation always carries an explicit port, even the HTTPS default 443 — the https-implies-443 elision only applies to the client base_url above). Gateway only listens on configured `port` (+ `httpsPort` when TLS on); cross-hive federation needs either an SRV record (`_matrix._tcp.chat.<swarm>` → port 80 / 443) OR `matrix.openFirewall = true` so peers reach tuwunel's federation port directly. Hyperhive is mostly closed/internal, so this rarely bites.
+
+## SPA fallback (Accept-header pattern)
+
+The per-agent UIs and the `chat.<swarm>` vhost serve a flutter/SPA bundle via the Accept-header pattern below. The dashboard vhost instead routes by **path** — see [Dashboard: path-based routing](#dashboard-path-based-routing-not-accept-header) below. Two requirements collide:
+
+- hard-refresh on a sub-route must serve `index.html` (SPA's client-side router takes over after JS bootstrap)
+- a non-navigation request that isn't an on-disk asset must NOT get HTML with the wrong content-type
+
+Solution: an `nginx http`-context `map $http_accept $<name>_spa_target { ... }` keyed on the request's Accept header. Browser navigations (`Accept: text/html,...`) get `index.html`; everything else (`Accept: image/*`, `*/*`, `application/json`, `text/event-stream`, …) gets a sentinel nonexistent path, so `try_files $uri $<name>_spa_target <final>` falls through to `<final>`. No extension allowlist, no `if` block, no regex heuristics.
+
+For matrix / per-agent static assets, `<final>` is `=404` (a missing asset is just missing).
+
+### Dashboard: path-based routing (not Accept-header)
+
+Every hive-c0re backend route lives under `/api/` plus the single `/webhook/knowledge` endpoint, so the dashboard vhost routes by **path**, not Accept header — deterministic, unlike a content-type split where the same URL could resolve differently depending on the caller's `Accept` header:
+
+- `location /api/` → hive-c0re (`7000`): all dashboard data, actions/mutations, and the two SSE streams (`/api/dashboard/stream`, `/api/build-logs/id/{id}/stream`). Carries `proxy_buffering off` + a 1d read timeout for the streams.
+- `location /webhook/` → hive-c0re: the knowledge webhook.
+- `location /` → the dashboard dist (from the `servedFrontend` nix-store path) with `try_files $uri /index.html` (SPA fallback).
+
+Each location carries a duplicated `auth_basic` block (separate locations don't inherit it). This keeps the gateway static-serving the dashboard dist while hive-c0re stays API-only — a frontend-only change doesn't rebuild or restart the core daemon. A new top-level c0re route prefix (beyond `/api` + `/webhook`) needs a matching `location` added to the dashboard vhost.
+
+## Local dev (`localHostsEntry`)
+
+`services.hyperhive.gateway.localHostsEntry = true` adds entries to the host's `/etc/hosts`:
+
+- `<hive-domain>` → `127.0.0.1`
+- `forge.<swarm>` → `127.0.0.1` (when forge.behindGateway)
+- `chat.<swarm>` → `127.0.0.1` (when matrix.gatewayHost set)
+- `auth.<swarm>` → `127.0.0.1` (when deploy.authelia)
+
+`lib.unique` de-dupes if any sub-domain happens to equal another entry. Operators with real DNS leave it off.
+
+## Sub-domain shape (rationale)
+
+Operator decision: sub-domain over sub-path for forge + matrix, sub-path for per-agent UIs.
+
+- forgejo's default `ROOT_URL = http://<host>/` works without any `X-Forwarded-Prefix` gymnastics — sub-domain hosting is the canonical Forgejo deploy shape.
+- matrix-spec deployments universally use `matrix.<server_name>` for the actual API listener — federation already expects this.
+- per-agent UIs are hyperhive-internal and base-path-aware specifically for `/agent/<name>/`. Sub-domain per agent would multiply DNS + TLS-per-subdomain cost without per-app config wins.
+- cookie / storage isolation: a future forge XSS can't reach the dashboard session because they're different origins.
+
+`services.hyperhive.{forge.domain,matrix.gatewayHost}` take the full hostname (`forge.darkest.space`, `git.example.com`) rather than a label that gets concatenated with hive-domain — operators want control over the full shape, not a forced `<label>.<hive-domain>` pattern.
+
+## Tuning knobs
+
+Per-vhost timeouts + body-size limits live in the location blocks:
+
+- forge `/` (forgejo): `client_max_body_size 1G` (LFS), `proxy_read_timeout 1h` (multi-GB clones), `proxyWebsockets = true` (live-update endpoints).
+- matrix `/_matrix/` (tuwunel): `client_max_body_size 50M` (media uploads), `proxy_read_timeout 1h` (long-poll `/sync`), CORS `*` (federation + cross-origin clients), `proxyWebsockets = true`.
+- per-agent `/agent/<name>/`: `proxy_read_timeout 1d` (long-lived SSE / WebSocket dashboards), `proxyWebsockets = true`, `X-Forwarded-Prefix` set so the harness can build absolute URLs when relative isn't enough.
+
+SSH for forge stays direct on `cfg.sshPort` — separate listener protocol, not HTTP-over-nginx.
+
+## Per-agent unix-socket upstream
+
+All agents bind their web UI on a unix-domain socket at
+`/run/hive-agent/<name>/web.sock` — the `HIVE_WEB_SOCKET` env var is
+now set unconditionally for every agent. The mechanism:
+
+1. **Agent side**. `HIVE_WEB_SOCKET=/run/hive-agent/<name>/web.sock`
+   is set on every harness service env; `web_ui::serve` binds a
+   `UnixListener` at that path.
+2. **Host side**. `hive-c0re` bind-mounts the per-agent subdir
+   (`/run/hive-agent/<name>/`) into the agent's container. Dir
+   bind, not file bind — file bind-mounts don't survive the
+   harness's `unlink + bind(2)` cycle on socket replace. Per-agent
+   subdir keeps each agent's container blind to siblings' sockets.
+
+   The dir is `0751`, owned by the agent's container uid/gid, so
+   nginx reaches `web.sock` through `o=--x` (traverse) and the socket's
+   own `0666`. The gateway is one of three principals sharing that dir
+   and does not own its ownership rules — see
+   [`docs/trust-boundary/boundary.md`](../trust-boundary/boundary.md#the-per-agent-socket-dir).
+3. **Marker gate**. After successful `bind_unix`, the harness drops
+   `<dir>/hyperhive-socket-bound` next to the socket. c0re's
+   `agent_sockets::write` filters its JSON map by marker presence —
+   only agents whose harness has actually bound the socket appear there.
+   (Legacy name `.bound` also accepted during the transition window.)
+4. **Gateway side**. `gateway_nginx::write` generates
+   `/var/lib/hive-gateway/conf/agents.conf` — a plain nginx include
+   file with one `location /agent/<name>/` block per agent. Always
+   a UDS upstream (`http://unix:/run/hive-agent/<name>/web.sock:/`);
+   if the socket is not yet bound, nginx returns 502 caught by the
+   `error_page 502 503 504 = /__hive_agent_unreachable` directive.
+   nginx includes `/var/lib/hive-gateway/conf/agents.conf` — the same
+   path c0re writes, since both run on the host.
+   After each write, c0re triggers the appropriate nginx action via
+   `hive-priv` (which is root; hive-c0re runs as the unprivileged
+   `hive-core` user and cannot act on a system unit).
+   `hive-priv` queries `ActiveState` and dispatches:
+   - active → `systemctl reload nginx` (SIGHUP, zero-downtime)
+   - failed → `systemctl reset-failed nginx` + `systemctl start nginx`
+   - otherwise → `systemctl start nginx`
+   This is an explicit trigger rather than a path unit watching the
+   file: the write and the reload belong in one causal chain c0re can
+   retry and report on (`RELOAD_PENDING`), not two units racing on an
+   inotify event.
+
+c0re regenerates `agents.conf` (and triggers a reload) on two
+triggers: every topology change (new/removed agents) and every 10s
+marker poll tick (`agent_sockets::spawn_poll`). `write()` is
+idempotent — skips the rename when content is unchanged. Failed reloads
+are retried automatically on subsequent poll ticks via
+`gateway_nginx::reload_if_pending`.
+
+`agents.conf` uses atomic `<path>.tmp` + `rename()` writes so a crashing
+c0re process never leaves a partial or unparseable file behind.
+
+## Dashboard link shape (gateway vs direct)
+
+When the gateway is in front, the SW4RM tab builds per-agent links
+as same-origin `/agent/<name>/…` URLs instead of the legacy direct
+`http://<host>:<container.port>/` TCP shape. The signal comes from
+`StateSnapshot.gateway_enabled`, sourced from the
+`HIVE_GATEWAY_ENABLED` env the c0re NixOS module now always sets
+(`services.hyperhive.gateway.enable` was removed — the gateway runs
+unconditionally alongside hyperhive), so this is effectively always
+true; the `false` branch is retained as a defensive fallback for the
+env being unset. Three render sites
+flip together: the primary agent-name link, the favicon fetch
+(`<url>/icon`), and the nav-strip `container`-kind links from
+`DashboardState.links` (`GET /api/dashboard-state`). `forge`-kind nav-strip links still
+resolve against `http://<host>:3000` (separate sub-domain transition
+tracked by `forge.behindGateway`); `external`-kind links are
+already absolute. See `docs/web-ui/dashboard.md::Container row` for the
+frontend-side derivation.
+
+## TLS modes
+
+The gateway always terminates TLS — self-signed is the implicit floor when
+nothing else is configured, so there is no http-only mode. Three modes,
+selected by which (if any) external TLS source is set:
+
+| mode | config | cert source | `.well-known` scheme |
+|---|---|---|---|
+| self-signed (default) | neither `tls.certDir` nor `tls.acme` set | host hive-CA signs a gateway leaf (RSA-4096) | `https` |
+| ACME (Let's Encrypt) | `tls.acme.enable = true` | nginx via HTTP-01 | `https` |
+| operator cert | `tls.certDir` set | read from the operator's dir | `https` |
+
+The `gateway.selfSignedTls` option has been **removed** — self-signed
+is now derived from the absence of `tls.certDir` / `tls.acme`. A config
+that still sets it fails eval with a removal message; use `tls.certDir`
+/ `tls.acme` to override the default.
+
+### ACME / Let's Encrypt (`tls.acme`)
+
+Simplest production path for operators with a public domain:
+
+```nix
+services.hyperhive.gateway = {
+  openFirewall  = true;
+  tls.acme = {
+    enable = true;
+    email  = "admin@example.com";
+  };
+};
+```
+
+nginx obtains and auto-renews certs via the ACME HTTP-01 challenge on `port` (default 80). Certs land in `/var/lib/acme/` on the host, managed by nixpkgs's `security.acme` in the ordinary way.
+
+**Requirements**: `services.hyperhive.domain` must be publicly DNS-resolvable to this host, and `openFirewall = true` so Let's Encrypt can reach `/.well-known/acme-challenge/`. Each active vhost (main domain, `forge.<swarm-domain>`, `chat.<swarm-domain>`) gets its own cert via separate ACME challenges — the swarm services default to names under `services.hyperhive.swarm.domain`, so **every one of those names must resolve to this host too**, not just the hive's own.
+
+Mutual exclusion: `tls.certDir` set together with `tls.acme.enable = true` fails an assertion — pick one external TLS source (or neither, for the self-signed default).
+
+### Self-signed TLS (default)
+
+On by default, and listens on `httpsPort` (default 443) on every vhost beside the plain-http `port` (default 80).
+
+The issuer is a **host-held hive CA**, not a bare self-signed leaf. A host service (`hive-tls-ca.service`, from the `hive-tls` module) generates a long-lived CA (`services.hyperhive.deploy.hive-controller.tls.caValidityDays`, default ~20y) under `services.hyperhive.deploy.hive-controller.tls.stateDir` (default `/var/lib/hive-tls`), then signs a gateway **leaf** (`leafValidityDays`, default 30d) with it. `hive-gateway-self-signed-cert` then imports the leaf into nginx's state dir (`/var/lib/hive-gateway/tls/{cert,key}.pem`).
+
+⚠️ **Do not collapse that import unit into pointing nginx at the CA dir.**
+It does two jobs, and skipping it has taken the gateway down in production
+before. It re-modes the leaf (`hive-tls-ca` writes the key `0600
+root:root`; nginx's pre-start `nginx -t` runs as the *nginx user*, so a
+`0600` key fails the config test and blocks the unit), and it guarantees
+**every cert path the nginx config names exists** — which is what the
+swarm-services fallback below is for.
+
+**Why a CA, not a bare leaf**: a bare self-signed leaf is its own trust anchor, so every regeneration is a new anchor every consumer must re-trust — and a runtime-generated leaf can't be wired into an agent's build-time trust store at all. With a stable CA, agents and federation peers trust it *once*; leaf rotation never re-breaks them.
+
+**What consumers trust**: `trust-bundle.pem` in the same state dir, not `ca.pem`. The hive CA is itself issued under the swarm root ([`swarm/ca.md`](../swarm/ca.md) has the hierarchy), and an intermediate is not a chain a verifier can terminate at — so the bundle carries the hive CA plus whatever it is rooted at. nginx is handed the leaf with the hive CA appended for the same reason. Everything that trusts the hive's TLS reads the bundle: agents (via `security.pki.certificateFiles`), the CI and forge containers, and a federating peer.
+
+**Why on by default**: matrix-dart-sdk (FluffyChat's SDK) hardcodes `https://<host>/.well-known/matrix/client` for homeserver discovery and refuses to fall back to plain http. Without TLS the browser client cannot bootstrap.
+
+**Cert shape**: leaf subject CN = bare hive domain; subjectAltName is `<hive>` plus wildcard `*.<hive>`, so all current and future sub-domain vhosts validate under the same leaf + the hive CA. A swarm service whose name is *not* under this hive's domain cannot be added here — the hive CA is name-constrained to `<hive>`, and a violating SAN invalidates the whole leaf, not just that name. Those names get the swarm-services leaf instead ([`swarm/ca.md`](../swarm/ca.md)).
+
+**Rotation**: `hive-tls-ca.service` is idempotent — it re-signs the leaf when it is missing or within 30 days of expiry, always under the same CA (so consumer trust is undisturbed). The CA itself is regenerated only if missing or already expired. To force a leaf rotation, delete `gateway.pem` under the state dir and restart the unit, then reload `nginx`.
+
+**Cert prompts**: browsers still warn once per host until the hive's `trust-bundle.pem` is added to the browser/OS trust store (an anchor, not the leaf, is the thing to trust). Agent trust is wired separately (see the agent-trust work for `/run/hive-ca`).
+
+### Operator-provided cert (`tls.certDir`)
+
+For operators with a real CA cert (Let's Encrypt, corporate CA, etc.):
+
+```nix
+services.hyperhive.gateway = {
+  tls.certDir = "/var/lib/acme/example.com";  # nixpkgs security.acme output dir
+  # tls.certName = "cert.pem";   # default — matches security.acme layout
+  # tls.keyName  = "key.pem";    # default — matches security.acme layout
+};
+```
+
+nginx reads the directory directly and uses `cert.pem` + `key.pem` (override `tls.certName`/`tls.keyName` for different filenames). Both modes listen on `httpsPort` (default 443) and emit `https://` in `.well-known` responses.
+
+`tls.certDir` and `tls.acme.enable` set together is an assertion error.
+
+**Key file permissions**: nixpkgs's `security.acme` outputs private keys as `0640 root:acme` by default. nginx runs as the `nginx` user and cannot read a key with that ownership. Fix with:
+
+```nix
+security.acme.certs."example.com".group = "nginx";
+```
+
+or make the key world-readable (`0644`) if your threat model allows it. nginx errors out at startup on a key it can't read — the error is explicit in the journal, not a silent failure.
+
+### Fronting with an external TLS terminator
+
+There is no http-only mode (see [TLS modes](#tls-modes) above). Two paths
+for an operator who wants their own TLS terminator:
+
+- give the gateway the real cert via `tls.certDir` (or `tls.acme`) so it
+  serves proper TLS directly — no separate proxy needed; or
+- front it over a **unix socket** rather than a plain-http TCP port (the
+  intended direction for "bring your own proxy" — the gateway is not meant
+  to expose an unencrypted TCP upstream).
+
+Because of this, `.well-known/matrix/{client,server}` discovery responses
+always advertise `https` (see [Discovery flow](#discovery-flow-matrix) above).
+
+## Firewall posture (host-level)
+
+The gateway is unconditional — `services.hyperhive.gateway.enable` was
+removed, there is no gateway-off mode. nginx is always the sole
+external entry point and routes to agents over the UDS upstream
+described above (see [Per-agent unix-socket
+upstream](#per-agent-unix-socket-upstream)), so the per-agent web-port
+range `8100..8999` stays closed on the host firewall
+unconditionally — opening it would defeat the single-front-door story.
+The hashed TCP port (`lifecycle::agent_web_port`) still exists as a
+fallback bind for an agent whose `HIVE_WEB_SOCKET` env somehow ends up
+unset, but nothing opens a matching firewall hole for it and the
+gateway itself never proxies through it.
+
+`services.hyperhive.gateway.openFirewall = true` opens both `port` and
+`httpsPort` — both are always served, since the gateway always terminates
+TLS (see [TLS modes](#tls-modes) above).
+
+Every agent hashes into the same port range (no special case), so
+one range opening covers every container.
+
+The dashboard port (`cfg.dashboardPort`, default 7000) is *not*
+listed in either case — it binds `127.0.0.1` only, so a firewall
+hole would be a no-op. Remote dashboard access flows through the
+gateway. Operators who opt out of the gateway lose external
+dashboard reach by design — the surface is privileged (approve /
+deny / destroy) and must not be exposed without a real reverse
+proxy in front.
+
+## `HIVE_FORGE_URL`: agents reach the forge via the gateway by domain
+
+Agents poll `HIVE_FORGE_URL` for Forgejo notifications + run all
+`hive-forge` calls against it. Network isolation is always on (the
+shared-netns mode was removed), so agents run in a private netns and
+can never reach the host's loopback.
+`nix/host-modules/hive-c0re/environment.nix` sets `HIVE_FORGE_URL` to
+`http://<forge.domain>` (default `forge.<swarm-domain>` — a swarm runs
+one forge; `services.hyperhive.domain` is required). Agents
+get the bridge dnsmasq as their resolver, resolve the hostname →
+bridge IP, then reach nginx on port 80 (the bridge firewall opens
+80+443). nginx proxies to forgejo — the same path an operator browser
+takes, no raw port exposure needed.
+
+## hive-forge container shape
+
+Private Forgejo wrapped in a nixos-container (`hive-forge`, not
+`h-*` — keeps c0re's lifecycle scanner out of the picture; the
+operator manages it via the standard `nixos-container` CLI). The
+container also keeps hive-forge from fighting any `services.forgejo`
+the operator already runs on the host — separate systemd namespace,
+separate state dir, separate port unless the operator deliberately
+collides.
+
+The forge container shares the host network namespace
+(`privateNetwork = false`), so forgejo's listeners look like a
+host-side service — nixos-container is here for state + systemd-unit
+isolation, not network isolation. Note this is the FORGE container;
+agent containers are network-isolated and reach the forge through the
+gateway by `forge.<swarm-domain>` (see `HIVE_FORGE_URL` above), not via
+the host's loopback.
+
+State lives at `/var/lib/nixos-containers/hive-forge/var/lib/forgejo/`
+and survives container restart / host reboot. To wipe, destroy the
+container.
+
+### Network and port configuration
+
+```nix
+services.hyperhive.forge = {
+  httpPort   = 3000;   # default — HTTP listener; outside hyperhive's 7000/8100-8999 range
+  sshPort    = 2222;   # default — git-over-SSH; kept off 22 so it doesn't collide with the host openssh
+  openFirewall = false; # default — expose httpPort + sshPort to the host firewall
+};
+```
+
+`httpPort` (default **3000**) is the port Forgejo's HTTP server binds to.
+It sits outside hyperhive's reserved ranges (dashboard 7000,
+agents 8100–8999) so a default install has no port fights. Change it
+only if you already have another process bound to 3000.
+
+`sshPort` (default **2222**) is the port Forgejo's built-in SSH server
+uses for `git clone/push/pull` over SSH (`git@<domain>:owner/repo.git`
+via `-p 2222`). Port 22 is left alone on the host for openssh.
+
+`openFirewall` (default **false**) controls whether `httpPort` and
+`sshPort` are opened in the host firewall. Off by default (secure by
+default): agents reach Forgejo through the gateway (`forge.<swarm-domain>` on
+the bridge), not the raw port, so no firewall hole is needed. Flip to
+`true` when you need:
+- The operator's browser to reach `http://<host>:<httpPort>/` directly
+  (not behind the gateway).
+- External git clients that push/pull via SSH directly to the host.
+
+Forgejo served through the gateway (`forge.behindGateway = true`) does
+not need `openFirewall` — the gateway's own `openFirewall` option covers
+that path.
+
+### `rootUrl` override
+
+```nix
+services.hyperhive.forge.rootUrl = "https://forge.example.com/";
+```
+
+`rootUrl` (default **null**) overrides the Forgejo `ROOT_URL` that is
+auto-derived from `forge.domain` + gateway state. The auto-derivation
+covers most cases:
+
+| Shape | Auto-derived `ROOT_URL` |
+|---|---|
+| `behindGateway = true` | `https://<forge.domain>/` (port suffix omitted when `gateway.httpsPort == 443`) |
+| `behindGateway = false` | `http://<forge.domain>:<httpPort>/` |
+
+The gateway always terminates TLS, so the `behindGateway = true` case is
+always advertised over `https://`; only the direct (`behindGateway =
+false`) shape stays `http://`. Set `rootUrl` explicitly when
+`forge.domain` resolves differently from the public URL, or for a
+genuinely bespoke shape (e.g. an external reverse proxy on a different
+host/path). Must end with `/` (Forgejo requirement; an assertion
+enforces this).
+
+## Per-agent static frontend split
+
+When `services.hyperhive.frontend` is configured, hive-c0re injects
+`HIVE_AGENT_FRONTEND_DIR = "${cfg.frontend}/agent"` into its service
+environment. The nginx include generator (`gateway_nginx::write`) reads
+this variable and, when set, emits split location blocks per agent
+instead of the legacy single-proxy block.
+
+**Location priority for `/agent/<name>/...`:**
+
+```nginx
+# 1. Compiled assets — content-addressed nix store path, cache forever
+location ^~ /agent/<name>/static/ {
+    alias <frontend>/static/;
+    expires 1y;
+    add_header Cache-Control "public, immutable, max-age=31536000";
+}
+
+# 2. Static dist + proxy fallback for dynamic paths
+location /agent/<name>/ {
+    alias <frontend>/;
+    try_files $uri $uri.html $uri/index.html @<name>_dynamic;
+}
+
+# 3. Proxy catchall — API, events, icon, send, login, …
+location @<name>_dynamic {
+    proxy_pass <upstream>;
+    proxy_set_header X-Forwarded-Prefix /agent/<name>;
+    proxy_intercept_errors on;
+    error_page 502 503 504 = /__hive_agent_unreachable;
+    # … (full proxy header block)
+}
+```
+
+**`try_files` resolution** (nginx applies the `alias` mapping before
+checking file existence):
+
+| request | resolved | outcome |
+| --- | --- | --- |
+| `/agent/iris/` | `<frontend>/index.html` | main agent page |
+| `/agent/iris/stats` | `<frontend>/stats.html` | stats page |
+| `/agent/iris/screen` | `<frontend>/screen.html` | screen page |
+| `/agent/iris/static/app.js` | caught by `^~` block first | served with immutable cache |
+| `/agent/iris/api/state` | no file match → `@iris_dynamic` | proxied to agent daemon |
+| `/agent/iris/events/live` | no file match → `@iris_dynamic` | proxied (SSE) |
+
+Adding a new HTML page to the frontend dist (`dist/<page>.html`)
+automatically makes it reachable at `/agent/<name>/<page>` — no
+generator change needed.
+
+**Why `^~` for `/static/`**: the `^~` prefix gives this block higher
+priority than the plain prefix `location /agent/<name>/`, so compiled
+JS/CSS assets skip `try_files` entirely and get the immutable cache
+headers. Nix store paths are content-addressed — the hash changes on
+any content change — so `max-age=31536000` is safe.
+
+**Why the nix store path resolves**: `HIVE_AGENT_FRONTEND_DIR` is a nix
+store path baked in at hive-c0re build time, and c0re (writing
+`agents.conf`) and nginx (serving files from it) are on the same machine,
+so they see the same store.
+
+**Graceful degradation**: if `HIVE_AGENT_FRONTEND_DIR` is empty or
+unset (e.g. a build that predates `cfg.frontend`), each agent gets the
+legacy single-proxy block and all traffic is forwarded to the agent
+daemon as before.
+
+**`extraFiles`**: per-agent `hyperhive.frontend.extraFiles` are in
+`mergedDist`, not in the base `cfg.frontend` dist. They are not under
+the nix-store `alias` path, so requests for them fall through
+`try_files` to `@<name>_dynamic` and are served by the agent daemon
+as before.
+
+## Per-agent error pages
+
+`/agent/<name>/` requests hit two failure modes; both get static
+HTML pages instead of nginx's default error chrome:
+
+- **Agent not found** (`/agent/<unknown>/...`) — name isn't in
+  `agentPortsTable`. nginx's prefix match falls back to the bare
+  `/agent/` catch-all, which `return 404`s and `error_page 404` rewrites
+  to `/__hive_agent_not_found` → serves `not-found.html` with a link
+  back to the dashboard.
+
+- **Agent unreachable** (`502 / 503 / 504` from `proxy_pass`) — the
+  per-agent harness isn't responding (container restarting, crash
+  recovery, etc.). `proxy_intercept_errors on` + `error_page 502 503
+  504 = /__hive_agent_unreachable` rewrites to `unreachable.html`.
+
+Both pages are built at deploy time via `pkgs.runCommand` (one nix
+derivation `hyperhive-agent-error-pages` with `not-found.html` +
+`unreachable.html` inside) and served via two `internal` nginx
+locations with `alias` to the exact file. `internal` keeps the
+files from being directly request-able by operators — only nginx's
+own error-handling can reach them.
+
+Page styling: minimal inline CSS matching the dashboard's catppuccin
+palette (`#1e1e2e` bg, `#cdd6f4` text, `#cba6f7` heading). No
+dependencies on the frontend dist — these pages render even when
+hive-c0re itself is down.
+
+Scope is intentionally narrow: a route earns a custom page when the
+default status code would point at the wrong component. The per-agent
+routes qualify (a 502 there means the harness is restarting, not that
+the gateway is broken), and so does `auth.<swarm>` — a dead authelia
+upstream almost always means the user store was never bootstrapped, and
+a bare 502 blames the proxy, which is the one part that is working.
+
+Forge / matrix / fluffychat still get nginx defaults: their upstreams
+being down means what the status code says, so a themed page would add
+styling and no information.
+
+## HTTP Basic auth
+
+`services.hyperhive.gateway.auth.enable = true` gates every request to
+the main vhost (`_`) behind HTTP Basic auth. nginx's built-in `auth_basic`
+module validates credentials; no extra service or host-side daemon is
+required.
+
+**Setup:**
+
+```nix
+services.hyperhive.gateway.auth = {
+  enable = true;
+  # realm = "hyperhive";  # optional, default shown
+};
+```
+
+The credential store lives at the fixed path
+`/var/lib/hive-gateway/conf/gateway.htpasswd` on the host. A tmpfiles
+rule pre-creates the file on first boot; no manual path configuration
+is required. nginx reads it at that path directly.
+
+Manage users with `hivectl gateway`. `hivectl` sends the request over the
+host admin socket and the `hive-c0re` daemon performs the write at its
+canonical path — no path is exposed to the CLI:
+
+```sh
+# Add or update a user (prompted for password):
+hivectl gateway create-user alice --password-stdin
+
+# Add with inline password (visible in shell history — avoid for sensitive creds):
+hivectl gateway create-user bob --password hunter2
+
+# Remove a user:
+hivectl gateway delete-user bob
+
+# List current usernames:
+hivectl gateway list-users
+```
+
+The daemon hashes passwords with BCrypt (cost 12) and writes
+`$2y$`-prefixed hashes that nginx accepts natively. No external
+`htpasswd` binary is required.
+
+**What is not gated:** per-agent UI routes emitted into `agents.conf`
+(served under `/agent/<name>/`) inherit no auth from `/` — nginx
+applies `auth_basic` per-location. Full per-agent coverage is a
+follow-up.
+
+**Realm:** the `WWW-Authenticate: Basic realm="..."` string browsers
+display in the credential dialog. Defaults to `"hyperhive"`. Must not
+contain `"` or `$`.
+
+**Custom 401 page:** when credentials are absent or wrong, nginx serves
+a Catppuccin-styled `unauthorized.html` page (built into the same Nix
+derivation as the agent error pages) that tells the operator which
+`hivectl` command to run to create a user. The response status is still
+`401` (`error_page 401 =401 /__hive_auth_unauthorized`) so browsers
+present the login dialog on the first visit — users who dismiss the
+dialog see the human-readable hint. The internal exact-match location
+(`= /__hive_auth_unauthorized`) beats `location /` in nginx's prefix
+ordering, preventing the subrequest from looping back through
+`auth_basic`.
+
+## Security headers
+
+The following headers are emitted at server scope on every gateway
+vhost (`_`, `forge.<swarm-domain>`, `chat.<swarm-domain>`):
+
+| Header | Value |
+|--------|-------|
+| `X-Frame-Options` | `SAMEORIGIN` |
+| `X-Content-Type-Options` | `nosniff` |
+| `Referrer-Policy` | `strict-origin-when-cross-origin` |
+
+nginx's `add_header` inheritance rule: a `location` block that sets its
+own `add_header` does **not** inherit server-scope headers. API locations
+that carry their own CORS headers (e.g. `/.well-known/matrix/client`,
+`/_matrix/`) are therefore unaffected. HTML-serving and proxy locations
+with no `add_header` of their own pick the security headers up
+automatically.
+
+### HSTS (`gateway.hsts`)
+
+HSTS is **opt-in** and disabled by default:
+
+```nix
+services.hyperhive.gateway.hsts = {
+  enable = true;          # default: false
+  maxAge = 31536000;      # default: 1 year (required for preload list)
+  includeSubDomains = true;  # default: true
+};
+```
+
+When enabled, a `Strict-Transport-Security: max-age=...[; includeSubDomains]`
+header is added alongside the other security headers.
+
+**Opt-in rationale**: HSTS pins HTTPS in the browser's preload cache;
+enabling it on a deployment that later loses TLS locks browsers out
+until `max-age` expires. Only enable when TLS is permanent.
+
+Since the gateway always terminates TLS (see [TLS modes](#tls-modes)
+above), an enabled HSTS header is always served over https — there is no
+TLS-less mode that could violate it.
+
+## Dialing another vhost by name (`verifiedProxyTo`)
+
+`vhost-lib.nix`'s `verifiedProxyTo` builds the `proxy_ssl_*` /
+`proxy_set_header` block a module uses to dial another service on this
+same gateway BY NAME over https, verified. One definition rather than a
+copy per module: nginx verifies nothing by default
+(`proxy_ssl_verify` is off), so a `proxy_pass https://…` without these
+lines is encrypted and unauthenticated. That failure is invisible — it
+works, and keeps working, against any certificate at all.
+
+Every line earns its place, each confirmed against a real nginx with
+the opposite arm run as a control:
+
+- `verify` + `depth` — the chain is leaf -> intermediate -> root.
+- `trusted_cert` — the bundle; nginx reads ALL certs in the file, which
+  the bundle's own doc warns is not true of every consumer.
+- `ssl_name` — checks the HOSTNAME too. Without it a chain-only check
+  accepts any certificate this CA ever signed, and for an internal CA
+  that is every service on the hive.
+- `server_name on` — sends SNI, or the far end cannot pick a cert.
+
+**⚠️ Session-cache footgun**: `proxy_ssl_session_reuse` is left at its
+default (on), deliberately — this is used on per-request auth
+subrequests, so the handshake it avoids is paid on every request. Worth
+knowing when testing though: the session cache is keyed by upstream
+address and NOT by trust config, so two locations pointing at one
+upstream with different trust do not verify independently.
+
+**⚠️ Host-header clobber footgun**: `verifiedProxyTo` also pins `Host`
+(and reinstates the rest of nginx's `recommendedProxySettings` header
+set) to the target `name` rather than leaving it to be filled in later.
+`name` here resolves back to THIS gateway — every consumer dials another
+vhost on the same nginx, not a separate host — and nginx picks the vhost
+to answer an HTTPS request from the `Host` header, not from the TLS SNI
+that `proxy_ssl_name` sends. `recommendedProxySettings`'s own `Host
+$host` (the CALLER's host, not the target) is textually appended by
+nixpkgs AFTER a location's `extraConfig` — so it always wins over a
+`proxy_set_header Host` written in the location body, and the subrequest
+loops back into the ORIGINAL vhost instead of reaching the target,
+recursing on its own `auth_request` until nginx's subrequest-depth limit
+turns it into a plain 500. Every call site sets `recommendedProxySettings
+= false` on the location for exactly this reason — nixpkgs' version
+would still clobber this one.
+

docs/networking/network.md

@@ -0,0 +1,277 @@
+# hive-network
+
+Host-side bridge + per-agent private-netns isolation — always on
+whenever hyperhive is enabled. Configured via
+`services.hyperhive.network.*`.
+
+> Isolation is the only mode — there is no shared-netns fallback. The
+> former `services.hyperhive.network.enable`,
+> `services.hyperhive.network.isolateContainers` and
+> `services.hyperhive.network.upstreamDns` options were removed; a
+> config that still sets one fails eval with a removal message.
+
+## Network map
+
+One picture of the whole hive. There are two planes: **infra
+containers share the host netns** and bind host ports directly;
+**compute containers (agents + CI) each get a private netns** behind
+the bridge. The unix-socket control plane rides the VFS and is
+untouched by any of it.
+
+```
+                        internet
+                           │  uplink NIC — NAT MASQUERADE for the
+                           │  bridge subnet (10.42.0.0/24 default)
+┌──────────────────────────┴─────────────────────────────────────────┐
+│ host netns — the host itself plus gateway / forge / matrix         │
+│                                                                    │
+│   nginx :80/:443                                  [hive-gateway]   │
+│   dnsmasq 10.42.0.1:53 (DNS) + :67 (DHCP)         [hive-gateway]   │
+│   forgejo :3000 http, :2222 git-ssh               [hive-forge]     │
+│   tuwunel :8008 client API                        [hive-matrix]    │
+│   hive-c0re dashboard 127.0.0.1:7000              (host service)   │
+│   wg-hive :51820/udp — swarm mesh, when enabled   (host iface)     │
+│                                                                    │
+│                  hive-br0   10.42.0.1/24                           │
+│              ┌──────────┼──────────────┐                           │
+└──────────────┼──────────┼──────────────┼───────────────────────────┘
+          vb-h-<a>    vb-h-<b>      vb-hive-ci      veth pairs
+               │          │              │
+          ┌────┴────┐ ┌───┴─────┐ ┌──────┴──┐   one private netns
+          │ agent a │ │ agent b │ │ hive-ci │   each; eth0 leases
+          │  eth0   │ │  eth0   │ │  eth0   │   from the DHCP pool
+          └─────────┘ └─────────┘ └─────────┘
+```
+
+| container      | netns                   | IPv4                 | listens / reached via                                                                        |
+| -------------- | ----------------------- | -------------------- | -------------------------------------------------------------------------------------------- |
+| `hive-gateway` | host (shared)           | host addresses       | nginx `:80`/`:443` (every vhost); dnsmasq `bridgeIp:53` + DHCP `:67` on the bridge            |
+| `hive-forge`   | host (shared)           | host addresses       | forgejo `:3000` http, `:2222` git-ssh; fronted by the `forge.<swarm-domain>` vhost            |
+| `hive-matrix`  | host (shared)           | host addresses       | tuwunel `:8008` (+ optional federation port); fronted by the matrix vhost                     |
+| `hive-ci`      | private, veth on bridge | DHCP pool            | outbound only (runner → forge); no inbound surface                                            |
+| `h-<agent>`    | private, veth on bridge | DHCP pool            | web UI via UDS `/run/hive-agent/<name>` → nginx sub-path; in-container UI port hashed 8100–8999 |
+
+The flows, end to end:
+
+- **DHCP** — agent `dhcpcd` broadcasts on `eth0` → veth → bridge →
+  host firewall (udp 67 hole) → dnsmasq pool → lease + router option.
+- **DNS** — agents and the service containers query `bridgeIp:53`; hive
+  zones are answered authoritatively with the bridge IP, everything else
+  forwards to the host's resolvers (see *Resolver behaviour* below). Each
+  container points its own `resolv.conf` there, and one that instead
+  inherits the host's resolves no swarm name at all — those records exist
+  only on the bridge.
+- **HTTP** — `forge.` and `chat.` (under `swarm.domain`) plus the hive's
+  own dashboard name resolve to the bridge IP, land on nginx
+  `:80`/`:443`, and proxy to forgejo
+  `:3000`, tuwunel `:8008`, hive-c0re `127.0.0.1:7000`, or a per-agent
+  UI unix socket.
+- **Internet egress** — agent default route points at the bridge IP;
+  the host forwards + masquerades out its uplink.
+- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
+  and reach each other's gateway/forge across it
+  ([`docs/swarm/`](../swarm/README.md)).
+- **Control plane (no network)** — per-agent broker socket
+  `/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
+  operator admin `/run/hyperhive/host.sock`, and the per-agent UI
+  sockets under `/run/hive-agent/` are unix domain sockets
+  bind-mounted through the VFS; private netns does not affect them.
+
+## Container shape (where dnsmasq lives)
+
+Co-located in the existing `hive-gateway` container — single
+front-door for both DNS and HTTP, saves a sibling container, single
+systemd-unit / state surface to monitor. The gateway shares host
+netns (`privateNetwork = false`) so dnsmasq's `bind-interfaces`
+listener on `bridgeIp` is on the host's bridge interface.
+
+## Configuration
+
+```nix
+{
+  services.hyperhive = {
+    enable = true;
+    hiveName = "pr1ma";
+    swarm.domain = "darkest.space";
+    swarm.hives.pr1ma = { };          # -> domain = pr1ma.darkest.space
+    # network.bridgeIp = "10.42.0.1"; # default
+  };
+}
+```
+
+Requires `services.hyperhive.domain` to be set — the dnsmasq resolver
+is authoritative for `<hive-domain>` and its sub-domains. You do not
+write it: it is read from this hive's entry in the swarm directory
+(`docs/swarm/README.md` § Hive identity config).
+
+## Bridge addressing
+
+Default subnet is `10.42.0.0/24`, host-side gateway at `10.42.0.1`.
+RFC 1918 space, unlikely to clash with operator's existing setup;
+override `bridgeIp` + `bridgePrefixLength` if a different range is
+already in use. `/24` gives 254 usable per-agent addresses — enough
+for any single-host hive; bigger swarms or tighter addressing
+schemes pick their own.
+
+## Resolver behaviour
+
+dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
+plus whatever swarm-service names this host contributes via
+`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
+(matrix) when this host runs those services, and `auth.<swarm-domain>`
+when it runs authelia — answering each with the bridge IP (where nginx
+is reachable). Note forge and matrix are swarm-domain names, not
+sub-domains of `<hive-domain>`: a swarm runs one forge and one
+homeserver, so their names belong to the swarm rather than to whichever
+hive happens to host them. Everything else is forwarded to the host's
+own resolvers: dnsmasq runs on the host and reads the host's
+`/etc/resolv.conf` directly. Containers don't need to know the
+upstream — they query the bridge IP and dnsmasq does the right thing
+per-name.
+
+There is deliberately no fallback `server=`: dnsmasq queries all known
+upstreams in parallel, so a hardcoded public resolver would take a share
+of normal traffic, not just cover the gap.
+
+dnsmasq runs on the host and reads the host's `/etc/resolv.conf`
+directly, so a network change (new router, new lease, laptop moving
+networks) reaches it the moment openresolv rewrites the file. There is
+nothing to synchronise and no unit watching for it.
+
+`bind-interfaces` + `interface = [ bridgeName "lo" ]` means the
+listener only accepts queries from the bridge interface (plus lo for
+container health-checks). External hosts can't reach it — no
+DNS-amplification surface even when the operator opens port 80 for
+gateway HTTP.
+
+`resolveLocalQueries = false` keeps dnsmasq out of the host's own
+resolution stack — the host's resolver (systemd-resolved, plain
+glibc nss, dnscrypt-proxy, etc.) keeps doing whatever the operator
+configured. The hive resolver is purely for inbound queries from
+agent containers.
+
+## Firewall posture
+
+`networking.firewall.interfaces.<bridge>.allowedUDPPorts = [ 53 67 ]`
+`networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 53 80 443 ]`
+
+- Port 53 opens the resolver on the bridge interface only. Other
+  interfaces stay closed. The hive resolver isn't an external-facing
+  service.
+- Port 67 (UDP) admits DHCP requests to the dnsmasq pool. dnsmasq
+  receives DHCP via a regular UDP socket (it does not use a
+  netfilter-bypassing raw socket), so the hole is mandatory — without
+  it containers never get a lease and fall back to 169.254.x.x.
+- Ports 80 and 443 let isolated agents reach nginx (gateway
+  container, shared host netns) for the forge sub-domain, per-agent
+  UI proxies, and any other HTTP services.
+
+The **host** firewall is the only firewall. The shared-netns infra
+containers (gateway, forge, matrix) set
+`networking.firewall.enable = false`: a NixOS firewall inside a
+shared-netns container runs against the *host* ruleset — at container
+boot its `firewall-start` flushes the `nixos-fw` chains, rebuilds them
+from the container's (empty) port list, and deletes the host's
+`nixos-nat-*` chains without recreating them, silently wiping the
+bridge holes above plus the agents' NAT. Private-netns containers
+(agents, hive-ci) may keep their own firewall — it is scoped to their
+namespace.
+
+### Reaching host services (`exposeHostPorts`)
+
+By default agents can only reach the host on 80/443 (+53 DNS), so a
+host-side service on another port — e.g. a dev OTLP collector you want
+agents to reach directly — is unreachable. (hyperhive's own telemetry
+needs none of this: `otel.enable` opens its collector's port itself, and
+`otel.endpoint` is the *upstream*, which no agent ever dials. See
+`docs/observability.md`.)
+
+`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
+listed TCP port `P` on the bridge-interface `allowedTCPPorts`, so an
+agent can connect to `<bridgeIp>:P` (point the collector endpoint at
+`http://<bridgeIp>:4318`, default `http://10.42.0.1:4318`).
+
+This is **firewall-only**: the host service must bind an address
+reachable from the bridge — `0.0.0.0` or the bridge IP — not loopback
+only. The bridge→`127.0.0.0/8` DROP rule (below) is unchanged, so a
+service bound to `127.0.0.1` only stays unreachable; rebind it to
+`0.0.0.0`.
+
+The port is reachable by **every** agent on the bridge subnet (like
+DNS/gateway), so only expose services safe for any agent to reach.
+
+## Container isolation
+
+Each agent container runs in a private network namespace with a dedicated
+veth pair attached to the bridge. The following table summarises what
+the nix side sets up unconditionally:
+
+| effect         | mechanism                                                                                                                             |
+| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| IP forwarding  | `boot.kernel.sysctl."net.ipv4.ip_forward" = 1`                                                                                        |
+| Internet NAT   | `networking.nat { enable = true; internalInterfaces = [ bridgeName ]; }` — MASQUERADE on packets leaving via any external NIC         |
+| Loopback DROP  | `networking.firewall.extraInputRules` — drops bridge-subnet → `127.0.0.0/8` traffic; defence-in-depth against routing table leaks     |
+| Gateway access | `networking.firewall.interfaces.<bridge>.allowedTCPPorts = [ 80 443 ]` — lets isolated agents (private netns, veth on bridge) reach nginx on the host  |
+| c0re signal    | `HIVE_NETWORK_BRIDGE`, `HIVE_NETWORK_SUBNET` in `systemd.services.hive-c0re.environment` — both **required**; `hive-c0re` refuses to start without them |
+
+`HIVE_NETWORK_SUBNET` is the host-side bridge IP + prefix (e.g.
+`10.42.0.1/24`), **not** the canonical network address. The Rust side
+must normalise (bitwise-AND with mask) before subnet membership checks or
+address arithmetic.
+
+### What the Rust side does
+
+`hive-c0re` reads `HIVE_NETWORK_BRIDGE` + `HIVE_NETWORK_SUBNET` and passes
+`PRIVATE_NETWORK=1`, `LOCAL_ADDRESS=` (empty), `HOST_ADDRESS=<bridge-ip>`,
+and `HOST_BRIDGE=<bridgeName>` via `lifecycle::set_nspawn_flags` when
+creating or updating containers. Both variables are validated **once at
+daemon startup**, not per container: they are process-global, so a
+missing or malformed value is a misconfigured daemon rather than one bad
+container, and failing at boot gives a single diagnostic instead of one
+per agent. There is no non-isolated mode to fall back to. `LOCAL_ADDRESS` is left empty so the
+container's dhcpcd acquires an address from the bridge dnsmasq pool
+(`networking.useDHCP = true` in `nix/agent-modules/network.nix`). This applies uniformly
+to all containers — agents and service containers alike.
+
+`HOST_ADDRESS` is the bridge gateway IP (the address part of
+`HIVE_NETWORK_SUBNET`, via `lifecycle::bridge_gateway_ip` — taken verbatim
+so a non-`.1` operator override still resolves to wherever the bridge
+actually lives). It is **load-bearing**: nixos-container's container-side
+network setup only installs a default route (`ip route add default via
+$HOST_ADDRESS`) when `HOST_ADDRESS` is non-empty. In bridge mode the
+host-side address/route setup is skipped, so writing it only affects the
+container's default route — without it the container comes up with an IP
+but no path off the bridge subnet (no internet, no `api.anthropic.com`).
+
+### How the isolated container gets its resolver
+
+nixos-container copies the **host's** `/etc/resolv.conf` into the container
+at every start. The host resolver (e.g. `127.0.0.53` from systemd-resolved,
+or a LAN router) is unreachable from a private netns and isn't
+authoritative for the hive's own zones, so it is replaced with the
+bridge dnsmasq at boot. Because the copy happens on every start, a
+declarative `environment.etc."resolv.conf"` would be clobbered — so the
+wiring is runtime:
+
+- `hive-priv` drops a marker file (`/etc/hyperhive-bridge-dns`, carrying the
+  gateway IP) into each container's `/etc`.
+- the `hyperhive-isolated-dns` oneshot (`nix/agent-modules/network.nix`), gated on that
+  marker, rewrites `/etc/resolv.conf` to `nameserver <gateway-ip>` at boot.
+  It is ordered `before` the harness (`hive-ag3nt`), the matrix daemon, and
+  `tea-login` so the resolver is correct before the first DNS lookup.
+
+**Why isolation is safe**: hive-c0re's control-plane sockets are unix
+domain sockets bind-mounted into containers, not network listeners — see
+the *Control plane (no network)* bullet under [Network
+map](#network-map) above. `PRIVATE_NETWORK=1` has no effect on a path
+that never touches the network stack.
+
+The nix side also enables IP forwarding + NAT (agents reach the internet
+through the host) and drops bridge-subnet → loopback traffic (defence-in-depth
+against a compromised agent reaching the c0re dashboard HTTP at
+`127.0.0.1`). Agents have no legitimate reason to reach the dashboard over
+loopback — the hive-c0re admin socket is a UDS, not TCP.
+
+## Cross-references
+
+- `docs/gateway.md` — vhost map + the gateway's other duties

docs/networking/snapshot-store.md

@@ -0,0 +1,224 @@
+# Snapshot store
+
+The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
+over the WireGuard mesh; a destination hive later pulls one back to
+complete a migration.
+
+Two things it is not, both worth stating because both are easy to
+assume:
+
+- **It is not the swarm controller**, and does not depend on one. It is
+  a NixOS host role: a btrfs subvolume tree, a socket-activated
+  receiver, and the `wg-hive` interface the swarm module already brings
+  up. That is why it can be deployed before any controller exists.
+- **It is not a backup product.** It happens to hold the data a backup
+  would hold, and it should be operated accordingly (see
+  [Operating it](#operating-it)) --- but nothing in it does scheduling,
+  verification, or restore orchestration.
+
+## Enabling it
+
+```nix
+services.hyperhive.snapshotStore = {
+  enable = true;
+  path = "/var/lib/hyperhive-snapshots";  # must be on btrfs
+  port = 51821;
+};
+
+# The mesh is a hard requirement, and is asserted:
+services.hyperhive.swarm.wireguard = {
+  enable = true;
+  address = "10.100.0.9/24";
+  privateKeyFile = "/etc/wireguard/hive.key";
+};
+```
+
+The store host is a swarm member like any other: it gets an entry in
+`services.hyperhive.swarm.hives`, the same directory every host holds. See
+[swarm/](../swarm/README.md) for the mesh itself.
+
+Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
+`c0re.enable` --- a store host runs no hive and would otherwise get no
+`wg-hive` interface at all.
+
+## Pointing a hive at it
+
+The block above configures the host that *receives*. Every hive that
+*pushes* separately needs to be told where the store is:
+
+```nix
+services.hyperhive.swarm.snapshotStore = {
+  address = "10.100.0.9";  # the store's mesh address, no prefix
+  port = 51821;            # optional; must match the receiver's port
+};
+```
+
+Two deliberate asymmetries in that pair, both easy to misread as
+inconsistency:
+
+- **`address` has no default.** It is a deployment fact a pushing hive
+  cannot derive, and a wrong guess means streaming an agent's state at
+  whatever happens to answer. Unset, a push fails naming this option.
+- **`port` does default** (`51821`), because it is a convention both
+  ends read from the same option docs --- a default there is
+  coordination, not a guess.
+
+Note the option lives under `swarm.*` while the receiving host's lives
+under `services.hyperhive.snapshotStore`. That is the distinction the
+two namespaces carry throughout: `swarm.*` describes *the swarm* as seen
+from this host, and a bare `services.hyperhive.<service>` describes *a
+role this host performs*. A store host sets both --- one to run the
+receiver, one only if it also runs a hive that pushes.
+
+With it set, `hivectl agent <name> subvol snapshot push <label>
+[--parent <label>]` streams a snapshot straight into the store. There is
+no destination argument, because a swarm has exactly one store (see
+[One subvolume per agent, not per hive](#one-subvolume-per-agent-not-per-hive)),
+and no credential argument, because the mesh is the authentication.
+
+## The mesh is the authentication
+
+There are no certificates here, and no key material of its own. That is
+deliberate rather than an omission.
+
+WireGuard's cryptokey routing already binds a peer's source address to
+its public key: the swarm module configures each peer with
+`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
+that address provably came from the holder of that private key. A
+packet that reaches the receiver has therefore already been
+authenticated by the kernel.
+
+Layering TLS client certs on top would authenticate *the same fact* a
+second time, and add a credential with an expiry --- a migration that
+fails because a renewal quietly didn't happen, discovered on the day
+you need to move an agent.
+
+## One subvolume per agent, not per hive
+
+The destination is keyed by **agent**.
+
+This is not cosmetic. After a migration, an agent's next incremental
+send arrives from a *different* hive than the previous one. Keying by
+hive would split that agent's snapshot chain across two directories,
+and `btrfs send -p` would fail to find its parent --- breaking exactly
+the case the store exists to serve.
+
+## What the sender can and cannot choose
+
+A `btrfs send` stream carries no notion of *which agent* it belongs to,
+and the subvolume name inside it is chosen by the sender. So the
+protocol is one `agent <name>` header line, then the raw stream.
+
+The rule that matters:
+
+> **The receiver owns the destination root. The sender-supplied name is
+> validated, never used as a path.**
+
+Validation is a whitelist --- `[A-Za-z0-9_-]+` and nothing else. No
+slash and no dot means neither directory traversal nor an absolute path
+can survive it. It is deliberately a whitelist and not a list of
+forbidden characters: a blocklist only ever excludes the attacks
+somebody already thought of.
+
+## Reachability
+
+The receiver is socket-activated, and the socket binds **this host's
+mesh address**, never a wildcard. Both the mesh being enabled and the
+address being set are assertions, not documentation --- bound to
+`0.0.0.0` this socket is an unauthenticated remote write into agent
+state.
+
+Binding is not sufficient on its own. NixOS's firewall is default-deny
+and filters in netfilter, *before* a packet reaches a bound socket, so
+the port is opened explicitly --- and scoped to the mesh interface:
+
+```nix
+networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
+```
+
+A host-wide `allowedTCPPorts` would open the port on every interface
+including a public NIC, leaving only the socket's bind address between
+the internet and a root `btrfs receive`.
+
+## Operating it
+
+### Confinement is the deployment's job
+
+`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
+The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
+narrow `ReadWritePaths` --- but those are **defence in depth, not a
+boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
+undo the namespace they set up.
+
+The boundary is the machine. The intended deployments are:
+
+- **a swarm**: the store is its own small VM. The machine is the
+  boundary, which is stronger than anything the unit could assert about
+  itself.
+- **all-in-one / local**: the store runs as a container on the c0re
+  host.
+
+The second is worth keeping deliberately, and not only for
+convenience: it means the confined path is exercised by every local
+deployment. The usual failure mode for an isolated variant is that
+nobody runs it day to day, so it rots and is discovered broken in
+production.
+
+⚠️ **The assumption to keep true over time:** the store host runs
+nothing else. That is true on day one and quietly false the day someone
+notices the box has spare disk. Nothing in the config objects when it
+stops being true.
+
+### It holds every agent's state from every hive
+
+Which makes it the highest-value target in the swarm by a wide margin,
+and means it should get the treatment a backup host gets --- restricted
+access, and a decision (rather than an omission) on encryption at rest.
+
+The trap is the label: this box holds backup-grade data while not being
+called a backup, so it can end up with backup-grade *exposure* and
+non-backup-grade *controls*. Nobody puts a migration staging area on
+the access-review list.
+
+### What a snapshot contains
+
+The snapshot covers an agent's **state subvolume**, which is the parent
+of `state/`, `claude/` and `harness/` (see
+[`docs/persistence.md`'s btrfs subvolume
+section](../agent-lifecycle/persistence.md#btrfs-subvolumes-for-varlibhyperhiveagentsname)
+for how and when that subvolume is created). Consequences:
+
+- The Claude session (`claude/`) travels, so a restored agent keeps its
+  live `--continue` session rather than needing to log in again.
+- `harness/` travels too, including `harness/bash-tasks/`. Task output
+  is part of an agent's working continuity, so this is wanted --- but it
+  means anything that has ever leaked into a task's captured output is
+  in the retained snapshots as well.
+
+It does **not** cover the agent's applied config (`/applied/<name>/`) or
+its topology entry, both of which live outside the subvolume. A restore
+therefore yields an agent's memory without its definition; closing that
+gap is tracked separately.
+
+### Retention
+
+Retention lives on the *sending* side (last-N by count, swept
+periodically), not here. Count rather than age is deliberate: a count
+is bounded by construction, whereas an age policy silently scales disk
+usage with how hot a hive runs.
+
+Per-agent or per-hive `btrfs qgroup` quotas are not configured yet.
+Without them one runaway hive can fill the store and take out every
+other hive's snapshots.
+
+## Not built yet
+
+**The pull side.** Push is safe with minimal authorisation because a
+hive can only ever write to a chain it owns. Pull is the direction that
+needs a policy: unrestricted, any compromised hive could read every
+agent's state from every other hive. It needs a notion of which hive
+currently owns which agent, and that ownership record lands with the
+swarm controller work.
+
+With a single hive the question is trivial --- the only peer owns
+everything it sends --- which is why the receive half ships first.

docs/process/conventions.md

@@ -0,0 +1,563 @@
+# Conventions
+
+Code-style and process expectations across the workspace. Most of these
+exist because something already went wrong without them.
+
+## Naming
+
+- Containers are length-bounded by `nixos-container` (≤ 11 chars).
+- Sub-agents are `h-<name>` with `<name>` ≤ 9 chars.
+- One agent is the bootstrap/root container, with a fixed name (`ruth` today).
+- `MAX_AGENT_NAME` in `hive-c0re/src/lifecycle/mod.rs` enforces the cap.
+- Per-agent web UI port = `WEB_PORT_BASE + FNV1a(name) % WEB_PORT_RANGE`
+  (8100..8999) for every agent; dashboard
+  `cfg.dashboardPort` (default 7000).
+
+## Hive identity (label + domain + display names)
+
+Four env vars cover the identity surface, read by
+`hive-agent/src/identity.rs`:
+
+- `HIVE_LABEL` — short, hive-local agent label (`iris`,
+  `damocles`). `label()` returns it; falls back to empty string if
+  the env var is missing so downstream callers can decide how to
+  surface "unknown agent" rather than getting a panic from this
+  module.
+- `HYPERHIVE_HIVE_DOMAIN` — the hive's canonical DNS domain (e.g.
+  `darkest.space`), set by `nix/host-modules/hive-c0re/environment.nix`
+  from `services.hyperhive.domain`. When configured, `qualified_label()`
+  returns `${label}@${domain}` (e.g. `iris@darkest.space`); when
+  unset (single-hive deployments, dev/test) it degrades to just
+  the short label so existing callers see no change. The
+  qualified form surfaces in the per-agent web UI title, the
+  system-prompt template, and `/api/state.qualified_label`.
+- `HYPERHIVE_HIVE_NAME` — human display name of this hive
+  (`pr1ma`). Read by `hive_name()`; `None` when unset.
+- `HYPERHIVE_SWARM_NAME` — human display name of the wider swarm
+  this hive belongs to (`constellat1on`). Read by `swarm_name()`;
+  federated hives at different DNS domains can share a swarm
+  name.
+
+`hive_name` + `swarm_name` are **distinct** from
+`HYPERHIVE_HIVE_DOMAIN`: the domain may carry the hive name as
+its leftmost label by convention, but the convention isn't
+machine-readable, and federated hives at different DNS domains
+can share a swarm name. Humans want both: the address
+(`@darkest.space`) AND the prose name (`pr1ma`). Matrix MXIDs
+still use the domain-based convention untouched.
+
+`qualify(label)` is the same shape as `qualified_label()` but
+applies to an arbitrary label the caller already has (e.g. a peer
+name from the broker); it's the right surface when rendering a
+peer's name when the caller knows it's hive-local.
+
+## Identity = socket
+
+There are no auth tokens on the per-agent unix sockets. The socket
+*path* identifies the principal; perms come from "who has the
+bind-mount." A sub-agent only sees its own `/run/hive/mcp.sock`;
+hive-c0re owns the host admin socket.
+
+### Wake injection
+
+`AgentRequest::Wake { from, body }` (and the manager-flavour mirror)
+is the wake-event-injection surface. Recipient is implicit — the
+agent the socket belongs to — and `from` is caller-chosen so the
+wake prompt can label the source verbatim (`"matrix: new message in
+#general"`, `"forge: PR #42 opened"`, etc.). Typical caller: an
+in-container background task (the matrix daemon, a scraper, the
+forge-notify webhook subscriber) that needs to signal "external work
+has arrived" without going through the broker as a peer agent.
+
+Identity = socket means anything that can connect to
+`/run/hive/mcp.sock` is implicitly trusted to inject wakes. That's
+fine: the bind-mount only exposes the socket inside the agent's own
+container, so the trust boundary is the container's process
+namespace, not the wire surface.
+
+## Recipient sentinels
+
+A few recipient names are reserved by the broker and have special
+meaning that ordinary agent labels can never collide with — agent
+name validation rejects any character outside `[a-z0-9_-]`, so the
+angle-bracket and asterisk shapes below are structurally safe.
+
+- `*` — broadcast: deliver to every running agent except the sender
+  (`socket_server::handle_send` fans out via `Coordinator::broadcast_send`).
+- `operator` — the human at the dashboard. Messages accumulate in the
+  inbox view; no agent ever `recv`'s them.
+- `<parent>` — the sender's parent per `topology.json`. Rewritten at
+  send time by `topology::resolve_recipient`: looks up
+  `parent_of(sender)` and falls back to `operator` when the sender is
+  a root agent (or absent from topology entirely). Lets agents address
+  their parent without learning the label, so runtime reparenting
+  propagates with zero agent-side restart.
+- `<children>` — fan-out to every direct descendant of the sender per
+  `topology.json`. Resolved in `socket_server::handle_send` via
+  `topology::children_of(sender)`: one message is delivered to each
+  child, bypassing the allow-list check (structural fan-out targets are
+  never user-listed peers). No-op for leaf agents (returns `Ok` when the
+  child set is empty). Lets a sub-manager nudge its subtree without
+  enumerating labels.
+
+When a `<children>` or `<parent>` send resolves to real recipients, the
+broker stores the *resolved* label(s) as the message recipient(s) — the
+dashboard and recv side see the real routes. The sentinels are purely
+send-time addressing conveniences.
+
+## Wire protocol
+
+JSON line-delimited over unix sockets in both directions (host admin
+/ manager / agent). SSE streams (`/dashboard/stream` on hive-c0re,
+`/events/stream` on the per-agent web UIs) are `text/event-stream`;
+each frame carries a `seq` field for the snapshot-dedupe dance
+(see `docs/web-ui.md`). Request/response types live in `hive-sh4re`
+— change them in one place. The dashboard event vocabulary lives
+in `hive-c0re::dashboard_events::DashboardEvent`.
+
+### Broker delivery + ack cycle
+
+`AgentRequest::Recv` is the only path that delivers messages to an
+agent. Always returns a list (`Messages { messages }`) — empty when
+nothing's pending, single-pop when `max = None` (default 1, the
+single-message behaviour), batched up to `max` when caller asks for
+more (server-side cap is 5; values above clamp silently). The wire
+request still carries an optional `wait_seconds` (long-poll the first
+message, once one arrives — or one is already pending — the call
+drains up to `max` in total): the harness's own turn-driving loop
+uses it internally (`hive-agent`'s `recv_next`, 180s). The
+agent-facing MCP `recv` tool does not expose this parameter — it
+always passes `wait_seconds: None`, an immediate peek.
+
+Per-row bookkeeping inside the broker:
+
+- `delivered_at = NOW` set on every popped row.
+- Each recipient has an in-memory `unacked_ids` list of every row
+  delivered since the last `AckTurn`.
+- `redelivered = true` on a row if `RequeueInflight` resurfaced it
+  (the harness prepends a "may already be handled" hint when this
+  flag is set so the per-message warning is visible).
+
+`AgentRequest::AckTurn` closes out the in-memory list — the harness
+fires it after `TurnOutcome::Ok`, marking every message popped since
+the last ack as fully handled. Claude doesn't see this surface; it's
+strictly a harness↔broker pairing. On `TurnOutcome::Failed` the
+harness intentionally skips the ack so the unacked rows stay
+in-flight in the DB and get picked up by the next requeue sweep.
+
+`AgentRequest::RequeueInflight` is the recovery pair: fired by the
+harness exactly once at boot, before the serve loop starts. Catches
+the crashed-mid-turn / OOM-killed / container-restarted cases where
+a previous harness session popped messages but never drove them to
+a clean turn-end. Resets `delivered_at` back to NULL on every
+unacked row (so the next `Recv` pops them again), and remembers
+each id in a per-recipient in-memory set so the next `Recv` can tag
+the row with `redelivered: true`. Idempotent + cheap when there's
+nothing in flight, so the at-boot fire is unconditional.
+
+`AgentRequest::AckUntil { up_to }` is the agent-facing bulk-triage
+escape hatch (`mcp__hyperhive__ack_until`). Unlike `AckTurn` it IS
+visible to claude: each recv row and wake prompt carries a
+`[msg #<id>]` marker (the broker row id; transient pings show no
+marker — their sentinel id 0 has nothing to ack), and
+`ack_until(up_to: n)` marks every one of the agent's rows with
+`id <= n` handled in a single UPDATE — pending and delivered alike.
+This bounds the redelivered-flood cost after a restart: instead of
+popping dozens of already-handled messages one turn at a time, the
+agent notes the highest id it has seen and acks up to it.
+Recipient-scoped (an agent can only ack its own rows); also drains
+the in-memory `unacked_ids` / `requeued_ids` bookkeeping below the
+cutoff so a later `AckTurn` doesn't double-update and a stale
+redelivery tag can't outlive its row. The operator-side sibling is
+the dashboard's "mark all read" (unbounded, per-agent).
+
+### Loose-ends wire shape
+
+`LooseEnd` is the per-row response shape for `GetLooseEnds` (both
+the agent-flavour and manager-flavour requests). Tagged enum so
+new thread kinds (forge PRs, long-running approvals from a
+privileged bot, etc.) can land later without breaking existing
+handlers. Each row carries enough context that the caller renders
+it directly as a bulleted list, no follow-up fetch needed.
+
+Per-flavour scoping is uniform across the two cancellable variants:
+
+- **agent-flavour** `GetLooseEnds` only surfaces rows the calling
+  agent has standing in. `Approval` rows only appear when the
+  calling agent is the manager (sub-agents don't submit
+  approvals). `Reminder` rows are scoped to `owner == self`.
+- **manager-flavour** `GetLooseEnds` lists every pending row in
+  the swarm — full audit view.
+
+Per-variant fields:
+
+- `Approval { id, agent, commit_ref, description?, age_seconds }`
+  — `agent` is the affected agent (target of the spawn / config
+  commit), not the asker. `description` is the manager's free-text
+  blurb shown on the dashboard card. `commit_ref` is the
+  kind-specific payload (see `docs/approvals.md::Approval kinds
+  (wire shapes)`).
+- `Reminder { id, owner, message, due_at, age_seconds }` —
+  `due_at` is the absolute time the scheduler is targeting (RFC
+  3339 on the wire, see *Timestamps on the wire* below); clients
+  compute time-until-fire against it.
+- `PendingMessages { count }` — undelivered inbox messages the
+  agent still owes itself a `recv` for. Informational + not
+  cancellable (drain with `recv`); only emitted when `count > 0`,
+  and surfaced first in the agent-flavour list as the most
+  actionable signal. Counted host-side from the broker
+  (`count_pending`), so it reflects what's genuinely still queued —
+  the wake-message that drove the current turn is already delivered
+  and not counted.
+- `UnreadMatrix { rooms, summary }` — unread matrix notifications.
+  Informational + not cancellable (clear with `mark_read`). Unlike
+  the others this is injected by the in-container harness, not
+  hive-c0re, because the matrix daemon lives inside the agent.
+
+`age_seconds` saturates at zero on any clock anomaly (back-step,
+unsynchronised wall clock, etc.) so the bulleted list never
+shows nonsense ages.
+
+`CancelLooseEnd { kind, id }` is the matching write surface. The
+`kind` enum (`Reminder` / `Approval`) selects which underlying store
+the dispatcher reaches into. `Reminder` cancels from either surface
+subject to an ownership check (the scheduling agent).
+`Approval` is manager-only — sub-agents don't submit approvals
+so they have nothing of their own to withdraw; their wire
+surface returns a clear error if they try. Cancelling an approval
+transitions the row to `ApprovalStatus::Cancelled` and fires
+`ApprovalResolved { status: "cancelled" }` so the dashboard pulls
+the card out of the pending pane.
+
+### Agent metadata
+
+`AgentRequest::GetAgentMeta { name }` returns identity + status for
+an agent. Self-introspection when `name = None` (replaces the older
+`Whoami` request); target query when `name = Some`.
+
+Response is `AgentMeta { name, running, hyperhive_rev,
+status_text, status_set_at, hive_name, swarm_name, matrix_accounts }`:
+
+- `hyperhive_rev`: `None` only when the configured flake URL has
+  no canonical path. Otherwise carries the rev the target is
+  currently pinned at.
+- `running`: whether the target's container is currently up. When
+  `false`, the host clears `status_text` / `status_set_at` —
+  on-disk values from before the stop are stale snapshots and
+  shouldn't be shown as live status. Defaults to `true` on the
+  wire (older harnesses never serialised it, and the host only
+  knew how to ask about live containers — keeps backwards-compat
+  with pre-running-field payloads).
+- `status_text` / `status_set_at`: last value written via
+  `SetStatus`, plus its unix timestamp. Both `None` when the
+  target has never set a status, when the agent name is unknown,
+  or when `running = false` (see above).
+- `hive_name` / `swarm_name`: display names read from
+  `HYPERHIVE_HIVE_NAME` / `HYPERHIVE_SWARM_NAME` env (sourced from
+  `services.hyperhive.hiveName` / `services.hyperhive.swarm.name`).
+  Both `None` when the options aren't configured.
+- `matrix_accounts`: one `MatrixIdentity` per configured + live matrix
+  account the agent can act as. Empty for agents with no matrix
+  provisioning.
+
+### Timestamps on the wire
+
+Timestamp fields that cross a JSON boundary (dashboard API + SSE,
+the wire structs in hive-sh4re) serialize as **RFC 3339 UTC strings**
+(`2026-07-02T18:30:00Z`) via `hive_sh4re::wire_time` — Rust keeps the
+fields as `i64` unix seconds internally, only the JSON representation
+changes, and deserialization leniently accepts both the string form
+and the legacy bare integer (rolling-deploy skew, persisted blobs).
+**Input-direction** fields agents compute as epoch (`first_fire_at_unix`,
+schedule-edit `next_fire_at_unix`, `Wakeup::At`) stay integers. The
+`*_unix` field *names* are kept for now — renaming is the wire-types
+refactor's concern. The dashboard frontend parses via
+`util.js::epochSec` wherever it needs arithmetic and feeds the string
+straight to `new Date(s)` for display.
+
+### HTTP error bodies
+
+Every HTTP API in this repo answers failures with **RFC 9457
+`application/problem+json`** (`{ type, title, status, detail }`), with the
+human-readable cause in `detail`. An endpoint of ours returning a bare string
+or a bespoke error shape is a **bug to file against the backend**, not
+something for the caller to work around.
+
+Use the `problem_details` crate (`features = ["axum"]`), which the daemons
+already depend on: type a handler `Result<_, ProblemDetails>` and hand
+`ProblemDetails::from_status_code(...).with_detail(...)` to `Err`.
+
+The reason is the consumer, not tidiness. The UIs show errors through one
+shared component with a copy button, so a caller has to know **which part of
+the body is the message**. A bare string forces it to treat the whole payload
+as prose, which is the difference between offering "copy the cause" and
+dumping a response — and the cause is frequently the entire diagnosis (a
+JetStream permission refusal, a TLS chain failure) rather than a summary.
+
+Not in scope: the `hivectl` host-admin and in-agent unix sockets. Those are a
+JSON-line protocol with their own result types; RFC 9457 is an HTTP format.
+
+## Tool groups
+
+The MCP tool surface an agent receives is derived from a set of named
+`ToolGroup` values (`hive_sh4re::permissions::ToolGroup`), not from a hardcoded
+binary flavor.
+
+| Group | Tools |
+|---|---|
+| `messaging` | `send`, `recv`, `ack_until` |
+| `meta` | `get_agent_meta` (`set_status` is always-on, see below) |
+| `inbox` | `get_loose_ends`, `cancel_loose_end`, `remind` |
+| `execution` | vestigial — `mcp__bash__run` / `mcp__bash__status` are always available unconditionally via `extraMcpServers`; this group's entries expand to non-existent `mcp__hyperhive__run` / `mcp__hyperhive__status` and have no effect. See `docs/tools/bash.md`. |
+| `lifecycle` | `kill`, `start`, `restart`, `update`, `list_containers` *(privileged)* |
+| `approvals` | `request_init_config`, `request_update_meta_inputs` *(privileged)* |
+| `scheduling` | `request_schedule_prompt`, `fire_schedule_now`, `cancel_schedule`, `edit_schedule`, `list_schedules` *(privileged)* |
+| `diagnostics` | `get_logs` *(privileged)* |
+| `forge` | `create_repo` — create git repos through hive-c0re (operator-gated merge) |
+| `web_tools` | none (gates the Claude built-ins `WebFetch`/`WebSearch`, not an MCP tool) |
+
+**Always-on tools** — `set_status`, `compact`, and `mark_todos_done` are
+exposed to every agent regardless of which groups it holds
+(`ToolGroup::ALWAYS_ON_TOOLS`). The operator dashboard depends on every agent
+being able to report its status chip, and the server-side `SetStatus` handler
+has no tool-group check (only length validation), so gating it would only
+desync the `--allowedTools` list from what the host actually accepts.
+Revoking `meta` therefore drops `get_agent_meta` but never `set_status`.
+`mark_todos_done` is here because todos are pushed to an agent independent of
+whether it holds `inbox` — an agent without that group still needs a way to
+clear them.
+
+**Config storage** — per-agent tool groups live in
+`/var/lib/hyperhive/meta/tool-groups.json` (hive-c0re-owned, committed to the
+meta repo alongside `topology.json`). Format: `{ "alice": ["messaging", "meta",
+"inbox", "lifecycle"], "bob": ["messaging", "meta", "inbox"] }`. An absent entry
+means "use role default". Tool permissions are intentionally NOT configurable
+from `agent.nix` — that file goes through the manager's approval flow, so
+letting it declare its own groups would let the manager grant itself any tool by
+submitting a config commit, bypassing the operator gate.
+
+**Setting groups** — the operator sets groups via the dashboard or
+`hive-c0re::tool_groups::set_groups(name, groups)`. After a change
+`meta::sync_agents` commits the updated file; the next agent rebuild picks up
+the new `HIVE_TOOL_GROUPS` env var. Agents with no entry get no var.
+
+**Runtime resolution** — at session start the harness reads `HIVE_TOOL_GROUPS`
+(a comma-separated list of snake_case group names injected by the meta renderer
+from `tool-groups.json`). Unrecognised tokens are logged and skipped. Falls back
+to `ToolGroup::AGENT_DEFAULT` (`messaging`, `meta`, `inbox`, `execution`) when
+the var is absent or empty.
+
+**Updating the surface** — when a new `#[tool]` fn is added to `AgentServer`
+in `hive-agent-mcp/src/mcp/mod.rs`, add its name to the matching `ToolGroup::tools()`
+slice in `hive-sh4re/src/permissions.rs`. That's the single source of truth;
+`mcp_config::allowed_mcp_tools` (in `hive-agent/src/mcp_config.rs`) reads it at
+session start.
+
+## Capabilities
+
+Capabilities gate system-level access that goes beyond the MCP tool surface —
+things an agent can *access*, not just *call*. Parallel to tool groups but
+orthogonal: an agent can have a tool group that registers a tool AND a capability
+that allows the underlying resource access.
+
+| Capability | Effect |
+|---|---|
+| `manage_root_agent` | may lifecycle-manage the root/manager agent via `kill`/`start`/`restart` |
+| `read_host_journal` | `get_host_journal` MCP tool is registered + `GET /journal-host` requests are served |
+| `query_agent_state` | may call `get_loose_ends` / `CountPendingReminders` targeting non-child agents |
+
+**Config storage** — per-agent capabilities live in
+`/var/lib/hyperhive/meta/capabilities.json` alongside `tool-groups.json`.
+Format: `{ "atlas": ["read_host_journal"], "ruth": ["manage_root_agent"] }`.
+An absent entry means "no extra capabilities". `render_flake` in `meta.rs`
+reads this file and injects `HIVE_CAPABILITIES` (comma-separated
+`snake_case` names) into each agent's systemd service env; absent entries emit
+no env var so agents without capabilities don't trigger a spurious rebuild.
+
+**Setting capabilities** — the operator sets capabilities via the
+C4P4B1L1T13S section in the dashboard's P3RM1SS10NS tab.
+`hive-c0re::capabilities::set_caps(name, caps)` is the write path.
+After a change `meta::sync_agents` commits the updated file; the next agent
+rebuild picks up the new `HIVE_CAPABILITIES` env var.
+
+**Runtime resolution** — at session start the harness reads `HIVE_CAPABILITIES`
+and resolves each token to a `Capability` variant. Unrecognised tokens are
+logged and skipped. An absent or empty var means no extra capabilities.
+
+**Capability NOT configurable from `agent.nix`** — same reasoning as tool
+groups: an agent that could grant its own capabilities via a config commit would
+bypass the operator approval gate.
+
+**Adding a new capability** — add a variant to `Capability` in
+`hive-sh4re/src/permissions.rs` + an arm to `as_str`. Add it to `Capability::ALL`
+(the source of truth for the permissions UI columns). Implement the access
+check in the relevant handler (`hive-c0re/src/socket_server/mod.rs`,
+`hive-c0re/src/socket_server/lifecycle_handlers.rs`, `coordinator.rs`, or a
+handler under `hive-c0re/src/dashboard/`).
+
+## Async forms
+
+Dashboard + per-agent mutating forms carry `data-async`; the shared
+`bindAsyncForms` `submit` listener (`frontend/packages/shared/src/forms.js`,
+imported as `@hive/shared/forms.js` and wired up from `tabs.js` on the
+dashboard and `app.js` on the per-agent UI) intercepts, shows a spinner,
+POSTs `application/x-www-form-urlencoded` (axum's `Form` extractor
+rejects multipart), calls `refreshState()` on success. New mutating
+forms should add `data-async` and optionally `data-confirm` (for a
+JS-side `confirm()` prompt) or `data-prompt="…"` (for a
+`window.prompt()` whose answer goes into a hidden input named by
+`data-prompt-field`, default `note`).
+
+`refreshState` defers automatically when `document.activeElement`
+sits inside a managed section so the operator's typing isn't lost;
+collapsible `<details data-restore-key=…>` survive the re-render
+via `snapshotOpenDetails` / `restoreOpenDetails`.
+
+## `rebuild` is the reconcile verb
+
+`job_queue::templates::rebuild` builds the DAG that reconciles a
+container to its wanted state: `write_dropins` (the nspawn-conf
+rewrite — `PRIVATE_NETWORK=1`, `HOST_ADDRESS` = the bridge gateway IP,
+sets `EXTRA_NSPAWN_FLAGS` — plus the systemd resource-limits drop-in)
+is folded into the `Swap` node, then `nixos-container update` + stop +
+start runs across the `StopForUpdate → Swap → RebuildBookkeeping`
+brace and the tail `Reconcile` node. `flake.nix` itself is no longer
+regenerated host-side on rebuild — it's tracked in the agent's
+proposed/applied repos and rides along on every fetch (see
+`docs/approvals.md::Two repos per agent`).
+
+Anything that changes per-container state on the host should be
+re-applied here so a manual `↻ R3BU1LD` from the dashboard is
+sufficient to recover.
+
+## Actions are factored
+
+`approve` / `deny` / `destroy` (and the lifecycle helper) live in
+`actions.rs` / `hive-c0re/src/dashboard/`. The admin socket and the dashboard
+POST handlers both call into them so the two surfaces never drift.
+
+## Commit messages
+
+Short, lowercase, no `Co-Authored-By` trailer. Imperative mood, no
+period. Body explains *why* if non-obvious; otherwise the subject
+alone is fine. Wrap at ~72 cols.
+
+## Commit before test
+
+Stage and commit when work *looks* ready, then run validation
+(`cargo check`, `nix flake check`, real deploy). Failures get a
+follow-up commit rather than an amend. The commit history is the
+work log; rewriting it loses signal.
+
+## Building & local checks
+
+Build through the **flake devshell**, not a bare toolchain — agent
+containers ship no global rust. `nix develop -c <cmd>` runs one
+command inside the project-pinned env (cargo/clippy/rustfmt plus the
+C compiler + `libsqlite3`/`ring` link deps); outside it a bare
+`cargo build` fails with `failed to find tool "cc"` / `cannot find
+-lsqlite3`. One command per invocation — agents run each task as a
+fresh non-interactive process, so there's no persistent shell to
+reuse.
+
+```sh
+nix develop -c cargo clippy --all-targets -- -D warnings
+nix develop -c cargo test
+nix fmt                       # treefmt — authoritative, NOT bare cargo fmt
+```
+
+`nix fmt` (treefmt) is the formatter CI gates on; bare `cargo fmt`
+misses the non-rust files treefmt also covers, so always run `nix
+fmt` before pushing.
+
+**Clippy discipline — never add `#[allow(clippy::…)]`.** All lints are
+CI-fatal at `-D warnings` (pedantic included); every warning that fires
+must be *fixed*, not silenced. Common patterns:
+
+- `too_many_lines` — extract a helper function or a sub-struct
+  (the `TurnAccum` extraction in `stats.rs` is a worked example).
+- `doc_markdown` (brand name without backticks in a doc comment) — add
+  backticks: `` `DOMPurify` `` instead of `DOMPurify`.
+- `must_use` / `unused_results` — actually handle or explicitly discard
+  the return value (`let _ = …` is fine when intentional).
+
+If a lint seems wrong for a specific call site, file an issue and ask
+mara — don't add `#[allow]` speculatively. The gate is intentional.
+
+**The devshell checks are not the full `nix flake check`.** Clippy /
+fmt / `cargo test` cover most gates, but `nix flake check` runs extra
+check derivations they don't:
+
+- **`hivectl-docs`** regenerates `docs/tools/hivectl-cli.md` from
+  hivectl's clap tree and **fails if the committed copy is stale**.
+  So **after any change to a hivectl verb or flag, regenerate it**:
+  ```sh
+  nix develop -c cargo run --bin hivectl -- markdown-docs > docs/tools/hivectl-cli.md
+  ```
+  clippy / fmt / `cargo test` all pass *without* this — only the
+  flake check catches the drift, and `ci-log` often can't show you
+  why (it 500s on a fast failure), so you're left guessing "builder
+  flake" when it's a stale doc.
+- **`swarmctl-docs`** is the same check for `swarmctl` /
+  `docs/tools/swarmctl-cli.md`:
+  ```sh
+  nix develop -c cargo run --bin swarmctl -- markdown-docs > docs/tools/swarmctl-cli.md
+  ```
+- there's also a flake `cargo-test` check and NixOS module
+  evaluation in the set.
+
+When local clippy/fmt/test pass but CI's `nix flake check` fails,
+**don't assume a transient builder problem** — reproduce the real
+gate locally: `nix flake check` (shares the build farm, use
+sparingly) or build just the suspect check, e.g. `nix build
+.#checks.x86_64-linux.hivectl-docs`.
+
+## Best-effort oneshot services
+
+The harness ships a family of one-shot systemd services that
+configure agent-side surfaces from values hive-c0re writes into
+the state dir at provisioning time:
+
+- `tea-login` — writes `~/.config/tea/config.yml` from the
+  `forge-token` written by `hive-c0re::forge::ensure_user_for`,
+  so `tea repos create` / `tea pulls create` work without
+  interactive prompts.
+- `forge-avatar-sync` — uploads `hyperhive.icon` SVG to the
+  agent's Forgejo profile, so the icon shows up on commits / PRs /
+  issue comments.
+
+(The matrix profile avatar is **not** a oneshot — `hive-matrix-daemon`
+sets it over its live authenticated Client; see
+`docs/persistence.md::matrix avatar`.)
+
+Shape contract — every one of these:
+
+1. **Always `exit 0`**, even on internal failure. A non-zero exit
+   would mark the unit `failed`, which in turn aborts
+   `nixos-container update` and blocks rebuilds. The agent's
+   capability surface is not allowed to gate the container
+   build.
+2. **No `set -e`** in the script body. Subshell failures must not
+   propagate. Use `... || true` on every external call that can
+   fail (forge unreachable, missing icon, parse error, etc.)
+3. **Skip silently when prerequisites are missing**: no token
+   file, no icon, no reachable upstream → `echo` a short skip
+   line + `exit 0`. The next boot tries again.
+4. **Wired to `multi-user.target`** so they run on every boot
+   (lets a rotated token / new icon take effect without
+   `systemctl restart` gymnastics).
+5. **Re-runnable**: a second invocation produces the same final
+   state (idempotent uploads, idempotent config rewrites). Used
+   by the `.path` watchers that re-fire on token appearance (see
+   `docs/persistence.md::Matrix per-agent daemon`).
+
+The artefact lives under the agent user's home where applicable
+(`~/.config/tea/config.yml`) and is chown'd to that user, but the
+service itself stays root-owned so the bootstrap ordering doesn't
+need a user-existence check before each fire.
+
+This pattern keeps the rebuild path resilient: any failure inside
+these services degrades the corresponding surface (no tea config,
+no avatar) but never blocks the container from coming up. The
+operator notices through `journalctl -u <unit>` rather than a
+broken switch-to-configuration.

docs/process/gotchas.md

@@ -0,0 +1,522 @@
+# Gotchas
+
+NixOS + nspawn quirks and lessons we hit the hard way. If something
+here looks unmotivated in the code, there's usually a story underneath.
+Grouped by area — jump to the section that matches what you're
+touching.
+
+## NixOS / nspawn containers
+
+### `nixos-container` doesn't expose `--bind` on the CLI
+
+The CLI doesn't accept `--bind`. Path is via `EXTRA_NSPAWN_FLAGS` in
+`/etc/nixos-containers/<NAME>.conf` — the start script
+(`/nix/store/.../container_-start`) expands it unquoted into the
+`systemd-nspawn` invocation. `lifecycle::host_config::set_nspawn_flags()`
+rewrites this line.
+
+### `/run/systemd/nspawn/*.nspawn` overrides are ignored
+
+`nixos-container`'s start script builds the nspawn command line
+directly. Dropping a `.nspawn` file under `/run/systemd/nspawn/`
+looks like the obvious extension point and does nothing. Use
+`EXTRA_NSPAWN_FLAGS` (above).
+
+### `boot.isNspawnContainer = true`
+
+Not `boot.isContainer = true`. Renamed in nixos-25.11+.
+
+### `nixos-container create` auto-assigns `HOST_ADDRESS` / `LOCAL_ADDRESS`
+
+…in the `.conf`. The start script's `if HOST_ADDRESS set →
+--network-veth` branch then forces a private netns — silently fatal
+for our web UIs (the bind is invisible from the host). Every agent
+container runs isolated: `hive-priv`'s `write_nspawn_flags` writes
+`PRIVATE_NETWORK=1` plus a veth pair onto the host bridge, `HOST_ADDRESS`
+set to the bridge gateway IP (so `nixos-container`'s in-container init
+installs a default route before the DHCP lease arrives), rest left for
+DHCP.
+
+### systemd service PATH ≠ host PATH
+
+The hive-c0re service sets `path = [ pkgs.git "/run/current-system/sw" ]`.
+In-container harness services do the same so anything an agent adds
+to its own `agent.nix` (`environment.systemPackages`) is visible to
+the `mcp__bash__run` MCP tool (and any other in-container process) without
+editing the service definition.
+`environment.HYPERHIVE_GIT` bakes git's absolute path in (read by
+`lifecycle::git_command()`) for the host.
+
+### `systemd.services.*.path` appends `/bin` to every entry
+
+NixOS's `systemd.services.<unit>.path` list feeds every entry through
+`lib.makeBinPath`, which **appends `/bin` unconditionally**. That's
+the right thing for Nix packages (their `outPath` is the store root,
+not the `bin/` subdir), but it bites when you pass a string that
+already ends with `/bin`:
+
+```nix
+# ❌ /run/wrappers/bin → /run/wrappers/bin/bin (does not exist)
+path = [ "/run/wrappers/bin" "/run/current-system/sw" ];
+
+# ✅ /run/wrappers → /run/wrappers/bin  (the real wrappers dir)
+path = [ "/run/wrappers" "/run/current-system/sw" ];
+```
+
+The bug is silent: `nix eval` succeeds, the unit starts, but PATH
+contains a non-existent directory. The first symptom is usually
+`sudo: must be owned by uid 0 and have the setuid bit set` because
+the setuid sudo wrapper lives at `/run/wrappers/bin/sudo` and
+the path entry resolves to `/run/wrappers/bin/bin` instead.
+
+### `RuntimeDirectoryPreserve = "yes"`
+
+…keeps `/run/hyperhive/` (and the per-agent sub-dirs) across
+hive-c0re restarts. Without it, every restart wipes bind sources and
+existing containers can't be started.
+
+### `RestrictAddressFamilies` fails as "Address family not supported by protocol"
+
+A unit whose `RestrictAddressFamilies` omits a family gets `EAFNOSUPPORT`
+(errno 97) back from `socket()`. Clients surface that as *"tcp open error:
+Address family not supported by protocol"* — the message names the
+**protocol** and never the **sandbox**, so it reads like a dead network, a
+missing route, or an IPv6 problem.
+
+⇒ On that error, read the unit before you touch the network.
+
+Two things to get right when a daemon needs outbound TCP:
+
+- list `AF_INET` **and** `AF_INET6` — omitting one leaves a client that
+  works until DNS hands back the other family;
+- list `AF_NETLINK` too. glibc's `getaddrinfo` opens a netlink socket to
+  enumerate local addresses before it returns any, so name resolution
+  fails without it even when `AF_INET` is allowed.
+
+**The directive is a claim about what the program does, and nothing
+re-checks it when the program changes.** A unit that only served a unix
+socket when it was written is correct at `[ "AF_UNIX" ]` and silently wrong
+the day someone adds an HTTP client. Check the unit in the same commit as
+the client — and when narrowing it, prefer a test that derives the required
+families from the code (which fails on the *next* client too) over one that
+asserts today's list.
+
+### `register_agent` is idempotent
+
+Drops any prior socket task before rebinding. Required so a
+hive-c0re restart followed by `rebuild alice` recreates the agent's
+socket without needing a clean reinstall.
+
+## Claude Code packaging & credentials
+
+### `claude-code` is unfree
+
+`claude-code` comes from the flake's main `nixpkgs` (nixos-26.05).
+It's unfree, so the agent modules set `config.allowUnfreePredicate`
+at the container level to whitelist `claude-code` specifically —
+scoped, only this one package. This is needed because each per-agent
+`nixosConfiguration` evaluates its own nixpkgs instance and the
+operator's host-level `allowUnfree` does **not** propagate in.
+Operators don't need to set anything on their side.
+
+That same isolation is why an agent can't pick a claude out of a
+*different* nixpkgs by itself: a container only ever sees the one
+nixpkgs the meta flake injects, so an `agent.nix` naming the host's
+`nixpkgs-unstable` has nothing to name. A release channel can trail
+unstable by weeks on this package, which is what
+`services.hyperhive.c0re.claudeCodePackage` is for — set it host-side
+and every agent runs that build.
+
+What crosses is the **store path**, not the derivation. Containers
+share the host's `/nix/store`, so the binary is already reachable
+inside them with its whole closure; hive-c0re writes the path into each
+agent's flake as a string literal and the agent module symlinks
+`bin/claude` onto PATH. Two things rule out the obvious alternatives: a
+`path:/nix/store/<pkg>` flake input is re-copied into the store as a
+reference-less `-source` (so the runtime closure never arrives), and
+`lib.types.package` fed a bare path runs `builtins.storePath`, which
+pure evaluation rejects. `hyperhive.docs.source` gets away with being
+an input only because a docs tree has no runtime dependencies.
+
+The `storePath` trap is worth spelling out, because it is not confined
+to options the operator writes: **any** option of type `package` fed a
+store-path *string* coerces through `lib.toDerivation`, i.e.
+`builtins.storePath`. `environment.systemPackages` and
+`systemd.services.<name>.path` both do it (the latter takes plain
+strings like `/run/wrappers` happily, but anything under
+`builtins.storeDir` is treated as a package). So a path handed to the
+container as text has to be wrapped in a real derivation — a symlink
+farm built from the interpolated string — before it can go anywhere a
+package is expected.
+
+The catch is that a path written into a generated flake is text, not a
+reference — the container's closure does not keep the binary alive.
+The **host** does: the package is interpolated into
+`/etc/hyperhive/serve.json`, so it lands in the host's system closure
+and is gc-rooted by the running generation. `builtins.toJSON` preserves
+string context, which is the load-bearing detail; discard the context
+anywhere on that path and `nix-collect-garbage` will eventually take
+the hive's `claude` out from under it. The price of the root is that an
+old `claude-code` can't be reclaimed until every agent has rebuilt past
+it and the old generations are gone.
+
+### Claude credentials are per-agent
+
+`/var/lib/hyperhive/agents/<name>/claude/` bind-mounts to
+`/home/<name>/.claude` (RW). Sharing one dir across agents is NOT viable —
+OAuth refresh tokens rotate, so any sibling refresh invalidates all
+the others. Login flow runs from the per-agent web UI; creds persist
+across `destroy`/recreate (`--purge` wipes them).
+
+### Persistent notes dir per agent
+
+`/var/lib/hyperhive/agents/<name>/state/` bind-mounts to
+`/agents/<name>/state` (RW; uniform for all agents).
+The harness exposes the same path
+via `$HYPERHIVE_STATE_DIR`. System prompts tell agents to keep
+durable knowledge here (`notes.md`, anything else) — the harness's own
+internal files (`hyperhive-events.sqlite`, `hyperhive-turn-stats.sqlite`,
+`hyperhive-model`) live in the separate `harness` dir instead, so they
+don't clutter what claude sees as "my notes dir" (see
+[`docs/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)). Survives `destroy`/recreate
+alongside the claude dir.
+
+## Networking & ports
+
+### Web UI ports collide on hash
+
+Sub-agent web UI ports are deterministic FNV-1a of the agent name
+modulo 900 (range 8100..8999). With ~30 agents the birthday-paradox
+collision rate gets meaningful; at 2–3 agents you can still get
+unlucky. Operator resolves a collision by renaming the offending
+agent (different hash → different port) and rebuilding. No state
+file, no probing, no port-allocation drift — the value is
+reproducible from just the name. Every agent hashes into
+8100..8999 via the same FNV-1a; dashboard
+at `cfg.dashboardPort` (default 7000).
+
+### Restart races on TCP bind
+
+Both the dashboard and per-agent web UI use `tokio::net::TcpSocket`
+with `SO_REUSEADDR` plus a retry-on-`AddrInUse` loop (12 tries,
+exponential backoff capped at 2s, ~22s total). REUSEADDR handles
+the `TIME_WAIT` case from a clean previous exit; retry covers the
+genuine "previous process is still alive during a systemd restart
+overlap" case. REUSEADDR does **not** allow two simultaneous
+`LISTEN` sockets on the same port (that would be `SO_REUSEPORT`,
+which we don't use) — exclusivity is preserved.
+
+## Approvals
+
+### Orphan approvals
+
+If state dirs are wiped out from under a pending approval (test
+scripts, manual `rm -rf`), the dashboard's next render marks them
+`failed` with note `"agent state dir missing"` so they fall out of
+`pending`. They stay in sqlite for audit.
+
+## Gateway / SPA serving
+
+### SPA fallback: use `Accept` header map, not `try_files ... /index.html`
+
+The naive nginx pattern for an SPA (`try_files $uri $uri/
+/index.html`) silently swallows asset 404s — a missing JS file
+returns `index.html` with a 200, so the JS runtime never loads and the
+page renders blank with no visible error. Extension allowlists (tried
+as an alternative) have the same maintenance problem: any new file
+extension the SPA ships breaks silently.
+
+The pattern that works (`nix/host-modules/hive-matrix.nix`, serving
+fluffychat at the matrix gateway vhost's root) keys the fallback on the
+HTTP `Accept` header:
+
+```nginx
+# Outside the server block (appendHttpConfig):
+map $http_accept $matrix_spa_target {
+  default          "/__matrix_spa_no_html_fallback";
+  "~*text/html"    "/index.html";
+}
+
+# Inside the location:
+try_files $uri $uri/ $matrix_spa_target =404;
+```
+
+Top-frame navigations always send `Accept: text/html,...` (chrome /
+firefox / safari are consistent). Asset fetches (`image/*`,
+`application/javascript`, `*/*`) don't carry `text/html`, so they
+fall through to the trailing `=404`. No extension list to maintain;
+no named-location indirection needed.
+
+## Build & dev workflow
+
+### Nix store `cp -r` preserves read-only bits
+
+Copying a nix store path with `cp -r src/. $out/` inside a
+`pkgs.runCommand` derivation preserves the read-only permissions of
+store files. Any subsequent write into the copied tree (adding new
+files in subdirectories) fails with `EPERM`. Fix: pass
+`--no-preserve=mode,ownership` so the output tree is writable.
+
+### `nix build flake#name` does not walk into `nixosConfigurations`
+
+`nix build` resolves the fragment (`#name`) against the flake's
+**top-level output attrs** — not against `nixosConfigurations`
+specifically. `nixos-container` and `nixos-rebuild` use their own
+internal convention that routes an agent name to
+`nixosConfigurations.<name>.config.system.build.toplevel`, but
+`nix build` has no such convention.
+
+```
+# ❌ silently builds the wrong thing (or errors if attr doesn't exist)
+nix build /var/lib/hyperhive/meta#argus.config.system.build.toplevel
+
+# ✅ explicit path nix build actually resolves
+nix build /var/lib/hyperhive/meta#nixosConfigurations.argus.config.system.build.toplevel
+```
+
+`lifecycle::prebuild_toplevel` hit this once by constructing the attr
+path as `{flake_ref}.config…` — which produced `meta#argus.config…`
+instead of `meta#nixosConfigurations.argus.config…`. The fix:
+`split_once('#')` to separate flake path from name, then template
+`{path}#nixosConfigurations.{name}.config.system.build.toplevel`.
+
+### Containerized nix-daemon needs `sandbox-fallback = true`
+
+Agent containers bind-mount the host's nix-daemon socket. nspawn
+containers don't get user-namespaces by default, so `nix build`
+invocations _inside_ the container can't set up the build sandbox
+and fail outright if the host daemon's
+`nix.settings.sandbox-fallback` is `false` (nixpkgs default).
+`nix/agent-modules/default.nix` does `lib.mkForce true` so builds
+fall back to unsandboxed local builds rather than failing. Security
+implications: `docs/security.md`.
+
+### Linking workspace binaries locally needs `nix develop`
+
+The Rust workspace links `libsqlite3-sys` (rusqlite) against the
+system `libsqlite3`. Agent containers carry no system libsqlite3 on
+the linker path, so a plain `cargo build` of any binary dies with
+`cannot find -lsqlite3` (deps and `ring` compile fine — only the
+final link fails). `cargo check` / `cargo clippy` still work in the
+ambient shell since they never link.
+
+Build + run binaries through the dev shell, which carries `sqlite`
+on `NIX_LDFLAGS`:
+
+```bash
+nix develop -c cargo build -p hive-c0re --bin hivectl
+nix develop -c cargo run -p hive-c0re --bin hivectl -- <args>
+```
+
+This is also how you regenerate committed generated docs locally —
+e.g. `docs/tools/hivectl-cli.md` via the `hivectl markdown-docs`
+subcommand (its `hivectl-docs` flake check otherwise only fails in
+CI on drift).
+
+### Split asset derivations away from the rust workspace
+
+`nix/packages/assets.nix` builds the branding SVG/PNG family + claude
+system-prompt template + claude-settings JSON as its own derivation,
+separate from the hive-ag3nt / hive-c0re crates. Reason: when the
+rust build's `src` was the whole repo tree, any tweak to
+`branding/agent-configs.svg` or `hive-ag3nt/prompts/system.md`
+invalidated the cargo cache and forced a full rebuild. crane (and
+naersk before it) couldn't see "these inputs are unused by rust" on
+its own — the split breaks the coupling at the derivation boundary.
+The agent-configs PNG is rendered from the SVG via `rsvg-convert` at
+build time; librsvg dependency lives here, not in the rust
+derivation's `nativeBuildInputs`.
+
+### `nix fmt` fails in a git worktree with "object not found"
+
+`nix fmt` (and any `nix` command that fetches a `git+file://` flake
+URL) uses libgit2 internally to compute `revCount` — the number of
+commits reachable from HEAD. This walk fails with:
+
+```
+error: getting Git object '<hash>': object not found (libgit2 error code = 9)
+```
+
+when a commit that was reachable at some earlier evaluation is now gone
+(GC'd, rebased away, or pruned). The failure is persistent: clearing
+`~/.cache/nix/{eval-cache-v6,gitv3,fetcher-cache-v4.sqlite}` does not
+help because the missing object is a structural gap in the git object
+graph itself, not in nix's caches.
+
+**Workaround: use a plain clone, not a git worktree.**
+
+```bash
+git clone http://<forge>/hyperhive/hyperhive.git ~/hh-work
+cd ~/hh-work && nix fmt
+```
+
+The root cause is specific to worktrees: a worktree shares the object
+store with its parent repo. If the parent repo's history was rewritten
+(rebase, force-push, `git gc --prune`) while the worktree was checked
+out at a branch tip that references the pruned commits via its reflog or
+history, libgit2's rev-walk encounters the gap. A plain clone has its
+own self-consistent object store and is immune to the issue.
+
+## Tooling
+
+### `hive-forge`: prefer over raw curl pipelines
+
+Full CLI reference: [`docs/tools/forge.md`](../tools/forge.md).
+Never use raw `curl` for forge access.
+
+## GUI (weston/VNC)
+
+### Weston VNC compositor (per-agent `hyperhive.gui.enable`)
+
+`nix/agent-modules/weston-vnc.nix` adds an optional Weston Wayland
+compositor with the VNC backend, surfaced as
+`hyperhive.gui.enable = true` per-agent. The harness's
+`/screen/ws` WebSocket relay (`docs/web-ui/agent.md::Per-agent endpoints`)
+connects to the compositor at `127.0.0.1:<vnc_port>`.
+
+- **Port allocation**: a **fixed** port (`hyperhive.gui.vncPort`,
+  default 5900). No per-agent hashing: network isolation is
+  unconditional (each agent has its own netns — see
+  `docs/network.md#container-isolation`), so the VNC port is
+  container-local and can't collide across agents. The harness learns
+  the port from the `HIVE_GUI_VNC_PORT` env var (set on the harness
+  service when `gui.enable`) — no marker file, no runtime hash. (Unlike
+  the agent **web-UI** port, which is still an FNV-1a hash because those
+  listen on the shared host stack — see `Web UI ports collide on hash`.)
+- **Non-root, shared user session**: weston runs as the agent's own
+  user (`hyperhive.user.name`, the same user hive-ag3nt runs as), not
+  root, so the GUI and the agent share one session. The runtime dir is a
+  fixed `/run/gui` (systemd `RuntimeDirectory=gui`, `0700`,
+  `RuntimeDirectoryPreserve=yes` so it survives weston restarts for the
+  wayland client sharing the `/run/gui/wayland-0` socket). Wayland
+  clients in the agent's config (e.g. bitburner electron) must run as the
+  same user with `XDG_RUNTIME_DIR=/run/gui`.
+- **One shared D-Bus session bus (`gui-dbus.service`)**: a single
+  persistent `dbus-daemon --session` bound at `/run/gui/bus`, run as the
+  agent user, ordered `before weston.service` (it shares the same
+  `RuntimeDirectory=gui`, creating the dir first). Chromium/electron via
+  ozone refuse to map an `xdg_toplevel` without a reachable session bus
+  ("Failed to connect to the bus" → binds `xdg_wm_base` then destroys it
+  = invisible window even though CDP works). The fix is **not** to wrap
+  each client in its own `dbus-run-session` (a private throwaway bus per
+  process — that's a _separate_ session, defeating the one-session
+  model); it's this one shared bus, whose address is exported as
+  `DBUS_SESSION_BUS_ADDRESS=unix:path=/run/gui/bus` via
+  `systemd.globalEnvironment` so weston, the harness and every GUI client
+  inherit it.
+- **Fixed Wayland socket name (`--socket=wayland-0`)**: weston is
+  launched with `--socket=wayland-0` so the socket path is
+  deterministic. `nix/agent-modules/weston-vnc.nix` exports `WAYLAND_DISPLAY=wayland-0`
+  and `XDG_RUNTIME_DIR=/run/gui` as global system environment
+  variables (gated on `hyperhive.gui.enable`) so every systemd service
+  in the container inherits them. Without this, services starting
+  Wayland clients could not find the compositor — libwayland falls
+  back to a headless display or errors out, the app "works" on a
+  second invisible display, and the VNC session shows a blank weston
+  desktop (#540 double-screen).
+- **VNC bind address**: weston's VNC backend has no CLI
+  bind-address flag (unlike the RDP backend's `--address`), so the
+  listener binds `0.0.0.0`. The harness relay only connects via
+  `127.0.0.1`; the host firewall blocks the per-agent VNC port range
+  from external access. A future weston.ini `[vnc] address=` will
+  let us restrict the bind directly once upstream supports it.
+- **PAM service name**: literal `weston-remote-access` — that's the
+  string libweston passes to `pam_start()` in `libweston/auth.c`.
+  Using `weston` falls back to the system default PAM stack and
+  rejects auth. The service is configured to `pam_permit.so` for
+  all three module types (auth / account / session) so the
+  browser's empty Apple-DH credentials (type 30) always pass —
+  neatvnc ≥ 0.9 calls the PAM auth callback regardless of
+  `weston.ini` `auth-method=none`, so the permit fallback is what
+  actually lets the empty-cred client through.
+- **`Type = "simple"` (not `notify`)**: `switch-to-configuration`
+  must never block on weston signalling readiness. A misconfigured
+  weston degrades to a `Restart=on-failure` loop visible in
+  `journalctl`, it does not abort the `nixos-container update`.
+  Same reasoning as the `tea-login` unit in `nix/agent-modules/forge.nix`.
+- **`[core] idle-time=0`**: disables weston's 300-second idle
+  timeout. Without it the VNC desktop fades to black and
+  desktop-shell shows its click-to-unlock screen — useless for an
+  agent desktop viewed over `/screen`. `idle-time=0` updates the
+  idle timer with a 0ms delay, which
+  `wl_event_source_timer_update` treats as "disarm", so the
+  compositor never goes idle and never locks.
+
+## Nix docs pipeline
+
+### Nix options reference (`nix/docs/default.nix`)
+
+`pkgs.nixosOptionsDoc` over two evaluated module trees:
+`hostEval` (a stub NixOS system loading the `nix/host-modules/` aggregator with every
+hyperhive subsystem `mkForce false` so heavy build inputs stay out of
+the eval) and `agentEval` (evaluates `agent.nix` fresh for the
+per-agent options tree).
+
+Five output trees consumed by `flake.nix`, all **markdown**:
+
+- `docs-host` — operator-facing host module options
+  (`services.hyperhive.*` minus the `swarm`/`deploy` subtrees below)
+- `docs-swarm` — swarm-wide facts, identical on every host in the swarm
+  (`services.hyperhive.swarm.*`)
+- `docs-deploy` — this host's own deployment decisions, necessarily
+  different per host (`services.hyperhive.deploy.*` — added by
+  `nix/host-modules/deploy.nix`, split out of `swarm.*` for exactly this
+  reason: see that file's own comment)
+- `docs-agent` — per-agent harness options (`hyperhive.*`
+  declared in `nix/agent-modules/`)
+- `docs` — bundle of `index.md` + `host.md` + `swarm.md` +
+  `deploy.md` + `agent.md`
+
+Pipeline:
+
+- CommonMark from `nixosOptionsDoc.optionsCommonMark` is the only
+  output — the source of truth, emitted as `.md`.
+- **HTML + CSS is rendered downstream by the website repo**
+  (`nix/options.nix` there), which consumes this bundle's `host.md` /
+  `agent.md`, renders them with `cmark-gfm`, and shares one
+  stylesheet (`docs.css`) across `/options/` and the prose `/docs/`
+  tree. Keeping rendering in the website means the theme has a single
+  home and the colours are shared.
+- `transformOptions` strips the nix-store prefix from option
+  declaration paths and rewrites them as forge URLs, so the
+  rendered docs link back to the source.
+
+Host options live entirely under `services.hyperhive.*`. The
+`pickSubtrees` filter is rooted at `["services" "hyperhive"]` so the
+options tree picks up everything under that root — picking against
+stray roots produces an empty tree and renders the host page as
+template chrome with no `<h2>` headers. `docs-host` then drops the
+`swarm`/`deploy` subtrees from that picked tree (`removeAttrs`, not a
+second `pickSubtrees` root — the one exclusion this file needs doesn't
+earn a general-purpose helper) since `docs-swarm`/`docs-deploy` each
+pick their own root instead.
+
+#### Docs drv stability: `nixSrc`
+
+Naively, the docs evaluation depends on `self` (the flake's store path),
+so every commit — even Rust-only or frontend-only changes — produces new
+docs drv hashes. The remote builder must rebuild docs from scratch for
+every PR branch, and if its store is full the build fails with a cached
+failure that blocks CI for the whole branch.
+
+The fix (`nix/docs/default.nix`):
+
+1. **`nixSrc`** — `builtins.path` on the `nix/` directory, wrapped in
+   `builtins.unsafeDiscardStringContext` to strip `self`'s store-path
+   context. The resulting store path is content-addressed from the nix/
+   file contents only. Docs drvs only change when a `.nix` file changes.
+
+2. The package options the modules consume (`hyperhive.packages.*`,
+   `services.hyperhive.c0re.*`) carry no in-module defaults and every
+   default that references them has a `defaultText`, so the doc walk
+   never forces a package — no stubs needed, and the Rust/frontend
+   build closure stays out of the eval.
+
+3. Both `hostEval` and `agentEval` are evaluated from `nixSrc` paths
+   (not `self`), so the docs drv dependency chain ends at `nixSrc`.
+
+Why `builtins.unsafeDiscardStringContext`? The path string
+`toString self + "/nix"` carries `self`'s string context, which would
+make `builtins.path` include `self` as a build dependency even after
+content-addressing the directory. Discarding the context makes the
+resulting `nixSrc` truly independent of `self`'s store path.

docs/process/pr-review-gate.md

@@ -0,0 +1,65 @@
+# The PR review gate
+
+What a review verdict means, why a reviewer shouldn't wait on CI to
+submit one, and what "armed to auto-merge" actually signals about the
+human review that already happened.
+
+## The gate has (up to) three parts, and they're per-repo settings
+
+Whether a PR can merge, and what counts toward "can", is configured
+per repo in its branch-protection settings — not a fact true of every
+hive or every repo. The pieces a repo *can* require:
+
+- **CI is green** — the repo's required status checks pass on the
+  PR's current head commit, if the repo requires any.
+- **Requested reviews are `APPROVED`** — and whether a review is
+  invalidated by a later commit ("stale") is itself a setting
+  (Forgejo's "dismiss stale approvals" branch-protection option), not
+  universal behavior.
+- **Someone with write access has armed the PR to merge** — a manual
+  merge once the required conditions hold, or Forgejo's auto-merge
+  (merges automatically the moment the other required conditions are
+  met).
+
+Where these are required, they're independent of each other. A
+reviewer only ever owns the review-approval piece — CI resolves (or
+doesn't) on its own regardless of what a review says, and merge-arming
+is someone else's call.
+
+## Reviewers: submit the verdict, don't gate it on CI
+
+Submit `hive-forge pr-reviews <pr> --approve` or `--request-changes`
+as soon as you've finished checking the diff — don't hold it back
+waiting for CI to go green first. Mention CI's current state in the
+review body if it's relevant (e.g. "approving; `nix flake check` is
+still running"), but don't gate the formal verdict on it: CI isn't a
+signal a reviewer waits on, it's a separate condition that resolves
+independently.
+
+## What arming auto-merge actually means
+
+Auto-merge isn't "no human ever looked at this." Whoever arms it has
+already judged the PR sound at a coarse level — the signal it sends is
+roughly *"apart from maybe minor tweaks a reviewer can still catch,
+I think this is fine."* That's the human-in-the-loop step, and it
+already happened. No large changes are expected to surface after
+that point — a reviewer's job past that point is to flag it if one
+does, not to assume none ever will.
+
+The practical consequence for a reviewer: on a repo where auto-merge
+may already be armed before your review lands, a plain `APPROVED` can
+be the last step before the merge actually happens, with no further
+review pass after yours. That's a reason to actually finish checking
+before approving — not a reason to hesitate over every small thing.
+Genuine, substantive doubt (a claim you haven't verified, a real
+correctness question) is worth a `request-changes` or a clarifying
+comment before approving; a stray style nit isn't the same category.
+
+## Re-review after a repo requires it
+
+Where a repo dismisses stale approvals on a new commit, a review you
+already gave stops counting the moment a follow-up commit lands — even
+one whose message reads as trivial ("just a wording fix", "just
+trimming comments"). Re-diff and re-verify before submitting a fresh
+verdict; don't take a small-sounding commit message as an accurate
+description of the diff.

docs/scheduler/ci.md

@@ -0,0 +1,238 @@
+# hive-ci: Forgejo Actions Runner
+
+The `hive-ci` module runs a Forgejo Actions runner in a `hive-ci` nixos-container,
+executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.
+
+## For operators
+
+**Enabling it is one line**: `services.hyperhive.deploy.forgejo.ci.enable = true`
+in the host NixOS config. No manual token provisioning — hive-c0re
+registers the runner with the forge automatically.
+
+Two things worth knowing before you rely on it:
+
+- **Only run CI for contributors you trust.** Builds run **unsandboxed**
+  (an nspawn limitation, not a choice) — a malicious build script in a
+  PR could make network requests or write to the container filesystem
+  during the build. Fine for a small trusted-team hive where everyone
+  already has forge access; if you take outside/fork contributions,
+  gate CI behind Forgejo's fork-PR-approval setting or restrict the
+  workflow to push-only triggers (forks can't push to your branches).
+  See *Security* below for the full threat model.
+- **Watch your disk.** CI builds through the host's nix store with no
+  automatic garbage collection of its own — a busy CI day can fill
+  `/nix/store` until every job fails with `ENOSPC`. Add the daily +
+  disk-pressure GC config from *Host store maintenance* below to your
+  host's NixOS config (not optional if you plan to actually use this).
+
+Everything below this point is implementation detail: exact
+auto-registration internals, container design, and the full security
+writeup.
+
+## CI checks
+
+Three jobs run on every PR (and on `workflow_dispatch` for manual re-triggers),
+defined in [`.forgejo/workflows/ci.yml`](../.forgejo/workflows/ci.yml). All
+three are required checks (forge branch protection) — a hit on any of them
+blocks merge.
+
+| Job | What it runs |
+| --- | --- |
+| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
+| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
+| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
+
+`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
+without an empty commit.
+
+**`ci-rerun --pr` verifies the code but does not reliably move the PR's own
+status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
+JSON): a genuine PR-triggered run carries `event: pull_request` with the
+`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
+carries `event: workflow_dispatch` with the real branch name instead.
+Forgejo's PR commit-status tracking appears scoped to `pull_request`-event
+runs specifically, so a dispatched run — even a genuinely successful one —
+never writes to that status slot. Reproduced with three dispatches on one
+PR over 20+ minutes, all genuinely green, the PR's own status unmoved
+throughout. **Always re-check `pr-status` after a dispatch and believe
+what it says — don't push a commit just to unwedge it, that's the
+empty-commit anti-pattern this verb exists to avoid.** If the code is
+verified green (`hive-forge ci-log --run <n> --job 0`, verdict is the
+last line) but the status is stuck, ask the operator to click "rerun" in
+the forge web UI — it's CSRF-gated, so only they can do it.
+
+### Running checks locally
+
+Don't run `nix flake check` directly — it dispatches to the shared build farm and
+wastes a remote-builder slot. Use the devshell equivalents instead:
+
+```sh
+nix develop -c cargo clippy --all-targets -- -D warnings
+nix develop -c cargo test
+nix develop -c treefmt          # same as nix fmt; treefmt covers rustfmt + nixfmt + taplo
+sh scripts/check-issue-refs.sh  # tracker-tag lint
+sh scripts/check-comment-blocks.sh  # comment-block lint
+```
+
+A git pre-push hook that automates the two lint checks is provided at
+`scripts/pre-push`. Install it once per clone:
+
+```sh
+ln -sf ../../scripts/pre-push .git/hooks/pre-push
+```
+
+After that, any `git push` automatically runs both lints and aborts with a
+diagnostic if either fails — catching the issue locally before CI sees it.
+Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
+slow); run those manually before pushing Rust changes.
+
+## Configuration reference
+
+The internal forge is always present (mandatory), so the runner always has a
+hive-forge instance to register against — nothing extra to enable beyond
+`services.hyperhive.deploy.forgejo.ci.enable = true` (see *For operators* above).
+
+Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
+admin panel), `concurrency` (parallel job capacity), `labels` (workflow
+targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration
+string e.g. `"3h"` — a job that exceeds it is killed so a hung or runaway
+build can't hold the runner's single slot indefinitely).
+
+## Container design
+
+- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It cannot reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `forge.behindGateway = true`.
+- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
+- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`.
+- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only.
+
+## Auto-registration flow
+
+Registration is **off the container's boot-critical path** — hive-c0re owns
+it and runs it out of band, so a slow forge or core-token never delays the
+container's start. Gotcha: don't gate the container's own start on a forge
+round-trip (a host-side unit that did this could exceed the nspawn start
+timeout and trip a restart loop) — registration must stay something
+hive-c0re drives after the container is already up. The core admin token is
+held only by hive-c0re on the host; only the runner registration token
+reaches the container.
+
+### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)
+
+Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — failures are logged and never abort the sweep; a healthy runner is never restarted.
+
+1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
+   - **200**: still registered — done, no restart.
+   - **404 / other non-200 / malformed**: stale — re-register (below).
+   - **transport error** (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner.
+2. If absent or stale: mint a fresh token from `GET /api/v1/admin/runners/registration-token`, then hand it to hive-priv's `RegisterCiRunner`, which (as root) writes `TOKEN=<real>` **in place** to the host env-file `/run/hive-ci/runner-token` (preserving the inode nspawn pinned into the container at start) and restarts `gitea-runner-hive.service` inside the container so it picks up the credential and registers.
+
+### Container side
+
+- The container boots immediately — nothing gates its start on registration.
+- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
+- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
+- **Convergence**: because the token write targets the *host* file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
+
+## Actions checkout mirror
+
+When `deploy.forgejo.ci.enable` is set, hive-c0re auto-seeds an
+`actions/checkout` pull-mirror on the local forge and sets Forgejo's
+`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI
+`uses: actions/checkout@vN` steps resolve entirely on loopback — no
+external DNS on the CI critical path.
+
+The mirror is seeded by **hive-c0re** itself during its forge
+provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module
+forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the
+`hive-c0re` service environment (JSON-encoded `[{upstream, dest}]`
+list). hive-c0re already holds the admin token for the rest of the
+forge provisioning sweep (orgs, agent accounts, etc.), so mirror
+seeding lives in the same place rather than a separate host-side unit.
+
+**General-purpose mirrors**: you can pre-seed any external repo as a
+pull-mirror via `services.hyperhive.swarm.forge.mirrors`:
+
+```nix
+services.hyperhive.swarm.forge.mirrors = [
+  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
+  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
+];
+```
+
+Each entry is created as a real Forgejo pull-mirror — not a one-off
+clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
+request), so a DNS blip during that sync will propagate back to the
+runner as a hard `git clone` failure. The
+`<owner>` org in `dest` is auto-created. Keep mirror dests out of the
+hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
+to avoid provisioning collisions.
+
+## Security: unsandboxed builds and trusted contributors
+
+**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks:
+
+### What unsandboxed builds mean
+
+nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
+
+A malicious `default.nix` or build script in a PR can therefore:
+
+- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **cannot** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped).
+- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
+
+The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.
+
+Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
+
+### Mitigation
+
+For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.
+
+For repos with external contributors or fork PRs:
+
+- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
+- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
+
+The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.
+
+## Host store maintenance (recommended)
+
+The CI runner builds derivations through the **host** nix-daemon — the
+hive-ci container shares the host store and has no daemon of its own. Build
+outputs accumulate in `/nix/store` with no automatic collection, and a busy
+CI day can fill the disk until every job fails fast with `ENOSPC`.
+
+Store GC is a **host-level** concern, so it belongs in the host's own NixOS
+configuration, not in the hyperhive service modules — a single service should
+not reach out and change the host's global nix-daemon options. Add the
+following to your host config:
+
+```nix
+{
+  # Daily GC: delete store paths not referenced by a live root and older
+  # than a day. Keeps the store bounded between builds.
+  nix.gc = {
+    automatic = true;
+    dates = "daily";
+    options = "--delete-older-than 1d";
+  };
+
+  # Disk-pressure GC: when free space drops below min-free mid-build, the
+  # daemon collects garbage up to max-free before continuing. This is the
+  # real-time net the daily timer can't provide — a same-day build burst is
+  # what fills the disk. Tune to your disk size.
+  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
+  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
+}
+```
+
+**Remote builders:** if CI dispatches builds to a remote builder (e.g. via
+`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
+store, so the same GC config should be applied wherever the builder runs —
+GC on the coordinator host won't reclaim space on the builder.
+
+## References
+
+- `nix/host-modules/hive-ci.nix`: runner configuration, auto-registration script, container setup.
+- `.forgejo/workflows/ci.yml`: workflow definition.
+- `docs/gotchas.md`: nix sandboxing limitations in containers.

docs/scheduler/coordinator.md

@@ -0,0 +1,552 @@
+# hive-c0re coordinator internals
+
+Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems.
+For the public API surface (dashboard, socket protocol) see `docs/conventions.md`
+and `docs/persistence.md`.
+
+---
+
+## Job queue
+
+**For the job queue as a general idea — graph of steps, shared resource
+slots, no hive-c0re specifics — see [`jobq.md`](jobq.md) instead.** This
+section covers the concrete node inventory hive-c0re builds on top of that
+engine, plus the internals (module layout, scheduler mechanics,
+resource/lease semantics).
+
+Every container/meta operation (rebuild, meta-update, first-spawn, power
+changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
+as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
+concurrency comes from the resource classes below, not from multiple workers.
+Special cases like graceful stop, deferred starts, and the meta-update
+cascade need no bespoke code paths — each is expressed as a DAG *shape*
+built from the same primitive nodes.
+
+### Two levels: DAG and node
+
+The **DAG** is the unit of cancel / approval-resolution and the
+dashboard group; the **node** is the unit of scheduling / execution /
+build-log. Deps are intra-DAG edges only (`AfterOk` by default:
+the dep must succeed, a failed/cancelled dep cancels the dependent —
+cancel-downstream). Cross-DAG ordering comes from the per-agent lease,
+never from edges between DAGs.
+
+A DAG is **declared, not described**: a template builds it through
+`hive_jobq::JobBuilder`, naming each node it depends on via the handle
+`b.node(kind)` handed back, and the builder inserts the nodes itself. A handle
+only exists for a node already declared, so every edge points backwards and a
+cycle cannot be written down — there is no submit-time validation pass, because
+there is no malformed spec to reject.
+
+### Node inventory (primitives)
+
+Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
+
+| Node             | Wraps                                                                                                                                          |
+| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `Prebuild`       | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) |
+| `Swap`           | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
+| `Create`         | first-spawn provisioning + `nixos-container create` (atomic build+create)                                                                       |
+| `MetaLock`       | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
+| `DeployWindow`   | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
+| `DeployApply`    | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild                                         |
+
+Cheap — no build slot:
+
+| Node            | Behavior                                                                                                                             |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
+| `MergeVerify`   | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
+| `DeployTail`    | the deploy's `AfterAny` compensation + bookkeeping tail — rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good, then mirrors the config repo to the forge. Infallible by construction |
+| `MetaSync`      | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
+| `Reconcile`     | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
+| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped                               |
+| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here) |
+| `AgentWindow`   | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
+| `Signal`        | set the graceful fence + kick, so the harness runs one stop-checkpoint turn                                                           |
+| `Drain`         | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way                              |
+| `WriteDropin`   | `set_nspawn_flags` + `set_resource_limits` + daemon-reload                                                                            |
+| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots                |
+| `Reparent`      | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
+
+There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
+with its commit under its internal `META_LOCK` mutex, so a standalone commit
+node would open a dirty-working-tree window between nodes.
+
+Two further layers protect the meta repo across *windows* that span multiple
+`META_LOCK` acquisitions — above all the approval deploy's prepare→finalize
+span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
+container build:
+
+- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
+  resource declared by every node kind that mutates the meta repo — `MetaSync`,
+  `MetaLock`, `WritePermFile`, `Reparent`, `Provision`'s agent registration, and
+  `DeployWindow` — the deploy subtree's root, which holds it across every
+  phase below it (it declares `Resource::MetaWindow`). Two meta
+  mutations can therefore never interleave, so no commit lands inside another
+  node's staged window. It is a queue resource rather than a runtime mutex
+  because a resource is held by a subtree root across its whole subtree, which
+  a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
+  multi-node deploy own one window. For the same reason the window must stay
+  *off* long store-only work: the rebuild's meta preamble is its own
+  `MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
+  toplevel build runs outside the window and `buildSlots > 1` still gives
+  concurrent rebuilds across agents.
+- **Path-limited commits**: the targeted meta committers (perm files,
+  topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped
+  dirty checks, so even a non-queue caller (boot migration, destroy's
+  `sync_agents`) can never sweep someone else's staged content into its
+  commit.
+
+### Every operation as a DAG
+
+The `stop` / `start` power ops write the durable `wanted` intent via a head
+`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
+so intent-write + reconcile is atomic per-agent. `restart` is the exception:
+it writes *no* intent (no `SetWanted` head) — it bounces the container and
+lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
+deliberately-stopped agent is not forced back up by a hive-wide restart. The
+hive-wide power ops — `restart`, `stop`, and `start` — take an agent *list*:
+a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
+subgraph each (independent roots, run concurrently on their own leases), not
+N separate DAGs.
+
+**These are built dynamically from each agent's live running state** (an
+async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
+not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
+a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
+(convergence guarantee — cheap, noops when already converged) is ALWAYS
+present; only the *mechanical* nodes (`Signal`/`Drain`/`StopForUpdate`) are
+state-conditional — skipped for a *down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
+closes the TOCTOU window: if an agent flips state between the `is_running`
+read and node exec, the tail `Reconcile` still converges it in-DAG (with
+`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
+sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
+agent's subgraph is a rebuild-then-start).
+
+```text
+rebuild(a):        MetaSync(a) → AgentWindow(a){ Prebuild(a) ∥ [Signal(a)→Drain(a) if graceful]; both →(after-ok) StopForUpdate(a) → Swap(a) →(after-ok) RebuildBookkeeping(a) } →(after-any) Reconcile(a)
+stop(a..):     online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
+               offline a: SetWanted(a,Off) → Reconcile(a)                    (N subgraphs, 1 DAG)
+restart(a..):  online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a)  (no SetWanted)
+               offline a: Reconcile(a)  (nothing to stop; Reconcile converges to existing wanted)
+start(a..):    a: SetWanted(a,Up) → Reconcile(a)   (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
+spawn(a):          [wanted=Up at approve]  Create(a) → WriteDropin(a) → Reconcile(a)
+perm-change(a):    WritePermFile(a) → «rebuild subgraph»
+meta-update(inp):  MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent
+boot:              (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
+                   plus Reconcile(a) for every drifted agent  (all ONE DAG)
+reparent(moves):   Reparent(moves)   (no rebuild — topology.json is read live)
+```
+
+Notable collapses:
+
+- **`rebuild` is one uniform shape** — no `was_running` branch.
+  `StopForUpdate` noops when already down; the tail `Reconcile` auto-noops the
+  start when `wanted = Offline` (a rebuild of a deliberately-stopped agent
+  leaves it stopped).
+- **The swap-failure recovery-start is structural**: `Reconcile` deps on
+  `Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
+  terminal ok *or* fail, bringing a wanted-up agent back on its old config.
+- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
+  next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
+- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
+  whole-hive graceful stop fires every agent's signal immediately and all
+  drains overlap; each DAG's tail `Reconcile` does the actual stop.
+- **The meta-update cascade grows in the same DAG on completion**:
+  `MetaLock`'s executor computes the affected agent set after the bump lands
+  and grows one `rebuild` subgraph per agent into its *own* DAG via
+  `append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
+  doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
+  bump appends nothing (no cancel-children dance). Same shape as the startup
+  sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
+  agent keeps crash-watch suppression during its `Swap`.
+
+### Desired-state (spec vs status)
+
+Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
+`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
+`container_view` remains the observed *status*; `Reconcile` nodes converge the
+two. Setting `wanted` is never a queued node: the power layer
+(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
+whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
+Power toggles never commit to the meta repo. Every operator power surface —
+dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
+rides the queue through that power layer, so intent, lease serialization,
+and crash-watch suppression can't drift per surface; the only direct starts
+left are the root-agent bootstrap and infra containers (no lease, no
+harness). Cancelling a still-queued power DAG reverts `wanted` to the
+observed state — a cancel means "don't do it", not "do it later". Agents
+without a row are seeded from observed state on first touch (running ⇒
+`Up`); destroy removes the row.
+
+The admin-socket responses carry the submitted DAG ids; `hivectl` polls
+`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
+glyph, template, agent, node chain — so
+CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
+non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild
+subgraphs, a `Reconcile` fanning its `Start`/`Stop`) join the same DAG, so
+they surface under that DAG's id in the same loop — no separate child DAGs.
+
+### Scheduler semantics
+
+A node is **ready** when it's `Queued`, every dep is satisfied, and its
+resources are free. Resources:
+
+1. **Build slots** — `services.hyperhive.c0re.buildSlots` permits (default 1),
+   held by nix-heavy nodes for the node's duration.
+2. **Per-agent lifecycle lease** — keyed on the **node's** agent (agent is
+   per-node; a DAG can span agents) and globally exclusive per agent across
+   all DAGs: acquired either at a container-affecting node (`SetWanted`,
+   `Reconcile`, `WriteDropin`, `Create`) or at a **brace** (`AgentWindow`,
+   `DeployWindow`) on behalf of a whole coordinated subtree; held by the owning
+   DAG until it's terminal, so two DAGs never interleave container ops on the
+   same agent. A DAG touching several agents holds one lease per agent.
+   (`SetWanted` is a store write, not a container op, but takes the lease anyway
+   so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
+   clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
+   `Prebuild`, `MetaLock`, `WritePermFile`, `Reparent` — they touch the store /
+   meta, not the running container, which is exactly why a stop can land while
+   another DAG's prebuild is still building. Also exempt, for a different
+   reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
+   `Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
+   container, but their `AgentWindow` brace holds the lease above them — see
+   _Braces_ below.
+
+#### Braces
+
+Templates otherwise declare a resource on **every** node that needs it, even
+when a parent already holds it, so the requirement belongs to the node rather
+than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
+exception: a pure-resource-holder root that declares on behalf of a subtree
+coordinated with itself, whose members then declare nothing.
+
+It is forced rather than stylistic. Declaring a resource means *"I need this
+exclusively"*, and the agent lease is single-unit — so **two siblings that both
+declared it could never run concurrently.** For a subtree whose whole point is
+concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
+the requirement truthfully on every node and running those nodes in parallel are
+mutually exclusive. One holder above them speaks for the subtree.
+
+This is the opposite of the failure the declare-your-own rule exists to prevent,
+not a relapse into it: there the requirement was *implicit*, inferred from a
+node's kind and true only by accident of placement. Here it is explicit, on one
+node, with the omission below it documented on the brace itself.
+
+Two consequences worth knowing:
+
+- **Flattening a chain under a brace is safe.** The stop chain used to nest
+  `Signal` over `Drain` over `StopForUpdate` specifically so the lease stayed
+  continuous — as independent siblings each would acquire it separately and
+  leave a gap another DAG could claim the agent in, mid-bounce. A brace supplies
+  that continuity directly, so the nesting is no longer load-bearing.
+- **Observability is unaffected.** `running_transients` keys off a node's
+  *payload* agent, not off a declared lease edge, so every child still lights its
+  own dashboard pill and still reports its own `takes_container_down` to the
+  crash watcher. A brace itself reports `false`: it parents the stopping nodes
+  but does not stop anything, and claiming otherwise would widen crash
+  suppression across the build and tail, where a vanished container is still a
+  real crash.
+
+Among simultaneously-ready nodes competing for a resource, DAG-submit order
+wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
+DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
+created on lease acquisition and dropped when the DAG settles terminal.
+
+The queue is in-memory only and lost on hive-c0re restart — deliberate:
+desired state is re-derived at boot from the DB + rev markers (see _Boot
+reconcile_), so there is no durable-recovery machinery to go wrong.
+
+### Cancel, history
+
+Submit-time dedup was removed with the agent-per-node move (a multi-agent DAG
+has no single agent to key a dedup on), so every submit enqueues a fresh DAG;
+whether any dedup needs reintroducing is tracked as a follow-up.
+
+Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't
+interruptible) — each op is one DAG now, so there are no child DAGs to cascade to.
+Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
+`Cancelled` / `Done`. The snapshot retains the 50 most recent terminal
+DAGs — a flat cap over the whole sorted list, not per template, since
+the dashboard renders one recent-builds list and one number bounds it.
+
+### Approvals
+
+`MergeConfigPr` approvals ride as a four-node deploy subtree:
+
+```
+DeployWindow (root — build slot + lease + meta window, no work of its own)
+├── MergeVerify                     drift gate, fetch, verify_commit
+├── DeployApply    AfterOk(verify)  park rollback ref, ff-merge, deploy
+└── DeployTail     AfterAny(apply)  compensate, mirror to forge
+```
+
+The root's resources are held across the whole subtree, so the two-phase
+`prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock`
+protected even though the phases are separate nodes. Splitting them buys
+three things a single opaque node couldn't have: per-phase visibility on the
+dashboard, a `MergeVerify` failure that provably mutated nothing, and a
+compensation step that survives a hive-c0re restart — the pre-merge
+`applied/main` is parked in `refs/hyperhive/rollback/<approval-id>`, not in a
+local variable, so `DeployTail` can still undo a half-finished deploy after a
+crash.
+
+`Spawn` and `UpdateMetaInputs` approvals map onto the ordinary `spawn` /
+`meta-update` shapes. The scheduler fires `actions::resolve_approval_dag`
+exactly once when **any** approval-carrying DAG settles terminal — deploys
+included, since their outcome is now the DAG's own state (including
+cancelled-while-queued, which fails the approval instead of dangling it).
+
+### Wire shape
+
+`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
+payload.** It says *the queue changed*; a client that wants to know how
+re-fetches `GET /api/jobq/graph`.
+
+That endpoint serves the graph generically (`hive-jobq-wire`): every node
+carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
+string — the kind *is* the phase label, there is no separate sub-step
+string) and free-form `data` for what only some kinds have (`agent`,
+`approval_id`, `inputs`, `build_log_id`). Group roots ride as ordinary
+nodes, so a group's state is just the root's own `state`.
+
+There is **no group-level `agent`** — agent is per-node, so one group can
+span agents; consumers derive a group's agent(s) from its nodes. Build logs
+are likewise **per-node**: the dashboard renders the node tree and keys the
+live-log panel off the running node.
+
+The event carries no payload by design: shipping a typed projection of
+the whole queue in the event itself would be a second rendering of the
+same graph that has to be kept in agreement by hand with the endpoint
+every consumer actually reads. Telling a client *when* to refetch is
+the event's whole job.
+
+---
+
+## Container view
+
+`container_view.rs` maintains an in-memory snapshot of every nixos-container's
+systemd service state. It is polled on coordinator startup and re-scanned after
+every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects
+the actual container status without a live `nixos-container list` call on each
+render.
+
+---
+
+## Boot reconcile
+
+On startup, `auto_update::run` classifies every agent by rev freshness (the
+per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
+vs the current flake path) and persisted `wanted` intent, then:
+
+1. **Config path** — when *any* marker is stale, submit one `Boot`
+   DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
+   `Rebuild` subgraph for each stale agent whose `wanted = Up` (topology-sorted,
+   parents first). Stale but wanted-offline agents get no boot-time nix work — their
+   rebuild happens on their next start (the start submit path upgrades a
+   stale start to rebuild+start), which is also why the lock bump runs even
+   when every stale agent is offline: those later start-upgrades must build
+   against the bumped lock. Each child rebuild's tail `Reconcile` brings the
+   agent (back) up, covering both the running-stale and stopped-but-wanted-up
+   cases.
+
+2. **Power path** — every agent whose observed state drifted from `wanted`
+   gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
+
+Booting with no config change performs **no meta commit** — only reconciles.
+The sweep reason records the rebuild / deferred / up-to-date counts so the
+operator sees at a glance how much work the boot triggered. Agents without an
+`agent_power` row are seeded from observed state during classification (the
+one-time migration; thereafter the DB is authoritative).
+
+## Meta flake
+
+`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`.
+This flake consumes every agent's applied config repo as a flake input and exports
+one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so
+meta's git log is the system-wide deploy audit trail.
+
+Key operations:
+
+- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set,
+  init the repo on first call, relock if the rendered contents changed, commit.
+  Called by spawn / destroy / startup migration.
+- **`prepare_deploy` + `finalize_deploy` / `abort_deploy`** — two-phase for the
+  `MergeConfigPr` deploy path so a failed `nixos-container update` leaves no orphan
+  commit in meta. Prepare writes the new lock without committing; finalize commits
+  with the deploy message; abort restores the lock.
+- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
+  sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
+  the scheduler fans out the agent rebuilds on completion.
+
+Every public `meta.rs` operation takes the module's internal `META_LOCK`
+mutex, so concurrent job-queue nodes (and the approval deploy pipeline) never
+race on the repo's `.git/index.lock`.
+
+---
+
+## Container lifecycle (`lifecycle.rs`)
+
+Every container operation ultimately calls into `lifecycle.rs`. Two paths exist:
+**rebuild** (existing container) and **spawn** (first-time creation).
+
+### Rebuild path (existing container)
+
+Goal: apply the new system profile and any `EXTRA_NSPAWN_FLAGS` / drop-in changes
+in a single start, with minimum downtime.
+
+`nixos-container update` only runs `systemctl reload container@<c>` when the
+container is already up (per `isContainerRunning` in `nixos-container.pl`). Stopping
+first turns `update` into a boot-style operation: it builds + `nix-env --set`s the
+new profile and skips the in-container `switch-to-configuration`. The subsequent
+`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
+one go, rather than the double-bounce a live `update` would trigger.
+
+Sequence for a rebuild DAG (each step is its own queue node):
+
+1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
+   meta-update cascade child) the per-agent relock. Short, and the only step
+   that mutates the meta repo, so it is the only one holding the global deploy
+   window.
+2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
+   The container keeps serving the previous generation while eval + fetch +
+   build happen out-of-band. `nixos-container update` then finds the result
+   cached and skips straight to the profile-swap. Build failures surface
+   here, before the running container is touched. (Runs even for a stopped
+   container — same total nix work, one uniform DAG shape.)
+3. `StopForUpdate` — bring the container down (noop when already stopped).
+4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
+   (near-instant after the prebuild).
+5. `Reconcile` — boot into the new generation when `wanted = Up`; the
+   in-container activation script transitions old → new. Holds no build
+   slot, so the next DAG's `Prebuild` overlaps the container boot.
+
+The approval deploy uses this same chain rather than a rebuild path of its
+own. Its `DeployApply` node does not build: it merges, opens the two-phase
+meta deploy, and returns the chain above as a subgraph the scheduler grafts
+into the live DAG under that node. A `FinalizeDeploy` node gated on the
+graft's completion then plants the deploy tag — so "did the agent come back
+up?" is answered by `Reconcile` succeeding, the same way it is for every
+other rebuild, instead of by a fused inline start.
+
+The grafted nodes land *inside* `DeployWindow`'s subtree, so they re-enter
+the meta window and build slot it already holds rather than deadlocking
+against it.
+
+### Cold-start fallback
+
+`start` after `update` can exit non-zero when packages are **removed** between
+generations: the old-generation activation script references units that no longer
+exist in the new closure, causing systemd to exit non-zero. The container may be
+half-started at that point.
+
+Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
+→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
+Both errors are preserved and surfaced if the cold-start also fails. The fallback
+lives in `lifecycle::start_with_fallback`, used by every `Reconcile` node's
+start action.
+
+### Spawn path (new container)
+
+For a first-time `create`, `nixos-container create` is atomic: if the build fails,
+no container record is left to clean up. A separate prebuild would just duplicate
+the eval, so it's skipped. Sequence: `create --flake meta#<name>` → write nspawn
+flags → `systemctl daemon-reload` → `start`.
+
+### Prebuild attr path
+
+`nix build` does not auto-resolve `meta#<name>` against `nixosConfigurations` the
+way `nixos-container` does internally. The explicit attr path
+`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel` is required;
+using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
+or the flake root directly — none of which exist in the rendered meta flake.
+
+---
+
+## Host-level resource + performance options
+
+A handful of `services.hyperhive.c0re.*` options tune container resource
+limits, build parallelism, and first-spawn latency.
+
+### Build slots
+
+`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
+(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
+concurrently. The default serializes all heavy nix work like the pre-DAG
+rebuild queue did; raise it on hosts with the cores/RAM to build several
+agent toplevels at once. Per-agent correctness is independent of the count —
+each agent's container-affecting ops serialize on its lifecycle lease
+regardless.
+
+### Container resource limits
+
+`agentCpuQuota` and `agentMemoryMax` map directly to systemd
+`CPUQuota=` and `MemoryMax=`. hive-c0re writes a
+`container@h-<name>.service.d/` drop-in file on each spawn and
+rebuild, so changes take effect on the next lifecycle op without
+requiring a host rebuild.
+
+The same drop-in carries `CPUWeight=` / `IOWeight=` from
+`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
+setting: the quota and the memory max are **hard caps** that throttle
+an agent even on a completely idle host, while the weights are cgroup
+v2 **relative shares** that only decide who yields *under contention*.
+A low-weight container still gets the whole machine when nothing else
+wants it.
+
+| Option                                   | Default  | Description                                                                                                                          |
+| ---------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
+| `services.hyperhive.c0re.agentCpuQuota`  | `"200%"` | CPU cap per agent, as a percentage of one core (`"200%"` = 2 cores). Raise if agents hit CPU limits during builds or heavy tool use. |
+| `services.hyperhive.c0re.agentMemoryMax` | `"4G"`   | Memory cap per agent. Raise for agents that run large nix builds or hold big in-memory data.                                         |
+| `services.hyperhive.c0re.agentCpuWeight` | `80`     | `cpu.weight` share per agent, `1`–`10000` or `null` to omit the setting. Kernel default is `100`, so `80` makes agents yield.        |
+| `services.hyperhive.c0re.agentIoWeight`  | `80`     | `io.weight` share per agent, same range and `null` handling. See the caveat below — it is a no-op on many hosts.                     |
+
+Two things to know about the weights:
+
+- They are **hive-wide** — unlike the caps there is no per-agent
+  override in `meta/resource-limits.json`, so every agent carries the
+  same value and the weight does *not* rank agents against each other.
+  What `80` buys is that agents yield to everything **not** on this
+  drop-in path: host services (nginx and dnsmasq among them) and the
+  infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,
+  `hive-matrix`), which stay at the kernel default of `100`.
+- `IOWeight=` is only honoured when the backing device runs the BFQ
+  scheduler or has blk-iocost QoS enabled. On a host using
+  `none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
+  and the kernel ignores it — harmless, but it will measure as nothing.
+  Check with `cat /sys/fs/cgroup/io.cost.qos`, and set the option to
+  `null` if you would rather not write a setting nothing reads.
+
+For a hive-wide cap across all containers together, set
+`systemd.slices.machine.serviceConfig.CPUQuota` in your NixOS
+config — all nspawn containers live in `machine.slice`.
+
+### Pre-building agent templates
+
+`preBuildAgentTemplates` (default `false`) causes the host NixOS
+build to pre-fetch the per-container system closures
+(`agent-base` + manager toplevels) into `/nix/store`, instead of
+leaving that work to the first `nixos-container start`. The
+trade-off:
+
+- **On** (recommended for x86_64 hosts that care about first-spawn
+  latency): the first `nixos-container start` for any new agent
+  completes in seconds because nothing is left to fetch. Cost:
+  the full nixpkgs runtime closure + claude-code + the harness
+  binary are added to the host system closure (low single-digit GB
+  additional).
+- **Off** (default): the host closure stays lean; the first spawn
+  does all the eval + fetch work at runtime (can take several
+  minutes on a fresh store).
+
+**Note**: toplevels are pinned to `x86_64-linux`. Enabling on an
+`aarch64` host forces a cross-compilation or remote-builder build,
+which is almost never desired. Leave off on non-x86 hosts.
+
+---
+
+## See also
+
+- `docs/approvals.md` — approval flow + scheduled prompts
+- `docs/persistence.md` — SQLite schema, state-dir layout
+- `docs/conventions.md` — wire protocol, recipient sentinels
+- `docs/agent-hierarchy.md` — topology and parent/child relations

docs/scheduler/jobq.md

@@ -0,0 +1,52 @@
+# The job queue, for operators
+
+Every container operation — rebuild, first-spawn, a config-PR deploy,
+power changes — runs through one shared job queue. This page explains
+what the job queue *is*, as a general idea, independent of what any one
+subsystem uses it for. For the hive-c0re-specific step catalogue and the
+engineering internals (scheduler, leases, resource windows) see
+[`coordinator.md`](coordinator.md) instead.
+
+## What the job queue is, in the abstract
+
+"jobq" is a generic engine for running many interdependent jobs under
+limited concurrency — it has no idea what a "container" or a "rebuild" is.
+Two ideas are all there is to it:
+
+- **A job is a small graph of steps**, not one opaque blob. Steps can
+  depend on each other (this step only starts once that one finishes), so
+  a big operation is really a short, ordered sequence — not a single
+  black box that's either "done" or "not done."
+- **A step can need a shared resource**, which only so many steps can hold
+  at once (a "slot"). If every currently-running step already holds the
+  slots it needs, a new step that wants the same one waits its turn —
+  that's the whole reason things queue instead of all firing at once.
+
+The engine's whole job is: whenever a step's ordering and resource needs
+are both satisfied, run it. It has no opinion on what the steps *do* —
+that's supplied by whoever builds the graph. hive-c0re is the one thing
+building graphs on it today, but nothing about the engine is specific to
+containers or rebuilds; there's nothing stopping another subsystem from
+using the same engine for its own unrelated queue.
+
+## Watching it happen
+
+Each **row** you see in a queue view (the **BU1LDS** page's R3BU1LD QU3U3
+— see [`web-ui/dashboard.md`](../web-ui/dashboard.md) — and swarm-ui's
+`/jobs` page both render the same underlying graph) is one job; the rows
+nested under it are that job's steps, in order (occasionally a couple run
+side by side). A step shows one of:
+
+| Glyph | Meaning                                                 |
+| ----- | -------------------------------------------------------- |
+| `⏸`   | queued, waiting its turn                                  |
+| `▶`   | running                                                   |
+| `◐`   | its own work is done, waiting on a step nested under it   |
+| `✔`   | finished successfully                                     |
+| `✖`   | failed                                                    |
+| `⊘`   | cancelled                                                 |
+| `·`   | skipped (not needed for this run)                         |
+
+A step that isn't needed for a given run shows as `·` rather than being
+left out of the tree entirely, so the same kind of operation keeps a
+recognizable shape run to run, whichever steps it actually needed.

docs/scheduler/observability.md

@@ -0,0 +1,387 @@
+# Observability (OpenTelemetry)
+
+hyperhive has built-in support for exporting per-agent Claude Code statistics —
+token usage, cost, tool call counts — to any OTLP-compatible collector via
+Claude Code's built-in OpenTelemetry integration.
+
+This is a **hive-wide** setting: one switch in the host NixOS config enables it
+for every agent container simultaneously. There is no per-agent opt-in or opt-out.
+
+## Enabling export
+
+```nix
+services.hyperhive.otel = {
+  enable   = true;
+  endpoint = "https://collector.example.com/otel";
+};
+```
+
+`enable` is the single gate. `endpoint` is where telemetry ends up after it
+leaves the swarm — optional, because the swarm's own metrics store
+(`deploy.victoriametrics`) is a destination in its own right. With both,
+telemetry goes to both. See
+[`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana).
+
+**There is exactly one way telemetry leaves a hive: through the collector that
+`enable` starts on the host.** Agents never talk to `endpoint` themselves —
+they export unauthenticated to a bridge address only their own containers can
+reach. That collector forwards to the swarm's
+([`swarm/services.md`](../swarm/services.md#telemetry-collector-otel)), which is
+the single process holding the upstream credential and the only writer to the
+swarm's store. No agent holds a copy, and neither does this hive.
+
+The hive collector reaches the swarm collector by its gateway name
+(`swarm.otel.domain`, default `otel.<swarm domain>`) — the same DNS-and-CA-trust
+shape every hive-to-swarm-service hop uses, not a URL an operator has to point
+anywhere. A hive that does not run the swarm's services still resolves that
+name through the gateway; nothing here needs setting for the split-host case.
+
+⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
+same host as the agents and restarts on failure, and telemetry is not the
+control plane — degraded telemetry is not degraded operation — but the export
+no longer survives independently of anything host-side.
+
+### what the agent→collector hop is and isn't
+
+**It has no application-level auth.** The receiver takes any OTLP that reaches
+it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
+port on the bridge interface only. So "unauthenticated to a bridge address"
+means *reachable from an agent container*, not *presents a credential*.
+
+The consequence, stated because it is a choice rather than an oversight: **any
+agent can push arbitrary OTLP, and it is forwarded on under the operator's
+credential.** Neither tier can tell a container's genuine Claude Code stats
+from anything else shaped like OTLP arriving on that port — including data
+smuggled out in resource attributes on an otherwise-legitimate export.
+
+That is a **different risk from the one the collector fixes**, and strictly
+smaller than what preceded it: before, every agent held the upstream credential
+itself, so it could do all of the above *and* use the token anywhere else. The
+collector removes the token and keeps the pipe. Agents are inside the trust
+boundary (`docs/security.md`: capability = accepted risk), so an agent being
+able to *send* is an accepted extension of that boundary — but it is not
+closed by this design, and nothing here should be read as closing it.
+
+**The `agent` label is self-reported, and no planned authentication changes
+that.** Treat it as a convenience for grouping dashboards, never as evidence of
+which container produced a sample: any agent that can reach this hive's
+collector can label its data as any other agent.
+
+Worth spelling out, because two different hops are in play and only one of them
+is getting a credential:
+
+- **agent→collector** (this section's hop) stays open on the bridge. Nothing
+  downstream can tell one agent's export from another's.
+- **hive→swarm** is where the planned ingest auth goes. The swarm tier stamps
+  `hive=` from the connection it authenticated, so *that* label becomes
+  unforgeable.
+
+So a verified `hive` is reachable and a verified `agent` is not — and that falls
+out of the topology rather than being a gap someone forgot to close. The swarm
+runs one collector, and the mechanism gives it no finer grain: a bearer-token
+check never reveals *which* token matched, and a receiver reads request metadata
+rather than the claims it authenticated with.
+
+If you need per-agent numbers you can act on, take them from the agent's own
+turn-stats rather than from a metric label.
+
+## Options reference
+
+Every `services.hyperhive.otel.*` option's full type/default/description/
+example is generated straight from the nix module (`nix/host-modules/
+otel.nix`) into [`/options/`](/options/) (host options — `nix build
+.#docs-host` for a local render). That page is kept honest by the build in a
+way a hand-copied version here cannot be, so it is the reference, not this
+doc. What follows is what a flat per-option listing can't express: the
+two-tier architecture, the security model, and how the options interact.
+
+## The two collectors
+
+Telemetry crosses two collectors, and which one you configure depends on what
+the host is:
+
+| | runs where | receives from | does |
+|---|---|---|---|
+| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
+| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
+
+An all-local host runs both, and needs nothing said about the hop between them.
+
+```nix
+services.hyperhive.otel = {
+  enable            = true;
+  endpoint          = "https://collector.example.com/otel";  # the upstream
+  headersCredential = "/run/secrets/otel-headers";           # only the swarm tier reads it
+};
+```
+
+**Why the hive tier isn't optional.** Exporting straight to `endpoint` means
+every agent needs the credential to authenticate — and the harness delivers
+that token into the agent's own `~/.claude/settings.json`, a file the agent can
+read. `0600` protects it from other containers, not from the agent itself. As
+long as the direct path stays *selectable*, that hole stays selectable; an
+option that can reintroduce it is a hole with extra steps.
+
+**Why the tiers stay separate on one box.** They are not collapsed when
+co-located: an all-local hive is a statement about *where* processes run, not
+about the shape of the deployment. A boundary that disappears locally is one
+the local deployment stops testing.
+
+**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
+redefines it — the agent-facing value is *derived*
+(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
+keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
+automatically; there is nothing to open by hand.
+
+### Authenticated ingest
+
+The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
+from whichever receiver accepted a sample. A hive therefore cannot report
+metrics as another hive, and cannot relabel its own by editing what it sends —
+the label is not taken from the payload at all.
+
+**On an all-local swarm there is nothing to set.** Each hive already has an
+identity, and its collector reads the secret that host's own authelia minted.
+
+**On a hive that does not host the swarm's services**, the secret has to arrive
+somehow — copy it across and name it:
+
+```nix
+services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";
+```
+
+**There is no unauthenticated mode.** A hive always presents an identity, so a
+missing credential is a build error rather than a quieter fallback — the
+collector has no anonymous route to accept samples on, and every path it serves
+belongs to exactly one hive.
+
+Getting the secret wrong shows up as the hive's collector logging 401s from the
+swarm tier and no metrics appearing for that hive.
+
+⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
+`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
+exporter takes an *address*: `https://host/path` is a legal
+`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
+address"*. The collector's config is validated at build time, so a mismatch is
+a build error naming the reason rather than telemetry silently going nowhere.
+
+## Network access
+
+Agent containers can only reach the host on ports 80 and 443 by default. To let
+them reach some other host-local service you run yourself — a database, a
+scratch HTTP endpoint — open its port on the bridge:
+
+```nix
+services.hyperhive.network.exposeHostPorts = [ 5432 ];
+```
+
+and point whatever consumes it at `10.42.0.1:5432` rather than loopback: inside
+a container, loopback is the *container*. The bridge IP is the host's address on
+the `hive-br0` bridge. The service must also bind an address the bridge can
+reach — a `127.0.0.1`-only listener stays unreachable no matter what the
+firewall allows. See `docs/network.md::Reaching host services` for details.
+
+⚠️ **None of this is needed for hyperhive's own telemetry** — `otel.enable`
+contributes the collector's port and derives the agent-facing endpoint itself.
+
+## Built-in resource labels
+
+The OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
+`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) are set **container
+wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every
+process in an agent container exports to the hive's collector without any
+per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
+yourself from a tool call or `hivectl shell`.
+
+Every agent's export therefore includes these resource attributes
+automatically:
+
+| Attribute | Value |
+|-----------|-------|
+| `service.name` | `hyperhive-agent` (constant) |
+| `agent` | agent logical name (e.g. `iris`) |
+| `hive` | hive display name (`services.hyperhive.hiveName`) |
+| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
+
+Additional labels can be appended via `extraResourceAttributes` (see option
+reference above); custom per-data-point labels can be passed with
+`hive-metric --labels` (see below).
+
+## Host-emitted container-resource metrics (hive-c0re)
+
+When OTEL is enabled, **hive-c0re itself** also exports each agent
+container's resource load — the same cgroup gauges shown on the dashboard
+LOAD tab — to this hive's own collector, exactly like an agent does and with
+no separate toggle. These come from the host, not the in-container Claude SDK,
+so they cover containers even when their agent is idle.
+
+Emitted via the OpenTelemetry Rust SDK, using the
+[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
+metric names + the standard `container.name` attribute where a spec metric
+exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
+`service.name = hyperhive-c0re`; each data point is tagged `container.name`
+(= the `h-<agent>` machine) and the hive `agent` label:
+
+| Metric | Unit | Kind | Source |
+|--------|------|------|--------|
+| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
+| `container.memory.usage` | `By` | gauge | `memory.current` |
+| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
+| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
+| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
+| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
+
+The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
+limit + peak, on-disk footprint, and an instantaneous cpu percent kept
+alongside the spec `container.cpu.time` counter for convenience). Hive
+labels (`hive`, `swarm`, …) ride on the resource via
+`extraResourceAttributes`.
+
+Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP
+(JSON) to the hive collector's bridge address, with no auth header — that
+first hop is unauthenticated for every producer on this host, and the upstream
+credential stays on the swarm tier.
+
+## Agent-emitted per-turn metrics (`hive-agent`)
+
+When OTEL is enabled, the harness itself (`hive-agent`) exports one small set
+of metrics per claude turn, recorded the moment the turn ends (not polled).
+These are deliberately the fields Claude Code's own built-in export (see
+above) can't know about — the harness's own wall-clock timing, what woke the
+turn, its own outcome classification, the loose-ends backlog, and session
+boundaries. Token usage, cost, and tool-call counts are **not** duplicated
+here; that's already covered by Claude's own export.
+
+| Metric | Unit | Kind | Attributes |
+|--------|------|------|------------|
+| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
+| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
+| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
+| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
+| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
+| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
+
+Resource attributes (`service.name`, `agent`, `hive`, `swarm`) come from the
+same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
+section — nothing extra to configure. Cadence follows
+`HYPERHIVE_OTEL_METRIC_INTERVAL_MS` (default 60s, same variable + default as
+`hive-c0re`'s container-resource export above) — that only controls how often
+the batched points are flushed to the collector, not how often they're
+recorded (every turn, always).
+
+## Hive-scoped metrics (hive-c0re)
+
+Everything above is measured **per agent**, tagged with the hive it runs in.
+These three are measured per **hive**, and carry no `agent` label — so a hive
+that hosts no agents still reports, and "this hive is quiet" is
+distinguishable from "this hive is gone". Select them with
+`{hive!="",agent=""}`.
+
+| Metric | Unit | Kind | Meaning |
+|--------|------|------|---------|
+| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
+| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
+| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
+
+Both levels are reported every cycle, `0` included, so a healthy hive is
+visible as zeros rather than as missing series.
+
+`hyperhive.hive.degraded` is what a dashboard should alert on: it is
+`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
+and with what the swarm controller sees. `hyperhive.hive.warnings` is the
+detail behind it — `warn`-level entries mean "an operator should look" and do
+**not** set `degraded`.
+
+Same cadence, transport and resource labels as the container metrics above.
+
+## VCS activity metrics (`swarm-controller`)
+
+`swarm-controller` registers a single instance-wide Forgejo webhook (a
+"global/system" hook, not scoped to any one org or repo) and counts commit
+and push activity as deliveries arrive — occurrence-driven, not polled.
+Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
+counts of durable rows (issues, comments, repos), and neither a commit nor a
+push is stored anywhere as a row to count.
+
+| Metric | Unit | Kind | Attributes |
+|--------|------|------|------------|
+| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
+| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
+
+A push with zero commits (a branch delete, or a force-push that doesn't add
+new commits) still increments `push.count`; `commit.count` only advances
+when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_OTLP_ENDPOINT`), cadence variable
+(`HYPERHIVE_OTEL_METRIC_INTERVAL_MS`) and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`
+resource-attribute channel as `swarm-controller`'s other OTEL exporter (its
+`hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that
+crate's own doc comment) — `service.name = swarm-controller` is set
+directly rather than read from the container environment, since
+`swarm-controller` is a standalone daemon, not a per-agent harness process.
+
+## Agent-emitted custom metrics (`hive-metric`)
+
+Agents can push arbitrary labeled metrics to the same OTEL collector via the
+`hive-metric` CLI tool, available in every agent container when
+`services.hyperhive.otel.enable = true`.
+
+### Usage
+
+```text
+hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...]
+```
+
+- `<name>` — metric name (e.g. `tasks_completed`, `latency_ms`).
+- `<value>` — numeric value (f64; integers and floats both accepted).
+- `--type counter|gauge` — metric kind: `counter` (increasing sum, default) or
+  `gauge` (instantaneous point-in-time value).
+- `--temporality delta|cumulative` — counter reporting mode (`counter` only,
+  ignored for `gauge`): `delta` (this call's own contribution, default — send
+  `1` each time and the collector accumulates) or `cumulative` (this call
+  reports the running total, which a stateless one-shot CLI can't track
+  itself).
+- `--labels key=value` — extra per-data-point labels. May be repeated.
+  The resource labels (agent, hive, swarm, service.name) are inherited
+  automatically from `OTEL_RESOURCE_ATTRIBUTES` — do not re-specify them.
+
+### Examples
+
+```text
+# Counter: one more task finished (delta is the default — no flag needed)
+hive-metric tasks_completed 1 --labels phase=scan
+
+# Gauge: current queue depth (absolute value — must use --type gauge)
+hive-metric queue_depth 17 --type gauge
+
+# Float gauge with multiple labels (instantaneous measurement)
+hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
+```
+
+### Error when OTEL is not configured
+
+When `services.hyperhive.otel.enable = false` (the default), the
+`OTEL_EXPORTER_OTLP_ENDPOINT` env var is not set and `hive-metric` exits
+with an informative error message. No silently-dropped metrics.
+
+### Wire format
+
+`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
+`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
+`OTEL_EXPORTER_OTLP_PROTOCOL` setting. Auth headers from
+`OTEL_EXPORTER_OTLP_HEADERS` are forwarded verbatim.
+
+## Metrics temporality
+
+OTEL export is configured with **cumulative** temporality by default
+(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
+overriding Claude Code's default of DELTA. This avoids silent metric drops in
+Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a
+delta-to-cumulative processor.
+
+**`hive-metric` counters are the one exception**, reporting delta by default
+(see above) — programmatically set on the exporter, which overrides this
+container-wide env var for that tool specifically. `--type gauge` is
+unaffected either way; gauges have no temporality. The hive-tier collector
+runs a `deltatocumulative` processor ahead of export, so a delta
+`hive-metric` counter still lands in VictoriaMetrics as a cumulative
+series — the standard `rate()`/`increase()` idioms work on it exactly like
+any other counter in this system, no special query needed.