mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: the handbook follows the three layers: swarm, hives, agents

Browse source 
The handbook's sections were agent-lifecycle, integrations, swarm, tools,
trust-boundary and web-ui, which do not follow how the system is built, so a
reader could not tell which layer a page is about. Its pages now sit in
swarm/, hives/ and agents/, with the accounts an agent can hold outside the
system in external-accounts/.

- swarm/: forge, matrix and the knowledge repo join the swarm pages, and so
  does the swarmctl reference.
- hives/: the dashboard, hivectl and its reference, and the operator/agent
  boundary, with a README that takes over the old web-ui and tools pages.
- agents/: the roster, persistence, config changes, the agent's own web page
  (was web-ui/agent) and the security model, with a README.
- external-accounts/: GitHub and GitLab.

Pages are moved whole; none is split yet. Every reference follows: relative
links in the docs, `docs/2-handbook/...` paths in code comments, the nix
checks that regenerate the CLI references and the ignore list, and docs/README.md,
whose handbook section is now by layer.
This commit is contained in:
iris 2026-11-07 11:20:48 +01:00 • committed by mara
commit 0e84fe8bc4
6 changed files with 393 additions and 363 deletions
Download patch file Download diff file Expand all files Collapse all files

24
docs/2-handbook/agents/README.md Normal file
View file

@ -0,0 +1,24 @@
# Agents
An **agent** is one Claude instance in its own NixOS container, with its own
state directory, forge account and matrix accounts. This section covers
running one: its identity, what survives a restart, how its config changes, what
its own page shows and how far you can trust it.
- **[Roster](agent-roster.md)** — the agents of a hive, and who may do what
to whom.
- **[Persistence](persistence.md)** — what survives a restart, a destroy and
a purge.
- **[Config changes](config-changes.md)** — how a change to an agent's config
flows from a pull request to its container.
- **[The agent's web page](web-page.md)** — the per-agent terminal view, its
actions and its endpoints.
- **[Security model](security.md)** — the trust model, the prompt-injection
threat model and credential isolation.
The swarm creates and configures agents: the create form and its archetypes
are in [`swarm/archetypes.md`](../swarm/archetypes.md), logging an agent in to
claude is in [`swarm/claude-login.md`](../swarm/claude-login.md), and you set
an agent's permissions with `swarmctl agent permissions` or on the swarm UI.
What an agent can do inside its container is in
[`4-implementation/agent-tools/`](../../4-implementation/agent-tools/README.md).

133
docs/2-handbook/agents/agent-roster.md Normal file
View file

@ -0,0 +1,133 @@
# Agent roster & privileges
<!-- vale write-good.Passive = NO -->
Agents are a **flat set**: the `manage_root_agent` permission scopes
which agents can manage which others.
<!-- vale write-good.Passive = YES -->
This doc covers what the roster file holds, the map-shaped alternative
`topology::all_agents` accepts, and what's hard-coded for the manager.
## Where the roster lives
"The roster" means two different, non-overlapping things depending on
scope. The [README](../../../README.md)'s control-plane row ("`swarm-controller`
holds the hive directory, the agent roster and the job graph") is the
**swarm-wide** one: `GET /api/agents` on swarm-controller reads
straight from the swarm's authelia identity store over
`swarm-authelia-bridge` (`swarm-controller/src/auth.rs::list_agent_identities`)
— "every agent the swarm holds an identity for" (`swarm-controller/src/routes/agents.rs`'s
`get_agents` doc comment). It's authoritative: an agent exists in the
swarm if and only if it has an identity there, and `swarmctl agent
create` (or the swarm UI, via `POST /api/agents`) writes that identity
once, by calling `ensure_agent_identity`.
This page is about a different, **hive-local** file: the hive-c0re-owned
`meta/` directory's `topology.json`, at
`/var/lib/hyperhive/meta/topology.json`:
```json
["alice", "bob", "ruth"]
```
One entry per agent _this hive_ has state/config for, in name order.
Unlike the swarm roster, it's a derived cache: the reconcile pass below
rebuilds it from what the hive observes locally (config repos cloned, containers spawned), and it
answers a narrower question than "does this agent exist": _which of
this hive's local agents the `ManageRootAgent` permission's bind-mounts
should cover._ `topology::all_agents` is the only reader that matters.
That reader is a permission boundary: the set it returns is what an agent
holding the `ManageRootAgent` permission gets bind-mounted (each other
agent's `state` read-write and `config` read-only; never `harness`). An
agent holding no permission sees its own dirs and nothing else:
`ManageRootAgent` is the only grant that reaches another agent's state
dir. See
[`persistence.md`](persistence.md)'s _Cross-agent access to state._
### Reading the map-shaped format
`topology.json` may also be a map of `name → parent | null`; the reader
accepts that shape and takes its keys as the roster. An empty
roster costs more than a cosmetic gap: every permission holder loses its
mounts until the next reconcile pass writes the array form.
### Why meta, not per-agent `agent.nix`
An agent shouldn't be able to add itself to a set that governs who may
reach its state dir. The roster IS a system-level fact; meta is where
system-level facts live.
### How `topology.json` gets updated
- **Read** — parsed into a set of names; a missing or unparsable file
degrades safely to "no agents" (covers a fresh install that hasn't
synced yet).
- **Reconcile** — runs alongside the periodic meta/flake regeneration.
Adds newly spawned agents, drops removed ones. Reconcile keeps agents
whose config repo exists but that haven't spawned yet, so the gap until
the container appears doesn't churn the file.
No write API and no operator verb reach this file. Reconcile derives it
from which agents exist, so the next pass overwrites a hand edit.
See `hive-c0re/src/agent_config/topology.rs` and `hive-c0re/src/meta.rs`'s
module docs for the exact call chain.
## Hard-coded manager behaviour
The **manager (`ruth`)** differs from other agents in naming/bootstrap
and its socket flavour; its permissions, prompt, tool allow-list and
state dirs work as for every other agent.
<!-- vale write-good.Passive = NO -->
- **Naming/bootstrap** — the manager's broker recipient name, state-dir
key, and nixos-container name are all `ruth` (container `h-ruth`).
Like every agent it comes from the swarm dashboard or
`swarmctl agent create`; `hive-c0re` creates nothing at boot.
Roster-wise, `ruth` is just another entry.
- **Wire-protocol** — `hive-core-agent-sock`'s unified `Request` enum
is the same on both socket flavours. Container lifecycle
ops (kill/start/restart/rebuild) go through the separate host-admin
socket `hivectl` speaks. `Wake` (inject a `from: <X>` message straight
into the caller's own inbox) is on this socket too, unprivileged on
both flavours; matrix, bash and forge don't call it — their
notifications push a todo on the harness's in-agent socket (see
[`docs/4-implementation/turn-loop/mcp.md`](../../4-implementation/turn-loop/mcp.md#waking-the-agent-from-inside-the-container)).
- **Storage/mounts** — the `manage_root_agent` permission, which the swarm
UI's setup assistant creates ruth with, gets a container
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
manage any agent's state dir; config isn't mounted, since a real
config change is a PR from a clone), plus an RO mount for
`/meta` (the hive's topology and limits files). Any agent holding it gets the same mounts.
- **Prompt** — identical for every container but one block (below):
`prompts/system.md`
carries `<!-- role:agent -->` / `<!-- role:manager -->` marker
blocks, and `prompt::render` filters for `"agent"` unconditionally,
manager included ("always `agent` role — there is only one role",
`hive-agent/src/prompt.rs`'s own module doc). The `role:manager`
blocks are dead in production, exercised only by a unit test
(`filter_blocks(SAMPLE, "role", "manager")`). The one block that differs
is `<!-- config:repo -->` vs `<!-- config:archetype -->`: an agent built
from an [archetype](../swarm/archetypes.md) is told it has no config repo.
- **Tool allow-list** — the MCP tools claude sees come from the agent's
permissions alone, the same mechanism for every agent
([`docs/4-implementation/turn-loop/mcp.md`](../../4-implementation/turn-loop/mcp.md)). Ruth's permissions
are the default plus `manage_root_agent`, stored for her like any grant.
- **State dirs** — _not_ treated as an exception: `HYPERHIVE_STATE_DIR` is
injected uniformly via `systemd.globalEnvironment` for every
container including the manager, so all token/state paths resolve
through it the same way everywhere.
<!-- vale write-good.Passive = YES -->
None of the above is a stable interface — treat the module doc
comments as the source of truth for exactly which checks exist today.
## Cross-references
- Milestone: "Agent privileges and sub-agents" (tracked internally)
- Audit table source: milestone comment (tracked internally)
- Operator/agent trust boundary (orthogonal axis): [`boundary.md`](../hives/boundary.md)

383
docs/2-handbook/agents/config-changes.md Normal file
View file

@ -0,0 +1,383 @@
# Config changes + system messages
Config changes are pull requests on the agent's config repo, which an
operator merges on the forge; that merge deploys them. The "System messages"
section covers the plain-text `system` messages hive-c0re sends agents.
## For operators
- **Config change** — an agent proposed a change to another agent's
config (or its own, via a sub-manager) as a pull request on
`agent-configs/<agent>`. Review it on the forge like any other PR, and
merge it there to deploy it. See [Config changes](#config-changes).
Everything below this point is the implementation detail behind that
flow.
## Config changes
Config changes flow through a **forge pull request** on the agent's
`agent-configs/<name>` repo — the same surface agents use for code PRs.
No MCP tool exists for config changes: opening the PR IS the request.
1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
tracked path, but `agent.nix` is the contract entry point), commits
with its own git identity, and pushes a branch + opens a PR with
`hive-forge` — the same way it would change any other repo.
The container has no config mount: the clone is how an agent reads its
own config as well as how it changes it.
Branch protection (the agent isn't on the `main` push allowlist;
merge and approvals allowlist = `operators` team; see "Forge config repos"
below) makes the agent a write collaborator that **can't merge its own
config PR**.
2. An operator reviews the PR **on the forge** (native diff, threaded
comments, CI status) and merges it there. Merging needs membership of
the `operators` team in `agent-configs`. The team starts empty: add
each operator by hand in the forge UI.
3. swarm-controller gets the `agent-configs` org's `pull_request`
delivery. A `closed` event with `merged: true` and base branch `main`
names the commit in `merge_commit_sha`.
4. The controller looks up which hive's wanted state places the agent
and publishes a deploy request carrying that commit on that hive's
deploy subject. With no such hive, or more than one, it deploys
nothing and logs a warning naming them.
5. If the closure the swarm declares for the agent is the one the hive last
applied (the `applied/.<n>.closure` marker), the hive does nothing.
Otherwise it queues a rebuild, which downloads the declared closure. The
request's commit plays no part: the hive keeps no copy of the config and
reads no forge head. An agent with no container on that hive yet gets its
first deploy from the declared closure.
This path runs **no eval-verify**. A merged config that doesn't
evaluate or build fails the rebuild, so the agent's rebuilds keep failing
until a fix merges. A failed rebuild shows on the hive like any other. A merge
whose webhook delivery never reaches the controller deploys nothing.
Creating a brand-new agent is swarm-level (`swarmctl agent create`,
`POST /api/agents` on the swarm controller), and no hive can originate
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
agent's config repo with a default `agent.nix` template, then asks the
target hive to deploy it.
An agent created from an [archetype](../swarm/archetypes.md) gets no
config repo: the swarm builds it from the archetype's module.
Changing what the template seeded isn't an exception: like every
later change, it's a PR on that config repo, made from a clone and
merged by an operator on the forge (see [Config changes](#config-changes)).
## Agent lifecycle
### Reminder delivery: file-path semantics
A reminder may carry a `file_path` (the agent-visible path inside its
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
1. **Translates** the container path to the host path
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
from outside the container.
2. **Validates** the path: rejects anything outside the agent's own state
subtree, containing `..` (path traversal), or with an empty relative
tail. On rejection hive-c0re skips the write and delivers the
original message inline with a warning — the reminder still fires.
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
canonicalizes the parent dir and re-verifies it lives under the agent's host
state root. hive-c0re opens the final file with
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
basename can't redirect the write to an arbitrary host path.
4. **Writes the body to disk** and delivers a short pointer message in its
place, keeping the agent's inbox / wake-prompt small while the agent
reads the bulky payload out of band.
`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
body strings before calling it.
### Destroy semantics
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
Stops + removes the nspawn container and drops the systemd drop-in.
Persistent state (claude credentials, `/state/` notes) is
**kept by default** — recreating the agent with the same name
reuses prior notes + login. With `purge = true` the agent's
`/var/lib/hyperhive/agents/<name>/` tree is also
wiped (creds + notes gone forever). The
root/bootstrap container is destroyable like any other, and stays gone
until you create ruth again from the swarm dashboard.
## Meta flake
A hive has no meta flake of its own. It builds every agent from the swarm's
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
It declares one flake input per agent (`agent-<n>.url =
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
agent's forge config repo, the one config PRs merge into) and one
`nixosConfigurations.<n>` output per agent. Each output wraps
`inputs.agent-<n>.nixosModules.default` with the identity +
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
(the dashboard port, the pinned claude-code, extra OTEL labels, the queue
address) aren't in it:
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
start (see `docs/5-project/process/conventions.md`).
A config that sits only in a local checkout, not on forge `main`, never
deploys.
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
state, bound read-only into the root agent's container as `/meta`:
`topology.json`, the swarm resource limits, and the permission seed files of a
hive that predates the swarm's permissions. `agent_set::refresh(agents:
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
nginx include in line with the live agents. `provision_container`, `destroy`,
`rebuild` and the startup migration call it. It refuses an empty agent list,
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
is unset, since every agent would come up with no forge.
## The swarm's meta flake
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
every agent any hive places (the pins:
[`docs/4-implementation/swarm/builder.md`](../../4-implementation/swarm/builder.md#meta-flake-lock)). The swarm
builder builds every agent from it, and a hive deploys that build. While the
swarm has recorded no build of an agent, that agent's deploy fails and names the
agent. The controller queues
its update of the repo before the deploy request for a merged config commit
and for a new agent, and sends the deploy only when that update succeeded and
the swarm builder built the agent from the updated repo (the
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
agent's record in `agent-closures` declares, whichever `internal/meta` commit
the builder built it from, and reads no forge head. The rebuild's `MetaSync`
node prepares it (the agent's archetype, its runtime directories, the agent
set), reads the record once and pins its store path. `Prebuild` and `Swap` use
the pin, so a record that changes mid-rebuild can't make `Swap` pull another
closure than `Prebuild` fetched, and the marker names the closure it applied. The
next autoupdate pass picks up the newer record. A first deploy has no
`MetaSync`; its `Create` reads the record once.
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
the agent's record in the swarm's `agent-closures` bucket (the controller
writes it from the builder's reply) and have `hive-priv` download the closure
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and
`CreateContainer`), which accepts only the path its last `RealiseClosure` for
that agent returned. On the wire the
`hive_priv_sock::PrivRequest` variants carry a required field:
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
A rebuild with no record, or none the builder pushed, fails with an error
that names the agent; see
[`docs/4-implementation/swarm/builder.md`](../../4-implementation/swarm/builder.md#rebuilds-download-the-closure).
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
runs the hyperhive of the controller's deploy, not its hive's.
- **Marker.** After a successful swap or create `hive-c0re` writes the closure's
store path to `applied/.<n>.closure`. The autoupdate pass
([below](#autoupdate-on-startup)) compares it with the record.
## Deploy markers per agent
```
/var/lib/hyperhive/applied/ deploy markers - core-only
└── .<name>.closure # store path of the last closure applied
/var/lib/hyperhive/meta/ plain hive state - core
├── topology.json # the live agent set
└── swarm-resource-limits.json # the swarm's per-agent caps
```
hive-c0re keeps no copy of an agent's config: the forge's `agent-configs/<name>`
`main` is the only one, and a config change is a PR against it. The marker
records the closure last applied, and nothing else. Hives that ran before the
marker existed have `.<name>.hyperhive-rev`, `.<name>.config-rev` and
`meta-rev/<name>` files instead. Nothing reads them any more, and you can
delete them.
Older hives also kept an *applied* repo at `applied/<name>` and a *proposed*
clone at `agents/<name>/config`. Nothing reads either any more, and you can
delete them.
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
The agent's `agent-configs/<n>/flake.nix`
is a fixed boilerplate that exports `nixosModules.default =
import ./agent.nix`; the meta flake imports that module and
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
### Forge config repos
The bundled `hive-forge` container runs on the swarm's forge host
(`deploy.forgejo.enable`, see [`2-handbook/swarm/services.md`](../swarm/services.md)).
Every agent created through the swarm has a repo in the private
`agent-configs` Forgejo org, created and seeded by swarm-controller.
hive-c0re pushes nothing to it, keeps no clone and reads nothing from it: forge
`main` advances only by an operator's merge.
Each agent is a **write collaborator on its own** `agent-configs/<name>`
repo — so it can push a branch and open a config PR — but not a member
of any other agent's, so it can't reach another agent's config through
the forge. Branch protection keeps the agent off the `main` push
allowlist and allowlists merging to the `operators` team only, so an
agent can't fast-forward its own config or self-merge its PR (see
[Config changes](#config-changes) above). hive-c0re passes the credential to
each `git` call inline, never writing it into a repo's config, so
no stored token sits in a repo an agent might one day read.
The dashboard deep-links into this org — a `config repo` link
per container row. See `docs/2-handbook/hives/dashboard.md`.
### Submitting agent's view of config repos
No agent has a config repo mounted, not even one holding
`ManageRootAgent`: reading another agent's config means cloning its repo
from the forge, subject to the forge's own access rules.
An agent submits a change to another agent's config the same way it
makes any other change: **clone the target agent's config repo from the
forge into its own state dir, commit on a branch, open a PR**, and let
the operator review and merge it. By design, no second, mount-shaped
path reaches the same file without the review.
Agents holding the `manage_root_agent` permission (granted per agent in
the swarm's `agent-permissions` record; see
`docs/5-project/process/conventions.md#permissions`) get additional host-side bind
mounts via `set_nspawn_flags`:
- `/var/lib/hyperhive/agents/<n>/state` → `/agents/<n>/state` (RW) for
**every** agent. The permission means "may manage any agent," so the mounted
set is every agent.
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the hive's plain state
(topology, swarm limits).
An agent without the permission only has its direct children's config
dirs.
⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts
an agent, after its current turn, when its `manage_root_agent` changes.
No agent holds the permission by default: an agent whose record leaves
`permissions` unset gets the default, which doesn't include
`manage_root_agent`, whatever the agent's name. The swarm UI's setup
assistant creates ruth with it. For a ruth that predates that, whose
permissions were unset, swarm-controller writes the default plus
`manage_root_agent` into her key once at start, so she keeps her mounts;
it never touches a key that's already set.
## Startup migrations (older hosts)
hive-c0re runs a couple of idempotent migrations on every startup so a
host set up before the current deploy scheme (described above)
converges to it automatically. Each phase is a no-op
once already applied:
- **Agent files**: refreshes the files that follow the agent set
(`agent_set::refresh`) from the current agent list. Set
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
- **Legacy meta repo**: deletes the leftover `.git` in
`/var/lib/hyperhive/meta/` and the
unread `/var/lib/hyperhive/forge-core-token`. The files beside the `.git`
stay. The forge-side `core/meta` repo and `core` account aren't touched.
No state loss in either migration: claude creds, `/state/` notes, the
events DB all survive. The root
agent keeps its session; sub-agents stay logged in.
## The root/bootstrap container
The root agent container runs through the **same lifecycle as
sub-agents**. Nothing creates it at hive boot: you create ruth from the
swarm dashboard like any agent, and an existing ruth container starts with
the hive like every other. The root agent can clone
`agent-configs/ruth` to read its own `agent.nix` and open a config PR there
for operator review, same as any other agent.
Differences from sub-agents:
- Reference docs: her `agent.nix` turns on `services.hyperhive.agent.docs.enable`.
Her container builds from `agent-base` like every agent, and the seeded
`agent.nix` is the same for every name, so a newly created ruth has no docs
until a config PR adds that line.
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
FNV-1a hash as every other agent (8100..8999 range).
- The root agent holds `manage_root_agent` (the swarm UI's setup
assistant creates her with it), for which `set_nspawn_flags` adds extra
binds: every agent's `state` dir (RW) under `/agents/<n>/state`, and the
hive's plain state under `/meta` (RO).
- Its socket comes from the generic per-agent registration, in the same
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
not a separate manager-only path.
**Migration note** (for older hosts): drop any `containers.root =
{ ... }` block from your host NixOS config, and any `services.hyperhive.ruthless`
definition, then create ruth from the swarm dashboard.
## Root-agent policy
The system prompt (`hive-agent/prompts/system.md`, rendered by
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
varies is which MCP tools it surfaces (gated by the agent's permissions,
which the swarm sets). No `role:manager` block renders only
for the root agent. The root agent's role-specific
behaviour comes from its AGENTS.md / agent-specific instructions, not
the system prompt template.
## System messages
Hive-c0re has no typed event that drives an immediate claude turn. A
message from sender `system` is plain inbox text, and the harness logs it
(`log_system_event` in `hive-agent/src/main.rs`) without parsing the body.
Today hive-c0re sends two: the `/knowledge updated` broadcast
(`workers/knowledge.rs`) and the wake message for a graceful stop
(`Coordinator::kick_agent`).
**A previously running container crashing** drives no agent turn.
`crash_watch` only `tracing::warn!`s `"container crash
detected"`; a swarm-level grafana alert matches that log line directly
and pages the operator, with no agent (and no per-hive dashboard
banner) in the loop. See
[`docs/4-implementation/scheduler/observability.md`](../../4-implementation/scheduler/observability.md#container-crash-alerting-hive-c0re).
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
when convenient" events with no reason to drive an immediate turn, so
they go out via `swarm_notices::notify` instead: `subsystem = "core"`,
`key = "<event>:<agent>"`, and a single free-text `summary`
(`rebuilt_todo_summary` renders `Rebuilt`'s `ok`/`note`/`sha`/`tag`
fields into that string).
To add a new lifecycle notice: if it's "FYI, check when convenient,"
call `swarm_notices::notify` — no new wire type needed. If it needs to
drive an immediate turn, decide first whether it actually needs an
*agent* in the loop at all (the crash case above didn't, once a
swarm-level alert could carry the same signal).
## Autoupdate on startup
`hive-c0re serve` runs `auto_update::run` in a background task right
after opening the coordinator. It enumerates managed containers and
rebuilds any whose closure marker differs from the store path the agent's
record in `agent-closures` declares. That's how a hive that missed the
swarm's deploy request catches up, and it repeats every five minutes, which
covers a request lost while the hive is up. Sub-agents and the root agent go
through the same graceful rebuild subgraph.
The comparison reads the record and the marker
(`/var/lib/hyperhive/applied/.<name>.closure`, written after a successful
swap or create) and nothing on the forge. Where the marker is missing, as for
every agent on the first boot after the marker arrived, the hive reads the
closure the container's profile
(`/nix/var/nix/profiles/per-container/h-<name>/system`) points at and writes
that as the marker, so the first pass catches an agent that doesn't run its declared
closure. Those first-boot rebuilds go through the same job queue as
any other: each holds one of the `buildSlots`, so they don't swap all at once.
It rebuilds nothing when the hive can't tell: no connection to the swarm queue,
no record, a closure the builder didn't push, or a profile it can't read or that
leaves the nix store. The periodic pass
attempts a declared store path once per process, and skips an agent while a
rebuild of it waits or runs, so a closure that fails to deploy doesn't
restart its agent every five minutes. Restarting `hive-c0re`, or a rebuild from
the agent's menu, tries again.
A stopped agent gets its rebuild at its next start: the start path upgrades
the start of a stale, stopped agent to a rebuild followed by the start. In
the window after a config merge, before the swarm's builder has recorded the
new closure, the rebuild deploys the previous one.
The dashboard shows no pending-update state: the hive applies a changed
declaration itself. The agent menu's rebuild queues the same
`job_queue::templates::rebuild` DAG, which downloads and applies the declared
closure again.

842
docs/2-handbook/agents/persistence.md Normal file
View file

@ -0,0 +1,842 @@
# Persistence + retention
Where state lives, what survives what, and how it's bounded.
## For operators
The short answer to "does anything disappear": **destroying an agent
keeps its state, purging it doesn't.**
- **`DESTR0Y`** (the default action) stops and removes the container
but keeps everything on disk — config history, claude login, `/state/`
notes, harness data. The agent shows up as a tombstone (K3PT ST4T3 on
the C0R3 page); `swarmctl agent create` with the same name and hive
recreates it from the kept state, **no re-login needed**.
- **`PURG3`** (opt-in, from the dashboard or `hivectl agent <name>
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
history, claude credentials, `/state/` notes, everything. **No
undo.** Only reach for this when you actually want the agent gone
for good. The agent's matrix encryption keys go with the state dir,
but an account whose key backup is on (see
[matrix](../swarm/matrix.md)) gets them back from its
homeserver when it next comes up, so its encrypted history stays
readable.
Beyond that:
- **Broker messages**: acked ones vacuum after 30 days; anything
undelivered or delivered-but-not-yet-acked is always kept, however
old.
- **An agent's own `/state/` notes and claude login survive every
restart and rebuild** — only an explicit purge (or a hive
`--purge`-style host operation, or the agent's own choices) touches
them.
- The **root/bootstrap agent has no guard**: destroying ruth is as final as
for any agent, and she stays gone until you create her again from the
swarm dashboard.
Everything below this point is implementation detail: exact table
schemas, file layouts, and internal migration mechanics.
## Sqlite databases
### `/var/lib/hyperhive/db/broker.sqlite` (host)
One table, the message queue:
- `messages` — every inter-agent / operator-authored message.
`sender / recipient / body / sent_at / delivered_at / acked_at /
in_reply_to / priority`. `in_reply_to` links a reply to its parent
row id; the dashboard and per-agent inbox render these as threaded
rows.
⚠️ The `mcp__hyperhive__remind` queue lives in a harness-local,
per-agent store — see [`/harness/` contents
below](#state-dirs-per-agent) for where reminders (and todos)
live.
The swarm owns power intent, so hive-c0re keeps none on disk. An older build
also kept a `kv` table (the agents running at the last `hivectl stop`) and an
`agent_power` table (the durable `wanted` intent) here; opening the file drops
both.
On a host whose hive-c0re once kept scheduled prompts, the file also
holds `scheduled_prompts` and `scheduled_prompt_targets`. Nothing opens
them; scheduled prompts live at swarm level
([`2-handbook/swarm/schedules.md`](../swarm/schedules.md)). They're safe to drop by
hand.
Retention:
- `Broker::vacuum_delivered` runs hourly via a tokio task in
`hive-c0re::main`. Drops acked message rows older than 30 days
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
rows are always kept — the harness `ack_turn`s only after a
successful turn, so `requeue_inflight` can still requeue
an unacked row on a crash.
### `/harness/hyperhive-events.sqlite` (per agent)
Lives inside each container's bind-mounted `/harness/` dir (host
path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
One table:
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
harness emits during turn loop execution.
The db runs in WAL mode, so `-wal` and `-shm` files sit beside it while
the harness is up. Copy or back up the three together, or the recent rows
sit in the `-wal` file until a checkpoint.
<!-- vale write-good.Passive = NO -->
The harness both writes and vacuums it. Cleanup runs in-container
because hive-c0re runs as the unprivileged `hive-core` user under
privsep and can't delete agent-owned files (host-side deletes hit
`PermissionDenied` on the bash-task trio and a readonly database error
here). `hive-agent`'s `vacuum::run`
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
**type-scoped**: it deletes only the verbose `stream` rows (the raw
claude `stream-json` deltas — one per text chunk / tool use, the bulk
of the file's size) older than 14 days, and keeps every other kind
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
small and carry the semantic per-turn history the operator scrolls
back through when debugging a regression. Age-only within the
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
history sooner than a quiet one. The trade-off (accepted): a
misbehaving harness can skip its own cleanup — but a compromised harness is
already inside the container trust boundary
([`docs/2-handbook/agents/security.md`](security.md)), and these are ephemeral local
artifacts, so cleaning them up where they live is the honest fix.
<!-- vale write-good.Passive = YES -->
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
setups). On open failure the `Bus` falls back to no-store mode
rather than crashing the harness — events still broadcast over SSE,
just nothing persisted.
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
Per-turn analytics sink. One row per claude turn captures
identity (`model`, `wake_from`, `result_kind`), timing
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
cache_read / cache_creation token counts), behaviour
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
snapshot metrics (`open_threads_count`,
`open_reminders_count` — fetched via the same socket the harness
already uses for `GetOpenThreads` + `CountPendingReminders`).
Bin-loop helpers `build_row` + `record` land each row at
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
the turn loop continue.
The `hive-bash-daemon` (not the harness) writes a sibling
`bash_commands(ts INTEGER, head TEXT)` table in the same
file: one row per executed bash task recording the normalised command head -
the basename of the first real command, looking past `cd repo &&`
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
backs the "favorite tools" view on the /stats page. Best-effort and created on first write
(`CREATE TABLE IF NOT EXISTS`), so it's absent until a bash
task runs.
turn-stats.sqlite has **no vacuum** — it's one small row per turn
(~hundreds of KB even over months), read directly by the `/stats` page,
so pruning it would only lose trend
history for no space gain.
### `/state/hyperhive-harness.json` (per agent)
Consolidated harness state file written atomically (`.tmp` + rename) by
`Bus::emit_status` whenever rate-limited or login-failed flags change.
Shape:
```json
{ "rate_limited": false, "needs_login": false, "active_model": "…" }
```
- `rate_limited` — set when the harness detects a 429 from the Claude
API; cleared by any subsequent status emit. Drives
`ContainerView.rate_limited` on the dashboard.
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
cleared by `"online"` status (re-auth completed). Drives the
`needs_login` flag alongside the `claude_has_session` check.
- `active_model` — the resolved Claude model for the dashboard badge.
The turn loop is the only writer today, but it still goes
read-modify-write under a shared in-process lock and merges into the
existing object rather than reconstructing it — so a second writer
would preserve fields it doesn't own, and the lock closes the
lost-update window between a writer's read and its rename. The lock
is in-process only, so it wouldn't serialise a writer running as a
separate process; none of today's writers are.
hive-c0re reads this file on each `build_all` sweep (~10s) via
`container_view::read_harness_flags`. Falls back to the individual
sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) when the
JSON is absent.
### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
Full stdout + stderr capture for every `nixos-container` / `nix
build` invocation the lifecycle layer fires. One row per invocation;
the row accumulates lines as the child runs.
Capturing the full stream (rather than a short tail buffer) matters
because real eval errors routinely run long — "tried alternatives"
blocks alone are often 30+ lines — so a truncated tail would cut off
the actual failure and leave only the host journal holding the
complete output. With this table the dashboard can surface the entire
log.
Three indices:
- `(agent, started_at)` — backs the per-agent latest-N lookup used
by the agent card chip.
- `(status, finished_at)` — backs the retention sweep that runs
as part of the existing hourly vacuum.
- `(node_id)` — added by a later migration so a build log row can be
looked up by the job-queue node it belongs to (a `hive_jobq` node is
immutable after insert, so hive-c0re records the link on the log row
instead); `node_id` is nullable.
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
log a warning on sqlite error and let the build continue. A failed
log row never blocks a rebuild.
Each operation also logs to the host journal, target `nixos-container`, and the
swarm log store collects those lines. Every line carries `build_id` (this
table's row id; empty without the store), `agent`, `kind` and `cmdline`. A
`start` line (`event=start`) opens the build, one line follows per line of
output, and a `finish` line (`event=finish`) closes it with `status` (`ok` or
`fail`) and `runtime_secs`. hive-c0re logs a nix warning or error at warn, and a failed
finish too.
A row is open (`finished_at IS NULL`) only while its child runs. At startup
`Coordinator::open` closes every row still open as `fail` and appends a
line saying hive-c0re stopped: the previous process died mid-build, and
the retention sweep never deletes open rows.
### `/harness/hyperhive-model` (per agent)
Single-line text file holding a claude model name. The harness only
reads it, at boot in `Bus::new` and again when the swarm clears the
model it set. It holds the model the agent runs when neither the swarm nor
`HIVE_DEFAULT_MODEL` (agent.nix) names one. Nothing in the harness
writes it. Path overridable via `HYPERHIVE_MODEL_FILE`. Survives
destroy/recreate, gone on `--purge`.
### `/harness/paused` (per agent)
Empty marker file. Its presence parks the agent's turn loop: the
harness keeps serving its web UI and MCP daemons but drives no turns,
and inbox messages queue unacked until it's removed (see
[turn loop](../../4-implementation/turn-loop/README.md#the-loop)).
<!-- vale write-good.Passive = NO -->
Unusually, it's read and written from **both** sides of the harness
bind-mount, and that's the whole design: the harness stats it
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
stats it on the host (`Coordinator::is_paused`) to populate the
`paused` field on the agent card, and creates/removes it
(`Coordinator::set_paused`) for `hivectl agent <name> pause|resume` and the
dashboard toggle. Because the file itself is the only shared state
there's no protocol between them, no round-trip into the container, and
pause keeps working when the harness is wedged or the container is
stopped.
<!-- vale write-good.Passive = YES -->
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
is the agent's own space to fill, and this is harness control state.
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
back paused after a restart, which is the intended behaviour rather
than an accident of storage.
## State dirs (per agent)
Under `/var/lib/hyperhive/agents/<name>/`:
- `config/` — unused. Older hives kept a proposed nix repo here; nothing
reads it any more, and you can delete it. An agent reads its config by
cloning `agent-configs/<name>` from the forge.
- `claude/` — claude OAuth credentials, bind-mounted RW to
`/home/<name>/.claude` inside the container.
- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
to `/agents/<name>/state` inside the container (uniform for
all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
the same path to in-container scripts. Notable files written here
by the harness:
- `hyperhive-status` — single-line free-text status string written
by `set_status`; cleared on explicit `set_status("")`. Read by
hive-c0re and the per-agent `/api/dashboard-state` endpoint to
surface the status chip on the dashboard. Absent when no status
has a value.
- `hyperhive-harness.json` — rate-limited / needs-login flags read
by the dashboard's async container-state fetch. See
`docs/2-handbook/hives/dashboard.md::Container row`.
The agent writes its own instructions to `AGENTS.md` here; claude and
opencode load it automatically from their cwd, `state/`. At boot,
`agents_md_migrate::run` renames a `state/CLAUDE.md` to `AGENTS.md`
when no `AGENTS.md` exists. When both exist it renames neither and
logs an error naming both paths: claude loads only `CLAUDE.md` while
one exists in its cwd or above. The harness also publishes the file
to the swarm, which reads that copy, and applies the git diffs the
swarm sends to edit it. A new agent's first start writes the text given at
create, only when no file exists. This file stays the only copy that counts:
[`docs/2-handbook/swarm/README.md`](../swarm/README.md#an-agents-agentsmd).
- `harness/` — harness-internal ephemeral state; not intended for
agent consumption. Bind-mounted to `/agents/<name>/harness`
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
- `bash-tasks/` — task JSON + stdout/stderr files for
background `mcp__bash__run` jobs. JSON files are
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
(full captured output). The harness's own hourly sweep
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
event rows above) deletes terminal task trios older than 48
hours; non-terminal (still-running) tasks are never deleted. This
sweep runs in-container, for the same privsep-ownership reason as
the events vacuum above.
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
and reminders, one small table each in a single file (in-container
daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
`hive-forge-notify` — upsert keyed todos here over the harness's
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
`get_loose_ends` output and clears a row on `mark_todo_done`).
At boot, `db_migrate::run` creates this file from
`hyperhive-todos.sqlite` and `hyperhive-reminders.sqlite` when those
are present. Also backs the `mcp__hyperhive__remind` queue.
The harness itself is also a producer, not just the socket server:
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
socket the out-of-process daemons above dial) alongside
`disk_watch::run` — an *in-process* todo producer that shares the
store + wake `Notify` directly rather than dialling its own socket.
`disk_watch` raises a keyed `disk` todo when the filesystem backing
this agent's state gets tight, naming the agent's own biggest
directories; it buckets the summary and carries no raw byte
counts, so an unchanged situation re-upserts as `changed == false`
and never re-wakes.
Retention, same hourly `hive-agent::vacuum::run` sweep as
`hyperhive-events.sqlite` below: it reaps delivered (soft-deleted) reminder
rows 14 days after delivery, kept that long only to serve
the trailing-window `ReminderRollup` stats, and reaps acked todo rows
30 days after acking (long enough that only a genuinely quiet
month triggers the "one spurious re-announcement" fallback a
reconciled producer like `disk_watch` relies on — see `todos.rs`'s
module doc). Un-acked todos and undelivered reminders are never
swept — same "audit trail, not cache" treatment as the c0re-side
tables above.
- `hyperhive-events.sqlite` — turn-loop event log.
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
- `hyperhive-model` / `hyperhive-effort` — single-line model name and
effort level files, read-only fallbacks.
- `work/` — the agent's working space for git worktrees, build dirs
and other large working data. Bind-mounted RW to
`/agents/<name>/work` inside the container and owned by the agent
user. It sits outside `state/` so a claude or subagent started under
`work/` doesn't walk up into `state/AGENTS.md` and load the agent's
own identity. Survives `destroy`/recreate, gone on `--purge`. Counted
in the per-agent disk metric (`measure_agent_disk`) together with
`state/` and the container rootfs. The dashboard doesn't serve it.
### The container root (per agent)
An agent container starts with `systemd-nspawn --ephemeral`
(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a
snapshot of `/var/lib/nixos-containers/h-<name>`, and the snapshot vanishes
at stop. The container loses anything the agent writes outside a bind: `/home/<name>`
apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is
`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container
`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir.
`/home/<name>` holds caches and nothing the agent should keep: worktrees,
clones and builds go in `work/`.
The nix profile needs one extra step. nix keeps a non-root user's profile in
`~/.local/state/nix`, so the agent's tmpfiles rules
(`nix/agent-modules/user.nix`) point that directory into
`/nix/var/nix/profiles/per-user/<name>` (the persistent bind) and recreate
`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or
`nix profile` survive a restart, and go with the container on `destroy`.
**The one-time salvage.** The first start of an agent whose root has content
runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the
`WriteNspawnFlags` that precedes every start, while the container is down:
1. Everything in the root except the cache paths (`tmp`, `var/cache`,
`var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is
copied to `work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, with
modes and mtimes kept. The agent finds its old `/home/<name>` there.
2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there).
3. hive-priv renames the old root to `h-<name>.pre-ephemeral-<YYYYMMDD>` and
logs the path. Nothing deletes it; the operator removes it when ready.
4. A fresh template root takes its place, carrying `etc/machine-id` (the old
id, so the journal directory and DHCP identity hold) and the marker
`etc/hyperhive-ephemeral-root`.
The salvage runs again only if the template holds content outside the cache
paths, which happens when a hive without this change started the container
persistently in between; the new copy gets a numbered directory beside the
first. A failed salvage leaves the root where it was and the container starts
without the flag (hive-priv logs the error), so a failure costs the switch,
never data.
**Checked before every start.** nixpkgs' start script runs as host root on
the root template, and its `mkdir -p`, `chmod`, `touch` and
`cp --remove-destination` follow a link the container planted at the paths
they touch. The container's limits drop-in
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
script, on systemd's own restarts too. It looks at each path with `lstat`,
never following a link, and fails the start with one line on stderr naming the
path and its type when:
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
exists and isn't a real directory;
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
(`etc/os-release` may also be a link, which the script skips and a booted
NixOS guest creates);
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
exists and isn't a link that resolves into `/nix/store`, or the `system`
file inside the resolved directory isn't a regular file.
Absent paths pass, so a first start works. The check neither deletes nor
repairs: the operator removes the offending entry, and the next start goes
through.
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
points at a build without the sub-command. A drop-in written by an older
points at a build without the sub-command. That build's store path has a GC
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
drop-in and replaced on each write, so a garbage collection can't remove the
binary a restart would run. The directory sits beside `gcroots/per-container`,
which is the only gcroots path bound into a container, so only root writes it.
Destroying the container removes the drop-in and then its root; the build stays
until the next collection. A drop-in written by an older hive-priv has no
`ExecStartPre=` and no root, and the container starts as before.
**Hyperhive's platform containers: warn only.** The forge, the matrix server and
the swarm services (15 containers, declared by their own modules) run the same
check as `hive-priv check-rootfs --warn-only <name>`, from `ExecStartPre=` on
their own `container@<name>` unit, through the symlink `/run/hive-rootfs-check`
that `hive-priv.nix` points at the current build, so a new hyperhive build does
not change the unit or restart the container. It evaluates every rule, logs each
violation
as a warning, and exits 0, so the container starts either way. The line reads
`rootfs check (warn-only): <name>: <message>` with journald's warning priority.
Strict refusal needs the live roots of all 15 to pass first, and the warnings
show which ones would not. A container the operator runs that hyperhive doesn't
declare gets no check at all: the modules name their own containers, and
nothing enumerates the host's `containers`.
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and
copies the data of a file the filesystem refuses to clone. Before it creates
anything, hive-priv sizes the root and compares it with the free space under
`work/`. A salvage whose files clone (tested with one file of the root) must
leave 2 GiB free. One that has to copy data must leave 15 % of the filesystem
free. When the free space is short, hive-priv skips the salvage, deletes
nothing, logs `salvage skipped: <container> need <size> free <size>`, and the
container starts persistent; the next start tries again. hive-priv checks a file
that fails to clone against the 2 GiB reserve again before it copies the data. A
salvage that fails partway removes the directory it created, and leaves the root
and any earlier salvage as they were. After a salvage, hive-priv logs `salvage: <container> <size> reflinked=<n>/<m> files`.
**Concurrency.** hive-priv holds an `flock` on
`/run/hive-priv/salvage-<container>.lock` (the unit's `RuntimeDirectory`, which
no container can write) for the whole `WriteNspawnFlags`, so a second request for
the same container waits and then finds the finished template. Across
containers, one salvage copies at a time, so the free space each one checks
excludes any copy still in flight. The request runs on the blocking
thread pool, so a long copy holds no async worker. hive-c0re's per-agent lease
already serialises the requests it sends; the lock covers hive-priv's own
guarantee.
The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`)
reads the guest's own `/var/log/journal` and pushes it over OTLP, so no
journal link to the host takes part; nspawn skips `--link-journal=try-guest`
in ephemeral mode without failing. The journal vanishes at stop, and with it
the lines written after the last push.
### User namespace (opt-in, per agent)
An agent whose config sets `services.hyperhive.agent.user.privateUsers = true`
runs in its own user namespace. Every other agent keeps the host's uids.
**Approving the config PR that sets it authorises a host-level change:** the
first start afterwards has hive-priv, as root, chown the agent's `state`,
`harness`, `work` and `claude` dirs once to the host's data owner. It deletes nothing.
**How the hive learns the flag.** It never evaluates the agent's nix. The swarm
builder reads `config.services.hyperhive.agent.user.privateUsers` from the
config it evaluates in its sandbox and puts it in the agent's closure record
(`private_users`). hive-c0re reads the record when it writes the container's
flags and sends the value to hive-priv with each start. An agent whose module
doesn't declare the option reads false. When the builder can't read the
option (its eval fails or times out), the record carries no flag, and so does a
record or builder from before the field. A record without a flag, a record
hive-c0re can't read, or none, leaves the agent as it is.
- **On disk the data belongs to one host uid:gid**,
`services.hyperhive.deploy.hive-controller.userns.dataOwner`
(default `60000:60000`). Inside the container that owner shows as the agent
user. hive-priv mounts the agent's `state/`, `harness/`, `work/` and
`claude/` dirs and its socket dir with `owneridmap`, and every other bind too
(config, `/knowledge`, the runtime dir), whose owner shows as root inside.
- **`/nix/store` has to be on ext4, btrfs, xfs or erofs.** nspawn mounts the
store into a namespaced container idmapped, and the kernel refuses that on
overlayfs and on some other filesystems. `WriteNspawnFlags` for an opted-in
agent checks the `statfs` type of `/nix/store` against an allowlist of those
four; the VM test's erofs store idmaps on the hive's kernel. The check
refuses every other filesystem, zfs and tmpfs included, even where a newer
kernel could idmap it, because the gate exists to stop a container that would
not boot. The error names the type and the fix, and
comes before anything moves. Check a host with
`findmnt -no FSTYPE /nix/store`.
- **The container starts with `--private-users=pick --private-users-ownership=map`.**
nspawn idmaps the root at mount time, so a different range for `pick` never
walks or chowns it.
- **The move happens once.** The first `WriteNspawnFlags` that finds the
agent's container stopped chowns those four dirs to `dataOwner` (ownership
only; it removes or copies nothing), writes
`/var/lib/hive-priv/userns/<name>`, and creates the mount points in the root
template owned by the agent user. A later request with the marker present
chowns nothing. A running container waits for its next stop. The socket dir
is on a tmpfs, so `EnsureAgentSocketDir` hands it to `dataOwner` at every
start. The walk works by open directory fd and follows no link, so an agent
or manager that can write the dirs can't redirect it to a host path.
- **A move that fails starts over.** hive-priv opens all four dirs and
checks the agent's ids before the first chown, so a link or file where a dir
goes refuses with nothing changed. It then writes the ids to
`/var/lib/hive-priv/userns/<name>.pending`. If the walk fails after that (a
chown error, a tree nested too deep), the next start repeats it from those
ids, then writes the marker and removes the pending file. hive-priv refuses a
pending file that doesn't hold `<owner> <uid> <gid>`, and names it.
- **The agent's gid must equal its uid.** `owneridmap` maps the gid of a bind
with the uid map, so a group other than the uid's has no mapping and writes
fail. The agent module asserts that `user.uid` and `user.gid` hold the
same number when `privateUsers` is on. Start the agent once with the pin
before turning it on: the move refuses an agent whose `state` dir has another
gid than uid, naming the agent.
- **Turning it off is manual.** hive-priv refuses to start an agent whose
marker exists while its record says false, and one whose `dataOwner`
changed. Chown the four dirs back, then delete the marker and any
`<name>.pending` beside it.
- **Keep a `ManageRootAgent` holder off it.** The other agents' dirs it mounts
aren't remapped.
The pause marker needs nothing extra: hive-priv gives it the owner of the
`harness/` dir. hive-c0re still creates agent dirs and new subvolumes as
`hive-core`, and only the first namespaced start chowns them. A dir created
after the move, such as a recreated subvolume, stays `hive-core`'s and the
agent can't write to it.
### Cross-agent access to state
An agent holding the `manage_root_agent` permission gets every other
agent's `state` dir bind-mounted **read-write** (`bind_child_agent_dirs` in
`lifecycle/host_config.rs`). The RW is deliberate, not an oversight: the
holder recovers other agents, which includes writing into their state (for
example seeding notes, clearing a stuck sentinel) as well as reading it.
This is the **only** cross-agent mount. An agent without that permission
sees its own dirs and nothing else.
**`harness` isn't mounted at all.** It holds that agent's own runtime
material - `bash-tasks/`, the turn-stats and event sqlite dbs - and
nothing argues for anyone else reading it, let alone writing it.
hive-c0re reads a harness dir **directly on the host** when it wants
those stats, which needs no mount into another container.
**`config` isn't mounted either.** A config change is a PR on that agent's
config repo, made from a clone and merged after review. Anyone who wants to
read the config clones `agent-configs/<name>` from the forge. A mount, even a
read-only one, would show an agent a tree that's not the one that governs it
whenever the forge has moved on.
Isolation still holds by default: a container has its *own* dirs and,
unless it holds the permission, nothing else.
Under `/var/lib/hyperhive/applied/` — the hive-c0re-only deploy markers
`.<name>.closure`: the store path of the closure the agent was last swapped to
or created from, or, where the file was missing, the closure the container's
profile pointed at when the hive first read it. No config lives here; the
forge's `agent-configs/<name>` is the only copy. A hive that predates this
finds per-agent `applied/<name>` repos here. Nothing reads them any more, and
you can delete them.
Under `/var/lib/hyperhive/meta/` — system-level config files, plain state
with no git history. The flake agents build from is the
swarm's `internal/meta`, not here. A hive that predates this finds a
leftover `.git` here; `migrate::run` deletes it at startup. Contents:
- `topology.json` — the agent roster (`["alice", "bob", "ruth"]`).
Written by `topology::reconcile` from `agent_set::refresh`; read by
`topology::all_agents`, which is the set the `manage_root_agent`
permission grants mounts over. The reader also accepts a map-shaped
file carrying a `parent` per agent, and keeps its keys.
- `swarm-resource-limits.json` — the caps the swarm sets for this
hive's agents
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
Written only by `workers::swarm_limits`, which copies each agent's key of
the swarm's `agent-limits` bucket here every 30 seconds. Read where
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
into the container — these are host-side caps on the container, so
the capped party never sets them. It sees only the resulting memory cap,
as a byte count or `infinity` in its runtime dir (below). Fallback is per
*field*: an absent file, absent agent, or absent field falls back to the
hive-wide `services.hyperhive.deploy.hive-controller.agentCpuQuota` / `agentMemoryMax`,
so an agent can have only its memory capped and still track the hive
default for CPU. Every value passes the validators first, and one bad
value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
shares in the same drop-in have **no** per-agent setting — they're
hive-wide only and come straight off `HiveEnv`, so this file has no field
for them.
- `resource-limits.json` — retired. Per-agent overrides set on the hive used
to live here; nothing reads or writes it now. An old copy in this directory
is harmless and you can delete it.
An agent holding `manage_root_agent`, normally ruth, has the meta dir
RO-mounted at `/meta/`.
Outside this directory, `/var/lib/hyperhive/swarm-agent-permissions.json`
holds the permissions the swarm grants this hive's agents, as
`workers::swarm_permissions` last read them from the swarm's
`agent-permissions` bucket (every 30 seconds): a copy, not the
record, which `swarmctl agent permissions` sets. `set_nspawn_flags`
reads it at every container start for `manage_root_agent`.
`/var/lib/hyperhive/swarm-agent-archetypes.json` holds the archetype each of
this hive's archetype agents runs, as last read from the swarm's
`agent-archetype` bucket at the agent's provision and each rebuild. An agent
with an entry has no config repo and no `agent-<n>` meta input; see
[archetypes](../swarm/archetypes.md).
hive-c0re creates containers from the closure the swarm builds from `internal/meta` (`nixosConfigurations.<n>`).
## Destroy vs purge
See [For operators](#for-operators) above for what each action does to
an agent's state. The mechanics, for completeness:
- `DESTR0Y` also drops the systemd drop-in; reviving the agent is
`swarmctl agent create` with the same name and hive, which reuses the
kept state.
- `PURG3` wipes `/var/lib/hyperhive/agents/<name>/` — the
union of everything `DESTR0Y` left behind.
`actions::destroy` has no guard for the root/bootstrap agent: it destroys ruth
like any other agent, and she stays gone until you create her again from the
swarm dashboard.
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
On a btrfs host, `lifecycle::ensure_agent_state_subvolume` creates a brand-new agent's state root as a
**btrfs subvolume** instead of a plain directory (progressive
enhancement). This is a no-op fallback on
non-btrfs hosts and for any agent whose root already exists, so
hive-c0re automigrates nothing: existing agents keep their plain dirs
until an explicit opt-in upgrade.
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
hive-c0re creates the per-agent subdirs (spawn / rebuild).
It skips the work when the root already exists; otherwise it asks
hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
root when the FS is btrfs (`statfs` magic gate) and chown it to the
`hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
succeed inside it.
- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
reuses it untouched.
- **PURG3 deletes it correctly:** `rmdir`/`remove_dir_all` can't remove
a subvolume root, so purge first calls hive-priv
(`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
actually a subvolume, then the normal `remove_dir_all` sweep covers
plain-dir agents.
Per-subvolume disk-usage accounting and optional quotas have since
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
`hivectl agent <name> quota show|set` reads/limits one agent's
subvolume usage through the same hive-priv-mediated path as
subvolume creation/deletion above.
This is the same subvolume `hivectl agent <name> subvol snapshot push`
sends to the swarm's snapshot store — see
[`docs/4-implementation/networking/snapshot-store.md`](../../4-implementation/networking/snapshot-store.md) for what a pushed
snapshot contains and how the store authenticates a sender.
The subvolume is the whole agent state root, so `work/` is part of every
snapshot and migration of it and can make them much larger.
## `/var/lib/swarm-controller/` (swarm-controller host only)
Only present on the one host running
`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
so it survives restarts and redeploys.
- `webhook-secret` — the HMAC key Forgejo signs the swarm's forge webhooks
with, as it was before the swarm secret store held it. **Keep it.** The
controller stores its value in the swarm secret store at
`swarm/controller/swarm-controller/forge/webhook-secret` the first time it
starts with that path empty, and reads the store from then on. The file is the
fallback while the store is unreachable or its policy doesn't grant the path
yet, and nothing else reads or removes it. A host with neither the file nor a
store value gets a new secret in the store, with nothing to configure.
<!-- vale write-good.Passive = NO -->
If the store can't answer and the file is unreadable or missing at startup, the daemon still starts and logs
`webhook secret unavailable`; the webhook endpoint then answers 503
rather than accepting deliveries it can't verify. Everything else the
controller serves is unaffected. The daemon keeps reading the store in the
background, at most five minutes apart, and the endpoint and the forge hook
registration start working when it answers. It writes a new secret only when
the store answers that it holds none, never because a read failed.
<!-- vale write-good.Passive = YES -->
- `operator-matrix-admins.json` — the matrix user ids the operator admin
sync has made homeserver admin, a JSON array. The sync revokes admin
only from these, so deleting the file leaves every current admin in
place; a missing file is an empty set. A file the controller can't
read or parse stops the sync, grants included, and the controller logs
`operator matrix admin: pass failed` on every pass. See
[`docs/2-handbook/swarm/matrix.md`](../swarm/matrix.md#operators-are-homeserver-admins).
## Run-time dirs
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
Without that, every restart wipes bind sources and existing
containers can't start.
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
(bind-mounted into the container as `/run/hive/mcp.sock`).
- `/run/hyperhive/agents/<name>/memory-max-bytes` — the agent's effective
`MemoryMax=` as a byte count (a percentage resolved against the host's
physical memory) or `infinity`, written by `lifecycle::write_dropins`
before every start. Absent only when hive-c0re can't tell the cap. The
container reads it at boot as `/run/hive/memory-max-bytes`
(`nix/agent-modules/memory-cap.nix`).
On startup, `Coordinator::register_agent` drops any prior socket
task before rebinding — idempotent so a hive-c0re restart followed
by `rebuild alice` recreates the agent's socket without a clean
reinstall.
## First-boot agent-user migration
The harness runs as a per-agent unix user inside the container
(`services.hyperhive.agent.user.name`, defaults to the agent's logical label so each
container has a uniquely named user). The activation script below
moves root-owned state to the agent user so the claude session survives.
`system.activationScripts.hive-agent-user-migrate` (in
`nix/agent-modules/user.nix`) runs on every activation,
marker-guarded so the substantive moves only happen once per
container lifetime:
1. **`${homeDir}` exists with the right ownership** — covers the
the first boot before `useradd`'s `createHome` has had a
chance to chown. Also re-applies on every rebuild in case the
meta-flake's per-agent name evolves (rare).
2. **Migrate any leftover `/root/.claude` content into
`${homeDir}/.claude`.** Marker
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
only written once there's nothing left to migrate, so a `cp`
failure leaves it absent and the next boot retries. The root is
ephemeral, so the marker vanishes at every stop and the step
re-runs; with no `/root/.claude` it only writes the marker again.
`cp -an` (no-clobber) so any pre-existing files at the new
location win — never blow over data already there, and so a
retry after a partial copy is as safe as the first attempt.
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
recursively so the agent user can read/write it. Wildcard
matches the single agent that container sees; `-h` skips
symlinks the agent might have planted.
4. **Chown the `~/.claude/` bind-mount** recursively. The harness reads
`~/.claude/` as the agent user to decide Online vs NeedsLogin in
`login::has_session`. Without the chown, credentials owned by
another uid get silently treated as "no session" and the operator
re-prompts every boot.
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
user**, `0751`, not recursive. hive-priv creates it `0751 root`
on the host before every start; the harness binds its sockets
there as the agent user. Why the mode matters:
[`boundary-mechanism.md`](../../4-implementation/trust-boundary/boundary-mechanism.md#the-per-agent-socket-dir).
A namespaced agent has the dir bound at `/agents/<name>/sockets` and
linked from `/run/hive-agent/<name>`; the activation chowns nothing, because
a mount point under `/run` would belong to root and the id mapping would
give the agent user no ids there. A later in-container activation sees the
link and logs `is a symlink or not a directory; not handing it`. The log
is harmless.
## Matrix per-agent daemon + token-arrival trigger
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
process per agent. Serves its MCP tools directly over
streamable-http (`services.hyperhive.agent.mcp.matrixHttpPort`, no stdio bridge —
same shape as `hive-bash-daemon`), emits hyperhive wake signals
on incoming room events via `/run/hive/mcp.sock`. Conditional on the
agent having at least one `services.hyperhive.agent.matrixAccounts`
entry — there is no separate enable switch, and the same condition gates
the daemon, its path watcher AND the autoinjected
`extraMcpServers.matrix` entry. The module declares the hive-internal
`main` entry whenever `services.hyperhive.agent.matrix.url` is non-null,
so on a real hive (where hive-c0re renders that URL per agent) every
agent has one; a `null` URL with no operator-declared account is the
"this agent has no matrix" state.
**First-boot ordering**: a token can arrive after the container comes
up — a declared token file, or a token the swarm mints into the store.
Without the path-trigger sibling
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
<this agent's state dir>/matrix-token*` — the trailing `*` also catches
a declared secondary token file like `matrix-token-ccc`), the daemon
would exit 0 quietly the first time it ran and the MCP would have no
daemon until the next restart. The `.path` unit makes the appearance
of the token re-fire the service so the daemon comes alive in the
same boot cycle as
provisioning. A token in the store changes no file, so a store-backed agent
also gets a timer that restarts the daemon five minutes after it last
exited. A running daemon re-lists its linked accounts in the store
every two minutes and, when the set changed, exits with status
75, which the unit's `RestartForceExitStatus` restarts. On the same
two-minute tick it tries again each secondary account that didn't come
up — a declared account with no token yet, a token the homeserver
rejected, or a homeserver that
was down — waiting two minutes after its first failure and twice as long
after each further one, up to sixteen minutes. It tries one account at a
time and counts a try that hasn't finished after two minutes as a failure,
so one hung account doesn't hold the others back. One that comes up joins
the running daemon; the primary and the other accounts aren't
restarted. The same token watcher also drives avatar setting: on a
restart the daemon re-runs each account's bring-up, which sets the
avatar (see below).
### matrix avatar (set by the daemon over the live Client)
`hive-matrix-daemon` itself publishes the agent icon (`services.hyperhive.agent.icon`, an SVG) as each matrix
account's profile avatar
(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
the daemon builds + restores an account's `Client` (authenticated,
pointed at that account's resolved homeserver), it calls matrix-sdk's
`account().upload_avatar()` — one call that uploads the media and sets
`avatar_url`. Because it reuses the live Client, there is no hardcoded
homeserver URL, no token re-read, and no token-file globbing: the daemon
already iterates every configured + dashboard-discovered account in its
bring-up loop, so it sets the avatar for **every** account.
Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
unit, gated on `services.hyperhive.agent.icon != null`. No icon configured → the env is
unset → `sync_avatar` returns early and sets no avatar.
<!-- vale write-good.Passive = NO -->
Idempotency is **per-account**: an `avatar-icon-hash` file in each
account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
skips the upload when unchanged, because every upload mints a fresh
`mxc://` URI that emits a profile state event in every joined room —
re-uploading identical bytes is timeline spam. A swarm-UI-linked
account gets its avatar when the daemon restarts on a changed account
listing (exit 75, which re-runs the per-account bring-up), so no separate
avatar trigger is needed. The daemon swallows avatar failures
(logged, non-fatal) so they never break account bring-up or sync.
<!-- vale write-good.Passive = YES -->

178
docs/2-handbook/agents/security.md Normal file
View file

@ -0,0 +1,178 @@
# Security model
## Agent trust model
The implementation docs document the specific mechanisms (nixbld isolation,
privilege separation; see the list at the end). This section frames the model they
serve: **what hyperhive defends, what it deliberately doesn't, and where the
operator is accepting risk.** It's the reference for "is it safe to give an
agent capability X?."
### The trust boundary is the container, not credential storage
<!-- vale write-good.Passive = NO -->
An agent is **trusted code running inside its own nspawn container**. The
boundary that matters is the container: a sub-agent can't see the host
netns, another agent's container, or another agent's state dir. Within its
own container the agent is privileged — it has **passwordless `sudo` by
default**. Isolating credentials _from the agent itself_ is therefore **not a
goal**: an agent can read its own tokens, its own `/home/<name>/.claude`, and
run arbitrary commands as root inside its container. (The narrow exception is
_cross-tenant_ leakage — for example the unsandboxed-nix-build `0600` token policy
(see [nix builds](../../4-implementation/trust-boundary/nix-builds.md)) stops a build's nixbld user reading the agent's own forge token. That hardens
the boundary; it doesn't sandbox the agent from itself.)
<!-- vale write-good.Passive = YES -->
<!-- vale write-good.Passive = NO -->
The corollary: **don't reason about security as "can the agent be stopped from
touching its credentials." Reason about it as "what's the blast radius if this
agent does the worst possible thing with everything it can reach."**
<!-- vale write-good.Passive = YES -->
### Scoped tokens bound the blast radius
Each agent gets its own scoped credentials, never shared:
- **forge token** → that agent's Forgejo account only (its own repos +
collaborator grants; can't act as another agent or as `core`).
- **matrix token** → that agent's matrix account only.
<!-- vale write-good.Passive = NO -->
_Its own_ account's scope bounds a compromised/confused agent's reach on the
forge or matrix, not the swarm's. This is the main thing standing
between "one agent does something unwise" and "the whole hive is affected."
<!-- vale write-good.Passive = YES -->
**Identity vs. secret (matrix).** The scoping is on the _secret_, not the
_identity_: an agent's matrix **token** is private to its own account, but its
matrix **identities** — the public handles (`name`, `user_id` `@user:server`,
`homeserver`) — are intentionally readable by any agent via `GetAgentMeta`, so
peers can find and address one another on a shared matrix instance. Only the
public handle crosses that boundary; the token never does.
### The swarm secret store isn't a boundary between hives
A hive reads its agents' credentials out of the swarm's secret store with its
own certificate. **Every hive's policy grants read on every agent's
credentials**, not only on the agents it hosts — so a compromised hive can read
the matrix token of an agent running on a different hive. That includes each
account's key-backup recovery passphrase, which with the token decrypts the
agent's whole encrypted history, not only messages from then on.
That's deliberate and it's the interim state, not the intent. An agent's
credential path doesn't name the hive hosting it (agents move between hives),
so a per-hive grant has to be an enumeration the controller re-emits whenever
the roster changes — and an enumeration that can drift or land out of order
advertises a boundary it doesn't actually hold. A wide grant that says what it
is beats a narrow one that only looks narrow.
What still holds: the grant is **read-only** (a hive can't write an agent's
credential, so it can't hand itself an agent's identity), and it reaches two
prefixes and nothing else in the store — every agent's credentials, and the
reader's _own_ entry under the hive namespace, which names the hive asking and
so widens nothing between them.
A hive reads no **swarm service's** OIDC client secret. Every reader of
`swarm/services/` presents a certificate that isn't a hive's: Grafana, the swarm
collector and the store forwarder each fetch theirs with a leaf of their own,
and the store's granter fetches the bao UI's.
Three services read **every agent's queue secret**
(`swarm/agents/<agent>/queue`), each under a leaf of its own whose policy
reaches that one leaf per agent: the queue's auth callout (`swarm-nats-auth`),
the log store's agent check (`swarm-logs-auth`, on the log store's host), and
the metrics store's (`swarm-metrics-agent-auth`, under the `swarm-metrics-auth`
leaf on the metrics store's host). Any one compromised can present any agent's
queue secret, at the queue and at both stores' read routes alike. The two store checks also read the
agent's `swarm/agents/<agent>/observability` leaf, which only the controller writes, and let
the agent through only with the `logs` or `metrics` permission it names.
### Per-hive credentials live under the hive kind
`swarm_secret_client::policy` scopes the **hive** stanza narrowly (unlike the
agent stanza above) because a hive's own path names it, so scoping to the
reader's own name costs nothing and drifts nowhere. That only holds if a
credential that must be one-per-hive is actually stored under the hive kind.
**A credential that must be one-per-hive goes under `Kind::Hive`**; putting it
under the agent prefix and expecting the grant to scope it — that's the mistake
this note exists to stop.
### Threat model: prompt injection → confused deputy
The realistic adversary **never needs to breach the container**. They supply
**untrusted input the agent reads and acts on**: a poisoned issue or PR
comment, a cloned repo's README/CI, a scraped webpage, a crafted matrix
message. The agent is the trusted, capable party; the _input_ is the
untrusted part. A successful injection turns the agent into a **confused
deputy** — it uses its legitimate capabilities (push, comment, deploy, run
shell) on the attacker's behalf.
Mitigations are therefore about **bounding capability and inserting human
checkpoints**, not about sandboxing the agent from its own tools:
- **Operator merges, not the agent** — an agent may _push_ branches, but a
**human (the operator) merges the PR**, keeping a person in the loop on the
highest-value action. On the **internal forge this is technically enforced,
not just convention**: agents can't create repos (`max_repo_creation = 0`),
and `main` on an `agent-configs/<name>` repo carries swarm-controller's
branch protection: merge and approval allowlisted to the `operators` team,
which swarm-controller converges on every config repo. An operator's merge
there is also what deploys the config. Existing `agents/<repo>` repos carry
the same merge gate.
An agent (a write collaborator, not a repo admin) can neither change those
settings nor merge its own PR. It's **not** set up for external VCS (GitHub
etc.), though — there, operator-merge is process + accepted risk, not a
technical control.
- **Operator sign-off** — config changes are config PRs an operator merges
on the forge (see
[`config-changes.md`](config-changes.md)), and
scheduled prompts are operator-only, set at swarm level
(see [`schedules.md`](../swarm/schedules.md)).
Before that merge, the swarm builder evaluates an unmerged config PR
off-hive, on any host that sets `deploy.builder.enable` (the
all-swarm-services switch defaults it on): it builds the
PR's head with a read-only forge token, in its own netns, as an untrusted
client of its host's nix daemon (see [`builder.md`](../../4-implementation/swarm/builder.md)).
### Capability = accepted risk
Every capability granted to an agent is a risk the operator is **explicitly
accepting**. The rule of thumb:
> **Don't give an agent access to something you can't afford to lose.**
If an agent can deploy to prod, you are accepting the risk of a dropped
production database (via injection or plain error). If that's unacceptable,
the answer is _don't grant the capability_ — not "grant it and hope the
sandbox holds," because there is no sandbox between an agent and the tools you
handed it.
### No autosandboxing of external tokens
hyperhive provisions and scopes its **own** per-agent forge + matrix tokens.
It does **not** automatically sandbox or scope **external** credentials
(GitHub PATs, cloud keys, third-party API tokens). The scope of an external
token is **operator-accepted risk**: if you drop a broadly scoped GitHub token
into an agent's config, that agent has exactly that reach, with no hyperhive
layer narrowing it. Scope external tokens tightly at the source (the external
provider) before handing them over.
## Agent-supplied names in paths
Wherever an agent-supplied name reaches a filesystem path, hyperhive validates
it first. The agent socket's `GetAgentMeta` takes `name` as a serde-validated
`hive_types::Ident` (or falls back to `Ident::parse` for the "self" case)
before building `agent_notes_dir(name)`, so a `..` component can't traverse.
## Implementation detail
- How in-container `nix build` stays apart from the credentials, and its
threat model: [nix builds](../../4-implementation/trust-boundary/nix-builds.md).
- How `hive-c0re` runs unprivileged and delegates root operations:
[privilege separation](../../4-implementation/trust-boundary/hive-c0re-privilege.md).

423
docs/2-handbook/agents/web-page.md Normal file
View file

@ -0,0 +1,423 @@
# Per-agent page
> Part of [Agents](README.md). See also:
> [Shape (shared)](../../4-implementation/web-ui/shape.md) · [Dashboard layout](../hives/dashboard.md)
Two fixed-position layers frame a full-viewport terminal: a header and
scrollable main content — plus a slide-in side panel for flyouts and long
content. The page has no message box: to write to an agent, use the swarm
UI's terminal.
## Header
Preact component tree (`Header.tsx` + `StatusChips.tsx` +
`MetaNav.tsx` + `HeaderPill.tsx`, wired together in `Root.tsx`) — see
`frontend/packages/agent/src/components/`. This section describes the
rendered component tree: every element is Preact component output.
**Fixed-overlay header** (`<header class="agent-header">`): frosted
glass — `backdrop-filter: blur` lets scrolled terminal rows show
through. Measures its own rendered height via `ResizeObserver`
(`Header.tsx`) and writes it to a CSS custom property the content
below reads for its offset, so content never overlaps the header
regardless of how many rows of badges/pills wrap onto an extra line
at a given viewport width. Two columns:
<!-- vale write-good.Passive = NO -->
- **Agent icon** (`<img class="agent-icon">`): fixed-size square
identity anchor — `width: 5em; height: 5em` with explicit pixel
sizing so the `<img>`'s intrinsic (large) dimensions don't push
the parent flex container open via `align-items: stretch`-driven
height feedback. Falls back to the dimmed hyperhive mark
(`/favicon.svg`) on load error (`/icon` 404s when the agent has no
`services.hyperhive.agent.icon` override — there is no server-side default image).
- **Main column** (`.agent-header-main`, `Header.tsx`): two rows —
title (`◆ <label> ◆`) and, when set, the "swarm / hive" identity
line. Nothing else lives here; both rows are short, fixed-shape text
that doesn't wrap in practice, so this column's height stays stable
regardless of how many badges or pills are showing (a design
correction mid-review — the first pass put the variable-width
content here instead, which is exactly the kind of thing the
`ResizeObserver` measurement above exists to catch even when a
layout choice reintroduces it).
- **Pills cluster** (`.agent-header-pills`, right-aligned): status
badges (`StatusChips.tsx`) and flyout triggers together in one row,
so the header reads as one identity zone + one status/actions zone:
- **Alive badge**: `● alive` (green) / `⊘ rate limited` (red) /
`◌ needs login` / `◌ logging in` / `○ offline` / `… connecting`.
- **State badge**: `💤 idle` / `🧠 thinking` / `📦 compacting` /
`○ offline` / `… booting` + age suffix.
- **Model and effort badges** (`model · <name>`, `effort · <level>`):
display only. Each tooltip says where the value comes from
(`state.model_source` / `state.effort_source`: set on the swarm,
agent.nix, an existing `hyperhive-model` / `hyperhive-effort` file, or
the built-in default); the model's also says what the last turn
resolved it to. Both are set on the swarm: the swarm UI's agents page,
or `swarmctl agent inference set`
([`inference.md`](../../4-implementation/swarm/inference.md)).
- On an ACP agent each badge shows the session's own value when it
doesn't offer the requested one. The effort badge is hidden
(`state.effort` is `null`) until the first turn has attached a
session, when the model offers no effort levels, and while a newly
set model waits for the next turn, since the levels come with the
model.
- **Ctx / cost badges**: `ctx · 142k` (last inference's prompt size,
tooltip shows % of context window) / `cost · 1.3M` (cumulative
tokens billed across every inference in the last turn).
- **Pause badge** (`⏸ pause` / `▶ resume`): toggles via the same
hive-c0re-owned `/api/pause/<name>` / `/api/resume/<name>`
endpoints the dashboard uses (pausing needs a write this
unprivileged process can't make directly).
- **Inbox / todos pills** (`📬 inbox · N` / `📋 todos · N`): hidden
when empty; selecting it opens the matching flyout in the side panel.
- **Profile avatar**: the signed-in user's initial, opening a popover with
their name and links to authelia's settings and sign-out pages, as on the
dashboard (`AgentUserMenu.tsx`). It shows only behind the gateway, where
the page shares the dashboard's origin and session; opened on the agent's
own port there is no sign-in and no avatar. The name comes from the
dashboard's `/api/whoami`, authelia's URL from the dashboard's
`/api/state`.
- **Links badge** (`🔗`): opens a popover listing `↑ dashboard`
(back-link to the host dashboard, `${dashboardBase}`)
followed by this agent's `agent_links()`-sourced navigation (stats,
screen when VNC is enabled, forge profile + config-repo mirror when
the agent has a forge account, any `services.hyperhive.agent.dashboardLinks`
extras) — all as real `<a>` elements, not a `Dropdown`-style
command list, so ctrl/middle-click and "copy link address" keep
working. Each `AgentLink.kind` resolves differently: `container` →
same-origin path; `forge` → `state.forge_public_url + url`, and
hive-agent drops the link entirely when that's unset (never guessed from
`<host>:3000`); `external` → already absolute. Same source
(`GET /api/state`'s `links` field) also feeds
`DashboardState.links` for the dashboard card's icon strip —
`agent_links()` in hive-agent is the single source of truth for
both. The dashboard link lives in this 🔗 popover. `cancel turn`,
`new session` and `log out` are rows of the state badge's menu, each
with a click-again confirm (see Actions below).
<!-- vale write-good.Passive = YES -->
Values throughout come from `GET /api/state`'s cold-load snapshot,
kept in sync afterwards by the SSE stream (see Live view below) —
`context_window_tokens` for the ctx badge tooltip, `qualified_label`
(the hive-qualified `name@domain` form, used for the browser tab title
so two tabs from different hives are distinguishable; the header's own
`◆ <label> ◆` stays short).
## Main content
**Main content** (`<main class="agent-main">`): fills the viewport
and scrolls behind the fixed header + footer.
<!-- vale write-good.Passive = NO -->
- `#status` overlay: empty when online; shows the login form / OAuth
URL when `status` is `needs_login_*`. The OAuth code input is
`type="password"` with a `👁 reveal` toggle that flips it back to
`text` on press so the operator can sanity-check the paste before
submit — avoids accidental on-screen token exposure to
shoulder-surfers or screenshots. `autocomplete="one-time-code"`
is the semantic value for OAuth codes (per WHATWG): browsers may
silently ignore `autocomplete="off"` on `type="password"`, but
browsers honour `one-time-code` and suppress the "save password
for this site?" prompt that would otherwise fire on submit.
- Terminal-wrap: live event tail (sticky-bottom autoscroll +
`↓ N new` pill when not at bottom). The pill is **anchored in
`.agent-main`**, not in `log.parentElement = .terminal-wrap`:
`.terminal-wrap` applies `backdrop-filter: blur` for the frost
effect, which creates a CSS stacking context — anchoring the
pill inside that context would confine its `z-index` below the
fixed header in the root stacking context, and it'd never
float. `.agent-main` has no backdrop-filter (no stacking-context
creators), so the pill's `z-index` reaches the root and properly
composites above the header. Geometry is unchanged —
`.agent-main` and `.terminal-wrap` both `inset: 0` fill the same
area.
<!-- vale write-good.Passive = YES -->
## Side panel
**Side panel** (slide-in from right): singleton shared with the
dashboard's side panel shape. Carries inbox and todos flyouts (opened
via the header pills) as well as long content (file previews, diffs,
journald logs). Inbox flyout: unread messages addressed to this agent
(`acked_at IS NULL`, newest-first, up to 30); reply messages indented
with `↳ reply ·` in amber. A `✓ mark all read` button appears in the
flyout header when the inbox is non-empty; selecting it confirms then
POSTs cross-origin to the core dashboard's `POST
/api/agent/{name}/mark-all-read`, which acks all pending messages for
this agent so the harness won't receive wake-prompts for them. A `{
marked: N }` pill surfaces the count. After the drain the inbox list
empties on reload (the filter is `acked_at IS NULL`, so drained
messages disappear). Todos flyout ("loose-ends v2"): the harness-local
todos other subsystems push at this agent (`GET /api/todos`) — matrix,
forge, and bash are the built-in producers, but any user-configured MCP
server can push its own via the same in-agent socket. Each row shows the
producing subsystem, an
optional source label, a summary, and age. A checkbox per row plus a
select-all / select-none / `✓ mark done` bulk row above the list POSTs
the checked ids to `POST /api/todos/mark-done`, which dismisses them
from the harness-local store (same effect as `cancel_loose_end(kind:
"todo")`, just from the web UI instead of the agent's own tool calls).
If the todo store errors or doesn't answer for any id, the endpoint returns
500 naming how many ids failed, and the panel shows `failed: …` instead of
`✓ marked done`.
## Live view
Each agent runs an `events::Bus`: a `tokio::sync::broadcast<LiveEvent>`
plus a sqlite-backed history at `/state/hyperhive-events.sqlite`.
The harness emits `TurnStart { from, body, unread }`,
`Stream(value)` (one per parsed stream-json line), `Note`,
`TurnEnd { ok, note }`. Each event also carries a `ts` (unix
seconds) — a flattened sibling of the event tag on both the live
SSE frame and the replayed history rows. The web UI:
- fetches `GET /events/history` on page load and replays the last
2000 events (oldest first);
- then subscribes to `GET /events/stream` (SSE) for live tail;
- shows a granular state badge above the terminal, driven
authoritatively from `/api/state.turn_state`. SSE turn_start /
turn_end still flip the badge instantly between renders;
- terminal-themed: phosphor mauve glow, Crust bg,
backdrop-filter blur, row fade-in slide-up.
The backfill/live-tail dedupe, sticky-bottom autoscroll, and "↓ N
new" pill are the shared terminal-pane mechanics described in
[Shape](../../4-implementation/web-ui/shape.md#shared-terminal-pane) — this page's log is one
instance of that same factory.
Per-stream rendering (see [`docs/4-implementation/web-ui/terminal-rendering.md`](../../4-implementation/web-ui/terminal-rendering.md) for
the full row taxonomy and dispatch logic):
- `Stream` `tool_use` →
- `Write` / `Edit`: collapsed `<details>` with a +/- diff body
(`-` lines from `input.old_string`, `+` lines from
`input.new_string` or every line of `input.content`).
Summary carries the path + line counts.
- others (`Read /path`, `Bash $ cmd`, `mcp__hyperhive__send →
operator: "..."`, etc.): flat one-line per-tool format with
per-tool salient-arg extraction (`fmtToolUse`).
- `Stream` `tool_result` short → flat `← ...`; long → collapsed
`<details>` `▸ ← Nl · headline` (select to expand full body).
- `Stream` `thinking` → `.thinking` row with a `💭 thinking …`
indicator.
- `Stream` `system` → handled by subtype: `plugin_install` and
`compact_boundary` emit muted notes; `commands_changed` emits an
expandable details row listing slash commands; `thinking_tokens`
updates a single in-place `🧠` counter; `init`, `result`, and
`rate_limit_event` drop silently (noise / used elsewhere); other
subtypes → muted `⚙ <subtype>` note.
- `Note` → `· text`.
- `TurnStart` → `◆ TURN ← <from>` with the wake-prompt body; a
muted `· HH:MM:SS` time suffix from the event `ts`.
- `TurnEnd` → `✓ turn ok` / `✗ turn fail — note`, with a
`· HH:MM:SS · <duration>` suffix (duration = end − the paired
turn-start), triggers a `refreshState()`. The time suffix is
guarded on a numeric `ts`, so rows degrade cleanly when absent.
## Actions
The state badge's menu carries the actions on the agent's claude:
- `cancel turn` — `POST /api/cancel` → SIGINT to the turn's claude and every
command it started (its process group), emits a Note. On an ACP agent
it sends the agent `session/cancel` instead, and SIGINT if the turn
hasn't ended 10s later. Offered while the state is `thinking`.
- `new session` — `POST /api/new-session` (confirms first).
Arms a one-shot on the Bus; next turn runs without
`--continue`, dropping the resume session entirely.
- `log out` — `POST /api/logout` (confirms first). Wipes OAuth
credential files, parks the agent in `needs_login`, and preserves
session history (`~/.claude/projects/`).
`POST /api/compact` sets the deferred `Bus::request_compact()` flag and
returns immediately. The harness runs the `/compact` at the next turn
boundary (end of the in-flight turn, or `turn::run_pending_compact` when
idle); output streams into the live panel, which ends with a `/compact done`
or `/compact failed: …` note. It works mid-turn, not only when idle. Nothing
on this page calls it now.
## Per-agent endpoints
Successful POSTs return 200 (no 303 redirects). Error responses
use semantic status codes: **400** for missing/malformed input
(`body` required, unknown model name, unrecognised effort level),
**409** for retryable state conflicts (turn in flight when
`/compact` fires), **500** only for genuine server/transport
failures. The matching
mutations fire `LiveEvent` variants on the per-agent bus, so the
client doesn't refetch `/api/state` on submit — the SSE stream
delivers the new state faster anyway.
The agent has no login flow of its own. An agent without credentials
reports `needs_login_idle`, the operator logs in from the swarm UI, and the
credentials arrive from the swarm secret store.
- `POST /api/cancel` — SIGINT the in-flight claude turn and the
commands it started. Emits a `LiveEvent::Note`.
- `POST /api/compact` — run `/compact` on the persistent session
(same MCP config + system prompt + allowed tools as a normal
turn — only the stdin payload differs). Flips state to
`Compacting` via `Bus::set_state`, which emits
`TurnStateChanged`.
- `POST /api/new-session` — arm a one-shot for the next turn to
drop `--continue`. Emits a `LiveEvent::Note`.
- `POST /api/logout` — three-step teardown that re-uses the
existing `wait_for_login` resumption path:
1. SIGINT any running claude (matches `/api/cancel`'s pattern —
idempotent no-op when nothing is running) so the credential
wipe doesn't race a mid-API-call turn.
2. Delete only the OAuth credential files
(`.credentials.json` + `mcp-needs-auth-cache.json` under
`~/.claude/`). **Preserves** session history files
(`projects/<hash>/*.jsonl`), sessions, shell-snapshots,
plans, settings, telemetry, and the dir itself, so
`claude --continue` keeps working after a fresh login.
⚠️ Deleting the whole `~/.claude/` directory instead breaks
session continuity — logout must stay narrowed to just the
credential files.
3. Flip `LoginState::NeedsLogin` + emit a `LiveEvent::Note`
naming the files it just deleted, then emit
`needs_login_idle`. The turn-loop's next iteration parks
into `wait_for_login`, which snapshots the credential dir
(now missing the wiped files) and resumes when a fresh
credentials file appears (a login from the swarm UI stores one,
and the agent installs it).
Always returns 200 with a body describing what happened — it
folds per-file errors into the response + the Note so the
operator sees them in the live panel rather than as an HTTP
error. Missing files (already logged out) count as
idempotent.
- `GET /api/state` — cold-load snapshot (`StateSnapshot`) consumed by
the frontend (`useAgentState.ts`) on page load. Includes `turn_state`,
`context_window_tokens`, `qualified_label`,
`model`, `model_source`, `effort`, `effort_source`, `links`, and
other fields described inline throughout this document. All
subsequent state updates arrive via SSE.
- `GET /api/dashboard-state` — lean snapshot of agent-owned fields
fetched once per running agent by the dashboard's container row to
get fresh values without relying on hive-c0re's periodic file-reads.
Returns `{ status_text?, status_set_at?, ctx_tokens?, context_window_tokens,
rate_limited, links }`. Only called when the container is running;
skipped (muted badges) when stopped. Also accessible via the gateway
at `/agent/<name>/api/dashboard-state`.
- `GET /api/todos` — harness-local todos snapshot (loose-ends v2)
consumed by the todos flyout (`refreshTodos` / `buildTodosList`).
See the todos-flyout paragraph above for the producer/subsystem
model and the bulk mark-done row.
- `GET /api/stats?window=24h|7d|30d|all` — time-bucketed turn analytics
`Snapshot` consumed by the `/stats` page. `all` ranges from the
earliest recorded turn with an adaptive bucket width.
- `GET /icon` — agent's icon as `image/svg+xml`. Returns
`/etc/hyperhive/icon.svg` (set via `services.hyperhive.agent.icon` in `agent.nix`)
when present, otherwise **404** — there is no server-side default.
Consumers (dashboard container row, this page's own header icon)
hit `/icon` optimistically and fall back client-side on load failure
to the frontend-bundled `/favicon.svg` rather than probing first.
- `GET /events/history` — replay buffer for the terminal.
- `GET /screen` — VNC viewer page (minimal RFB-over-WebSocket
renderer — deliberately thin, just enough to display the
desktop + forward pointer + keyboard. A production-grade viewer
would vendor noVNC; this file ships the minimal in-tree variant).
Only accessible when `services.hyperhive.agent.gui.enable = true` in the agent's
`agent.nix`; the harness shows a 🖥 screen link in the state row
when `gui_vnc_port` is present. A `← agent` back-link in the page
header returns to the main per-agent page. Controls: `⤢ fit` CSS-downscales
the canvas to the window via `relayoutCanvas()` setting explicit
pixel dimensions on the canvas — *not* CSS `max-width/max-height`,
because a flex item's automatic minimum size (`min-width: auto`
resolves to the canvas's intrinsic framebuffer resolution) silently
clamps `max-*` back up, making fit mode a no-op that just centred +
clipped the oversized canvas. The fit-mode rules pin the canvas
with `flex: none; min-width: 0; min-height: 0` so the JS-set size
sticks. `⤡ match size` sends an RFB `SetDesktopSize` request so the
server (weston) changes its real output resolution to the window
dimensions; enabled once the server advertises the
`ExtendedDesktopSize` pseudo-encoding (`-308` rect in the header).
Fit-mode state persists in `localStorage` (`screen-fit`); default
is on. `sendPointer` rescales pointer coordinates so presses
land on the right pixel regardless of CSS scale.
- `GET /screen/ws` — raw RFB byte relay: proxies WebSocket
frames to the weston VNC server at `127.0.0.1:<vnc_port>`.
Transparent to any RFB variant. VNC port comes from the
`HIVE_GUI_VNC_PORT` env var (a fixed port set on the harness
service when `services.hyperhive.agent.gui.enable`; see `weston-vnc.nix`).
- `GET|POST /extra/<name>/…` — **extra web proxies** declared via
`services.hyperhive.agent.extraWebProxies` in `agent.nix` (serialised to the
`HIVE_EXTRA_WEB_PROXIES` env var as a JSON object
`{"<name>": "<upstream_url>"}`). Each entry mounts a transparent
reverse-proxy at `/extra/<name>/` that forwards every request (method,
headers, body) to the configured upstream, strips hop-by-hop headers
on both sides (repeated response headers such as two `Set-Cookie` lines
are all forwarded), and buffers the full response body (MVP — SSE
connections will appear as one large response rather than streaming).
The `/extra/` namespace ensures user-declared proxies can never
conflict with native agent endpoints. Upstream values are either an
`http(s)://` URL (forwarded via `reqwest`) or a Unix domain socket,
spelled `unix:<path>` (for example `unix:/run/myapp/http.sock`) — dialed
directly with a raw HTTP/1.1 client per request, since `reqwest` has
no UDS transport. Implemented in `web_ui/proxy.rs`.
Bus events (new vocabulary on `/events/stream`):
- `status_changed { status }` — `online` / `rate_limited` /
`needs_login_idle`. Drives the
alive-badge. The harness sets `rate_limited` when it detects a
429 response and clears it when the retry sleep expires.
- `model_changed { model }` — drives the model chip. The harness emits it
when it applies or clears a swarm value.
- `effort_changed { effort }` — drives the effort chip. The harness
emits it when it applies or clears a swarm value.
- `token_usage_changed { ctx: TokenUsage, cost: TokenUsage }`
— drives the ctx + cost badges. Emitted from
`Bus::record_turn_usage` at turn-end; `ctx` is the last
inference's usage (current context size), `cost` is the
cumulative across every inference (the `result` event's
totals).
- `turn_state_changed { state, since_unix }` — drives the
state badge (`idle`/`thinking`/`compacting`).
## Stats page
`GET /stats` is a separate per-agent page (served by the
harness, linked from the per-agent page's `📊 stats →` and from
each dashboard container row). A `← live` back-link in the page
header returns to the main per-agent page. Turn analytics, read-only, from
`/state/hyperhive-turn-stats.sqlite`. `GET /api/stats?window=
24h|7d|30d|all` returns a time-bucketed `Snapshot`; the page renders
it with Chart.js (bundled into `stats.js` via esbuild — no CDN
dependency). Charts: turns, duration (p50 · p95 · avg), context
tokens, token cost per bucket, a **turns-by-model** stacked bar
(model choice drives token cost, so it sits directly under the
cost chart), doughnuts for tool / wake-source / result mix, and a
**result-trend** stacked bar — per-bucket `result_counts` so
error / rate-limit / compaction outcomes are visible over time
(the doughnut shows only the window total).
A **favorite tools** doughnut shows the most-run shell commands —
normalised `bash_commands` heads written per bash task by the
hive-bash-daemon capture: the basename of the *first real command*,
looking past `cd repo &&` prefixes, env-assignments, and
prefix-runners like `sudo` / `env` (so `cd /repo && cargo build`
records `cargo`, not `cd`). Read via `bash_breakdown`; the card
stays hidden until the agent has run a bash command (a missing
`bash_commands` table degrades to an empty list), so it never
renders an empty chart.
A summary chip row carries window totals, plus two
token-efficiency chips derived from the bucket sums: **cache
hit-rate** (`cache_read` over all input-side tokens) and
**tokens/turn**. When `reminder_stats` is present (fetched via
`ReminderRollup` RPC and merged into the snapshot in
`web_ui/stats.rs::api_stats`) three more chips appear: **reminders
scheduled / delivered / pending** for the window. When the
per-session capture has data, a **first-turn ctx** chip shows the
input tokens of the most recent fresh claude session's first turn —
a proxy for system-prompt + AGENTS.md sprawl (a fresh session's first
turn pays the full static prefix uncached). It's derived in `stats.rs`
(`first_turn_ctx`: the first turn, `ORDER BY started_at LIMIT 1`, of
the latest `sessions` row in the window) and stays out of the JSON
until the `sessions` / `turn_stats.session_id` capture has rows — so
the chip stays hidden on older dbs (inert-until-capture).
`stats.rs` opens the sqlite db read-only and degrades to an
empty snapshot on any error — the page is decorative, never
authoritative.