docs: persistence and config changes become one operator page
The operator parts of the persistence and config-changes pages merge into agents/persistence.md. The sqlite tables, state dirs, mounts and migrations move to 4-implementation/agent-state/persistence.md, and the deploy path, meta flake, deploy markers, startup migrations, root container, system messages and autoupdate move to 4-implementation/agent-state/config-changes.md. Links, code comments and nix comments that named the old sections point at the new pages.
This commit is contained in:
iris
mara
parent
a36b9745b3
commit
0ae34b1cd0
34 changed files with 919 additions and 881 deletions
|
|
@ -15,11 +15,9 @@ docs/4-implementation/agent-tools/swarm-metrics-cli.md
|
|||
# indentation normalisation, so these files need manual re-wrap before
|
||||
# adding them back to the formatter scope.
|
||||
AGENTS.md
|
||||
docs/2-handbook/agents/config-changes.md
|
||||
docs/5-project/process/conventions.md
|
||||
docs/4-implementation/networking/gateway.md
|
||||
docs/2-handbook/swarm/matrix.md
|
||||
docs/2-handbook/agents/persistence.md
|
||||
docs/4-implementation/web-ui/terminal-rendering.md
|
||||
docs/4-implementation/agent-tools/forge.md
|
||||
docs/4-implementation/agent-tools/matrix.md
|
||||
|
|
|
|||
|
|
@ -192,7 +192,7 @@ pick the page for your task rather than reading front to back. Some
|
|||
common entry points:
|
||||
|
||||
- connecting hives into a swarm → [`2-handbook/swarm/`](docs/2-handbook/swarm/README.md)
|
||||
- how config changes flow → [`2-handbook/agents/config-changes.md`](docs/2-handbook/agents/config-changes.md)
|
||||
- how config changes flow → [`2-handbook/agents/persistence.md`](docs/2-handbook/agents/persistence.md#config-changes)
|
||||
- trust boundary + threat model → [`hives/boundary.md`](docs/2-handbook/hives/boundary.md), [`agents/security.md`](docs/2-handbook/agents/security.md)
|
||||
- what an agent's turn looks like → [`4-implementation/turn-loop/`](docs/4-implementation/turn-loop/README.md)
|
||||
- every `services.hyperhive.*` option → [options reference](https://hyperhive.darkest.space/options/)
|
||||
|
|
|
|||
|
|
@ -217,7 +217,7 @@ swarm UI's job view for progress.
|
|||
|
||||
Later config changes are PRs on `agent-configs/<name>`; you merge them on the
|
||||
forge, and the merge deploys them. →
|
||||
[`2-handbook/agents/config-changes.md`](../2-handbook/agents/config-changes.md#config-changes)
|
||||
[`2-handbook/agents/persistence.md`](../2-handbook/agents/persistence.md#config-changes)
|
||||
|
||||
## After setup: use the swarm UI
|
||||
|
||||
|
|
|
|||
|
|
@ -7,10 +7,9 @@ its own page shows and how far you can trust it.
|
|||
|
||||
- **[Roster](agent-roster.md)** — the agents of a hive, and who may do what
|
||||
to whom.
|
||||
- **[Persistence](persistence.md)** — what survives a restart, a destroy and
|
||||
a purge.
|
||||
- **[Config changes](config-changes.md)** — how a change to an agent's config
|
||||
flows from a pull request to its container.
|
||||
- **[Persistence and config changes](persistence.md)** — what survives a
|
||||
restart, a destroy and a purge, and how a change to an agent's config flows
|
||||
from a pull request to its container.
|
||||
- **[The agent's web page](web-page.md)** — the per-agent terminal view, its
|
||||
actions and its endpoints.
|
||||
- **[Security model](security.md)** — the trust model, the prompt-injection
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ agent's `state` dir, read-write, at `/agents/<name>/state`; never `config` or
|
|||
`harness`). An agent holding no permission sees its own dirs and nothing else:
|
||||
`ManageRootAgent` is the only grant that reaches another agent's state
|
||||
dir. See
|
||||
[`persistence.md`](persistence.md)'s _Cross-agent access to state._
|
||||
[`persistence.md`](../../4-implementation/agent-state/persistence.md)'s _Cross-agent access to state._
|
||||
|
||||
### Reading the map-shaped format
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
# Persistence + retention
|
||||
# Persistence and config changes
|
||||
|
||||
Where state lives, what survives what, and how it's bounded.
|
||||
What survives what, and how a change to an agent's config reaches it.
|
||||
|
||||
## For operators
|
||||
## Persistence
|
||||
|
||||
The short answer to "does anything disappear": **destroying an agent
|
||||
keeps its state, purging it doesn't.**
|
||||
|
|
@ -13,7 +13,7 @@ keeps its state, purging it doesn't.**
|
|||
tab of the dashboard); `swarmctl agent create` with the same name and hive
|
||||
recreates it from the kept state, **no re-login needed**.
|
||||
- **`PURG3`** (opt-in, from the dashboard or `hivectl agent <name>
|
||||
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
|
||||
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
|
||||
history, claude credentials, `/state/` notes, everything. **No
|
||||
undo.** Only reach for this when you actually want the agent gone
|
||||
for good. The agent's matrix encryption keys go with the state dir,
|
||||
|
|
@ -35,811 +35,36 @@ Beyond that:
|
|||
for any agent, and she stays gone until you create her again from the
|
||||
swarm dashboard.
|
||||
|
||||
Everything below this point is implementation detail: exact table
|
||||
schemas, file layouts, and internal migration mechanics.
|
||||
|
||||
## Sqlite databases
|
||||
|
||||
### `/var/lib/hyperhive/db/broker.sqlite` (host)
|
||||
|
||||
One table, the message queue:
|
||||
|
||||
- `messages` — every inter-agent / operator-authored message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
in_reply_to / priority`. `in_reply_to` links a reply to its parent
|
||||
row id; the dashboard and per-agent inbox render these as threaded
|
||||
rows.
|
||||
|
||||
⚠️ The `mcp__hyperhive__remind` queue lives in a harness-local,
|
||||
per-agent store — see [`/harness/` contents
|
||||
below](#state-dirs-per-agent) for where reminders (and todos)
|
||||
live.
|
||||
|
||||
The swarm owns power intent, so hive-c0re keeps none on disk. An older build
|
||||
also kept a `kv` table (the agents running at the last `hivectl stop`) and an
|
||||
`agent_power` table (the durable `wanted` intent) here; opening the file drops
|
||||
both.
|
||||
|
||||
On a host whose hive-c0re once kept scheduled prompts, the file also
|
||||
holds `scheduled_prompts` and `scheduled_prompt_targets`. Nothing opens
|
||||
them; scheduled prompts live at swarm level
|
||||
([`2-handbook/swarm/schedules.md`](../swarm/schedules.md)). They're safe to drop by
|
||||
hand.
|
||||
|
||||
Retention:
|
||||
|
||||
- `Broker::vacuum_delivered` runs hourly via a tokio task in
|
||||
`hive-c0re::main`. Drops acked message rows older than 30 days
|
||||
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
|
||||
rows are always kept — the harness `ack_turn`s only after a
|
||||
successful turn, so `requeue_inflight` can still requeue
|
||||
an unacked row on a crash.
|
||||
|
||||
### `/harness/hyperhive-events.sqlite` (per agent)
|
||||
|
||||
Lives inside each container's bind-mounted `/harness/` dir (host
|
||||
path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
|
||||
One table:
|
||||
|
||||
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
|
||||
harness emits during turn loop execution.
|
||||
|
||||
The db runs in WAL mode, so `-wal` and `-shm` files sit beside it while
|
||||
the harness is up. Copy or back up the three together, or the recent rows
|
||||
sit in the `-wal` file until a checkpoint.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
The harness both writes and vacuums it. Cleanup runs in-container
|
||||
because hive-c0re runs as the unprivileged `hive-core` user under
|
||||
privsep and can't delete agent-owned files (host-side deletes hit
|
||||
`PermissionDenied` on the bash-task trio and a readonly database error
|
||||
here). `hive-agent`'s `vacuum::run`
|
||||
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
|
||||
**type-scoped**: it deletes only the verbose `stream` rows (the raw
|
||||
claude `stream-json` deltas — one per text chunk / tool use, the bulk
|
||||
of the file's size) older than 14 days, and keeps every other kind
|
||||
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
|
||||
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
|
||||
small and carry the semantic per-turn history the operator scrolls
|
||||
back through when debugging a regression. Age-only within the
|
||||
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
|
||||
history sooner than a quiet one. The trade-off (accepted): a
|
||||
misbehaving harness can skip its own cleanup — but a compromised harness is
|
||||
already inside the container trust boundary
|
||||
([`docs/2-handbook/agents/security.md`](security.md)), and these are ephemeral local
|
||||
artifacts, so cleaning them up where they live is the honest fix.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
|
||||
setups). On open failure the `Bus` falls back to no-store mode
|
||||
rather than crashing the harness — events still broadcast over SSE,
|
||||
just nothing persisted.
|
||||
|
||||
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
|
||||
|
||||
Per-turn analytics sink. One row per claude turn captures
|
||||
identity (`model`, `wake_from`, `result_kind`), timing
|
||||
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
|
||||
cache_read / cache_creation token counts), behaviour
|
||||
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
|
||||
snapshot metrics (`open_threads_count`,
|
||||
`open_reminders_count` — fetched via the same socket the harness
|
||||
already uses for `GetOpenThreads` + `CountPendingReminders`).
|
||||
Bin-loop helpers `build_row` + `record` land each row at
|
||||
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
|
||||
the turn loop continue.
|
||||
|
||||
The `hive-bash-daemon` (not the harness) writes a sibling
|
||||
`bash_commands(ts INTEGER, head TEXT)` table in the same
|
||||
file: one row per executed bash task recording the normalised command head -
|
||||
the basename of the first real command, looking past `cd repo &&`
|
||||
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
|
||||
backs the "favorite tools" view on the /stats page. Best-effort and created on first write
|
||||
(`CREATE TABLE IF NOT EXISTS`), so it's absent until a bash
|
||||
task runs.
|
||||
|
||||
turn-stats.sqlite has **no vacuum** — it's one small row per turn
|
||||
(~hundreds of KB even over months), read directly by the `/stats` page,
|
||||
so pruning it would only lose trend
|
||||
history for no space gain.
|
||||
|
||||
### `/state/hyperhive-harness.json` (per agent)
|
||||
|
||||
Consolidated harness state file written atomically (`.tmp` + rename) by
|
||||
`Bus::emit_status` whenever rate-limited or login-failed flags change.
|
||||
Shape:
|
||||
|
||||
```json
|
||||
{ "rate_limited": false, "needs_login": false, "active_model": "…" }
|
||||
```
|
||||
|
||||
- `rate_limited` — set when the harness detects a 429 from the Claude
|
||||
API; cleared by any subsequent status emit. Drives
|
||||
`ContainerView.rate_limited` on the dashboard.
|
||||
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
|
||||
cleared by `"online"` status (re-auth completed). Drives the
|
||||
`needs_login` flag alongside the `claude_has_session` check.
|
||||
- `active_model` — the resolved Claude model. The hive dashboard no longer shows it; hive-c0re still reads it into the container view.
|
||||
|
||||
The turn loop is the only writer today, but it still goes
|
||||
read-modify-write under a shared in-process lock and merges into the
|
||||
existing object rather than reconstructing it — so a second writer
|
||||
would preserve fields it doesn't own, and the lock closes the
|
||||
lost-update window between a writer's read and its rename. The lock
|
||||
is in-process only, so it wouldn't serialise a writer running as a
|
||||
separate process; none of today's writers are.
|
||||
|
||||
hive-c0re reads this file on each `build_all` sweep (~10s) via
|
||||
`container_view::read_harness_flags`. Falls back to the individual
|
||||
sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) when the
|
||||
JSON is absent.
|
||||
|
||||
### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
|
||||
|
||||
Full stdout + stderr capture for every `nixos-container` / `nix
|
||||
build` invocation the lifecycle layer fires. One row per invocation;
|
||||
the row accumulates lines as the child runs.
|
||||
|
||||
Capturing the full stream (rather than a short tail buffer) matters
|
||||
because real eval errors routinely run long — "tried alternatives"
|
||||
blocks alone are often 30+ lines — so a truncated tail would cut off
|
||||
the actual failure and leave only the host journal holding the
|
||||
complete output. With this table the dashboard can surface the entire
|
||||
log.
|
||||
|
||||
Three indices:
|
||||
- `(agent, started_at)` — backs the per-agent latest-N lookup used
|
||||
by the agent card chip.
|
||||
- `(status, finished_at)` — backs the retention sweep that runs
|
||||
as part of the existing hourly vacuum.
|
||||
- `(node_id)` — added by a later migration so a build log row can be
|
||||
looked up by the job-queue node it belongs to (a `hive_jobq` node is
|
||||
immutable after insert, so hive-c0re records the link on the log row
|
||||
instead); `node_id` is nullable.
|
||||
|
||||
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
|
||||
log a warning on sqlite error and let the build continue. A failed
|
||||
log row never blocks a rebuild.
|
||||
|
||||
Each operation also logs to the host journal, target `nixos-container`, and the
|
||||
swarm log store collects those lines. Every line carries `build_id` (this
|
||||
table's row id; empty without the store), `agent`, `kind` and `cmdline`. A
|
||||
`start` line (`event=start`) opens the build, one line follows per line of
|
||||
output, and a `finish` line (`event=finish`) closes it with `status` (`ok` or
|
||||
`fail`) and `runtime_secs`. hive-c0re logs a nix warning or error at warn, and a failed
|
||||
finish too.
|
||||
|
||||
A row is open (`finished_at IS NULL`) only while its child runs. At startup
|
||||
`Coordinator::open` closes every row still open as `fail` and appends a
|
||||
line saying hive-c0re stopped: the previous process died mid-build, and
|
||||
the retention sweep never deletes open rows.
|
||||
|
||||
### `/harness/hyperhive-model` (per agent)
|
||||
|
||||
Single-line text file holding a claude model name. The harness only
|
||||
reads it, at boot in `Bus::new` and again when the swarm clears the
|
||||
model it set. It holds the model the agent runs when neither the swarm nor
|
||||
`HIVE_DEFAULT_MODEL` (agent.nix) names one. Nothing in the harness
|
||||
writes it. Path overridable via `HYPERHIVE_MODEL_FILE`. Survives
|
||||
destroy/recreate, gone on `--purge`.
|
||||
|
||||
### `/harness/paused` (per agent)
|
||||
|
||||
Marker file, empty when an operator paused the agent. Its presence parks the agent's turn loop: the
|
||||
harness keeps serving its web UI and MCP daemons but drives no turns,
|
||||
and inbox messages queue unacked until it's removed (see
|
||||
[turn loop](../../4-implementation/turn-loop/README.md#the-loop)).
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
Unusually, it's read and written from **both** sides of the harness
|
||||
bind-mount, and that's the whole design: the harness stats it
|
||||
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
|
||||
stats it on the host (`Coordinator::is_paused`) to populate the
|
||||
`paused` field on the agent card, and asks hive-priv to create or remove it
|
||||
(`Coordinator::set_paused`; hive-c0re itself is unprivileged) for `hivectl agent <name> pause|resume` and the
|
||||
dashboard toggle. Because the file itself is the only shared state
|
||||
there's no protocol between them, no round-trip into the container, and
|
||||
pause keeps working when the harness is wedged or the container is
|
||||
stopped.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
When the harness pauses itself, after three todo wakes in a row that skipped
|
||||
`get_loose_ends`, it writes the reason into the marker. The agent's turn-state
|
||||
header reports a non-empty marker as `paused_by_hive`
|
||||
([swarm](../swarm/README.md)), and the swarm then declares the
|
||||
agent `paused` so the hive's wanted-state convergence doesn't resume it.
|
||||
|
||||
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
||||
is the agent's own space to fill, and this is harness control state.
|
||||
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
||||
back paused after a restart, which is the intended behaviour rather
|
||||
than an accident of storage.
|
||||
|
||||
## State dirs (per agent)
|
||||
|
||||
Under `/var/lib/hyperhive/agents/<name>/`:
|
||||
|
||||
- `config/` — unused. Older hives kept a proposed nix repo here; nothing
|
||||
reads it any more, and you can delete it. An agent reads its config by
|
||||
cloning `agent-configs/<name>` from the forge.
|
||||
- `claude/` — claude OAuth credentials, bind-mounted RW to
|
||||
`/home/<name>/.claude` inside the container.
|
||||
- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
|
||||
to `/agents/<name>/state` inside the container (uniform for
|
||||
all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
|
||||
the same path to in-container scripts. Notable files written here
|
||||
by the harness:
|
||||
- `hyperhive-status` — single-line free-text status string written
|
||||
by `set_status`; cleared on explicit `set_status("")`. Read by
|
||||
hive-c0re and the per-agent `/api/dashboard-state` endpoint to
|
||||
surface the status chip on the dashboard. Absent when no status
|
||||
has a value.
|
||||
- `hyperhive-harness.json` — rate-limited / needs-login flags read
|
||||
by the dashboard's async container-state fetch. See
|
||||
`docs/2-handbook/hives/dashboard.md::Container row`.
|
||||
|
||||
The agent writes its own instructions to `AGENTS.md` here; claude and
|
||||
opencode load it automatically from their cwd, `state/`. At boot,
|
||||
`agents_md_migrate::run` renames a `state/CLAUDE.md` to `AGENTS.md`
|
||||
when no `AGENTS.md` exists. When both exist it renames neither and
|
||||
logs an error naming both paths: claude loads only `CLAUDE.md` while
|
||||
one exists in its cwd or above. The harness also publishes the file
|
||||
to the swarm, which reads that copy, and applies the git diffs the
|
||||
swarm sends to edit it. A new agent's first start writes the text given at
|
||||
create, only when no file exists. This file stays the only copy that counts:
|
||||
[`docs/4-implementation/swarm/controller.md`](../../4-implementation/swarm/controller.md#an-agents-agentsmd).
|
||||
- `harness/` — harness-internal ephemeral state; not intended for
|
||||
agent consumption. Bind-mounted to `/agents/<name>/harness`
|
||||
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
|
||||
- `bash-tasks/` — task JSON + stdout/stderr files for
|
||||
background `mcp__bash__run` jobs. JSON files are
|
||||
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
|
||||
(full captured output). The harness's own hourly sweep
|
||||
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
|
||||
event rows above) deletes terminal task trios older than 48
|
||||
hours; non-terminal (still-running) tasks are never deleted. This
|
||||
sweep runs in-container, for the same privsep-ownership reason as
|
||||
the events vacuum above.
|
||||
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
|
||||
and reminders, one small table each in a single file (in-container
|
||||
daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
|
||||
`hive-forge-notify` — upsert keyed todos here over the harness's
|
||||
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
|
||||
`get_loose_ends` output and clears a row on `mark_todo_done`).
|
||||
At boot, `db_migrate::run` creates this file from
|
||||
`hyperhive-todos.sqlite` and `hyperhive-reminders.sqlite` when those
|
||||
are present. Also backs the `mcp__hyperhive__remind` queue.
|
||||
|
||||
The harness itself is also a producer, not just the socket server:
|
||||
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
|
||||
socket the out-of-process daemons above dial) alongside
|
||||
`disk_watch::run` — an *in-process* todo producer that shares the
|
||||
store + wake `Notify` directly rather than dialling its own socket.
|
||||
`disk_watch` raises a keyed `disk` todo when the filesystem backing
|
||||
this agent's state gets tight, naming the agent's own biggest
|
||||
directories; it buckets the summary and carries no raw byte
|
||||
counts, so an unchanged situation re-upserts as `changed == false`
|
||||
and never re-wakes.
|
||||
|
||||
Retention, same hourly `hive-agent::vacuum::run` sweep as
|
||||
`hyperhive-events.sqlite` below: it reaps delivered (soft-deleted) reminder
|
||||
rows 14 days after delivery, kept that long only to serve
|
||||
the trailing-window `ReminderRollup` stats, and reaps acked todo rows
|
||||
30 days after acking (long enough that only a genuinely quiet
|
||||
month triggers the "one spurious re-announcement" fallback a
|
||||
reconciled producer like `disk_watch` relies on — see `todos.rs`'s
|
||||
module doc). Un-acked todos and undelivered reminders are never
|
||||
swept — same "audit trail, not cache" treatment as the c0re-side
|
||||
tables above.
|
||||
- `hyperhive-events.sqlite` — turn-loop event log.
|
||||
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
|
||||
- `hyperhive-model` / `hyperhive-effort` — single-line model name and
|
||||
effort level files, read-only fallbacks.
|
||||
- `work/` — the agent's working space for git worktrees, build dirs
|
||||
and other large working data. Bind-mounted RW to
|
||||
`/agents/<name>/work` inside the container and owned by the agent
|
||||
user. It sits outside `state/` so a claude or subagent started under
|
||||
`work/` doesn't walk up into `state/AGENTS.md` and load the agent's
|
||||
own identity. Survives `destroy`/recreate, gone on `--purge`. Counted
|
||||
in the per-agent disk metric (`measure_agent_disk`) together with
|
||||
`state/` and the container rootfs. The dashboard doesn't serve it.
|
||||
|
||||
### The container root (per agent)
|
||||
|
||||
An agent container starts with `systemd-nspawn --ephemeral`
|
||||
(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a
|
||||
snapshot of `/var/lib/nixos-containers/h-<name>`, and the snapshot vanishes
|
||||
at stop. The container loses anything the agent writes outside a bind: `/home/<name>`
|
||||
apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is
|
||||
`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container
|
||||
`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir.
|
||||
`/home/<name>` holds caches and nothing the agent should keep: worktrees,
|
||||
clones and builds go in `work/`.
|
||||
|
||||
The nix profile needs one extra step. nix keeps a non-root user's profile in
|
||||
`~/.local/state/nix`, so the agent's tmpfiles rules
|
||||
(`nix/agent-modules/user.nix`) point that directory into
|
||||
`/nix/var/nix/profiles/per-user/<name>` (the persistent bind) and recreate
|
||||
`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or
|
||||
`nix profile` survive a restart, and go with the container on `destroy`.
|
||||
|
||||
**The one-time salvage.** The first start of an agent whose root has content
|
||||
runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the
|
||||
`WriteNspawnFlags` that precedes every start, while the container is down:
|
||||
|
||||
1. Everything in the root except the cache paths (`tmp`, `var/cache`,
|
||||
`var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is
|
||||
copied to `work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, with
|
||||
modes and mtimes kept. The agent finds its old `/home/<name>` there.
|
||||
2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there).
|
||||
3. hive-priv renames the old root to `h-<name>.pre-ephemeral-<YYYYMMDD>` and
|
||||
logs the path. Nothing deletes it; the operator removes it when ready.
|
||||
4. A fresh template root takes its place, carrying `etc/machine-id` (the old
|
||||
id, so the journal directory and DHCP identity hold) and the marker
|
||||
`etc/hyperhive-ephemeral-root`.
|
||||
|
||||
The salvage runs again only if the template holds content outside the cache
|
||||
paths, which happens when a hive without this change started the container
|
||||
persistently in between; the new copy gets a numbered directory beside the
|
||||
first. A failed salvage leaves the root where it was and the container starts
|
||||
without the flag (hive-priv logs the error), so a failure costs the switch,
|
||||
never data.
|
||||
|
||||
**Checked before every start.** nixpkgs' start script runs as host root on
|
||||
the root template, and its `mkdir -p`, `chmod`, `touch` and
|
||||
`cp --remove-destination` follow a link the container planted at the paths
|
||||
they touch. The container's limits drop-in
|
||||
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
|
||||
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
|
||||
script, on systemd's own restarts too. It looks at each path with `lstat`,
|
||||
never following a link, and fails the start with one line on stderr naming the
|
||||
path and its type when:
|
||||
|
||||
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
|
||||
exists and isn't a real directory;
|
||||
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
|
||||
(`etc/os-release` may also be a link, which the script skips and a booted
|
||||
NixOS guest creates);
|
||||
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
|
||||
exists and isn't a link that resolves into `/nix/store`, or the `system`
|
||||
file inside the resolved directory isn't a regular file.
|
||||
|
||||
Absent paths pass, so a first start works. The check neither deletes nor
|
||||
repairs: the operator removes the offending entry, and the next start goes
|
||||
through.
|
||||
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
|
||||
points at a build without the sub-command. A drop-in written by an older
|
||||
points at a build without the sub-command. That build's store path has a GC
|
||||
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
|
||||
drop-in and replaced on each write, so a garbage collection can't remove the
|
||||
binary a restart would run. The directory sits beside `gcroots/per-container`,
|
||||
which is the only gcroots path bound into a container, so only root writes it.
|
||||
Destroying the container removes the drop-in and then its root; the build stays
|
||||
until the next collection. A drop-in written by an older hive-priv has no
|
||||
`ExecStartPre=` and no root, and the container starts as before.
|
||||
|
||||
**Hyperhive's platform containers: warn only.** The forge, the matrix server and
|
||||
the swarm services (15 containers, declared by their own modules) run the same
|
||||
check as `hive-priv check-rootfs --warn-only <name>`, from `ExecStartPre=` on
|
||||
their own `container@<name>` unit, through the symlink `/run/hive-rootfs-check`
|
||||
that `hive-priv.nix` points at the current build, so a new hyperhive build does
|
||||
not change the unit or restart the container. It evaluates every rule, logs each
|
||||
violation
|
||||
as a warning, and exits 0, so the container starts either way. The line reads
|
||||
`rootfs check (warn-only): <name>: <message>` with journald's warning priority.
|
||||
Strict refusal needs the live roots of all 15 to pass first, and the warnings
|
||||
show which ones would not. A container the operator runs that hyperhive doesn't
|
||||
declare gets no check at all: the modules name their own containers, and
|
||||
nothing enumerates the host's `containers`.
|
||||
|
||||
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
|
||||
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and
|
||||
copies the data of a file the filesystem refuses to clone. Before it creates
|
||||
anything, hive-priv sizes the root and compares it with the free space under
|
||||
`work/`. A salvage whose files clone (tested with one file of the root) must
|
||||
leave 2 GiB free. One that has to copy data must leave 15 % of the filesystem
|
||||
free. When the free space is short, hive-priv skips the salvage, deletes
|
||||
nothing, logs `salvage skipped: <container> need <size> free <size>`, and the
|
||||
container starts persistent; the next start tries again. hive-priv checks a file
|
||||
that fails to clone against the 2 GiB reserve again before it copies the data. A
|
||||
salvage that fails partway removes the directory it created, and leaves the root
|
||||
and any earlier salvage as they were. After a salvage, hive-priv logs `salvage: <container> <size> reflinked=<n>/<m> files`.
|
||||
|
||||
**Concurrency.** hive-priv holds an `flock` on
|
||||
`/run/hive-priv/salvage-<container>.lock` (the unit's `RuntimeDirectory`, which
|
||||
no container can write) for the whole `WriteNspawnFlags`, so a second request for
|
||||
the same container waits and then finds the finished template. Across
|
||||
containers, one salvage copies at a time, so the free space each one checks
|
||||
excludes any copy still in flight. The request runs on the blocking
|
||||
thread pool, so a long copy holds no async worker. hive-c0re's per-agent lease
|
||||
already serialises the requests it sends; the lock covers hive-priv's own
|
||||
guarantee.
|
||||
|
||||
The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`)
|
||||
reads the guest's own `/var/log/journal` and pushes it over OTLP, so no
|
||||
journal link to the host takes part; nspawn skips `--link-journal=try-guest`
|
||||
in ephemeral mode without failing. The journal vanishes at stop, and with it
|
||||
the lines written after the last push.
|
||||
|
||||
### User namespace (opt-in, per agent)
|
||||
|
||||
An agent whose config sets `services.hyperhive.agent.user.privateUsers = true`
|
||||
runs in its own user namespace. Every other agent keeps the host's uids.
|
||||
|
||||
**Approving the config PR that sets it authorises a host-level change:** the
|
||||
first start afterwards has hive-priv, as root, chown the agent's `state`,
|
||||
`harness`, `work` and `claude` dirs once to the host's data owner. It deletes nothing.
|
||||
|
||||
**How the hive learns the flag.** It never evaluates the agent's nix. The swarm
|
||||
builder reads `config.services.hyperhive.agent.user.privateUsers` from the
|
||||
config it evaluates in its sandbox and puts it in the agent's closure record
|
||||
(`private_users`). hive-c0re reads the record when it writes the container's
|
||||
flags and sends the value to hive-priv with each start. An agent whose module
|
||||
doesn't declare the option reads false. When the builder can't read the
|
||||
option (its eval fails or times out), the record carries no flag, and so does a
|
||||
record or builder from before the field. A record without a flag, a record
|
||||
hive-c0re can't read, or none, leaves the agent as it is.
|
||||
|
||||
- **On disk the data belongs to one host uid:gid**,
|
||||
`services.hyperhive.deploy.hive-controller.userns.dataOwner`
|
||||
(default `60000:60000`). Inside the container that owner shows as the agent
|
||||
user. hive-priv mounts the agent's `state/`, `harness/`, `work/` and
|
||||
`claude/` dirs and its socket dir with `owneridmap`, and every other bind too
|
||||
(config, `/knowledge`, the runtime dir), whose owner shows as root inside.
|
||||
- **`/nix/store` has to be on ext4, btrfs, xfs or erofs.** nspawn mounts the
|
||||
store into a namespaced container idmapped, and the kernel refuses that on
|
||||
overlayfs and on some other filesystems. `WriteNspawnFlags` for an opted-in
|
||||
agent checks the `statfs` type of `/nix/store` against an allowlist of those
|
||||
four; the VM test's erofs store idmaps on the hive's kernel. The check
|
||||
refuses every other filesystem, zfs and tmpfs included, even where a newer
|
||||
kernel could idmap it, because the gate exists to stop a container that would
|
||||
not boot. The error names the type and the fix, and
|
||||
comes before anything moves. Check a host with
|
||||
`findmnt -no FSTYPE /nix/store`.
|
||||
- **The container starts with `--private-users=pick --private-users-ownership=map`.**
|
||||
nspawn idmaps the root at mount time, so a different range for `pick` never
|
||||
walks or chowns it.
|
||||
- **The move happens once.** The first `WriteNspawnFlags` that finds the
|
||||
agent's container stopped chowns those four dirs to `dataOwner` (ownership
|
||||
only; it removes or copies nothing), writes
|
||||
`/var/lib/hive-priv/userns/<name>`, and creates the mount points in the root
|
||||
template owned by the agent user. A later request with the marker present
|
||||
chowns nothing. A running container waits for its next stop. The socket dir
|
||||
is on a tmpfs, so `EnsureAgentSocketDir` hands it to `dataOwner` at every
|
||||
start. The walk works by open directory fd and follows no link, so an agent
|
||||
or manager that can write the dirs can't redirect it to a host path.
|
||||
- **A move that fails starts over.** hive-priv opens all four dirs and
|
||||
checks the agent's ids before the first chown, so a link or file where a dir
|
||||
goes refuses with nothing changed. It then writes the ids to
|
||||
`/var/lib/hive-priv/userns/<name>.pending`. If the walk fails after that (a
|
||||
chown error, a tree nested too deep), the next start repeats it from those
|
||||
ids, then writes the marker and removes the pending file. hive-priv refuses a
|
||||
pending file that doesn't hold `<owner> <uid> <gid>`, and names it.
|
||||
- **The agent's gid must equal its uid.** `owneridmap` maps the gid of a bind
|
||||
with the uid map, so a group other than the uid's has no mapping and writes
|
||||
fail. The agent module asserts that `user.uid` and `user.gid` hold the
|
||||
same number when `privateUsers` is on. Start the agent once with the pin
|
||||
before turning it on: the move refuses an agent whose `state` dir has another
|
||||
gid than uid, naming the agent.
|
||||
- **Turning it off is manual.** hive-priv refuses to start an agent whose
|
||||
marker exists while its record says false, and one whose `dataOwner`
|
||||
changed. Chown the four dirs back, then delete the marker and any
|
||||
`<name>.pending` beside it.
|
||||
- **Keep a `ManageRootAgent` holder off it.** The other agents' dirs it mounts
|
||||
aren't remapped.
|
||||
|
||||
The pause marker needs nothing extra: hive-priv gives it the owner of the
|
||||
`harness/` dir. hive-c0re still creates agent dirs and new subvolumes as
|
||||
`hive-core`, and only the first namespaced start chowns them. A dir created
|
||||
after the move, such as a recreated subvolume, stays `hive-core`'s and the
|
||||
agent can't write to it.
|
||||
|
||||
### Cross-agent access to state
|
||||
|
||||
An agent holding the `manage_root_agent` permission gets every other
|
||||
agent's `state` dir bind-mounted **read-write** (`bind_child_agent_dirs` in
|
||||
`lifecycle/host_config.rs`). The RW is deliberate, not an oversight: the
|
||||
holder recovers other agents, which includes writing into their state (for
|
||||
example seeding notes, clearing a stuck sentinel) as well as reading it.
|
||||
|
||||
This is the **only** cross-agent mount. An agent without that permission
|
||||
sees its own dirs and nothing else.
|
||||
|
||||
**`harness` isn't mounted at all.** It holds that agent's own runtime
|
||||
material - `bash-tasks/`, the turn-stats and event sqlite dbs - and
|
||||
nothing argues for anyone else reading it, let alone writing it.
|
||||
hive-c0re reads a harness dir **directly on the host** when it wants
|
||||
those stats, which needs no mount into another container.
|
||||
|
||||
**`config` isn't mounted either.** A config change is a PR on that agent's
|
||||
config repo, made from a clone and merged after review. Anyone who wants to
|
||||
read the config clones `agent-configs/<name>` from the forge. A mount, even a
|
||||
read-only one, would show an agent a tree that's not the one that governs it
|
||||
whenever the forge has moved on.
|
||||
|
||||
Isolation still holds by default: a container has its *own* dirs and,
|
||||
unless it holds the permission, nothing else.
|
||||
|
||||
Under `/var/lib/hyperhive/applied/` — the hive-c0re-only deploy markers
|
||||
`.<name>.closure`: the store path of the closure the agent was last swapped to
|
||||
or created from, or, where the file was missing, the closure the container's
|
||||
profile pointed at when the hive first read it. No config lives here; the
|
||||
forge's `agent-configs/<name>` is the only copy. A hive that predates this
|
||||
finds per-agent `applied/<name>` repos here. Nothing reads them any more, and
|
||||
you can delete them.
|
||||
|
||||
Under `/var/lib/hyperhive/meta/` — system-level config files, plain state
|
||||
with no git history. The flake agents build from is the
|
||||
swarm's `internal/meta`, not here. A hive that predates this finds a
|
||||
leftover `.git` here; `migrate::run` deletes it at startup. Contents:
|
||||
|
||||
- `topology.json` — the agent roster (`["alice", "bob", "ruth"]`).
|
||||
Written by `topology::reconcile` from `agent_set::refresh`; read by
|
||||
`topology::all_agents`, which is the set the `manage_root_agent`
|
||||
permission grants mounts over. The reader also accepts a map-shaped
|
||||
file carrying a `parent` per agent, and keeps its keys.
|
||||
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
||||
hive's agents
|
||||
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
||||
Written only by `workers::swarm_limits`, which copies each agent's key of
|
||||
the swarm's `agent-limits` bucket here every 30 seconds. Read where
|
||||
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
|
||||
into the container — these are host-side caps on the container, so
|
||||
the capped party never sets them. It sees only the resulting memory cap,
|
||||
as a byte count or `infinity` in its runtime dir (below). Fallback is per
|
||||
*field*: an absent file, absent agent, or absent field falls back to the
|
||||
hive-wide `services.hyperhive.deploy.hive-controller.agentCpuQuota` / `agentMemoryMax`,
|
||||
so an agent can have only its memory capped and still track the hive
|
||||
default for CPU. Every value passes the validators first, and one bad
|
||||
value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
|
||||
shares in the same drop-in have **no** per-agent setting — they're
|
||||
hive-wide only and come straight off `HiveEnv`, so this file has no field
|
||||
for them.
|
||||
- `resource-limits.json` — retired. Per-agent overrides set on the hive used
|
||||
to live here; nothing reads or writes it now. An old copy in this directory
|
||||
is harmless and you can delete it.
|
||||
|
||||
An agent holding `manage_root_agent`, normally ruth, has the meta dir
|
||||
RO-mounted at `/meta/`.
|
||||
|
||||
Outside this directory, `/var/lib/hyperhive/swarm-agent-permissions.json`
|
||||
holds the permissions the swarm grants this hive's agents, as
|
||||
`workers::swarm_permissions` last read them from the swarm's
|
||||
`agent-permissions` bucket (every 30 seconds): a copy, not the
|
||||
record, which `swarmctl agent permissions` sets. `set_nspawn_flags`
|
||||
reads it at every container start for `manage_root_agent`.
|
||||
|
||||
`/var/lib/hyperhive/swarm-agent-archetypes.json` holds the archetype each of
|
||||
this hive's archetype agents runs, as last read from the swarm's
|
||||
`agent-archetype` bucket at the agent's provision and each rebuild. An agent
|
||||
with an entry has no config repo and no `agent-<n>` meta input; see
|
||||
[archetypes](../swarm/archetypes.md).
|
||||
|
||||
hive-c0re creates containers from the closure the swarm builds from `internal/meta` (`nixosConfigurations.<n>`).
|
||||
|
||||
## Destroy vs purge
|
||||
|
||||
See [For operators](#for-operators) above for what each action does to
|
||||
an agent's state. The mechanics, for completeness:
|
||||
|
||||
- `DESTR0Y` also drops the systemd drop-in; reviving the agent is
|
||||
`swarmctl agent create` with the same name and hive, which reuses the
|
||||
kept state.
|
||||
- `PURG3` wipes `/var/lib/hyperhive/agents/<name>/` — the
|
||||
union of everything `DESTR0Y` left behind.
|
||||
|
||||
`actions::destroy` has no guard for the root/bootstrap agent: it destroys ruth
|
||||
like any other agent, and she stays gone until you create her again from the
|
||||
swarm dashboard.
|
||||
|
||||
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
|
||||
|
||||
On a btrfs host, `lifecycle::ensure_agent_state_subvolume` creates a brand-new agent's state root as a
|
||||
**btrfs subvolume** instead of a plain directory (progressive
|
||||
enhancement). This is a no-op fallback on
|
||||
non-btrfs hosts and for any agent whose root already exists, so
|
||||
hive-c0re automigrates nothing: existing agents keep their plain dirs
|
||||
until an explicit opt-in upgrade.
|
||||
|
||||
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
|
||||
hive-c0re creates the per-agent subdirs (spawn / rebuild).
|
||||
It skips the work when the root already exists; otherwise it asks
|
||||
hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
|
||||
root when the FS is btrfs (`statfs` magic gate) and chown it to the
|
||||
`hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
|
||||
succeed inside it.
|
||||
- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
|
||||
reuses it untouched.
|
||||
- **PURG3 deletes it correctly:** `rmdir`/`remove_dir_all` can't remove
|
||||
a subvolume root, so purge first calls hive-priv
|
||||
(`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
|
||||
actually a subvolume, then the normal `remove_dir_all` sweep covers
|
||||
plain-dir agents.
|
||||
|
||||
Per-subvolume disk-usage accounting and optional quotas have since
|
||||
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
|
||||
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
|
||||
`hivectl agent <name> quota show|set` reads/limits one agent's
|
||||
subvolume usage through the same hive-priv-mediated path as
|
||||
subvolume creation/deletion above.
|
||||
|
||||
The old snapshot store no longer exists; swarm-archive will replace it for agent-state backups, and nothing copies the subvolume off the host in the meantime.
|
||||
|
||||
The subvolume is the whole agent state root, so `work/` is part of every
|
||||
snapshot and migration of it and can make them much larger.
|
||||
|
||||
## `/var/lib/swarm-controller/` (swarm-controller host only)
|
||||
|
||||
Only present on the one host running
|
||||
`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
|
||||
so it survives restarts and redeploys.
|
||||
|
||||
- `webhook-secret` — the HMAC key Forgejo signs the swarm's forge webhooks
|
||||
with, as it was before the swarm secret store held it. **Keep it.** The
|
||||
controller stores its value in the swarm secret store at
|
||||
`swarm/controller/swarm-controller/forge/webhook-secret` the first time it
|
||||
starts with that path empty, and reads the store from then on. The file is the
|
||||
fallback while the store is unreachable or its policy doesn't grant the path
|
||||
yet, and nothing else reads or removes it. A host with neither the file nor a
|
||||
store value gets a new secret in the store, with nothing to configure.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
If the store can't answer and the file is unreadable or missing at startup, the daemon still starts and logs
|
||||
`webhook secret unavailable`; the webhook endpoint then answers 503
|
||||
rather than accepting deliveries it can't verify. Everything else the
|
||||
controller serves is unaffected. The daemon keeps reading the store in the
|
||||
background, at most five minutes apart, and the endpoint and the forge hook
|
||||
registration start working when it answers. It writes a new secret only when
|
||||
the store answers that it holds none, never because a read failed.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
- `operator-matrix-admins.json` — the matrix user ids the operator admin
|
||||
sync has made homeserver admin, a JSON array. The sync revokes admin
|
||||
only from these, so deleting the file leaves every current admin in
|
||||
place; a missing file is an empty set. A file the controller can't
|
||||
read or parse stops the sync, grants included, and the controller logs
|
||||
`operator matrix admin: pass failed` on every pass. See
|
||||
[`docs/2-handbook/swarm/matrix.md`](../swarm/matrix.md#operators-are-homeserver-admins).
|
||||
|
||||
## Run-time dirs
|
||||
|
||||
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
||||
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
|
||||
Without that, every restart wipes bind sources and existing
|
||||
containers can't start.
|
||||
|
||||
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
|
||||
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
|
||||
(bind-mounted into the container as `/run/hive/mcp.sock`).
|
||||
- `/run/hyperhive/agents/<name>/memory-max-bytes` — the agent's effective
|
||||
`MemoryMax=` as a byte count (a percentage resolved against the host's
|
||||
physical memory) or `infinity`, written by `lifecycle::write_dropins`
|
||||
before every start. Absent only when hive-c0re can't tell the cap. The
|
||||
container reads it at boot as `/run/hive/memory-max-bytes`
|
||||
(`nix/agent-modules/memory-cap.nix`).
|
||||
|
||||
On startup, `Coordinator::register_agent` drops any prior socket
|
||||
task before rebinding — idempotent so a hive-c0re restart followed
|
||||
by `rebuild alice` recreates the agent's socket without a clean
|
||||
reinstall.
|
||||
|
||||
## First-boot agent-user migration
|
||||
|
||||
The harness runs as a per-agent unix user inside the container
|
||||
(`services.hyperhive.agent.user.name`, defaults to the agent's logical label so each
|
||||
container has a uniquely named user). The activation script below
|
||||
moves root-owned state to the agent user so the claude session survives.
|
||||
|
||||
`system.activationScripts.hive-agent-user-migrate` (in
|
||||
`nix/agent-modules/user.nix`) runs on every activation,
|
||||
marker-guarded so the substantive moves only happen once per
|
||||
container lifetime:
|
||||
|
||||
1. **`${homeDir}` exists with the right ownership** — covers the
|
||||
the first boot before `useradd`'s `createHome` has had a
|
||||
chance to chown. Also re-applies on every rebuild in case the
|
||||
meta-flake's per-agent name evolves (rare).
|
||||
2. **Migrate any leftover `/root/.claude` content into
|
||||
`${homeDir}/.claude`.** Marker
|
||||
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
|
||||
only written once there's nothing left to migrate, so a `cp`
|
||||
failure leaves it absent and the next boot retries. The root is
|
||||
ephemeral, so the marker vanishes at every stop and the step
|
||||
re-runs; with no `/root/.claude` it only writes the marker again.
|
||||
`cp -an` (no-clobber) so any pre-existing files at the new
|
||||
location win — never blow over data already there, and so a
|
||||
retry after a partial copy is as safe as the first attempt.
|
||||
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
|
||||
recursively so the agent user can read/write it. Wildcard
|
||||
matches the single agent that container sees; `-h` skips
|
||||
symlinks the agent might have planted.
|
||||
4. **Chown the `~/.claude/` bind-mount** recursively. The harness reads
|
||||
`~/.claude/` as the agent user to decide Online vs NeedsLogin in
|
||||
`login::has_session`. Without the chown, credentials owned by
|
||||
another uid get silently treated as "no session" and the operator
|
||||
re-prompts every boot.
|
||||
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
|
||||
user**, `0751`, not recursive. hive-priv creates it `0751 root`
|
||||
on the host before every start; the harness binds its sockets
|
||||
there as the agent user. Why the mode matters:
|
||||
[`boundary-mechanism.md`](../../4-implementation/trust-boundary/boundary-mechanism.md#the-per-agent-socket-dir).
|
||||
A namespaced agent has the dir bound at `/agents/<name>/sockets` and
|
||||
linked from `/run/hive-agent/<name>`; the activation chowns nothing, because
|
||||
a mount point under `/run` would belong to root and the id mapping would
|
||||
give the agent user no ids there. A later in-container activation sees the
|
||||
link and logs `is a symlink or not a directory; not handing it`. The log
|
||||
is harmless.
|
||||
|
||||
## Matrix per-agent daemon + token-arrival trigger
|
||||
|
||||
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
|
||||
process per agent. Serves its MCP tools directly over
|
||||
streamable-http (`services.hyperhive.agent.mcp.matrixHttpPort`, no stdio bridge —
|
||||
same shape as `hive-bash-daemon`), emits hyperhive wake signals
|
||||
on incoming room events via `/run/hive/mcp.sock`. Conditional on the
|
||||
agent having at least one `services.hyperhive.agent.matrixAccounts`
|
||||
entry — there is no separate enable switch, and the same condition gates
|
||||
the daemon, its path watcher AND the autoinjected
|
||||
`extraMcpServers.matrix` entry. The module declares the hive-internal
|
||||
`main` entry whenever `services.hyperhive.agent.matrix.url` is non-null,
|
||||
so on a real hive (where hive-c0re renders that URL per agent) every
|
||||
agent has one; a `null` URL with no operator-declared account is the
|
||||
"this agent has no matrix" state.
|
||||
|
||||
**First-boot ordering**: a token can arrive after the container comes
|
||||
up — a declared token file, or a token the swarm mints into the store.
|
||||
Without the path-trigger sibling
|
||||
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
|
||||
<this agent's state dir>/matrix-token*` — the trailing `*` also catches
|
||||
a declared secondary token file like `matrix-token-ccc`), the daemon
|
||||
would exit 0 quietly the first time it ran and the MCP would have no
|
||||
daemon until the next restart. The `.path` unit makes the appearance
|
||||
of the token re-fire the service so the daemon comes alive in the
|
||||
same boot cycle as
|
||||
provisioning. A token in the store changes no file, so a store-backed agent
|
||||
also gets a timer that restarts the daemon five minutes after it last
|
||||
exited. A running daemon re-lists its linked accounts in the store
|
||||
every two minutes and, when the set changed, exits with status
|
||||
75, which the unit's `RestartForceExitStatus` restarts. On the same
|
||||
two-minute tick it tries again each secondary account that didn't come
|
||||
up — a declared account with no token yet, a token the homeserver
|
||||
rejected, or a homeserver that
|
||||
was down — waiting two minutes after its first failure and twice as long
|
||||
after each further one, up to sixteen minutes. It tries one account at a
|
||||
time and counts a try that hasn't finished after two minutes as a failure,
|
||||
so one hung account doesn't hold the others back. One that comes up joins
|
||||
the running daemon; the primary and the other accounts aren't
|
||||
restarted. The same token watcher also drives avatar setting: on a
|
||||
restart the daemon re-runs each account's bring-up, which sets the
|
||||
avatar (see below).
|
||||
|
||||
### matrix avatar (set by the daemon over the live Client)
|
||||
|
||||
`hive-matrix-daemon` itself publishes the agent icon (`services.hyperhive.agent.icon`, an SVG) as each matrix
|
||||
account's profile avatar
|
||||
(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
|
||||
the daemon builds + restores an account's `Client` (authenticated,
|
||||
pointed at that account's resolved homeserver), it calls matrix-sdk's
|
||||
`account().upload_avatar()` — one call that uploads the media and sets
|
||||
`avatar_url`. Because it reuses the live Client, there is no hardcoded
|
||||
homeserver URL, no token re-read, and no token-file globbing: the daemon
|
||||
already iterates every configured + dashboard-discovered account in its
|
||||
bring-up loop, so it sets the avatar for **every** account.
|
||||
|
||||
Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
|
||||
librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
|
||||
unit, gated on `services.hyperhive.agent.icon != null`. No icon configured → the env is
|
||||
unset → `sync_avatar` returns early and sets no avatar.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
Idempotency is **per-account**: an `avatar-icon-hash` file in each
|
||||
account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
|
||||
skips the upload when unchanged, because every upload mints a fresh
|
||||
`mxc://` URI that emits a profile state event in every joined room —
|
||||
re-uploading identical bytes is timeline spam. A swarm-UI-linked
|
||||
account gets its avatar when the daemon restarts on a changed account
|
||||
listing (exit 75, which re-runs the per-account bring-up), so no separate
|
||||
avatar trigger is needed. The daemon swallows avatar failures
|
||||
(logged, non-fatal) so they never break account bring-up or sync.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
The tables, state directories, mounts and migrations behind this are in
|
||||
[`persistence.md`](../../4-implementation/agent-state/persistence.md).
|
||||
|
||||
## Config changes
|
||||
|
||||
A config change is a pull request on the agent's config repo,
|
||||
`agent-configs/<name>`. No tool exists for it: opening the PR is the request.
|
||||
|
||||
1. The agent clones its config repo, edits it (`agent.nix` is the entry point),
|
||||
pushes a branch and opens the PR with `hive-forge`. Branch protection lets it
|
||||
push branches but not `main`, so it **can't merge its own config PR**. An agent
|
||||
can propose a change to another agent's config the same way.
|
||||
2. You review the PR on the forge, with its diff, comments and CI status, and
|
||||
merge it there. Merging needs membership of the `operators` team in
|
||||
`agent-configs`. The team starts empty: add each operator by hand in the forge
|
||||
UI.
|
||||
3. The merge deploys. The swarm controller finds the hive that runs the agent and
|
||||
tells it to rebuild.
|
||||
|
||||
A merge runs **no evaluation check**. A merged config that doesn't evaluate or
|
||||
build fails the rebuild, and the agent's rebuilds keep failing until a fix
|
||||
merges. A failed rebuild shows on the hive like any other. A merge whose webhook
|
||||
never reaches the controller deploys nothing.
|
||||
|
||||
You create a brand-new agent at swarm level (`swarmctl agent create`, or the
|
||||
swarm UI), not from a hive. The controller seeds its config repo with a default
|
||||
`agent.nix`, and every later change is a PR like any other. An agent created
|
||||
from an [archetype](../swarm/archetypes.md) gets no config repo: the swarm builds
|
||||
it from the archetype's module.
|
||||
|
||||
The full path, the meta flake, deploy markers, startup migrations, the root
|
||||
container, the `system` messages hive-c0re sends and autoupdate:
|
||||
[`config-changes.md`](../../4-implementation/agent-state/config-changes.md).
|
||||
|
|
|
|||
|
|
@ -131,7 +131,7 @@ checkpoints**, not about sandboxing the agent from its own tools:
|
|||
technical control.
|
||||
- **Operator sign-off** — config changes are config PRs an operator merges
|
||||
on the forge (see
|
||||
[`config-changes.md`](config-changes.md)), and
|
||||
[`persistence.md`](persistence.md#config-changes)), and
|
||||
scheduled prompts are operator-only, set at swarm level
|
||||
(see [`schedules.md`](../swarm/schedules.md)).
|
||||
Before that merge, the swarm builder evaluates an unmerged config PR
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ Select the name to open its terminal and watch it work.
|
|||
**Review a config change.** Agent config changes (new packages, env vars, MCP
|
||||
servers) are pull requests on the agent's `agent-configs` repo. Review and
|
||||
merge them on the forge; the merge deploys the change. See
|
||||
[`config-changes.md`](../agents/config-changes.md#config-changes).
|
||||
[`persistence.md`](../agents/persistence.md#config-changes).
|
||||
|
||||
**Start, stop, restart or rebuild an agent.** Select one or more agents on
|
||||
4G3NTS (select the icon) and use the selection bar, or use the per-agent `⋮`
|
||||
|
|
|
|||
|
|
@ -53,7 +53,7 @@ Two things live in the `agent-configs` Forgejo organization:
|
|||
branch-protected: only the `operators` team can merge, and the agent can
|
||||
neither push `main` nor self-merge. An operator's merge in the Forgejo UI
|
||||
deploys the merged commit (see
|
||||
[config-changes.md § Config changes](../agents/config-changes.md#config-changes)).
|
||||
[persistence.md § Config changes](../agents/persistence.md#config-changes)).
|
||||
Repos stay private, so an agent can't read another agent's config.
|
||||
|
||||
The `hive-forge` CLI wraps the Forgejo REST API with the agent's own token;
|
||||
|
|
|
|||
370
docs/4-implementation/agent-state/config-changes.md
Normal file
370
docs/4-implementation/agent-state/config-changes.md
Normal file
|
|
@ -0,0 +1,370 @@
|
|||
# Config changes: deploy path, meta flake and system messages
|
||||
|
||||
How a merged config PR reaches an agent, the meta flake and deploy markers
|
||||
behind it, the startup migrations, the root container, the `system` messages
|
||||
hive-c0re sends, and autoupdate. The operator's flow is in
|
||||
[persistence and config changes](config-changes.md#config-changes).
|
||||
|
||||
## Config changes
|
||||
|
||||
Config changes flow through a **forge pull request** on the agent's
|
||||
`agent-configs/<name>` repo — the same surface agents use for code PRs.
|
||||
No MCP tool exists for config changes: opening the PR IS the request.
|
||||
|
||||
1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
|
||||
tracked path, but `agent.nix` is the contract entry point), commits
|
||||
with its own git identity, and pushes a branch + opens a PR with
|
||||
`hive-forge` — the same way it would change any other repo.
|
||||
The container has no config mount: the clone is how an agent reads its
|
||||
own config as well as how it changes it.
|
||||
Branch protection (the agent isn't on the `main` push allowlist;
|
||||
merge and approvals allowlist = `operators` team; see "Forge config repos"
|
||||
below) makes the agent a write collaborator that **can't merge its own
|
||||
config PR**.
|
||||
2. An operator reviews the PR **on the forge** (native diff, threaded
|
||||
comments, CI status) and merges it there. Merging needs membership of
|
||||
the `operators` team in `agent-configs`. The team starts empty: add
|
||||
each operator by hand in the forge UI.
|
||||
3. swarm-controller gets the `agent-configs` org's `pull_request`
|
||||
delivery. A `closed` event with `merged: true` and base branch `main`
|
||||
names the commit in `merge_commit_sha`.
|
||||
4. The controller looks up which hive's wanted state places the agent
|
||||
and publishes a deploy request carrying that commit on that hive's
|
||||
deploy subject. With no such hive, or more than one, it deploys
|
||||
nothing and logs a warning naming them.
|
||||
5. If the closure the swarm declares for the agent is the one the hive last
|
||||
applied (the `applied/.<n>.closure` marker), the hive does nothing.
|
||||
Otherwise it queues a rebuild, which downloads the declared closure. The
|
||||
request's commit plays no part: the hive keeps no copy of the config and
|
||||
reads no forge head. An agent with no container on that hive yet gets its
|
||||
first deploy from the declared closure.
|
||||
|
||||
This path runs **no eval-verify**. A merged config that doesn't
|
||||
evaluate or build fails the rebuild, so the agent's rebuilds keep failing
|
||||
until a fix merges. A failed rebuild shows on the hive like any other. A merge
|
||||
whose webhook delivery never reaches the controller deploys nothing.
|
||||
|
||||
Creating a brand-new agent is swarm-level (`swarmctl agent create`,
|
||||
`POST /api/agents` on the swarm controller), and no hive can originate
|
||||
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
|
||||
agent's config repo with a default `agent.nix` template, then asks the
|
||||
target hive to deploy it.
|
||||
An agent created from an [archetype](../../2-handbook/swarm/archetypes.md) gets no
|
||||
config repo: the swarm builds it from the archetype's module.
|
||||
|
||||
Changing what the template seeded isn't an exception: like every
|
||||
later change, it's a PR on that config repo, made from a clone and
|
||||
merged by an operator on the forge (see [Config changes](#config-changes)).
|
||||
|
||||
## Agent lifecycle
|
||||
|
||||
### Reminder delivery: file-path semantics
|
||||
|
||||
A reminder may carry a `file_path` (the agent-visible path inside its
|
||||
container, for example `/agents/<name>/state/foo.md`). The agent's own
|
||||
harness (`hive-agent`, inside the container) delivers the reminder when it
|
||||
falls due:
|
||||
|
||||
1. **Validates** the path: it must resolve under the agent's own state
|
||||
directory, with no `..` components and a non-empty relative tail. On
|
||||
rejection, or on a write failure, the harness delivers the original
|
||||
message inline with a warning note — the reminder still fires.
|
||||
2. **Writes the body to disk** at that path and delivers a short pointer
|
||||
message in its place, keeping the agent's inbox / wake-prompt small
|
||||
while the agent reads the bulky payload out of band.
|
||||
|
||||
The write happens inside the agent's container, so hive-c0re takes no
|
||||
part in it and there is no symlink-escape defence: the harness is the
|
||||
agent, and the container boundary is the sandbox.
|
||||
|
||||
### Destroy semantics
|
||||
|
||||
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
|
||||
Stops + removes the nspawn container and drops the systemd drop-in.
|
||||
Persistent state (claude credentials, `/state/` notes) is
|
||||
**kept by default** — recreating the agent with the same name
|
||||
reuses prior notes + login. With `purge = true` the agent's
|
||||
`/var/lib/hyperhive/agents/<name>/` tree is also
|
||||
wiped (creds + notes gone forever). The
|
||||
root/bootstrap container is destroyable like any other, and stays gone
|
||||
until you create ruth again from the swarm dashboard.
|
||||
|
||||
## Meta flake
|
||||
|
||||
A hive has no meta flake of its own. It builds every agent from the swarm's
|
||||
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
|
||||
It declares one flake input per agent (`agent-<n>.url =
|
||||
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
|
||||
agent's forge config repo, the one config PRs merge into) and one
|
||||
`nixosConfigurations.<n>` output per agent. Each output wraps
|
||||
`inputs.agent-<n>.nixosModules.default` with the identity +
|
||||
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
|
||||
(the dashboard port, the pinned claude-code, extra OTEL labels, the queue
|
||||
address) aren't in it:
|
||||
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
|
||||
start (see `docs/5-project/process/conventions.md`).
|
||||
A config that sits only in a local checkout, not on forge `main`, never
|
||||
deploys.
|
||||
|
||||
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
|
||||
state, bound read-only into the root agent's container as `/meta`:
|
||||
`topology.json` and the swarm resource limits. The permissions the swarm
|
||||
grants live outside it, in `/var/lib/hyperhive/swarm-agent-permissions.json`.
|
||||
`agent_set::refresh(agents:
|
||||
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
|
||||
nginx include in line with the live agents. `provision_container`, `destroy`,
|
||||
`rebuild` and the startup migration call it. It refuses an empty agent list,
|
||||
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
|
||||
is unset, since every agent would come up with no forge.
|
||||
|
||||
## The swarm's meta flake
|
||||
|
||||
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
|
||||
every agent any hive places (the pins:
|
||||
[`docs/4-implementation/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). The swarm
|
||||
builder builds every agent from it, and a hive deploys that build. While the
|
||||
swarm has recorded no build of an agent, that agent's deploy fails and names the
|
||||
agent. The controller queues
|
||||
its update of the repo before the deploy request for a merged config commit
|
||||
and for a new agent, and sends the deploy only when that update succeeded and
|
||||
the swarm builder built the agent from the updated repo (the
|
||||
[deploy gate](../swarm/builder.md#deploy-gate)). A failed build, a closure the builder didn't push or a failed copy of the hive's cache facts skips the deploy, and the timer declares them again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
|
||||
|
||||
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
|
||||
agent's record in `agent-closures` declares, whichever `internal/meta` commit
|
||||
the builder built it from, and reads no forge head. The rebuild's `MetaSync`
|
||||
node prepares it (the agent's archetype, its runtime directories, the agent
|
||||
set), reads the record once and pins its store path. `Prebuild` and `Swap` use
|
||||
the pin, so a record that changes mid-rebuild can't make `Swap` pull another
|
||||
closure than `Prebuild` fetched, and the marker names the closure it applied. The
|
||||
next autoupdate pass picks up the newer record. A first deploy has no
|
||||
`MetaSync`; its `Create` reads the record once.
|
||||
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
|
||||
the agent's record in the swarm's `agent-closures` bucket (the controller
|
||||
writes it from the builder's reply) and pull the closure it names from the
|
||||
swarm cache through the host's nix daemon. `Swap` and `Create` then give
|
||||
`hive-priv` the store path (`toplevel` on `UpdateContainer` and
|
||||
`CreateContainer`), which accepts only a path in the store that carries the
|
||||
swarm cache's signature. On the wire the
|
||||
`hive_priv_sock::PrivRequest` variants carry a required field:
|
||||
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
|
||||
A rebuild with no record, or none the builder pushed, fails with an error
|
||||
that names the agent; see
|
||||
[`docs/4-implementation/swarm/builder.md`](../swarm/builder.md#rebuilds-download-the-closure).
|
||||
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
||||
runs the hyperhive of the controller's deploy, not its hive's.
|
||||
- **Marker.** After a successful swap or create `hive-c0re` writes the closure's
|
||||
store path to `applied/.<n>.closure`. The autoupdate pass
|
||||
([below](#autoupdate-on-startup)) compares it with the record.
|
||||
|
||||
## Deploy markers per agent
|
||||
|
||||
```
|
||||
/var/lib/hyperhive/applied/ deploy markers - core-only
|
||||
└── .<name>.closure # store path of the last closure applied
|
||||
|
||||
/var/lib/hyperhive/meta/ plain hive state - core
|
||||
├── topology.json # the live agent set
|
||||
└── swarm-resource-limits.json # the swarm's per-agent caps
|
||||
```
|
||||
|
||||
hive-c0re keeps no copy of an agent's config: the forge's `agent-configs/<name>`
|
||||
`main` is the only one, and a config change is a PR against it. The marker
|
||||
records the closure last applied, and nothing else. Hives that ran before the
|
||||
marker existed have `.<name>.hyperhive-rev`, `.<name>.config-rev` and
|
||||
`meta-rev/<name>` files instead. Nothing reads them any more, and you can
|
||||
delete them.
|
||||
|
||||
Older hives also kept an _applied_ repo at `applied/<name>` and a _proposed_
|
||||
clone at `agents/<name>/config`. Nothing reads either any more, and you can
|
||||
delete them.
|
||||
|
||||
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
|
||||
The agent's `agent-configs/<n>/flake.nix`
|
||||
is a fixed boilerplate that exports `nixosModules.default =
|
||||
import ./agent.nix`; the meta flake imports that module and
|
||||
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
|
||||
|
||||
### Forge config repos
|
||||
|
||||
The bundled `hive-forge` container runs on the swarm's forge host
|
||||
(`deploy.forgejo.enable`, see [`2-handbook/swarm/services.md`](../../2-handbook/swarm/services.md)).
|
||||
Every agent created through the swarm has a repo in the private
|
||||
`agent-configs` Forgejo org, created and seeded by swarm-controller.
|
||||
hive-c0re pushes nothing to it, keeps no clone and reads nothing from it: forge
|
||||
`main` advances only by an operator's merge.
|
||||
|
||||
Each agent is a **write collaborator on its own** `agent-configs/<name>`
|
||||
repo — so it can push a branch and open a config PR — but not a member
|
||||
of any other agent's, so it can't reach another agent's config through
|
||||
the forge. Branch protection keeps the agent off the `main` push
|
||||
allowlist and allowlists merging to the `operators` team only, so an
|
||||
agent can't fast-forward its own config or self-merge its PR (see
|
||||
[Config changes](#config-changes) above). hive-c0re passes the credential to
|
||||
each `git` call inline, never writing it into a repo's config, so
|
||||
no stored token sits in a repo an agent might one day read.
|
||||
|
||||
The dashboard deep-links into this org — a `config repo` link
|
||||
per container row. See `docs/2-handbook/hives/dashboard.md`.
|
||||
|
||||
### Submitting agent's view of config repos
|
||||
|
||||
No agent has a config repo mounted, not even one holding
|
||||
`ManageRootAgent`: reading another agent's config means cloning its repo
|
||||
from the forge, subject to the forge's own access rules.
|
||||
|
||||
An agent submits a change to another agent's config the same way it
|
||||
makes any other change: **clone the target agent's config repo from the
|
||||
forge into its own state dir, commit on a branch, open a PR**, and let
|
||||
the operator review and merge it. By design, no second, mount-shaped
|
||||
path reaches the same file without the review.
|
||||
|
||||
Agents holding the `manage_root_agent` permission (granted per agent in
|
||||
the swarm's `agent-permissions` record; see
|
||||
`docs/5-project/process/conventions.md#permissions`) get additional host-side bind
|
||||
mounts via `set_nspawn_flags`:
|
||||
|
||||
- `/var/lib/hyperhive/agents/<n>/state` → `/agents/<n>/state` (RW) for
|
||||
**every** agent. The permission means "may manage any agent," so the mounted
|
||||
set is every agent.
|
||||
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the hive's plain state
|
||||
(topology, swarm limits).
|
||||
|
||||
An agent without the permission gets only its own `state`, `harness` and
|
||||
`work` dirs: no other agent's dirs and no `/meta`.
|
||||
|
||||
⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts
|
||||
an agent, after its current turn, when its `manage_root_agent` changes.
|
||||
|
||||
No agent holds the permission by default: an agent whose record leaves
|
||||
`permissions` unset gets the default, which doesn't include
|
||||
`manage_root_agent`, whatever the agent's name. The swarm UI's setup
|
||||
assistant creates ruth with it. For a ruth that predates that, whose
|
||||
permissions were unset, swarm-controller writes the default plus
|
||||
`manage_root_agent` into her key once at start, so she keeps her mounts;
|
||||
it never touches a key that's already set.
|
||||
|
||||
## Startup migrations (older hosts)
|
||||
|
||||
hive-c0re runs a couple of idempotent migrations on every startup so a
|
||||
host set up before the current deploy scheme (described above)
|
||||
converges to it automatically. Each phase is a no-op
|
||||
once already applied:
|
||||
|
||||
- **Agent files**: refreshes the files that follow the agent set
|
||||
(`agent_set::refresh`) from the current agent list. Set
|
||||
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
||||
- **Legacy meta repo**: deletes the leftover `.git` in
|
||||
`/var/lib/hyperhive/meta/` and the
|
||||
unread `/var/lib/hyperhive/forge-core-token`. The files beside the `.git`
|
||||
stay. The forge-side `core/meta` repo and `core` account aren't touched.
|
||||
|
||||
No state loss in either migration: claude creds, `/state/` notes, the
|
||||
events DB all survive. The root
|
||||
agent keeps its session; sub-agents stay logged in.
|
||||
|
||||
## The root/bootstrap container
|
||||
|
||||
The root agent container runs through the **same lifecycle as
|
||||
sub-agents**. Nothing creates it at hive boot: you create ruth from the
|
||||
swarm dashboard like any agent, and an existing ruth container starts with
|
||||
the hive like every other. The root agent can clone
|
||||
`agent-configs/ruth` to read its own `agent.nix` and open a config PR there
|
||||
for operator review, same as any other agent.
|
||||
|
||||
Differences from sub-agents:
|
||||
|
||||
- Reference docs: her `agent.nix` turns on `services.hyperhive.agent.docs.enable`.
|
||||
Her container builds from `agent-base` like every agent, and the seeded
|
||||
`agent.nix` is the same for every name, so a newly created ruth has no docs
|
||||
until a config PR adds that line.
|
||||
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
||||
FNV-1a hash as every other agent (8100..8999 range).
|
||||
- The root agent holds `manage_root_agent` (the swarm UI's setup
|
||||
assistant creates her with it), for which `set_nspawn_flags` adds extra
|
||||
binds: every agent's `state` dir (RW) under `/agents/<n>/state`, and the
|
||||
hive's plain state under `/meta` (RO).
|
||||
- Its socket comes from the generic per-agent registration, in the same
|
||||
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
||||
not a separate manager-only path.
|
||||
|
||||
**Migration note** (for older hosts): drop any `containers.root =
|
||||
{ ... }` block from your host NixOS config, and any `services.hyperhive.ruthless`
|
||||
definition, then create ruth from the swarm dashboard.
|
||||
|
||||
## Root-agent policy
|
||||
|
||||
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
||||
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
||||
varies is which MCP tools it surfaces (gated by the agent's permissions,
|
||||
which the swarm sets). No `role:manager` block renders only
|
||||
for the root agent. The root agent's role-specific
|
||||
behaviour comes from its AGENTS.md / agent-specific instructions, not
|
||||
the system prompt template.
|
||||
|
||||
## System messages
|
||||
|
||||
Hive-c0re has no typed event that drives an immediate claude turn. A
|
||||
message from sender `system` is plain inbox text, and the harness logs it
|
||||
(`log_system_event` in `hive-agent/src/main.rs`) without parsing the body.
|
||||
Today hive-c0re sends two: the `/knowledge updated` broadcast
|
||||
(`workers/knowledge.rs`) and the wake message for a graceful stop
|
||||
(`Coordinator::kick_agent`).
|
||||
|
||||
**A previously running container crashing** drives no agent turn.
|
||||
`crash_watch` only `tracing::warn!`s `"container crash
|
||||
detected"`; a swarm-level grafana alert matches that log line directly
|
||||
and pages the operator, with no agent (and no per-hive dashboard
|
||||
banner) in the loop. See
|
||||
[`docs/4-implementation/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re).
|
||||
|
||||
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
||||
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
||||
when convenient" events with no reason to drive an immediate turn, so
|
||||
they go out via `swarm_notices::notify` instead: `subsystem = "core"`,
|
||||
`key = "<event>:<agent>"`, and a single free-text `summary`
|
||||
(`rebuilt_todo_summary` renders `Rebuilt`'s `ok`/`note`/`sha`/`tag`
|
||||
fields into that string).
|
||||
|
||||
To add a new lifecycle notice: if it's "FYI, check when convenient,"
|
||||
call `swarm_notices::notify` — no new wire type needed. If it needs to
|
||||
drive an immediate turn, decide first whether it actually needs an
|
||||
_agent_ in the loop at all (the crash case above didn't, once a
|
||||
swarm-level alert could carry the same signal).
|
||||
|
||||
## Autoupdate on startup
|
||||
|
||||
`hive-c0re serve` runs `auto_update::run` in a background task right
|
||||
after opening the coordinator. It enumerates managed containers and
|
||||
rebuilds any whose closure marker differs from the store path the agent's
|
||||
record in `agent-closures` declares. That's how a hive that missed the
|
||||
swarm's deploy request catches up, and it repeats every five minutes, which
|
||||
covers a request lost while the hive is up. Sub-agents and the root agent go
|
||||
through the same graceful rebuild subgraph.
|
||||
|
||||
The comparison reads the record and the marker
|
||||
(`/var/lib/hyperhive/applied/.<name>.closure`, written after a successful
|
||||
swap or create) and nothing on the forge. Where the marker is missing, as for
|
||||
every agent on the first boot after the marker arrived, the hive reads the
|
||||
closure the container's profile
|
||||
(`/nix/var/nix/profiles/per-container/h-<name>/system`) points at and writes
|
||||
that as the marker, so the first pass catches an agent that doesn't run its declared
|
||||
closure. Those first-boot rebuilds go through the same job queue as
|
||||
any other: each holds one of the `buildSlots`, so they don't swap all at once.
|
||||
It rebuilds nothing when the hive can't tell: no connection to the swarm queue,
|
||||
no record, a closure the builder didn't push, or a profile it can't read or that
|
||||
leaves the nix store. The periodic pass
|
||||
attempts a declared store path once per process, and skips an agent while a
|
||||
rebuild of it waits or runs, so a closure that fails to deploy doesn't
|
||||
restart its agent every five minutes. Restarting `hive-c0re`, or a rebuild from
|
||||
the agent's menu, tries again.
|
||||
|
||||
A stopped agent gets its rebuild at its next start: the start path upgrades
|
||||
the start of a stale, stopped agent to a rebuild followed by the start. In
|
||||
the window after a config merge, before the swarm's builder has recorded the
|
||||
new closure, the rebuild deploys the previous one.
|
||||
|
||||
The dashboard shows no pending-update state: the hive applies a changed
|
||||
declaration itself. The agent menu's rebuild queues the same
|
||||
`job_queue::templates::rebuild` DAG, which downloads and applies the declared
|
||||
closure again.
|
||||
822
docs/4-implementation/agent-state/persistence.md
Normal file
822
docs/4-implementation/agent-state/persistence.md
Normal file
|
|
@ -0,0 +1,822 @@
|
|||
# Persistence: sqlite, state dirs and run-time layout
|
||||
|
||||
The exact tables, file layouts, mounts and migrations behind what survives
|
||||
what. The operator's summary, destroy against purge, is in
|
||||
[persistence and config changes](../../2-handbook/agents/persistence.md).
|
||||
|
||||
## Sqlite databases
|
||||
|
||||
### `/var/lib/hyperhive/db/broker.sqlite` (host)
|
||||
|
||||
One table, the message queue:
|
||||
|
||||
- `messages` — every inter-agent / operator-authored message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
in_reply_to / priority`. `in_reply_to` links a reply to its parent
|
||||
row id; the dashboard and per-agent inbox render these as threaded
|
||||
rows.
|
||||
|
||||
⚠️ The `mcp__hyperhive__remind` queue lives in a harness-local,
|
||||
per-agent store — see [`/harness/` contents
|
||||
below](#state-dirs-per-agent) for where reminders (and todos)
|
||||
live.
|
||||
|
||||
The swarm owns power intent, so hive-c0re keeps none on disk. An older build
|
||||
also kept a `kv` table (the agents running at the last `hivectl stop`) and an
|
||||
`agent_power` table (the durable `wanted` intent) here; opening the file drops
|
||||
both.
|
||||
|
||||
On a host whose hive-c0re once kept scheduled prompts, the file also
|
||||
holds `scheduled_prompts` and `scheduled_prompt_targets`. Nothing opens
|
||||
them; scheduled prompts live at swarm level
|
||||
([`2-handbook/swarm/schedules.md`](../../2-handbook/swarm/schedules.md)). They're safe to drop by
|
||||
hand.
|
||||
|
||||
Retention:
|
||||
|
||||
- `Broker::vacuum_delivered` runs hourly via a tokio task in
|
||||
`hive-c0re::main`. Drops acked message rows older than 30 days
|
||||
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
|
||||
rows are always kept — the harness `ack_turn`s only after a
|
||||
successful turn, so `requeue_inflight` can still requeue
|
||||
an unacked row on a crash.
|
||||
|
||||
### `/harness/hyperhive-events.sqlite` (per agent)
|
||||
|
||||
Lives inside each container's bind-mounted `/harness/` dir (host
|
||||
path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
|
||||
One table:
|
||||
|
||||
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
|
||||
harness emits during turn loop execution.
|
||||
|
||||
The db runs in WAL mode, so `-wal` and `-shm` files sit beside it while
|
||||
the harness is up. Copy or back up the three together, or the recent rows
|
||||
sit in the `-wal` file until a checkpoint.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
The harness both writes and vacuums it. Cleanup runs in-container
|
||||
because hive-c0re runs as the unprivileged `hive-core` user under
|
||||
privsep and can't delete agent-owned files (host-side deletes hit
|
||||
`PermissionDenied` on the bash-task trio and a readonly database error
|
||||
here). `hive-agent`'s `vacuum::run`
|
||||
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
|
||||
**type-scoped**: it deletes only the verbose `stream` rows (the raw
|
||||
claude `stream-json` deltas — one per text chunk / tool use, the bulk
|
||||
of the file's size) older than 14 days, and keeps every other kind
|
||||
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
|
||||
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
|
||||
small and carry the semantic per-turn history the operator scrolls
|
||||
back through when debugging a regression. Age-only within the
|
||||
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
|
||||
history sooner than a quiet one. The trade-off (accepted): a
|
||||
misbehaving harness can skip its own cleanup — but a compromised harness is
|
||||
already inside the container trust boundary
|
||||
([`docs/2-handbook/agents/security.md`](../../2-handbook/agents/security.md)), and these are ephemeral local
|
||||
artifacts, so cleaning them up where they live is the honest fix.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
|
||||
setups). On open failure the `Bus` falls back to no-store mode
|
||||
rather than crashing the harness — events still broadcast over SSE,
|
||||
just nothing persisted.
|
||||
|
||||
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
|
||||
|
||||
Per-turn analytics sink. One row per claude turn captures
|
||||
identity (`model`, `wake_from`, `result_kind`), timing
|
||||
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
|
||||
cache_read / cache_creation token counts), behaviour
|
||||
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
|
||||
snapshot metrics (`open_threads_count`,
|
||||
`open_reminders_count` — fetched via the same socket the harness
|
||||
already uses for `GetOpenThreads` + `CountPendingReminders`).
|
||||
Bin-loop helpers `build_row` + `record` land each row at
|
||||
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
|
||||
the turn loop continue.
|
||||
|
||||
The `hive-bash-daemon` (not the harness) writes a sibling
|
||||
`bash_commands(ts INTEGER, head TEXT)` table in the same
|
||||
file: one row per executed bash task recording the normalised command head -
|
||||
the basename of the first real command, looking past `cd repo &&`
|
||||
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
|
||||
backs the "favorite tools" view on the /stats page. Best-effort and created on first write
|
||||
(`CREATE TABLE IF NOT EXISTS`), so it's absent until a bash
|
||||
task runs.
|
||||
|
||||
turn-stats.sqlite has **no vacuum** — it's one small row per turn
|
||||
(~hundreds of KB even over months), read directly by the `/stats` page,
|
||||
so pruning it would only lose trend
|
||||
history for no space gain.
|
||||
|
||||
### `/state/hyperhive-harness.json` (per agent)
|
||||
|
||||
Consolidated harness state file written atomically (`.tmp` + rename) by
|
||||
`Bus::emit_status` whenever rate-limited or login-failed flags change.
|
||||
Shape:
|
||||
|
||||
```json
|
||||
{ "rate_limited": false, "needs_login": false, "active_model": "…" }
|
||||
```
|
||||
|
||||
- `rate_limited` — set when the harness detects a 429 from the Claude
|
||||
API; cleared by any subsequent status emit. Drives
|
||||
`ContainerView.rate_limited` on the dashboard.
|
||||
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
|
||||
cleared by `"online"` status (re-auth completed). Drives the
|
||||
`needs_login` flag alongside the `claude_has_session` check.
|
||||
- `active_model` — the resolved Claude model for the dashboard badge.
|
||||
|
||||
The turn loop is the only writer today, but it still goes
|
||||
read-modify-write under a shared in-process lock and merges into the
|
||||
existing object rather than reconstructing it — so a second writer
|
||||
would preserve fields it doesn't own, and the lock closes the
|
||||
lost-update window between a writer's read and its rename. The lock
|
||||
is in-process only, so it wouldn't serialise a writer running as a
|
||||
separate process; none of today's writers are.
|
||||
|
||||
hive-c0re reads this file on each `build_all` sweep (~10s) via
|
||||
`container_view::read_harness_flags`. Falls back to the individual
|
||||
sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) when the
|
||||
JSON is absent.
|
||||
|
||||
### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
|
||||
|
||||
Full stdout + stderr capture for every `nixos-container` / `nix
|
||||
build` invocation the lifecycle layer fires. One row per invocation;
|
||||
the row accumulates lines as the child runs.
|
||||
|
||||
Capturing the full stream (rather than a short tail buffer) matters
|
||||
because real eval errors routinely run long — "tried alternatives"
|
||||
blocks alone are often 30+ lines — so a truncated tail would cut off
|
||||
the actual failure and leave only the host journal holding the
|
||||
complete output. With this table the dashboard can surface the entire
|
||||
log.
|
||||
|
||||
Three indices:
|
||||
|
||||
- `(agent, started_at)` — backs the per-agent latest-N lookup used
|
||||
by the agent card chip.
|
||||
- `(status, finished_at)` — backs the retention sweep that runs
|
||||
as part of the existing hourly vacuum.
|
||||
- `(node_id)` — added by a later migration so a build log row can be
|
||||
looked up by the job-queue node it belongs to (a `hive_jobq` node is
|
||||
immutable after insert, so hive-c0re records the link on the log row
|
||||
instead); `node_id` is nullable.
|
||||
|
||||
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
|
||||
log a warning on sqlite error and let the build continue. A failed
|
||||
log row never blocks a rebuild.
|
||||
|
||||
Each operation also logs to the host journal, target `nixos-container`, and the
|
||||
swarm log store collects those lines. Every line carries `build_id` (this
|
||||
table's row id; empty without the store), `agent`, `kind` and `cmdline`. A
|
||||
`start` line (`event=start`) opens the build, one line follows per line of
|
||||
output, and a `finish` line (`event=finish`) closes it with `status` (`ok` or
|
||||
`fail`) and `runtime_secs`. hive-c0re logs a nix warning or error at warn, and a failed
|
||||
finish too.
|
||||
|
||||
A row is open (`finished_at IS NULL`) only while its child runs. At startup
|
||||
`Coordinator::open` closes every row still open as `fail` and appends a
|
||||
line saying hive-c0re stopped: the previous process died mid-build, and
|
||||
the retention sweep never deletes open rows.
|
||||
|
||||
### `/harness/hyperhive-model` (per agent)
|
||||
|
||||
Single-line text file holding a claude model name. The harness only
|
||||
reads it, at boot in `Bus::new` and again when the swarm clears the
|
||||
model it set. It holds the model the agent runs when neither the swarm nor
|
||||
`HIVE_DEFAULT_MODEL` (agent.nix) names one. Nothing in the harness
|
||||
writes it. Path overridable via `HYPERHIVE_MODEL_FILE`. Survives
|
||||
destroy/recreate, gone on `--purge`.
|
||||
|
||||
### `/harness/paused` (per agent)
|
||||
|
||||
Marker file, empty when an operator paused the agent. Its presence parks the agent's turn loop: the
|
||||
harness keeps serving its web UI and MCP daemons but drives no turns,
|
||||
and inbox messages queue unacked until it's removed (see
|
||||
[turn loop](../turn-loop/README.md#the-loop)).
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
Unusually, it's read and written from **both** sides of the harness
|
||||
bind-mount, and that's the whole design: the harness stats it
|
||||
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
|
||||
stats it on the host (`Coordinator::is_paused`) to populate the
|
||||
`paused` field on the agent card, and asks hive-priv to create or remove it
|
||||
(`Coordinator::set_paused`; hive-c0re itself is unprivileged) for `hivectl agent <name> pause|resume` and the
|
||||
dashboard toggle. Because the file itself is the only shared state
|
||||
there's no protocol between them, no round-trip into the container, and
|
||||
pause keeps working when the harness is wedged or the container is
|
||||
stopped.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
When the harness pauses itself, after three todo wakes in a row that skipped
|
||||
`get_loose_ends`, it writes the reason into the marker. The agent's turn-state
|
||||
header reports a non-empty marker as `paused_by_hive`
|
||||
([swarm](../../2-handbook/swarm/README.md)), and the swarm then declares the
|
||||
agent `paused` so the hive's wanted-state convergence doesn't resume it.
|
||||
|
||||
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
||||
is the agent's own space to fill, and this is harness control state.
|
||||
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
||||
back paused after a restart, which is the intended behaviour rather
|
||||
than an accident of storage.
|
||||
|
||||
## State dirs (per agent)
|
||||
|
||||
Under `/var/lib/hyperhive/agents/<name>/`:
|
||||
|
||||
- `config/` — unused. Older hives kept a proposed nix repo here; nothing
|
||||
reads it any more, and you can delete it. An agent reads its config by
|
||||
cloning `agent-configs/<name>` from the forge.
|
||||
- `claude/` — claude OAuth credentials, bind-mounted RW to
|
||||
`/home/<name>/.claude` inside the container.
|
||||
- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
|
||||
to `/agents/<name>/state` inside the container (uniform for
|
||||
all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
|
||||
the same path to in-container scripts. Notable files written here
|
||||
by the harness:
|
||||
- `hyperhive-status` — single-line free-text status string written
|
||||
by `set_status`; cleared on explicit `set_status("")`. Read by
|
||||
hive-c0re and the per-agent `/api/dashboard-state` endpoint to
|
||||
surface the status chip on the dashboard. Absent when no status
|
||||
has a value.
|
||||
- `hyperhive-harness.json` — rate-limited / needs-login flags read
|
||||
by the dashboard's async container-state fetch. See
|
||||
`docs/2-handbook/hives/dashboard.md::Container row`.
|
||||
|
||||
The agent writes its own instructions to `AGENTS.md` here; claude and
|
||||
opencode load it automatically from their cwd, `state/`. At boot,
|
||||
`agents_md_migrate::run` renames a `state/CLAUDE.md` to `AGENTS.md`
|
||||
when no `AGENTS.md` exists. When both exist it renames neither and
|
||||
logs an error naming both paths: claude loads only `CLAUDE.md` while
|
||||
one exists in its cwd or above. The harness also publishes the file
|
||||
to the swarm, which reads that copy, and applies the git diffs the
|
||||
swarm sends to edit it. A new agent's first start writes the text given at
|
||||
create, only when no file exists. This file stays the only copy that counts:
|
||||
[`docs/4-implementation/swarm/controller.md`](../swarm/controller.md#an-agents-agentsmd).
|
||||
|
||||
- `harness/` — harness-internal ephemeral state; not intended for
|
||||
agent consumption. Bind-mounted to `/agents/<name>/harness`
|
||||
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
|
||||
- `bash-tasks/` — task JSON + stdout/stderr files for
|
||||
background `mcp__bash__run` jobs. JSON files are
|
||||
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
|
||||
(full captured output). The harness's own hourly sweep
|
||||
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
|
||||
event rows above) deletes terminal task trios older than 48
|
||||
hours; non-terminal (still-running) tasks are never deleted. This
|
||||
sweep runs in-container, for the same privsep-ownership reason as
|
||||
the events vacuum above.
|
||||
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
|
||||
and reminders, one small table each in a single file (in-container
|
||||
daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
|
||||
`hive-forge-notify` — upsert keyed todos here over the harness's
|
||||
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
|
||||
`get_loose_ends` output and clears a row on `mark_todo_done`).
|
||||
At boot, `db_migrate::run` creates this file from
|
||||
`hyperhive-todos.sqlite` and `hyperhive-reminders.sqlite` when those
|
||||
are present. Also backs the `mcp__hyperhive__remind` queue.
|
||||
|
||||
The harness itself is also a producer, not just the socket server:
|
||||
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
|
||||
socket the out-of-process daemons above dial) alongside
|
||||
`disk_watch::run` — an _in-process_ todo producer that shares the
|
||||
store + wake `Notify` directly rather than dialling its own socket.
|
||||
`disk_watch` raises a keyed `disk` todo when the filesystem backing
|
||||
this agent's state gets tight, naming the agent's own biggest
|
||||
directories; it buckets the summary and carries no raw byte
|
||||
counts, so an unchanged situation re-upserts as `changed == false`
|
||||
and never re-wakes.
|
||||
|
||||
Retention, same hourly `hive-agent::vacuum::run` sweep as
|
||||
`hyperhive-events.sqlite` below: it reaps delivered (soft-deleted) reminder
|
||||
rows 14 days after delivery, kept that long only to serve
|
||||
the trailing-window `ReminderRollup` stats, and reaps acked todo rows
|
||||
30 days after acking (long enough that only a genuinely quiet
|
||||
month triggers the "one spurious re-announcement" fallback a
|
||||
reconciled producer like `disk_watch` relies on — see `todos.rs`'s
|
||||
module doc). Un-acked todos and undelivered reminders are never
|
||||
swept — same "audit trail, not cache" treatment as the c0re-side
|
||||
tables above.
|
||||
|
||||
- `hyperhive-events.sqlite` — turn-loop event log.
|
||||
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
|
||||
- `hyperhive-model` / `hyperhive-effort` — single-line model name and
|
||||
effort level files, read-only fallbacks.
|
||||
|
||||
- `work/` — the agent's working space for git worktrees, build dirs
|
||||
and other large working data. Bind-mounted RW to
|
||||
`/agents/<name>/work` inside the container and owned by the agent
|
||||
user. It sits outside `state/` so a claude or subagent started under
|
||||
`work/` doesn't walk up into `state/AGENTS.md` and load the agent's
|
||||
own identity. Survives `destroy`/recreate, gone on `--purge`. Counted
|
||||
in the per-agent disk metric (`measure_agent_disk`) together with
|
||||
`state/` and the container rootfs. The dashboard doesn't serve it.
|
||||
|
||||
### The container root (per agent)
|
||||
|
||||
An agent container starts with `systemd-nspawn --ephemeral`
|
||||
(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a
|
||||
snapshot of `/var/lib/nixos-containers/h-<name>`, and the snapshot vanishes
|
||||
at stop. The container loses anything the agent writes outside a bind: `/home/<name>`
|
||||
apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is
|
||||
`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container
|
||||
`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir.
|
||||
`/home/<name>` holds caches and nothing the agent should keep: worktrees,
|
||||
clones and builds go in `work/`.
|
||||
|
||||
The nix profile needs one extra step. nix keeps a non-root user's profile in
|
||||
`~/.local/state/nix`, so the agent's tmpfiles rules
|
||||
(`nix/agent-modules/user.nix`) point that directory into
|
||||
`/nix/var/nix/profiles/per-user/<name>` (the persistent bind) and recreate
|
||||
`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or
|
||||
`nix profile` survive a restart, and go with the container on `destroy`.
|
||||
|
||||
**The one-time salvage.** The first start of an agent whose root has content
|
||||
runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the
|
||||
`WriteNspawnFlags` that precedes every start, while the container is down:
|
||||
|
||||
1. Everything in the root except the cache paths (`tmp`, `var/cache`,
|
||||
`var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is
|
||||
copied to `work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, with
|
||||
modes and mtimes kept. The agent finds its old `/home/<name>` there.
|
||||
2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there).
|
||||
3. hive-priv renames the old root to `h-<name>.pre-ephemeral-<YYYYMMDD>` and
|
||||
logs the path. Nothing deletes it; the operator removes it when ready.
|
||||
4. A fresh template root takes its place, carrying `etc/machine-id` (the old
|
||||
id, so the journal directory and DHCP identity hold) and the marker
|
||||
`etc/hyperhive-ephemeral-root`.
|
||||
|
||||
The salvage runs again only if the template holds content outside the cache
|
||||
paths, which happens when a hive without this change started the container
|
||||
persistently in between; the new copy gets a numbered directory beside the
|
||||
first. A failed salvage leaves the root where it was and the container starts
|
||||
without the flag (hive-priv logs the error), so a failure costs the switch,
|
||||
never data.
|
||||
|
||||
**Checked before every start.** nixpkgs' start script runs as host root on
|
||||
the root template, and its `mkdir -p`, `chmod`, `touch` and
|
||||
`cp --remove-destination` follow a link the container planted at the paths
|
||||
they touch. The container's limits drop-in
|
||||
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
|
||||
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
|
||||
script, on systemd's own restarts too. It looks at each path with `lstat`,
|
||||
never following a link, and fails the start with one line on stderr naming the
|
||||
path and its type when:
|
||||
|
||||
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
|
||||
exists and isn't a real directory;
|
||||
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
|
||||
(`etc/os-release` may also be a link, which the script skips and a booted
|
||||
NixOS guest creates);
|
||||
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
|
||||
exists and isn't a link that resolves into `/nix/store`, or the `system`
|
||||
file inside the resolved directory isn't a regular file.
|
||||
|
||||
Absent paths pass, so a first start works. The check neither deletes nor
|
||||
repairs: the operator removes the offending entry, and the next start goes
|
||||
through.
|
||||
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
|
||||
points at a build without the sub-command. A drop-in written by an older
|
||||
points at a build without the sub-command. That build's store path has a GC
|
||||
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
|
||||
drop-in and replaced on each write, so a garbage collection can't remove the
|
||||
binary a restart would run. The directory sits beside `gcroots/per-container`,
|
||||
which is the only gcroots path bound into a container, so only root writes it.
|
||||
Destroying the container removes the drop-in and then its root; the build stays
|
||||
until the next collection. A drop-in written by an older hive-priv has no
|
||||
`ExecStartPre=` and no root, and the container starts as before.
|
||||
|
||||
**Hyperhive's platform containers: warn only.** The forge, the matrix server and
|
||||
the swarm services (15 containers, declared by their own modules) run the same
|
||||
check as `hive-priv check-rootfs --warn-only <name>`, from `ExecStartPre=` on
|
||||
their own `container@<name>` unit, through the symlink `/run/hive-rootfs-check`
|
||||
that `hive-priv.nix` points at the current build, so a new hyperhive build does
|
||||
not change the unit or restart the container. It evaluates every rule, logs each
|
||||
violation
|
||||
as a warning, and exits 0, so the container starts either way. The line reads
|
||||
`rootfs check (warn-only): <name>: <message>` with journald's warning priority.
|
||||
Strict refusal needs the live roots of all 15 to pass first, and the warnings
|
||||
show which ones would not. A container the operator runs that hyperhive doesn't
|
||||
declare gets no check at all: the modules name their own containers, and
|
||||
nothing enumerates the host's `containers`.
|
||||
|
||||
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
|
||||
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and
|
||||
copies the data of a file the filesystem refuses to clone. Before it creates
|
||||
anything, hive-priv sizes the root and compares it with the free space under
|
||||
`work/`. A salvage whose files clone (tested with one file of the root) must
|
||||
leave 2 GiB free. One that has to copy data must leave 15 % of the filesystem
|
||||
free. When the free space is short, hive-priv skips the salvage, deletes
|
||||
nothing, logs `salvage skipped: <container> need <size> free <size>`, and the
|
||||
container starts persistent; the next start tries again. hive-priv checks a file
|
||||
that fails to clone against the 2 GiB reserve again before it copies the data. A
|
||||
salvage that fails partway removes the directory it created, and leaves the root
|
||||
and any earlier salvage as they were. After a salvage, hive-priv logs `salvage: <container> <size> reflinked=<n>/<m> files`.
|
||||
|
||||
**Concurrency.** hive-priv holds an `flock` on
|
||||
`/run/hive-priv/salvage-<container>.lock` (the unit's `RuntimeDirectory`, which
|
||||
no container can write) for the whole `WriteNspawnFlags`, so a second request for
|
||||
the same container waits and then finds the finished template. Across
|
||||
containers, one salvage copies at a time, so the free space each one checks
|
||||
excludes any copy still in flight. The request runs on the blocking
|
||||
thread pool, so a long copy holds no async worker. hive-c0re's per-agent lease
|
||||
already serialises the requests it sends; the lock covers hive-priv's own
|
||||
guarantee.
|
||||
|
||||
The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`)
|
||||
reads the guest's own `/var/log/journal` and pushes it over OTLP, so no
|
||||
journal link to the host takes part; nspawn skips `--link-journal=try-guest`
|
||||
in ephemeral mode without failing. The journal vanishes at stop, and with it
|
||||
the lines written after the last push.
|
||||
|
||||
### User namespace (opt-in, per agent)
|
||||
|
||||
An agent whose config sets `services.hyperhive.agent.user.privateUsers = true`
|
||||
runs in its own user namespace. Every other agent keeps the host's uids.
|
||||
|
||||
**Approving the config PR that sets it authorises a host-level change:** the
|
||||
first start afterwards has hive-priv, as root, chown the agent's `state`,
|
||||
`harness`, `work` and `claude` dirs once to the host's data owner. It deletes nothing.
|
||||
|
||||
**How the hive learns the flag.** It never evaluates the agent's nix. The swarm
|
||||
builder reads `config.services.hyperhive.agent.user.privateUsers` from the
|
||||
config it evaluates in its sandbox and puts it in the agent's closure record
|
||||
(`private_users`). hive-c0re reads the record when it writes the container's
|
||||
flags and sends the value to hive-priv with each start. An agent whose module
|
||||
doesn't declare the option reads false. When the builder can't read the
|
||||
option (its eval fails or times out), the record carries no flag, and so does a
|
||||
record or builder from before the field. A record without a flag, a record
|
||||
hive-c0re can't read, or none, leaves the agent as it is.
|
||||
|
||||
- **On disk the data belongs to one host uid:gid**,
|
||||
`services.hyperhive.deploy.hive-controller.userns.dataOwner`
|
||||
(default `60000:60000`). Inside the container that owner shows as the agent
|
||||
user. hive-priv mounts the agent's `state/`, `harness/`, `work/` and
|
||||
`claude/` dirs and its socket dir with `owneridmap`, and every other bind too
|
||||
(config, `/knowledge`, the runtime dir), whose owner shows as root inside.
|
||||
- **`/nix/store` has to be on ext4, btrfs, xfs or erofs.** nspawn mounts the
|
||||
store into a namespaced container idmapped, and the kernel refuses that on
|
||||
overlayfs and on some other filesystems. `WriteNspawnFlags` for an opted-in
|
||||
agent checks the `statfs` type of `/nix/store` against an allowlist of those
|
||||
four; the VM test's erofs store idmaps on the hive's kernel. The check
|
||||
refuses every other filesystem, zfs and tmpfs included, even where a newer
|
||||
kernel could idmap it, because the gate exists to stop a container that would
|
||||
not boot. The error names the type and the fix, and
|
||||
comes before anything moves. Check a host with
|
||||
`findmnt -no FSTYPE /nix/store`.
|
||||
- **The container starts with `--private-users=pick --private-users-ownership=map`.**
|
||||
nspawn idmaps the root at mount time, so a different range for `pick` never
|
||||
walks or chowns it.
|
||||
- **The move happens once.** The first `WriteNspawnFlags` that finds the
|
||||
agent's container stopped chowns those four dirs to `dataOwner` (ownership
|
||||
only; it removes or copies nothing), writes
|
||||
`/var/lib/hive-priv/userns/<name>`, and creates the mount points in the root
|
||||
template owned by the agent user. A later request with the marker present
|
||||
chowns nothing. A running container waits for its next stop. The socket dir
|
||||
is on a tmpfs, so `EnsureAgentSocketDir` hands it to `dataOwner` at every
|
||||
start. The walk works by open directory fd and follows no link, so an agent
|
||||
or manager that can write the dirs can't redirect it to a host path.
|
||||
- **A move that fails starts over.** hive-priv opens all four dirs and
|
||||
checks the agent's ids before the first chown, so a link or file where a dir
|
||||
goes refuses with nothing changed. It then writes the ids to
|
||||
`/var/lib/hive-priv/userns/<name>.pending`. If the walk fails after that (a
|
||||
chown error, a tree nested too deep), the next start repeats it from those
|
||||
ids, then writes the marker and removes the pending file. hive-priv refuses a
|
||||
pending file that doesn't hold `<owner> <uid> <gid>`, and names it.
|
||||
- **The agent's gid must equal its uid.** `owneridmap` maps the gid of a bind
|
||||
with the uid map, so a group other than the uid's has no mapping and writes
|
||||
fail. The agent module asserts that `user.uid` and `user.gid` hold the
|
||||
same number when `privateUsers` is on. Start the agent once with the pin
|
||||
before turning it on: the move refuses an agent whose `state` dir has another
|
||||
gid than uid, naming the agent.
|
||||
- **Turning it off is manual.** hive-priv refuses to start an agent whose
|
||||
marker exists while its record says false, and one whose `dataOwner`
|
||||
changed. Chown the four dirs back, then delete the marker and any
|
||||
`<name>.pending` beside it.
|
||||
- **Keep a `ManageRootAgent` holder off it.** The other agents' dirs it mounts
|
||||
aren't remapped.
|
||||
|
||||
The pause marker needs nothing extra: hive-priv gives it the owner of the
|
||||
`harness/` dir. hive-c0re still creates agent dirs and new subvolumes as
|
||||
`hive-core`, and only the first namespaced start chowns them. A dir created
|
||||
after the move, such as a recreated subvolume, stays `hive-core`'s and the
|
||||
agent can't write to it.
|
||||
|
||||
### Cross-agent access to state
|
||||
|
||||
An agent holding the `manage_root_agent` permission gets every other
|
||||
agent's `state` dir bind-mounted **read-write** (`bind_child_agent_dirs` in
|
||||
`lifecycle/host_config.rs`). The RW is deliberate, not an oversight: the
|
||||
holder recovers other agents, which includes writing into their state (for
|
||||
example seeding notes, clearing a stuck sentinel) as well as reading it.
|
||||
|
||||
This is the **only** cross-agent mount. An agent without that permission
|
||||
sees its own dirs and nothing else.
|
||||
|
||||
**`harness` isn't mounted at all.** It holds that agent's own runtime
|
||||
material - `bash-tasks/`, the turn-stats and event sqlite dbs - and
|
||||
nothing argues for anyone else reading it, let alone writing it.
|
||||
hive-c0re reads a harness dir **directly on the host** when it wants
|
||||
those stats, which needs no mount into another container.
|
||||
|
||||
**`config` isn't mounted either.** A config change is a PR on that agent's
|
||||
config repo, made from a clone and merged after review. Anyone who wants to
|
||||
read the config clones `agent-configs/<name>` from the forge. A mount, even a
|
||||
read-only one, would show an agent a tree that's not the one that governs it
|
||||
whenever the forge has moved on.
|
||||
|
||||
Isolation still holds by default: a container has its _own_ dirs and,
|
||||
unless it holds the permission, nothing else.
|
||||
|
||||
Under `/var/lib/hyperhive/applied/` — the hive-c0re-only deploy markers
|
||||
`.<name>.closure`: the store path of the closure the agent was last swapped to
|
||||
or created from, or, where the file was missing, the closure the container's
|
||||
profile pointed at when the hive first read it. No config lives here; the
|
||||
forge's `agent-configs/<name>` is the only copy. A hive that predates this
|
||||
finds per-agent `applied/<name>` repos here. Nothing reads them any more, and
|
||||
you can delete them.
|
||||
|
||||
Under `/var/lib/hyperhive/meta/` — system-level config files, plain state
|
||||
with no git history. The flake agents build from is the
|
||||
swarm's `internal/meta`, not here. A hive that predates this finds a
|
||||
leftover `.git` here; `migrate::run` deletes it at startup. Contents:
|
||||
|
||||
- `topology.json` — the agent roster (`["alice", "bob", "ruth"]`).
|
||||
Written by `topology::reconcile` from `agent_set::refresh`; read by
|
||||
`topology::all_agents`, which is the set the `manage_root_agent`
|
||||
permission grants mounts over. The reader also accepts a map-shaped
|
||||
file carrying a `parent` per agent, and keeps its keys.
|
||||
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
||||
hive's agents
|
||||
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
||||
Written only by `workers::swarm_limits`, which copies each agent's key of
|
||||
the swarm's `agent-limits` bucket here every 30 seconds. Read where
|
||||
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
|
||||
into the container — these are host-side caps on the container, so
|
||||
the capped party never sets them. It sees only the resulting memory cap,
|
||||
as a byte count or `infinity` in its runtime dir (below). Fallback is per
|
||||
_field_: an absent file, absent agent, or absent field falls back to the
|
||||
hive-wide `services.hyperhive.deploy.hive-controller.agentCpuQuota` / `agentMemoryMax`,
|
||||
so an agent can have only its memory capped and still track the hive
|
||||
default for CPU. Every value passes the validators first, and one bad
|
||||
value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
|
||||
shares in the same drop-in have **no** per-agent setting — they're
|
||||
hive-wide only and come straight off `HiveEnv`, so this file has no field
|
||||
for them.
|
||||
- `resource-limits.json` — retired. Per-agent overrides set on the hive used
|
||||
to live here; nothing reads or writes it now. An old copy in this directory
|
||||
is harmless and you can delete it.
|
||||
|
||||
An agent holding `manage_root_agent`, normally ruth, has the meta dir
|
||||
RO-mounted at `/meta/`.
|
||||
|
||||
Outside this directory, `/var/lib/hyperhive/swarm-agent-permissions.json`
|
||||
holds the permissions the swarm grants this hive's agents, as
|
||||
`workers::swarm_permissions` last read them from the swarm's
|
||||
`agent-permissions` bucket (every 30 seconds): a copy, not the
|
||||
record, which `swarmctl agent permissions` sets. `set_nspawn_flags`
|
||||
reads it at every container start for `manage_root_agent`.
|
||||
|
||||
`/var/lib/hyperhive/swarm-agent-archetypes.json` holds the archetype each of
|
||||
this hive's archetype agents runs, as last read from the swarm's
|
||||
`agent-archetype` bucket at the agent's provision and each rebuild. An agent
|
||||
with an entry has no config repo and no `agent-<n>` meta input; see
|
||||
[archetypes](../../2-handbook/swarm/archetypes.md).
|
||||
|
||||
hive-c0re creates containers from the closure the swarm builds from `internal/meta` (`nixosConfigurations.<n>`).
|
||||
|
||||
## Destroy vs purge
|
||||
|
||||
See [For operators](#for-operators) above for what each action does to
|
||||
an agent's state. The mechanics, for completeness:
|
||||
|
||||
- `DESTR0Y` also drops the systemd drop-in; reviving the agent is
|
||||
`swarmctl agent create` with the same name and hive, which reuses the
|
||||
kept state.
|
||||
- `PURG3` wipes `/var/lib/hyperhive/agents/<name>/` — the
|
||||
union of everything `DESTR0Y` left behind.
|
||||
|
||||
`actions::destroy` has no guard for the root/bootstrap agent: it destroys ruth
|
||||
like any other agent, and she stays gone until you create her again from the
|
||||
swarm dashboard.
|
||||
|
||||
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
|
||||
|
||||
On a btrfs host, `lifecycle::ensure_agent_state_subvolume` creates a brand-new agent's state root as a
|
||||
**btrfs subvolume** instead of a plain directory (progressive
|
||||
enhancement). This is a no-op fallback on
|
||||
non-btrfs hosts and for any agent whose root already exists, so
|
||||
hive-c0re automigrates nothing: existing agents keep their plain dirs
|
||||
until an explicit opt-in upgrade.
|
||||
|
||||
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
|
||||
hive-c0re creates the per-agent subdirs (spawn / rebuild).
|
||||
It skips the work when the root already exists; otherwise it asks
|
||||
hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
|
||||
root when the FS is btrfs (`statfs` magic gate) and chown it to the
|
||||
`hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
|
||||
succeed inside it.
|
||||
- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
|
||||
reuses it untouched.
|
||||
- **PURG3 deletes it correctly:** `rmdir`/`remove_dir_all` can't remove
|
||||
a subvolume root, so purge first calls hive-priv
|
||||
(`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
|
||||
actually a subvolume, then the normal `remove_dir_all` sweep covers
|
||||
plain-dir agents.
|
||||
|
||||
Per-subvolume disk-usage accounting and optional quotas have since
|
||||
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
|
||||
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
|
||||
`hivectl agent <name> quota show|set` reads/limits one agent's
|
||||
subvolume usage through the same hive-priv-mediated path as
|
||||
subvolume creation/deletion above.
|
||||
|
||||
The old snapshot store no longer exists; swarm-archive will replace it for agent-state backups, and nothing copies the subvolume off the host in the meantime.
|
||||
|
||||
The subvolume is the whole agent state root, so `work/` is part of every
|
||||
snapshot and migration of it and can make them much larger.
|
||||
|
||||
## `/var/lib/swarm-controller/` (swarm-controller host only)
|
||||
|
||||
Only present on the one host running
|
||||
`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
|
||||
so it survives restarts and redeploys.
|
||||
|
||||
- `webhook-secret` — the HMAC key Forgejo signs the swarm's forge webhooks
|
||||
with, as it was before the swarm secret store held it. **Keep it.** The
|
||||
controller stores its value in the swarm secret store at
|
||||
`swarm/controller/swarm-controller/forge/webhook-secret` the first time it
|
||||
starts with that path empty, and reads the store from then on. The file is the
|
||||
fallback while the store is unreachable or its policy doesn't grant the path
|
||||
yet, and nothing else reads or removes it. A host with neither the file nor a
|
||||
store value gets a new secret in the store, with nothing to configure.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
If the store can't answer and the file is unreadable or missing at startup, the daemon still starts and logs
|
||||
`webhook secret unavailable`; the webhook endpoint then answers 503
|
||||
rather than accepting deliveries it can't verify. Everything else the
|
||||
controller serves is unaffected. The daemon keeps reading the store in the
|
||||
background, at most five minutes apart, and the endpoint and the forge hook
|
||||
registration start working when it answers. It writes a new secret only when
|
||||
the store answers that it holds none, never because a read failed.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
- `operator-matrix-admins.json` — the matrix user ids the operator admin
|
||||
sync has made homeserver admin, a JSON array. The sync revokes admin
|
||||
only from these, so deleting the file leaves every current admin in
|
||||
place; a missing file is an empty set. A file the controller can't
|
||||
read or parse stops the sync, grants included, and the controller logs
|
||||
`operator matrix admin: pass failed` on every pass. See
|
||||
[`docs/2-handbook/swarm/matrix.md`](../../2-handbook/swarm/matrix.md#operators-are-homeserver-admins).
|
||||
|
||||
## Run-time dirs
|
||||
|
||||
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
||||
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
|
||||
Without that, every restart wipes bind sources and existing
|
||||
containers can't start.
|
||||
|
||||
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
|
||||
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
|
||||
(bind-mounted into the container as `/run/hive/mcp.sock`).
|
||||
- `/run/hyperhive/agents/<name>/memory-max-bytes` — the agent's effective
|
||||
`MemoryMax=` as a byte count (a percentage resolved against the host's
|
||||
physical memory) or `infinity`, written by `lifecycle::write_dropins`
|
||||
before every start. Absent only when hive-c0re can't tell the cap. The
|
||||
container reads it at boot as `/run/hive/memory-max-bytes`
|
||||
(`nix/agent-modules/memory-cap.nix`).
|
||||
|
||||
On startup, `Coordinator::register_agent` drops any prior socket
|
||||
task before rebinding — idempotent so a hive-c0re restart followed
|
||||
by `rebuild alice` recreates the agent's socket without a clean
|
||||
reinstall.
|
||||
|
||||
## First-boot agent-user migration
|
||||
|
||||
The harness runs as a per-agent unix user inside the container
|
||||
(`services.hyperhive.agent.user.name`, defaults to the agent's logical label so each
|
||||
container has a uniquely named user). The activation script below
|
||||
moves root-owned state to the agent user so the claude session survives.
|
||||
|
||||
`system.activationScripts.hive-agent-user-migrate` (in
|
||||
`nix/agent-modules/user.nix`) runs on every activation,
|
||||
marker-guarded so the substantive moves only happen once per
|
||||
container lifetime:
|
||||
|
||||
1. **`${homeDir}` exists with the right ownership** — covers the
|
||||
the first boot before `useradd`'s `createHome` has had a
|
||||
chance to chown. Also re-applies on every rebuild in case the
|
||||
meta-flake's per-agent name evolves (rare).
|
||||
2. **Migrate any leftover `/root/.claude` content into
|
||||
`${homeDir}/.claude`.** Marker
|
||||
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
|
||||
only written once there's nothing left to migrate, so a `cp`
|
||||
failure leaves it absent and the next boot retries. The root is
|
||||
ephemeral, so the marker vanishes at every stop and the step
|
||||
re-runs; with no `/root/.claude` it only writes the marker again.
|
||||
`cp -an` (no-clobber) so any pre-existing files at the new
|
||||
location win — never blow over data already there, and so a
|
||||
retry after a partial copy is as safe as the first attempt.
|
||||
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
|
||||
recursively so the agent user can read/write it. Wildcard
|
||||
matches the single agent that container sees; `-h` skips
|
||||
symlinks the agent might have planted.
|
||||
4. **Chown the `~/.claude/` bind-mount** recursively. The harness reads
|
||||
`~/.claude/` as the agent user to decide Online vs NeedsLogin in
|
||||
`login::has_session`. Without the chown, credentials owned by
|
||||
another uid get silently treated as "no session" and the operator
|
||||
re-prompts every boot.
|
||||
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
|
||||
user**, `0751`, not recursive. hive-priv creates it `0751 root`
|
||||
on the host before every start; the harness binds its sockets
|
||||
there as the agent user. Why the mode matters:
|
||||
[`boundary-mechanism.md`](../trust-boundary/boundary-mechanism.md#the-per-agent-socket-dir).
|
||||
A namespaced agent has the dir bound at `/agents/<name>/sockets` and
|
||||
linked from `/run/hive-agent/<name>`; the activation chowns nothing, because
|
||||
a mount point under `/run` would belong to root and the id mapping would
|
||||
give the agent user no ids there. A later in-container activation sees the
|
||||
link and logs `is a symlink or not a directory; not handing it`. The log
|
||||
is harmless.
|
||||
|
||||
## Matrix per-agent daemon + token-arrival trigger
|
||||
|
||||
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
|
||||
process per agent. Serves its MCP tools directly over
|
||||
streamable-http (`services.hyperhive.agent.mcp.matrixHttpPort`, no stdio bridge —
|
||||
same shape as `hive-bash-daemon`), emits hyperhive wake signals
|
||||
on incoming room events via `/run/hive/mcp.sock`. Conditional on the
|
||||
agent having at least one `services.hyperhive.agent.matrixAccounts`
|
||||
entry — there is no separate enable switch, and the same condition gates
|
||||
the daemon, its path watcher AND the autoinjected
|
||||
`extraMcpServers.matrix` entry. The module declares the hive-internal
|
||||
`main` entry whenever `services.hyperhive.agent.matrix.url` is non-null,
|
||||
so on a real hive (where hive-c0re renders that URL per agent) every
|
||||
agent has one; a `null` URL with no operator-declared account is the
|
||||
"this agent has no matrix" state.
|
||||
|
||||
**First-boot ordering**: a token can arrive after the container comes
|
||||
up — a declared token file, or a token the swarm mints into the store.
|
||||
Without the path-trigger sibling
|
||||
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
|
||||
<this agent's state dir>/matrix-token*` — the trailing `*` also catches
|
||||
a declared secondary token file like `matrix-token-ccc`), the daemon
|
||||
would exit 0 quietly the first time it ran and the MCP would have no
|
||||
daemon until the next restart. The `.path` unit makes the appearance
|
||||
of the token re-fire the service so the daemon comes alive in the
|
||||
same boot cycle as
|
||||
provisioning. A token in the store changes no file, so a store-backed agent
|
||||
also gets a timer that restarts the daemon five minutes after it last
|
||||
exited. A running daemon re-lists its linked accounts in the store
|
||||
every two minutes and, when the set changed, exits with status
|
||||
75, which the unit's `RestartForceExitStatus` restarts. On the same
|
||||
two-minute tick it tries again each secondary account that didn't come
|
||||
up — a declared account with no token yet, a token the homeserver
|
||||
rejected, or a homeserver that
|
||||
was down — waiting two minutes after its first failure and twice as long
|
||||
after each further one, up to sixteen minutes. It tries one account at a
|
||||
time and counts a try that hasn't finished after two minutes as a failure,
|
||||
so one hung account doesn't hold the others back. One that comes up joins
|
||||
the running daemon; the primary and the other accounts aren't
|
||||
restarted. The same token watcher also drives avatar setting: on a
|
||||
restart the daemon re-runs each account's bring-up, which sets the
|
||||
avatar (see below).
|
||||
|
||||
### matrix avatar (set by the daemon over the live Client)
|
||||
|
||||
`hive-matrix-daemon` itself publishes the agent icon (`services.hyperhive.agent.icon`, an SVG) as each matrix
|
||||
account's profile avatar
|
||||
(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
|
||||
the daemon builds + restores an account's `Client` (authenticated,
|
||||
pointed at that account's resolved homeserver), it calls matrix-sdk's
|
||||
`account().upload_avatar()` — one call that uploads the media and sets
|
||||
`avatar_url`. Because it reuses the live Client, there is no hardcoded
|
||||
homeserver URL, no token re-read, and no token-file globbing: the daemon
|
||||
already iterates every configured + dashboard-discovered account in its
|
||||
bring-up loop, so it sets the avatar for **every** account.
|
||||
|
||||
Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
|
||||
librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
|
||||
unit, gated on `services.hyperhive.agent.icon != null`. No icon configured → the env is
|
||||
unset → `sync_avatar` returns early and sets no avatar.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
Idempotency is **per-account**: an `avatar-icon-hash` file in each
|
||||
account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
|
||||
skips the upload when unchanged, because every upload mints a fresh
|
||||
`mxc://` URI that emits a profile state event in every joined room —
|
||||
re-uploading identical bytes is timeline spam. A swarm-UI-linked
|
||||
account gets its avatar when the daemon restarts on a changed account
|
||||
listing (exit 75, which re-runs the per-account bring-up), so no separate
|
||||
avatar trigger is needed. The daemon swallows avatar failures
|
||||
(logged, non-fatal) so they never break account bring-up or sync.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems.
|
||||
For the public API surface (dashboard, socket protocol) see `docs/5-project/process/conventions.md`
|
||||
and `docs/2-handbook/agents/persistence.md`.
|
||||
and `docs/4-implementation/agent-state/persistence.md`.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -328,7 +328,7 @@ boot with no operator action in this process never finds drift of its own.
|
|||
|
||||
Agents build from the swarm's `internal/meta` flake, which swarm-controller
|
||||
commits and the swarm builder builds from (see
|
||||
[Config changes](../../2-handbook/agents/config-changes.md#the-swarms-meta-flake)).
|
||||
[Config changes](../../4-implementation/agent-state/config-changes.md#the-swarms-meta-flake)).
|
||||
The hive keeps no flake of its own and deploys the closure the swarm builds
|
||||
from it, as the agent's `agent-closures` record declares (`attic/closure.rs`).
|
||||
`agent_set.rs` keeps the
|
||||
|
|
@ -536,7 +536,7 @@ config — all nspawn containers live in `machine.slice`.
|
|||
|
||||
## See also
|
||||
|
||||
- `docs/2-handbook/agents/config-changes.md` — config PRs, meta flake, system messages
|
||||
- `docs/2-handbook/agents/persistence.md` — SQLite schema, state-dir layout
|
||||
- `docs/4-implementation/agent-state/config-changes.md` — config PRs, meta flake, system messages
|
||||
- `docs/4-implementation/agent-state/persistence.md` — SQLite schema, state-dir layout
|
||||
- `docs/5-project/process/conventions.md` — wire protocol, recipient sentinels
|
||||
- `docs/2-handbook/agents/agent-roster.md` — the agent roster and privilege
|
||||
|
|
|
|||
|
|
@ -179,7 +179,7 @@ the deploy node, and the graph shows the update as failed and the deploy as
|
|||
skipped. Each ten-minute retry of a failed update declares a fresh build and
|
||||
deploy behind it for every agent whose latest deploy the failed update skipped,
|
||||
so the run that succeeds sends them (see
|
||||
[Config changes](../../2-handbook/agents/config-changes.md#the-swarms-meta-flake)). A render
|
||||
[Config changes](../../4-implementation/agent-state/config-changes.md#the-swarms-meta-flake)). A render
|
||||
equal to the committed `flake.nix` sends no lock.
|
||||
|
||||
The controller sends a deploy to every agent a hive places whenever the pin in
|
||||
|
|
|
|||
|
|
@ -395,7 +395,7 @@ them, and the notification router, not the controller, decides what they mean.
|
|||
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
|
||||
of its `merge_commit_sha` on the one hive whose wanted state places the
|
||||
agent; with no such hive, or more than one, the controller deploys nothing. See
|
||||
[config-changes.md § Config changes](../../2-handbook/agents/config-changes.md#config-changes).
|
||||
[config-changes.md § Config changes](../../4-implementation/agent-state/config-changes.md#config-changes).
|
||||
A merge whose `closed` delivery never arrives deploys nothing either; the
|
||||
operator recovers by redelivering that delivery from the forge hook page,
|
||||
which re-enters the same handler and re-queues the deploy.
|
||||
|
|
@ -422,7 +422,7 @@ host also serves the swarm UI vhost — that's what publishes the
|
|||
endpoint, and a hook the forge can't reach would collect failed
|
||||
deliveries while looking healthy. The controller generates the HMAC
|
||||
secret on first start and keeps it
|
||||
(see [`docs/2-handbook/agents/persistence.md`](../../2-handbook/agents/persistence.md)).
|
||||
(see [`docs/4-implementation/agent-state/persistence.md`](../../4-implementation/agent-state/persistence.md)).
|
||||
|
||||
To check it's working, push to `internal/knowledge` and look for
|
||||
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
|
||||
|
|
|
|||
|
|
@ -69,13 +69,13 @@ Two things live in the `agent-configs` Forgejo organization:
|
|||
allowlists are the `operators` team, and the agent can neither push
|
||||
`main` directly nor self-merge. An operator's merge in the Forgejo UI
|
||||
deploys the merged commit (see
|
||||
[config-changes.md § Config changes](../../2-handbook/agents/config-changes.md#config-changes)).
|
||||
[config-changes.md § Config changes](../../4-implementation/agent-state/config-changes.md#config-changes)).
|
||||
hive-c0re pushes nothing to this org and reads nothing from it.
|
||||
Repos stay private, so an agent can't read another
|
||||
agent's config. The agent's **persistent meta flake input** is this repo's `main` head,
|
||||
pinned by sha in `internal/meta` (`agent-<n>.url =
|
||||
git+<forge>/agent-configs/<n>.git?rev=<sha>`; see
|
||||
[config-changes.md § Meta flake](../../2-handbook/agents/config-changes.md#meta-flake)).
|
||||
[config-changes.md § Meta flake](../../4-implementation/agent-state/config-changes.md#meta-flake)).
|
||||
|
||||
The `hive-forge` CLI (separate workspace crate, see
|
||||
[`README.md`](../../README.md) file map) wraps the Forgejo REST API
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ root:root`, no symlink followed) and leaves an existing one alone. The
|
|||
container's own activation (`hive-agent-user-migrate`) then chowns it to
|
||||
the agent user and sets `0751`. Without a user namespace
|
||||
that uid is the host inode's owner. (A namespaced agent, see "User namespace" in
|
||||
`docs/2-handbook/agents/persistence.md`, gets the dir from `EnsureAgentSocketDir`
|
||||
`docs/4-implementation/agent-state/persistence.md`, gets the dir from `EnsureAgentSocketDir`
|
||||
owned by its data owner instead, because the container can't chown through
|
||||
the id map.) Don't add another host-side chown or chmod:
|
||||
two owners of one path revert each other. Until the container activates,
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ agents) runs:
|
|||
queued and unacked, so a resume drains the backlog instead of
|
||||
losing it; reminders and todo wakes buffer in their channels. Set
|
||||
it with `hivectl agent <name> pause` or the dashboard toggle; see
|
||||
[persistence](../../2-handbook/agents/persistence.md#-harnesspaused-per-agent).
|
||||
[persistence](../../4-implementation/agent-state/persistence.md#-harnesspaused-per-agent).
|
||||
1. Long-poll `Recv` on its socket. The host-side broker
|
||||
(`broker.rs::recv_blocking_batch`) returns immediately if there's
|
||||
a pending message, otherwise waits up to 30 s for a broker `Sent`
|
||||
|
|
@ -125,7 +125,7 @@ dial. Matrix / bash / forge-notify daemons and the in-process
|
|||
`disk_watch` todo producer (low state-disk space) are the built-in
|
||||
producers, but the socket accepts any `subsystem` marker — a
|
||||
user-configured MCP server can push its own todos the same way. See
|
||||
[`docs/2-handbook/agents/persistence.md`](../../2-handbook/agents/persistence.md#state-dirs-per-agent) for
|
||||
[`docs/4-implementation/agent-state/persistence.md`](../../4-implementation/agent-state/persistence.md#state-dirs-per-agent) for
|
||||
what each built-in todo producer watches and how the store + `get_loose_ends`
|
||||
merge work.
|
||||
|
||||
|
|
|
|||
|
|
@ -216,7 +216,7 @@ agent's durable bind-mounted dir. AGENTS.md autoload walks
|
|||
upward from `/state` — drop a per-agent AGENTS.md there if you
|
||||
want long-term hints that survive destroy/recreate. A `CLAUDE.md`
|
||||
in `/state` or above it shadows `AGENTS.md`: claude then loads only
|
||||
the `CLAUDE.md` (see `docs/2-handbook/agents/persistence.md` for the
|
||||
the `CLAUDE.md` (see `docs/4-implementation/agent-state/persistence.md` for the
|
||||
boot-time rename).
|
||||
|
||||
The wake prompt is intentionally minimal: the popped message's
|
||||
|
|
|
|||
|
|
@ -330,7 +330,7 @@ KEYS
|
|||
The file lives in the agent's bind-mounted **harness** dir, not `state/`
|
||||
— `harness/` survives container rebuilds exactly like `state/` does, but
|
||||
(unlike `state/`) is never bind-mounted into a parent agent's container
|
||||
(see `docs/2-handbook/agents/persistence.md`'s "Parent access to child state"), so an API
|
||||
(see `docs/4-implementation/agent-state/persistence.md`'s "Parent access to child state"), so an API
|
||||
key placed here is reachable by nothing but this agent and the host. The
|
||||
file doesn't stay root-owned, though: the container's activation chowns the
|
||||
whole harness dir to the agent, so after the next boot the file is agent-owned
|
||||
|
|
|
|||
|
|
@ -128,7 +128,7 @@ hive_name?, swarm_name?, matrix_accounts? }`. `matrix_accounts` is a
|
|||
[`docs/4-implementation/agent-tools/subagent.md`](../agent-tools/subagent.md).
|
||||
- **Config changes** — no permission: they go through a forge PR on
|
||||
`agent-configs/<name>` — see
|
||||
[`docs/2-handbook/agents/config-changes.md`](../../2-handbook/agents/config-changes.md).
|
||||
[`docs/4-implementation/agent-state/config-changes.md`](../../4-implementation/agent-state/config-changes.md).
|
||||
- **Web egress** (`web_tools`) — enables Claude's built-in `WebFetch`
|
||||
and `WebSearch` tools (not MCP tools; added directly to the
|
||||
`--allowedTools` list). Off by default.
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue