docs: restructure into topic subdirectories, collapse duplicated index
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
This commit is contained in:
iris
mara
parent
e4a22b4190
commit
07b62612b0
124 changed files with 301 additions and 377 deletions
|
|
@ -1,593 +0,0 @@
|
|||
# Persistence + retention
|
||||
|
||||
Where state lives, what survives what, and how it's bounded.
|
||||
|
||||
## For operators
|
||||
|
||||
The short answer to "will I lose anything": **destroying an agent
|
||||
keeps its state, purging it doesn't.**
|
||||
|
||||
- **`DESTR0Y`** (the default action) stops and removes the container
|
||||
but keeps everything on disk — config history, claude login, `/state/`
|
||||
notes, harness data. The agent shows up as a tombstone (K3PT ST4T3 on
|
||||
the C0R3 page) with a `⊕ R3V1V3` button that recreates it from the
|
||||
kept state, **no re-login needed**.
|
||||
- **`PURG3`** (opt-in, from the dashboard or `hivectl agent <name>
|
||||
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
|
||||
history, claude credentials, `/state/` notes, everything. **No
|
||||
undo.** Only reach for this when you actually want the agent gone
|
||||
for good.
|
||||
|
||||
Beyond that:
|
||||
|
||||
- **Approvals are kept forever** — they're an audit trail, not a
|
||||
cache. Nothing about them ever ages out.
|
||||
- **Broker messages**: acked ones vacuum after 30 days; anything
|
||||
undelivered or delivered-but-not-yet-acked is always kept, however
|
||||
old.
|
||||
- **An agent's own `/state/` notes and claude login survive every
|
||||
restart and rebuild** — only an explicit purge (or a hive
|
||||
`--purge`-style host operation, or the agent's own choices) touches
|
||||
them.
|
||||
- The **root/bootstrap agent is special**: it isn't really destroyable
|
||||
in practice — hive-c0re recreates it automatically on its next
|
||||
startup if it's ever gone.
|
||||
|
||||
Everything below this point is implementation detail: exact table
|
||||
schemas, file layouts, and internal migration mechanics.
|
||||
|
||||
## Sqlite databases
|
||||
|
||||
### `/var/lib/hyperhive/db/broker.sqlite` (host)
|
||||
|
||||
Seven tables, all in one file — three queues, a small key/value
|
||||
table, the schedule header/targets split, and the per-agent
|
||||
power-intent registry:
|
||||
|
||||
- `messages` — every inter-agent / operator-bound message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
in_reply_to / priority`. `in_reply_to` links a reply to its parent
|
||||
row id; the dashboard and per-agent inbox render these as threaded
|
||||
rows.
|
||||
- `kv` — small persistent key/value store (`key PK / value`) for
|
||||
host-side bookkeeping that doesn't warrant its own table.
|
||||
|
||||
⚠️ The `mcp__hyperhive__remind` queue is **not** here any more: it
|
||||
moved to a harness-local, per-agent store as part of the
|
||||
loose-ends-v2 migration — see [`/harness/` contents
|
||||
below](#state-dirs-per-agent) for where reminders (and todos)
|
||||
actually live now.
|
||||
- `approvals` — the queue. `agent / kind (merge_config_pr | spawn |
|
||||
init_config | update_meta_inputs | schedule_prompt) /
|
||||
commit_ref / requested_at / status / resolved_at / note`.
|
||||
- `scheduled_prompts` — recurring + one-shot prompt queue.
|
||||
`owner / body / interval_seconds (NULL = one-shot) /
|
||||
next_fire_at_unix / created_at_unix / source ("operator" or
|
||||
"approval:<id>") / cancelled_at_unix / description`. `owner`
|
||||
drives cancel-permission checks (operator vs the submitting
|
||||
agent). Cancelled rows are tombstoned and reaped by the worker
|
||||
on its next pass.
|
||||
- `scheduled_prompt_targets` — per-target state for each schedule.
|
||||
`schedule_id / target / cancelled_at_unix /
|
||||
last_fired_at_unix / last_result`. `ON DELETE CASCADE` from
|
||||
`scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
|
||||
per connection (set at open).
|
||||
- `agent_power` — one tiny row per agent: `agent PK / wanted (up |
|
||||
offline) / updated_at`, owned by `hive-c0re/src/stores/power.rs`.
|
||||
This is the durable power *intent* the job queue reconciles the
|
||||
observed container state against; intent survives hive-c0re
|
||||
restarts even though in-flight queue work doesn't. See
|
||||
[`docs/coordinator.md`'s Desired-state
|
||||
section](coordinator.md#desired-state-spec-vs-status) for who
|
||||
writes and reads it and how reconciliation works.
|
||||
|
||||
Retention:
|
||||
|
||||
- `Broker::vacuum_delivered` runs hourly via a tokio task in
|
||||
`hive-c0re::main`. Drops acked message rows older than 30 days
|
||||
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
|
||||
rows are always kept — the harness `ack_turn`s only after a
|
||||
successful turn, so an unacked row can still be requeued via
|
||||
`requeue_inflight` on a crash.
|
||||
- Approvals are kept indefinitely — an audit trail. `actions::destroy`
|
||||
rows stay visible to anything that queries by id.
|
||||
- Scheduled prompts: one-shot rows are deleted on fire by the
|
||||
worker; recurring rows live until the operator cancels them
|
||||
(`cancel_schedule` MCP / dashboard ✗) which tombstones via
|
||||
`cancelled_at_unix`, then `reap_cancelled` drops the row on
|
||||
the next worker pass.
|
||||
- `agent_power` rows live until the agent is destroyed (one row per
|
||||
agent — nothing to vacuum).
|
||||
|
||||
### `/harness/hyperhive-events.sqlite` (per agent)
|
||||
|
||||
Lives inside each container's bind-mounted `/harness/` dir (host
|
||||
path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
|
||||
One table:
|
||||
|
||||
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
|
||||
harness emits during turn loop execution.
|
||||
|
||||
The harness both writes and vacuums it — this used to be a host-side
|
||||
sweep, but hive-c0re runs as the unprivileged `hive-core` user under
|
||||
privsep and can't delete agent-owned files (host-side deletes hit
|
||||
`PermissionDenied` on the bash-task trio and a readonly-database error
|
||||
here), so cleanup moved in-container. `hive-agent`'s `vacuum::run`
|
||||
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
|
||||
**type-scoped**: it deletes only the verbose `stream` rows (the raw
|
||||
claude `stream-json` deltas — one per text chunk / tool use, the bulk
|
||||
of the file's size) older than 14 days, and keeps every other kind
|
||||
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
|
||||
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
|
||||
small and carry the semantic per-turn history the operator scrolls
|
||||
back through when debugging a regression. Age-only within the
|
||||
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
|
||||
history sooner than a quiet one. The trade-off (accepted): a
|
||||
misbehaving harness could now skip its own cleanup, which the old
|
||||
host-side sweep was meant to prevent — but a compromised harness is
|
||||
already inside the container trust boundary
|
||||
([`docs/security.md`](security.md)), and these are ephemeral local
|
||||
artifacts, so cleaning them up where they live is the honest fix.
|
||||
|
||||
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
|
||||
setups). On open failure the `Bus` falls back to no-store mode
|
||||
rather than crashing the harness — events still broadcast over SSE,
|
||||
just nothing persisted.
|
||||
|
||||
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
|
||||
|
||||
Per-turn analytics sink. One row per claude turn captures
|
||||
identity (`model`, `wake_from`, `result_kind`), timing
|
||||
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
|
||||
cache_read / cache_creation token counts), behaviour
|
||||
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
|
||||
snapshot metrics (`open_threads_count`,
|
||||
`open_reminders_count` — fetched via the same socket the harness
|
||||
already uses for `GetOpenThreads` + `CountPendingReminders`).
|
||||
Bin-loop helpers `build_row` + `record` land each row at
|
||||
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
|
||||
the turn loop continue.
|
||||
|
||||
A sibling `bash_commands(ts INTEGER, head TEXT)` table in the same
|
||||
file is written by the `hive-bash-daemon` (not the harness): one
|
||||
row per executed bash task recording the normalised command head -
|
||||
the basename of the first real command, looking past `cd repo &&`
|
||||
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
|
||||
backs the "favorite tools" view on the /stats page (aggregated
|
||||
host-side). Best-effort and created on first write
|
||||
(`CREATE TABLE IF NOT EXISTS`), so it's simply absent until a bash
|
||||
task runs.
|
||||
|
||||
turn-stats.sqlite has **no vacuum** — it's one tiny row per turn
|
||||
(~hundreds of KB even over months), read directly by the `/stats` page
|
||||
and the hive-wide stats view, so pruning it would only lose trend
|
||||
history for no space gain.
|
||||
|
||||
### `/state/hyperhive-harness.json` (per agent)
|
||||
|
||||
Consolidated harness state file written atomically (`.tmp` + rename) by
|
||||
`Bus::emit_status` whenever rate-limited or login-failed flags change.
|
||||
Shape:
|
||||
|
||||
```json
|
||||
{ "rate_limited": false, "needs_login": false, "active_model": "…" }
|
||||
```
|
||||
|
||||
- `rate_limited` — set when the harness detects a 429 from the Claude
|
||||
API; cleared by any subsequent status emit. Drives
|
||||
`ContainerView.rate_limited` on the dashboard.
|
||||
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
|
||||
cleared by `"online"` status (re-auth completed). Drives the
|
||||
`needs_login` flag alongside the `claude_has_session` check.
|
||||
- `active_model` — the resolved Claude model for the dashboard badge.
|
||||
|
||||
The turn loop is the only writer today, but it still goes
|
||||
read-modify-write under a shared in-process lock and merges into the
|
||||
existing object rather than reconstructing it — so a second writer
|
||||
would preserve fields it doesn't own, and the lock closes the
|
||||
lost-update window between a writer's read and its rename. The lock
|
||||
is in-process only, so it wouldn't serialise a writer running as a
|
||||
separate process; none of today's writers are.
|
||||
|
||||
hive-c0re reads this file on each `build_all` sweep (~10s) via
|
||||
`container_view::read_harness_flags`. Falls back to the legacy individual
|
||||
sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) if the
|
||||
JSON is absent, so existing containers keep working through the transition
|
||||
window before their next rebuild.
|
||||
|
||||
### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
|
||||
|
||||
Full stdout + stderr capture for every `nixos-container` / `nix
|
||||
build` invocation the lifecycle layer fires. One row per invocation;
|
||||
the row accumulates lines as the child runs.
|
||||
|
||||
Capturing the full stream (rather than a short tail buffer) matters
|
||||
because real eval errors routinely run long — "tried alternatives"
|
||||
blocks alone are often 30+ lines — so a truncated tail would cut off
|
||||
the actual failure and leave only the host journal holding the
|
||||
complete output. With this table the dashboard can surface the entire
|
||||
log.
|
||||
|
||||
Three indices:
|
||||
- `(agent, started_at)` — backs the per-agent latest-N lookup used
|
||||
by the agent card chip.
|
||||
- `(status, finished_at)` — backs the retention sweep that runs
|
||||
as part of the existing hourly vacuum.
|
||||
- `(node_id)` — added by a later migration so a build log row can be
|
||||
looked up by the job-queue node it belongs to (a `hive_jobq` node is
|
||||
immutable after insert, so the link is recorded on the log row
|
||||
instead); legacy rows predating the column keep `node_id IS NULL`.
|
||||
|
||||
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
|
||||
log a warning on sqlite error and let the build continue. A failed
|
||||
log row never blocks a rebuild.
|
||||
|
||||
### `/harness/hyperhive-model` (per agent)
|
||||
|
||||
Single-line text file holding the claude model name currently
|
||||
selected for this agent (default `haiku` when absent). Written by
|
||||
`Bus::set_model` whenever the operator flips it via `/model
|
||||
<name>` in the web terminal. Read once at harness boot in
|
||||
`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
|
||||
Survives destroy/recreate, gone on `--purge`.
|
||||
|
||||
### `/harness/paused` (per agent)
|
||||
|
||||
Empty marker file. Its presence parks the agent's turn loop: the
|
||||
harness keeps serving its web UI and MCP daemons but drives no turns,
|
||||
and inbox messages queue unacked until it's removed (see
|
||||
[turn loop](turn-loop/README.md#the-loop)).
|
||||
|
||||
Unusually, it's read and written from **both** sides of the harness
|
||||
bind-mount, and that's the whole design: the harness stats it
|
||||
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
|
||||
stats it on the host (`Coordinator::is_paused`) to populate the
|
||||
`paused` field on the agent card, and creates/removes it
|
||||
(`Coordinator::set_paused`) for `hivectl agent <name> pause|resume` and the
|
||||
dashboard toggle. Because the file itself is the only shared state
|
||||
there's no protocol between them, no round-trip into the container, and
|
||||
pause keeps working when the harness is wedged or the container is
|
||||
stopped.
|
||||
|
||||
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
||||
is the agent's own space to fill, and this is harness control state.
|
||||
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
||||
back paused after a restart, which is the intended behaviour rather
|
||||
than an accident of storage.
|
||||
|
||||
## State dirs (per agent)
|
||||
|
||||
Under `/var/lib/hyperhive/agents/<name>/`:
|
||||
|
||||
- `config/` — the proposed nix repo (root-agent-editable). Bind-mounted
|
||||
**read-only** to `/agents/<name>/config` inside the sub-agent's own
|
||||
container so the agent can inspect what defines it and request
|
||||
precise changes from the root agent; RW into the root agent via the
|
||||
`/agents` tree bind.
|
||||
- `claude/` — claude OAuth credentials, bind-mounted RW to
|
||||
`/home/<name>/.claude` inside the container.
|
||||
- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
|
||||
to `/agents/<name>/state` inside the container (uniform for
|
||||
all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
|
||||
the same path to in-container scripts. Notable files written here
|
||||
by the harness:
|
||||
- `hyperhive-status` — single-line free-text status string written
|
||||
by `set_status`; cleared on explicit `set_status("")`. Read by
|
||||
hive-c0re and the per-agent `/api/dashboard-state` endpoint to
|
||||
surface the status chip on the dashboard. Absent when no status
|
||||
is set.
|
||||
- `hyperhive-harness.json` — rate-limited / needs-login flags read
|
||||
by the dashboard's async container-state fetch. See
|
||||
`docs/web-ui/dashboard.md::Container row`.
|
||||
- `harness/` — harness-internal ephemeral state; not intended for
|
||||
agent consumption. Bind-mounted to `/agents/<name>/harness`
|
||||
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
|
||||
- `bash-tasks/` — task JSON + stdout/stderr files for
|
||||
background `mcp__bash__run` jobs. JSON files are
|
||||
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
|
||||
(full captured output). The harness's own hourly sweep
|
||||
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
|
||||
event rows above) deletes terminal task trios older than 48
|
||||
hours; non-terminal (still-running) tasks are never deleted. This
|
||||
used to be a host-side `hive-c0re` vacuum, moved in-container for
|
||||
the same privsep-ownership reason as the events vacuum above.
|
||||
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
|
||||
and reminders, one small table each in a single file (in-container
|
||||
daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
|
||||
`hive-forge-notify` — upsert keyed todos here over the harness's
|
||||
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
|
||||
`get_loose_ends` output and clears a row on `mark_todo_done`).
|
||||
Replaces three formerly-separate files
|
||||
(`hyperhive-todos.sqlite`, `hyperhive-reminders.sqlite`, and the
|
||||
old file-based `mcp-loose-ends/` scanner before that) — a one-time
|
||||
boot migration (`db_migrate::run`) folds the legacy files into this
|
||||
path the first time a harness boots after the upgrade. Also backs
|
||||
the `mcp__hyperhive__remind` queue, which moved from a host-side
|
||||
`broker.sqlite` table to this per-agent store as part of the same
|
||||
migration.
|
||||
|
||||
The harness itself is also a producer, not just the socket server:
|
||||
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
|
||||
socket the out-of-process daemons above dial) alongside
|
||||
`disk_watch::run` — an *in-process* todo producer that shares the
|
||||
store + wake `Notify` directly rather than dialling its own socket.
|
||||
`disk_watch` raises a keyed `disk` todo when the filesystem backing
|
||||
this agent's state gets tight, naming the agent's own biggest
|
||||
directories; the summary is bucketed and carries no raw byte
|
||||
counts, so an unchanged situation re-upserts as `changed == false`
|
||||
and never re-wakes.
|
||||
|
||||
Retention, same hourly `hive-agent::vacuum::run` sweep as
|
||||
`hyperhive-events.sqlite` below: delivered (soft-deleted) reminder
|
||||
rows are reaped 14 days after delivery, kept that long only to serve
|
||||
the trailing-window `ReminderRollup` stats; acked todo rows are
|
||||
reaped 30 days after acking (long enough that only a genuinely quiet
|
||||
month triggers the "one spurious re-announcement" fallback a
|
||||
reconciled producer like `disk_watch` relies on — see `todos.rs`'s
|
||||
module doc). Un-acked todos and undelivered reminders are never
|
||||
swept — same "audit trail, not cache" treatment as the c0re-side
|
||||
tables above.
|
||||
- `hyperhive-events.sqlite` — turn-loop event log.
|
||||
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
|
||||
- `hyperhive-model` — single-line model name override file.
|
||||
|
||||
### Parent access to child state
|
||||
|
||||
A parent agent gets each direct child's `state` dir bind-mounted
|
||||
**read-write** and its `config` dir **read-only**
|
||||
(`bind_child_agent_dirs` in `lifecycle/host_config.rs`). The RW on
|
||||
`state` is deliberate, not an oversight: a parent manages its children,
|
||||
which includes writing into a child's state for recovery (e.g. seeding
|
||||
notes, clearing a stuck sentinel) as well as reading it.
|
||||
|
||||
**`harness` is not mounted at all.** It holds the child's own runtime
|
||||
material — `bash-tasks/`, the turn-stats and event sqlite dbs — and
|
||||
nothing argues for a parent reading it, let alone writing it. hive-c0re
|
||||
reads a child's harness dir **directly on the host** when it wants
|
||||
those stats, which needs no mount into the parent.
|
||||
|
||||
**`config` is read-only, including for the parent.** A config change is
|
||||
a PR on the child's config repo, made from a clone and merged after
|
||||
review — so the bind-mounted `config` dir is a *copy to read*, never a
|
||||
tree anyone edits in place. Mounting it writable would leave a second
|
||||
path to the same file that skips the review entirely, which makes the
|
||||
boundary a convention rather than a permission.
|
||||
|
||||
⚠️ Not to be confused with the seeding done when an `InitConfig`
|
||||
approval resolves: that writes the child's initial config repo as
|
||||
**hive-c0re, against the host path**, and `read_only` on a bind
|
||||
constrains writers *inside* a container only. The two are unrelated —
|
||||
conflating them is an easy way to reason your way into thinking this
|
||||
mount should be writable when it shouldn't.
|
||||
|
||||
Per-child isolation still holds: a container only ever has its *own*
|
||||
dirs plus its direct children's bind-mounted, never a sibling's.
|
||||
|
||||
Under `/var/lib/hyperhive/applied/<name>/` — the hive-c0re-only
|
||||
applied repo. Tracks `flake.nix` (module-only boilerplate; never
|
||||
edited after first spawn) + `agent.nix` (the actual config; the
|
||||
root agent's edits land here via the approval flow) + any other
|
||||
files committed via the approval flow. `.git/` carries the proposal /
|
||||
approved / building / deployed / failed / denied tag history.
|
||||
|
||||
Under `/var/lib/hyperhive/meta/` — the swarm-wide deploy flake plus
|
||||
system-level config files. Single git repo for the whole host; every
|
||||
hive-c0re mutation that should survive a restart is committed here.
|
||||
Contents:
|
||||
|
||||
- `flake.nix` — declares one `nixpkgs` input per agent + one
|
||||
`nixosConfigurations.<n>` output per agent. `flake.lock` is the
|
||||
canonical "what's deployed where." The git log is the deploy
|
||||
audit trail (one commit per successful deploy or hyperhive bump).
|
||||
- `topology.json` — parent/child agent graph
|
||||
(`{ "alice": "root", "bob": "alice", "root": null }`).
|
||||
Written by `topology::apply_set_parent` (the pure move-validating
|
||||
transform) via `meta::bulk_commit_topology` (the committer — see the
|
||||
`Reparent` node in [`docs/coordinator.md`](coordinator.md)); read by
|
||||
the dashboard, the renderer, and `<parent>` / `<children>` recipient
|
||||
resolution.
|
||||
- `tool-groups.json` — per-agent MCP tool group grants
|
||||
(`{ "alice": ["messaging", "inbox", "execution"] }`). Written by
|
||||
`tool_groups::set_groups`; injected as `HIVE_TOOL_GROUPS` env
|
||||
var into each agent's container.
|
||||
- `capabilities.json` — per-agent capability grants
|
||||
(`{ "atlas": ["read_host_journal"] }`). Written by
|
||||
`capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env
|
||||
var. Absent agents have no extra capabilities.
|
||||
- `resource-limits.json` — per-agent container resource overrides
|
||||
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
||||
Written by `resource_limits::set_limits`; read where the systemd
|
||||
drop-in is generated (`lifecycle::write_dropins`), **not** injected
|
||||
into the container — these are host-side caps on the container, so
|
||||
the capped party never sees or sets them. Fallback is per *field*:
|
||||
an absent file, absent agent, or absent field falls back to the
|
||||
hive-wide `services.hyperhive.agentCpuQuota` / `agentMemoryMax`,
|
||||
so an agent can override only its memory and still track the hive
|
||||
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
|
||||
drop-in have **no** per-agent override — they are hive-wide only and
|
||||
come straight off `HiveEnv`, so this file has no field for them.
|
||||
|
||||
The root agent has the meta dir RO-mounted at `/meta/`.
|
||||
|
||||
There is no longer a `.meta-migration-done` marker: the
|
||||
one-shot container repoint it guarded has been removed, since
|
||||
containers are rendered onto `meta#<n>` at creation. A stale
|
||||
marker file left over from an older hive is inert and can be
|
||||
deleted.
|
||||
|
||||
## Destroy vs purge
|
||||
|
||||
See [For operators](#for-operators) above for what each action does to
|
||||
an agent's state. The mechanics, for completeness:
|
||||
|
||||
- `DESTR0Y` also drops the systemd drop-in and fails any pending
|
||||
approvals; the tombstone's `⊕ R3V1V3` button queues a Spawn approval
|
||||
that reuses the kept state on approve.
|
||||
- `PURG3` wipes `/var/lib/hyperhive/{agents,applied}/<name>/` — the
|
||||
union of everything `DESTR0Y` left behind.
|
||||
|
||||
The root/bootstrap agent's specialness is implemented as a soft policy
|
||||
guard in `actions::destroy` that refuses to destroy it, backstopped by
|
||||
`auto_update::ensure_root_agent`, which recreates it on the next
|
||||
hive-c0re startup if it's ever absent (bypassing the approval queue,
|
||||
as required infrastructure) — so even without the guard, destroying it
|
||||
would only be transient.
|
||||
|
||||
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
|
||||
|
||||
On a btrfs host, a brand-new agent's state root is created as a
|
||||
**btrfs subvolume** instead of a plain directory (progressive
|
||||
enhancement — see the #1762 lane). This is a no-op fallback on
|
||||
non-btrfs hosts and for any agent whose root already exists, so
|
||||
nothing is auto-migrated: existing agents keep their plain dirs
|
||||
until an explicit opt-in upgrade.
|
||||
|
||||
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
|
||||
the per-agent subdirs are created (spawn / rebuild / InitConfig).
|
||||
It skips the work when the root already exists; otherwise it asks
|
||||
hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
|
||||
root when the FS is btrfs (`statfs` magic gate) and chown it to the
|
||||
`hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
|
||||
succeed inside it.
|
||||
- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
|
||||
reuses it untouched.
|
||||
- **PURG3 deletes it correctly:** a subvolume root can't be removed
|
||||
with `rmdir`/`remove_dir_all`, so purge first calls hive-priv
|
||||
(`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
|
||||
actually a subvolume, then the normal `remove_dir_all` sweep covers
|
||||
plain-dir agents + the applied dir.
|
||||
|
||||
Per-subvolume disk-usage accounting and optional quotas have since
|
||||
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
|
||||
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
|
||||
`hivectl agent <name> quota show|set` reads/limits one agent's
|
||||
subvolume usage through the same hive-priv-mediated path as
|
||||
subvolume creation/deletion above.
|
||||
|
||||
This is the same subvolume `hivectl agent <name> subvol snapshot push`
|
||||
sends to the swarm's snapshot store — see
|
||||
[`docs/snapshot-store.md`](snapshot-store.md) for what a pushed
|
||||
snapshot contains and how the store authenticates a sender.
|
||||
|
||||
## `/var/lib/swarm-controller/` (swarm-controller host only)
|
||||
|
||||
Only present on the one host running
|
||||
`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
|
||||
so it survives restarts and redeploys.
|
||||
|
||||
- `webhook-secret` — the HMAC key the swarm's forge webhooks are signed
|
||||
with. **Keep it.** It is handed to Forgejo when a hook is registered,
|
||||
so replacing the file means every subsequent delivery fails
|
||||
verification until the hook is re-registered with the new value. It is
|
||||
generated automatically on first start; there is nothing to configure.
|
||||
|
||||
If the file is unreadable at startup the daemon still starts and logs
|
||||
`webhook secret unavailable`; the webhook endpoint then answers 503
|
||||
rather than accepting deliveries it cannot verify. Everything else the
|
||||
controller serves is unaffected.
|
||||
|
||||
## Run-time dirs
|
||||
|
||||
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
||||
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
|
||||
Without that, every restart wipes bind sources and existing
|
||||
containers can't be started.
|
||||
|
||||
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
|
||||
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
|
||||
(bind-mounted into the container as `/run/hive/mcp.sock`).
|
||||
|
||||
On startup, `Coordinator::register_agent` drops any prior socket
|
||||
task before rebinding — idempotent so a hive-c0re restart followed
|
||||
by `rebuild alice` recreates the agent's socket without a clean
|
||||
reinstall.
|
||||
|
||||
## First-boot agent-user migration
|
||||
|
||||
The harness runs as a per-agent unix user inside the container
|
||||
(`hyperhive.user.name`, defaults to the agent's logical label so each
|
||||
container has a uniquely-named user). Operators with legacy root-owned
|
||||
state dirs need a one-time data shuffle so they don't lose their claude
|
||||
session.
|
||||
|
||||
`system.activationScripts.hive-agent-user-migrate` (in
|
||||
`nix/agent-modules/user.nix`) runs on every activation,
|
||||
marker-guarded so the substantive moves only happen once per
|
||||
container lifetime:
|
||||
|
||||
1. **`${homeDir}` exists with the right ownership** — covers the
|
||||
very first boot before `useradd`'s `createHome` has had a
|
||||
chance to chown. Also re-applies on every rebuild in case the
|
||||
meta-flake's per-agent name evolves (rare).
|
||||
2. **Migrate any leftover `/root/.claude` content into
|
||||
`${homeDir}/.claude`** — legacy `claude` wrote to root's
|
||||
empty home; the bind mount didn't exist yet. Marker
|
||||
(`/var/lib/hive-agent-user-migrated`) guards single-shot.
|
||||
`cp -an` (no-clobber) so any pre-existing files at the new
|
||||
location win — never blow over data already there.
|
||||
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
|
||||
recursively so the agent user can read/write it. Wildcard
|
||||
matches the single agent that container sees; `-h` skips
|
||||
symlinks the agent might have planted.
|
||||
4. **Chown the `~/.claude/` bind-mount** recursively. Legacy
|
||||
`claude` wrote `.credentials.json` 0600 root:root; the
|
||||
current harness reads `~/.claude/` as the agent user to decide
|
||||
Online vs NeedsLogin in `login::has_session`. Without the
|
||||
chown the existing credentials get silently treated as "no
|
||||
session" and the operator re-prompts every boot.
|
||||
|
||||
The activation script will eventually become unnecessary once no
|
||||
operators have legacy root-owned state dirs left to migrate; drop
|
||||
the body + marker check at that point.
|
||||
|
||||
## Matrix per-agent daemon + token-arrival trigger
|
||||
|
||||
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
|
||||
process per agent. Serves its MCP tools directly over
|
||||
streamable-http (`hyperhive.mcp.matrixHttpPort`, no stdio bridge —
|
||||
same shape as `hive-bash-daemon`), emits hyperhive wake signals
|
||||
on incoming room events via `/run/hive/mcp.sock`. Conditional on
|
||||
`hyperhive.matrix.enable` (which both the daemon AND the
|
||||
auto-injected `extraMcpServers.matrix` entry read).
|
||||
|
||||
**First-boot ordering**: hive-c0re provisions the matrix token AFTER
|
||||
agent containers come up. Without the path-trigger sibling
|
||||
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
|
||||
/agents/*/state/matrix-token*` — the trailing `*` also catches a
|
||||
secondary multi-account token like `matrix-token-ccc`), the daemon
|
||||
would exit 0 quietly the first time it ran and the MCP would have no
|
||||
backend until the next restart. The `.path` unit makes the appearance
|
||||
of the token re-fire the service so the daemon comes alive in the
|
||||
same boot cycle as
|
||||
provisioning. The same token watcher also drives avatar setting: on a
|
||||
restart the daemon re-runs each account's bring-up, which sets the
|
||||
avatar (see below).
|
||||
|
||||
### matrix avatar (set by the daemon over the live Client)
|
||||
|
||||
The agent icon (`hyperhive.icon`, an SVG) is published as each matrix
|
||||
account's profile avatar by `hive-matrix-daemon` itself
|
||||
(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
|
||||
the daemon builds + restores an account's `Client` (authenticated,
|
||||
pointed at that account's resolved homeserver), it calls matrix-sdk's
|
||||
`account().upload_avatar()` — one call that uploads the media and sets
|
||||
`avatar_url`. Because it reuses the live Client, there is no hardcoded
|
||||
homeserver URL, no token re-read, and no token-file globbing: the daemon
|
||||
already iterates every configured + dashboard-discovered account in its
|
||||
bring-up loop, so the avatar is set for **every** account.
|
||||
|
||||
Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
|
||||
librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
|
||||
unit, gated on `hyperhive.icon != null`. No icon configured → the env is
|
||||
unset → `sync_avatar` returns early and no avatar is set.
|
||||
|
||||
Idempotency is **per-account**: an `avatar-icon-hash` file in each
|
||||
account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
|
||||
skips the upload when unchanged, because every upload mints a fresh
|
||||
`mxc://` URI that emits a profile state event in every joined room —
|
||||
re-uploading identical bytes is timeline spam. A dashboard-provisioned
|
||||
account gets its avatar when the `systemd.paths.hive-matrix-daemon` token
|
||||
watcher restarts the daemon (which re-runs the per-account bring-up), so
|
||||
no separate avatar trigger is needed. Avatar failures are swallowed
|
||||
(logged, non-fatal) so they never break account bring-up or sync.
|
||||
|
||||
Loading…
Reference in a new issue