309879dba0
iris's #718 scope: move substantive design context from `#` comment blocks in `nix/` to corresponding `docs/` files, leave short references in code. iris handed it back to me on #10114 since nix/ is my lane + #775 established the pattern. First pass — three highest-density blocks in harness-base.nix: 1. **First-boot agent-user migration** (~70 lines → `~20 lines code + short ref` in the activation script). Substantive prose moves to new `docs/persistence.md::First-boot agent-user migration (post-#658)` section explaining the 4 steps the script performs + the eventual removability of the marker-guarded body. 2. **nix-daemon `sandbox-fallback = true`** (10-line block → 5-line ref). New `docs/gotchas.md::Containerized nix-daemon needs sandbox-fallback = true` section covers the user-namespaces rationale + nixpkgs-default override. 3. **Matrix daemon + token-arrival trigger** (~50 lines across two systemd units → ~10 lines code + short refs). New `docs/persistence.md::Matrix per-agent daemon + token-arrival trigger` covers the socket-path rationale, the runtime-dir ownership story, and the first-boot ordering pattern. Net: harness-base.nix -84 lines, docs +74 lines. Substantive design context moves to durable docs; in-code refs follow iris's pattern from her #712 batches (`see docs/<file>::<section>`). Follow-ups: hive-c0re.nix, hive-forge.nix, hive-matrix.nix (already trimmed via #775 but a couple of remaining blocks could go), and the smaller files in #718's scope table. Shipping this first to get the pattern reviewed before larger batches. Verified: `nix eval` on agent-base toplevel still resolves.
263 lines
12 KiB
Markdown
263 lines
12 KiB
Markdown
# Persistence + retention
|
|
|
|
Where state lives, what survives what, and how it's bounded.
|
|
|
|
## Three sqlite databases
|
|
|
|
### `/var/lib/hyperhive/broker.sqlite` (host)
|
|
|
|
Six tables, all in one file — four queues plus the schedule
|
|
header/targets split:
|
|
|
|
- `messages` — every inter-agent / operator-bound message.
|
|
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
|
in_reply_to`. `in_reply_to` links a reply to its parent row id;
|
|
the dashboard and per-agent inbox render these as threaded rows.
|
|
- `reminders` — `mcp__hyperhive__remind` queue.
|
|
`agent / message / file_path / due_at / created_at / sent_at /
|
|
attempt_count / last_error`. `file_path` set when a body
|
|
exceeded the inline soft-cap and got auto-spilled to a file
|
|
under the agent's state dir; the worker delivers a short
|
|
pointer instead. `attempt_count` / `last_error` accumulate
|
|
on delivery-failed retries.
|
|
- `approvals` — the queue. `agent / kind (apply_commit | spawn |
|
|
init_config | update_meta_inputs | schedule_prompt) /
|
|
commit_ref / requested_at / status / resolved_at / note`.
|
|
- `operator_questions` — `ask` / `answer` queue (despite the
|
|
table name, stores both operator-targeted + agent-to-agent
|
|
questions since the `ask` rename).
|
|
`asker / question / options_json / multi / asked_at /
|
|
deadline_at (ttl) / answered_at / answer / target`. `target IS
|
|
NULL` = operator path (dashboard); `target = '<agent>'` = peer
|
|
Q&A (`HelperEvent::QuestionAsked` pushed into target's inbox,
|
|
answered via `Answer` request). Migrated via `ALTER TABLE ADD
|
|
COLUMN` against `pragma_table_info`.
|
|
- `scheduled_prompts` — recurring + one-shot prompt queue
|
|
(closes #444). `owner / body / interval_seconds (NULL = one-shot) /
|
|
next_fire_at_unix / created_at_unix / source ("operator" or
|
|
"approval:<id>") / cancelled_at_unix / description`. `owner`
|
|
drives cancel-permission checks (operator vs the submitting
|
|
agent). Cancelled rows are tombstoned and reaped by the worker
|
|
on its next pass.
|
|
- `scheduled_prompt_targets` — per-target state for each schedule.
|
|
`schedule_id / target / cancelled_at_unix /
|
|
last_fired_at_unix / last_result`. `ON DELETE CASCADE` from
|
|
`scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
|
|
per connection (set at open).
|
|
|
|
Retention:
|
|
|
|
- `Broker::vacuum_delivered` runs hourly via a tokio task in
|
|
`hive-c0re::main`. Drops acked message rows older than 30 days
|
|
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
|
|
rows are always kept — the harness `ack_turn`s only after a
|
|
successful turn, so an unacked row can still be requeued via
|
|
`requeue_inflight` on a crash.
|
|
- Approvals and questions are kept indefinitely — both are
|
|
audit trails. `actions::destroy` and answered questions stay
|
|
visible to anything that queries by id.
|
|
- Reminder rows are kept after `sent_at` is set (audit trail);
|
|
no automatic vacuum today.
|
|
- Scheduled prompts: one-shot rows are deleted on fire by the
|
|
worker; recurring rows live until the operator cancels them
|
|
(`cancel_schedule` MCP / dashboard ✗) which tombstones via
|
|
`cancelled_at_unix`, then `reap_cancelled` drops the row on
|
|
the next worker pass.
|
|
|
|
### `/state/hyperhive-events.sqlite` (per agent)
|
|
|
|
Lives inside each container's bind-mounted `/state/` dir (host
|
|
path: `/var/lib/hyperhive/agents/<name>/state/hyperhive-events.sqlite`).
|
|
One table:
|
|
|
|
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
|
|
harness emits during turn loop execution.
|
|
|
|
The harness writes; the host vacuums. `hive-c0re::events_vacuum`
|
|
runs hourly and sweeps every existing agent state dir, deleting
|
|
rows older than 7 days. Age-only — no row cap — so a chatty turn
|
|
doesn't lose history sooner than a quiet one; disk pressure on a
|
|
sustained burst is the cheaper problem to have. Centralising
|
|
retention on the host means a misbehaving harness can't disable
|
|
its own vacuum and agents don't need any cleanup wiring of their
|
|
own.
|
|
|
|
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/state`
|
|
setups). On open failure the `Bus` falls back to no-store mode
|
|
rather than crashing the harness — events still broadcast over SSE,
|
|
just nothing persisted.
|
|
|
|
### `/state/hyperhive-turn-stats.sqlite` (per agent)
|
|
|
|
Per-turn analytics sink. One row per claude turn captures
|
|
identity (`model`, `wake_from`, `result_kind`), timing
|
|
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
|
|
cache_read / cache_creation token counts), behaviour
|
|
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
|
|
snapshot metrics (`open_threads_count`,
|
|
`open_reminders_count` — fetched via the same socket the harness
|
|
already uses for `GetOpenThreads` + `CountPendingReminders`).
|
|
Bin-loop helpers `build_row` + `record` land each row at
|
|
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
|
|
the turn loop continue.
|
|
|
|
No host-side vacuum yet — tracked as forge issue
|
|
[#10](http://localhost:3000/hyperhive/hyperhive/issues/10)
|
|
(target retention ~90 days, age-only sweep like events_vacuum).
|
|
|
|
### `/state/hyperhive-rate-limited` (per agent)
|
|
|
|
Sentinel file written by `Bus::emit_status("rate_limited")` when the
|
|
harness detects a 429 / rate-limit response from the Claude API, and
|
|
removed when the retry sleep expires (any subsequent status emit
|
|
clears it). The file's presence is checked by hive-c0re's
|
|
`container_view::is_rate_limited` on each `build_all` sweep (~10s) to
|
|
populate `ContainerView.rate_limited` for the dashboard. Survives a
|
|
harness restart (the Bus reads it back at boot and restores the flag),
|
|
so the badge remains accurate if hive-c0re restarts while the harness
|
|
is mid-sleep.
|
|
|
|
### `/state/hyperhive-model` (per agent)
|
|
|
|
Single-line text file holding the claude model name currently
|
|
selected for this agent (default `haiku` when absent). Written by
|
|
`Bus::set_model` whenever the operator flips it via `/model
|
|
<name>` in the web terminal. Read once at harness boot in
|
|
`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
|
|
Survives destroy/recreate, gone on `--purge`.
|
|
|
|
## State dirs (per agent)
|
|
|
|
Under `/var/lib/hyperhive/agents/<name>/`:
|
|
|
|
- `config/` — the proposed nix repo (manager-editable). Bind-mounted
|
|
**read-only** to `/agents/<name>/config` inside the sub-agent's own
|
|
container so the agent can inspect what defines it and request
|
|
precise changes from the manager; RW into the manager via the
|
|
`/agents` tree bind.
|
|
- `claude/` — claude OAuth credentials, bind-mounted RW to
|
|
`/home/<name>/.claude` inside the container (post-#658 — was
|
|
`/root/.claude` pre-#658 when every harness ran as root).
|
|
- `state/` — durable notes, the events.sqlite db, and the
|
|
turn-stats sqlite db. Bind-mounted to `/agents/<name>/state`
|
|
inside the container (uniform for sub-agents + manager
|
|
post-#604). The `$HYPERHIVE_STATE_DIR` env var exposes the
|
|
same path to in-container scripts.
|
|
|
|
Under `/var/lib/hyperhive/applied/<name>/` — the hive-c0re-only
|
|
applied repo. Tracks `flake.nix` (module-only boilerplate; never
|
|
edited after first spawn) + `agent.nix` (the actual config; the
|
|
manager's edits land here via the approval flow) + any other
|
|
files the manager committed. `.git/` carries the proposal /
|
|
approved / building / deployed / failed / denied tag history.
|
|
|
|
Under `/var/lib/hyperhive/meta/` — the swarm-wide deploy flake.
|
|
Single repo for the whole host; `flake.nix` declares one input
|
|
per agent + one `nixosConfigurations.<n>` output per agent;
|
|
`flake.lock` is the canonical "what's deployed where." The git
|
|
log is the deploy audit trail (one commit per successful
|
|
deploy or hyperhive bump). Manager has this RO-mounted at
|
|
`/meta/`.
|
|
|
|
Marker file `/var/lib/hyperhive/.meta-migration-done` is
|
|
written by the startup migration after every container has
|
|
been repointed at `meta#<n>`. Removing it forces a re-run on
|
|
next hive-c0re start (idempotent — only the actual repoint
|
|
step would re-fire).
|
|
|
|
## Destroy vs purge
|
|
|
|
- `DESTR0Y` (default) — stops + removes the nspawn container,
|
|
drops the systemd drop-in, fails any pending approvals. State
|
|
dirs stay put; the agent appears in the dashboard's K3PT ST4T3
|
|
section as a tombstone with `⊕ R3V1V3` and `PURG3` actions.
|
|
`R3V1V3` queues a Spawn approval that reuses the kept state on
|
|
approve (no re-login).
|
|
- `PURG3` (opt-in via the dashboard button or
|
|
`hive-c0re destroy --purge <name>`) — DESTR0Y plus wipes
|
|
`/var/lib/hyperhive/{agents,applied}/<name>/`. Config history,
|
|
claude creds, /state/ notes, and the events db are all gone.
|
|
No undo.
|
|
|
|
The manager is non-destroyable from both paths (declarative
|
|
container; would fight with the host's NixOS config).
|
|
|
|
## Run-time dirs
|
|
|
|
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
|
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
|
|
Without that, every restart wipes bind sources and existing
|
|
containers can't be started.
|
|
|
|
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
|
|
- `/run/hyperhive/manager/mcp.sock` — manager-privileged socket.
|
|
- `/run/hyperhive/agents/<name>/mcp.sock` — per-sub-agent socket
|
|
(bind-mounted into the container as `/run/hive/mcp.sock`).
|
|
|
|
On startup, `Coordinator::register_agent` drops any prior socket
|
|
task before rebinding — idempotent so a hive-c0re restart followed
|
|
by `rebuild alice` recreates the agent's socket without a clean
|
|
reinstall.
|
|
|
|
## First-boot agent-user migration (post-#658)
|
|
|
|
Pre-#658 the harness ran as root inside the container. #658 dropped
|
|
to a per-agent unix user (`hyperhive.user.name`, defaults to the
|
|
agent's logical label so each container has a uniquely-named user).
|
|
The transition needs a one-time data shuffle so existing operators
|
|
who deployed pre-#658 don't lose their claude session.
|
|
|
|
`system.activationScripts.hive-agent-user-migrate` (in
|
|
`nix/templates/harness-base.nix`) runs on every activation,
|
|
marker-guarded so the substantive moves only happen once per
|
|
container lifetime:
|
|
|
|
1. **`${homeDir}` exists with the right ownership** — covers the
|
|
very first boot before `useradd`'s `createHome` has had a
|
|
chance to chown. Also re-applies on every rebuild in case the
|
|
meta-flake's per-agent name evolves (rare).
|
|
2. **Migrate any leftover `/root/.claude` content into
|
|
`${homeDir}/.claude`** — pre-#658 `claude` wrote to root's
|
|
empty home; the bind mount didn't exist yet. Marker
|
|
(`/var/lib/hive-agent-user-migrated`) guards single-shot.
|
|
`cp -an` (no-clobber) so any pre-existing files at the new
|
|
location win — never blow over data already there.
|
|
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
|
|
recursively so the new agent user can read/write it. Wildcard
|
|
matches the single agent that container sees; `-h` skips
|
|
symlinks the agent might have planted.
|
|
4. **Chown the `~/.claude/` bind-mount** recursively. Pre-#658
|
|
`claude` wrote `.credentials.json` 0600 root:root; post-#658
|
|
the harness reads `~/.claude/` as the agent user to decide
|
|
Online vs NeedsLogin in `login::has_session`. Without the
|
|
chown the existing credentials get silently treated as "no
|
|
session" and the operator re-prompts every boot.
|
|
|
|
The activation script will eventually become unnecessary once no
|
|
operators have pre-#658 state dirs left to migrate; drop the body
|
|
+ marker check at that point.
|
|
|
|
## Matrix per-agent daemon + token-arrival trigger
|
|
|
|
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
|
|
process per agent. Holds the unix socket the stdio
|
|
`hive-matrix-mcp` bridge talks to, emits hyperhive wake signals
|
|
on incoming room events via `/run/hive/mcp.sock`. Conditional on
|
|
`hyperhive.matrix.enable` (which both the daemon AND the
|
|
auto-injected `extraMcpServers.matrix` entry read).
|
|
|
|
Socket path lives inside the systemd-managed runtime dir
|
|
(`RuntimeDirectory = "hive-matrix"` → `/run/hive-matrix/`, owned by
|
|
the agent user) so the daemon can bind without needing root over
|
|
`/run/` itself. Both daemon + bridge agree on the path via the
|
|
`HIVE_MATRIX_SOCKET` env var.
|
|
|
|
**First-boot ordering**: hive-c0re provisions the matrix token AFTER
|
|
agent containers come up. Without the path-trigger sibling
|
|
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
|
|
/agents/*/state/matrix-token`), the daemon would exit 0 quietly the
|
|
first time it ran and the MCP would have no backend until the next
|
|
restart. The `.path` unit makes the appearance of the token re-fire
|
|
the service so the daemon comes alive in the same boot cycle as
|
|
provisioning. `matrix-avatar-sync.path` uses the same pattern for
|
|
the icon-upload oneshot (#571).
|