hyperhive/docs/persistence.md

# Persistence + retention

Where state lives, what survives what, and how it's bounded.

## Three sqlite databases

### `/var/lib/hyperhive/broker.sqlite` (host)

Six tables, all in one file — four queues plus the schedule
header/targets split:

- `messages` — every inter-agent / operator-bound message.
  `sender / recipient / body / sent_at / delivered_at / acked_at /
  in_reply_to`. `in_reply_to` links a reply to its parent row id;
  the dashboard and per-agent inbox render these as threaded rows.
- `reminders` — `mcp__hyperhive__remind` queue.
  `agent / message / file_path / due_at / created_at / sent_at /
  attempt_count / last_error`. `file_path` set when a body
  exceeded the inline soft-cap and got auto-spilled to a file
  under the agent's state dir; the worker delivers a short
  pointer instead. `attempt_count` / `last_error` accumulate
  on delivery-failed retries.
- `approvals` — the queue. `agent / kind (apply_commit | spawn |
  init_config | update_meta_inputs | schedule_prompt) /
  commit_ref / requested_at / status / resolved_at / note`.
- `operator_questions` — `ask` / `answer` queue (despite the
  table name, stores both operator-targeted + agent-to-agent
  questions since the `ask` rename).
  `asker / question / options_json / multi / asked_at /
  deadline_at (ttl) / answered_at / answer / target`. `target IS
  NULL` = operator path (dashboard); `target = '<agent>'` = peer
  Q&A (`HelperEvent::QuestionAsked` pushed into target's inbox,
  answered via `Answer` request). Migrated via `ALTER TABLE ADD
  COLUMN` against `pragma_table_info`.
- `scheduled_prompts` — recurring + one-shot prompt queue
  (closes #444). `owner / body / interval_seconds (NULL = one-shot) /
  next_fire_at_unix / created_at_unix / source ("operator" or
  "approval:<id>") / cancelled_at_unix / description`. `owner`
  drives cancel-permission checks (operator vs the submitting
  agent). Cancelled rows are tombstoned and reaped by the worker
  on its next pass.
- `scheduled_prompt_targets` — per-target state for each schedule.
  `schedule_id / target / cancelled_at_unix /
  last_fired_at_unix / last_result`. `ON DELETE CASCADE` from
  `scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
  per connection (set at open).

Retention:

- `Broker::vacuum_delivered` runs hourly via a tokio task in
  `hive-c0re::main`. Drops acked message rows older than 30 days
  (`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
  rows are always kept — the harness `ack_turn`s only after a
  successful turn, so an unacked row can still be requeued via
  `requeue_inflight` on a crash.
- Approvals and questions are kept indefinitely — both are
  audit trails. `actions::destroy` and answered questions stay
  visible to anything that queries by id.
- Reminder rows are kept after `sent_at` is set (audit trail);
  no automatic vacuum today.
- Scheduled prompts: one-shot rows are deleted on fire by the
  worker; recurring rows live until the operator cancels them
  (`cancel_schedule` MCP / dashboard ✗) which tombstones via
  `cancelled_at_unix`, then `reap_cancelled` drops the row on
  the next worker pass.

### `/state/hyperhive-events.sqlite` (per agent)

Lives inside each container's bind-mounted `/state/` dir (host
path: `/var/lib/hyperhive/agents/<name>/state/hyperhive-events.sqlite`).
One table:

- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
  harness emits during turn loop execution.

The harness writes; the host vacuums. `hive-c0re::events_vacuum`
runs hourly and sweeps every existing agent state dir, deleting
rows older than 7 days. Age-only — no row cap — so a chatty turn
doesn't lose history sooner than a quiet one; disk pressure on a
sustained burst is the cheaper problem to have. Centralising
retention on the host means a misbehaving harness can't disable
its own vacuum and agents don't need any cleanup wiring of their
own.

Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/state`
setups). On open failure the `Bus` falls back to no-store mode
rather than crashing the harness — events still broadcast over SSE,
just nothing persisted.

### `/state/hyperhive-turn-stats.sqlite` (per agent)

Per-turn analytics sink. One row per claude turn captures
identity (`model`, `wake_from`, `result_kind`), timing
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
cache_read / cache_creation token counts), behaviour
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
snapshot metrics (`open_threads_count`,
`open_reminders_count` — fetched via the same socket the harness
already uses for `GetOpenThreads` + `CountPendingReminders`).
Bin-loop helpers `build_row` + `record` land each row at
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
the turn loop continue.

No host-side vacuum yet — tracked as forge issue
[#10](http://localhost:3000/hyperhive/hyperhive/issues/10)
(target retention ~90 days, age-only sweep like events_vacuum).

### `/state/hyperhive-rate-limited` (per agent)

Sentinel file written by `Bus::emit_status("rate_limited")` when the
harness detects a 429 / rate-limit response from the Claude API, and
removed when the retry sleep expires (any subsequent status emit
clears it). The file's presence is checked by hive-c0re's
`container_view::is_rate_limited` on each `build_all` sweep (~10s) to
populate `ContainerView.rate_limited` for the dashboard. Survives a
harness restart (the Bus reads it back at boot and restores the flag),
so the badge remains accurate if hive-c0re restarts while the harness
is mid-sleep.

### `/state/hyperhive-model` (per agent)

Single-line text file holding the claude model name currently
selected for this agent (default `haiku` when absent). Written by
`Bus::set_model` whenever the operator flips it via `/model
<name>` in the web terminal. Read once at harness boot in
`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
Survives destroy/recreate, gone on `--purge`.

## State dirs (per agent)

Under `/var/lib/hyperhive/agents/<name>/`:

- `config/` — the proposed nix repo (manager-editable). Bind-mounted
  **read-only** to `/agents/<name>/config` inside the sub-agent's own
  container so the agent can inspect what defines it and request
  precise changes from the manager; RW into the manager via the
  `/agents` tree bind.
- `claude/` — claude OAuth credentials, bind-mounted RW to
  `/root/.claude` inside the container.
- `state/` — durable notes, the events.sqlite db, and the
  turn-stats sqlite db. Bind-mounted to `/agents/<name>/state`
  inside the container (the manager still uses the legacy
  `/state` mount point — same host path either way).

Under `/var/lib/hyperhive/applied/<name>/` — the hive-c0re-only
applied repo. Tracks `flake.nix` (module-only boilerplate; never
edited after first spawn) + `agent.nix` (the actual config; the
manager's edits land here via the approval flow) + any other
files the manager committed. `.git/` carries the proposal /
approved / building / deployed / failed / denied tag history.

Under `/var/lib/hyperhive/meta/` — the swarm-wide deploy flake.
Single repo for the whole host; `flake.nix` declares one input
per agent + one `nixosConfigurations.<n>` output per agent;
`flake.lock` is the canonical "what's deployed where." The git
log is the deploy audit trail (one commit per successful
deploy or hyperhive bump). Manager has this RO-mounted at
`/meta/`.

Marker file `/var/lib/hyperhive/.meta-migration-done` is
written by the startup migration after every container has
been repointed at `meta#<n>`. Removing it forces a re-run on
next hive-c0re start (idempotent — only the actual repoint
step would re-fire).

## Destroy vs purge

- `DESTR0Y` (default) — stops + removes the nspawn container,
  drops the systemd drop-in, fails any pending approvals. State
  dirs stay put; the agent appears in the dashboard's K3PT ST4T3
  section as a tombstone with `⊕ R3V1V3` and `PURG3` actions.
  `R3V1V3` queues a Spawn approval that reuses the kept state on
  approve (no re-login).
- `PURG3` (opt-in via the dashboard button or
  `hive-c0re destroy --purge <name>`) — DESTR0Y plus wipes
  `/var/lib/hyperhive/{agents,applied}/<name>/`. Config history,
  claude creds, /state/ notes, and the events db are all gone.
  No undo.

The manager is non-destroyable from both paths (declarative
container; would fight with the host's NixOS config).

## Run-time dirs

`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
Without that, every restart wipes bind sources and existing
containers can't be started.

- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
- `/run/hyperhive/manager/mcp.sock` — manager-privileged socket.
- `/run/hyperhive/agents/<name>/mcp.sock` — per-sub-agent socket
  (bind-mounted into the container as `/run/hive/mcp.sock`).

On startup, `Coordinator::register_agent` drops any prior socket
task before rebinding — idempotent so a hive-c0re restart followed
by `rebuild alice` recreates the agent's socket without a clean
reinstall.