hyperhive/docs/persistence.md

Persistence + retention

Where state lives, what survives what, and how it's bounded.

Three sqlite databases

/var/lib/hyperhive/broker.sqlite (host)

Six tables, all in one file — four queues plus the schedule header/targets split:

• messages — every inter-agent / operator-bound message. sender / recipient / body / sent_at / delivered_at / acked_at / in_reply_to. in_reply_to links a reply to its parent row id; the dashboard and per-agent inbox render these as threaded rows.
• reminders — mcp__hyperhive__remind queue. agent / message / file_path / due_at / created_at / sent_at / attempt_count / last_error. file_path set when a body exceeded the inline soft-cap and got auto-spilled to a file under the agent's state dir; the worker delivers a short pointer instead. attempt_count / last_error accumulate on delivery-failed retries.
• approvals — the queue. agent / kind (apply_commit | spawn | init_config | update_meta_inputs | schedule_prompt) / commit_ref / requested_at / status / resolved_at / note.
• operator_questions — ask / answer queue (despite the table name, stores both operator-targeted + agent-to-agent questions since the ask rename). asker / question / options_json / multi / asked_at / deadline_at (ttl) / answered_at / answer / target. target IS NULL = operator path (dashboard); target = '<agent>' = peer Q&A (HelperEvent::QuestionAsked pushed into target's inbox, answered via Answer request). Migrated via ALTER TABLE ADD COLUMN against pragma_table_info.
• scheduled_prompts — recurring + one-shot prompt queue. owner / body / interval_seconds (NULL = one-shot) / next_fire_at_unix / created_at_unix / source ("operator" or "approval:<id>") / cancelled_at_unix / description. owner drives cancel-permission checks (operator vs the submitting agent). Cancelled rows are tombstoned and reaped by the worker on its next pass.
• scheduled_prompt_targets — per-target state for each schedule. schedule_id / target / cancelled_at_unix / last_fired_at_unix / last_result. ON DELETE CASCADE from scheduled_prompts(id) — requires PRAGMA foreign_keys = ON per connection (set at open).

Retention:

• Broker::vacuum_delivered runs hourly via a tokio task in hive-c0re::main. Drops acked message rows older than 30 days (acked_at IS NOT NULL). Undelivered + delivered-but-not-acked rows are always kept — the harness ack_turns only after a successful turn, so an unacked row can still be requeued via requeue_inflight on a crash.
• Approvals and questions are kept indefinitely — both are audit trails. actions::destroy and answered questions stay visible to anything that queries by id.
• Reminder rows are kept after sent_at is set (audit trail); no automatic vacuum today.
• Scheduled prompts: one-shot rows are deleted on fire by the worker; recurring rows live until the operator cancels them (cancel_schedule MCP / dashboard ✗) which tombstones via cancelled_at_unix, then reap_cancelled drops the row on the next worker pass.

/state/hyperhive-events.sqlite (per agent)

Lives inside each container's bind-mounted /state/ dir (host path: /var/lib/hyperhive/agents/<name>/state/hyperhive-events.sqlite). One table:

• events(id, ts, kind, payload_json) — every LiveEvent the harness emits during turn loop execution.

The harness writes; the host vacuums. hive-c0re::events_vacuum runs hourly and sweeps every existing agent state dir, deleting rows older than 7 days. Age-only — no row cap — so a chatty turn doesn't lose history sooner than a quiet one; disk pressure on a sustained burst is the cheaper problem to have. Centralising retention on the host means a misbehaving harness can't disable its own vacuum and agents don't need any cleanup wiring of their own.

Path overridable via HYPERHIVE_EVENTS_DB (for dev / no-/state setups). On open failure the Bus falls back to no-store mode rather than crashing the harness — events still broadcast over SSE, just nothing persisted.

/state/hyperhive-turn-stats.sqlite (per agent)

Per-turn analytics sink. One row per claude turn captures identity (model, wake_from, result_kind), timing (started_at, ended_at, duration_ms), cost (input / output / cache_read / cache_creation token counts), behaviour (tool_call_count + tool_call_breakdown_json), and post-turn snapshot metrics (open_threads_count, open_reminders_count — fetched via the same socket the harness already uses for GetOpenThreads + CountPendingReminders). Bin-loop helpers build_row + record land each row at turn_end; writes are best-effort, a sqlite hiccup logs + lets the turn loop continue.

No host-side vacuum yet — tracked separately. Target retention ~90 days, age-only sweep like events_vacuum.

/state/hyperhive-rate-limited (per agent)

Sentinel file written by Bus::emit_status("rate_limited") when the harness detects a 429 / rate-limit response from the Claude API, and removed when the retry sleep expires (any subsequent status emit clears it). The file's presence is checked by hive-c0re's container_view::is_rate_limited on each build_all sweep (~10s) to populate ContainerView.rate_limited for the dashboard. Survives a harness restart (the Bus reads it back at boot and restores the flag), so the badge remains accurate if hive-c0re restarts while the harness is mid-sleep.

/state/hyperhive-model (per agent)

Single-line text file holding the claude model name currently selected for this agent (default haiku when absent). Written by Bus::set_model whenever the operator flips it via /model <name> in the web terminal. Read once at harness boot in Bus::new. Path overridable via HYPERHIVE_MODEL_FILE. Survives destroy/recreate, gone on --purge.

State dirs (per agent)

Under /var/lib/hyperhive/agents/<name>/:

• config/ — the proposed nix repo (manager-editable). Bind-mounted read-only to /agents/<name>/config inside the sub-agent's own container so the agent can inspect what defines it and request precise changes from the manager; RW into the manager via the /agents tree bind.
• claude/ — claude OAuth credentials, bind-mounted RW to /home/<name>/.claude inside the container.
• state/ — durable notes, the events.sqlite db, and the turn-stats sqlite db. Bind-mounted to /agents/<name>/state inside the container (uniform for sub-agents + manager). The $HYPERHIVE_STATE_DIR env var exposes the same path to in-container scripts.

Under /var/lib/hyperhive/applied/<name>/ — the hive-c0re-only applied repo. Tracks flake.nix (module-only boilerplate; never edited after first spawn) + agent.nix (the actual config; the manager's edits land here via the approval flow) + any other files the manager committed. .git/ carries the proposal / approved / building / deployed / failed / denied tag history.

Under /var/lib/hyperhive/meta/ — the swarm-wide deploy flake. Single repo for the whole host; flake.nix declares one input per agent + one nixosConfigurations.<n> output per agent; flake.lock is the canonical "what's deployed where." The git log is the deploy audit trail (one commit per successful deploy or hyperhive bump). Manager has this RO-mounted at /meta/.

Marker file /var/lib/hyperhive/.meta-migration-done is written by the startup migration after every container has been repointed at meta#<n>. Removing it forces a re-run on next hive-c0re start (idempotent — only the actual repoint step would re-fire).

Destroy vs purge

• DESTR0Y (default) — stops + removes the nspawn container, drops the systemd drop-in, fails any pending approvals. State dirs stay put; the agent appears in the dashboard's K3PT ST4T3 section as a tombstone with ⊕ R3V1V3 and PURG3 actions. R3V1V3 queues a Spawn approval that reuses the kept state on approve (no re-login).
• PURG3 (opt-in via the dashboard button or hive-c0re destroy --purge <name>) — DESTR0Y plus wipes /var/lib/hyperhive/{agents,applied}/<name>/. Config history, claude creds, /state/ notes, and the events db are all gone. No undo.

The manager is non-destroyable from both paths (declarative container; would fight with the host's NixOS config).

Run-time dirs

/run/hyperhive/ is tmpfs-backed (systemd RuntimeDirectory=) but preserved across hive-c0re restarts via RuntimeDirectoryPreserve=yes. Without that, every restart wipes bind sources and existing containers can't be started.

• /run/hyperhive/host.sock — admin socket (host-side CLI).
• /run/hyperhive/manager/mcp.sock — manager-privileged socket.
• /run/hyperhive/agents/<name>/mcp.sock — per-sub-agent socket (bind-mounted into the container as /run/hive/mcp.sock).

On startup, Coordinator::register_agent drops any prior socket task before rebinding — idempotent so a hive-c0re restart followed by rebuild alice recreates the agent's socket without a clean reinstall.

First-boot agent-user migration

The harness runs as a per-agent unix user inside the container (hyperhive.user.name, defaults to the agent's logical label so each container has a uniquely-named user). Operators with legacy root-owned state dirs need a one-time data shuffle so they don't lose their claude session.

system.activationScripts.hive-agent-user-migrate (in nix/templates/harness-base.nix) runs on every activation, marker-guarded so the substantive moves only happen once per container lifetime:

1. ${homeDir} exists with the right ownership — covers the very first boot before useradd's createHome has had a chance to chown. Also re-applies on every rebuild in case the meta-flake's per-agent name evolves (rare).
2. Migrate any leftover /root/.claude content into ${homeDir}/.claude — legacy claude wrote to root's empty home; the bind mount didn't exist yet. Marker (/var/lib/hive-agent-user-migrated) guards single-shot. cp -an (no-clobber) so any pre-existing files at the new location win — never blow over data already there.
3. Chown the bind-mounted state dir (/agents/*/state) recursively so the agent user can read/write it. Wildcard matches the single agent that container sees; -h skips symlinks the agent might have planted.
4. Chown the ~/.claude/ bind-mount recursively. Legacy claude wrote .credentials.json 0600 root:root; the current harness reads ~/.claude/ as the agent user to decide Online vs NeedsLogin in login::has_session. Without the chown the existing credentials get silently treated as "no session" and the operator re-prompts every boot.

The activation script will eventually become unnecessary once no operators have legacy root-owned state dirs left to migrate; drop the body + marker check at that point.

Matrix per-agent daemon + token-arrival trigger

hive-matrix-daemon is a long-running matrix-sdk Client + sync process per agent. Holds the unix socket the stdio hive-matrix-mcp bridge talks to, emits hyperhive wake signals on incoming room events via /run/hive/mcp.sock. Conditional on hyperhive.matrix.enable (which both the daemon AND the auto-injected extraMcpServers.matrix entry read).

Socket path lives inside the systemd-managed runtime dir (RuntimeDirectory = "hive-matrix" → /run/hive-matrix/, owned by the agent user) so the daemon can bind without needing root over /run/ itself. Both daemon + bridge agree on the path via the HIVE_MATRIX_SOCKET env var.

First-boot ordering: hive-c0re provisions the matrix token AFTER agent containers come up. Without the path-trigger sibling (systemd.paths.hive-matrix-daemon, PathExistsGlob = /agents/*/state/matrix-token), the daemon would exit 0 quietly the first time it ran and the MCP would have no backend until the next restart. The .path unit makes the appearance of the token re-fire the service so the daemon comes alive in the same boot cycle as provisioning. matrix-avatar-sync.path uses the same pattern for the icon-upload oneshot.