diff --git a/docs/persistence.md b/docs/persistence.md index dda2a6be..679b2dd3 100644 --- a/docs/persistence.md +++ b/docs/persistence.md @@ -40,20 +40,23 @@ schemas, file layouts, and internal migration mechanics. ### `/var/lib/hyperhive/db/broker.sqlite` (host) -Seven tables, all in one file — four queues, the schedule -header/targets split, and the per-agent power-intent registry: +Seven tables, all in one file — three queues, a small key/value +table, the schedule header/targets split, and the per-agent +power-intent registry: - `messages` — every inter-agent / operator-bound message. `sender / recipient / body / sent_at / delivered_at / acked_at / - in_reply_to`. `in_reply_to` links a reply to its parent row id; - the dashboard and per-agent inbox render these as threaded rows. -- `reminders` — `mcp__hyperhive__remind` queue. - `agent / message / file_path / due_at / created_at / sent_at / - attempt_count / last_error`. `file_path` set when a body - exceeded the inline soft-cap and got auto-spilled to a file - under the agent's state dir; the worker delivers a short - pointer instead. `attempt_count` / `last_error` accumulate - on delivery-failed retries. + in_reply_to / priority`. `in_reply_to` links a reply to its parent + row id; the dashboard and per-agent inbox render these as threaded + rows. +- `kv` — small persistent key/value store (`key PK / value`) for + host-side bookkeeping that doesn't warrant its own table. + + ⚠️ The `mcp__hyperhive__remind` queue is **not** here any more: it + moved to a harness-local, per-agent store as part of the + loose-ends-v2 migration — see [`/harness/` contents + below](#state-dirs-per-agent) for where reminders (and todos, and + the questions mirror) actually live now. - `approvals` — the queue. `agent / kind (merge_config_pr | spawn | init_config | update_meta_inputs | schedule_prompt) / commit_ref / requested_at / status / resolved_at / note`. @@ -98,8 +101,6 @@ Retention: - Approvals and questions are kept indefinitely — both are audit trails. `actions::destroy` and answered questions stay visible to anything that queries by id. -- Reminder rows are kept after `sent_at` is set (audit trail); - no automatic vacuum today. - Scheduled prompts: one-shot rows are deleted on fire by the worker; recurring rows live until the operator cancels them (`cancel_schedule` MCP / dashboard ✗) which tombstones via @@ -117,20 +118,26 @@ One table: - `events(id, ts, kind, payload_json)` — every `LiveEvent` the harness emits during turn loop execution. -The harness writes; the host vacuums. `hive-c0re::events_vacuum` -runs hourly and sweeps every existing agent harness dir. Retention -is **type-scoped**: it deletes only the verbose `stream` rows (the -raw claude `stream-json` deltas — one per text chunk / tool use, the -bulk of the file's size) older than 14 days, and keeps every other -kind (`turn_start`, `turn_end`, `note`, `status_changed`, -`model_changed`, `token_usage_changed`, `turn_state_changed`) -indefinitely — those are small and carry the semantic per-turn -history the operator scrolls back through when debugging a -regression. Age-only within the `stream` kind — no row cap — so a -chatty turn doesn't lose its stream history sooner than a quiet one. -Centralising retention on the host means a misbehaving harness can't -disable its own vacuum and agents don't need any cleanup wiring of -their own. +The harness both writes and vacuums it — this used to be a host-side +sweep, but hive-c0re runs as the unprivileged `hive-core` user under +privsep and can't delete agent-owned files (host-side deletes hit +`PermissionDenied` on the bash-task trio and a readonly-database error +here), so cleanup moved in-container. `hive-agent`'s `vacuum::run` +(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is +**type-scoped**: it deletes only the verbose `stream` rows (the raw +claude `stream-json` deltas — one per text chunk / tool use, the bulk +of the file's size) older than 14 days, and keeps every other kind +(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`, +`token_usage_changed`, `turn_state_changed`) indefinitely — those are +small and carry the semantic per-turn history the operator scrolls +back through when debugging a regression. Age-only within the +`stream` kind — no row cap — so a chatty turn doesn't lose its stream +history sooner than a quiet one. The trade-off (accepted): a +misbehaving harness could now skip its own cleanup, which the old +host-side sweep was meant to prevent — but a compromised harness is +already inside the container trust boundary +([`docs/security.md`](security.md)), and these are ephemeral local +artifacts, so cleaning them up where they live is the honest fix. Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness` setups). On open failure the `Bus` falls back to no-store mode @@ -211,11 +218,15 @@ the actual failure and leave only the host journal holding the complete output. With this table the dashboard can surface the entire log. -Two indices: +Three indices: - `(agent, started_at)` — backs the per-agent latest-N lookup used by the agent card chip. - `(status, finished_at)` — backs the retention sweep that runs as part of the existing hourly vacuum. +- `(node_id)` — added by a later migration so a build log row can be + looked up by the job-queue node it belongs to (a `hive_jobq` node is + immutable after insert, so the link is recorded on the log row + instead); legacy rows predating the column keep `node_id IS NULL`. Writes are best-effort: `append_stdout` / `append_stderr` / `finish` log a warning on sqlite error and let the build continue. A failed @@ -239,7 +250,7 @@ and inbox messages queue unacked until it's removed (see Unusually, it's read and written from **both** sides of the harness bind-mount, and that's the whole design: the harness stats it -in-container via `hive_agent::paths::paused_marker`, while hive-c0re +in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re stats it on the host (`Coordinator::is_paused`) to populate the `paused` field on the agent card, and creates/removes it (`Coordinator::set_paused`) for `hivectl agent pause|resume` and the @@ -284,15 +295,26 @@ Under `/var/lib/hyperhive/agents//`: - `bash-tasks/` — task JSON + stdout/stderr files for background `mcp__bash__run` jobs. JSON files are `.json` (status + tails), `.out` / `.err` - (full captured output). `hive-c0re::bash_tasks_vacuum` runs - hourly and deletes terminal task trios older than 48 hours; - non-terminal (still-running) tasks are never deleted by vacuum. - - `hyperhive-todos.sqlite` — loose-ends-v2 todo store. In-container - daemons (`hive-bash-daemon`, `hive-matrix-daemon`, `hive-forge-notify`) - upsert keyed todos here over the harness's in-agent socket - (`HIVE_AGENT_SOCKET`); the harness merges them into `get_loose_ends` - output and clears a row on `mark_todo_done`. Replaced the old - file-based `mcp-loose-ends/` scanner. + (full captured output). The harness's own hourly sweep + (`hive-agent`'s `vacuum::run`, same one that ages out `stream` + event rows above) deletes terminal task trios older than 48 + hours; non-terminal (still-running) tasks are never deleted. This + used to be a host-side `hive-c0re` vacuum, moved in-container for + the same privsep-ownership reason as the events vacuum above. + - `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos, + reminders, and a questions mirror, one small table each in a single + file (in-container daemons — `hive-bash-daemon`, `hive-matrix-daemon`, + `hive-forge-notify` — upsert keyed todos here over the harness's + in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into + `get_loose_ends` output and clears a row on `mark_todo_done`). + Replaces three formerly-separate files + (`hyperhive-todos.sqlite`, `hyperhive-reminders.sqlite`, and the + old file-based `mcp-loose-ends/` scanner before that) — a one-time + boot migration (`db_migrate::run`) folds the legacy files into this + path the first time a harness boots after the upgrade. Also backs + the `mcp__hyperhive__remind` queue, which moved from a host-side + `broker.sqlite` table to this per-agent store as part of the same + migration. The harness itself is also a producer, not just the socket server: boot wiring's `spawn_todo_socket` starts `todo_server::run` (the @@ -358,8 +380,11 @@ Contents: audit trail (one commit per successful deploy or hyperhive bump). - `topology.json` — parent/child agent graph (`{ "alice": "root", "bob": "alice", "root": null }`). - Written by `topology::set_parent`; read by the dashboard, the - renderer, and `` / `` recipient resolution. + Written by `topology::apply_set_parent` (the pure move-validating + transform) via `meta::bulk_commit_topology` (the committer — see the + `Reparent` node in [`docs/coordinator.md`](coordinator.md)); read by + the dashboard, the renderer, and `` / `` recipient + resolution. - `tool-groups.json` — per-agent MCP tool group grants (`{ "alice": ["messaging", "inbox", "execution"] }`). Written by `tool_groups::set_groups`; injected as `HIVE_TOOL_GROUPS` env @@ -431,8 +456,12 @@ until an explicit opt-in upgrade. actually a subvolume, then the normal `remove_dir_all` sweep covers plain-dir agents + the applied dir. -Per-subvolume disk-usage accounting and optional quotas are a -follow-up (the qgroup work), not part of the base migration. +Per-subvolume disk-usage accounting and optional quotas have since +landed as the qgroup work: `hivectl quota-enable` turns on btrfs +qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and +`hivectl agent quota show|set` reads/limits one agent's +subvolume usage through the same hive-priv-mediated path as +subvolume creation/deletion above. This is the same subvolume `hivectl agent subvol snapshot push` sends to the swarm's snapshot store — see @@ -506,10 +535,12 @@ auto-injected `extraMcpServers.matrix` entry read). **First-boot ordering**: hive-c0re provisions the matrix token AFTER agent containers come up. Without the path-trigger sibling (`systemd.paths.hive-matrix-daemon`, `PathExistsGlob = -/agents/*/state/matrix-token`), the daemon would exit 0 quietly the -first time it ran and the MCP would have no backend until the next -restart. The `.path` unit makes the appearance of the token re-fire -the service so the daemon comes alive in the same boot cycle as +/agents/*/state/matrix-token*` — the trailing `*` also catches a +secondary multi-account token like `matrix-token-ccc`), the daemon +would exit 0 quietly the first time it ran and the MCP would have no +backend until the next restart. The `.path` unit makes the appearance +of the token re-fire the service so the daemon comes alive in the +same boot cycle as provisioning. The same token watcher also drives avatar setting: on a restart the daemon re-runs each account's bring-up, which sets the avatar (see below).