# Approvals + helper events The approval queue is hyperhive's pivot: nothing that changes the shape of an agent (its config, whether it exists) happens without an operator click. The submitting agent — any agent with the `approvals` tool group, which manages the config of its **direct children** (the root agent for top-level agents; a sub-manager for its own subtree) — is the policy gate in front of that queue; helper events are how it stays informed about what happens after a decision lands. ## End-to-end approval flow Config changes flow through a **forge pull request** on the agent's `agent-configs/` repo — the same surface agents use for code PRs. There is no bespoke MCP tool for config changes: opening the PR IS the request. 1. The submitting agent (the child's parent, holding the `approvals` tool group) edits files in the child's proposed config repo (any tracked path, but `agent.nix` is the contract entry point), commits with its own git identity, and pushes a branch + opens a PR on `agent-configs/` with `hive-forge`. The parent's container has the child's proposed config repo bind-mounted read-write at `/agents//config/` (topology-driven via `set_nspawn_flags`; the agent's *own* config at `/agents//config/` is read-only). Branch protection (push/merge whitelist = `core`, approvals whitelist = operator team; see "Forge mirror" and #1787) makes the agent a write collaborator that **cannot merge its own config PR**. 2. hive-c0re's `/webhook/config-pr` endpoint receives the Forgejo `pull_request` event (opened / synchronized / reopened) and queues a `MergeConfigPr` approval; a poll fallback catches any missed webhook. The approval row stores the PR **number** (`commit_ref`) and the PR **head sha at queue time** (`fetched_sha` — the "reviewed" sha). If the PR head later moves, the stale approval is superseded by a fresh one pinned to the new head, so the operator always reviews what will actually deploy. 3. The operator reviews the PR **on the forge** (native diff, threaded comments, CI status) and sees a matching card on the dashboard with a "review PR on forge" deep link. They click ◆ APPR0VE (or `hivectl approvals approve ` on the CLI) once satisfied. 4. On approve, a deploy DAG runs three phases under a resource-holding `DeployWindow` root (see *Queue templates* below): - `MergeVerify` re-reads the live PR head and **aborts if it drifted** from the reviewed `fetched_sha` (the submitter must push again, which queues a fresh approval); then fetches that head into the applied repo and **eval-verifies** it — a flake eval on a throwaway checkout. This is the trust gate: it relies on c0re's own eval, not on any in-repo (agent-forgeable) signal like a CI status. Nothing is mutated in this phase, so a rejection here leaves the forge and the applied repo exactly as they were. - `DeployApply` parks the pre-merge `applied/main` in `refs/hyperhive/rollback/`, then fast-forward-merges the reviewed head to the forge config repo's `main` (this IS the merge — a `core`-authenticated ff-merge pinned to the reviewed sha, so a moved PR head can't substitute bytes), and runs the deploy proper (`deploy_applied_target`): ff `applied/main`, two-phase meta deploy, container rebuild. On success it drops the rollback ref and plants `deployed/`. - `DeployTail` runs on **every** outcome, including a cancel-cascade. If the rollback ref survived, the deploy never confirmed good: it rolls `applied/main` back, resyncs the working tree, and aborts the staged meta lock, so the agent stays on its last-good tree. Then it mirrors the config repo (and its new deploy tag) to the forge. The rollback state lives in a **git ref, not a local variable**, on purpose: hive-c0re can restart between the apply and the tail, and the tail still has to know what to undo when it does. 5. `HelperEvent::ApprovalResolved` (and `Rebuilt`) land in the **submitting agent's** inbox via `notify_submitter`, carrying both the canonical sha and the terminal tag (the approval row carries a `submitter` column recording the agent the change is for). ### Withdrawing a pending approval The submitting agent can call `cancel_loose_end(kind: "approval", id)` to withdraw an approval that hasn't been acted on yet. The row transitions to `ApprovalStatus::Cancelled` (distinct from `Denied`/`Failed`), the dashboard pulls the card out of the pending pane, and `ApprovalResolved { status: "cancelled" }` fires on the root agent + dashboard channels. Approvals that have already been approved/denied/failed return an error — the resolution is final once the operator (or a lifecycle failure) acted on the row. The socket refuses the `approval` kind with a clear error for any agent that lacks the `approvals` tool group: only an agent with that group submits approvals (for its direct children), so an agent without it has nothing of its own to withdraw. `InitConfig` approvals seed a brand-new agent's proposed config repo. On approve, hive-c0re seeds it with a default `agent.nix` template and sends `HelperEvent::ConfigReady { agent }` to the submitting agent's inbox via `notify_submitter`. The submitting agent reviews, edits, and commits the template; the operator then **spawns** the agent (the `Spawn` approval / `◆ R3QU3ST SP4WN` button), which creates the container from that config. From then on, config changes go through a forge PR (`MergeConfigPr`) — never a fresh spawn. This gives the submitting agent (and operator) an explicit review gate on the initial configuration before any container is created. ### Approval kinds (wire shapes) `ApprovalKind` carries five variants; each maps to a different `commit_ref` encoding because that field is overloaded as the kind-specific payload carrier. - `MergeConfigPr` — the config-change flow. Triggered automatically: when an agent opens (or force-pushes) a PR on its `agent-configs/` forge repo, hive-c0re's `/webhook/config-pr` endpoint receives the Forgejo pull_request event and queues this approval row. No MCP tool call needed — the forge PR IS the request. `commit_ref` stores the **PR number** (decimal), and `fetched_sha` is the PR **head sha at queue time** (the "reviewed" sha). On approve, the deploy DAG's `MergeVerify` phase re-reads the live PR head and aborts if it drifted from `fetched_sha` (submitter must push again to re-trigger), then fetches that head into the applied repo and eval-verifies it; `DeployApply` fast-forward-merges the forge config repo's `main` to it (the merge) and runs `deploy_applied_target`; `DeployTail` compensates on failure. Never a first spawn. - `Spawn` — direct container creation from the agent's config repo. `commit_ref` is empty. Submitted via `HostRequest::RequestSpawn` (operator-gated, the `◆ R3QU3ST SP4WN` dashboard button + `hivectl agents request-spawn` CLI). The host-level `HostRequest::Spawn` variant bypasses the approval queue entirely — privileged-context use only (operator on the host shell, test scripts, one-off recoveries). This is the **canonical first-spawn**: a new agent's `InitConfig` seeds its config repo, the submitting agent customises it, then the operator spawns to create the container. Subsequent config changes go through a `MergeConfigPr` PR. - `InitConfig` — `commit_ref` is empty; the variant just gates "seed the proposed repo with the default template" against operator approval. Step 1 of the two-step spawn flow above. - `UpdateMetaInputs` — `commit_ref` stores the JSON-encoded inputs array (`"[]"` = all inputs, `"[\"nixpkgs\"]"` = just nixpkgs, etc.). `agent` field is set to the requesting root agent. On approve hive-c0re runs `nix flake update [inputs...]` on the meta flake and commits the resulting lock changes. - `SchedulePrompt` — `commit_ref` stores the JSON-encoded `SchedulePromptPayload` (target list, body, schedule) so the approval row carries the full submission verbatim. On approve hive-c0re inserts a row into `scheduled_prompts` with `source = approval:`; the worker fans the body out as inbox messages to each target at the scheduled time, recurring when `interval_seconds` is set. ### Scheduled prompts (submit paths) Two ways a row lands in `scheduled_prompts`: - **Operator-direct** (`source = "operator"`): the operator adds a schedule through the dashboard form. Lands in the table immediately, no approval gate — operator action is already the trust boundary. - **Agent-requested** (`source = "approval:"`): an agent submits a `RequestSchedulePrompt` through its MCP socket (the `request_schedule_prompt` tool, `scheduling` group). An `ApprovalKind::SchedulePrompt` row is queued; on approve, hive-c0re inserts the schedule row with `source = approval:` so the audit trail points back at the operator decision (above). No self-target shortcut: even agent-self schedules need approval. The existing `remind` MCP tool stays the quick self-wake path (no approval, lands directly in the agent's own inbox); this module is the bigger, multi-recipient, operator-visible thing. ### Scheduled prompt worker (catch-up clamp) When hive-c0re comes back from being down, the worker sees rows whose `next_fire_at_unix` is well in the past. For recurring rows that would mean firing N delayed pulses in a row — spammy and useless. Instead the worker fires **once** per row and bumps `next_fire_at_unix` to the next interval slot ≥ `now`, recording how many cycles were skipped in `last_result` (per-target). Operators see "fired late, caught up from 17 skipped" instead of 17 wake-up storms. One-shot rows fire once (if past due, on the next worker pass) and are deleted by the worker; recurring rows survive until cancelled. `targets` is its own table (`scheduled_prompt_targets`) so partial cancellation flips a single row and the dashboard can show last-fired / last-result per recipient. Cancelling every target reaps the parent row on the next worker pass. ### Missing-target failure When a target name doesn't resolve to a known agent (container destroyed, operator typo, etc.) the worker: 1. Records `last_result = "no such agent: "` on the per-target row. 2. Sends a single advisory `Message` from `system` to `operator` naming the schedule, target, and reason. 3. Continues fanning out to the other live targets. Transient broker errors (sqlite lock contention, etc.) get the same `last_result` annotation plus a `tracing::warn`, and then: - **Recurring rows** re-arm to the next interval slot — the retry self-heals on the next worker pass. - **One-shot rows** are deleted unconditionally after their single fan-out pass; a broker error on a one-shot is not retried (the operator advisory and `last_result` are the only audit trail). ### Reminder delivery: file-path semantics A reminder may carry a `file_path` (the agent-visible path inside its container, e.g. `/agents//state/foo.md`). On delivery hive-c0re: 1. **Translates** the container path to the host path (`/var/lib/hyperhive/agents//state/foo.md`) so c0re can write from outside the container. 2. **Validates** the path: rejects anything outside the agent's own state subtree, containing `..` (path traversal), or with an empty relative tail. On rejection the write is skipped and the original message is delivered inline with a warning — the reminder still fires. 3. **Defends against symlink escape**: after `create_dir_all`, the parent dir is canonicalized and re-verified to live under the agent's host state root. The final file is opened with `O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the basename cannot redirect the write to an arbitrary host path. 4. **Writes the body to disk** and delivers a short pointer message in its place, keeping the agent's inbox / wake-prompt small while the bulky payload is read out of band. Atomicity of the inbox INSERT + `reminders.sent_at` UPDATE is handled inside `Broker::deliver_reminders_batch`; the scheduler only computes the body strings before calling it. ### Destroy semantics `HostRequest::Destroy { name, purge }` is the lifecycle tear-down, not an approval. Stops + removes the nspawn container, drops the systemd drop-in, fails any pending approvals. Persistent state (proposed/applied repos, claude credentials, `/state/` notes) is **kept by default** — recreating the agent with the same name reuses prior config + login. With `purge = true` the agent's `/var/lib/hyperhive/{agents,applied}//` trees are also wiped (config history + creds + notes gone forever). The root/bootstrap container is destroyable like any other — hive-c0re recreates it on the next startup if it's absent, so destroying it is transient. ## Meta flake The hive-c0re-owned repo at `/var/lib/hyperhive/meta/` declares one flake input per agent (`agent-.url = "git+http:///agent-configs/.git"`) and one `nixosConfigurations.` output per agent. Each output wraps `inputs.agent-.nixosModules.default` with the identity + `HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module that `setup_applied` used to generate inline. Containers run against `--flake /var/lib/hyperhive/meta#`. The declared input url is the agent's **forge config repo** (the same `agent-configs/` the config-PR flow lands approved changes on), so the meta flake references a reviewable, reproducible source rather than a local checkout. hive-c0re authenticates that `git+http` fetch via a git credential helper that reads the live forge-core token — no token in the url or the lock. The deploy and manual-rebuild paths, however, do **not** re-lock from the forge: they `--override-input agent- git+file:///var/lib/hyperhive/applied/`, locking the exact config that `verify_commit` gated and `applied//main` was fast-forwarded to. That keeps a deploy/rebuild reproducible and independent of forge reachability — rebuilds fire on crash-restart and meta bumps, not just config PRs — while the declared url stays the forge. `sync_agents` re-renders + re-locks the persistent input; a plain `nix flake lock` leaves an existing applied override in place (it only re-locks when the declared url itself changes), so the forge-declared / applied-deployed split is stable. Per-deploy lock flow (two-phase), spread across the deploy subtree's nodes — each phase is its own node, so the queue can show which one is running and a restart resumes at node granularity: 1. `DeployApply` → `meta::prepare_deploy(name)` runs `nix flake lock --update-input agent-` without committing. Working tree of meta now points the input at `applied//main` (which the deploy already fast-forwarded to the reviewed PR head). 2. The rebuild subgraph `DeployApply` grows into the DAG builds and swaps the container (`Prebuild → StopForUpdate → Swap → PostSwap`, plus `Reconcile`). Nix evaluates against the staged lock. 3. On success — `FinalizeDeploy` drops the rollback ref, plants `deployed/`, then `meta::finalize_deploy(name, sha, "deployed/ ")` stages `flake.lock` and commits with `deploy deployed/ `. Meta's git log gains one entry per successful deploy. 4. On failure — the `DeployTail` node runs `meta::abort_deploy()` (`git restore flake.lock`) so the meta history shows only successes; the failure stays as an annotated `failed/` tag in `applied/`. The tail runs on every outcome, so this also covers a hive-c0re restart mid-build: the staged lock is dropped and `applied/main` rolled back from the parked `refs/hyperhive/rollback/`. Single-phase variants exist for paths without rollback semantics: `meta::lock_update_for_rebuild(name)` for the manual `↻ R3BU1LD` button (commits if the lock changed) and `meta::lock_update_hyperhive()` for the auto-update flake-rev bump (one shot before per-agent rebuilds, commits if the lock changed). `meta::sync_agents(hyperhive_flake, dashboard_port, &agents)` is the idempotent reconciler called by `spawn`, `destroy`, `rebuild`, and the startup migration. Renders `flake.nix` from the agent list; if it differs from disk, runs `nix flake lock` + commits as `regenerate meta flake` (or `seed meta from N agent(s)` on the very first call). The root agent has `/meta` RO-bound inside its container: `git -C /meta log --oneline` is the swarm-wide deploy log, `cat /meta/flake.lock | jq '.nodes["agent-"].locked'` resolves which sha each agent is pinned at right now. Dashboard surfaces the same info as a `deployed:` chip per container row. ## Two repos per agent ``` /var/lib/hyperhive/agents//config/ proposed — submitting agent RW └── # any files the submitting # agent wants in the commit. # agent.nix is the # convention entry # point; flake.nix is # tracked boilerplate # (submitting agent doesn't # edit it). /var/lib/hyperhive/applied// applied — core-only ├── .git/ # tag-rich history ├── flake.nix # tracked, fixed │ # boilerplate exporting │ # nixosModules.default ├── agent.nix # working tree of main └── # also tracked /var/lib/hyperhive/meta/ swarm-wide flake — core ├── .git/ # one commit per successful │ # deploy ├── flake.nix # generated from agent set └── flake.lock # pins each agent's sha ``` Why two physical repos: the submitting agent's `/agents//config/` is RW — a buggy or hostile agent can `git clean -fdx` its own proposed tree. The applied repo is never bind-mounted (except the read-only `.git` exposure described below) so a destructive move inside the container cannot reach it. The container's `--flake` ref is `/var/lib/hyperhive/meta#` (see "Meta flake" above). The agent's own `applied//flake.nix` is a fixed boilerplate that exports `nixosModules.default = import ./agent.nix`; the meta flake imports that module and wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT`. ### Tag state machine Each deploy leaves a tag on the underlying commit inside the applied repo: | Tag | When | Annotated? | |---|---|---| | `deployed/` | rebuild succeeded — `main` ff's here | no | | `failed/` | rebuild failed | yes (body = error) | `deployed/0` is planted at first spawn. `applied/main` is always the latest `deployed/*`. A `failed/` tree stays browsable forever — `git log --tags` in the applied repo is the audit trail. A denied or failed config PR carries no extra state on the forge side: the PR stays open, and the submitter pushes again (or closes it) to retry. ### Dispatch via the job queue Long-running approval work — `MergeConfigPr`, `UpdateMetaInputs`, `Spawn` — no longer runs inline inside `actions::approve`. Instead the approval handler submits a DAG to the global job queue (`docs/coordinator.md::Job queue`): | `ApprovalKind` | DAG submitted | source | |---|---|---| | `MergeConfigPr` | `rebuild` (`DeployWindow` root + `MergeVerify → DeployApply` + `DeployTail`) | `approval` | | `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` | | `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` | | `InitConfig` | — runs inline (sub-second git seed) | — | | `SchedulePrompt` | — runs inline (single sqlite insert) | — | The DAG carries the originating `approval_id`, surfaced on the node that owns it — for a deploy that's the `DeployWindow` root, so the dashboard renders one approval card, not four. **Every** queued kind resolves through `actions::resolve_approval_dag` when its DAG settles terminal: the deploy's phases are ordinary queue nodes, so the DAG's own terminal state is the authoritative outcome. That hook fires the matching `HelperEvent::*` via `finish_approval`, derives the `Rebuilt` event's terminal tag (verifying the tag actually resolves in the applied repo — a pre-merge rejection plants none), posts the failing build log back to the config PR, and for a spawn runs the post-spawn forge bookkeeping. Two visible consequences: - **Operator dashboard**: after clicking APPR0VE the work-in-progress shows up on the *rebuild queue* card (`/api/state.rebuild_queue` + live `rebuild_queue_changed` events), not on the approvals panel (which already moved the row to "approved"). A long meta-update cascade renders as a parent DAG with one child rebuild per affected agent — see `docs/web-ui.md` for the layout. - **Cancellation**: the dashboard's *× cancel* button on a still-queued DAG calls `POST /api/rebuild-queue/{id}/cancel`, which flips it to `Cancelled` before any node runs (and fails the approval row instead of leaving it dangling). Returns `{"cancelled": true}` on success, `{"cancelled": false}` once any node started — terminal states can't be retroactively rewritten. The `approval` source + `approval_id` mean a tail-end build failure surfaces back as a failed approval row, not just a silent queue entry. `manual` (dashboard ↻ R3BU1LD) and `auto_update` (boot reconcile) DAGs use the same queue but skip the approval plumbing. ### Forge mirror The bundled `hive-forge` container is mandatory (it deploys with hyperhive), and hive-c0re mirrors every agent's applied repo into a private `agent-configs` Forgejo org. `forge::push_config()` pushes `applied/main` plus every tag to `agent-configs/` after each ref mutation: the spawn that seeds `deployed/0`, every successful deploy (which plants `deployed/`) or failed build (`failed/`), and a sweep at startup. Pushes are best-effort — a missing or stopped forge never blocks a deploy. Each agent is a **write collaborator on its own** `agent-configs/` repo — so it can push a branch and open a config PR — but not a member of any other agent's, so it can't reach another agent's config through the forge. Branch protection keeps `main` push/merge `core`-only with operator-team approval, so an agent can't fast-forward its own config or self-merge its PR (see the End-to-end flow + #1787). The tokenised push URL is passed inline to `git push`, never written into `applied//.git/config`; that repo is RO-bind-mounted into the root agent, and a stored token would leak core's admin credential to an agent. The dashboard deep-links into this org — a `config repo` link per container row and a `review PR on forge` link per config-PR approval card. See `docs/web-ui.md`. ### Submitting agent's view of config repos Every parent agent's container has its **direct children's** proposed config repos bind-mounted read-write (topology-driven: `lifecycle.rs` calls `bind_child_agent_dirs` for each entry in `topology::children_of(agent_name)`). An agent with the `approvals` tool group can therefore edit, commit, and submit changes for any of its direct children directly inside its container at `/agents//config/`. Agents holding the `can_manage_top_level_agents` topology role (defined as `ROLE_CAN_MANAGE_TOP_LEVEL_AGENTS` in `hive-c0re/src/agent_config/topology.rs`) get additional host-side bind mounts via `set_nspawn_flags`: - `/var/lib/hyperhive/agents/` → `/agents/` (RW) — all top-level agents' proposed repos (not just direct children). - `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every agent's authoritative applied repo, including `.git`. - `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide deploy flake. The root agent holds this role; a sub-manager that only manages a subtree does not, and only has its direct children's config dirs. Each proposed repo (`/agents//config/`) is pre-configured with `applied` as a git remote pointing at `/applied//.git`. Useful incantations from inside an agent with the full `/applied` mount: ```sh git -C /agents//config fetch applied git -C /agents//config log applied/main --oneline git -C /agents//config show applied/refs/tags/deployed/ git -C /agents//config show applied/refs/tags/failed/ # body = build error git -C /agents//config show applied/refs/tags/denied/ # body = operator note git -C /agents//config rebase applied/main # base in-flight work on what's deployed git -C /meta log --oneline # swarm-wide deploy history cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))' ``` The RO binds block push at the kernel level — git plumbing inside the container cannot corrupt either authoritative repo. ## Migration from the pre-tag / pre-meta schemes Both overhauls (tag-driven flow + meta flake) ship in-place migrations that run on every hive-c0re startup. Idempotent; each phase is a no-op once already applied. Behaviour: - Tag-driven phase: assumes the operator ran the one-shot `git tag deployed/0 main` script (see commit history / earlier docs revisions) once per agent. Tagging is non-destructive: it doesn't touch live containers, state dirs, or claude creds. - Meta-flake phase: rewrites each `applied//flake.nix` to the module-only boilerplate, wires the `applied` remote in each proposed repo, bootstraps the meta repo from the current agent list, and `nixos-container update`s every container at `meta#`. The expensive last step is guarded by `/var/lib/hyperhive/.meta-migration-done` so it only runs once across hive-c0re restarts. Set `HIVE_SKIP_META_MIGRATION=1` on the service to defer. No state loss in either migration. claude creds, /state/ notes, the events DB, proposed history, and applied history all survive. The root agent keeps its session; sub-agents stay logged in. ## The root/bootstrap container is hive-c0re-managed The root agent container runs through the **same lifecycle as sub-agents**. On `hive-c0re serve` startup, if `ruth` is missing, hive-c0re creates it. The root agent's flake lives at `/var/lib/hyperhive/applied/ruth/`; its proposed config at `/var/lib/hyperhive/agents/ruth/config/`. The root agent can edit its own `agent.nix` (visible inside the container at `/agents/ruth/config/`) and open a config PR on `agent-configs/ruth` for operator approval, same as any other agent. Differences from sub-agents: - `flake.nix` extends `hyperhive.nixosConfigurations.manager` (vs `agent-base`). - Web UI port via `lifecycle::agent_web_port("ruth")` — same FNV-1a hash as every other agent (8100..8999 range). - `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents` → `/agents` (RW) so the root agent can edit per-agent proposed repos, and `/var/lib/hyperhive/applied` → `/applied` (RO) so the root agent can `git fetch` deployed/failed/denied tags from any agent's authoritative applied repo (see "Root-agent view of applied" below). - First-deploy spawn bypasses the approval queue (the root agent is required infrastructure). - Per-agent socket lives at `/run/hyperhive/manager/`, owned by `manager_server::start`. **Migration note** (for older hosts): drop any `containers.root = { ... }` block from your host NixOS config. hyperhive creates and updates the root agent itself. ## Root-agent policy The system prompt (`hive-ag3nt/prompts/system.md`, rendered via `hive_ag3nt::prompt::render`) is the **same for every agent**; what varies is which MCP tools are surfaced (gated by tool groups and capabilities in `agent.nix`). There is no `role:manager` block that renders only for the root agent. The root agent's approval-gating behaviour comes from its CLAUDE.md / agent-specific instructions, not the system prompt template. `ask(question, options?, multi?, ttl_seconds?, to?)` is available to **any agent** — it queues a question and returns the id immediately. When `to` is omitted (or `"operator"`) the question shows up on the dashboard; when `to` is another agent's name, the recipient receives a `HelperEvent::QuestionAsked` and answers via their own `answer` tool. Either way the answer arrives back as `HelperEvent::QuestionAnswered { id, question, answer, answerer }` in the asker's inbox. Storage is `hive-c0re::operator_questions` (sqlite) — same table, with a nullable `target` column (NULL = operator). Dispatch goes through `hive-c0re/src/questions.rs::{handle_ask, handle_answer}`. The answer flow is: ``` POST /answer-question/{id} agent: Answer { id, answer } → OperatorQuestions::answer(_, _, "operator") → questions::handle_answer → notify_agent(asker, QuestionAnswered { → OperatorQuestions::answer(_, _, agent) answerer: "operator", ... }) → notify_agent(asker, QuestionAnswered { answerer: agent, ... }) ``` Two more paths resolve a pending question with a sentinel answer: - `POST /cancel-question/{id}` (✗ CANC3L button on the dashboard) resolves with `[cancelled]`. The asking agent sees a terminal state and can fall back. - `ttl_seconds` deadline: a tokio watchdog spawned at submit time fires `answer(id, "[expired]")` once the ttl runs out. Already- resolved races no-op. The dashboard surfaces a `⏳ MM:SS` chip on each pending question with a deadline. ## Helper events to the submitting agent `Coordinator::notify_submitter(approval_id, &HelperEvent)` routes the event to the agent that originally submitted the approval (looked up from the `submitter` column on the `approvals` table). The harness delivers it as a regular `system` inbox message so it drives a normal claude turn. Legacy approval rows that predate the submitter column fall back to the root agent. Variants (`hive_sh4re::HelperEvent`): - `ApprovalResolved { id, agent, commit_ref, status, note }` — fired by `actions::approve` + `actions::deny` whenever an approval transitions to its terminal state. - `Spawned { agent, ok, note }` — the `Spawn` approval DAG + admin `HostRequest::Spawn`. - `Rebuilt { agent, ok, note }` — `auto_update::rebuild_agent` (covers startup scan + manual `/rebuild` from dashboard) + the `MergeConfigPr` deploy. - `Killed { agent }` — admin `HostRequest::Kill` + dashboard `/kill` + the `Kill` MCP tool. - `Destroyed { agent }` — `actions::destroy`. - `ContainerCrash { agent, note }` — `crash_watch`: a previously- running container went away with no operator-initiated transient state (Stopping / Restarting / Destroying / Rebuilding) AND no such transient was cleared in the last 30s (`RECENT_TRANSIENT_GRACE` tombstone, three `POLL_INTERVAL`s — closes the race where a lifecycle op finishes between two crash-watch polls and the container shows briefly as "stopped without transient" before the next start). The root agent can `start` it again or escalate. - `NeedsLogin { agent }` — sub-agent has no claude session yet. The root agent can't act directly (interactive OAuth); typically flags the operator. - `LoggedIn { agent }` — sub-agent just completed login. The root agent often greets the agent on this event. - `ConfigReady { agent }` — a new agent's proposed config repo was just seeded (post-`InitConfig` approval). The root agent can now edit `/agents//config/agent.nix`, commit the changes, and hand off to the operator to **spawn** the agent (the `Spawn` approval creates the container from that config). - `NeedsUpdate { agent }` — sub-agent's recorded flake rev is stale. The root agent calls `update(name)` to rebuild — idempotent, no approval required. - `QuestionAnswered { id, question, answer, answerer }` — dashboard `/answer-question/{id}` (answerer = `"operator"`), peer `Answer` request (answerer = agent name), or ttl watchdog expiry (answerer = `"ttl-watchdog"`, answer = `"[expired]"`). - `QuestionAsked { id, asker, question, options, multi }` — fired when an agent calls `Ask { to: Some(), ... }`. The recipient responds via `Answer { id, answer }` and the asker sees the matching `QuestionAnswered`. Optional `sha` field on `ApprovalResolved` and `Rebuilt` carries the canonical hive-c0re-vouched commit sha. Optional `tag` on the same two carries the deploy bookkeeping tag — `deployed/` on a successful build or `failed/` on a failed one, planted by the `MergeConfigPr` deploy. Both fields are `Option`: `None` on the paths that don't deploy a new commit (spawn / init_config / meta-update / deny, and `auto_update::rebuild_agent` reapplying the existing main, or the dashboard `↻ R3BU1LD` button when the lock didn't move). When set, `git show ` against `/agents//applied.git` inside the bootstrap container yields the exact tree that was referenced. To add a new event: new `HelperEvent` variant + call sites + update `prompts/system.md` (`` block, the lifecycle- event list) so the root agent knows the new shape. ## Auto-update on startup `hive-c0re serve` runs `auto_update::run` in a background task right after opening the coordinator. It enumerates managed containers and rebuilds any whose recorded hyperhive rev differs from the current one — sub-agents and the root agent go through the same `lifecycle::rebuild` path. "Rev" = canonical filesystem path of `cfg.hyperhiveFlake`. Marker file: `/var/lib/hyperhive/applied/..hyperhive-rev`. If the flake input has no canonical path (e.g. a `github:` URL), auto-update is a no-op — rebuild manually. The dashboard surfaces pending updates per agent: a clickable "needs update ↻" badge appears whenever the marker differs from current rev. The badge POSTs `/api/rebuild/`, calling the same `auto_update::rebuild_agent` path so manual triggers and the startup scan can't drift. When at least one container is stale, a top-level `↻ UPD4TE 4LL` button appears that loops over every stale container.