hyperhive/docs/approvals.md

Approvals + manager + helper events

The approval queue is hyperhive's pivot: nothing that changes the shape of an agent (its config, whether it exists) happens without an operator click. The manager (hm1nd) is the policy gate in front of that queue; helper events are how it stays informed about what happens after a decision lands.

End-to-end approval flow

1. Manager edits files under /agents/<name>/config/ (any tracked path, but agent.nix is the contract entry point) and commits with its own git identity.
2. Manager submits the commit sha via request_apply_commit(agent, commit_ref). commit_ref must be a commit sha (7-40 hex chars, short or full) — a branch or tag name is rejected so the approval pins an immutable commit.
3. hive-c0re immediately fetches that commit from the proposed repo into the applied repo and tags it proposal/<id>. It resolves the sha locally against the proposed repo, fetches all of proposed's heads into applied's object db, then tags the resolved commit — git fetch <remote> <sha>:<dst> can't fetch by a bare sha (the left side of a refspec is a remote ref name), so the resolution happens on hive-c0re's side. The approval row stores both the manager-supplied sha and the canonical hive-c0re-vouched sha. From here on the proposed repo is irrelevant for this approval — the manager can amend, force-push, or rm -rf the proposed repo and the queued approval still points at an immutable git object inside applied. 3a. Flake validation (ApplyCommit only): after the proposal tag is planted, hive-c0re reads proposal/<id>:flake.lock and runs two checks. If either check fails, no pending approval is created for the operator — the row is marked failed and surfaces on the dashboard with the validation message:
   • Stale lock — materialises the commit in a temp worktree, runs nix flake lock (no --update-input flags, so it only fills missing entries), and rejects if the committed flake.lock differs from the result. Triggered when the manager added or removed inputs in flake.nix without re-running nix flake lock. Fix: run nix flake lock in the config repo, commit, and re-submit.
   • Duplicate inputs — groups lock nodes by their canonical original field; rejects if two or more nodes share the same source. This usually means an input is missing inputs.<x>.inputs.nixpkgs.follows = "nixpkgs". Fix: add the follows directive, re-lock, and re-submit. Both checks only flag new violations — agents whose lock already carried duplicates before this check was added are unaffected until a coordinated config-change pass via manager.
4. Operator sees the proposal as a card on the dashboard — a full multi-file diff, toggleable between three bases (vs the running tree / vs the last approved proposal / vs the previous queued proposal) — and clicks ◆ APPR0VE (or hive-c0re approve <id> on the CLI).
5. hive-c0re moves the working tree to proposal/<id> and runs the build under a sequence of tags (see below). On success, applied/main fast-forwards to the proposal commit. On failure, main stays put and the working tree resets back to the previous deployed commit.
6. HelperEvent::ApprovalResolved (and Rebuilt for the ApplyCommit kind) land in the manager's inbox, carrying both the canonical sha and the terminal tag.

Withdrawing a pending approval

The manager can call cancel_loose_end(kind: "approval", id) to withdraw an approval that hasn't been acted on yet. The row transitions to ApprovalStatus::Cancelled (distinct from Denied/Failed), the dashboard pulls the card out of the pending pane, and ApprovalResolved { status: "cancelled" } fires on the manager + dashboard channels. Approvals that have already been approved/denied/failed return an error — the resolution is final once the operator (or a lifecycle failure) acted on the row.

Sub-agent surface refuses the approval kind with a clear error: sub-agents don't submit approvals, so they have nothing of their own to withdraw. Manager-only.

InitConfig approvals are the first step in a two-step spawn flow. On approve, hive-c0re seeds the proposed config repo with a default agent.nix template and sends the manager HelperEvent::ConfigReady { agent }. The manager then reviews, edits, and commits the template before calling request_apply_commit to proceed to an ApplyCommit approval. The first ApplyCommit creates the container; subsequent ones rebuild it with new config. This gives the manager (and operator) an explicit review gate on the initial configuration before any container is created.

Meta flake

The hive-c0re-owned repo at /var/lib/hyperhive/meta/ declares one flake input per agent (agent-<n>.url = "git+file:///var/lib/hyperhive/applied/<n>") and one nixosConfigurations.<n> output per agent. Each output wraps inputs.agent-<n>.nixosModules.default with the identity + HIVE_PORT / HIVE_LABEL / HIVE_DASHBOARD_PORT injection module that setup_applied used to generate inline. Containers run against --flake /var/lib/hyperhive/meta#<n>.

Per-deploy lock flow (two-phase, owned by actions::run_apply_commit → meta::{prepare,finalize,abort} _deploy):

1. meta::prepare_deploy(name) runs nix flake lock --update-input agent-<n> without committing. Working tree of meta now points the input at applied/<n>/main (which run_apply_commit already fast-forwarded to proposal/<id>).
2. lifecycle::rebuild_no_meta runs nixos-container update <c> --flake meta#<name>. Nix evaluates against the staged lock.
3. On success — meta::finalize_deploy(name, sha, "deployed/ <id>") stages flake.lock and commits with deploy <n> deployed/<id> <sha12>. Meta's git log gains one entry per successful deploy.
4. On failure — meta::abort_deploy() runs git restore flake.lock so the meta history shows only successes; the failure stays as an annotated failed/<id> tag in applied/<n>.

Single-phase variants exist for paths without rollback semantics: meta::lock_update_for_rebuild(name) for the manual ↻ R3BU1LD button (commits if the lock changed) and meta::lock_update_hyperhive() for the auto-update flake-rev bump (one shot before per-agent rebuilds, commits if the lock changed).

meta::sync_agents(hyperhive_flake, dashboard_port, &agents) is the idempotent reconciler called by spawn, destroy, rebuild, and the startup migration. Renders flake.nix from the agent list; if it differs from disk, runs nix flake lock + commits as regenerate meta flake (or seed meta from N agent(s) on the very first call).

The manager has /meta RO-bound inside its container: git -C /meta log --oneline is the swarm-wide deploy log, cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked' resolves which sha each agent is pinned at right now. Dashboard surfaces the same info as a deployed:<sha12> chip per container row.

Two repos per agent

/var/lib/hyperhive/agents/<name>/config/    proposed — manager RW
└── <anything>                              # any files the manager
                                            # wants in the commit.
                                            # agent.nix is the
                                            # convention entry
                                            # point; flake.nix is
                                            # tracked boilerplate
                                            # (manager doesn't edit
                                            # it).

/var/lib/hyperhive/applied/<name>/          applied — core-only
├── .git/                                   # tag-rich history
├── flake.nix                               # tracked, fixed
│                                           # boilerplate exporting
│                                           # nixosModules.default
├── agent.nix                               # working tree of main
└── <other manager files>                   # also tracked

/var/lib/hyperhive/meta/                    swarm-wide flake — core
├── .git/                                   # one commit per successful
│                                           # deploy
├── flake.nix                               # generated from agent set
└── flake.lock                              # pins each agent's sha

Why two physical repos: the manager's /agents/<n>/config/ is RW — a buggy or hostile agent can git clean -fdx its own proposed tree. The applied repo is never bind-mounted (except the read-only .git exposure described below) so a destructive move inside the container cannot reach it.

The container's --flake ref is /var/lib/hyperhive/meta#<name> (see "Meta flake" above). The agent's own applied/<n>/flake.nix is a fixed boilerplate that exports nixosModules.default = import ./agent.nix; the meta flake imports that module and wraps it with identity + HIVE_PORT / HIVE_LABEL / HIVE_DASHBOARD_PORT.

Tag state machine

Every approval id walks through a fixed set of tags on the underlying commit inside the applied repo:

Tag	When	Annotated?
proposal/<id>	request_apply_commit, after fetch	no
approved/<id>	operator approve	no
building/<id>	rebuild started	no
deployed/<id>	rebuild succeeded — main ff's here	no
failed/<id>	rebuild failed	yes (body = error)
denied/<id>	operator deny	yes (body = operator note)

applied/main is always the latest deployed/*. denied/ and failed/ are terminal; the manager submits a new commit + new approval id to retry. Because tags are first-class git objects, rejected and failed trees stay browsable forever — git log --tags in the applied repo is the audit trail.

Dispatch via rebuild_queue

Long-running approval work — ApplyCommit, UpdateMetaInputs, Spawn — no longer runs inline inside actions::approve. Instead the approval handler enqueues a QueueEntry into the global rebuild_queue:

ApprovalKind	QueueKind queued	QueueSource
ApplyCommit	Rebuild	Approval
UpdateMetaInputs	MetaUpdate	Approval
Spawn	Spawn	Approval
InitConfig	— runs inline (sub-second git seed)	—
SchedulePrompt	— runs inline (single sqlite insert)	—

Each queue entry carries the originating approval_id so the worker can re-fetch the approval row when it dispatches, run the kind-specific pipeline (run_approval_apply_commit / run_approval_update_meta_inputs / run_approval_spawn), and fire the matching HelperEvent::* on completion via finish_approval.

Two visible consequences:

• Operator dashboard: after clicking APPR0VE the work-in-progress shows up on the rebuild queue card (POST /api/state.rebuild_queue
  • live rebuild_queue_changed events), not on the approvals panel (which already moved the row to "approved"). A long meta-update cascade renders as a parent entry with one child per per-agent rebuild — see docs/web-ui.md for the layout.
• Cancellation: the dashboard's × cancel button on a Queued entry calls POST /api/rebuild-queue/{id}/cancel, which flips the entry to Cancelled before the worker dispatches it. Returns {"cancelled": true} on success, {"cancelled": false} if the entry already left Queued (running / done / failed) — terminal states can't be retroactively rewritten.

QueueSource::Approval carries the approval_id so a tail-end build failure surfaces back as a failed approval row, not just a silent queue entry. QueueSource::Manual (dashboard ↻ R3BU1LD) and QueueSource::AutoUpdate (boot-time sweep) use the same queue but skip the approval row plumbing.

Forge mirror

When the bundled hive-forge container is running — on by default, hyperhive.forge.enable — hive-c0re mirrors every agent's applied repo into a private agent-configs Forgejo org. forge::push_config(<name>) pushes applied/main plus every tag to agent-configs/<name> after each ref mutation: the spawn that seeds deployed/0, every request_apply_commit (which plants proposal/<id>), every approve / deny, and a sweep at startup. Pushes are best-effort — a missing or stopped forge never blocks a deploy.

The org is private and agents are not members, so only the core user (a Forgejo site admin) can read it: an agent can't reach another agent's config — or even its own — through the forge. The tokenised push URL is passed inline to git push, never written into applied/<n>/.git/config; that repo is RO-bind-mounted into the manager, and a stored token would leak core's admin credential to an agent.

The dashboard deep-links into this org — a config repo link per container row and a commit on forge link per approval card. See docs/web-ui.md.

Manager view of applied + meta

The manager container gets three host-side bind mounts via set_nspawn_flags:

• /var/lib/hyperhive/agents/ → /agents/ (RW) — proposed repos. Manager edits + commits per-agent config here.
• /var/lib/hyperhive/applied/ → /applied/ (RO) — every agent's authoritative applied repo, including .git.
• /var/lib/hyperhive/meta/ → /meta/ (RO) — the swarm-wide deploy flake.

Each proposed repo (/agents/<n>/config/) is pre-configured with applied as a git remote pointing at /applied/<n>/.git. Useful incantations from inside the manager:

git -C /agents/<n>/config fetch applied
git -C /agents/<n>/config log applied/main --oneline
git -C /agents/<n>/config show applied/refs/tags/deployed/<id>
git -C /agents/<n>/config show applied/refs/tags/failed/<id>   # body = build error
git -C /agents/<n>/config show applied/refs/tags/denied/<id>   # body = operator note
git -C /agents/<n>/config rebase applied/main                 # base in-flight work on what's deployed

git -C /meta log --oneline                                    # swarm-wide deploy history
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'

The RO binds block push at the kernel level, so the manager can only fetch / read — git plumbing inside the container cannot corrupt either authoritative repo.

Migration from the pre-tag / pre-meta schemes

Both overhauls (tag-driven flow + meta flake) ship in-place migrations that run on every hive-c0re startup. Idempotent; each phase is a no-op once already applied. Behaviour:

• Tag-driven phase: assumes the operator ran the one-shot git tag deployed/0 main script (see commit history / earlier docs revisions) once per agent. Tagging is non-destructive: it doesn't touch live containers, state dirs, or claude creds.
• Meta-flake phase: rewrites each applied/<n>/flake.nix to the module-only boilerplate, wires the applied remote in each proposed repo, bootstraps the meta repo from the current agent list, and nixos-container updates every container at meta#<n>. The expensive last step is guarded by /var/lib/hyperhive/.meta-migration-done so it only runs once across hive-c0re restarts. Set HIVE_SKIP_META_MIGRATION=1 on the service to defer.

No state loss in either migration. claude creds, /state/ notes, the events DB, proposed history, and applied history all survive. The manager keeps its session; sub-agents stay logged in.

Manager (hm1nd) is hive-c0re-managed

The manager container runs through the same lifecycle as sub-agents. On hive-c0re serve startup, if hm1nd is missing, hive-c0re creates it. The manager's flake lives at /var/lib/hyperhive/applied/hm1nd/; its proposed config at /var/lib/hyperhive/agents/hm1nd/config/. Manager can edit its own agent.nix (visible inside the container at /agents/hm1nd/config/) and submit request_apply_commit("hm1nd", <sha>) for operator approval.

Differences from sub-agents:

• flake.nix extends hyperhive.nixosConfigurations.manager (vs agent-base).
• Container name is hm1nd (no h- prefix).
• Web UI port via lifecycle::agent_web_port("hm1nd") — same FNV-1a hash as every other agent (8100..8999 range).
• set_nspawn_flags adds two extra binds: /var/lib/hyperhive/agents → /agents (RW) so the manager can edit per-agent proposed repos, and /var/lib/hyperhive/applied → /applied (RO) so the manager can git fetch deployed/failed/denied tags from any agent's authoritative applied repo (see "Manager view of applied" below).
• First-deploy spawn bypasses the approval queue (manager is required infrastructure).
• Per-agent socket lives at /run/hyperhive/manager/, owned by manager_server::start.

Migration note (for older hosts): drop any containers.hm1nd = { ... } block from your host NixOS config. hyperhive creates and updates the manager itself.

Manager policy

From hive-ag3nt/prompts/system.md (<!-- role:manager --> block, rendered via hive_ag3nt::prompt::render): the manager does NOT rubber-stamp sub-agent config requests. It verifies (role match, package legitimacy, cheaper alternative, blast radius) before committing and calling request_apply_commit.

For ambiguous cases or anything that needs human signal, the manager calls ask(question, options?, multi?, ttl_seconds?, to?) — queues the question and returns the id immediately. When to is omitted (or "operator") the question shows up on the dashboard; when to is a sub-agent's name, the recipient receives a HelperEvent::QuestionAsked and answers via their own answer tool. Either way the answer arrives back as HelperEvent::QuestionAnswered { id, question, answer, answerer } in the asker's inbox. Storage is hive-c0re::operator_questions (sqlite) — same table, with a nullable target column (NULL = operator). Dispatch goes through hive-c0re/src/questions.rs::{handle_ask, handle_answer} so both the agent + manager surfaces stay aligned. The answer flow is:

POST /answer-question/{id}                       agent: Answer { id, answer }
  → OperatorQuestions::answer(_, _, "operator")    → questions::handle_answer
  → notify_agent(asker, QuestionAnswered {         → OperatorQuestions::answer(_, _, agent)
       answerer: "operator", ... })                → notify_agent(asker, QuestionAnswered {
                                                       answerer: agent, ... })

Two more paths resolve a pending question with a sentinel answer:

• POST /cancel-question/{id} (✗ CANC3L button on the dashboard) resolves with [cancelled]. The manager sees a terminal state and can fall back.
• ttl_seconds deadline: a tokio watchdog spawned at submit time fires answer(id, "[expired]") once the ttl runs out. Already- resolved races no-op. The dashboard surfaces a ⏳ MM:SS chip on each pending question with a deadline.

Helper events to the manager

Coordinator::notify_manager(&HelperEvent) enqueues an inbox message from sender system with the event JSON in the body. The manager harness no longer short-circuits these — they drive a regular claude turn so the manager can react. Variants (hive_sh4re::HelperEvent):

• ApprovalResolved { id, agent, commit_ref, status, note } — fired by actions::approve + actions::deny whenever an approval transitions to its terminal state.
• Spawned { agent, ok, note } — actions::approve (first-time ApplyCommit-kind) + admin HostRequest::Spawn (deprecated).
• Rebuilt { agent, ok, note } — auto_update::rebuild_agent (covers startup scan + manual /rebuild from dashboard) + actions::approve (ApplyCommit).
• Killed { agent } — admin HostRequest::Kill + dashboard /kill + manager Kill MCP tool.
• Destroyed { agent } — actions::destroy.
• ContainerCrash { agent, note } — crash_watch: a previously- running container went away with no operator-initiated transient state (Stopping / Restarting / Destroying / Rebuilding) AND no such transient was cleared in the last 30s (RECENT_TRANSIENT_GRACE tombstone, three POLL_INTERVALs — closes the race where a lifecycle op finishes between two crash-watch polls and the container shows briefly as "stopped without transient" before the next start). Manager can start it again or escalate.
• NeedsLogin { agent } — sub-agent has no claude session yet. Manager can't act directly (interactive OAuth); typically flags the operator.
• LoggedIn { agent } — sub-agent just completed login. Manager often greets the agent on this event.
• ConfigReady { agent } — a new agent's proposed config repo was just seeded (post-InitConfig approval). The manager can now edit /agents/<agent>/config/agent.nix, commit the changes, and submit request_apply_commit with the commit sha to create the container (first ApplyCommit also triggers spawn bookkeeping).
• NeedsUpdate { agent } — sub-agent's recorded flake rev is stale. Manager calls update(name) to rebuild — idempotent, no approval required.
• QuestionAnswered { id, question, answer, answerer } — dashboard /answer-question/{id} (answerer = "operator"), peer Answer request (answerer = agent name), or ttl watchdog expiry (answerer = "ttl-watchdog", answer = "[expired]").
• QuestionAsked { id, asker, question, options, multi } — fired when an agent calls Ask { to: Some(<this-agent>), ... }. The recipient responds via Answer { id, answer } and the asker sees the matching QuestionAnswered.

To add a new event: new HelperEvent variant + call sites + update prompts/system.md (<!-- role:manager --> block, the lifecycle- event list) so the manager knows the new shape.

Auto-update on startup

hive-c0re serve runs auto_update::run in a background task right after opening the coordinator. It enumerates managed containers and rebuilds any whose recorded hyperhive rev differs from the current one — sub-agents and manager go through the same lifecycle::rebuild path.

"Rev" = canonical filesystem path of cfg.hyperhiveFlake. Marker file: /var/lib/hyperhive/applied/.<name>.hyperhive-rev. If the flake input has no canonical path (e.g. a github: URL), auto-update is a no-op — rebuild manually.

The dashboard surfaces pending updates per agent: a clickable "needs update ↻" badge appears whenever the marker differs from current rev. The badge POSTs /rebuild/<name>, calling the same auto_update::rebuild_agent path so manual triggers and the startup scan can't drift. When at least one container is stale, a top-level ↻ UPD4TE 4LL button appears that loops over every stale container.