coordinator.md rewrites the queue section (node inventory, DAG shapes, resources, desired-state reconciliation, boot reconcile); approvals.md + persistence.md + hivectl --graceful help updated to match. agent_power lives in broker.sqlite like approvals/questions (own connection + busy timeout) instead of a separate db file.
32 KiB
Approvals + helper events
The approval queue is hyperhive's pivot: nothing that changes the
shape of an agent (its config, whether it exists) happens without an
operator click. The submitting agent — any agent with the approvals
tool group, which manages the config of its direct children (the
root agent for top-level agents; a sub-manager for its own subtree) — is
the policy gate in front of that queue; helper events are how it stays
informed about what happens after a decision lands.
End-to-end approval flow
- The submitting agent (the child's parent, holding the
approvalstool group) edits files in the child's proposed config repo (any tracked path, butagent.nixis the contract entry point) and commits with its own git identity. The parent's container has the child's proposed config repo bind-mounted read-write at/agents/<name>/config/(topology-driven viaset_nspawn_flags; the agent's own config at/agents/<self>/config/is read-only). - The submitting agent submits the commit sha via
request_apply_commit(agent, commit_ref).commit_refmust be a commit sha (7-40 hex chars, short or full) — a branch or tag name is rejected so the approval pins an immutable commit. - hive-c0re immediately fetches that commit from the proposed
repo into the applied repo and tags it
proposal/<id>. It resolves the sha locally against the proposed repo, fetches all of proposed's heads into applied's object db, then tags the resolved commit —git fetch <remote> <sha>:<dst>can't fetch by a bare sha (the left side of a refspec is a remote ref name), so the resolution happens on hive-c0re's side. The approval row stores both the submitted sha and the canonical hive-c0re-vouched sha. From here on the proposed repo is irrelevant for this approval — the submitter can amend, force-push, orrm -rfthe proposed repo and the queued approval still points at an immutable git object inside applied. 3a. Flake validation (ApplyCommit only): after the proposal tag is planted, hive-c0re readsproposal/<id>:flake.lockand runs two checks. If either check fails, no pending approval is created for the operator — the row is marked failed and surfaces on the dashboard with the validation message:- Stale lock — materialises the commit in a temp worktree,
runs
nix flake lock(no--update-inputflags, so it only fills missing entries), and rejects if the committedflake.lockdiffers from the result. Triggered when the submitter added or removedinputsinflake.nixwithout re-runningnix flake lock. Fix: runnix flake lockin the config repo, commit, and re-submit. - Duplicate inputs — groups lock nodes by their canonical
originalfield; rejects if two or more nodes share the same source. This usually means an input is missinginputs.<x>.inputs.nixpkgs.follows = "nixpkgs". Fix: add thefollowsdirective, re-lock, and re-submit. Both checks only flag new violations — agents whose lock already carried duplicates before this check was added are unaffected until a coordinated config-change pass.
- Stale lock — materialises the commit in a temp worktree,
runs
- Operator sees the proposal as a card on the dashboard — a
full multi-file diff, toggleable between three bases (vs the
running tree / vs the last approved proposal / vs the
previous queued proposal) — and clicks ◆ APPR0VE (or
hive-c0re approve <id>on the CLI). - hive-c0re moves the working tree to
proposal/<id>and runs the build under a sequence of tags (see below). On success,applied/mainfast-forwards to the proposal commit. On failure, main stays put and the working tree resets back to the previous deployed commit. HelperEvent::ApprovalResolved(andRebuiltfor the ApplyCommit kind) land in the submitting agent's inbox, carrying both the canonical sha and the terminal tag. Helper events route to the submitting agent vianotify_submitter(the approval row carries asubmittercolumn recording which agent calledrequest_apply_commitorrequest_init_config).
Withdrawing a pending approval
The submitting agent can call cancel_loose_end(kind: "approval", id) to
withdraw an approval that hasn't been acted on yet.
The row transitions to ApprovalStatus::Cancelled (distinct from
Denied/Failed), the dashboard pulls the card out of the
pending pane, and ApprovalResolved { status: "cancelled" } fires
on the root agent + dashboard channels. Approvals that have already
been approved/denied/failed return an error — the resolution is
final once the operator (or a lifecycle failure) acted on the row.
The socket refuses the approval kind with a clear error for any
agent that lacks the approvals tool group: only an agent with that
group submits approvals (for its direct children), so an agent
without it has nothing of its own to withdraw.
InitConfig approvals are the first step in a two-step spawn
flow. On approve, hive-c0re seeds the proposed config repo with
a default agent.nix template and sends HelperEvent::ConfigReady { agent }
to the submitting agent's inbox via notify_submitter. The submitting
agent then reviews,
edits, and commits the template before calling request_apply_commit
to proceed to an ApplyCommit approval. The first ApplyCommit
creates the container; subsequent ones rebuild it with new config.
This gives the submitting agent (and operator) an explicit review gate on the
initial configuration before any container is created.
Approval kinds (wire shapes)
ApprovalKind carries six variants; each maps to a different
commit_ref encoding because that field is overloaded as the
kind-specific payload carrier.
ApplyCommit—commit_refis the submitted git sha (7-40 hex chars). The canonical, hive-c0re-vouched sha after the proposal fetch lives infetched_shaon the sameApprovalrow (onlyApplyCommitpopulates it). See the End-to-end flow above.MergeConfigPr— the PR-based config flow's counterpart toApplyCommit.commit_refstores the PR number (decimal), andfetched_shais the PR head sha the operator reviewed. On approve,run_merge_config_prre-reads the live PR head and aborts if it drifted fromfetched_sha(re-review), then fetches that head into the applied repo, eval-verifies it, fast-forwards the forge config repo'smainto it (the merge), marks the PR merged (best-effort —mainis already there), and runs the same shared deploy tail asApplyCommit(deploy_applied_target). Never a first spawn.Spawn— direct container creation under the defaultagent.nixtemplate.commit_refis empty. Submitted viaHostRequest::RequestSpawn(operator-gated, the◆ R3QU3ST SP4WNdashboard button +hive-c0re request-spawnCLI). The host-levelHostRequest::Spawnvariant bypasses the approval queue entirely — privileged-context use only (operator on the host shell, test scripts, one-off recoveries). The agent-sideRequestSpawnis gone; the submitting agent goes through theInitConfig→ApplyCommittwo-step instead so the spawn captures the customised config.InitConfig—commit_refis empty; the variant just gates "seed the proposed repo with the default template" against operator approval. Step 1 of the two-step spawn flow above.UpdateMetaInputs—commit_refstores the JSON-encoded inputs array ("[]"= all inputs,"[\"nixpkgs\"]"= just nixpkgs, etc.).agentfield is set to the requesting root agent. On approve hive-c0re runsnix flake update [inputs...]on the meta flake and commits the resulting lock changes.SchedulePrompt—commit_refstores the JSON-encodedSchedulePromptPayload(target list, body, schedule) so the approval row carries the full submission verbatim. On approve hive-c0re inserts a row intoscheduled_promptswithsource = approval:<id>; the worker fans the body out as inbox messages to each target at the scheduled time, recurring wheninterval_secondsis set.
Scheduled prompts (submit paths)
Two ways a row lands in scheduled_prompts:
- Operator-direct (
source = "operator"): the operator adds a schedule through the dashboard form. Lands in the table immediately, no approval gate — operator action is already the trust boundary. - Agent-requested (
source = "approval:<id>"): an agent submits aRequestSchedulePromptthrough its MCP socket (therequest_schedule_prompttool,schedulinggroup). AnApprovalKind::SchedulePromptrow is queued; on approve, hive-c0re inserts the schedule row withsource = approval:<id>so the audit trail points back at the operator decision (above).
No self-target shortcut: even agent-self schedules need approval. The existing remind MCP tool stays the quick self-wake path (no approval, lands directly in the agent's own inbox); this module is the bigger, multi-recipient, operator-visible thing.
Scheduled prompt worker (catch-up clamp)
When hive-c0re comes back from being down, the worker sees rows whose next_fire_at_unix is well in the past. For recurring rows that would mean firing N delayed pulses in a row — spammy and useless. Instead the worker fires once per row and bumps next_fire_at_unix to the next interval slot ≥ now, recording how many cycles were skipped in last_result (per-target). Operators see "fired late, caught up from 17 skipped" instead of 17 wake-up storms.
One-shot rows fire once (if past due, on the next worker pass) and are deleted by the worker; recurring rows survive until cancelled.
targets is its own table (scheduled_prompt_targets) so partial cancellation flips a single row and the dashboard can show last-fired / last-result per recipient. Cancelling every target reaps the parent row on the next worker pass.
Missing-target failure
When a target name doesn't resolve to a known agent (container destroyed, operator typo, etc.) the worker:
- Records
last_result = "no such agent: <name>"on the per-target row. - Sends a single advisory
Messagefromsystemtooperatornaming the schedule, target, and reason. - Continues fanning out to the other live targets.
Transient broker errors (sqlite lock contention, etc.) get the same
last_result annotation plus a tracing::warn, and then:
- Recurring rows re-arm to the next interval slot — the retry self-heals on the next worker pass.
- One-shot rows are deleted unconditionally after their single
fan-out pass; a broker error on a one-shot is not retried (the
operator advisory and
last_resultare the only audit trail).
Reminder delivery: file-path semantics
A reminder may carry a file_path (the agent-visible path inside its
container, e.g. /agents/<name>/state/foo.md). On delivery hive-c0re:
- Translates the container path to the host path
(
/var/lib/hyperhive/agents/<name>/state/foo.md) so c0re can write from outside the container. - Validates the path: rejects anything outside the agent's own state
subtree, containing
..(path traversal), or with an empty relative tail. On rejection the write is skipped and the original message is delivered inline with a warning — the reminder still fires. - Defends against symlink escape: after
create_dir_all, the parent dir is canonicalized and re-verified to live under the agent's host state root. The final file is opened withO_NOFOLLOW | O_CREAT | O_TRUNCso an existing symlink at the basename cannot redirect the write to an arbitrary host path. - Writes the body to disk and delivers a short pointer message in its place, keeping the agent's inbox / wake-prompt small while the bulky payload is read out of band.
Atomicity of the inbox INSERT + reminders.sent_at UPDATE is handled
inside Broker::deliver_reminders_batch; the scheduler only computes the
body strings before calling it.
Destroy semantics
HostRequest::Destroy { name, purge } is the lifecycle tear-down,
not an approval. Stops + removes the nspawn container, drops the
systemd drop-in, fails any pending approvals. Persistent state
(proposed/applied repos, claude credentials, /state/ notes) is
kept by default — recreating the agent with the same name
reuses prior config + login. With purge = true the agent's
/var/lib/hyperhive/{agents,applied}/<name>/ trees are also
wiped (config history + creds + notes gone forever). The
root/bootstrap container is destroyable like any other — hive-c0re
recreates it on the next startup if it's absent, so destroying it is
transient.
Meta flake
The hive-c0re-owned repo at /var/lib/hyperhive/meta/
declares one flake input per agent (agent-<n>.url = "git+file:///var/lib/hyperhive/applied/<n>") and one
nixosConfigurations.<n> output per agent. Each output wraps
inputs.agent-<n>.nixosModules.default with the identity +
HIVE_PORT / HIVE_LABEL / HIVE_DASHBOARD_PORT injection
module that setup_applied used to generate inline.
Containers run against --flake /var/lib/hyperhive/meta#<n>.
Per-deploy lock flow (two-phase, owned by
actions::run_apply_commit → meta::{prepare,finalize,abort} _deploy):
meta::prepare_deploy(name)runsnix flake lock --update-input agent-<n>without committing. Working tree of meta now points the input atapplied/<n>/main(whichrun_apply_commitalready fast-forwarded toproposal/<id>).lifecycle::rebuild_no_metarunsnixos-container update <c> --flake meta#<name>. Nix evaluates against the staged lock.- On success —
meta::finalize_deploy(name, sha, "deployed/ <id>")stagesflake.lockand commits withdeploy <n> deployed/<id> <sha12>. Meta's git log gains one entry per successful deploy. - On failure —
meta::abort_deploy()runsgit restore flake.lockso the meta history shows only successes; the failure stays as an annotatedfailed/<id>tag inapplied/<n>.
Single-phase variants exist for paths without
rollback semantics: meta::lock_update_for_rebuild(name) for
the manual ↻ R3BU1LD button (commits if the lock changed)
and meta::lock_update_hyperhive() for the
auto-update flake-rev bump (one shot before per-agent
rebuilds, commits if the lock changed).
meta::sync_agents(hyperhive_flake, dashboard_port, &agents)
is the idempotent reconciler called by spawn, destroy,
rebuild, and the startup migration. Renders flake.nix
from the agent list; if it differs from disk, runs
nix flake lock + commits as regenerate meta flake (or
seed meta from N agent(s) on the very first call).
The root agent has /meta RO-bound inside its container:
git -C /meta log --oneline is the swarm-wide deploy log,
cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'
resolves which sha each agent is pinned at right now.
Dashboard surfaces the same info as a deployed:<sha12> chip
per container row.
Two repos per agent
/var/lib/hyperhive/agents/<name>/config/ proposed — submitting agent RW
└── <anything> # any files the submitting
# agent wants in the commit.
# agent.nix is the
# convention entry
# point; flake.nix is
# tracked boilerplate
# (submitting agent doesn't
# edit it).
/var/lib/hyperhive/applied/<name>/ applied — core-only
├── .git/ # tag-rich history
├── flake.nix # tracked, fixed
│ # boilerplate exporting
│ # nixosModules.default
├── agent.nix # working tree of main
└── <other committed files> # also tracked
/var/lib/hyperhive/meta/ swarm-wide flake — core
├── .git/ # one commit per successful
│ # deploy
├── flake.nix # generated from agent set
└── flake.lock # pins each agent's sha
Why two physical repos: the submitting agent's /agents/<n>/config/ is
RW — a buggy or hostile agent can git clean -fdx its own
proposed tree. The applied repo is never bind-mounted (except
the read-only .git exposure described below) so a destructive
move inside the container cannot reach it.
The container's --flake ref is /var/lib/hyperhive/meta#<name>
(see "Meta flake" above). The agent's own applied/<n>/flake.nix
is a fixed boilerplate that exports nixosModules.default = import ./agent.nix; the meta flake imports that module and
wraps it with identity + HIVE_PORT / HIVE_LABEL /
HIVE_DASHBOARD_PORT.
Tag state machine
Every approval id walks through a fixed set of tags on the underlying commit inside the applied repo:
| Tag | When | Annotated? |
|---|---|---|
proposal/<id> |
request_apply_commit, after fetch | no |
approved/<id> |
operator approve | no |
building/<id> |
rebuild started | no |
deployed/<id> |
rebuild succeeded — main ff's here |
no |
failed/<id> |
rebuild failed | yes (body = error) |
denied/<id> |
operator deny | yes (body = operator note) |
applied/main is always the latest deployed/*. denied/ and
failed/ are terminal; the submitting agent submits a new commit + new
approval id to retry. Because tags are first-class git objects,
rejected and failed trees stay browsable forever — git log --tags in the applied repo is the audit trail.
Dispatch via the job queue
Long-running approval work — ApplyCommit, UpdateMetaInputs,
Spawn — no longer runs inline inside actions::approve. Instead
the approval handler submits a DAG to the global job queue
(docs/coordinator.md::Job queue):
ApprovalKind |
DAG submitted | source |
|---|---|---|
ApplyCommit |
rebuild (single opaque ApprovalDeploy node) |
approval |
MergeConfigPr |
rebuild (single opaque ApprovalDeploy node) |
approval |
UpdateMetaInputs |
meta_update (MetaLock + rebuild fan-out) |
approval |
Spawn |
spawn (Create → WriteDropin → Reconcile) |
approval |
InitConfig |
— runs inline (sub-second git seed) | — |
SchedulePrompt |
— runs inline (single sqlite insert) | — |
The DAG carries the originating approval_id. The ApprovalDeploy
node runs the kind-specific pipeline (run_approval_apply_commit /
run_approval_merge_config_pr — the two-phase meta deploy stays
inside actions.rs) and fires the matching HelperEvent::* via
finish_approval itself; Spawn and UpdateMetaInputs DAGs resolve
through actions::resolve_approval_dag when the DAG settles terminal
(a spawn additionally runs the post-spawn forge bookkeeping there).
Two visible consequences:
- Operator dashboard: after clicking APPR0VE the work-in-progress
shows up on the rebuild queue card (
/api/state.rebuild_queue- live
rebuild_queue_changedevents), not on the approvals panel (which already moved the row to "approved"). A long meta-update cascade renders as a parent DAG with one child rebuild per affected agent — seedocs/web-ui.mdfor the layout.
- live
- Cancellation: the dashboard's × cancel button on a still-queued
DAG calls
POST /api/rebuild-queue/{id}/cancel, which flips it toCancelledbefore any node runs (and fails the approval row instead of leaving it dangling). Returns{"cancelled": true}on success,{"cancelled": false}once any node started — terminal states can't be retroactively rewritten.
The approval source + approval_id mean a tail-end build failure
surfaces back as a failed approval row, not just a silent queue
entry. manual (dashboard ↻ R3BU1LD) and auto_update (boot
reconcile) DAGs use the same queue but skip the approval plumbing.
Forge mirror
The bundled hive-forge container is mandatory (it deploys with
hyperhive), and hive-c0re mirrors every agent's applied repo into a
private agent-configs Forgejo org. forge::push_config(<name>) pushes applied/main plus
every tag to agent-configs/<name> after each ref mutation:
the spawn that seeds deployed/0, every request_apply_commit
(which plants proposal/<id>), every approve / deny, and a
sweep at startup. Pushes are best-effort — a missing or stopped
forge never blocks a deploy.
The org is private and agents are not members, so only the
core user (a Forgejo site admin) can read it: an agent can't
reach another agent's config — or even its own — through the
forge. The tokenised push URL is passed inline to git push,
never written into applied/<n>/.git/config; that repo is
RO-bind-mounted into the root agent, and a stored token would leak
core's admin credential to an agent.
The dashboard deep-links into this org — a config repo link
per container row and a commit on forge link per approval
card. See docs/web-ui.md.
Submitting agent's view of config repos
Every parent agent's container has its direct children's proposed
config repos bind-mounted read-write (topology-driven: lifecycle.rs
calls bind_child_agent_dirs for each entry in
topology::children_of(agent_name)). An agent with the approvals
tool group can therefore edit, commit, and submit changes for any of
its direct children directly inside its container at /agents/<child>/config/.
Agents holding the can_manage_top_level_agents topology role
(defined as ROLE_CAN_MANAGE_TOP_LEVEL_AGENTS in hive-c0re/src/topology.rs)
get additional host-side bind mounts via set_nspawn_flags:
/var/lib/hyperhive/agents/→/agents/(RW) — all top-level agents' proposed repos (not just direct children)./var/lib/hyperhive/applied/→/applied/(RO) — every agent's authoritative applied repo, including.git./var/lib/hyperhive/meta/→/meta/(RO) — the swarm-wide deploy flake.
The root agent holds this role; a sub-manager that only manages a subtree does not, and only has its direct children's config dirs.
Each proposed repo (/agents/<n>/config/) is pre-configured
with applied as a git remote pointing at
/applied/<n>/.git. Useful incantations from inside an agent with
the full /applied mount:
git -C /agents/<n>/config fetch applied
git -C /agents/<n>/config log applied/main --oneline
git -C /agents/<n>/config show applied/refs/tags/deployed/<id>
git -C /agents/<n>/config show applied/refs/tags/failed/<id> # body = build error
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
git -C /meta log --oneline # swarm-wide deploy history
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
The RO binds block push at the kernel level — git plumbing inside the container cannot corrupt either authoritative repo.
Migration from the pre-tag / pre-meta schemes
Both overhauls (tag-driven flow + meta flake) ship in-place migrations that run on every hive-c0re startup. Idempotent; each phase is a no-op once already applied. Behaviour:
- Tag-driven phase: assumes the operator ran the one-shot
git tag deployed/0 mainscript (see commit history / earlier docs revisions) once per agent. Tagging is non-destructive: it doesn't touch live containers, state dirs, or claude creds. - Meta-flake phase: rewrites each
applied/<n>/flake.nixto the module-only boilerplate, wires theappliedremote in each proposed repo, bootstraps the meta repo from the current agent list, andnixos-container updates every container atmeta#<n>. The expensive last step is guarded by/var/lib/hyperhive/.meta-migration-doneso it only runs once across hive-c0re restarts. SetHIVE_SKIP_META_MIGRATION=1on the service to defer.
No state loss in either migration. claude creds, /state/ notes, the events DB, proposed history, and applied history all survive. The root agent keeps its session; sub-agents stay logged in.
The root/bootstrap container is hive-c0re-managed
The root agent container runs through the same lifecycle as
sub-agents. On hive-c0re serve startup, if ruth is missing,
hive-c0re creates it. The root agent's flake lives at
/var/lib/hyperhive/applied/ruth/; its proposed config at
/var/lib/hyperhive/agents/ruth/config/. The root agent can edit its own
agent.nix (visible inside the container at /agents/ruth/config/)
and submit request_apply_commit("ruth", <sha>) for operator
approval.
Differences from sub-agents:
flake.nixextendshyperhive.nixosConfigurations.manager(vsagent-base).- Web UI port via
lifecycle::agent_web_port("ruth")— same FNV-1a hash as every other agent (8100..8999 range). set_nspawn_flagsadds two extra binds:/var/lib/hyperhive/agents→/agents(RW) so the root agent can edit per-agent proposed repos, and/var/lib/hyperhive/applied→/applied(RO) so the root agent cangit fetchdeployed/failed/denied tags from any agent's authoritative applied repo (see "Root-agent view of applied" below).- First-deploy spawn bypasses the approval queue (the root agent is required infrastructure).
- Per-agent socket lives at
/run/hyperhive/manager/, owned bymanager_server::start.
Migration note (for older hosts): drop any containers.root = { ... } block from your host NixOS config. hyperhive creates and
updates the root agent itself.
Root-agent policy
The system prompt (hive-ag3nt/prompts/system.md, rendered via
hive_ag3nt::prompt::render) is the same for every agent; what
varies is which MCP tools are surfaced (gated by tool groups and
capabilities in agent.nix). There is no role:manager block that
renders only for the root agent. The root agent's approval-gating
behaviour comes from its CLAUDE.md / agent-specific instructions, not
the system prompt template.
ask(question, options?, multi?, ttl_seconds?, to?) is available to
any agent — it queues a question and returns the id immediately.
When to is omitted (or "operator") the question shows up on the
dashboard; when to is another agent's name, the recipient receives a
HelperEvent::QuestionAsked and answers via their own answer
tool. Either way the answer arrives back as
HelperEvent::QuestionAnswered { id, question, answer, answerer }
in the asker's inbox. Storage is hive-c0re::operator_questions
(sqlite) — same table, with a nullable target column
(NULL = operator). Dispatch goes through
hive-c0re/src/questions.rs::{handle_ask, handle_answer}. The answer flow is:
POST /answer-question/{id} agent: Answer { id, answer }
→ OperatorQuestions::answer(_, _, "operator") → questions::handle_answer
→ notify_agent(asker, QuestionAnswered { → OperatorQuestions::answer(_, _, agent)
answerer: "operator", ... }) → notify_agent(asker, QuestionAnswered {
answerer: agent, ... })
Two more paths resolve a pending question with a sentinel answer:
POST /cancel-question/{id}(✗ CANC3L button on the dashboard) resolves with[cancelled]. The asking agent sees a terminal state and can fall back.ttl_secondsdeadline: a tokio watchdog spawned at submit time firesanswer(id, "[expired]")once the ttl runs out. Already- resolved races no-op. The dashboard surfaces a⏳ MM:SSchip on each pending question with a deadline.
Helper events to the submitting agent
Coordinator::notify_submitter(approval_id, &HelperEvent) routes the
event to the agent that originally submitted the approval (looked up from
the submitter column on the approvals table). The harness delivers it
as a regular system inbox message so it drives a normal claude turn.
Legacy approval rows that predate the submitter column fall back to the
root agent. Variants (hive_sh4re::HelperEvent):
ApprovalResolved { id, agent, commit_ref, status, note }— fired byactions::approve+actions::denywhenever an approval transitions to its terminal state.Spawned { agent, ok, note }—actions::approve(first-time ApplyCommit-kind) + adminHostRequest::Spawn(deprecated).Rebuilt { agent, ok, note }—auto_update::rebuild_agent(covers startup scan + manual/rebuildfrom dashboard) +actions::approve(ApplyCommit).Killed { agent }— adminHostRequest::Kill+ dashboard/kill+ theKillMCP tool.Destroyed { agent }—actions::destroy.ContainerCrash { agent, note }—crash_watch: a previously- running container went away with no operator-initiated transient state (Stopping / Restarting / Destroying / Rebuilding) AND no such transient was cleared in the last 30s (RECENT_TRANSIENT_GRACEtombstone, threePOLL_INTERVALs — closes the race where a lifecycle op finishes between two crash-watch polls and the container shows briefly as "stopped without transient" before the next start). The root agent canstartit again or escalate.NeedsLogin { agent }— sub-agent has no claude session yet. The root agent can't act directly (interactive OAuth); typically flags the operator.LoggedIn { agent }— sub-agent just completed login. The root agent often greets the agent on this event.ConfigReady { agent }— a new agent's proposed config repo was just seeded (post-InitConfigapproval). The root agent can now edit/agents/<agent>/config/agent.nix, commit the changes, and submitrequest_apply_commitwith the commit sha to create the container (first ApplyCommit also triggers spawn bookkeeping).NeedsUpdate { agent }— sub-agent's recorded flake rev is stale. The root agent callsupdate(name)to rebuild — idempotent, no approval required.QuestionAnswered { id, question, answer, answerer }— dashboard/answer-question/{id}(answerer ="operator"), peerAnswerrequest (answerer = agent name), or ttl watchdog expiry (answerer ="ttl-watchdog", answer ="[expired]").QuestionAsked { id, asker, question, options, multi }— fired when an agent callsAsk { to: Some(<this-agent>), ... }. The recipient responds viaAnswer { id, answer }and the asker sees the matchingQuestionAnswered.
Optional sha field on ApprovalResolved, Spawned, and Rebuilt
carries the canonical hive-c0re-vouched commit sha. Optional tag
on ApprovalResolved and Rebuilt only — the spawn path always
lands at deployed/0, so the tag is implicit and not echoed. The
tag values for the variants that do carry it: deployed/<id> /
failed/<id> / denied/<id> for approval-driven flows;
approved/<id> for the rare bare-approval case where no underlying
action runs. Both fields are Option: None on the rebuild paths
that don't change the deployed commit (e.g.
auto_update::rebuild_agent reapplying the existing main, or the
dashboard ↻ R3BU1LD button when the lock didn't move). When set,
git show <sha> against /agents/<n>/applied.git inside the
bootstrap container yields the exact tree that was referenced.
To add a new event: new HelperEvent variant + call sites + update
prompts/system.md (<!-- role:manager --> block, the lifecycle-
event list) so the root agent knows the new shape.
Auto-update on startup
hive-c0re serve runs auto_update::run in a background task right
after opening the coordinator. It enumerates managed containers and
rebuilds any whose recorded hyperhive rev differs from the current
one — sub-agents and the root agent go through the same lifecycle::rebuild
path.
"Rev" = canonical filesystem path of cfg.hyperhiveFlake. Marker
file: /var/lib/hyperhive/applied/.<name>.hyperhive-rev. If the
flake input has no canonical path (e.g. a github: URL),
auto-update is a no-op — rebuild manually.
The dashboard surfaces pending updates per agent: a clickable
"needs update ↻" badge appears whenever the marker differs from
current rev. The badge POSTs /api/rebuild/<name>, calling the same
auto_update::rebuild_agent path so manual triggers and the
startup scan can't drift. When at least one container is stale, a
top-level ↻ UPD4TE 4LL button appears that loops over every
stale container.