hyperhive/docs/agent-lifecycle/config-changes.md

Config changes + helper events

Config changes are pull requests on the agent's config repo, which an operator merges on the forge; that merge deploys them. Helper events are how agents stay informed about lifecycle changes that need a reaction.

For operators

• Config change — an agent proposed a change to another agent's config (or its own, via a sub-manager) as a pull request on agent-configs/<agent>. Review it on the forge like any other PR, and merge it there to deploy it. See Config changes.

Everything below this point is the implementation detail behind that flow.

Config changes

Config changes flow through a forge pull request on the agent's agent-configs/<name> repo — the same surface agents use for code PRs. No MCP tool exists for config changes: opening the PR IS the request.

1. The submitting agent clones agent-configs/<name>, edits it there (any tracked path, but agent.nix is the contract entry point), commits with its own git identity, and pushes a branch + opens a PR with hive-forge — the same way it would change any other repo. The container has no config mount: the clone is how an agent reads its own config as well as how it changes it. Branch protection (the agent isn't on the main push allowlist; merge and approvals allowlist = operators team; see "Forge config repos" below) makes the agent a write collaborator that can't merge its own config PR.
2. An operator reviews the PR on the forge (native diff, threaded comments, CI status) and merges it there. Merging needs membership of the operators team in agent-configs. The team starts empty: add each operator by hand in the forge UI.
3. swarm-controller gets the agent-configs org's pull_request delivery. A closed event with merged: true and base branch main names the commit in merge_commit_sha.
4. The controller looks up which hive's wanted state places the agent and publishes a deploy request carrying that commit on that hive's deploy subject. With no such hive, or more than one, it deploys nothing and logs a warning naming them.
5. If the hive's applied/main already is that commit, it does nothing. Otherwise it fetches the config repo's main. A commit that applied/main already contains is a no-op too: requests for two merges can arrive in either order. A commit that descends from applied/main is fast-forwarded to and a rebuild queued. A failure before the rebuild (the fetch, or a commit that has diverged from applied/main) deploys nothing and logs swarm events: rev not applied; nothing queued at error, which fires the Grafana alert config rev refused. An agent with no container on that hive yet ignores the commit; its first deploy builds what it seeds.

This path runs no eval-verify. A merged config that doesn't evaluate or build fails the rebuild, and applied/main stays at that commit, so the agent's rebuilds keep failing until a fix merges. A failed rebuild shows on the hive like any other. A merge whose webhook delivery never reaches the controller deploys nothing.

Creating a brand-new agent is swarm-level (swarmctl agent create, POST /api/agents on the swarm controller), and no hive can originate an agent. The swarm controller's InitAgentConfigRepo job seeds the agent's config repo with a default agent.nix template, then asks the target hive to deploy it. An agent created from an archetype gets no config repo: its hive builds it from the archetype's module.

Changing what the template seeded isn't an exception: like every later change, it's a PR on that config repo, made from a clone and merged by an operator on the forge (see Config changes).

Agent lifecycle

Reminder delivery: file-path semantics

A reminder may carry a file_path (the agent-visible path inside its container, for example /agents/<name>/state/foo.md). On delivery hive-c0re:

1. Translates the container path to the host path (/var/lib/hyperhive/agents/<name>/state/foo.md) so c0re can write from outside the container.
2. Validates the path: rejects anything outside the agent's own state subtree, containing .. (path traversal), or with an empty relative tail. On rejection hive-c0re skips the write and delivers the original message inline with a warning — the reminder still fires.
3. Defends against symlink escape: after create_dir_all, hive-c0re canonicalizes the parent dir and re-verifies it lives under the agent's host state root. hive-c0re opens the final file with O_NOFOLLOW | O_CREAT | O_TRUNC so an existing symlink at the basename can't redirect the write to an arbitrary host path.
4. Writes the body to disk and delivers a short pointer message in its place, keeping the agent's inbox / wake-prompt small while the agent reads the bulky payload out of band.

Broker::deliver_reminders_batch handles atomicity of the inbox INSERT + reminders.sent_at UPDATE; the scheduler only computes the body strings before calling it.

Destroy semantics

HostRequest::Destroy { name, purge } is the lifecycle tear-down. Stops + removes the nspawn container and drops the systemd drop-in. Persistent state (the applied repo, claude credentials, /state/ notes) is kept by default — recreating the agent with the same name reuses prior config + login. With purge = true the agent's /var/lib/hyperhive/{agents,applied}/<name>/ trees are also wiped (config history + creds + notes gone forever). The root/bootstrap container is destroyable like any other, and stays gone until you create ruth again from the swarm dashboard.

Meta flake

A hive has no meta flake of its own. It builds every agent from the swarm's internal/meta, which swarm-controller commits (below). It declares one flake input per agent (agent-<n>.url = "git+<forge>/agent-configs/<n>.git?rev=<sha>", the head of main on the agent's forge config repo, the one config PRs merge into) and one nixosConfigurations.<n> output per agent. Each output wraps inputs.agent-<n>.nixosModules.default with the identity + HIVE_PORT / HIVE_LABEL injection module. Values that differ per host (the dashboard port, the pinned claude-code, extra OTEL labels, the queue address) aren't in it: hive-priv writes them to the container's /etc/hyperhive-hive-facts at start (see docs/process/conventions.md). A config that sits only in the hive's applied/<n> mirror, not on forge main, never deploys.

What hive-c0re still keeps under /var/lib/hyperhive/meta/ is plain state, bound read-only into the root agent's container as /meta: topology.json, the swarm resource limits, and the permission seed files of a hive that predates the swarm's permissions. agent_set::refresh(agents: &[AgentSpec]) keeps topology.json, the gateway's agent-socket map and its nginx include in line with the live agents. provision_container, destroy, rebuild and the startup migration call it. It refuses an empty agent list, which means nixos-container list failed, and it fails when HIVE_FORGE_URL is unset, since every agent would come up with no forge.

The dashboard's deployed:<sha12> chip per container row shows the config commit the agent's last successful build pinned.

The swarm's meta flake

swarm-controller commits internal/meta: one flake.nix and flake.lock for every agent any hive places (the pins: docs/swarm/builder.md). The swarm builder builds every agent from it, and a hive deploys that build. While the repo has no commit that pins an agent's current config, that agent's deploy fails and names the agent. The controller queues its update of the repo before the deploy request for a merged config commit and for a new agent, and sends the deploy only when that update succeeded and the swarm builder built the agent from the updated repo (the deploy gate). A failed build skips the deploy, and the timer builds again.

• Choosing the commit. The rebuild's MetaSync node reads the repo's default-branch head with the hive's read token (forge::reader_token()) and takes it when its flake.nix builds the agent the way the agent needs now: a custom agent from the config commit at the head of agent-configs/<n> main, an archetype agent from the archetype it has now. An agent switched between custom and archetype, or to another archetype, fails that test. A repo with no commit, a head that lacks the agent and a head behind the config each fail the deploy with an error that names the agent. MetaSync records the commit, with the hyperhive and config commits the head pins, in hive-c0re's own state, /var/lib/hyperhive/meta-rev/<n>, which the agent can't write (the container mounts its runtime directory read-write); Prebuild and Swap read that record, so both expect the closure of one commit. A first deploy resolves the commit in its Create node, since it has no MetaSync.
• The closure. The hive builds nothing. Prebuild, Swap and Create read the agent's record in the swarm's agent-closures bucket (the controller writes it from the builder's reply) and have hive-priv download the closure it names from the swarm cache (RealiseClosure). Swap and Create then give hive-priv the store path (toplevel on UpdateContainer and CreateContainer), which accepts only the path its last RealiseClosure for that agent returned. On the wire the hive_priv_sock::PrivRequest variants carry a required field: {"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}. The three cases when the swarm's closure isn't the one this hive expects are in docs/swarm/builder.md.
• Hyperhive. The commit pins one hyperhive rev for the swarm, so an agent runs the hyperhive of the controller's deploy, not its hive's.
• Markers. After a successful build hive-c0re writes the hyperhive commit the head pins to applied/.<n>.hyperhive-rev, and the config commit to applied/.<n>.config-rev (none for an archetype agent). The boot reconcile (below) and the dashboard's update chip read them.

The applied repo per agent

/var/lib/hyperhive/applied/<name>/          applied - core-only
├── .git/                                   # tag-rich history
├── flake.nix                               # tracked, fixed
│                                           # boilerplate exporting
│                                           # nixosModules.default
├── agent.nix                               # working tree of main
└── <other committed files>                 # also tracked

/var/lib/hyperhive/meta/                    plain hive state - core
├── topology.json                           # the live agent set
└── swarm-resource-limits.json              # the swarm's per-agent caps

hive-c0re seeds the applied repo at first spawn from the forge's agent-configs/<name> main, or from a template when the forge has none yet, and fast-forwards it to each merged config commit that deploys. An agent never edits it: it isn't mounted into any container, and a config change is a PR against the agent's forge repo. Older hives also kept a proposed clone at agents/<name>/config. Nothing reads it any more, and you can delete it.

The container's --flake ref is internal/meta (see "Meta flake" above). The agent's own applied/<n>/flake.nix is a fixed boilerplate that exports nixosModules.default = import ./agent.nix; the meta flake imports that module and wraps it with identity + HIVE_PORT / HIVE_LABEL.

Tag state machine

hive-c0re plants deployed/0 on the seed commit at first spawn. A merged config commit that deploys plants no tag: the merged PR on the forge and internal/meta's commits record it. A config PR nobody merges carries no extra state on the forge side: the PR stays open, and the submitter pushes again (or closes it) to retry.

Forge config repos

The bundled hive-forge container runs on the swarm's forge host (deploy.forgejo.enable, see ../swarm/services.md). Every agent created through the swarm has a repo in the private agent-configs Forgejo org, created and seeded by swarm-controller. hive-c0re pushes nothing to it: the applied repo's tags (deployed/0) exist only locally, and forge main advances only by an operator's merge. hive-c0re reads from it - reseeding a purged applied, catching applied up, and and seeding applied at first spawn (forge::fetch_config_main_into_applied).

A legacy agent spawned by the hive alone has no swarm-seeded repo; if its agent-configs/<name> has no main, a first spawn falls back to the local template, and reseeding a purged applied fails with a destroy --purge and re-spawn instruction.

Each agent is a write collaborator on its own agent-configs/<name> repo — so it can push a branch and open a config PR — but not a member of any other agent's, so it can't reach another agent's config through the forge. Branch protection keeps the agent off the main push allowlist and allowlists merging to the operators team only, so an agent can't fast-forward its own config or self-merge its PR (see Config changes above). hive-c0re passes the credential to each git call inline, never writing it into applied/<n>/.git/config, so no stored token sits in a repo an agent might one day read.

The dashboard deep-links into this org — a config repo link per container row. See docs/web-ui/dashboard.md.

Submitting agent's view of config repos

No agent has a config repo mounted, not even one holding ManageRootAgent: reading another agent's config means cloning its repo from the forge, subject to the forge's own access rules.

An agent submits a change to another agent's config the same way it makes any other change: clone the target agent's config repo from the forge into its own state dir, commit on a branch, open a PR, and let the operator review and merge it. By design, no second, mount-shaped path reaches the same file without the review.

Agents holding the manage_root_agent permission (granted per agent in the swarm's agent-permissions record; see docs/process/conventions.md#permissions) get additional host-side bind mounts via set_nspawn_flags:

• /var/lib/hyperhive/agents/<n>/state → /agents/<n>/state (RW) for every agent. The permission means "may manage any agent," so the mounted set is every agent.
• /var/lib/hyperhive/meta/ → /meta/ (RO) — the hive's plain state (topology, swarm limits).

An agent without the permission only has its direct children's config dirs.

⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts an agent, after its current turn, when its manage_root_agent changes.

No agent holds the permission by default: an agent whose record leaves permissions unset gets the default, which doesn't include manage_root_agent, whatever the agent's name. The swarm UI's setup assistant creates ruth with it. For a ruth that predates that, whose permissions were unset, swarm-controller writes the default plus manage_root_agent into her key once at start, so she keeps her mounts; it never touches a key that's already set.

Startup migrations (older hosts)

hive-c0re runs a couple of idempotent migrations on every startup so a host set up before the tag-driven-deploy scheme (described above) converges to it automatically. Each phase is a no-op once already applied:

• Tags: hive-c0re tags agents from before the tag-driven scheme deployed/0 on main once. Non-destructive — it doesn't touch live containers, state dirs, or claude creds.
• Agent files: rewrites each applied/<n>/flake.nix to the module-only boilerplate and refreshes the files that follow the agent set (agent_set::refresh) from the current agent list. Set HIVE_SKIP_META_MIGRATION=1 on the service to defer this phase.
• Legacy meta repo: deletes the leftover .git in /var/lib/hyperhive/meta/ and the unread /var/lib/hyperhive/forge-core-token. The files beside the .git stay. The forge-side core/meta repo and core account aren't touched.

No state loss in either migration: claude creds, /state/ notes, the events DB, and the applied history all survive. The root agent keeps its session; sub-agents stay logged in.

The root/bootstrap container

The root agent container runs through the same lifecycle as sub-agents. Nothing creates it at hive boot: you create ruth from the swarm dashboard like any agent, and an existing ruth container starts with the hive like every other. The root agent's flake lives at /var/lib/hyperhive/applied/ruth/. The root agent can clone agent-configs/ruth to read its own agent.nix and open a config PR there for operator review, same as any other agent.

Differences from sub-agents:

• Reference docs: her agent.nix turns on services.hyperhive.agent.docs.enable. Her container builds from agent-base like every agent, and the seeded agent.nix is the same for every name, so a newly created ruth has no docs until a config PR adds that line.
• Web UI port via lifecycle::agent_web_port("ruth") — same FNV-1a hash as every other agent (8100..8999 range).
• The root agent holds manage_root_agent (the swarm UI's setup assistant creates her with it), for which set_nspawn_flags adds extra binds: every agent's state dir (RW) under /agents/<n>/state, and the hive's plain state under /meta (RO).
• Its socket comes from the generic per-agent registration, in the same per-agent runtime dir as any other agent (/run/hyperhive/agents/ruth/), not a separate manager-only path.

Migration note (for older hosts): drop any containers.root = { ... } block from your host NixOS config, and any services.hyperhive.ruthless definition, then create ruth from the swarm dashboard.

Root-agent policy

The system prompt (hive-agent/prompts/system.md, rendered by hive-agent/src/prompt.rs) is the same for every agent; what varies is which MCP tools it surfaces (gated by the agent's permissions, which the swarm sets). No role:manager block renders only for the root agent. The root agent's role-specific behaviour comes from its AGENTS.md / agent-specific instructions, not the system prompt template.

Helper events

hive_sh4re::manager::HelperEvent is the wire type for an event urgent enough to drive an immediate claude turn (delivered as a regular system inbox message, same recv path as any other message). It currently has one variant, NeedsUpdate { agent } (sub-agent's recorded flake rev is stale, operator rebuilds from the dashboard) — and nothing in the tree constructs it; this removal dropped the push path (Coordinator::notify_agent) once its last caller (notify_manager, used only for the crash event below) went away. HelperEvent stays as the type for whatever comes along that genuinely needs to interrupt an agent's turn loop, not because anything live uses it today.

A previously running container crashing used to be exactly that case (ContainerCrash, routed to the root agent via Coordinator::notify_manager) — it no longer drives any agent turn at all. crash_watch now only tracing::warn!s "container crash detected"; a swarm-level grafana alert matches that log line directly and pages the operator, with no agent (and no per-hive dashboard banner) in the loop. See docs/scheduler/observability.md.

The remaining lower-urgency lifecycle notices — Rebuilt, Killed, Destroyed, NeedsLogin, LoggedIn — are "FYI, check when convenient" events with no reason to drive an immediate turn, so they go out via swarm_notices::notify instead of a HelperEvent: subsystem = "core", key = "<event>:<agent>", and a single free-text summary (rebuilt_todo_summary renders Rebuilt's ok/note/sha/tag fields into that string).

To add a new lifecycle notice: if it's "FYI, check when convenient," call swarm_notices::notify — no new wire type needed. If it needs to drive an immediate turn, decide first whether it actually needs an agent in the loop at all (the crash case above didn't, once a swarm-level alert could carry the same signal) — only reach for a new HelperEvent variant + notify_agent-equivalent call site + prompts/system.md update when the answer is yes.

Autoupdate on startup

hive-c0re serve runs auto_update::run in a background task right after opening the coordinator. It enumerates managed containers and rebuilds any whose recorded hyperhive rev differs from the one internal/meta pins now. That's how a hive that missed the swarm's fan-out catches up. Sub-agents and the root agent go through the same graceful rebuild subgraph.

"Rev" = the hyperhive commit pinned by the head of internal/meta, as the hive reads it with its forge token. Marker file: /var/lib/hyperhive/applied/.<name>.hyperhive-rev, written after a successful build. If the hive can't read the pin (no forge token, the forge down, internal/meta not published yet), the scan rebuilds nothing; a rebuild would have no flake to build from either. A hive deploy alone doesn't rebuild agents.

A stopped agent gets its rebuild at its next start: the start path upgrades the start of a stale, stopped agent to a rebuild followed by the start. In the window after a config merge, before the swarm's update has pinned the new config, that rebuild fails with the error that names the agent, and the agent stays stopped until the update lands and you ask for the start again.

The dashboard surfaces pending config changes per agent: a clickable "needs update ↻" badge appears whenever the agent's applied repo has a commit that its last build didn't pin. The badge POSTs /api/rebuild/<name>, which inserts the same job_queue::templates::rebuild DAG so manual triggers and the startup scan can't drift. When at least one container is stale, a top-level ↻ UPD4TE 4LL button appears that loops over every stale container.