# Config changes + helper events Config changes are pull requests on the agent's config repo, which an operator merges on the forge; that merge deploys them. Helper events are how agents stay informed about lifecycle changes that need a reaction. ## For operators - **Config change** — an agent proposed a change to another agent's config (or its own, via a sub-manager) as a pull request on `agent-configs/`. Review it on the forge like any other PR, and merge it there to deploy it. See [Config changes](#config-changes). Everything below this point is the implementation detail behind that flow. ## Config changes Config changes flow through a **forge pull request** on the agent's `agent-configs/` repo — the same surface agents use for code PRs. No MCP tool exists for config changes: opening the PR IS the request. 1. The submitting agent **clones** `agent-configs/`, edits it there (any tracked path, but `agent.nix` is the contract entry point), commits with its own git identity, and pushes a branch + opens a PR with `hive-forge` — the same way it would change any other repo. The bind-mounted `/agents//config/` is a **copy for reading** a config, not the tree to edit: authoring in place there produces no PR. (it's currently mounted read-write, which is a defect tracked separately, not an authoring path.) Branch protection (the agent isn't on the `main` push allowlist; merge and approvals allowlist = `operators` team; see "Forge config repos" below) makes the agent a write collaborator that **can't merge its own config PR**. 2. An operator reviews the PR **on the forge** (native diff, threaded comments, CI status) and merges it there. Merging needs membership of the `operators` team in `agent-configs`. The team starts empty: add each operator by hand in the forge UI. 3. swarm-controller gets the `agent-configs` org's `pull_request` delivery. A `closed` event with `merged: true` and base branch `main` names the commit in `merge_commit_sha`. 4. The controller looks up which hive's wanted state places the agent and publishes a deploy request carrying that commit on that hive's deploy subject. With no such hive, or more than one, it deploys nothing and logs a warning naming them. 5. If the hive's `applied/main` already is that commit, it does nothing. Otherwise it fetches the config repo's `main`. A commit that `applied/main` already contains is a no-op too: requests for two merges can arrive in either order. A commit that descends from `applied/main` is fast-forwarded to and a rebuild queued. A failure before the rebuild (the fetch, or a commit that has diverged from `applied/main`) deploys nothing and logs `swarm events: rev not applied; nothing queued` at error, which fires the Grafana alert `config rev refused`. An agent with no container on that hive yet ignores the commit; its first deploy builds what it seeds. This path runs **no eval-verify**. A merged config that doesn't evaluate or build fails the rebuild, and `applied/main` stays at that commit, so the agent's rebuilds keep failing until a fix merges. A failed rebuild shows on the hive like any other. A merge whose webhook delivery never reaches the controller deploys nothing. Creating a brand-new agent is swarm-level (`swarmctl agent create`, `POST /api/agents` on the swarm controller), and no hive can originate an agent. The swarm controller's `InitAgentConfigRepo` job seeds the agent's config repo with a default `agent.nix` template, then asks the target hive to deploy it. An agent created from an [archetype](../swarm/archetypes.md) gets no config repo: its hive builds it from the archetype's module. Changing what the template seeded isn't a special case: like every later change, it's a PR on that config repo, made from a clone and merged by an operator on the forge (see [Config changes](#config-changes)). ## Agent lifecycle ### Reminder delivery: file-path semantics A reminder may carry a `file_path` (the agent-visible path inside its container, for example `/agents//state/foo.md`). On delivery hive-c0re: 1. **Translates** the container path to the host path (`/var/lib/hyperhive/agents//state/foo.md`) so c0re can write from outside the container. 2. **Validates** the path: rejects anything outside the agent's own state subtree, containing `..` (path traversal), or with an empty relative tail. On rejection hive-c0re skips the write and delivers the original message inline with a warning — the reminder still fires. 3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re canonicalizes the parent dir and re-verifies it lives under the agent's host state root. hive-c0re opens the final file with `O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the basename can't redirect the write to an arbitrary host path. 4. **Writes the body to disk** and delivers a short pointer message in its place, keeping the agent's inbox / wake-prompt small while the agent reads the bulky payload out of band. `Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the body strings before calling it. ### Destroy semantics `HostRequest::Destroy { name, purge }` is the lifecycle tear-down. Stops + removes the nspawn container and drops the systemd drop-in. Persistent state (proposed/applied repos, claude credentials, `/state/` notes) is **kept by default** — recreating the agent with the same name reuses prior config + login. With `purge = true` the agent's `/var/lib/hyperhive/{agents,applied}//` trees are also wiped (config history + creds + notes gone forever). The root/bootstrap container is destroyable like any other, and stays gone until you create ruth again from the swarm dashboard. ## Meta flake A hive has no meta flake of its own. It builds every agent from the swarm's `internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)). It declares one flake input per agent (`agent-.url = "git+/agent-configs/.git?rev="`, the head of `main` on the agent's forge config repo, the one config PRs merge into) and one `nixosConfigurations.` output per agent. Each output wraps `inputs.agent-.nixosModules.default` with the identity + `HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host (the dashboard port, the pinned claude-code, extra OTEL labels, the queue address) aren't in it: `hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at start (see `docs/process/conventions.md`). A config that sits only in the hive's `applied/` mirror, not on forge `main`, never deploys. What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain state, bound read-only into the root agent's container as `/meta`: `topology.json`, the swarm resource limits, and the permission seed files of a hive that predates the swarm's permissions. `agent_set::refresh(agents: &[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its nginx include in line with the live agents. `provision_container`, `destroy`, `rebuild` and the startup migration call it. It refuses an empty agent list, which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL` is unset, since every agent would come up with no forge. The dashboard's `deployed:` chip per container row shows the config commit the agent's last successful build pinned. ## The swarm's meta flake swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for every agent any hive places (the pins: [`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). A hive builds every agent from it. While the repo has no commit that pins an agent's current config, that agent's build fails and names the agent. The controller queues its update of the repo before the deploy request for a merged config commit and for a new agent, and sends the deploy only when that update succeeded. - **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's default-branch head with the hive's read token (`forge::reader_token()`) and takes it when its `flake.nix` builds the agent the way the agent needs now: a custom agent from the config commit at the head of `agent-configs/` `main`, an archetype agent from the archetype it has now. An agent switched between custom and archetype, or to another archetype, fails that test. A hive without the forge credential file that nix's git reads, a repo with no commit, a head that lacks the agent and a head behind the config each fail the build with an error that names the agent. `MetaSync` records the commit, with the hyperhive and config commits the head pins, in `hive-c0re`'s own state, `/var/lib/hyperhive/meta-rev/`, which the agent can't write (the container mounts its runtime directory read-write); `Prebuild` and `Swap` read that record, so both build one flake. A first deploy resolves the commit in its `Create` node, since it has no `MetaSync`. - **The build.** `hive-c0re` prebuilds `git+/internal/meta.git?rev=#nixosConfigurations.…` and names the commit to `hive-priv` (`meta_rev` on `UpdateContainer` and `CreateContainer`). `hive-priv` accepts only a 40-digit lowercase hex commit and builds the flake from its own forge URL (`HIVE_PRIV_FORGE_URL`); the URL never comes from a request. On the wire the `hive_priv_sock::PrivRequest` variants carry a required field: `{"op":"update_container","name":"atlas","stream":true,"meta_rev":"<40 hex>"}`. - **Fetching.** nix fetches the forge's inputs with `git`. Both `hive-c0re` and `hive-priv` carry a `$HOME/.gitconfig` that answers for the forge host through `git-credential-hive-forge` (the hive's forge read token, published to `/run/hyperhive/forge-hive-token`) and rewrites the forge's `https` URL, which `internal/meta`'s lock names its inputs by, to `http`. - **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent runs the hyperhive of the controller's deploy, not its hive's. - **Markers.** After a successful build `hive-c0re` writes the hyperhive commit the head pins to `applied/..hyperhive-rev`, and the config commit to `applied/..config-rev` (none for an archetype agent). The boot reconcile ([below](#autoupdate-on-startup)) and the dashboard's update chip read them. ## Two repos per agent ``` /var/lib/hyperhive/agents//config/ proposed — parent mount is RO └── # any files the submitting # agent wants in the commit. # agent.nix is the # convention entry # point; flake.nix is # tracked boilerplate # (submitting agent doesn't # edit it). /var/lib/hyperhive/applied// applied — core-only ├── .git/ # tag-rich history ├── flake.nix # tracked, fixed │ # boilerplate exporting │ # nixosModules.default ├── agent.nix # working tree of main └── # also tracked /var/lib/hyperhive/meta/ plain hive state — core ├── topology.json # the live agent set └── swarm-resource-limits.json # the swarm's per-agent caps ``` Why two physical repos: the submitting agent's `/agents//config/` is RW — a buggy or hostile agent can `git clean -fdx` its own proposed tree. The applied repo is never bind-mounted (except the read-only `.git` exposure described below) so a destructive move inside the container can't reach it. The container's `--flake` ref is `internal/meta` (see "Meta flake" above). The agent's own `applied//flake.nix` is a fixed boilerplate that exports `nixosModules.default = import ./agent.nix`; the meta flake imports that module and wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`. ### Tag state machine hive-c0re plants `deployed/0` on the seed commit at first spawn. A merged config commit that deploys plants no tag: the merged PR on the forge and `internal/meta`'s commits record it. A config PR nobody merges carries no extra state on the forge side: the PR stays open, and the submitter pushes again (or closes it) to retry. ### Forge config repos The bundled `hive-forge` container runs on the swarm's forge host (`deploy.forgejo.enable`, see [`../swarm/services.md`](../swarm/services.md)). Every agent created through the swarm has a repo in the private `agent-configs` Forgejo org, created and seeded by swarm-controller. hive-c0re pushes nothing to it: the applied repo's tags (`deployed/0`) exist only locally, and forge `main` advances only by an operator's merge. hive-c0re reads from it — reseeding a purged `applied` (`forge::fetch_config_main_into_applied`), catching `applied` up, and seeding `proposed` at first spawn (`forge::clone_config_into_proposed`). A legacy agent spawned by the hive alone has no swarm-seeded repo; if its `agent-configs/` has no `main`, a first spawn falls back to the local template, and reseeding a purged `applied` fails with a `destroy --purge` and re-spawn instruction. Each agent is a **write collaborator on its own** `agent-configs/` repo — so it can push a branch and open a config PR — but not a member of any other agent's, so it can't reach another agent's config through the forge. Branch protection keeps the agent off the `main` push allowlist and allowlists merging to the `operators` team only, so an agent can't fast-forward its own config or self-merge its PR (see [Config changes](#config-changes) above). hive-c0re passes the credential to each `git` call inline, never writing it into `applied//.git/config`; that repo is RO-bind-mounted into the root agent, and a stored token would leak the hive's forge credential to an agent. The dashboard deep-links into this org — a `config repo` link per container row. See `docs/web-ui/dashboard.md`. ### Submitting agent's view of config repos An agent holding `ManageRootAgent` has every other agent's config repo bind-mounted **read-only** (`hive-c0re/src/lifecycle/host_config.rs` calls `bind_child_agent_dirs` for each entry in `topology::all_agents()`). it's a copy to *read* another agent's current config — not an editing surface. An agent without the permission sees no other agent's config at all. An agent submits a change to another agent's config the same way it makes any other change: **clone the target agent's config repo from the forge into its own state dir, commit on a branch, open a PR**, and let the operator review and merge it. By design, no second, mount-shaped path reaches the same file without the review. Agents holding the `manage_root_agent` permission (granted per agent in the swarm's `agent-permissions` record; see `docs/process/conventions.md#permissions`) get additional host-side bind mounts via `set_nspawn_flags`: - `/var/lib/hyperhive/agents/` → `/agents/` (RW) — **every** agent's proposed repo, not just direct children. The permission means "may manage any agent", so the mounted set is every agent. - `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every agent's authoritative applied repo, including `.git`. - `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the hive's plain state (topology, swarm limits). An agent without the permission only has its direct children's config dirs. ⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts an agent, after its current turn, when its `manage_root_agent` changes. No agent holds the permission by default: an agent whose record leaves `permissions` unset gets the default, which doesn't include `manage_root_agent`, whatever the agent's name. The swarm UI's setup assistant creates ruth with it. For a ruth that predates that, whose permissions were unset, swarm-controller writes the default plus `manage_root_agent` into her key once at start, so she keeps her mounts; it never touches a key that's already set. Each proposed repo (`/agents//config/`) is pre-configured with `applied` as a git remote pointing at `/applied//.git`. Useful incantations from inside an agent with the full `/applied` mount: ```sh git -C /agents//config fetch applied git -C /agents//config log applied/main --oneline git -C /agents//config show applied/refs/tags/deployed/0 # the seed commit git -C /agents//config show applied/refs/tags/denied/ # body = operator note git -C /agents//config rebase applied/main # base in-flight work on what's deployed ``` The RO binds block push at the kernel level — git plumbing inside the container can't corrupt either authoritative repo. ## Startup migrations (older hosts) hive-c0re runs a couple of idempotent migrations on every startup so a host set up before the tag-driven-deploy scheme (described above) converges to it automatically. Each phase is a no-op once already applied: - **Tags**: hive-c0re tags agents from before the tag-driven scheme `deployed/0` on `main` once. Non-destructive — it doesn't touch live containers, state dirs, or claude creds. - **Agent files**: rewrites each `applied//flake.nix` to the module-only boilerplate, wires the `applied` remote in each proposed repo, and refreshes the files that follow the agent set (`agent_set::refresh`) from the current agent list. Set `HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase. No state loss in either migration: claude creds, `/state/` notes, the events DB, and both proposed + applied history all survive. The root agent keeps its session; sub-agents stay logged in. ## The root/bootstrap container The root agent container runs through the **same lifecycle as sub-agents**. Nothing creates it at hive boot: you create ruth from the swarm dashboard like any agent, and an existing ruth container starts with the hive like every other. The root agent's flake lives at `/var/lib/hyperhive/applied/ruth/`; its proposed config at `/var/lib/hyperhive/agents/ruth/config/`. The root agent can edit its own `agent.nix` (visible inside the container at `/agents/ruth/config/`) and open a config PR on `agent-configs/ruth` for operator review, same as any other agent. Differences from sub-agents: - Reference docs: her `agent.nix` turns on `services.hyperhive.agent.docs.enable`. Her container builds from `agent-base` like every agent, and the seeded `agent.nix` is the same for every name, so a newly created ruth has no docs until a config PR adds that line. - Web UI port via `lifecycle::agent_web_port("ruth")` — same FNV-1a hash as every other agent (8100..8999 range). - The root agent holds `manage_root_agent` (the swarm UI's setup assistant creates her with it), for which `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents` → `/agents` (RW) so the root agent can edit per-agent proposed repos, and `/var/lib/hyperhive/applied` → `/applied` (RO) so the root agent can `git fetch` deployed/failed/denied tags from any agent's authoritative applied repo (see "Root-agent view of applied" below). - Its socket comes from the generic per-agent registration, in the same per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`), not a special manager-only path. **Migration note** (for older hosts): drop any `containers.root = { ... }` block from your host NixOS config, and any `services.hyperhive.ruthless` definition, then create ruth from the swarm dashboard. ## Root-agent policy The system prompt (`hive-agent/prompts/system.md`, rendered by `hive-agent/src/prompt.rs`) is the **same for every agent**; what varies is which MCP tools it surfaces (gated by the agent's permissions, which the swarm sets). No `role:manager` block renders only for the root agent. The root agent's role-specific behaviour comes from its AGENTS.md / agent-specific instructions, not the system prompt template. ## Helper events `hive_sh4re::manager::HelperEvent` is the wire type for an event urgent enough to drive an immediate claude turn (delivered as a regular `system` inbox message, same `recv` path as any other message). It currently has one variant, `NeedsUpdate { agent }` (sub-agent's recorded flake rev is stale, operator rebuilds from the dashboard) — and nothing in the tree constructs it; this removal dropped the push path (`Coordinator::notify_agent`) once its last caller (`notify_manager`, used only for the crash event below) went away. `HelperEvent` stays as the type for whatever comes along that genuinely needs to interrupt an agent's turn loop, not because anything live uses it today. **A previously-running container crashing** used to be exactly that case (`ContainerCrash`, routed to the root agent via `Coordinator::notify_manager`) — it no longer drives any agent turn at all. `crash_watch` now only `tracing::warn!`s `"container crash detected"`; a swarm-level grafana alert matches that log line directly and pages the operator, with no agent (and no per-hive dashboard banner) in the loop. See [`docs/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re). The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`, `Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check when convenient" events with no reason to drive an immediate turn, so they go out via `swarm_notices::notify` instead of a `HelperEvent`: `subsystem = "core"`, `key = ":"`, and a single free-text `summary` (`rebuilt_todo_summary` renders `Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string). To add a new lifecycle notice: if it's "FYI, check when convenient," call `swarm_notices::notify` — no new wire type needed. If it needs to drive an immediate turn, decide first whether it actually needs an *agent* in the loop at all (the crash case above didn't, once a swarm-level alert could carry the same signal) — only reach for a new `HelperEvent` variant + `notify_agent`-equivalent call site + `prompts/system.md` update when the answer is yes. ## Autoupdate on startup `hive-c0re serve` runs `auto_update::run` in a background task right after opening the coordinator. It enumerates managed containers and rebuilds any whose recorded hyperhive rev differs from the one `internal/meta` pins now. That's how a hive that missed the swarm's fan-out catches up. Sub-agents and the root agent go through the same graceful rebuild subgraph. "Rev" = the hyperhive commit pinned by the head of `internal/meta`, as the hive reads it with its forge token. Marker file: `/var/lib/hyperhive/applied/..hyperhive-rev`, written after a successful build. If the hive can't read the pin (no forge token, the forge down, `internal/meta` not published yet), the scan rebuilds nothing; a rebuild would have no flake to build from either. A hive deploy alone doesn't rebuild agents. A stopped agent gets its rebuild at its next start: the start path upgrades the start of a stale, stopped agent to a rebuild followed by the start. In the window after a config merge, before the swarm's update has pinned the new config, that rebuild fails with the error that names the agent, and the agent stays stopped until the update lands and you ask for the start again. The dashboard surfaces pending config changes per agent: a clickable "needs update ↻" badge appears whenever the agent's applied repo has a commit that its last build didn't pin. The badge POSTs `/api/rebuild/`, which inserts the same `job_queue::templates::rebuild` DAG so manual triggers and the startup scan can't drift. When at least one container is stale, a top-level `↻ UPD4TE 4LL` button appears that loops over every stale container.