hyperhive/docs/agent-lifecycle/config-changes.md

# Config changes + helper events

Config changes are pull requests on the agent's config repo, which an
operator merges on the forge; that merge deploys them. Helper events are
how agents stay informed about lifecycle changes that need a reaction.

## For operators

- **Config change** — an agent proposed a change to another agent's
  config (or its own, via a sub-manager) as a pull request on
  `agent-configs/<agent>`. Review it on the forge like any other PR, and
  merge it there to deploy it. See [Config changes](#config-changes).

Everything below this point is the implementation detail behind that
flow.

## Config changes

Config changes flow through a **forge pull request** on the agent's
`agent-configs/<name>` repo — the same surface agents use for code PRs.
No MCP tool exists for config changes: opening the PR IS the request.

1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
   tracked path, but `agent.nix` is the contract entry point), commits
   with its own git identity, and pushes a branch + opens a PR with
   `hive-forge` — the same way it would change any other repo.
   The bind-mounted `/agents/<name>/config/` is a **copy for reading** a
   config, not the tree to edit: authoring in place there produces no PR.
   (it's currently mounted read-write, which is a defect tracked
   separately, not an authoring path.)
   Branch protection (the agent isn't on the `main` push allowlist;
   merge and approvals allowlist = `operators` team; see "Forge config repos"
   below) makes the agent a write collaborator that **can't merge its own
   config PR**.
2. An operator reviews the PR **on the forge** (native diff, threaded
   comments, CI status) and merges it there. Merging needs membership of
   the `operators` team in `agent-configs`. The team starts empty: add
   each operator by hand in the forge UI.
3. swarm-controller gets the `agent-configs` org's `pull_request`
   delivery. A `closed` event with `merged: true` and base branch `main`
   names the commit in `merge_commit_sha`.
4. The controller looks up which hive's wanted state places the agent
   and publishes a deploy request carrying that commit on that hive's
   deploy subject. With no such hive, or more than one, it deploys
   nothing and logs a warning naming them.
5. If the hive's `applied/main` already is that commit, it does nothing.
   Otherwise it fetches the config repo's `main`. A commit that
   `applied/main` already contains is a no-op too: requests for two merges
   can arrive in either order. A commit that descends from `applied/main`
   is fast-forwarded to and a rebuild queued. A failure before the rebuild
   (the fetch, or a commit that has diverged from `applied/main`) deploys
   nothing and logs `swarm events: rev not
   applied; nothing queued` at error, which fires the Grafana alert
   `config rev refused`. An agent with no container on that hive yet ignores
   the commit; its first deploy builds what it seeds.

This path runs **no eval-verify**. A merged config that doesn't
evaluate or build fails the rebuild, and `applied/main` stays at that
commit, so the agent's rebuilds keep failing until a fix merges. A
failed rebuild shows on the hive like any other. A merge whose webhook delivery never reaches the controller
deploys nothing.

Creating a brand-new agent is swarm-level (`swarmctl agent create`,
`POST /api/agents` on the swarm controller), and no hive can originate
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
agent's config repo with a default `agent.nix` template, then asks the
target hive to deploy it.

Changing what the template seeded isn't a special case: like every
later change, it's a PR on that config repo, made from a clone and
merged by an operator on the forge (see [Config changes](#config-changes)).

## Agent lifecycle

### Reminder delivery: file-path semantics

A reminder may carry a `file_path` (the agent-visible path inside its
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:

1. **Translates** the container path to the host path
   (`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
   from outside the container.
2. **Validates** the path: rejects anything outside the agent's own state
   subtree, containing `..` (path traversal), or with an empty relative
   tail. On rejection hive-c0re skips the write and delivers the
   original message inline with a warning — the reminder still fires.
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
   canonicalizes the parent dir and re-verifies it lives under the agent's host
   state root. hive-c0re opens the final file with
   `O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
   basename can't redirect the write to an arbitrary host path.
4. **Writes the body to disk** and delivers a short pointer message in its
   place, keeping the agent's inbox / wake-prompt small while the agent
   reads the bulky payload out of band.

`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
body strings before calling it.

### Destroy semantics

`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
Stops + removes the nspawn container and drops the systemd drop-in.
Persistent state (proposed/applied repos, claude credentials, `/state/`
notes) is
**kept by default** — recreating the agent with the same name
reuses prior config + login. With `purge = true` the agent's
`/var/lib/hyperhive/{agents,applied}/<name>/` trees are also
wiped (config history + creds + notes gone forever). The
root/bootstrap container is destroyable like any other — hive-c0re
recreates it on the next startup if it's absent, so destroying it's
transient.

## Meta flake

The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
declares one flake input per agent (`agent-<n>.url =
"git+http://<forge>/agent-configs/<n>.git"`) and one
`nixosConfigurations.<n>` output per agent. Each output wraps
`inputs.agent-<n>.nixosModules.default` with the identity +
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module.
Containers run against `--flake /var/lib/hyperhive/meta#<n>`.

The declared input url is the agent's **forge config repo** (the
same `agent-configs/<n>` config PRs merge into), so the meta flake
references a reviewable, reproducible source rather than a local
checkout. hive-c0re authenticates that `git+http` fetch via a git
credential helper that reads the live forge-core token — no token in
the url or the lock. The rebuild paths, however, do **not** re-lock
from the forge: they `--override-input agent-<n>
git+file:///var/lib/hyperhive/applied/<n>`, locking the exact config
`applied/<n>/main` was fast-forwarded to. That keeps a rebuild
reproducible and independent of forge reachability — rebuilds fire on
crash-restart and meta bumps, not just config merges — while the
declared url stays the forge. `sync_agents` re-renders + re-locks the
persistent input; a plain `nix flake lock` leaves an existing applied
override in place (it only re-locks when the declared url itself
changes), so the forge-declared / applied-deployed split is stable.

Lock updates are single-phase and commit when the lock changed:
`meta::lock_update_for_rebuild(name)` relocks one agent's input for a
relocking rebuild (the manual `↻ R3BU1LD` button, and the rebuild a
merged config commit queues), and `meta::lock_update_hyperhive()` is the
autoupdate flake-rev bump (one shot before per-agent rebuilds).

`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
per-hive config — is the idempotent reconciler called by `spawn`,
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
from the agent list; if it differs from disk, runs
`nix flake lock` + commits as `regenerate meta flake` (or
`seed meta from N agent(s)` on the first call).

The root agent has `/meta` RO-bound inside its container:
`git -C /meta log --oneline` is the swarm-wide deploy log,
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
resolves which sha the flake pins each agent at right now.
Dashboard surfaces the same info as a `deployed:<sha12>` chip
per container row.

## Two repos per agent

```
/var/lib/hyperhive/agents/<name>/config/    proposed — parent mount is RO
└── <anything>                              # any files the submitting
                                            # agent wants in the commit.
                                            # agent.nix is the
                                            # convention entry
                                            # point; flake.nix is
                                            # tracked boilerplate
                                            # (submitting agent doesn't
                                            # edit it).

/var/lib/hyperhive/applied/<name>/          applied — core-only
├── .git/                                   # tag-rich history
├── flake.nix                               # tracked, fixed
│                                           # boilerplate exporting
│                                           # nixosModules.default
├── agent.nix                               # working tree of main
└── <other committed files>                 # also tracked

/var/lib/hyperhive/meta/                    swarm-wide flake — core
├── .git/                                   # one commit per lock
│                                           # change
├── flake.nix                               # generated from agent set
└── flake.lock                              # pins each agent's sha
```

Why two physical repos: the submitting agent's `/agents/<n>/config/` is
RW — a buggy or hostile agent can `git clean -fdx` its own
proposed tree. The applied repo is never bind-mounted (except
the read-only `.git` exposure described below) so a destructive
move inside the container can't reach it.

The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
is a fixed boilerplate that exports `nixosModules.default =
import ./agent.nix`; the meta flake imports that module and
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` /
`HIVE_DASHBOARD_PORT`.

### Tag state machine

hive-c0re plants `deployed/0` on the seed commit at first spawn. A
merged config commit that deploys plants no tag: the merged PR on the
forge and meta's lock commits record it. A config PR nobody merges
carries no extra state on the forge side: the PR stays open, and
the submitter pushes again (or closes it) to retry.

### Forge config repos

The bundled `hive-forge` container runs on the swarm's forge host
(`deploy.forgejo.enable`, see [`../swarm/services.md`](../swarm/services.md)).
Every agent created through the swarm has a repo in the private
`agent-configs` Forgejo org, created and seeded by swarm-controller.
hive-c0re pushes nothing to it: the applied repo's tags (`deployed/0`)
exist only locally, and forge `main` advances only by an operator's merge.
hive-c0re reads from it — reseeding a purged `applied`
(`forge::fetch_config_main_into_applied`), catching `applied` up, and
seeding `proposed` at first spawn (`forge::clone_config_into_proposed`).

A legacy agent spawned by the hive alone has no swarm-seeded repo; if its
`agent-configs/<name>` has no `main`, a first spawn falls back to the local
template, and reseeding a purged `applied` fails with a `destroy --purge`
and re-spawn instruction.

Each agent is a **write collaborator on its own** `agent-configs/<name>`
repo — so it can push a branch and open a config PR — but not a member
of any other agent's, so it can't reach another agent's config through
the forge. Branch protection keeps the agent off the `main` push
allowlist and allowlists merging to the `operators` team only, so an
agent can't fast-forward its own config or self-merge its PR (see
[Config changes](#config-changes) above). hive-c0re passes the credential to
each `git` call inline, never writing it into
`applied/<n>/.git/config`; that repo is RO-bind-mounted into the root
agent, and a stored token would leak core's admin credential to an
agent.

The dashboard deep-links into this org — a `config repo` link
per container row. See `docs/web-ui/dashboard.md`.

### Submitting agent's view of config repos

An agent holding `ManageRootAgent` has every other agent's config repo
bind-mounted **read-only** (`hive-c0re/src/lifecycle/host_config.rs`
calls `bind_child_agent_dirs` for each entry in
`topology::all_agents()`). it's a copy to *read* another agent's
current config — not an editing surface. An agent without the
capability sees no other agent's config at all.

An agent submits a change to another agent's config the same way it
makes any other change: **clone the target agent's config repo from the
forge into its own state dir, commit on a branch, open a PR**, and let
the operator review and merge it. By design, no second, mount-shaped
path reaches the same file without the review.

Agents holding the `manage_root_agent` capability (granted per agent in
`capabilities.json`; see `hive-sh4re/src/permissions.rs`) get additional
host-side bind mounts via `set_nspawn_flags`:

- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — **every** agent's
  proposed repo, not just direct children. The capability means "may
  manage any agent", so the mounted set is every agent.
- `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every agent's
  authoritative applied repo, including `.git`.
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide
  deploy flake.

An agent without the capability only has its direct children's config
dirs.

⚠️ nspawn bakes bind flags at container start, so granting or
revoking this capability doesn't change any mount until that agent's
container is rebuilt/restarted.

The root agent gets the capability by default, seeded on its autodeploy
path (`workers::auto_update::ensure_root_agent`) so the recovery mounts
are there from its first container. That seed only fires while
`capabilities.json` doesn't exist yet: any grant or revoke through
the dashboard creates the file, so a revoked root-agent grant stays
revoked and isn't re-applied on the next hive-c0re restart.

Each proposed repo (`/agents/<n>/config/`) is pre-configured
with `applied` as a git remote pointing at
`/applied/<n>/.git`. Useful incantations from inside an agent with
the full `/applied` mount:

```sh
git -C /agents/<n>/config fetch applied
git -C /agents/<n>/config log applied/main --oneline
git -C /agents/<n>/config show applied/refs/tags/deployed/0     # the seed commit
git -C /agents/<n>/config show applied/refs/tags/denied/<id>   # body = operator note
git -C /agents/<n>/config rebase applied/main                 # base in-flight work on what's deployed

git -C /meta log --oneline                                    # swarm-wide deploy history
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
```

The RO binds block push at the kernel level — git plumbing inside the
container can't corrupt either authoritative repo.

## Startup migrations (older hosts)

hive-c0re runs a couple of idempotent migrations on every startup so a
host set up before the tag-driven-deploy + meta-flake scheme (both
described above) converges to it automatically. Each phase is a no-op
once already applied:

- **Tags**: hive-c0re tags agents from before the tag-driven scheme
  `deployed/0` on `main` once. Non-destructive — it doesn't touch live
  containers, state dirs, or claude creds.
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
  module-only boilerplate, wires the `applied` remote in each proposed
  repo, and bootstraps the meta repo from the current agent list. Set
  `HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.

No state loss in either migration: claude creds, `/state/` notes, the
events DB, and both proposed + applied history all survive. The root
agent keeps its session; sub-agents stay logged in.

## The root/bootstrap container is hive-c0re-managed

The root agent container runs through the **same lifecycle as
sub-agents**. On `hive-c0re serve` startup, if `ruth` is missing,
hive-c0re creates it. The root agent's flake lives at
`/var/lib/hyperhive/applied/ruth/`; its proposed config at
`/var/lib/hyperhive/agents/ruth/config/`. The root agent can edit its own
`agent.nix` (visible inside the container at `/agents/ruth/config/`)
and open a config PR on `agent-configs/ruth` for operator review,
same as any other agent.

Differences from sub-agents:

- `flake.nix` extends `hyperhive.nixosConfigurations.ruth`
  (vs `agent-base`).
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
  FNV-1a hash as every other agent (8100..8999 range).
- `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents`
  → `/agents` (RW) so the root agent can edit per-agent proposed repos,
  and `/var/lib/hyperhive/applied` → `/applied` (RO) so the root agent
  can `git fetch` deployed/failed/denied tags from any agent's
  authoritative applied repo (see "Root-agent view of applied" below).
- `socket_server::start_manager` binds the root agent's socket,
  pure transport with no dedicated helpers — it uses the same
  per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
  not a special manager-only path.

**Migration note** (for older hosts): drop any `containers.root =
{ ... }` block from your host NixOS config. hyperhive creates and
updates the root agent itself.

## Root-agent policy

The system prompt (`hive-agent/prompts/system.md`, rendered by
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
varies is which MCP tools it surfaces (gated by tool groups and
capabilities in `agent.nix`). No `role:manager` block renders only
for the root agent. The root agent's role-specific
behaviour comes from its CLAUDE.md / agent-specific instructions, not
the system prompt template.

## Helper events

`Coordinator::notify_agent(agent, &HelperEvent)` pushes the event into
the agent's inbox, and the harness delivers it as a regular `system`
inbox message so it drives a normal claude turn. A "FYI, check when
convenient" event doesn't need a message — hive-c0re publishes those as
swarm lifecycle notices instead (`swarm_notices::notify`). Variants
(`hive_sh4re::manager::HelperEvent`):

- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
  running container went away with no operator-initiated transient
  state (Stopping / Restarting / Destroying / Rebuilding) AND nothing
  cleared that transient in the last 30s (`RECENT_TRANSIENT_GRACE`
  tombstone, three `POLL_INTERVAL`s — closes the race where a
  lifecycle op finishes between two crash-watch polls and the
  container shows briefly as "stopped without transient" before
  the next start). When `systemctl is-active` fails or runs past its
  10s bound on a poll, the container keeps its previous state for that
  poll, so an unreadable unit never counts as a crash.
  The root agent escalates to the operator, who starts it again from
  the dashboard.
- `NeedsUpdate { agent }` — sub-agent's recorded flake rev is
  stale. The operator rebuilds it from the dashboard.

The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
when convenient" events with no reason to drive an immediate turn, so
they go out via `swarm_notices::notify` (see above) instead
of a `HelperEvent`: `subsystem = "core"`, `key = "<event>:<agent>"`,
and a single free-text `summary` (`rebuilt_todo_summary` renders
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).

To add a new lifecycle notice: if it needs to drive an immediate turn
(something genuinely urgent, like `ContainerCrash`), add a
`HelperEvent` variant + call sites + update `prompts/system.md`'s
message-event list. If it's "FYI, check when convenient," call
`swarm_notices::notify` instead — no new wire type
needed.

## Autoupdate on startup

`hive-c0re serve` runs `auto_update::run` in a background task right
after opening the coordinator. It enumerates managed containers and
rebuilds any whose recorded hyperhive rev differs from the current
one — sub-agents and the root agent go through the same
`job_queue::templates::rebuild` DAG.

"Rev" = canonical filesystem path of
`services.hyperhive.c0re.hyperhiveFlake`. Marker
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
flake input has no canonical path (for example a `github:` URL),
autoupdate is a no-op — rebuild manually.

The dashboard surfaces pending updates per agent: a clickable
"needs update ↻" badge appears whenever the marker differs from
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
same `job_queue::templates::rebuild` DAG so manual triggers and the
startup scan can't drift. When at least one container is stale, a
top-level `↻ UPD4TE 4LL` button appears that loops over every
stale container.