393 lines
20 KiB
Markdown
393 lines
20 KiB
Markdown
# Config changes + helper events
|
|
|
|
Config changes are pull requests on the agent's config repo, which an
|
|
operator merges on the forge; that merge deploys them. Helper events are
|
|
how agents stay informed about lifecycle changes that need a reaction.
|
|
|
|
## For operators
|
|
|
|
- **Config change** — an agent proposed a change to another agent's
|
|
config (or its own, via a sub-manager) as a pull request on
|
|
`agent-configs/<agent>`. Review it on the forge like any other PR, and
|
|
merge it there to deploy it. See [Config changes](#config-changes).
|
|
|
|
Everything below this point is the implementation detail behind that
|
|
flow.
|
|
|
|
## Config changes
|
|
|
|
Config changes flow through a **forge pull request** on the agent's
|
|
`agent-configs/<name>` repo — the same surface agents use for code PRs.
|
|
No MCP tool exists for config changes: opening the PR IS the request.
|
|
|
|
1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
|
|
tracked path, but `agent.nix` is the contract entry point), commits
|
|
with its own git identity, and pushes a branch + opens a PR with
|
|
`hive-forge` — the same way it would change any other repo.
|
|
The container has no config mount: the clone is how an agent reads its
|
|
own config as well as how it changes it.
|
|
Branch protection (the agent isn't on the `main` push allowlist;
|
|
merge and approvals allowlist = `operators` team; see "Forge config repos"
|
|
below) makes the agent a write collaborator that **can't merge its own
|
|
config PR**.
|
|
2. An operator reviews the PR **on the forge** (native diff, threaded
|
|
comments, CI status) and merges it there. Merging needs membership of
|
|
the `operators` team in `agent-configs`. The team starts empty: add
|
|
each operator by hand in the forge UI.
|
|
3. swarm-controller gets the `agent-configs` org's `pull_request`
|
|
delivery. A `closed` event with `merged: true` and base branch `main`
|
|
names the commit in `merge_commit_sha`.
|
|
4. The controller looks up which hive's wanted state places the agent
|
|
and publishes a deploy request carrying that commit on that hive's
|
|
deploy subject. With no such hive, or more than one, it deploys
|
|
nothing and logs a warning naming them.
|
|
5. If the commit already is the one the agent's last successful build pinned
|
|
(the `applied/.<n>.config-rev` marker), the hive does nothing. Otherwise it
|
|
queues a rebuild, which resolves its pins from `internal/meta` like any
|
|
rebuild. The request's commit only feeds that check: the hive keeps no copy
|
|
of the config. An agent with no container on that hive yet ignores the
|
|
commit; its first deploy builds whatever `internal/meta` pins.
|
|
|
|
This path runs **no eval-verify**. A merged config that doesn't
|
|
evaluate or build fails the rebuild, so the agent's rebuilds keep failing
|
|
until a fix merges. A failed rebuild shows on the hive like any other. A merge
|
|
whose webhook delivery never reaches the controller deploys nothing.
|
|
|
|
Creating a brand-new agent is swarm-level (`swarmctl agent create`,
|
|
`POST /api/agents` on the swarm controller), and no hive can originate
|
|
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
|
|
agent's config repo with a default `agent.nix` template, then asks the
|
|
target hive to deploy it.
|
|
An agent created from an [archetype](../swarm/archetypes.md) gets no
|
|
config repo: its hive builds it from the archetype's module.
|
|
|
|
Changing what the template seeded isn't an exception: like every
|
|
later change, it's a PR on that config repo, made from a clone and
|
|
merged by an operator on the forge (see [Config changes](#config-changes)).
|
|
|
|
## Agent lifecycle
|
|
|
|
### Reminder delivery: file-path semantics
|
|
|
|
A reminder may carry a `file_path` (the agent-visible path inside its
|
|
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
|
|
|
1. **Translates** the container path to the host path
|
|
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
|
|
from outside the container.
|
|
2. **Validates** the path: rejects anything outside the agent's own state
|
|
subtree, containing `..` (path traversal), or with an empty relative
|
|
tail. On rejection hive-c0re skips the write and delivers the
|
|
original message inline with a warning — the reminder still fires.
|
|
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
|
|
canonicalizes the parent dir and re-verifies it lives under the agent's host
|
|
state root. hive-c0re opens the final file with
|
|
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
|
|
basename can't redirect the write to an arbitrary host path.
|
|
4. **Writes the body to disk** and delivers a short pointer message in its
|
|
place, keeping the agent's inbox / wake-prompt small while the agent
|
|
reads the bulky payload out of band.
|
|
|
|
`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
|
|
body strings before calling it.
|
|
|
|
### Destroy semantics
|
|
|
|
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
|
|
Stops + removes the nspawn container and drops the systemd drop-in.
|
|
Persistent state (claude credentials, `/state/` notes) is
|
|
**kept by default** — recreating the agent with the same name
|
|
reuses prior notes + login. With `purge = true` the agent's
|
|
`/var/lib/hyperhive/agents/<name>/` tree is also
|
|
wiped (creds + notes gone forever). The
|
|
root/bootstrap container is destroyable like any other, and stays gone
|
|
until you create ruth again from the swarm dashboard.
|
|
|
|
## Meta flake
|
|
|
|
A hive has no meta flake of its own. It builds every agent from the swarm's
|
|
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
|
|
It declares one flake input per agent (`agent-<n>.url =
|
|
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
|
|
agent's forge config repo, the one config PRs merge into) and one
|
|
`nixosConfigurations.<n>` output per agent. Each output wraps
|
|
`inputs.agent-<n>.nixosModules.default` with the identity +
|
|
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
|
|
(the dashboard port, the pinned claude-code, extra OTEL labels, the queue
|
|
address) aren't in it:
|
|
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
|
|
start (see `docs/process/conventions.md`).
|
|
A config that sits only in a local checkout, not on forge `main`, never
|
|
deploys.
|
|
|
|
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
|
|
state, bound read-only into the root agent's container as `/meta`:
|
|
`topology.json`, the swarm resource limits, and the permission seed files of a
|
|
hive that predates the swarm's permissions. `agent_set::refresh(agents:
|
|
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
|
|
nginx include in line with the live agents. `provision_container`, `destroy`,
|
|
`rebuild` and the startup migration call it. It refuses an empty agent list,
|
|
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
|
|
is unset, since every agent would come up with no forge.
|
|
|
|
The dashboard's `deployed:<sha12>` chip per container row shows the config
|
|
commit the agent's last successful build pinned.
|
|
|
|
## The swarm's meta flake
|
|
|
|
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
|
|
every agent any hive places (the pins:
|
|
[`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). The swarm
|
|
builder builds every agent from it, and a hive deploys that build. While the
|
|
repo has no commit that pins an agent's current config, that agent's deploy
|
|
fails and names the agent. The controller queues
|
|
its update of the repo before the deploy request for a merged config commit
|
|
and for a new agent, and sends the deploy only when that update succeeded and
|
|
the swarm builder built the agent from the updated repo (the
|
|
[deploy gate](../swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again.
|
|
|
|
- **Choosing the commit.** The rebuild's `MetaSync` node reads the repo's
|
|
default-branch head with the hive's read token (`forge::reader_token()`)
|
|
and takes it when its `flake.nix` builds the agent the way the agent needs now:
|
|
a custom agent from the config commit at the head of `agent-configs/<n>`
|
|
`main`, an archetype agent from the archetype it has now. An agent switched
|
|
between custom and archetype, or to another archetype, fails that test.
|
|
A repo with no commit, a head that lacks the agent and a head behind the
|
|
config each fail the deploy with an error that names the agent. `MetaSync` records the commit,
|
|
with the hyperhive and config commits the head pins, in `hive-c0re`'s own state,
|
|
`/var/lib/hyperhive/meta-rev/<n>`, which the agent can't write (the container mounts its runtime
|
|
directory read-write); `Prebuild` and `Swap` read that
|
|
record, so both expect the closure of one commit. A first deploy resolves the commit in its `Create`
|
|
node, since it has no `MetaSync`.
|
|
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
|
|
the agent's record in the swarm's `agent-closures` bucket (the controller
|
|
writes it from the builder's reply) and have `hive-priv` download the closure
|
|
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then
|
|
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and
|
|
`CreateContainer`), which accepts only the path its last `RealiseClosure` for
|
|
that agent returned. On the wire the
|
|
`hive_priv_sock::PrivRequest` variants carry a required field:
|
|
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
|
|
The three cases when the swarm's closure isn't the one this hive expects
|
|
are in [`docs/swarm/builder.md`](../swarm/builder.md#rebuilds-download-the-closure).
|
|
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
|
runs the hyperhive of the controller's deploy, not its hive's.
|
|
- **Markers.** After a successful build `hive-c0re` writes the hyperhive
|
|
commit the head pins to `applied/.<n>.hyperhive-rev`, and the config
|
|
commit to `applied/.<n>.config-rev` (none for an archetype agent). The boot
|
|
reconcile ([below](#autoupdate-on-startup)) and the dashboard's update chip
|
|
read them.
|
|
|
|
## Deploy markers per agent
|
|
|
|
```
|
|
/var/lib/hyperhive/applied/ deploy markers - core-only
|
|
├── .<name>.hyperhive-rev # hyperhive commit of the last good build
|
|
└── .<name>.config-rev # config commit of the last good build
|
|
|
|
/var/lib/hyperhive/meta/ plain hive state - core
|
|
├── topology.json # the live agent set
|
|
└── swarm-resource-limits.json # the swarm's per-agent caps
|
|
```
|
|
|
|
hive-c0re keeps no copy of an agent's config: the forge's `agent-configs/<name>`
|
|
`main` is the only one, and a config change is a PR against it. The two markers
|
|
record what the agent's last successful build pinned, and nothing else.
|
|
|
|
Older hives also kept an *applied* repo at `applied/<name>` and a *proposed*
|
|
clone at `agents/<name>/config`. Nothing reads either any more, and you can
|
|
delete them.
|
|
|
|
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
|
|
The agent's `agent-configs/<n>/flake.nix`
|
|
is a fixed boilerplate that exports `nixosModules.default =
|
|
import ./agent.nix`; the meta flake imports that module and
|
|
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
|
|
|
|
### Forge config repos
|
|
|
|
The bundled `hive-forge` container runs on the swarm's forge host
|
|
(`deploy.forgejo.enable`, see [`../swarm/services.md`](../swarm/services.md)).
|
|
Every agent created through the swarm has a repo in the private
|
|
`agent-configs` Forgejo org, created and seeded by swarm-controller.
|
|
hive-c0re pushes nothing to it and keeps no clone: forge `main` advances only
|
|
by an operator's merge. The hive reads the head of `main`, to resolve the
|
|
commit `internal/meta` pins.
|
|
|
|
Each agent is a **write collaborator on its own** `agent-configs/<name>`
|
|
repo — so it can push a branch and open a config PR — but not a member
|
|
of any other agent's, so it can't reach another agent's config through
|
|
the forge. Branch protection keeps the agent off the `main` push
|
|
allowlist and allowlists merging to the `operators` team only, so an
|
|
agent can't fast-forward its own config or self-merge its PR (see
|
|
[Config changes](#config-changes) above). hive-c0re passes the credential to
|
|
each `git` call inline, never writing it into a repo's config, so
|
|
no stored token sits in a repo an agent might one day read.
|
|
|
|
The dashboard deep-links into this org — a `config repo` link
|
|
per container row. See `docs/web-ui/dashboard.md`.
|
|
|
|
### Submitting agent's view of config repos
|
|
|
|
No agent has a config repo mounted, not even one holding
|
|
`ManageRootAgent`: reading another agent's config means cloning its repo
|
|
from the forge, subject to the forge's own access rules.
|
|
|
|
An agent submits a change to another agent's config the same way it
|
|
makes any other change: **clone the target agent's config repo from the
|
|
forge into its own state dir, commit on a branch, open a PR**, and let
|
|
the operator review and merge it. By design, no second, mount-shaped
|
|
path reaches the same file without the review.
|
|
|
|
Agents holding the `manage_root_agent` permission (granted per agent in
|
|
the swarm's `agent-permissions` record; see
|
|
`docs/process/conventions.md#permissions`) get additional host-side bind
|
|
mounts via `set_nspawn_flags`:
|
|
|
|
- `/var/lib/hyperhive/agents/<n>/state` → `/agents/<n>/state` (RW) for
|
|
**every** agent. The permission means "may manage any agent," so the mounted
|
|
set is every agent.
|
|
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the hive's plain state
|
|
(topology, swarm limits).
|
|
|
|
An agent without the permission only has its direct children's config
|
|
dirs.
|
|
|
|
⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts
|
|
an agent, after its current turn, when its `manage_root_agent` changes.
|
|
|
|
No agent holds the permission by default: an agent whose record leaves
|
|
`permissions` unset gets the default, which doesn't include
|
|
`manage_root_agent`, whatever the agent's name. The swarm UI's setup
|
|
assistant creates ruth with it. For a ruth that predates that, whose
|
|
permissions were unset, swarm-controller writes the default plus
|
|
`manage_root_agent` into her key once at start, so she keeps her mounts;
|
|
it never touches a key that's already set.
|
|
|
|
## Startup migrations (older hosts)
|
|
|
|
hive-c0re runs a couple of idempotent migrations on every startup so a
|
|
host set up before the current deploy scheme (described above)
|
|
converges to it automatically. Each phase is a no-op
|
|
once already applied:
|
|
|
|
- **Agent files**: refreshes the files that follow the agent set
|
|
(`agent_set::refresh`) from the current agent list. Set
|
|
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
|
- **Legacy meta repo**: deletes the leftover `.git` in
|
|
`/var/lib/hyperhive/meta/` and the
|
|
unread `/var/lib/hyperhive/forge-core-token`. The files beside the `.git`
|
|
stay. The forge-side `core/meta` repo and `core` account aren't touched.
|
|
|
|
No state loss in either migration: claude creds, `/state/` notes, the
|
|
events DB all survive. The root
|
|
agent keeps its session; sub-agents stay logged in.
|
|
|
|
## The root/bootstrap container
|
|
|
|
The root agent container runs through the **same lifecycle as
|
|
sub-agents**. Nothing creates it at hive boot: you create ruth from the
|
|
swarm dashboard like any agent, and an existing ruth container starts with
|
|
the hive like every other. The root agent can clone
|
|
`agent-configs/ruth` to read its own `agent.nix` and open a config PR there
|
|
for operator review, same as any other agent.
|
|
|
|
Differences from sub-agents:
|
|
|
|
- Reference docs: her `agent.nix` turns on `services.hyperhive.agent.docs.enable`.
|
|
Her container builds from `agent-base` like every agent, and the seeded
|
|
`agent.nix` is the same for every name, so a newly created ruth has no docs
|
|
until a config PR adds that line.
|
|
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
|
FNV-1a hash as every other agent (8100..8999 range).
|
|
- The root agent holds `manage_root_agent` (the swarm UI's setup
|
|
assistant creates her with it), for which `set_nspawn_flags` adds extra
|
|
binds: every agent's `state` dir (RW) under `/agents/<n>/state`, and the
|
|
hive's plain state under `/meta` (RO).
|
|
- Its socket comes from the generic per-agent registration, in the same
|
|
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
|
not a separate manager-only path.
|
|
|
|
**Migration note** (for older hosts): drop any `containers.root =
|
|
{ ... }` block from your host NixOS config, and any `services.hyperhive.ruthless`
|
|
definition, then create ruth from the swarm dashboard.
|
|
|
|
## Root-agent policy
|
|
|
|
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
|
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
|
varies is which MCP tools it surfaces (gated by the agent's permissions,
|
|
which the swarm sets). No `role:manager` block renders only
|
|
for the root agent. The root agent's role-specific
|
|
behaviour comes from its AGENTS.md / agent-specific instructions, not
|
|
the system prompt template.
|
|
|
|
## Helper events
|
|
|
|
`hive_sh4re::manager::HelperEvent` is the wire type for an event urgent
|
|
enough to drive an immediate claude turn (delivered as a regular
|
|
`system` inbox message, same `recv` path as any other message). It
|
|
currently has one variant, `NeedsUpdate { agent }` (sub-agent's
|
|
recorded flake rev is stale, operator rebuilds from the dashboard) —
|
|
and nothing in the tree constructs it; this removal dropped the push path
|
|
(`Coordinator::notify_agent`) once its last caller
|
|
(`notify_manager`, used only for the crash event below) went away.
|
|
`HelperEvent` stays as the type for whatever comes along that
|
|
genuinely needs to interrupt an agent's turn loop, not because
|
|
anything live uses it today.
|
|
|
|
**A previously running container crashing** used to be exactly that
|
|
case (`ContainerCrash`, routed to the root agent via
|
|
`Coordinator::notify_manager`) — it no longer drives any agent turn at
|
|
all. `crash_watch` now only `tracing::warn!`s `"container crash
|
|
detected"`; a swarm-level grafana alert matches that log line directly
|
|
and pages the operator, with no agent (and no per-hive dashboard
|
|
banner) in the loop. See
|
|
[`docs/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re).
|
|
|
|
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
|
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
|
when convenient" events with no reason to drive an immediate turn, so
|
|
they go out via `swarm_notices::notify` instead
|
|
of a `HelperEvent`: `subsystem = "core"`, `key = "<event>:<agent>"`,
|
|
and a single free-text `summary` (`rebuilt_todo_summary` renders
|
|
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
|
|
|
|
To add a new lifecycle notice: if it's "FYI, check when convenient,"
|
|
call `swarm_notices::notify` — no new wire type needed. If it needs to
|
|
drive an immediate turn, decide first whether it actually needs an
|
|
*agent* in the loop at all (the crash case above didn't, once a
|
|
swarm-level alert could carry the same signal) — only reach for a new
|
|
`HelperEvent` variant + `notify_agent`-equivalent call site +
|
|
`prompts/system.md` update when the answer is yes.
|
|
|
|
## Autoupdate on startup
|
|
|
|
`hive-c0re serve` runs `auto_update::run` in a background task right
|
|
after opening the coordinator. It enumerates managed containers and
|
|
rebuilds any whose recorded hyperhive rev differs from the one
|
|
`internal/meta` pins now. That's how a hive that missed the swarm's
|
|
fan-out catches up. Sub-agents and the root agent go through the same
|
|
graceful rebuild subgraph.
|
|
|
|
"Rev" = the hyperhive commit pinned by the head of `internal/meta`, as
|
|
the hive reads it with its forge token. Marker
|
|
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`, written after
|
|
a successful build. If the hive can't read the pin (no forge token, the
|
|
forge down, `internal/meta` not published yet), the scan rebuilds
|
|
nothing; a rebuild would have no flake to build from either. A hive deploy
|
|
alone doesn't rebuild agents.
|
|
|
|
A stopped agent gets its rebuild at its next start: the start path upgrades
|
|
the start of a stale, stopped agent to a rebuild followed by the start. In
|
|
the window after a config merge, before the swarm's update has pinned the new
|
|
config, that rebuild fails with the error that names the agent, and the
|
|
agent stays stopped until the update lands and you ask for the start again.
|
|
|
|
The dashboard surfaces pending config changes per agent: a clickable
|
|
"needs update ↻" badge appears whenever the config commit that `internal/meta`
|
|
pins for the agent's next build differs from the one its last build pinned. The badge POSTs `/api/rebuild/<name>`, which inserts the
|
|
same `job_queue::templates::rebuild` DAG so manual triggers and the
|
|
startup scan can't drift. When at least one container is stale, a
|
|
top-level `↻ UPD4TE 4LL` button appears that loops over every
|
|
stale container.
|