90e8158f37
ruth is created from the swarm dashboard like any agent. An existing ruth container starts at boot through the boot reconcile, like every other agent. A definition of services.hyperhive.ruthless now fails evaluation with a removal message.
417 lines
21 KiB
Markdown
417 lines
21 KiB
Markdown
# Config changes + helper events
|
|
|
|
Config changes are pull requests on the agent's config repo, which an
|
|
operator merges on the forge; that merge deploys them. Helper events are
|
|
how agents stay informed about lifecycle changes that need a reaction.
|
|
|
|
## For operators
|
|
|
|
- **Config change** — an agent proposed a change to another agent's
|
|
config (or its own, via a sub-manager) as a pull request on
|
|
`agent-configs/<agent>`. Review it on the forge like any other PR, and
|
|
merge it there to deploy it. See [Config changes](#config-changes).
|
|
|
|
Everything below this point is the implementation detail behind that
|
|
flow.
|
|
|
|
## Config changes
|
|
|
|
Config changes flow through a **forge pull request** on the agent's
|
|
`agent-configs/<name>` repo — the same surface agents use for code PRs.
|
|
No MCP tool exists for config changes: opening the PR IS the request.
|
|
|
|
1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
|
|
tracked path, but `agent.nix` is the contract entry point), commits
|
|
with its own git identity, and pushes a branch + opens a PR with
|
|
`hive-forge` — the same way it would change any other repo.
|
|
The bind-mounted `/agents/<name>/config/` is a **copy for reading** a
|
|
config, not the tree to edit: authoring in place there produces no PR.
|
|
(it's currently mounted read-write, which is a defect tracked
|
|
separately, not an authoring path.)
|
|
Branch protection (the agent isn't on the `main` push allowlist;
|
|
merge and approvals allowlist = `operators` team; see "Forge config repos"
|
|
below) makes the agent a write collaborator that **can't merge its own
|
|
config PR**.
|
|
2. An operator reviews the PR **on the forge** (native diff, threaded
|
|
comments, CI status) and merges it there. Merging needs membership of
|
|
the `operators` team in `agent-configs`. The team starts empty: add
|
|
each operator by hand in the forge UI.
|
|
3. swarm-controller gets the `agent-configs` org's `pull_request`
|
|
delivery. A `closed` event with `merged: true` and base branch `main`
|
|
names the commit in `merge_commit_sha`.
|
|
4. The controller looks up which hive's wanted state places the agent
|
|
and publishes a deploy request carrying that commit on that hive's
|
|
deploy subject. With no such hive, or more than one, it deploys
|
|
nothing and logs a warning naming them.
|
|
5. If the hive's `applied/main` already is that commit, it does nothing.
|
|
Otherwise it fetches the config repo's `main`. A commit that
|
|
`applied/main` already contains is a no-op too: requests for two merges
|
|
can arrive in either order. A commit that descends from `applied/main`
|
|
is fast-forwarded to and a rebuild queued. A failure before the rebuild
|
|
(the fetch, or a commit that has diverged from `applied/main`) deploys
|
|
nothing and logs `swarm events: rev not
|
|
applied; nothing queued` at error, which fires the Grafana alert
|
|
`config rev refused`. An agent with no container on that hive yet ignores
|
|
the commit; its first deploy builds what it seeds.
|
|
|
|
This path runs **no eval-verify**. A merged config that doesn't
|
|
evaluate or build fails the rebuild, and `applied/main` stays at that
|
|
commit, so the agent's rebuilds keep failing until a fix merges. A
|
|
failed rebuild shows on the hive like any other. A merge whose webhook delivery never reaches the controller
|
|
deploys nothing.
|
|
|
|
Creating a brand-new agent is swarm-level (`swarmctl agent create`,
|
|
`POST /api/agents` on the swarm controller), and no hive can originate
|
|
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
|
|
agent's config repo with a default `agent.nix` template, then asks the
|
|
target hive to deploy it.
|
|
|
|
Changing what the template seeded isn't a special case: like every
|
|
later change, it's a PR on that config repo, made from a clone and
|
|
merged by an operator on the forge (see [Config changes](#config-changes)).
|
|
|
|
## Agent lifecycle
|
|
|
|
### Reminder delivery: file-path semantics
|
|
|
|
A reminder may carry a `file_path` (the agent-visible path inside its
|
|
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
|
|
|
1. **Translates** the container path to the host path
|
|
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
|
|
from outside the container.
|
|
2. **Validates** the path: rejects anything outside the agent's own state
|
|
subtree, containing `..` (path traversal), or with an empty relative
|
|
tail. On rejection hive-c0re skips the write and delivers the
|
|
original message inline with a warning — the reminder still fires.
|
|
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
|
|
canonicalizes the parent dir and re-verifies it lives under the agent's host
|
|
state root. hive-c0re opens the final file with
|
|
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
|
|
basename can't redirect the write to an arbitrary host path.
|
|
4. **Writes the body to disk** and delivers a short pointer message in its
|
|
place, keeping the agent's inbox / wake-prompt small while the agent
|
|
reads the bulky payload out of band.
|
|
|
|
`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
|
|
body strings before calling it.
|
|
|
|
### Destroy semantics
|
|
|
|
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
|
|
Stops + removes the nspawn container and drops the systemd drop-in.
|
|
Persistent state (proposed/applied repos, claude credentials, `/state/`
|
|
notes) is
|
|
**kept by default** — recreating the agent with the same name
|
|
reuses prior config + login. With `purge = true` the agent's
|
|
`/var/lib/hyperhive/{agents,applied}/<name>/` trees are also
|
|
wiped (config history + creds + notes gone forever). The
|
|
root/bootstrap container is destroyable like any other, and stays gone
|
|
until you create ruth again from the swarm dashboard.
|
|
|
|
## Meta flake
|
|
|
|
The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
|
|
declares one flake input per agent (`agent-<n>.url =
|
|
"git+file://<applied>"`, the agent's local `applied` mirror) and one
|
|
`nixosConfigurations.<n>` output per agent. Each output wraps
|
|
`inputs.agent-<n>.nixosModules.default` with the identity +
|
|
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module.
|
|
Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
|
|
|
|
The declared input url is the agent's **local `applied` mirror**
|
|
(`/var/lib/hyperhive/applied/<n>`) of its forge config repo
|
|
(`agent-configs/<n>`, the one config PRs merge into). The meta flake
|
|
therefore resolves entirely from on-disk state: building, and booting,
|
|
never depend on the forge being reachable. hive-c0re fills the mirror
|
|
by fast-forwarding `applied/<n>/main` to the merged config commit,
|
|
reading the forge with the hive's read token
|
|
(`forge::reader_token()`); the forge repo is the review and audit
|
|
surface, not a flake input. The rebuild paths re-lock the input with
|
|
`--override-input agent-<n> git+file:///var/lib/hyperhive/applied/<n>`,
|
|
locking the exact config `applied/<n>/main` was fast-forwarded to.
|
|
Rebuilds fire on crash-restart and meta bumps, not just config
|
|
merges, so the lock stays on that head. `sync_agents` re-renders and
|
|
re-locks the persistent input.
|
|
|
|
Lock updates are single-phase and commit when the lock changed:
|
|
`meta::lock_update_for_rebuild(name)` relocks one agent's input for a
|
|
relocking rebuild (the manual `↻ R3BU1LD` button, and the rebuild a
|
|
merged config commit queues), and `meta::lock_update_hyperhive()` is the
|
|
autoupdate flake-rev bump (one shot before per-agent rebuilds).
|
|
|
|
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
|
|
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
|
|
per-hive config — is the idempotent reconciler called by `provision_container`,
|
|
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
|
|
from the agent list; if it differs from disk, runs
|
|
`nix flake lock` + commits as `regenerate meta flake` (or
|
|
`seed meta from N agent(s)` on the first call).
|
|
|
|
The root agent has `/meta` RO-bound inside its container:
|
|
`git -C /meta log --oneline` is the swarm-wide deploy log,
|
|
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
|
|
resolves which sha the flake pins each agent at right now.
|
|
Dashboard surfaces the same info as a `deployed:<sha12>` chip
|
|
per container row.
|
|
|
|
## Two repos per agent
|
|
|
|
```
|
|
/var/lib/hyperhive/agents/<name>/config/ proposed — parent mount is RO
|
|
└── <anything> # any files the submitting
|
|
# agent wants in the commit.
|
|
# agent.nix is the
|
|
# convention entry
|
|
# point; flake.nix is
|
|
# tracked boilerplate
|
|
# (submitting agent doesn't
|
|
# edit it).
|
|
|
|
/var/lib/hyperhive/applied/<name>/ applied — core-only
|
|
├── .git/ # tag-rich history
|
|
├── flake.nix # tracked, fixed
|
|
│ # boilerplate exporting
|
|
│ # nixosModules.default
|
|
├── agent.nix # working tree of main
|
|
└── <other committed files> # also tracked
|
|
|
|
/var/lib/hyperhive/meta/ swarm-wide flake — core
|
|
├── .git/ # one commit per lock
|
|
│ # change
|
|
├── flake.nix # generated from agent set
|
|
└── flake.lock # pins each agent's sha
|
|
```
|
|
|
|
Why two physical repos: the submitting agent's `/agents/<n>/config/` is
|
|
RW — a buggy or hostile agent can `git clean -fdx` its own
|
|
proposed tree. The applied repo is never bind-mounted (except
|
|
the read-only `.git` exposure described below) so a destructive
|
|
move inside the container can't reach it.
|
|
|
|
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
|
|
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
|
|
is a fixed boilerplate that exports `nixosModules.default =
|
|
import ./agent.nix`; the meta flake imports that module and
|
|
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` /
|
|
`HIVE_DASHBOARD_PORT`.
|
|
|
|
### Tag state machine
|
|
|
|
hive-c0re plants `deployed/0` on the seed commit at first spawn. A
|
|
merged config commit that deploys plants no tag: the merged PR on the
|
|
forge and meta's lock commits record it. A config PR nobody merges
|
|
carries no extra state on the forge side: the PR stays open, and
|
|
the submitter pushes again (or closes it) to retry.
|
|
|
|
### Forge config repos
|
|
|
|
The bundled `hive-forge` container runs on the swarm's forge host
|
|
(`deploy.forgejo.enable`, see [`../swarm/services.md`](../swarm/services.md)).
|
|
Every agent created through the swarm has a repo in the private
|
|
`agent-configs` Forgejo org, created and seeded by swarm-controller.
|
|
hive-c0re pushes nothing to it: the applied repo's tags (`deployed/0`)
|
|
exist only locally, and forge `main` advances only by an operator's merge.
|
|
hive-c0re reads from it — reseeding a purged `applied`
|
|
(`forge::fetch_config_main_into_applied`), catching `applied` up, and
|
|
seeding `proposed` at first spawn (`forge::clone_config_into_proposed`).
|
|
|
|
A legacy agent spawned by the hive alone has no swarm-seeded repo; if its
|
|
`agent-configs/<name>` has no `main`, a first spawn falls back to the local
|
|
template, and reseeding a purged `applied` fails with a `destroy --purge`
|
|
and re-spawn instruction.
|
|
|
|
Each agent is a **write collaborator on its own** `agent-configs/<name>`
|
|
repo — so it can push a branch and open a config PR — but not a member
|
|
of any other agent's, so it can't reach another agent's config through
|
|
the forge. Branch protection keeps the agent off the `main` push
|
|
allowlist and allowlists merging to the `operators` team only, so an
|
|
agent can't fast-forward its own config or self-merge its PR (see
|
|
[Config changes](#config-changes) above). hive-c0re passes the credential to
|
|
each `git` call inline, never writing it into
|
|
`applied/<n>/.git/config`; that repo is RO-bind-mounted into the root
|
|
agent, and a stored token would leak core's admin credential to an
|
|
agent.
|
|
|
|
The dashboard deep-links into this org — a `config repo` link
|
|
per container row. See `docs/web-ui/dashboard.md`.
|
|
|
|
### Submitting agent's view of config repos
|
|
|
|
An agent holding `ManageRootAgent` has every other agent's config repo
|
|
bind-mounted **read-only** (`hive-c0re/src/lifecycle/host_config.rs`
|
|
calls `bind_child_agent_dirs` for each entry in
|
|
`topology::all_agents()`). it's a copy to *read* another agent's
|
|
current config — not an editing surface. An agent without the
|
|
permission sees no other agent's config at all.
|
|
|
|
An agent submits a change to another agent's config the same way it
|
|
makes any other change: **clone the target agent's config repo from the
|
|
forge into its own state dir, commit on a branch, open a PR**, and let
|
|
the operator review and merge it. By design, no second, mount-shaped
|
|
path reaches the same file without the review.
|
|
|
|
Agents holding the `manage_root_agent` permission (granted per agent in
|
|
the swarm's `agent-permissions` record; see
|
|
`docs/process/conventions.md#permissions`) get additional host-side bind
|
|
mounts via `set_nspawn_flags`:
|
|
|
|
- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — **every** agent's
|
|
proposed repo, not just direct children. The permission means "may
|
|
manage any agent", so the mounted set is every agent.
|
|
- `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every agent's
|
|
authoritative applied repo, including `.git`.
|
|
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide
|
|
deploy flake.
|
|
|
|
An agent without the permission only has its direct children's config
|
|
dirs.
|
|
|
|
⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts
|
|
an agent, after its current turn, when its `manage_root_agent` changes.
|
|
|
|
The root agent holds the permission by default: an agent whose record
|
|
leaves `permissions` unset gets the default, which includes
|
|
`manage_root_agent` for `ruth` and for no other agent. Setting her
|
|
permissions without it revokes it.
|
|
|
|
Each proposed repo (`/agents/<n>/config/`) is pre-configured
|
|
with `applied` as a git remote pointing at
|
|
`/applied/<n>/.git`. Useful incantations from inside an agent with
|
|
the full `/applied` mount:
|
|
|
|
```sh
|
|
git -C /agents/<n>/config fetch applied
|
|
git -C /agents/<n>/config log applied/main --oneline
|
|
git -C /agents/<n>/config show applied/refs/tags/deployed/0 # the seed commit
|
|
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
|
|
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
|
|
|
|
git -C /meta log --oneline # swarm-wide deploy history
|
|
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
|
|
```
|
|
|
|
The RO binds block push at the kernel level — git plumbing inside the
|
|
container can't corrupt either authoritative repo.
|
|
|
|
## Startup migrations (older hosts)
|
|
|
|
hive-c0re runs a couple of idempotent migrations on every startup so a
|
|
host set up before the tag-driven-deploy + meta-flake scheme (both
|
|
described above) converges to it automatically. Each phase is a no-op
|
|
once already applied:
|
|
|
|
- **Tags**: hive-c0re tags agents from before the tag-driven scheme
|
|
`deployed/0` on `main` once. Non-destructive — it doesn't touch live
|
|
containers, state dirs, or claude creds.
|
|
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
|
|
module-only boilerplate, wires the `applied` remote in each proposed
|
|
repo, and bootstraps the meta repo from the current agent list. Set
|
|
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
|
|
|
No state loss in either migration: claude creds, `/state/` notes, the
|
|
events DB, and both proposed + applied history all survive. The root
|
|
agent keeps its session; sub-agents stay logged in.
|
|
|
|
## The root/bootstrap container
|
|
|
|
The root agent container runs through the **same lifecycle as
|
|
sub-agents**. Nothing creates it at hive boot: you create ruth from the
|
|
swarm dashboard like any agent, and an existing ruth container starts with
|
|
the hive like every other. The root agent's flake lives at
|
|
`/var/lib/hyperhive/applied/ruth/`; its proposed config at
|
|
`/var/lib/hyperhive/agents/ruth/config/`. The root agent can edit its own
|
|
`agent.nix` (visible inside the container at `/agents/ruth/config/`)
|
|
and open a config PR on `agent-configs/ruth` for operator review,
|
|
same as any other agent.
|
|
|
|
Differences from sub-agents:
|
|
|
|
- `flake.nix` extends `hyperhive.nixosConfigurations.ruth`
|
|
(vs `agent-base`).
|
|
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
|
FNV-1a hash as every other agent (8100..8999 range).
|
|
- `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents`
|
|
→ `/agents` (RW) so the root agent can edit per-agent proposed repos,
|
|
and `/var/lib/hyperhive/applied` → `/applied` (RO) so the root agent
|
|
can `git fetch` deployed/failed/denied tags from any agent's
|
|
authoritative applied repo (see "Root-agent view of applied" below).
|
|
- `socket_server::start_manager` binds the root agent's socket,
|
|
pure transport with no dedicated helpers — it uses the same
|
|
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
|
not a special manager-only path.
|
|
|
|
**Migration note** (for older hosts): drop any `containers.root =
|
|
{ ... }` block from your host NixOS config, and any `services.hyperhive.ruthless`
|
|
definition, then create ruth from the swarm dashboard.
|
|
|
|
## Root-agent policy
|
|
|
|
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
|
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
|
varies is which MCP tools it surfaces (gated by the agent's permissions,
|
|
which the swarm sets). No `role:manager` block renders only
|
|
for the root agent. The root agent's role-specific
|
|
behaviour comes from its CLAUDE.md / agent-specific instructions, not
|
|
the system prompt template.
|
|
|
|
## Helper events
|
|
|
|
`hive_sh4re::manager::HelperEvent` is the wire type for an event urgent
|
|
enough to drive an immediate claude turn (delivered as a regular
|
|
`system` inbox message, same `recv` path as any other message). It
|
|
currently has one variant, `NeedsUpdate { agent }` (sub-agent's
|
|
recorded flake rev is stale, operator rebuilds from the dashboard) —
|
|
and nothing in the tree constructs it; this removal dropped the push path
|
|
(`Coordinator::notify_agent`) once its last caller
|
|
(`notify_manager`, used only for the crash event below) went away.
|
|
`HelperEvent` stays as the type for whatever comes along that
|
|
genuinely needs to interrupt an agent's turn loop, not because
|
|
anything live uses it today.
|
|
|
|
**A previously-running container crashing** used to be exactly that
|
|
case (`ContainerCrash`, routed to the root agent via
|
|
`Coordinator::notify_manager`) — it no longer drives any agent turn at
|
|
all. `crash_watch` now only `tracing::warn!`s `"container crash
|
|
detected"`; a swarm-level grafana alert matches that log line directly
|
|
and pages the operator, with no agent (and no per-hive dashboard
|
|
banner) in the loop. See
|
|
[`docs/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re).
|
|
|
|
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
|
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
|
when convenient" events with no reason to drive an immediate turn, so
|
|
they go out via `swarm_notices::notify` instead
|
|
of a `HelperEvent`: `subsystem = "core"`, `key = "<event>:<agent>"`,
|
|
and a single free-text `summary` (`rebuilt_todo_summary` renders
|
|
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
|
|
|
|
To add a new lifecycle notice: if it's "FYI, check when convenient,"
|
|
call `swarm_notices::notify` — no new wire type needed. If it needs to
|
|
drive an immediate turn, decide first whether it actually needs an
|
|
*agent* in the loop at all (the crash case above didn't, once a
|
|
swarm-level alert could carry the same signal) — only reach for a new
|
|
`HelperEvent` variant + `notify_agent`-equivalent call site +
|
|
`prompts/system.md` update when the answer is yes.
|
|
|
|
## Autoupdate on startup
|
|
|
|
`hive-c0re serve` runs `auto_update::run` in a background task right
|
|
after opening the coordinator. It enumerates managed containers and
|
|
rebuilds any whose recorded hyperhive rev differs from the current
|
|
one — sub-agents and the root agent go through the same
|
|
`job_queue::templates::rebuild` DAG.
|
|
|
|
"Rev" = canonical filesystem path of
|
|
`services.hyperhive.c0re.hyperhiveFlake`. Marker
|
|
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
|
|
flake input has no canonical path (for example a `github:` URL),
|
|
autoupdate is a no-op — rebuild manually.
|
|
|
|
The dashboard surfaces pending updates per agent: a clickable
|
|
"needs update ↻" badge appears whenever the marker differs from
|
|
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
|
|
same `job_queue::templates::rebuild` DAG so manual triggers and the
|
|
startup scan can't drift. When at least one container is stale, a
|
|
top-level `↻ UPD4TE 4LL` button appears that loops over every
|
|
stale container.
|