383 lines
20 KiB
Markdown
383 lines
20 KiB
Markdown
# Config changes + system messages
|
|
|
|
Config changes are pull requests on the agent's config repo, which an
|
|
operator merges on the forge; that merge deploys them. The "System messages"
|
|
section covers the plain-text `system` messages hive-c0re sends agents.
|
|
|
|
## For operators
|
|
|
|
- **Config change** — an agent proposed a change to another agent's
|
|
config (or its own, via a sub-manager) as a pull request on
|
|
`agent-configs/<agent>`. Review it on the forge like any other PR, and
|
|
merge it there to deploy it. See [Config changes](#config-changes).
|
|
|
|
Everything below this point is the implementation detail behind that
|
|
flow.
|
|
|
|
## Config changes
|
|
|
|
Config changes flow through a **forge pull request** on the agent's
|
|
`agent-configs/<name>` repo — the same surface agents use for code PRs.
|
|
No MCP tool exists for config changes: opening the PR IS the request.
|
|
|
|
1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
|
|
tracked path, but `agent.nix` is the contract entry point), commits
|
|
with its own git identity, and pushes a branch + opens a PR with
|
|
`hive-forge` — the same way it would change any other repo.
|
|
The container has no config mount: the clone is how an agent reads its
|
|
own config as well as how it changes it.
|
|
Branch protection (the agent isn't on the `main` push allowlist;
|
|
merge and approvals allowlist = `operators` team; see "Forge config repos"
|
|
below) makes the agent a write collaborator that **can't merge its own
|
|
config PR**.
|
|
2. An operator reviews the PR **on the forge** (native diff, threaded
|
|
comments, CI status) and merges it there. Merging needs membership of
|
|
the `operators` team in `agent-configs`. The team starts empty: add
|
|
each operator by hand in the forge UI.
|
|
3. swarm-controller gets the `agent-configs` org's `pull_request`
|
|
delivery. A `closed` event with `merged: true` and base branch `main`
|
|
names the commit in `merge_commit_sha`.
|
|
4. The controller looks up which hive's wanted state places the agent
|
|
and publishes a deploy request carrying that commit on that hive's
|
|
deploy subject. With no such hive, or more than one, it deploys
|
|
nothing and logs a warning naming them.
|
|
5. If the closure the swarm declares for the agent is the one the hive last
|
|
applied (the `applied/.<n>.closure` marker), the hive does nothing.
|
|
Otherwise it queues a rebuild, which downloads the declared closure. The
|
|
request's commit plays no part: the hive keeps no copy of the config and
|
|
reads no forge head. An agent with no container on that hive yet gets its
|
|
first deploy from the declared closure.
|
|
|
|
This path runs **no eval-verify**. A merged config that doesn't
|
|
evaluate or build fails the rebuild, so the agent's rebuilds keep failing
|
|
until a fix merges. A failed rebuild shows on the hive like any other. A merge
|
|
whose webhook delivery never reaches the controller deploys nothing.
|
|
|
|
Creating a brand-new agent is swarm-level (`swarmctl agent create`,
|
|
`POST /api/agents` on the swarm controller), and no hive can originate
|
|
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
|
|
agent's config repo with a default `agent.nix` template, then asks the
|
|
target hive to deploy it.
|
|
An agent created from an [archetype](../swarm/archetypes.md) gets no
|
|
config repo: the swarm builds it from the archetype's module.
|
|
|
|
Changing what the template seeded isn't an exception: like every
|
|
later change, it's a PR on that config repo, made from a clone and
|
|
merged by an operator on the forge (see [Config changes](#config-changes)).
|
|
|
|
## Agent lifecycle
|
|
|
|
### Reminder delivery: file-path semantics
|
|
|
|
A reminder may carry a `file_path` (the agent-visible path inside its
|
|
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
|
|
|
1. **Translates** the container path to the host path
|
|
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
|
|
from outside the container.
|
|
2. **Validates** the path: rejects anything outside the agent's own state
|
|
subtree, containing `..` (path traversal), or with an empty relative
|
|
tail. On rejection hive-c0re skips the write and delivers the
|
|
original message inline with a warning — the reminder still fires.
|
|
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
|
|
canonicalizes the parent dir and re-verifies it lives under the agent's host
|
|
state root. hive-c0re opens the final file with
|
|
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
|
|
basename can't redirect the write to an arbitrary host path.
|
|
4. **Writes the body to disk** and delivers a short pointer message in its
|
|
place, keeping the agent's inbox / wake-prompt small while the agent
|
|
reads the bulky payload out of band.
|
|
|
|
`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
|
|
body strings before calling it.
|
|
|
|
### Destroy semantics
|
|
|
|
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
|
|
Stops + removes the nspawn container and drops the systemd drop-in.
|
|
Persistent state (claude credentials, `/state/` notes) is
|
|
**kept by default** — recreating the agent with the same name
|
|
reuses prior notes + login. With `purge = true` the agent's
|
|
`/var/lib/hyperhive/agents/<name>/` tree is also
|
|
wiped (creds + notes gone forever). The
|
|
root/bootstrap container is destroyable like any other, and stays gone
|
|
until you create ruth again from the swarm dashboard.
|
|
|
|
## Meta flake
|
|
|
|
A hive has no meta flake of its own. It builds every agent from the swarm's
|
|
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
|
|
It declares one flake input per agent (`agent-<n>.url =
|
|
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
|
|
agent's forge config repo, the one config PRs merge into) and one
|
|
`nixosConfigurations.<n>` output per agent. Each output wraps
|
|
`inputs.agent-<n>.nixosModules.default` with the identity +
|
|
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
|
|
(the dashboard port, the pinned claude-code, extra OTEL labels, the queue
|
|
address) aren't in it:
|
|
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
|
|
start (see `docs/process/conventions.md`).
|
|
A config that sits only in a local checkout, not on forge `main`, never
|
|
deploys.
|
|
|
|
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
|
|
state, bound read-only into the root agent's container as `/meta`:
|
|
`topology.json`, the swarm resource limits, and the permission seed files of a
|
|
hive that predates the swarm's permissions. `agent_set::refresh(agents:
|
|
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
|
|
nginx include in line with the live agents. `provision_container`, `destroy`,
|
|
`rebuild` and the startup migration call it. It refuses an empty agent list,
|
|
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
|
|
is unset, since every agent would come up with no forge.
|
|
|
|
## The swarm's meta flake
|
|
|
|
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
|
|
every agent any hive places (the pins:
|
|
[`docs/swarm/builder.md`](../swarm/builder.md#meta-flake-lock)). The swarm
|
|
builder builds every agent from it, and a hive deploys that build. While the
|
|
swarm has recorded no build of an agent, that agent's deploy fails and names the
|
|
agent. The controller queues
|
|
its update of the repo before the deploy request for a merged config commit
|
|
and for a new agent, and sends the deploy only when that update succeeded and
|
|
the swarm builder built the agent from the updated repo (the
|
|
[deploy gate](../swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
|
|
|
|
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
|
|
agent's record in `agent-closures` declares, whichever `internal/meta` commit
|
|
the builder built it from, and reads no forge head. The rebuild's `MetaSync`
|
|
node prepares it (the agent's archetype, its runtime directories, the agent
|
|
set), reads the record once and pins its store path. `Prebuild` and `Swap` use
|
|
the pin, so a record that changes mid-rebuild can't make `Swap` pull another
|
|
closure than `Prebuild` fetched, and the marker names the closure it applied. The
|
|
next autoupdate pass picks up the newer record. A first deploy has no
|
|
`MetaSync`; its `Create` reads the record once.
|
|
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
|
|
the agent's record in the swarm's `agent-closures` bucket (the controller
|
|
writes it from the builder's reply) and have `hive-priv` download the closure
|
|
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then
|
|
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and
|
|
`CreateContainer`), which accepts only the path its last `RealiseClosure` for
|
|
that agent returned. On the wire the
|
|
`hive_priv_sock::PrivRequest` variants carry a required field:
|
|
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
|
|
A rebuild with no record, or none the builder pushed, fails with an error
|
|
that names the agent; see
|
|
[`docs/swarm/builder.md`](../swarm/builder.md#rebuilds-download-the-closure).
|
|
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
|
runs the hyperhive of the controller's deploy, not its hive's.
|
|
- **Marker.** After a successful swap or create `hive-c0re` writes the closure's
|
|
store path to `applied/.<n>.closure`. The autoupdate pass
|
|
([below](#autoupdate-on-startup)) compares it with the record.
|
|
|
|
## Deploy markers per agent
|
|
|
|
```
|
|
/var/lib/hyperhive/applied/ deploy markers - core-only
|
|
└── .<name>.closure # store path of the last closure applied
|
|
|
|
/var/lib/hyperhive/meta/ plain hive state - core
|
|
├── topology.json # the live agent set
|
|
└── swarm-resource-limits.json # the swarm's per-agent caps
|
|
```
|
|
|
|
hive-c0re keeps no copy of an agent's config: the forge's `agent-configs/<name>`
|
|
`main` is the only one, and a config change is a PR against it. The marker
|
|
records the closure last applied, and nothing else. Hives that ran before the
|
|
marker existed have `.<name>.hyperhive-rev`, `.<name>.config-rev` and
|
|
`meta-rev/<name>` files instead. Nothing reads them any more, and you can
|
|
delete them.
|
|
|
|
Older hives also kept an *applied* repo at `applied/<name>` and a *proposed*
|
|
clone at `agents/<name>/config`. Nothing reads either any more, and you can
|
|
delete them.
|
|
|
|
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
|
|
The agent's `agent-configs/<n>/flake.nix`
|
|
is a fixed boilerplate that exports `nixosModules.default =
|
|
import ./agent.nix`; the meta flake imports that module and
|
|
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
|
|
|
|
### Forge config repos
|
|
|
|
The bundled `hive-forge` container runs on the swarm's forge host
|
|
(`deploy.forgejo.enable`, see [`../swarm/services.md`](../swarm/services.md)).
|
|
Every agent created through the swarm has a repo in the private
|
|
`agent-configs` Forgejo org, created and seeded by swarm-controller.
|
|
hive-c0re pushes nothing to it, keeps no clone and reads nothing from it: forge
|
|
`main` advances only by an operator's merge.
|
|
|
|
Each agent is a **write collaborator on its own** `agent-configs/<name>`
|
|
repo — so it can push a branch and open a config PR — but not a member
|
|
of any other agent's, so it can't reach another agent's config through
|
|
the forge. Branch protection keeps the agent off the `main` push
|
|
allowlist and allowlists merging to the `operators` team only, so an
|
|
agent can't fast-forward its own config or self-merge its PR (see
|
|
[Config changes](#config-changes) above). hive-c0re passes the credential to
|
|
each `git` call inline, never writing it into a repo's config, so
|
|
no stored token sits in a repo an agent might one day read.
|
|
|
|
The dashboard deep-links into this org — a `config repo` link
|
|
per container row. See `docs/web-ui/dashboard.md`.
|
|
|
|
### Submitting agent's view of config repos
|
|
|
|
No agent has a config repo mounted, not even one holding
|
|
`ManageRootAgent`: reading another agent's config means cloning its repo
|
|
from the forge, subject to the forge's own access rules.
|
|
|
|
An agent submits a change to another agent's config the same way it
|
|
makes any other change: **clone the target agent's config repo from the
|
|
forge into its own state dir, commit on a branch, open a PR**, and let
|
|
the operator review and merge it. By design, no second, mount-shaped
|
|
path reaches the same file without the review.
|
|
|
|
Agents holding the `manage_root_agent` permission (granted per agent in
|
|
the swarm's `agent-permissions` record; see
|
|
`docs/process/conventions.md#permissions`) get additional host-side bind
|
|
mounts via `set_nspawn_flags`:
|
|
|
|
- `/var/lib/hyperhive/agents/<n>/state` → `/agents/<n>/state` (RW) for
|
|
**every** agent. The permission means "may manage any agent," so the mounted
|
|
set is every agent.
|
|
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the hive's plain state
|
|
(topology, swarm limits).
|
|
|
|
An agent without the permission only has its direct children's config
|
|
dirs.
|
|
|
|
⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts
|
|
an agent, after its current turn, when its `manage_root_agent` changes.
|
|
|
|
No agent holds the permission by default: an agent whose record leaves
|
|
`permissions` unset gets the default, which doesn't include
|
|
`manage_root_agent`, whatever the agent's name. The swarm UI's setup
|
|
assistant creates ruth with it. For a ruth that predates that, whose
|
|
permissions were unset, swarm-controller writes the default plus
|
|
`manage_root_agent` into her key once at start, so she keeps her mounts;
|
|
it never touches a key that's already set.
|
|
|
|
## Startup migrations (older hosts)
|
|
|
|
hive-c0re runs a couple of idempotent migrations on every startup so a
|
|
host set up before the current deploy scheme (described above)
|
|
converges to it automatically. Each phase is a no-op
|
|
once already applied:
|
|
|
|
- **Agent files**: refreshes the files that follow the agent set
|
|
(`agent_set::refresh`) from the current agent list. Set
|
|
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
|
- **Legacy meta repo**: deletes the leftover `.git` in
|
|
`/var/lib/hyperhive/meta/` and the
|
|
unread `/var/lib/hyperhive/forge-core-token`. The files beside the `.git`
|
|
stay. The forge-side `core/meta` repo and `core` account aren't touched.
|
|
|
|
No state loss in either migration: claude creds, `/state/` notes, the
|
|
events DB all survive. The root
|
|
agent keeps its session; sub-agents stay logged in.
|
|
|
|
## The root/bootstrap container
|
|
|
|
The root agent container runs through the **same lifecycle as
|
|
sub-agents**. Nothing creates it at hive boot: you create ruth from the
|
|
swarm dashboard like any agent, and an existing ruth container starts with
|
|
the hive like every other. The root agent can clone
|
|
`agent-configs/ruth` to read its own `agent.nix` and open a config PR there
|
|
for operator review, same as any other agent.
|
|
|
|
Differences from sub-agents:
|
|
|
|
- Reference docs: her `agent.nix` turns on `services.hyperhive.agent.docs.enable`.
|
|
Her container builds from `agent-base` like every agent, and the seeded
|
|
`agent.nix` is the same for every name, so a newly created ruth has no docs
|
|
until a config PR adds that line.
|
|
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
|
FNV-1a hash as every other agent (8100..8999 range).
|
|
- The root agent holds `manage_root_agent` (the swarm UI's setup
|
|
assistant creates her with it), for which `set_nspawn_flags` adds extra
|
|
binds: every agent's `state` dir (RW) under `/agents/<n>/state`, and the
|
|
hive's plain state under `/meta` (RO).
|
|
- Its socket comes from the generic per-agent registration, in the same
|
|
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
|
not a separate manager-only path.
|
|
|
|
**Migration note** (for older hosts): drop any `containers.root =
|
|
{ ... }` block from your host NixOS config, and any `services.hyperhive.ruthless`
|
|
definition, then create ruth from the swarm dashboard.
|
|
|
|
## Root-agent policy
|
|
|
|
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
|
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
|
varies is which MCP tools it surfaces (gated by the agent's permissions,
|
|
which the swarm sets). No `role:manager` block renders only
|
|
for the root agent. The root agent's role-specific
|
|
behaviour comes from its AGENTS.md / agent-specific instructions, not
|
|
the system prompt template.
|
|
|
|
## System messages
|
|
|
|
Hive-c0re has no typed event that drives an immediate claude turn. A
|
|
message from sender `system` is plain inbox text, and the harness logs it
|
|
(`log_system_event` in `hive-agent/src/main.rs`) without parsing the body.
|
|
Today hive-c0re sends two: the `/knowledge updated` broadcast
|
|
(`workers/knowledge.rs`) and the wake message for a graceful stop
|
|
(`Coordinator::kick_agent`).
|
|
|
|
**A previously running container crashing** drives no agent turn.
|
|
`crash_watch` only `tracing::warn!`s `"container crash
|
|
detected"`; a swarm-level grafana alert matches that log line directly
|
|
and pages the operator, with no agent (and no per-hive dashboard
|
|
banner) in the loop. See
|
|
[`docs/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re).
|
|
|
|
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
|
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
|
when convenient" events with no reason to drive an immediate turn, so
|
|
they go out via `swarm_notices::notify` instead: `subsystem = "core"`,
|
|
`key = "<event>:<agent>"`, and a single free-text `summary`
|
|
(`rebuilt_todo_summary` renders `Rebuilt`'s `ok`/`note`/`sha`/`tag`
|
|
fields into that string).
|
|
|
|
To add a new lifecycle notice: if it's "FYI, check when convenient,"
|
|
call `swarm_notices::notify` — no new wire type needed. If it needs to
|
|
drive an immediate turn, decide first whether it actually needs an
|
|
*agent* in the loop at all (the crash case above didn't, once a
|
|
swarm-level alert could carry the same signal).
|
|
|
|
## Autoupdate on startup
|
|
|
|
`hive-c0re serve` runs `auto_update::run` in a background task right
|
|
after opening the coordinator. It enumerates managed containers and
|
|
rebuilds any whose closure marker differs from the store path the agent's
|
|
record in `agent-closures` declares. That's how a hive that missed the
|
|
swarm's deploy request catches up, and it repeats every five minutes, which
|
|
covers a request lost while the hive is up. Sub-agents and the root agent go
|
|
through the same graceful rebuild subgraph.
|
|
|
|
The comparison reads the record and the marker
|
|
(`/var/lib/hyperhive/applied/.<name>.closure`, written after a successful
|
|
swap or create) and nothing on the forge. Where the marker is missing, as for
|
|
every agent on the first boot after the marker arrived, the hive reads the
|
|
closure the container's profile
|
|
(`/nix/var/nix/profiles/per-container/h-<name>/system`) points at and writes
|
|
that as the marker, so the first pass catches an agent that doesn't run its declared
|
|
closure. Those first-boot rebuilds go through the same job queue as
|
|
any other: each holds one of the `buildSlots`, so they don't swap all at once.
|
|
It rebuilds nothing when the hive can't tell: no connection to the swarm queue,
|
|
no record, a closure the builder didn't push, or a profile it can't read or that
|
|
leaves the nix store. The periodic pass
|
|
attempts a declared store path once per process, and skips an agent while a
|
|
rebuild of it waits or runs, so a closure that fails to deploy doesn't
|
|
restart its agent every five minutes. Restarting `hive-c0re`, or a rebuild from
|
|
the agent's menu, tries again.
|
|
|
|
A stopped agent gets its rebuild at its next start: the start path upgrades
|
|
the start of a stale, stopped agent to a rebuild followed by the start. In
|
|
the window after a config merge, before the swarm's builder has recorded the
|
|
new closure, the rebuild deploys the previous one.
|
|
|
|
The dashboard shows no pending-update state: the hive applies a changed
|
|
declaration itself. The agent menu's rebuild queues the same
|
|
`job_queue::templates::rebuild` DAG, which downloads and applies the declared
|
|
closure again.
|