docs: restructure into topic subdirectories, collapse duplicated index
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
This commit is contained in:
iris
mara
parent
e4a22b4190
commit
07b62612b0
124 changed files with 301 additions and 377 deletions
246
docs/agent-lifecycle/agent-hierarchy.md
Normal file
246
docs/agent-lifecycle/agent-hierarchy.md
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
# Agent hierarchy & privileges
|
||||
|
||||
Every agent has a place in an operator-editable parent/child tree, used
|
||||
to scope which agents can manage which others. This doc covers how the
|
||||
tree is stored and edited today, the rules that are meant to run on top
|
||||
of it once enforcement is finished, and where the manager still gets
|
||||
special-cased in the meantime. Tracking issue: hyperhive#361
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361`).
|
||||
|
||||
## Where the tree lives
|
||||
|
||||
Topology lives in the hive-c0re-owned **meta repo**, alongside
|
||||
`flake.nix`, at `/var/lib/hyperhive/meta/topology.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"ruth": null,
|
||||
"alice": null,
|
||||
"bob": "alice"
|
||||
}
|
||||
```
|
||||
|
||||
`null` = root-level agent. New agents **default to root** — there is no
|
||||
structural manager that everything hangs under. Hierarchy is built
|
||||
explicitly: an agent that requests a sub-agent gets a
|
||||
requester-as-parent edge written at its `init_config` approval (so
|
||||
`bob` above was spawned by `alice`), and the operator can reparent any
|
||||
agent, including the bootstrap container (`ruth`) — it's just another
|
||||
root. The manager is reparentable like any other agent; there's no
|
||||
"structurally root" carve-out. Its privileges live on its MCP socket,
|
||||
not its tree position (see *Manager special-casing today* below).
|
||||
|
||||
### Reparenting
|
||||
|
||||
- CLI: `hivectl agent <child> set-parent --parent <new>` (or `--root`
|
||||
to promote). Exactly one of `--parent` / `--root` is required.
|
||||
- Dashboard: `POST /api/topology/set-parent` (form fields `child`,
|
||||
optional `new_parent` — absent / empty ⇒ promote to root).
|
||||
- Wire: `HostRequest::SetParent { child, new_parent: Option<String> }`.
|
||||
|
||||
All three go through the same validation, which refuses:
|
||||
|
||||
- unknown `child` / `new_parent` (typo guard),
|
||||
- self-parenting,
|
||||
- cycles (a bounded ancestor walk — moving the manager under one of
|
||||
its own descendants is the only real safety concern here, and it's
|
||||
caught the same way as any other agent).
|
||||
|
||||
Setting a parent to its current value is a no-op (no disk write). A
|
||||
successful change triggers an immediate rescan, so connected dashboard
|
||||
viewers see the tree repaint without polling.
|
||||
|
||||
### Why meta, not per-agent `agent.nix`
|
||||
|
||||
An agent shouldn't be able to claim a parent without that parent's
|
||||
consent, and operator-driven re-parenting shouldn't require touching
|
||||
the moved agent's config. Topology IS a system-level concern; meta is
|
||||
where system-level facts live.
|
||||
|
||||
### How `topology.json` gets updated
|
||||
|
||||
- **Read** — parsed into an agent→parent map; a missing or unparsable
|
||||
file degrades safely to "every agent is root" (covers a fresh
|
||||
install that hasn't synced yet).
|
||||
- **Reconcile** — runs alongside the periodic meta/flake regeneration.
|
||||
New agents default to root unless they already carry an explicit
|
||||
parent edge from an `init_config` approval; existing entries
|
||||
(including operator overrides) are preserved; removed agents drop.
|
||||
Agents that are approved but not yet spawned keep their edge too, so
|
||||
it survives the gap until the container actually appears.
|
||||
- **Inject** — each container's parent (if any) is exposed to its own
|
||||
environment as `HIVE_PARENT`, so the harness / system-prompt
|
||||
renderer can see it.
|
||||
- **Surface** — every rescan re-reads `topology.json` and populates
|
||||
`ContainerView.parent`, which the dashboard renders as a tree.
|
||||
|
||||
See `hive-c0re/src/agent_config/topology.rs` and `hive-c0re/src/meta.rs`'s
|
||||
module docs for the exact call chain.
|
||||
|
||||
### Current limitation: state-dir visibility lags topology
|
||||
|
||||
Reparenting today is purely a JSON edit. Only the top-level manager
|
||||
(`root`) gets `/var/lib/hyperhive/agents` bind-mounted at `/agents` in
|
||||
its container, so sub-agents don't yet see their would-be children's
|
||||
state dirs. Once sub-manager bind mounts land alongside capability
|
||||
enforcement, reparenting will grow a companion
|
||||
umount-old / mount-new / restart-cascade step.
|
||||
|
||||
## Planned topology semantics (once ancestor-based enforcement lands)
|
||||
|
||||
| operation | who can do it |
|
||||
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
|
||||
| `kill` / `start` / `restart` / `update` (any descendant) | any ancestor |
|
||||
| `request_init_config` (spawn a new child) | any agent, child added under self |
|
||||
| config change via forge PR (any descendant's config) | any ancestor |
|
||||
| `get_logs` (any descendant) | any ancestor |
|
||||
| moderate reminders (cancel any open thread of a descendant) | any ancestor |
|
||||
| `send` / `recv` routing | parent ↔ same-parent siblings ↔ self ↔ descendants; explicit allow-list for anyone else |
|
||||
| `request_update_meta_inputs` (bump meta lock) | root agents only (today: just `manager`) |
|
||||
|
||||
"Ancestor" walks `ContainerView.parent` chains; cycles are guarded by a
|
||||
visited-set at dispatch time (a malformed `topology.json` can't lock
|
||||
the dispatcher into a loop).
|
||||
|
||||
## Manager special-casing today
|
||||
|
||||
Enforcement of the ancestor rules above isn't fully wired yet, so the
|
||||
**manager (`ruth`) still gets some hard-coded special treatment**
|
||||
other agents don't:
|
||||
|
||||
- **Naming/bootstrap** — the manager's broker recipient name, state-dir
|
||||
key, and nixos-container name are all `ruth` (container `h-ruth`).
|
||||
`hive-c0re` spawns it directly at boot if missing, with no operator
|
||||
approval step — every other agent goes through `request_init_config`
|
||||
→ approval. Topology-wise, `ruth` is still just another root agent.
|
||||
- **Wire-protocol** — the privileged `Request` variants
|
||||
(`RequestInitConfig`; `Kill` / `Start` / `Restart` / `Update`;
|
||||
`GetLogs`; `RequestUpdateMetaInputs`) — marked `*(privileged)*` in
|
||||
`hive-core-agent-sock`'s unified `Request` enum — are reachable only
|
||||
from the manager's socket flavour today. Planned rule for each is in the
|
||||
table above ("any agent, child added under self" for init-config,
|
||||
"any ancestor" for lifecycle/logs); `RequestUpdateMetaInputs` stays
|
||||
a root-only capability even post-milestone, not a topology rule.
|
||||
One exception: `Wake` (inject a `from: <X>` message into the
|
||||
caller's own inbox) isn't really privileged — every per-agent daemon
|
||||
(e.g. `hive-forge-notify`) needs it, and sub-agents already have the
|
||||
equivalent on their own socket.
|
||||
- **Storage/mounts** — only the manager container gets
|
||||
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
|
||||
manage any agent's state dir — config is not authored there, since a
|
||||
real config change is a PR from a clone), plus RO mounts for
|
||||
`/applied` (diff against what's deployed) and `/meta` (system-wide
|
||||
deploy log). Planned: each agent gets RW to `/agents/<descendant>/`
|
||||
for just its own subtree — the manager's full-forest RW becomes the
|
||||
"root's subtree is everything" case of that same rule. RO `/meta`
|
||||
access will be gated on a "meta read" capability; only
|
||||
`request_update_meta_inputs` writes `flake.lock`, gated by its own
|
||||
capability.
|
||||
- **Prompt/tools** — the system prompt uses `<!-- role:agent -->` /
|
||||
`<!-- role:manager -->` marker blocks, and a `Flavor::{Agent,
|
||||
Manager}` switch picks the MCP tool allow-list claude sees. Both are
|
||||
already parametrised on a single flavour value, so the planned
|
||||
per-capability-group version (`cap:<group>` prompt blocks + a
|
||||
matching tool allow-list) is additive rather than a rewrite.
|
||||
- **State dirs** — *not* special-cased: `HYPERHIVE_STATE_DIR` is
|
||||
injected uniformly via `systemd.globalEnvironment` for every
|
||||
container including the manager, so all token/state paths resolve
|
||||
through it the same way everywhere.
|
||||
- **Scattered ownership checks** — a handful of independent
|
||||
manager-only overrides exist across `hive-c0re` today: loose-ends
|
||||
visibility (manager sees hive-wide, sub-agents only their own),
|
||||
`destroy` refusing to act on the manager, and crash-watch skipping
|
||||
the manager (it auto-restarts via systemd instead of going through
|
||||
the crash-watch loop). Each is planned to become an
|
||||
ancestor/descendant check instead of a manager-name check — see the
|
||||
module docs for `loose_ends.rs`, `stores/broker.rs`, `actions.rs`,
|
||||
and `workers/crash_watch.rs` for the current owner-check logic in
|
||||
each. (Question/answer routing and its own manager-override cancel
|
||||
path — formerly `hive-c0re/src/questions.rs` and
|
||||
`stores/operator_questions.rs` — has been removed entirely; reminder
|
||||
cancellation is now handled fully in-agent, see the note on
|
||||
`CancelLooseEndKind::Reminder` in `hive-c0re/src/socket_server/mod.rs`.)
|
||||
|
||||
None of the above is a stable interface — treat the module doc
|
||||
comments as the source of truth for exactly which checks exist today.
|
||||
|
||||
## Future work: sub-agents inside the same container
|
||||
|
||||
When enabled for an agent, it will be able to spawn temporary
|
||||
"sub-agents" that run inside its own container — lighter than a full
|
||||
nspawn agent. Open questions, not yet wired:
|
||||
|
||||
- Inherit caps from parent, or take an explicit narrower set?
|
||||
- Survive container restart, or always ephemeral?
|
||||
- Inbox: separate from parent, or shared?
|
||||
- Filesystem: share parent's `/state` RW, or a sub-dir?
|
||||
- Identity: distinct broker recipient name, or address the parent?
|
||||
|
||||
## Harness systemd unit shape
|
||||
|
||||
One harness serve binary (`hive-agent`, with its `hive-agent-mcp`
|
||||
sibling), one shared `nix/agent-modules/` tree, one service unit
|
||||
(`systemd.services.hive-agent`) for all agents. There is no separate
|
||||
manager service name or role distinction in the harness — privilege
|
||||
differences live server-side in the broker socket (which tool groups
|
||||
and manager-surface calls each agent receives).
|
||||
|
||||
`agent.nix` and `ruth.nix` both import the shared `nix/agent-modules/`.
|
||||
`ruth.nix` additionally sets forge defaults to suppress the
|
||||
subscription/participation firehose so ruth's inbox stays focused on
|
||||
direct mentions, reviews, and assignments.
|
||||
|
||||
### Environment variables set on the unit
|
||||
|
||||
- `HOME = /home/<userName>` — systemd defaults `HOME` to `/` for
|
||||
services without `User=` set; with the per-agent user the harness
|
||||
needs the right home so claude finds its bind-mounted `~/.claude/`
|
||||
session dir.
|
||||
- `HIVE_STATIC_DIR = <mergedDist>` — `tower_http::ServeDir` root for
|
||||
the per-agent web UI; merged dist = agent default + every
|
||||
`hyperhive.frontend.extraFiles` overlay.
|
||||
- `HIVE_ASSETS_DIR = pkgs.hyperhive-assets/share/hyperhive` — set
|
||||
directly on the unit, **not** via `environment.variables`, because
|
||||
the latter only populates `/etc/profile` which systemd services
|
||||
don't inherit.
|
||||
|
||||
### `PATH` setup (the wrapper-dir trick)
|
||||
|
||||
```nix
|
||||
path = [ "/run/wrappers" "/run/current-system/sw" ];
|
||||
```
|
||||
|
||||
`/run/wrappers` (not `/run/wrappers/bin`) comes first so setuid
|
||||
wrappers — notably `sudo` — resolve before bare nix-store binaries; see
|
||||
[`docs/process/gotchas.md`](../process/gotchas.md) ("`systemd.services.*.path` appends
|
||||
`/bin` to every entry") for why the trailing `/bin` matters in
|
||||
general. It's load-bearing here because the harness runs as the
|
||||
per-agent user: without the wrapper dir on `PATH`, `sudo` resolves to
|
||||
the non-setuid nix-store binary and every
|
||||
`hyperhive.user.passwordlessSudo` grant fails with "must be owned by
|
||||
uid 0 and have the setuid bit set."
|
||||
|
||||
### `serviceConfig` highlights
|
||||
|
||||
- `ExecStart = pkgs.hyperhive/bin/hive-agent` — same binary for every
|
||||
agent.
|
||||
- `Restart = on-failure`, `RestartSec = 2` — keeps the harness
|
||||
resilient across transient crashes without thundering retries.
|
||||
- `RuntimeDirectory = "hive-config"` → `/run/hive-config/` owned by
|
||||
`User=`, auto-cleared on stop. The harness writes regenerated
|
||||
`claude-{mcp-config,settings,system-prompt}` files there
|
||||
(`paths::config_dir`). Deliberately separate from `/run/hive`, which
|
||||
the host bind-mounts in root-owned and which holds hive-c0re's
|
||||
`mcp.sock`.
|
||||
- `User = Group = userName` — drops root inside the container; sudo is
|
||||
the explicit escalation surface (`hyperhive.user.passwordlessSudo`).
|
||||
|
||||
## Cross-references
|
||||
|
||||
- Milestone: "Agent privileges and sub-agents"
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361`)
|
||||
- Dashboard render: "show agent topology in container list"
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/363`)
|
||||
- Audit table source: milestone comment
|
||||
(`$HIVE_FORGE_URL/hyperhive/hyperhive/issues/361#issuecomment-3335`)
|
||||
- Operator/agent trust boundary (orthogonal axis): [`boundary.md`](../trust-boundary/boundary.md)
|
||||
679
docs/agent-lifecycle/approvals.md
Normal file
679
docs/agent-lifecycle/approvals.md
Normal file
|
|
@ -0,0 +1,679 @@
|
|||
# Approvals + helper events
|
||||
|
||||
The approval queue is hyperhive's pivot: nothing that changes the
|
||||
shape of an agent (its config, whether it exists) happens without an
|
||||
operator click. The submitting agent — any agent with the `approvals`
|
||||
tool group, which manages the config of its **direct children** (the
|
||||
root agent for top-level agents; a sub-manager for its own subtree) — is
|
||||
the policy gate in front of that queue; helper events are how it stays
|
||||
informed about what happens after a decision lands.
|
||||
|
||||
## For operators
|
||||
|
||||
Every add/remove/change to an agent lands on your dashboard's Y3R
|
||||
C4LL tab (or `hivectl approvals pending` / `approve <id>` from the
|
||||
CLI) before it takes effect. What you'll see, and what to do with it:
|
||||
|
||||
- **Config change** (`MergeConfigPr`) — an agent proposed a change to
|
||||
another agent's config (or its own, via a sub-manager) as a forge
|
||||
pull request. Review the diff on the forge — the dashboard card
|
||||
links straight to it, same as reviewing any other PR. Approving
|
||||
triggers the deploy automatically: hive-c0re re-verifies the PR
|
||||
hasn't moved since you looked at it, evaluates it (a dry run,
|
||||
nothing applied yet), merges it, and rebuilds the container. If
|
||||
anything in that chain fails, the change rolls back automatically —
|
||||
the agent stays on its last-good config, no recovery action needed
|
||||
from you.
|
||||
- **New agent** (`InitConfig` then `Spawn`) — creating a brand-new
|
||||
agent is two approvals. `InitConfig` creates the config repo and
|
||||
seeds it from a template; `Spawn` creates the container from that
|
||||
config. Tailoring the template first is not a separate mechanism —
|
||||
it's the config-change flow above, a PR you review like any other. Every later change goes through the config-change flow
|
||||
above — there's no repeat "spawn" for an existing agent.
|
||||
- **Meta/flake update** (`UpdateMetaInputs`) — an agent asked to bump
|
||||
one or more Nix flake inputs (or all of them). Approving runs the
|
||||
update and commits the lock change; it doesn't rebuild anything by
|
||||
itself.
|
||||
- **Scheduled prompt** (`SchedulePrompt`) — an agent asked to schedule
|
||||
a message to one or more inboxes at a future time. You can also add
|
||||
schedules yourself directly from the SCH3DUL3S tab, which skips this
|
||||
approval step entirely — the gate here is specifically for an
|
||||
*agent* asking to schedule something, not for you doing it.
|
||||
|
||||
Don't want to approve something? **Deny it** (`DENY` on the dashboard
|
||||
card, or `hivectl approvals deny <id>`) — nothing runs. Either way the
|
||||
submitting agent is always notified their request was denied; what's
|
||||
optional is only the reason text, which you can add on the dashboard's
|
||||
prompt (cancelling that prompt aborts the whole deny, not just the
|
||||
reason) but not from the CLI. Denying is final: a denied approval
|
||||
can't be re-approved later, the agent has to submit a fresh one (a new
|
||||
PR, a new request).
|
||||
|
||||
Everything below this point is the implementation detail behind that
|
||||
flow.
|
||||
|
||||
## End-to-end approval flow
|
||||
|
||||
Config changes flow through a **forge pull request** on the agent's
|
||||
`agent-configs/<name>` repo — the same surface agents use for code PRs.
|
||||
There is no bespoke MCP tool for config changes: opening the PR IS the
|
||||
request.
|
||||
|
||||
1. The submitting agent (the child's parent, holding the `approvals`
|
||||
tool group) **clones** `agent-configs/<name>`, edits it there (any
|
||||
tracked path, but `agent.nix` is the contract entry point), commits
|
||||
with its own git identity, and pushes a branch + opens a PR with
|
||||
`hive-forge` — the same way it would change any other repo.
|
||||
The bind-mounted `/agents/<name>/config/` is a **copy for reading** a
|
||||
config, not the tree to edit: authoring in place there produces no PR
|
||||
and no approval. (It is currently mounted read-write, which is a
|
||||
defect tracked separately, not an authoring path.)
|
||||
Branch protection (push/merge whitelist = `core`, approvals whitelist
|
||||
= operator team; see "Forge mirror" and #1787) makes the agent a
|
||||
write collaborator that **cannot merge its own config PR**.
|
||||
2. hive-c0re's `/webhook/config-pr` endpoint receives the Forgejo
|
||||
`pull_request` event (opened / synchronized / reopened) and queues a
|
||||
`MergeConfigPr` approval; a poll fallback catches any missed webhook.
|
||||
The approval row stores the PR **number** (`commit_ref`) and the PR
|
||||
**head sha at queue time** (`fetched_sha` — the "reviewed" sha). If
|
||||
the PR head later moves, the stale approval is superseded by a fresh
|
||||
one pinned to the new head, so the operator always reviews what will
|
||||
actually deploy.
|
||||
3. The operator reviews the PR **on the forge** (native diff, threaded
|
||||
comments, CI status) and sees a matching card on the dashboard with a
|
||||
"review PR on forge" deep link. They click ◆ APPR0VE (or
|
||||
`hivectl approvals approve <id>` on the CLI) once satisfied.
|
||||
4. On approve, a deploy DAG runs three phases under a resource-holding
|
||||
`DeployWindow` root (see *Queue templates* below):
|
||||
- `MergeVerify` re-reads the live PR head and **aborts if it drifted**
|
||||
from the reviewed `fetched_sha` (the submitter must push again,
|
||||
which queues a fresh approval); then fetches that head into the
|
||||
applied repo and **eval-verifies** it — a flake eval on a throwaway
|
||||
checkout. This is the trust gate: it relies on c0re's own eval, not
|
||||
on any in-repo (agent-forgeable) signal like a CI status. Nothing is
|
||||
mutated in this phase, so a rejection here leaves the forge and the
|
||||
applied repo exactly as they were.
|
||||
- `DeployApply` parks the pre-merge `applied/main` in
|
||||
`refs/hyperhive/rollback/<approval-id>`, then fast-forward-merges
|
||||
the reviewed head to the forge config repo's `main` (this IS the
|
||||
merge — a `core`-authenticated ff-merge pinned to the reviewed sha,
|
||||
so a moved PR head can't substitute bytes), and runs the deploy
|
||||
proper (`deploy_applied_target`): ff `applied/main`, two-phase meta
|
||||
deploy, container rebuild. On success it drops the rollback ref and
|
||||
plants `deployed/<id>`.
|
||||
- `DeployTail` runs on **every** outcome, including a cancel-cascade.
|
||||
If the rollback ref survived, the deploy never confirmed good: it
|
||||
rolls `applied/main` back, resyncs the working tree, and aborts the
|
||||
staged meta lock, so the agent stays on its last-good tree. Then it
|
||||
mirrors the config repo (and its new deploy tag) to the forge.
|
||||
|
||||
The rollback state lives in a **git ref, not a local variable**, on
|
||||
purpose: hive-c0re can restart between the apply and the tail, and the
|
||||
tail still has to know what to undo when it does.
|
||||
5. `HelperEvent::ApprovalResolved` (and `Rebuilt`) land in the
|
||||
**submitting agent's** inbox via `notify_submitter`, carrying both the
|
||||
canonical sha and the terminal tag (the approval row carries a
|
||||
`submitter` column recording the agent the change is for).
|
||||
|
||||
### Withdrawing a pending approval
|
||||
|
||||
The submitting agent can call `cancel_loose_end(kind: "approval", id)` to
|
||||
withdraw an approval that hasn't been acted on yet.
|
||||
The row transitions to `ApprovalStatus::Cancelled` (distinct from
|
||||
`Denied`/`Failed`), the dashboard pulls the card out of the
|
||||
pending pane, and `ApprovalResolved { status: "cancelled" }` fires
|
||||
on the root agent + dashboard channels. Approvals that have already
|
||||
been approved/denied/failed return an error — the resolution is
|
||||
final once the operator (or a lifecycle failure) acted on the row.
|
||||
|
||||
The socket refuses the `approval` kind with a clear error for any
|
||||
agent that lacks the `approvals` tool group: only an agent with that
|
||||
group submits approvals (for its direct children), so an agent
|
||||
without it has nothing of its own to withdraw.
|
||||
|
||||
`InitConfig` approvals create a brand-new agent's config repo. On
|
||||
approve, hive-c0re seeds it with a default `agent.nix` template and
|
||||
pushes a todo (`push_todo_submitter`) into the submitting agent's
|
||||
in-container store. The operator then **spawns** the agent (the
|
||||
`Spawn` approval / `◆ R3QU3ST SP4WN` button), which creates the
|
||||
container from that config.
|
||||
|
||||
Changing what the template seeded is not a special case: like every
|
||||
later change, it's a PR on that config repo (`MergeConfigPr`), made
|
||||
from a clone, reviewed and approved by the operator. The PR flow is
|
||||
the one path — an operator can equally drive both steps herself
|
||||
through the web UI or the forge.
|
||||
|
||||
### Approval kinds (wire shapes)
|
||||
|
||||
`ApprovalKind` carries five variants; each maps to a different
|
||||
`commit_ref` encoding because that field is overloaded as the
|
||||
kind-specific payload carrier.
|
||||
|
||||
- `MergeConfigPr` — the config-change flow. Triggered automatically:
|
||||
when an agent opens (or force-pushes) a PR on its
|
||||
`agent-configs/<agent>` forge repo, hive-c0re's `/webhook/config-pr`
|
||||
endpoint receives the Forgejo pull_request event and queues this
|
||||
approval row. No MCP tool call needed — the forge PR IS the request.
|
||||
`commit_ref` stores the **PR number** (decimal), and `fetched_sha` is
|
||||
the PR **head sha at queue time** (the "reviewed" sha). On approve,
|
||||
the deploy DAG's `MergeVerify` phase re-reads the live PR head and
|
||||
aborts if it drifted from `fetched_sha` (submitter must push again to
|
||||
re-trigger), then fetches that head into the applied repo and
|
||||
eval-verifies it; `DeployApply` fast-forward-merges the forge config
|
||||
repo's `main` to it (the merge) and runs `deploy_applied_target`;
|
||||
`DeployTail` compensates on failure. Never a first spawn.
|
||||
- `Spawn` — direct container creation from the agent's config repo.
|
||||
`commit_ref` is empty. Submitted via `HostRequest::RequestSpawn`
|
||||
(operator-gated, the `◆ R3QU3ST SP4WN` dashboard button +
|
||||
`hivectl agent <name> request-create` CLI). The host-level `HostRequest::Spawn`
|
||||
variant bypasses the approval queue entirely — privileged-context use
|
||||
only (operator on the host shell, test scripts, one-off recoveries;
|
||||
`hivectl agent <name> create`). This is the **canonical first-spawn**: a new agent's `InitConfig`
|
||||
seeds its config repo, the submitting agent customises it, then the
|
||||
operator spawns to create the container. Subsequent config changes go
|
||||
through a `MergeConfigPr` PR.
|
||||
- `InitConfig` — `commit_ref` is empty; the variant just gates
|
||||
"seed the proposed repo with the default template" against
|
||||
operator approval. Step 1 of the two-step spawn flow above.
|
||||
- `UpdateMetaInputs` — `commit_ref` stores the JSON-encoded inputs
|
||||
array (`"[]"` = all inputs, `"[\"nixpkgs\"]"` = just nixpkgs,
|
||||
etc.). `agent` field is set to the requesting root agent.
|
||||
On approve hive-c0re runs `nix flake update [inputs...]` on the
|
||||
meta flake and commits the resulting lock changes.
|
||||
- `SchedulePrompt` — `commit_ref` stores the JSON-encoded
|
||||
`SchedulePromptPayload` (target list, body, schedule) so the
|
||||
approval row carries the full submission verbatim. On approve
|
||||
hive-c0re inserts a row into `scheduled_prompts` with
|
||||
`source = approval:<id>`; the worker fans the body out as
|
||||
inbox messages to each target at the scheduled time, recurring
|
||||
when `interval_seconds` is set.
|
||||
|
||||
### Scheduled prompts (submit paths)
|
||||
|
||||
Two ways a row lands in `scheduled_prompts`:
|
||||
|
||||
- **Operator-direct** (`source = "operator"`): the operator adds a schedule through the dashboard form. Lands in the table immediately, no approval gate — operator action is already the trust boundary.
|
||||
- **Agent-requested** (`source = "approval:<id>"`): an agent submits a `RequestSchedulePrompt` through its MCP socket (the `request_schedule_prompt` tool, `scheduling` group). An `ApprovalKind::SchedulePrompt` row is queued; on approve, hive-c0re inserts the schedule row with `source = approval:<id>` so the audit trail points back at the operator decision (above).
|
||||
|
||||
No self-target shortcut: even agent-self schedules need approval. The existing `remind` MCP tool stays the quick self-wake path (no approval, lands directly in the agent's own inbox); this module is the bigger, multi-recipient, operator-visible thing.
|
||||
|
||||
### Scheduled prompt worker (catch-up clamp)
|
||||
|
||||
When hive-c0re comes back from being down, the worker sees rows whose `next_fire_at_unix` is well in the past. For recurring rows that would mean firing N delayed pulses in a row — spammy and useless. Instead the worker fires **once** per row and bumps `next_fire_at_unix` to the next interval slot ≥ `now`, recording how many cycles were skipped in `last_result` (per-target). Operators see "fired late, caught up from 17 skipped" instead of 17 wake-up storms.
|
||||
|
||||
One-shot rows fire once (if past due, on the next worker pass) and are deleted by the worker; recurring rows survive until cancelled.
|
||||
|
||||
`targets` is its own table (`scheduled_prompt_targets`) so partial cancellation flips a single row and the dashboard can show last-fired / last-result per recipient. Cancelling every target reaps the parent row on the next worker pass.
|
||||
|
||||
### Scheduled prompt delivery: todo, not a broker message
|
||||
|
||||
An agent target's delivery is `push_todo` (`Coordinator::push_todo`,
|
||||
`docs/coordinator.md` covers the mechanism generally), not a broker
|
||||
`Message` — a scheduled prompt wakes its target with a todo instead of
|
||||
driving an immediate turn, by design. `key = "schedule:<id>"` per
|
||||
target drives `push_todo`'s own upsert-by-key dedup: a re-fire of the
|
||||
*same schedule* against a target that hasn't reviewed the last one
|
||||
collapses into that one todo instead of stacking up.
|
||||
|
||||
**`operator` is the one exception** — it's a valid schedule target but
|
||||
has no in-container todo inbox, so it keeps the original broker
|
||||
`Message` path (the dashboard mirrors `to == operator` into its own
|
||||
pane, same as before).
|
||||
|
||||
### Missing-target failure
|
||||
|
||||
When a target name doesn't resolve to a known agent (container
|
||||
destroyed, operator typo, etc.) the worker:
|
||||
|
||||
1. Records `last_result = "no such agent: <name>"` on the
|
||||
per-target row.
|
||||
2. Sends a single advisory `Message` from `system` to `operator`
|
||||
naming the schedule, target, and reason. This one stays a `Message`
|
||||
regardless of target type — it's a to-operator advisory about a
|
||||
broken schedule, not the schedule's own delivery.
|
||||
3. Continues fanning out to the other live targets.
|
||||
|
||||
Transient broker errors (sqlite lock contention, etc.) get the same
|
||||
`last_result` annotation plus a `tracing::warn`, and then:
|
||||
|
||||
- **Recurring rows** re-arm to the next interval slot — the retry
|
||||
self-heals on the next worker pass.
|
||||
- **One-shot rows** are deleted unconditionally after their single
|
||||
fan-out pass; a broker error on a one-shot is not retried (the
|
||||
operator advisory and `last_result` are the only audit trail).
|
||||
|
||||
### Reminder delivery: file-path semantics
|
||||
|
||||
A reminder may carry a `file_path` (the agent-visible path inside its
|
||||
container, e.g. `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
||||
|
||||
1. **Translates** the container path to the host path
|
||||
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
|
||||
from outside the container.
|
||||
2. **Validates** the path: rejects anything outside the agent's own state
|
||||
subtree, containing `..` (path traversal), or with an empty relative
|
||||
tail. On rejection the write is skipped and the original message is
|
||||
delivered inline with a warning — the reminder still fires.
|
||||
3. **Defends against symlink escape**: after `create_dir_all`, the parent
|
||||
dir is canonicalized and re-verified to live under the agent's host
|
||||
state root. The final file is opened with
|
||||
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
|
||||
basename cannot redirect the write to an arbitrary host path.
|
||||
4. **Writes the body to disk** and delivers a short pointer message in its
|
||||
place, keeping the agent's inbox / wake-prompt small while the bulky
|
||||
payload is read out of band.
|
||||
|
||||
Atomicity of the inbox INSERT + `reminders.sent_at` UPDATE is handled
|
||||
inside `Broker::deliver_reminders_batch`; the scheduler only computes the
|
||||
body strings before calling it.
|
||||
|
||||
### Destroy semantics
|
||||
|
||||
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down,
|
||||
not an approval. Stops + removes the nspawn container, drops the
|
||||
systemd drop-in, fails any pending approvals. Persistent state
|
||||
(proposed/applied repos, claude credentials, `/state/` notes) is
|
||||
**kept by default** — recreating the agent with the same name
|
||||
reuses prior config + login. With `purge = true` the agent's
|
||||
`/var/lib/hyperhive/{agents,applied}/<name>/` trees are also
|
||||
wiped (config history + creds + notes gone forever). The
|
||||
root/bootstrap container is destroyable like any other — hive-c0re
|
||||
recreates it on the next startup if it's absent, so destroying it is
|
||||
transient.
|
||||
|
||||
## Meta flake
|
||||
|
||||
The hive-c0re-owned repo at `/var/lib/hyperhive/meta/`
|
||||
declares one flake input per agent (`agent-<n>.url =
|
||||
"git+http://<forge>/agent-configs/<n>.git"`) and one
|
||||
`nixosConfigurations.<n>` output per agent. Each output wraps
|
||||
`inputs.agent-<n>.nixosModules.default` with the identity +
|
||||
`HIVE_PORT` / `HIVE_LABEL` / `HIVE_DASHBOARD_PORT` injection module.
|
||||
Containers run against `--flake /var/lib/hyperhive/meta#<n>`.
|
||||
|
||||
The declared input url is the agent's **forge config repo** (the
|
||||
same `agent-configs/<n>` the config-PR flow lands approved changes
|
||||
on), so the meta flake references a reviewable, reproducible source
|
||||
rather than a local checkout. hive-c0re authenticates that
|
||||
`git+http` fetch via a git credential helper that reads the live
|
||||
forge-core token — no token in the url or the lock. The deploy and
|
||||
manual-rebuild paths, however, do **not** re-lock from the forge:
|
||||
they `--override-input agent-<n>
|
||||
git+file:///var/lib/hyperhive/applied/<n>`, locking the exact config
|
||||
that `verify_commit` gated and `applied/<n>/main` was
|
||||
fast-forwarded to. That keeps a deploy/rebuild reproducible and
|
||||
independent of forge reachability — rebuilds fire on crash-restart
|
||||
and meta bumps, not just config PRs — while the declared url stays
|
||||
the forge. `sync_agents` re-renders + re-locks the persistent input;
|
||||
a plain `nix flake lock` leaves an existing applied override in
|
||||
place (it only re-locks when the declared url itself changes), so
|
||||
the forge-declared / applied-deployed split is stable.
|
||||
|
||||
Per-deploy lock flow (two-phase), spread across the deploy subtree's
|
||||
nodes — each phase is its own node, so the queue can show which one is
|
||||
running and a restart resumes at node granularity:
|
||||
|
||||
1. `DeployApply` → `meta::prepare_deploy(name)` runs
|
||||
`nix flake lock --update-input agent-<n>` without
|
||||
committing. Working tree of meta now points the input at
|
||||
`applied/<n>/main` (which the deploy already fast-forwarded to
|
||||
the reviewed PR head).
|
||||
2. The rebuild subgraph `DeployApply` grows into the DAG builds and
|
||||
swaps the container (`AgentWindow` bracing `Prebuild → StopForUpdate
|
||||
→ Swap → RebuildBookkeeping`, plus `Reconcile`). Nix evaluates
|
||||
against the staged lock.
|
||||
3. On success — `FinalizeDeploy` drops the rollback ref, plants
|
||||
`deployed/<id>`, then `meta::finalize_deploy(name, sha, "deployed/
|
||||
<id>")` stages `flake.lock` and commits with
|
||||
`deploy <n> deployed/<id> <sha12>`. Meta's git log gains
|
||||
one entry per successful deploy.
|
||||
4. On failure — the `DeployTail` node runs `meta::abort_deploy()`
|
||||
(`git restore flake.lock`) so the meta history shows only
|
||||
successes; the failure stays as an annotated `failed/<id>`
|
||||
tag in `applied/<n>`. The tail runs on every outcome, so this
|
||||
also covers a hive-c0re restart mid-build: the staged lock is
|
||||
dropped and `applied/main` rolled back from the parked
|
||||
`refs/hyperhive/rollback/<id>`.
|
||||
|
||||
Single-phase variants exist for paths without
|
||||
rollback semantics: `meta::lock_update_for_rebuild(name)` for
|
||||
the manual `↻ R3BU1LD` button (commits if the lock changed)
|
||||
and `meta::lock_update_hyperhive()` for the
|
||||
auto-update flake-rev bump (one shot before per-agent
|
||||
rebuilds, commits if the lock changed).
|
||||
|
||||
`meta::sync_agents(hive: &HiveEnv, agents: &[AgentSpec])` — `hive`
|
||||
carries `hyperhive_flake`, `dashboard_port`, and the rest of the
|
||||
per-hive config — is the idempotent reconciler called by `spawn`,
|
||||
`destroy`, `rebuild`, and the startup migration. Renders `flake.nix`
|
||||
from the agent list; if it differs from disk, runs
|
||||
`nix flake lock` + commits as `regenerate meta flake` (or
|
||||
`seed meta from N agent(s)` on the very first call).
|
||||
|
||||
The root agent has `/meta` RO-bound inside its container:
|
||||
`git -C /meta log --oneline` is the swarm-wide deploy log,
|
||||
`cat /meta/flake.lock | jq '.nodes["agent-<n>"].locked'`
|
||||
resolves which sha each agent is pinned at right now.
|
||||
Dashboard surfaces the same info as a `deployed:<sha12>` chip
|
||||
per container row.
|
||||
|
||||
## Two repos per agent
|
||||
|
||||
```
|
||||
/var/lib/hyperhive/agents/<name>/config/ proposed — parent mount is RO
|
||||
└── <anything> # any files the submitting
|
||||
# agent wants in the commit.
|
||||
# agent.nix is the
|
||||
# convention entry
|
||||
# point; flake.nix is
|
||||
# tracked boilerplate
|
||||
# (submitting agent doesn't
|
||||
# edit it).
|
||||
|
||||
/var/lib/hyperhive/applied/<name>/ applied — core-only
|
||||
├── .git/ # tag-rich history
|
||||
├── flake.nix # tracked, fixed
|
||||
│ # boilerplate exporting
|
||||
│ # nixosModules.default
|
||||
├── agent.nix # working tree of main
|
||||
└── <other committed files> # also tracked
|
||||
|
||||
/var/lib/hyperhive/meta/ swarm-wide flake — core
|
||||
├── .git/ # one commit per successful
|
||||
│ # deploy
|
||||
├── flake.nix # generated from agent set
|
||||
└── flake.lock # pins each agent's sha
|
||||
```
|
||||
|
||||
Why two physical repos: the submitting agent's `/agents/<n>/config/` is
|
||||
RW — a buggy or hostile agent can `git clean -fdx` its own
|
||||
proposed tree. The applied repo is never bind-mounted (except
|
||||
the read-only `.git` exposure described below) so a destructive
|
||||
move inside the container cannot reach it.
|
||||
|
||||
The container's `--flake` ref is `/var/lib/hyperhive/meta#<name>`
|
||||
(see "Meta flake" above). The agent's own `applied/<n>/flake.nix`
|
||||
is a fixed boilerplate that exports `nixosModules.default =
|
||||
import ./agent.nix`; the meta flake imports that module and
|
||||
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL` /
|
||||
`HIVE_DASHBOARD_PORT`.
|
||||
|
||||
### Tag state machine
|
||||
|
||||
Each deploy leaves a tag on the underlying commit inside the applied
|
||||
repo:
|
||||
|
||||
| Tag | When | Annotated? |
|
||||
|---|---|---|
|
||||
| `deployed/<id>` | rebuild succeeded — `main` ff's here | no |
|
||||
| `failed/<id>` | rebuild failed | yes (body = error) |
|
||||
|
||||
`deployed/0` is planted at first spawn. `applied/main` is always the
|
||||
latest `deployed/*`. A `failed/` tree stays browsable forever — `git log
|
||||
--tags` in the applied repo is the audit trail. A denied or failed config
|
||||
PR carries no extra state on the forge side: the PR stays open, and the
|
||||
submitter pushes again (or closes it) to retry.
|
||||
|
||||
### Dispatch via the job queue
|
||||
|
||||
Long-running approval work — `MergeConfigPr`, `UpdateMetaInputs`,
|
||||
`Spawn` — runs as a DAG on the global job queue
|
||||
(`docs/coordinator.md::Job queue`), submitted by the approval handler
|
||||
rather than run inline:
|
||||
|
||||
| `ApprovalKind` | DAG submitted | source |
|
||||
|---|---|---|
|
||||
| `MergeConfigPr` | `rebuild` (`DeployWindow` root + `MergeVerify → DeployApply` + `DeployTail`) | `approval` |
|
||||
| `UpdateMetaInputs` | `meta_update` (`MetaLock` + rebuild fan-out) | `approval` |
|
||||
| `Spawn` | `spawn` (`Create → WriteDropin → Reconcile`) | `approval` |
|
||||
| `InitConfig` | — runs inline (sub-second git seed) | — |
|
||||
| `SchedulePrompt` | — runs inline (single sqlite insert) | — |
|
||||
|
||||
The DAG carries the originating `approval_id`, surfaced on the node that
|
||||
owns it — for a deploy that's the `DeployWindow` root, so the dashboard
|
||||
renders one approval card, not four. **Every** queued kind resolves
|
||||
through `actions::resolve_approval_dag` when its DAG settles terminal:
|
||||
the deploy's phases are ordinary queue nodes, so the DAG's own terminal
|
||||
state is the authoritative outcome. That hook fires the matching
|
||||
`HelperEvent::*` via `finish_approval`, derives the `Rebuilt` event's
|
||||
terminal tag (verifying the tag actually resolves in the applied repo —
|
||||
a pre-merge rejection plants none), posts the failing build log back to
|
||||
the config PR, and for a spawn runs the post-spawn forge bookkeeping.
|
||||
|
||||
Two visible consequences:
|
||||
|
||||
- **Operator dashboard**: after clicking APPR0VE the work-in-progress
|
||||
shows up on the *rebuild queue* card (`GET /api/jobq/graph`, refetched
|
||||
on every `rebuild_queue_changed` tick), not on the approvals panel
|
||||
(which already moved the row to "approved"). A long meta-update
|
||||
cascade renders as a parent DAG with one child rebuild per affected
|
||||
agent — see `docs/web-ui.md` for the layout.
|
||||
- **Cancellation**: the dashboard's *× cancel* button on a still-queued
|
||||
DAG calls `POST /api/rebuild-queue/{id}/cancel`, which flips it to
|
||||
`Cancelled` before any node runs (and fails the approval row instead
|
||||
of leaving it dangling). Returns `{"cancelled": true}` on success,
|
||||
`{"cancelled": false}` once any node started — terminal states can't
|
||||
be retroactively rewritten.
|
||||
|
||||
The `approval` source + `approval_id` mean a tail-end build failure
|
||||
surfaces back as a failed approval row, not just a silent queue
|
||||
entry. `manual` (dashboard ↻ R3BU1LD) and `auto_update` (boot
|
||||
reconcile) DAGs use the same queue but skip the approval plumbing.
|
||||
|
||||
### Forge mirror
|
||||
|
||||
The bundled `hive-forge` container is mandatory (it deploys with
|
||||
hyperhive), and hive-c0re mirrors every agent's applied repo into a
|
||||
private `agent-configs` Forgejo org. `forge::push_config(<name>)` pushes `applied/main` plus
|
||||
every tag to `agent-configs/<name>` after each ref mutation:
|
||||
the spawn that seeds `deployed/0`, every successful deploy (which
|
||||
plants `deployed/<id>`) or failed build (`failed/<id>`), and a
|
||||
sweep at startup. Pushes are best-effort — a missing or stopped
|
||||
forge never blocks a deploy.
|
||||
|
||||
Each agent is a **write collaborator on its own** `agent-configs/<name>`
|
||||
repo — so it can push a branch and open a config PR — but not a member
|
||||
of any other agent's, so it can't reach another agent's config through
|
||||
the forge. Branch protection keeps `main` push/merge `core`-only with
|
||||
operator-team approval, so an agent can't fast-forward its own config or
|
||||
self-merge its PR (see the End-to-end flow + #1787). The tokenised push
|
||||
URL is passed inline to `git push`, never written into
|
||||
`applied/<n>/.git/config`; that repo is RO-bind-mounted into the root
|
||||
agent, and a stored token would leak core's admin credential to an
|
||||
agent.
|
||||
|
||||
The dashboard deep-links into this org — a `config repo` link
|
||||
per container row and a `review PR on forge` link per config-PR
|
||||
approval card. See `docs/web-ui.md`.
|
||||
|
||||
### Submitting agent's view of config repos
|
||||
|
||||
Every parent agent's container has its **direct children's** config
|
||||
repos bind-mounted **read-only** (topology-driven:
|
||||
`hive-c0re/src/lifecycle/host_config.rs` calls `bind_child_agent_dirs`
|
||||
for each entry in
|
||||
`topology::children_of(agent_name)`). It is a copy to *read* a child's
|
||||
current config — not an editing surface.
|
||||
|
||||
An agent with the `approvals` tool group submits a change the same way
|
||||
any other change is made: **clone the child's config repo from the
|
||||
forge into its own state dir, commit on a branch, open a PR**, and let
|
||||
the operator review and approve it. There is deliberately no second,
|
||||
mount-shaped path that reaches the same file without the review.
|
||||
|
||||
Agents holding the `can_manage_top_level_agents` topology role (see
|
||||
`hive-c0re/src/agent_config/topology.rs`) get additional host-side
|
||||
bind mounts via `set_nspawn_flags`:
|
||||
|
||||
- `/var/lib/hyperhive/agents/` → `/agents/` (RW) — all top-level
|
||||
agents' proposed repos (not just direct children).
|
||||
- `/var/lib/hyperhive/applied/` → `/applied/` (RO) — every agent's
|
||||
authoritative applied repo, including `.git`.
|
||||
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the swarm-wide
|
||||
deploy flake.
|
||||
|
||||
The root agent holds this role; a sub-manager that only manages a
|
||||
subtree does not, and only has its direct children's config dirs.
|
||||
|
||||
Each proposed repo (`/agents/<n>/config/`) is pre-configured
|
||||
with `applied` as a git remote pointing at
|
||||
`/applied/<n>/.git`. Useful incantations from inside an agent with
|
||||
the full `/applied` mount:
|
||||
|
||||
```sh
|
||||
git -C /agents/<n>/config fetch applied
|
||||
git -C /agents/<n>/config log applied/main --oneline
|
||||
git -C /agents/<n>/config show applied/refs/tags/deployed/<id>
|
||||
git -C /agents/<n>/config show applied/refs/tags/failed/<id> # body = build error
|
||||
git -C /agents/<n>/config show applied/refs/tags/denied/<id> # body = operator note
|
||||
git -C /agents/<n>/config rebase applied/main # base in-flight work on what's deployed
|
||||
|
||||
git -C /meta log --oneline # swarm-wide deploy history
|
||||
cat /meta/flake.lock | jq '.nodes | with_entries(select(.key | startswith("agent-")))'
|
||||
```
|
||||
|
||||
The RO binds block push at the kernel level — git plumbing inside the
|
||||
container cannot corrupt either authoritative repo.
|
||||
|
||||
## Startup migrations (older hosts)
|
||||
|
||||
hive-c0re runs a couple of idempotent migrations on every startup so a
|
||||
host set up before the tag-driven-deploy + meta-flake scheme (both
|
||||
described above) converges to it automatically. Each phase is a no-op
|
||||
once already applied:
|
||||
|
||||
- **Tags**: agents from before the tag-driven scheme are tagged
|
||||
`deployed/0` on `main` once. Non-destructive — it doesn't touch live
|
||||
containers, state dirs, or claude creds.
|
||||
- **Meta flake**: rewrites each `applied/<n>/flake.nix` to the
|
||||
module-only boilerplate, wires the `applied` remote in each proposed
|
||||
repo, and bootstraps the meta repo from the current agent list. Set
|
||||
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
||||
|
||||
No state loss in either migration: claude creds, `/state/` notes, the
|
||||
events DB, and both proposed + applied history all survive. The root
|
||||
agent keeps its session; sub-agents stay logged in.
|
||||
|
||||
## The root/bootstrap container is hive-c0re-managed
|
||||
|
||||
The root agent container runs through the **same lifecycle as
|
||||
sub-agents**. On `hive-c0re serve` startup, if `ruth` is missing,
|
||||
hive-c0re creates it. The root agent's flake lives at
|
||||
`/var/lib/hyperhive/applied/ruth/`; its proposed config at
|
||||
`/var/lib/hyperhive/agents/ruth/config/`. The root agent can edit its own
|
||||
`agent.nix` (visible inside the container at `/agents/ruth/config/`)
|
||||
and open a config PR on `agent-configs/ruth` for operator approval,
|
||||
same as any other agent.
|
||||
|
||||
Differences from sub-agents:
|
||||
|
||||
- `flake.nix` extends `hyperhive.nixosConfigurations.ruth`
|
||||
(vs `agent-base`).
|
||||
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
||||
FNV-1a hash as every other agent (8100..8999 range).
|
||||
- `set_nspawn_flags` adds two extra binds: `/var/lib/hyperhive/agents`
|
||||
→ `/agents` (RW) so the root agent can edit per-agent proposed repos,
|
||||
and `/var/lib/hyperhive/applied` → `/applied` (RO) so the root agent
|
||||
can `git fetch` deployed/failed/denied tags from any agent's
|
||||
authoritative applied repo (see "Root-agent view of applied" below).
|
||||
- First-deploy spawn bypasses the approval queue (the root agent is
|
||||
required infrastructure).
|
||||
- The root agent's socket is bound by `socket_server::start_manager`,
|
||||
pure transport with no dedicated helpers — it uses the same
|
||||
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
||||
not a special manager-only path.
|
||||
|
||||
**Migration note** (for older hosts): drop any `containers.root =
|
||||
{ ... }` block from your host NixOS config. hyperhive creates and
|
||||
updates the root agent itself.
|
||||
|
||||
## Root-agent policy
|
||||
|
||||
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
||||
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
||||
varies is which MCP tools are surfaced (gated by tool groups and
|
||||
capabilities in `agent.nix`). There is no `role:manager` block that
|
||||
renders only for the root agent. The root agent's approval-gating
|
||||
behaviour comes from its CLAUDE.md / agent-specific instructions, not
|
||||
the system prompt template.
|
||||
|
||||
## Helper events to the submitting agent
|
||||
|
||||
`Coordinator::notify_submitter(approval_id, &HelperEvent)` routes the
|
||||
event to the agent that originally submitted the approval (looked up from
|
||||
the `submitter` column on the `approvals` table). The harness delivers it
|
||||
as a regular `system` inbox message so it drives a normal claude turn.
|
||||
`finish_approval` fires an `ApprovalResolved` HelperEvent this way for
|
||||
**every** approval kind's terminal state, `Spawn` included. A
|
||||
"FYI, check when convenient" event doesn't need a message — those go
|
||||
through `Coordinator::push_todo`/`push_todo_submitter` instead, a direct
|
||||
live dial of the target agent's in-container todo socket (same
|
||||
`UpsertTodo` request in-container producers use); `finish_approval` fires
|
||||
one of these too for `InitConfig`/`Spawn`/`MergeConfigPr`, *in addition to*
|
||||
the `ApprovalResolved` HelperEvent above, not instead of it. Legacy
|
||||
approval rows that predate the submitter column fall back to the
|
||||
root agent. Variants (`hive_sh4re::manager::HelperEvent`):
|
||||
|
||||
- `ApprovalResolved { id, agent, commit_ref, status, note }` —
|
||||
fired by `actions::approve` + `actions::deny` whenever an
|
||||
approval transitions to its terminal state.
|
||||
- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
|
||||
running container went away with no operator-initiated transient
|
||||
state (Stopping / Restarting / Destroying / Rebuilding) AND no
|
||||
such transient was cleared in the last 30s (`RECENT_TRANSIENT_GRACE`
|
||||
tombstone, three `POLL_INTERVAL`s — closes the race where a
|
||||
lifecycle op finishes between two crash-watch polls and the
|
||||
container shows briefly as "stopped without transient" before
|
||||
the next start). The root agent can `start` it again or escalate.
|
||||
- `NeedsUpdate { agent }` — sub-agent's recorded flake rev is
|
||||
stale. The root agent calls `update(name)` to rebuild — idempotent,
|
||||
no approval required.
|
||||
|
||||
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
||||
`Destroyed`, `NeedsLogin`, `LoggedIn`, `ConfigReady` — are "FYI, check
|
||||
when convenient" events with no reason to drive an immediate turn, so
|
||||
they deliver via `push_todo`/`push_todo_submitter` (see above) instead
|
||||
of `HelperEvent`: an `agent_todo_socket` push instead of a broker
|
||||
message, `subsystem = "core"`, `key = "<event>:<agent>"` for dedup,
|
||||
and a single free-text `summary` (`rebuilt_todo_summary` renders
|
||||
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
|
||||
|
||||
Optional `sha` field on `ApprovalResolved` carries the canonical
|
||||
hive-c0re-vouched commit sha. Optional `tag` carries the deploy
|
||||
bookkeeping tag — `deployed/<id>` on a successful build or
|
||||
`failed/<id>` on a failed one, planted by the `MergeConfigPr` deploy.
|
||||
Both fields are `Option`: `None` on the paths that don't deploy a new
|
||||
commit (spawn / init_config / meta-update / deny, and the auto-update
|
||||
sweep's `job_queue::templates::rebuild` reapplying the existing main,
|
||||
or the dashboard `↻ R3BU1LD` button when the lock didn't move). When set,
|
||||
`git show <sha>` against `/applied/<n>/.git` inside the
|
||||
bootstrap container yields the exact tree that was referenced.
|
||||
|
||||
To add a new lifecycle notice: if it needs to drive an immediate turn
|
||||
(something genuinely urgent, like `ContainerCrash`), add a
|
||||
`HelperEvent` variant + call sites + update `prompts/system.md`'s
|
||||
message-event list. If it's "FYI, check when convenient," call
|
||||
`push_todo`/`push_todo_submitter` directly instead — no new wire type
|
||||
needed.
|
||||
|
||||
## Auto-update on startup
|
||||
|
||||
`hive-c0re serve` runs `auto_update::run` in a background task right
|
||||
after opening the coordinator. It enumerates managed containers and
|
||||
rebuilds any whose recorded hyperhive rev differs from the current
|
||||
one — sub-agents and the root agent go through the same
|
||||
`job_queue::templates::rebuild` DAG.
|
||||
|
||||
"Rev" = canonical filesystem path of `cfg.hyperhiveFlake`. Marker
|
||||
file: `/var/lib/hyperhive/applied/.<name>.hyperhive-rev`. If the
|
||||
flake input has no canonical path (e.g. a `github:` URL),
|
||||
auto-update is a no-op — rebuild manually.
|
||||
|
||||
The dashboard surfaces pending updates per agent: a clickable
|
||||
"needs update ↻" badge appears whenever the marker differs from
|
||||
current rev. The badge POSTs `/api/rebuild/<name>`, which inserts the
|
||||
same `job_queue::templates::rebuild` DAG so manual triggers and the
|
||||
startup scan can't drift. When at least one container is stale, a
|
||||
top-level `↻ UPD4TE 4LL` button appears that loops over every
|
||||
stale container.
|
||||
593
docs/agent-lifecycle/persistence.md
Normal file
593
docs/agent-lifecycle/persistence.md
Normal file
|
|
@ -0,0 +1,593 @@
|
|||
# Persistence + retention
|
||||
|
||||
Where state lives, what survives what, and how it's bounded.
|
||||
|
||||
## For operators
|
||||
|
||||
The short answer to "will I lose anything": **destroying an agent
|
||||
keeps its state, purging it doesn't.**
|
||||
|
||||
- **`DESTR0Y`** (the default action) stops and removes the container
|
||||
but keeps everything on disk — config history, claude login, `/state/`
|
||||
notes, harness data. The agent shows up as a tombstone (K3PT ST4T3 on
|
||||
the C0R3 page) with a `⊕ R3V1V3` button that recreates it from the
|
||||
kept state, **no re-login needed**.
|
||||
- **`PURG3`** (opt-in, from the dashboard or `hivectl agent <name>
|
||||
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
|
||||
history, claude credentials, `/state/` notes, everything. **No
|
||||
undo.** Only reach for this when you actually want the agent gone
|
||||
for good.
|
||||
|
||||
Beyond that:
|
||||
|
||||
- **Approvals are kept forever** — they're an audit trail, not a
|
||||
cache. Nothing about them ever ages out.
|
||||
- **Broker messages**: acked ones vacuum after 30 days; anything
|
||||
undelivered or delivered-but-not-yet-acked is always kept, however
|
||||
old.
|
||||
- **An agent's own `/state/` notes and claude login survive every
|
||||
restart and rebuild** — only an explicit purge (or a hive
|
||||
`--purge`-style host operation, or the agent's own choices) touches
|
||||
them.
|
||||
- The **root/bootstrap agent is special**: it isn't really destroyable
|
||||
in practice — hive-c0re recreates it automatically on its next
|
||||
startup if it's ever gone.
|
||||
|
||||
Everything below this point is implementation detail: exact table
|
||||
schemas, file layouts, and internal migration mechanics.
|
||||
|
||||
## Sqlite databases
|
||||
|
||||
### `/var/lib/hyperhive/db/broker.sqlite` (host)
|
||||
|
||||
Seven tables, all in one file — three queues, a small key/value
|
||||
table, the schedule header/targets split, and the per-agent
|
||||
power-intent registry:
|
||||
|
||||
- `messages` — every inter-agent / operator-bound message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
in_reply_to / priority`. `in_reply_to` links a reply to its parent
|
||||
row id; the dashboard and per-agent inbox render these as threaded
|
||||
rows.
|
||||
- `kv` — small persistent key/value store (`key PK / value`) for
|
||||
host-side bookkeeping that doesn't warrant its own table.
|
||||
|
||||
⚠️ The `mcp__hyperhive__remind` queue is **not** here any more: it
|
||||
moved to a harness-local, per-agent store as part of the
|
||||
loose-ends-v2 migration — see [`/harness/` contents
|
||||
below](#state-dirs-per-agent) for where reminders (and todos)
|
||||
actually live now.
|
||||
- `approvals` — the queue. `agent / kind (merge_config_pr | spawn |
|
||||
init_config | update_meta_inputs | schedule_prompt) /
|
||||
commit_ref / requested_at / status / resolved_at / note`.
|
||||
- `scheduled_prompts` — recurring + one-shot prompt queue.
|
||||
`owner / body / interval_seconds (NULL = one-shot) /
|
||||
next_fire_at_unix / created_at_unix / source ("operator" or
|
||||
"approval:<id>") / cancelled_at_unix / description`. `owner`
|
||||
drives cancel-permission checks (operator vs the submitting
|
||||
agent). Cancelled rows are tombstoned and reaped by the worker
|
||||
on its next pass.
|
||||
- `scheduled_prompt_targets` — per-target state for each schedule.
|
||||
`schedule_id / target / cancelled_at_unix /
|
||||
last_fired_at_unix / last_result`. `ON DELETE CASCADE` from
|
||||
`scheduled_prompts(id)` — requires `PRAGMA foreign_keys = ON`
|
||||
per connection (set at open).
|
||||
- `agent_power` — one tiny row per agent: `agent PK / wanted (up |
|
||||
offline) / updated_at`, owned by `hive-c0re/src/stores/power.rs`.
|
||||
This is the durable power *intent* the job queue reconciles the
|
||||
observed container state against; intent survives hive-c0re
|
||||
restarts even though in-flight queue work doesn't. See
|
||||
[`docs/scheduler/coordinator.md`'s Desired-state
|
||||
section](../scheduler/coordinator.md#desired-state-spec-vs-status) for who
|
||||
writes and reads it and how reconciliation works.
|
||||
|
||||
Retention:
|
||||
|
||||
- `Broker::vacuum_delivered` runs hourly via a tokio task in
|
||||
`hive-c0re::main`. Drops acked message rows older than 30 days
|
||||
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
|
||||
rows are always kept — the harness `ack_turn`s only after a
|
||||
successful turn, so an unacked row can still be requeued via
|
||||
`requeue_inflight` on a crash.
|
||||
- Approvals are kept indefinitely — an audit trail. `actions::destroy`
|
||||
rows stay visible to anything that queries by id.
|
||||
- Scheduled prompts: one-shot rows are deleted on fire by the
|
||||
worker; recurring rows live until the operator cancels them
|
||||
(`cancel_schedule` MCP / dashboard ✗) which tombstones via
|
||||
`cancelled_at_unix`, then `reap_cancelled` drops the row on
|
||||
the next worker pass.
|
||||
- `agent_power` rows live until the agent is destroyed (one row per
|
||||
agent — nothing to vacuum).
|
||||
|
||||
### `/harness/hyperhive-events.sqlite` (per agent)
|
||||
|
||||
Lives inside each container's bind-mounted `/harness/` dir (host
|
||||
path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
|
||||
One table:
|
||||
|
||||
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
|
||||
harness emits during turn loop execution.
|
||||
|
||||
The harness both writes and vacuums it — this used to be a host-side
|
||||
sweep, but hive-c0re runs as the unprivileged `hive-core` user under
|
||||
privsep and can't delete agent-owned files (host-side deletes hit
|
||||
`PermissionDenied` on the bash-task trio and a readonly-database error
|
||||
here), so cleanup moved in-container. `hive-agent`'s `vacuum::run`
|
||||
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
|
||||
**type-scoped**: it deletes only the verbose `stream` rows (the raw
|
||||
claude `stream-json` deltas — one per text chunk / tool use, the bulk
|
||||
of the file's size) older than 14 days, and keeps every other kind
|
||||
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
|
||||
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
|
||||
small and carry the semantic per-turn history the operator scrolls
|
||||
back through when debugging a regression. Age-only within the
|
||||
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
|
||||
history sooner than a quiet one. The trade-off (accepted): a
|
||||
misbehaving harness could now skip its own cleanup, which the old
|
||||
host-side sweep was meant to prevent — but a compromised harness is
|
||||
already inside the container trust boundary
|
||||
([`docs/trust-boundary/security.md`](../trust-boundary/security.md)), and these are ephemeral local
|
||||
artifacts, so cleaning them up where they live is the honest fix.
|
||||
|
||||
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
|
||||
setups). On open failure the `Bus` falls back to no-store mode
|
||||
rather than crashing the harness — events still broadcast over SSE,
|
||||
just nothing persisted.
|
||||
|
||||
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
|
||||
|
||||
Per-turn analytics sink. One row per claude turn captures
|
||||
identity (`model`, `wake_from`, `result_kind`), timing
|
||||
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
|
||||
cache_read / cache_creation token counts), behaviour
|
||||
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
|
||||
snapshot metrics (`open_threads_count`,
|
||||
`open_reminders_count` — fetched via the same socket the harness
|
||||
already uses for `GetOpenThreads` + `CountPendingReminders`).
|
||||
Bin-loop helpers `build_row` + `record` land each row at
|
||||
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
|
||||
the turn loop continue.
|
||||
|
||||
A sibling `bash_commands(ts INTEGER, head TEXT)` table in the same
|
||||
file is written by the `hive-bash-daemon` (not the harness): one
|
||||
row per executed bash task recording the normalised command head -
|
||||
the basename of the first real command, looking past `cd repo &&`
|
||||
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
|
||||
backs the "favorite tools" view on the /stats page (aggregated
|
||||
host-side). Best-effort and created on first write
|
||||
(`CREATE TABLE IF NOT EXISTS`), so it's simply absent until a bash
|
||||
task runs.
|
||||
|
||||
turn-stats.sqlite has **no vacuum** — it's one tiny row per turn
|
||||
(~hundreds of KB even over months), read directly by the `/stats` page
|
||||
and the hive-wide stats view, so pruning it would only lose trend
|
||||
history for no space gain.
|
||||
|
||||
### `/state/hyperhive-harness.json` (per agent)
|
||||
|
||||
Consolidated harness state file written atomically (`.tmp` + rename) by
|
||||
`Bus::emit_status` whenever rate-limited or login-failed flags change.
|
||||
Shape:
|
||||
|
||||
```json
|
||||
{ "rate_limited": false, "needs_login": false, "active_model": "…" }
|
||||
```
|
||||
|
||||
- `rate_limited` — set when the harness detects a 429 from the Claude
|
||||
API; cleared by any subsequent status emit. Drives
|
||||
`ContainerView.rate_limited` on the dashboard.
|
||||
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
|
||||
cleared by `"online"` status (re-auth completed). Drives the
|
||||
`needs_login` flag alongside the `claude_has_session` check.
|
||||
- `active_model` — the resolved Claude model for the dashboard badge.
|
||||
|
||||
The turn loop is the only writer today, but it still goes
|
||||
read-modify-write under a shared in-process lock and merges into the
|
||||
existing object rather than reconstructing it — so a second writer
|
||||
would preserve fields it doesn't own, and the lock closes the
|
||||
lost-update window between a writer's read and its rename. The lock
|
||||
is in-process only, so it wouldn't serialise a writer running as a
|
||||
separate process; none of today's writers are.
|
||||
|
||||
hive-c0re reads this file on each `build_all` sweep (~10s) via
|
||||
`container_view::read_harness_flags`. Falls back to the legacy individual
|
||||
sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) if the
|
||||
JSON is absent, so existing containers keep working through the transition
|
||||
window before their next rebuild.
|
||||
|
||||
### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
|
||||
|
||||
Full stdout + stderr capture for every `nixos-container` / `nix
|
||||
build` invocation the lifecycle layer fires. One row per invocation;
|
||||
the row accumulates lines as the child runs.
|
||||
|
||||
Capturing the full stream (rather than a short tail buffer) matters
|
||||
because real eval errors routinely run long — "tried alternatives"
|
||||
blocks alone are often 30+ lines — so a truncated tail would cut off
|
||||
the actual failure and leave only the host journal holding the
|
||||
complete output. With this table the dashboard can surface the entire
|
||||
log.
|
||||
|
||||
Three indices:
|
||||
- `(agent, started_at)` — backs the per-agent latest-N lookup used
|
||||
by the agent card chip.
|
||||
- `(status, finished_at)` — backs the retention sweep that runs
|
||||
as part of the existing hourly vacuum.
|
||||
- `(node_id)` — added by a later migration so a build log row can be
|
||||
looked up by the job-queue node it belongs to (a `hive_jobq` node is
|
||||
immutable after insert, so the link is recorded on the log row
|
||||
instead); legacy rows predating the column keep `node_id IS NULL`.
|
||||
|
||||
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
|
||||
log a warning on sqlite error and let the build continue. A failed
|
||||
log row never blocks a rebuild.
|
||||
|
||||
### `/harness/hyperhive-model` (per agent)
|
||||
|
||||
Single-line text file holding the claude model name currently
|
||||
selected for this agent (default `haiku` when absent). Written by
|
||||
`Bus::set_model` whenever the operator flips it via `/model
|
||||
<name>` in the web terminal. Read once at harness boot in
|
||||
`Bus::new`. Path overridable via `HYPERHIVE_MODEL_FILE`.
|
||||
Survives destroy/recreate, gone on `--purge`.
|
||||
|
||||
### `/harness/paused` (per agent)
|
||||
|
||||
Empty marker file. Its presence parks the agent's turn loop: the
|
||||
harness keeps serving its web UI and MCP daemons but drives no turns,
|
||||
and inbox messages queue unacked until it's removed (see
|
||||
[turn loop](../turn-loop/README.md#the-loop)).
|
||||
|
||||
Unusually, it's read and written from **both** sides of the harness
|
||||
bind-mount, and that's the whole design: the harness stats it
|
||||
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
|
||||
stats it on the host (`Coordinator::is_paused`) to populate the
|
||||
`paused` field on the agent card, and creates/removes it
|
||||
(`Coordinator::set_paused`) for `hivectl agent <name> pause|resume` and the
|
||||
dashboard toggle. Because the file itself is the only shared state
|
||||
there's no protocol between them, no round-trip into the container, and
|
||||
pause keeps working when the harness is wedged or the container is
|
||||
stopped.
|
||||
|
||||
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
||||
is the agent's own space to fill, and this is harness control state.
|
||||
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
||||
back paused after a restart, which is the intended behaviour rather
|
||||
than an accident of storage.
|
||||
|
||||
## State dirs (per agent)
|
||||
|
||||
Under `/var/lib/hyperhive/agents/<name>/`:
|
||||
|
||||
- `config/` — the proposed nix repo (root-agent-editable). Bind-mounted
|
||||
**read-only** to `/agents/<name>/config` inside the sub-agent's own
|
||||
container so the agent can inspect what defines it and request
|
||||
precise changes from the root agent; RW into the root agent via the
|
||||
`/agents` tree bind.
|
||||
- `claude/` — claude OAuth credentials, bind-mounted RW to
|
||||
`/home/<name>/.claude` inside the container.
|
||||
- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
|
||||
to `/agents/<name>/state` inside the container (uniform for
|
||||
all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
|
||||
the same path to in-container scripts. Notable files written here
|
||||
by the harness:
|
||||
- `hyperhive-status` — single-line free-text status string written
|
||||
by `set_status`; cleared on explicit `set_status("")`. Read by
|
||||
hive-c0re and the per-agent `/api/dashboard-state` endpoint to
|
||||
surface the status chip on the dashboard. Absent when no status
|
||||
is set.
|
||||
- `hyperhive-harness.json` — rate-limited / needs-login flags read
|
||||
by the dashboard's async container-state fetch. See
|
||||
`docs/web-ui/dashboard.md::Container row`.
|
||||
- `harness/` — harness-internal ephemeral state; not intended for
|
||||
agent consumption. Bind-mounted to `/agents/<name>/harness`
|
||||
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
|
||||
- `bash-tasks/` — task JSON + stdout/stderr files for
|
||||
background `mcp__bash__run` jobs. JSON files are
|
||||
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
|
||||
(full captured output). The harness's own hourly sweep
|
||||
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
|
||||
event rows above) deletes terminal task trios older than 48
|
||||
hours; non-terminal (still-running) tasks are never deleted. This
|
||||
used to be a host-side `hive-c0re` vacuum, moved in-container for
|
||||
the same privsep-ownership reason as the events vacuum above.
|
||||
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
|
||||
and reminders, one small table each in a single file (in-container
|
||||
daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
|
||||
`hive-forge-notify` — upsert keyed todos here over the harness's
|
||||
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
|
||||
`get_loose_ends` output and clears a row on `mark_todo_done`).
|
||||
Replaces three formerly-separate files
|
||||
(`hyperhive-todos.sqlite`, `hyperhive-reminders.sqlite`, and the
|
||||
old file-based `mcp-loose-ends/` scanner before that) — a one-time
|
||||
boot migration (`db_migrate::run`) folds the legacy files into this
|
||||
path the first time a harness boots after the upgrade. Also backs
|
||||
the `mcp__hyperhive__remind` queue, which moved from a host-side
|
||||
`broker.sqlite` table to this per-agent store as part of the same
|
||||
migration.
|
||||
|
||||
The harness itself is also a producer, not just the socket server:
|
||||
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
|
||||
socket the out-of-process daemons above dial) alongside
|
||||
`disk_watch::run` — an *in-process* todo producer that shares the
|
||||
store + wake `Notify` directly rather than dialling its own socket.
|
||||
`disk_watch` raises a keyed `disk` todo when the filesystem backing
|
||||
this agent's state gets tight, naming the agent's own biggest
|
||||
directories; the summary is bucketed and carries no raw byte
|
||||
counts, so an unchanged situation re-upserts as `changed == false`
|
||||
and never re-wakes.
|
||||
|
||||
Retention, same hourly `hive-agent::vacuum::run` sweep as
|
||||
`hyperhive-events.sqlite` below: delivered (soft-deleted) reminder
|
||||
rows are reaped 14 days after delivery, kept that long only to serve
|
||||
the trailing-window `ReminderRollup` stats; acked todo rows are
|
||||
reaped 30 days after acking (long enough that only a genuinely quiet
|
||||
month triggers the "one spurious re-announcement" fallback a
|
||||
reconciled producer like `disk_watch` relies on — see `todos.rs`'s
|
||||
module doc). Un-acked todos and undelivered reminders are never
|
||||
swept — same "audit trail, not cache" treatment as the c0re-side
|
||||
tables above.
|
||||
- `hyperhive-events.sqlite` — turn-loop event log.
|
||||
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
|
||||
- `hyperhive-model` — single-line model name override file.
|
||||
|
||||
### Parent access to child state
|
||||
|
||||
A parent agent gets each direct child's `state` dir bind-mounted
|
||||
**read-write** and its `config` dir **read-only**
|
||||
(`bind_child_agent_dirs` in `lifecycle/host_config.rs`). The RW on
|
||||
`state` is deliberate, not an oversight: a parent manages its children,
|
||||
which includes writing into a child's state for recovery (e.g. seeding
|
||||
notes, clearing a stuck sentinel) as well as reading it.
|
||||
|
||||
**`harness` is not mounted at all.** It holds the child's own runtime
|
||||
material — `bash-tasks/`, the turn-stats and event sqlite dbs — and
|
||||
nothing argues for a parent reading it, let alone writing it. hive-c0re
|
||||
reads a child's harness dir **directly on the host** when it wants
|
||||
those stats, which needs no mount into the parent.
|
||||
|
||||
**`config` is read-only, including for the parent.** A config change is
|
||||
a PR on the child's config repo, made from a clone and merged after
|
||||
review — so the bind-mounted `config` dir is a *copy to read*, never a
|
||||
tree anyone edits in place. Mounting it writable would leave a second
|
||||
path to the same file that skips the review entirely, which makes the
|
||||
boundary a convention rather than a permission.
|
||||
|
||||
⚠️ Not to be confused with the seeding done when an `InitConfig`
|
||||
approval resolves: that writes the child's initial config repo as
|
||||
**hive-c0re, against the host path**, and `read_only` on a bind
|
||||
constrains writers *inside* a container only. The two are unrelated —
|
||||
conflating them is an easy way to reason your way into thinking this
|
||||
mount should be writable when it shouldn't.
|
||||
|
||||
Per-child isolation still holds: a container only ever has its *own*
|
||||
dirs plus its direct children's bind-mounted, never a sibling's.
|
||||
|
||||
Under `/var/lib/hyperhive/applied/<name>/` — the hive-c0re-only
|
||||
applied repo. Tracks `flake.nix` (module-only boilerplate; never
|
||||
edited after first spawn) + `agent.nix` (the actual config; the
|
||||
root agent's edits land here via the approval flow) + any other
|
||||
files committed via the approval flow. `.git/` carries the proposal /
|
||||
approved / building / deployed / failed / denied tag history.
|
||||
|
||||
Under `/var/lib/hyperhive/meta/` — the swarm-wide deploy flake plus
|
||||
system-level config files. Single git repo for the whole host; every
|
||||
hive-c0re mutation that should survive a restart is committed here.
|
||||
Contents:
|
||||
|
||||
- `flake.nix` — declares one `nixpkgs` input per agent + one
|
||||
`nixosConfigurations.<n>` output per agent. `flake.lock` is the
|
||||
canonical "what's deployed where." The git log is the deploy
|
||||
audit trail (one commit per successful deploy or hyperhive bump).
|
||||
- `topology.json` — parent/child agent graph
|
||||
(`{ "alice": "root", "bob": "alice", "root": null }`).
|
||||
Written by `topology::apply_set_parent` (the pure move-validating
|
||||
transform) via `meta::bulk_commit_topology` (the committer — see the
|
||||
`Reparent` node in [`docs/scheduler/coordinator.md`](../scheduler/coordinator.md)); read by
|
||||
the dashboard, the renderer, and `<parent>` / `<children>` recipient
|
||||
resolution.
|
||||
- `tool-groups.json` — per-agent MCP tool group grants
|
||||
(`{ "alice": ["messaging", "inbox", "execution"] }`). Written by
|
||||
`tool_groups::set_groups`; injected as `HIVE_TOOL_GROUPS` env
|
||||
var into each agent's container.
|
||||
- `capabilities.json` — per-agent capability grants
|
||||
(`{ "atlas": ["read_host_journal"] }`). Written by
|
||||
`capabilities::set_caps`; injected as `HIVE_CAPABILITIES` env
|
||||
var. Absent agents have no extra capabilities.
|
||||
- `resource-limits.json` — per-agent container resource overrides
|
||||
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
||||
Written by `resource_limits::set_limits`; read where the systemd
|
||||
drop-in is generated (`lifecycle::write_dropins`), **not** injected
|
||||
into the container — these are host-side caps on the container, so
|
||||
the capped party never sees or sets them. Fallback is per *field*:
|
||||
an absent file, absent agent, or absent field falls back to the
|
||||
hive-wide `services.hyperhive.agentCpuQuota` / `agentMemoryMax`,
|
||||
so an agent can override only its memory and still track the hive
|
||||
default for CPU. The `CPUWeight=` / `IOWeight=` shares in the same
|
||||
drop-in have **no** per-agent override — they are hive-wide only and
|
||||
come straight off `HiveEnv`, so this file has no field for them.
|
||||
|
||||
The root agent has the meta dir RO-mounted at `/meta/`.
|
||||
|
||||
There is no longer a `.meta-migration-done` marker: the
|
||||
one-shot container repoint it guarded has been removed, since
|
||||
containers are rendered onto `meta#<n>` at creation. A stale
|
||||
marker file left over from an older hive is inert and can be
|
||||
deleted.
|
||||
|
||||
## Destroy vs purge
|
||||
|
||||
See [For operators](#for-operators) above for what each action does to
|
||||
an agent's state. The mechanics, for completeness:
|
||||
|
||||
- `DESTR0Y` also drops the systemd drop-in and fails any pending
|
||||
approvals; the tombstone's `⊕ R3V1V3` button queues a Spawn approval
|
||||
that reuses the kept state on approve.
|
||||
- `PURG3` wipes `/var/lib/hyperhive/{agents,applied}/<name>/` — the
|
||||
union of everything `DESTR0Y` left behind.
|
||||
|
||||
The root/bootstrap agent's specialness is implemented as a soft policy
|
||||
guard in `actions::destroy` that refuses to destroy it, backstopped by
|
||||
`auto_update::ensure_root_agent`, which recreates it on the next
|
||||
hive-c0re startup if it's ever absent (bypassing the approval queue,
|
||||
as required infrastructure) — so even without the guard, destroying it
|
||||
would only be transient.
|
||||
|
||||
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
|
||||
|
||||
On a btrfs host, a brand-new agent's state root is created as a
|
||||
**btrfs subvolume** instead of a plain directory (progressive
|
||||
enhancement — see the #1762 lane). This is a no-op fallback on
|
||||
non-btrfs hosts and for any agent whose root already exists, so
|
||||
nothing is auto-migrated: existing agents keep their plain dirs
|
||||
until an explicit opt-in upgrade.
|
||||
|
||||
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
|
||||
the per-agent subdirs are created (spawn / rebuild / InitConfig).
|
||||
It skips the work when the root already exists; otherwise it asks
|
||||
hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
|
||||
root when the FS is btrfs (`statfs` magic gate) and chown it to the
|
||||
`hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
|
||||
succeed inside it.
|
||||
- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
|
||||
reuses it untouched.
|
||||
- **PURG3 deletes it correctly:** a subvolume root can't be removed
|
||||
with `rmdir`/`remove_dir_all`, so purge first calls hive-priv
|
||||
(`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
|
||||
actually a subvolume, then the normal `remove_dir_all` sweep covers
|
||||
plain-dir agents + the applied dir.
|
||||
|
||||
Per-subvolume disk-usage accounting and optional quotas have since
|
||||
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
|
||||
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
|
||||
`hivectl agent <name> quota show|set` reads/limits one agent's
|
||||
subvolume usage through the same hive-priv-mediated path as
|
||||
subvolume creation/deletion above.
|
||||
|
||||
This is the same subvolume `hivectl agent <name> subvol snapshot push`
|
||||
sends to the swarm's snapshot store — see
|
||||
[`docs/snapshot-store.md`](../networking/snapshot-store.md) for what a pushed
|
||||
snapshot contains and how the store authenticates a sender.
|
||||
|
||||
## `/var/lib/swarm-controller/` (swarm-controller host only)
|
||||
|
||||
Only present on the one host running
|
||||
`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
|
||||
so it survives restarts and redeploys.
|
||||
|
||||
- `webhook-secret` — the HMAC key the swarm's forge webhooks are signed
|
||||
with. **Keep it.** It is handed to Forgejo when a hook is registered,
|
||||
so replacing the file means every subsequent delivery fails
|
||||
verification until the hook is re-registered with the new value. It is
|
||||
generated automatically on first start; there is nothing to configure.
|
||||
|
||||
If the file is unreadable at startup the daemon still starts and logs
|
||||
`webhook secret unavailable`; the webhook endpoint then answers 503
|
||||
rather than accepting deliveries it cannot verify. Everything else the
|
||||
controller serves is unaffected.
|
||||
|
||||
## Run-time dirs
|
||||
|
||||
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
||||
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
|
||||
Without that, every restart wipes bind sources and existing
|
||||
containers can't be started.
|
||||
|
||||
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
|
||||
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
|
||||
(bind-mounted into the container as `/run/hive/mcp.sock`).
|
||||
|
||||
On startup, `Coordinator::register_agent` drops any prior socket
|
||||
task before rebinding — idempotent so a hive-c0re restart followed
|
||||
by `rebuild alice` recreates the agent's socket without a clean
|
||||
reinstall.
|
||||
|
||||
## First-boot agent-user migration
|
||||
|
||||
The harness runs as a per-agent unix user inside the container
|
||||
(`hyperhive.user.name`, defaults to the agent's logical label so each
|
||||
container has a uniquely-named user). Operators with legacy root-owned
|
||||
state dirs need a one-time data shuffle so they don't lose their claude
|
||||
session.
|
||||
|
||||
`system.activationScripts.hive-agent-user-migrate` (in
|
||||
`nix/agent-modules/user.nix`) runs on every activation,
|
||||
marker-guarded so the substantive moves only happen once per
|
||||
container lifetime:
|
||||
|
||||
1. **`${homeDir}` exists with the right ownership** — covers the
|
||||
very first boot before `useradd`'s `createHome` has had a
|
||||
chance to chown. Also re-applies on every rebuild in case the
|
||||
meta-flake's per-agent name evolves (rare).
|
||||
2. **Migrate any leftover `/root/.claude` content into
|
||||
`${homeDir}/.claude`** — legacy `claude` wrote to root's
|
||||
empty home; the bind mount didn't exist yet. Marker
|
||||
(`/var/lib/hive-agent-user-migrated`) guards single-shot.
|
||||
`cp -an` (no-clobber) so any pre-existing files at the new
|
||||
location win — never blow over data already there.
|
||||
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
|
||||
recursively so the agent user can read/write it. Wildcard
|
||||
matches the single agent that container sees; `-h` skips
|
||||
symlinks the agent might have planted.
|
||||
4. **Chown the `~/.claude/` bind-mount** recursively. Legacy
|
||||
`claude` wrote `.credentials.json` 0600 root:root; the
|
||||
current harness reads `~/.claude/` as the agent user to decide
|
||||
Online vs NeedsLogin in `login::has_session`. Without the
|
||||
chown the existing credentials get silently treated as "no
|
||||
session" and the operator re-prompts every boot.
|
||||
|
||||
The activation script will eventually become unnecessary once no
|
||||
operators have legacy root-owned state dirs left to migrate; drop
|
||||
the body + marker check at that point.
|
||||
|
||||
## Matrix per-agent daemon + token-arrival trigger
|
||||
|
||||
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
|
||||
process per agent. Serves its MCP tools directly over
|
||||
streamable-http (`hyperhive.mcp.matrixHttpPort`, no stdio bridge —
|
||||
same shape as `hive-bash-daemon`), emits hyperhive wake signals
|
||||
on incoming room events via `/run/hive/mcp.sock`. Conditional on
|
||||
`hyperhive.matrix.enable` (which both the daemon AND the
|
||||
auto-injected `extraMcpServers.matrix` entry read).
|
||||
|
||||
**First-boot ordering**: hive-c0re provisions the matrix token AFTER
|
||||
agent containers come up. Without the path-trigger sibling
|
||||
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
|
||||
/agents/*/state/matrix-token*` — the trailing `*` also catches a
|
||||
secondary multi-account token like `matrix-token-ccc`), the daemon
|
||||
would exit 0 quietly the first time it ran and the MCP would have no
|
||||
backend until the next restart. The `.path` unit makes the appearance
|
||||
of the token re-fire the service so the daemon comes alive in the
|
||||
same boot cycle as
|
||||
provisioning. The same token watcher also drives avatar setting: on a
|
||||
restart the daemon re-runs each account's bring-up, which sets the
|
||||
avatar (see below).
|
||||
|
||||
### matrix avatar (set by the daemon over the live Client)
|
||||
|
||||
The agent icon (`hyperhive.icon`, an SVG) is published as each matrix
|
||||
account's profile avatar by `hive-matrix-daemon` itself
|
||||
(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
|
||||
the daemon builds + restores an account's `Client` (authenticated,
|
||||
pointed at that account's resolved homeserver), it calls matrix-sdk's
|
||||
`account().upload_avatar()` — one call that uploads the media and sets
|
||||
`avatar_url`. Because it reuses the live Client, there is no hardcoded
|
||||
homeserver URL, no token re-read, and no token-file globbing: the daemon
|
||||
already iterates every configured + dashboard-discovered account in its
|
||||
bring-up loop, so the avatar is set for **every** account.
|
||||
|
||||
Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
|
||||
librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
|
||||
unit, gated on `hyperhive.icon != null`. No icon configured → the env is
|
||||
unset → `sync_avatar` returns early and no avatar is set.
|
||||
|
||||
Idempotency is **per-account**: an `avatar-icon-hash` file in each
|
||||
account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
|
||||
skips the upload when unchanged, because every upload mints a fresh
|
||||
`mxc://` URI that emits a profile state event in every joined room —
|
||||
re-uploading identical bytes is timeline spam. A dashboard-provisioned
|
||||
account gets its avatar when the `systemd.paths.hive-matrix-daemon` token
|
||||
watcher restarts the daemon (which re-runs the per-account bring-up), so
|
||||
no separate avatar trigger is needed. Avatar failures are swallowed
|
||||
(logged, non-fatal) so they never break account bring-up or sync.
|
||||
|
||||
Loading…
Reference in a new issue