docs: the handbook follows the three layers: swarm, hives, agents
The handbook's sections were agent-lifecycle, integrations, swarm, tools, trust-boundary and web-ui, which do not follow how the system is built, so a reader could not tell which layer a page is about. Its pages now sit in swarm/, hives/ and agents/, with the accounts an agent can hold outside the system in external-accounts/. - swarm/: forge, matrix and the knowledge repo join the swarm pages, and so does the swarmctl reference. - hives/: the dashboard, hivectl and its reference, and the operator/agent boundary, with a README that takes over the old web-ui and tools pages. - agents/: the roster, persistence, config changes, the agent's own web page (was web-ui/agent) and the security model, with a README. - external-accounts/: GitHub and GitLab. Pages are moved whole; none is split yet. Every reference follows: relative links in the docs, `docs/2-handbook/...` paths in code comments, the nix checks that regenerate the CLI references and the ignore list, and docs/README.md, whose handbook section is now by layer.
This commit is contained in:
iris
mara
parent
263b21021a
commit
0e84fe8bc4
95 changed files with 393 additions and 363 deletions
|
|
@ -4,8 +4,8 @@
|
|||
# generator, not the formatter, owns these files' layout. Every CLI
|
||||
# reference doc belongs here — adding one and forgetting this line is
|
||||
# what made swarmctl's fail.
|
||||
docs/2-handbook/tools/hivectl-cli.md
|
||||
docs/2-handbook/tools/swarmctl-cli.md
|
||||
docs/2-handbook/hives/hivectl-cli.md
|
||||
docs/2-handbook/swarm/swarmctl-cli.md
|
||||
docs/4-implementation/agent-tools/forge-cli.md
|
||||
docs/4-implementation/agent-tools/swarm-logs-cli.md
|
||||
docs/4-implementation/agent-tools/swarm-metrics-cli.md
|
||||
|
|
@ -15,14 +15,14 @@ docs/4-implementation/agent-tools/swarm-metrics-cli.md
|
|||
# indentation normalisation, so these files need manual re-wrap before
|
||||
# adding them back to the formatter scope.
|
||||
AGENTS.md
|
||||
docs/2-handbook/agent-lifecycle/config-changes.md
|
||||
docs/2-handbook/agents/config-changes.md
|
||||
docs/5-project/process/conventions.md
|
||||
docs/4-implementation/networking/gateway.md
|
||||
docs/2-handbook/integrations/matrix.md
|
||||
docs/2-handbook/agent-lifecycle/persistence.md
|
||||
docs/2-handbook/swarm/matrix.md
|
||||
docs/2-handbook/agents/persistence.md
|
||||
docs/4-implementation/web-ui/terminal-rendering.md
|
||||
docs/4-implementation/agent-tools/forge.md
|
||||
docs/4-implementation/agent-tools/matrix.md
|
||||
docs/4-implementation/turn-loop/README.md
|
||||
docs/2-handbook/web-ui/agent.md
|
||||
docs/2-handbook/web-ui/dashboard.md
|
||||
docs/2-handbook/agents/web-page.md
|
||||
docs/2-handbook/hives/dashboard.md
|
||||
|
|
|
|||
16
AGENTS.md
16
AGENTS.md
|
|
@ -10,12 +10,12 @@ when you need depth on a subsystem. This file is the index.
|
|||
the env var — inside an agent container `localhost` is the *agent*, not
|
||||
the forge, so a hardcoded loopback address fails to connect.
|
||||
- Operator/agent trust-boundary design:
|
||||
**[docs/2-handbook/trust-boundary/boundary.md](docs/2-handbook/trust-boundary/boundary.md)** (`area/ops` issues
|
||||
**[docs/2-handbook/hives/boundary.md](docs/2-handbook/hives/boundary.md)** (`area/ops` issues
|
||||
for the deployment/gateway/privsep work — the separator is a **slash**,
|
||||
and `--label area:ops` now fails with `did you mean "area/ops"?`).
|
||||
- Agent trust model (trust boundary, prompt-injection threat model,
|
||||
capability = accepted risk), credential isolation + sandbox threat model:
|
||||
**[docs/2-handbook/trust-boundary/security.md](docs/2-handbook/trust-boundary/security.md)**.
|
||||
**[docs/2-handbook/agents/security.md](docs/2-handbook/agents/security.md)**.
|
||||
- Odd-looking nspawn / container plumbing usually has a story:
|
||||
**[docs/5-project/process/gotchas.md](docs/5-project/process/gotchas.md)**.
|
||||
|
||||
|
|
@ -33,8 +33,8 @@ nix fmt # treefmt is the CI gate, NOT bare cargo fmt
|
|||
|
||||
Changed a `hivectl` / `swarmctl` verb or flag? Regenerate its doc — only
|
||||
CI's `nix flake check` catches the drift:
|
||||
`nix develop -c cargo run --bin hivectl -- markdown-docs > docs/2-handbook/tools/hivectl-cli.md`
|
||||
(same for `swarmctl` → `docs/2-handbook/tools/swarmctl-cli.md`). CI red but local
|
||||
`nix develop -c cargo run --bin hivectl -- markdown-docs > docs/2-handbook/hives/hivectl-cli.md`
|
||||
(same for `swarmctl` → `docs/2-handbook/swarm/swarmctl-cli.md`). CI red but local
|
||||
green → build the suspect check (e.g. `nix build
|
||||
.#checks.x86_64-linux.hivectl-docs`) before blaming the builder.
|
||||
|
||||
|
|
@ -58,10 +58,10 @@ Per-hive (host + in-container):
|
|||
crate.
|
||||
- **`hivectl/`** — operator CLI; talks to `hive-c0re` over the host admin
|
||||
socket, does NOT link it. Verb reference (CI-enforced):
|
||||
[`docs/2-handbook/tools/hivectl-cli.md`](docs/2-handbook/tools/hivectl-cli.md).
|
||||
[`docs/2-handbook/hives/hivectl-cli.md`](docs/2-handbook/hives/hivectl-cli.md).
|
||||
- **`hive-priv/`** — minimal root helper, socket-activated at
|
||||
`/run/hive/priv.sock`; the few root ops (bind-mount edits, nsenter)
|
||||
`hive-c0re` delegates. See `docs/2-handbook/trust-boundary/boundary.md`.
|
||||
`hive-c0re` delegates. See `docs/2-handbook/hives/boundary.md`.
|
||||
- **`hive-agent/`** — the in-container serve loop: turn-loop *policy*
|
||||
(`turn.rs`) over `hive-runtime`, per-agent web UI, event + turn-stats
|
||||
sinks, system-prompt renderer, claude launch config (`mcp_config.rs`).
|
||||
|
|
@ -130,7 +130,7 @@ Swarm-level:
|
|||
**`users.yml` in place**, shared with `swarm-authelia-bridge`; that
|
||||
crate's README says what both writers must uphold. Verb reference
|
||||
(CI-enforced):
|
||||
[`docs/2-handbook/tools/swarmctl-cli.md`](docs/2-handbook/tools/swarmctl-cli.md).
|
||||
[`docs/2-handbook/swarm/swarmctl-cli.md`](docs/2-handbook/swarm/swarmctl-cli.md).
|
||||
- **`swarm-authelia-bridge/`** (+ **`swarm-authelia-bridge-sock/`** wire
|
||||
types) — the other writer of authelia's users DB; runs inside
|
||||
`swarm-authelia`.
|
||||
|
|
@ -225,7 +225,7 @@ in this file and drifted.
|
|||
- **Operator-facing doc with real implementation detail mixed in**: wrap
|
||||
the implementation part in a `<details><summary>…</summary>` block
|
||||
(blank line after `<summary>` so the markdown inside renders) rather
|
||||
than a sibling page — see `docs/2-handbook/integrations/github.md`'s
|
||||
than a sibling page — see `docs/2-handbook/external-accounts/github.md`'s
|
||||
"Implementation" section. It only collapses in a rendered browser, and
|
||||
the `<summary>` line isn't markdown-processed, so keep it plain prose
|
||||
with no backticks.
|
||||
|
|
|
|||
16
README.md
16
README.md
|
|
@ -92,13 +92,13 @@ box it gets:
|
|||
credentials, and notifications turned into todos. Optional linked
|
||||
accounts on other forges, GitLab (`glab` + push), and GitHub (`gh` + push) via
|
||||
`services.hyperhive.agent.github.enable`. →
|
||||
[forge](docs/2-handbook/integrations/forge.md) · [gitlab](docs/2-handbook/integrations/gitlab.md) ·
|
||||
[github](docs/2-handbook/integrations/github.md)
|
||||
[forge](docs/2-handbook/swarm/forge.md) · [gitlab](docs/2-handbook/external-accounts/gitlab.md) ·
|
||||
[github](docs/2-handbook/external-accounts/github.md)
|
||||
- **matrix** — its own account(s) on the swarm homeserver, as MCP tools. →
|
||||
[matrix](docs/4-implementation/agent-tools/matrix.md)
|
||||
- **knowledge** — the swarm's shared `internal/knowledge` repo, read-only
|
||||
at `/knowledge`, plus hyperhive's own docs in the container. →
|
||||
[knowledge](docs/2-handbook/integrations/knowledge.md)
|
||||
[knowledge](docs/2-handbook/swarm/knowledge.md)
|
||||
- **web UI** — a per-agent page behind the gateway: live terminal of the
|
||||
turn stream, model / effort switching, cancel.
|
||||
- **GUI** (opt-in, `services.hyperhive.agent.gui.enable`) — a Wayland
|
||||
|
|
@ -175,11 +175,11 @@ once, top to bottom.
|
|||
[`docs/2-handbook/swarm/ui.md`](docs/2-handbook/swarm/ui.md)
|
||||
- **`swarmctl`** — the swarm's CLI, on the swarm-controller host: create
|
||||
agents and mint their identities, manage SSO users, make forge admins.
|
||||
→ [reference](docs/2-handbook/tools/swarmctl-cli.md)
|
||||
→ [reference](docs/2-handbook/swarm/swarmctl-cli.md)
|
||||
- **per-hive** — each hive's dashboard (container state,
|
||||
rebuild queue) and `hivectl` for host-level administration and
|
||||
container shells. → [`docs/2-handbook/web-ui/`](docs/2-handbook/web-ui/README.md) ·
|
||||
[hivectl](docs/2-handbook/tools/hivectl.md)
|
||||
container shells. → [`docs/2-handbook/hives/`](docs/2-handbook/hives/README.md) ·
|
||||
[hivectl](docs/2-handbook/hives/hivectl.md)
|
||||
|
||||
You are an SSO user too: `swarmctl user add <you> --group admins`, and a
|
||||
first login through authelia creates your forge and matrix accounts;
|
||||
|
|
@ -193,8 +193,8 @@ pick the page for your task rather than reading front to back. Some
|
|||
common entry points:
|
||||
|
||||
- connecting hives into a swarm → [`2-handbook/swarm/`](docs/2-handbook/swarm/README.md)
|
||||
- how config changes flow → [`2-handbook/agent-lifecycle/config-changes.md`](docs/2-handbook/agent-lifecycle/config-changes.md)
|
||||
- trust boundary + threat model → [`trust-boundary/`](docs/2-handbook/trust-boundary/boundary.md)
|
||||
- how config changes flow → [`2-handbook/agents/config-changes.md`](docs/2-handbook/agents/config-changes.md)
|
||||
- trust boundary + threat model → [`hives/boundary.md`](docs/2-handbook/hives/boundary.md), [`agents/security.md`](docs/2-handbook/agents/security.md)
|
||||
- what an agent's turn looks like → [`4-implementation/turn-loop/`](docs/4-implementation/turn-loop/README.md)
|
||||
- every `services.hyperhive.*` option → [options reference](https://hyperhive.darkest.space/options/)
|
||||
|
||||
|
|
|
|||
|
|
@ -114,7 +114,7 @@ container keeps starting with the hive meanwhile.
|
|||
From here on, ask ruth. Open her card on `/agents` in the swarm UI and use the
|
||||
message box under her terminal. She is the manager agent: her MCP surface is the
|
||||
privileged one that configures agents, and she can recover the others. Changes to
|
||||
an agent's config arrive as PRs you merge. → [`2-handbook/agent-lifecycle/agent-roster.md`](../2-handbook/agent-lifecycle/agent-roster.md)
|
||||
an agent's config arrive as PRs you merge. → [`2-handbook/agents/agent-roster.md`](../2-handbook/agents/agent-roster.md)
|
||||
|
||||
To create an agent yourself instead, use the swarm UI, or:
|
||||
|
||||
|
|
@ -129,7 +129,7 @@ swarm UI's job view for progress.
|
|||
|
||||
Later config changes are PRs on `agent-configs/iris`; you merge them on the
|
||||
forge, and the merge deploys them. →
|
||||
[`2-handbook/agent-lifecycle/config-changes.md`](../2-handbook/agent-lifecycle/config-changes.md#config-changes)
|
||||
[`2-handbook/agents/config-changes.md`](../2-handbook/agents/config-changes.md#config-changes)
|
||||
|
||||
## After setup: use the swarm UI
|
||||
|
||||
|
|
|
|||
24
docs/2-handbook/agents/README.md
Normal file
24
docs/2-handbook/agents/README.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Agents
|
||||
|
||||
An **agent** is one Claude instance in its own NixOS container, with its own
|
||||
state directory, forge account and matrix accounts. This section covers
|
||||
running one: its identity, what survives a restart, how its config changes, what
|
||||
its own page shows and how far you can trust it.
|
||||
|
||||
- **[Roster](agent-roster.md)** — the agents of a hive, and who may do what
|
||||
to whom.
|
||||
- **[Persistence](persistence.md)** — what survives a restart, a destroy and
|
||||
a purge.
|
||||
- **[Config changes](config-changes.md)** — how a change to an agent's config
|
||||
flows from a pull request to its container.
|
||||
- **[The agent's web page](web-page.md)** — the per-agent terminal view, its
|
||||
actions and its endpoints.
|
||||
- **[Security model](security.md)** — the trust model, the prompt-injection
|
||||
threat model and credential isolation.
|
||||
|
||||
The swarm creates and configures agents: the create form and its archetypes
|
||||
are in [`swarm/archetypes.md`](../swarm/archetypes.md), logging an agent in to
|
||||
claude is in [`swarm/claude-login.md`](../swarm/claude-login.md), and you set
|
||||
an agent's permissions with `swarmctl agent permissions` or on the swarm UI.
|
||||
What an agent can do inside its container is in
|
||||
[`4-implementation/agent-tools/`](../../4-implementation/agent-tools/README.md).
|
||||
133
docs/2-handbook/agents/agent-roster.md
Normal file
133
docs/2-handbook/agents/agent-roster.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
# Agent roster & privileges
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
Agents are a **flat set**: the `manage_root_agent` permission scopes
|
||||
which agents can manage which others.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
This doc covers what the roster file holds, the map-shaped alternative
|
||||
`topology::all_agents` accepts, and what's hard-coded for the manager.
|
||||
|
||||
## Where the roster lives
|
||||
|
||||
"The roster" means two different, non-overlapping things depending on
|
||||
scope. The [README](../../../README.md)'s control-plane row ("`swarm-controller`
|
||||
holds the hive directory, the agent roster and the job graph") is the
|
||||
**swarm-wide** one: `GET /api/agents` on swarm-controller reads
|
||||
straight from the swarm's authelia identity store over
|
||||
`swarm-authelia-bridge` (`swarm-controller/src/auth.rs::list_agent_identities`)
|
||||
— "every agent the swarm holds an identity for" (`swarm-controller/src/routes/agents.rs`'s
|
||||
`get_agents` doc comment). It's authoritative: an agent exists in the
|
||||
swarm if and only if it has an identity there, and `swarmctl agent
|
||||
create` (or the swarm UI, via `POST /api/agents`) writes that identity
|
||||
once, by calling `ensure_agent_identity`.
|
||||
|
||||
This page is about a different, **hive-local** file: the hive-c0re-owned
|
||||
`meta/` directory's `topology.json`, at
|
||||
`/var/lib/hyperhive/meta/topology.json`:
|
||||
|
||||
```json
|
||||
["alice", "bob", "ruth"]
|
||||
```
|
||||
|
||||
One entry per agent _this hive_ has state/config for, in name order.
|
||||
Unlike the swarm roster, it's a derived cache: the reconcile pass below
|
||||
rebuilds it from what the hive observes locally (config repos cloned, containers spawned), and it
|
||||
answers a narrower question than "does this agent exist": _which of
|
||||
this hive's local agents the `ManageRootAgent` permission's bind-mounts
|
||||
should cover._ `topology::all_agents` is the only reader that matters.
|
||||
|
||||
That reader is a permission boundary: the set it returns is what an agent
|
||||
holding the `ManageRootAgent` permission gets bind-mounted (each other
|
||||
agent's `state` read-write and `config` read-only; never `harness`). An
|
||||
agent holding no permission sees its own dirs and nothing else:
|
||||
`ManageRootAgent` is the only grant that reaches another agent's state
|
||||
dir. See
|
||||
[`persistence.md`](persistence.md)'s _Cross-agent access to state._
|
||||
|
||||
### Reading the map-shaped format
|
||||
|
||||
`topology.json` may also be a map of `name → parent | null`; the reader
|
||||
accepts that shape and takes its keys as the roster. An empty
|
||||
roster costs more than a cosmetic gap: every permission holder loses its
|
||||
mounts until the next reconcile pass writes the array form.
|
||||
|
||||
### Why meta, not per-agent `agent.nix`
|
||||
|
||||
An agent shouldn't be able to add itself to a set that governs who may
|
||||
reach its state dir. The roster IS a system-level fact; meta is where
|
||||
system-level facts live.
|
||||
|
||||
### How `topology.json` gets updated
|
||||
|
||||
- **Read** — parsed into a set of names; a missing or unparsable file
|
||||
degrades safely to "no agents" (covers a fresh install that hasn't
|
||||
synced yet).
|
||||
- **Reconcile** — runs alongside the periodic meta/flake regeneration.
|
||||
Adds newly spawned agents, drops removed ones. Reconcile keeps agents
|
||||
whose config repo exists but that haven't spawned yet, so the gap until
|
||||
the container appears doesn't churn the file.
|
||||
|
||||
No write API and no operator verb reach this file. Reconcile derives it
|
||||
from which agents exist, so the next pass overwrites a hand edit.
|
||||
|
||||
See `hive-c0re/src/agent_config/topology.rs` and `hive-c0re/src/meta.rs`'s
|
||||
module docs for the exact call chain.
|
||||
|
||||
## Hard-coded manager behaviour
|
||||
|
||||
The **manager (`ruth`)** differs from other agents in naming/bootstrap
|
||||
and its socket flavour; its permissions, prompt, tool allow-list and
|
||||
state dirs work as for every other agent.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
- **Naming/bootstrap** — the manager's broker recipient name, state-dir
|
||||
key, and nixos-container name are all `ruth` (container `h-ruth`).
|
||||
Like every agent it comes from the swarm dashboard or
|
||||
`swarmctl agent create`; `hive-c0re` creates nothing at boot.
|
||||
Roster-wise, `ruth` is just another entry.
|
||||
- **Wire-protocol** — `hive-core-agent-sock`'s unified `Request` enum
|
||||
is the same on both socket flavours. Container lifecycle
|
||||
ops (kill/start/restart/rebuild) go through the separate host-admin
|
||||
socket `hivectl` speaks. `Wake` (inject a `from: <X>` message straight
|
||||
into the caller's own inbox) is on this socket too, unprivileged on
|
||||
both flavours; matrix, bash and forge don't call it — their
|
||||
notifications push a todo on the harness's in-agent socket (see
|
||||
[`docs/4-implementation/turn-loop/mcp.md`](../../4-implementation/turn-loop/mcp.md#waking-the-agent-from-inside-the-container)).
|
||||
- **Storage/mounts** — the `manage_root_agent` permission, which the swarm
|
||||
UI's setup assistant creates ruth with, gets a container
|
||||
`/var/lib/hyperhive/agents` bind-mounted RW at `/agents` (so it can
|
||||
manage any agent's state dir; config isn't mounted, since a real
|
||||
config change is a PR from a clone), plus an RO mount for
|
||||
`/meta` (the hive's topology and limits files). Any agent holding it gets the same mounts.
|
||||
- **Prompt** — identical for every container but one block (below):
|
||||
`prompts/system.md`
|
||||
carries `<!-- role:agent -->` / `<!-- role:manager -->` marker
|
||||
blocks, and `prompt::render` filters for `"agent"` unconditionally,
|
||||
manager included ("always `agent` role — there is only one role",
|
||||
`hive-agent/src/prompt.rs`'s own module doc). The `role:manager`
|
||||
blocks are dead in production, exercised only by a unit test
|
||||
(`filter_blocks(SAMPLE, "role", "manager")`). The one block that differs
|
||||
is `<!-- config:repo -->` vs `<!-- config:archetype -->`: an agent built
|
||||
from an [archetype](../swarm/archetypes.md) is told it has no config repo.
|
||||
- **Tool allow-list** — the MCP tools claude sees come from the agent's
|
||||
permissions alone, the same mechanism for every agent
|
||||
([`docs/4-implementation/turn-loop/mcp.md`](../../4-implementation/turn-loop/mcp.md)). Ruth's permissions
|
||||
are the default plus `manage_root_agent`, stored for her like any grant.
|
||||
- **State dirs** — _not_ treated as an exception: `HYPERHIVE_STATE_DIR` is
|
||||
injected uniformly via `systemd.globalEnvironment` for every
|
||||
container including the manager, so all token/state paths resolve
|
||||
through it the same way everywhere.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
None of the above is a stable interface — treat the module doc
|
||||
comments as the source of truth for exactly which checks exist today.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- Milestone: "Agent privileges and sub-agents" (tracked internally)
|
||||
- Audit table source: milestone comment (tracked internally)
|
||||
- Operator/agent trust boundary (orthogonal axis): [`boundary.md`](../hives/boundary.md)
|
||||
383
docs/2-handbook/agents/config-changes.md
Normal file
383
docs/2-handbook/agents/config-changes.md
Normal file
|
|
@ -0,0 +1,383 @@
|
|||
# Config changes + system messages
|
||||
|
||||
Config changes are pull requests on the agent's config repo, which an
|
||||
operator merges on the forge; that merge deploys them. The "System messages"
|
||||
section covers the plain-text `system` messages hive-c0re sends agents.
|
||||
|
||||
## For operators
|
||||
|
||||
- **Config change** — an agent proposed a change to another agent's
|
||||
config (or its own, via a sub-manager) as a pull request on
|
||||
`agent-configs/<agent>`. Review it on the forge like any other PR, and
|
||||
merge it there to deploy it. See [Config changes](#config-changes).
|
||||
|
||||
Everything below this point is the implementation detail behind that
|
||||
flow.
|
||||
|
||||
## Config changes
|
||||
|
||||
Config changes flow through a **forge pull request** on the agent's
|
||||
`agent-configs/<name>` repo — the same surface agents use for code PRs.
|
||||
No MCP tool exists for config changes: opening the PR IS the request.
|
||||
|
||||
1. The submitting agent **clones** `agent-configs/<name>`, edits it there (any
|
||||
tracked path, but `agent.nix` is the contract entry point), commits
|
||||
with its own git identity, and pushes a branch + opens a PR with
|
||||
`hive-forge` — the same way it would change any other repo.
|
||||
The container has no config mount: the clone is how an agent reads its
|
||||
own config as well as how it changes it.
|
||||
Branch protection (the agent isn't on the `main` push allowlist;
|
||||
merge and approvals allowlist = `operators` team; see "Forge config repos"
|
||||
below) makes the agent a write collaborator that **can't merge its own
|
||||
config PR**.
|
||||
2. An operator reviews the PR **on the forge** (native diff, threaded
|
||||
comments, CI status) and merges it there. Merging needs membership of
|
||||
the `operators` team in `agent-configs`. The team starts empty: add
|
||||
each operator by hand in the forge UI.
|
||||
3. swarm-controller gets the `agent-configs` org's `pull_request`
|
||||
delivery. A `closed` event with `merged: true` and base branch `main`
|
||||
names the commit in `merge_commit_sha`.
|
||||
4. The controller looks up which hive's wanted state places the agent
|
||||
and publishes a deploy request carrying that commit on that hive's
|
||||
deploy subject. With no such hive, or more than one, it deploys
|
||||
nothing and logs a warning naming them.
|
||||
5. If the closure the swarm declares for the agent is the one the hive last
|
||||
applied (the `applied/.<n>.closure` marker), the hive does nothing.
|
||||
Otherwise it queues a rebuild, which downloads the declared closure. The
|
||||
request's commit plays no part: the hive keeps no copy of the config and
|
||||
reads no forge head. An agent with no container on that hive yet gets its
|
||||
first deploy from the declared closure.
|
||||
|
||||
This path runs **no eval-verify**. A merged config that doesn't
|
||||
evaluate or build fails the rebuild, so the agent's rebuilds keep failing
|
||||
until a fix merges. A failed rebuild shows on the hive like any other. A merge
|
||||
whose webhook delivery never reaches the controller deploys nothing.
|
||||
|
||||
Creating a brand-new agent is swarm-level (`swarmctl agent create`,
|
||||
`POST /api/agents` on the swarm controller), and no hive can originate
|
||||
an agent. The swarm controller's `InitAgentConfigRepo` job seeds the
|
||||
agent's config repo with a default `agent.nix` template, then asks the
|
||||
target hive to deploy it.
|
||||
An agent created from an [archetype](../swarm/archetypes.md) gets no
|
||||
config repo: the swarm builds it from the archetype's module.
|
||||
|
||||
Changing what the template seeded isn't an exception: like every
|
||||
later change, it's a PR on that config repo, made from a clone and
|
||||
merged by an operator on the forge (see [Config changes](#config-changes)).
|
||||
|
||||
## Agent lifecycle
|
||||
|
||||
### Reminder delivery: file-path semantics
|
||||
|
||||
A reminder may carry a `file_path` (the agent-visible path inside its
|
||||
container, for example `/agents/<name>/state/foo.md`). On delivery hive-c0re:
|
||||
|
||||
1. **Translates** the container path to the host path
|
||||
(`/var/lib/hyperhive/agents/<name>/state/foo.md`) so c0re can write
|
||||
from outside the container.
|
||||
2. **Validates** the path: rejects anything outside the agent's own state
|
||||
subtree, containing `..` (path traversal), or with an empty relative
|
||||
tail. On rejection hive-c0re skips the write and delivers the
|
||||
original message inline with a warning — the reminder still fires.
|
||||
3. **Defends against symlink escape**: after `create_dir_all`, hive-c0re
|
||||
canonicalizes the parent dir and re-verifies it lives under the agent's host
|
||||
state root. hive-c0re opens the final file with
|
||||
`O_NOFOLLOW | O_CREAT | O_TRUNC` so an existing symlink at the
|
||||
basename can't redirect the write to an arbitrary host path.
|
||||
4. **Writes the body to disk** and delivers a short pointer message in its
|
||||
place, keeping the agent's inbox / wake-prompt small while the agent
|
||||
reads the bulky payload out of band.
|
||||
|
||||
`Broker::deliver_reminders_batch` handles atomicity of the inbox INSERT + `reminders.sent_at` UPDATE; the scheduler only computes the
|
||||
body strings before calling it.
|
||||
|
||||
### Destroy semantics
|
||||
|
||||
`HostRequest::Destroy { name, purge }` is the lifecycle tear-down.
|
||||
Stops + removes the nspawn container and drops the systemd drop-in.
|
||||
Persistent state (claude credentials, `/state/` notes) is
|
||||
**kept by default** — recreating the agent with the same name
|
||||
reuses prior notes + login. With `purge = true` the agent's
|
||||
`/var/lib/hyperhive/agents/<name>/` tree is also
|
||||
wiped (creds + notes gone forever). The
|
||||
root/bootstrap container is destroyable like any other, and stays gone
|
||||
until you create ruth again from the swarm dashboard.
|
||||
|
||||
## Meta flake
|
||||
|
||||
A hive has no meta flake of its own. It builds every agent from the swarm's
|
||||
`internal/meta`, which swarm-controller commits ([below](#the-swarms-meta-flake)).
|
||||
It declares one flake input per agent (`agent-<n>.url =
|
||||
"git+<forge>/agent-configs/<n>.git?rev=<sha>"`, the head of `main` on the
|
||||
agent's forge config repo, the one config PRs merge into) and one
|
||||
`nixosConfigurations.<n>` output per agent. Each output wraps
|
||||
`inputs.agent-<n>.nixosModules.default` with the identity +
|
||||
`HIVE_PORT` / `HIVE_LABEL` injection module. Values that differ per host
|
||||
(the dashboard port, the pinned claude-code, extra OTEL labels, the queue
|
||||
address) aren't in it:
|
||||
`hive-priv` writes them to the container's `/etc/hyperhive-hive-facts` at
|
||||
start (see `docs/5-project/process/conventions.md`).
|
||||
A config that sits only in a local checkout, not on forge `main`, never
|
||||
deploys.
|
||||
|
||||
What `hive-c0re` still keeps under `/var/lib/hyperhive/meta/` is plain
|
||||
state, bound read-only into the root agent's container as `/meta`:
|
||||
`topology.json`, the swarm resource limits, and the permission seed files of a
|
||||
hive that predates the swarm's permissions. `agent_set::refresh(agents:
|
||||
&[AgentSpec])` keeps `topology.json`, the gateway's agent-socket map and its
|
||||
nginx include in line with the live agents. `provision_container`, `destroy`,
|
||||
`rebuild` and the startup migration call it. It refuses an empty agent list,
|
||||
which means `nixos-container list` failed, and it fails when `HIVE_FORGE_URL`
|
||||
is unset, since every agent would come up with no forge.
|
||||
|
||||
## The swarm's meta flake
|
||||
|
||||
swarm-controller commits `internal/meta`: one `flake.nix` and `flake.lock` for
|
||||
every agent any hive places (the pins:
|
||||
[`docs/4-implementation/swarm/builder.md`](../../4-implementation/swarm/builder.md#meta-flake-lock)). The swarm
|
||||
builder builds every agent from it, and a hive deploys that build. While the
|
||||
swarm has recorded no build of an agent, that agent's deploy fails and names the
|
||||
agent. The controller queues
|
||||
its update of the repo before the deploy request for a merged config commit
|
||||
and for a new agent, and sends the deploy only when that update succeeded and
|
||||
the swarm builder built the agent from the updated repo (the
|
||||
[deploy gate](../../4-implementation/swarm/builder.md#deploy-gate)). A failed build skips the deploy, and the timer builds again. A merge whose webhook delivery never arrives gets the same update, build and deploy within five minutes, from the controller's drift pass (see the paragraph on config drift in the deploy gate section).
|
||||
|
||||
- **Choosing the closure.** The hive doesn't choose. It deploys the closure the
|
||||
agent's record in `agent-closures` declares, whichever `internal/meta` commit
|
||||
the builder built it from, and reads no forge head. The rebuild's `MetaSync`
|
||||
node prepares it (the agent's archetype, its runtime directories, the agent
|
||||
set), reads the record once and pins its store path. `Prebuild` and `Swap` use
|
||||
the pin, so a record that changes mid-rebuild can't make `Swap` pull another
|
||||
closure than `Prebuild` fetched, and the marker names the closure it applied. The
|
||||
next autoupdate pass picks up the newer record. A first deploy has no
|
||||
`MetaSync`; its `Create` reads the record once.
|
||||
- **The closure.** The hive builds nothing. `Prebuild`, `Swap` and `Create` read
|
||||
the agent's record in the swarm's `agent-closures` bucket (the controller
|
||||
writes it from the builder's reply) and have `hive-priv` download the closure
|
||||
it names from the swarm cache (`RealiseClosure`). `Swap` and `Create` then
|
||||
give `hive-priv` the store path (`toplevel` on `UpdateContainer` and
|
||||
`CreateContainer`), which accepts only the path its last `RealiseClosure` for
|
||||
that agent returned. On the wire the
|
||||
`hive_priv_sock::PrivRequest` variants carry a required field:
|
||||
`{"op":"update_container","name":"atlas","stream":true,"toplevel":"/nix/store/<hash>-nixos-system-atlas"}`.
|
||||
A rebuild with no record, or none the builder pushed, fails with an error
|
||||
that names the agent; see
|
||||
[`docs/4-implementation/swarm/builder.md`](../../4-implementation/swarm/builder.md#rebuilds-download-the-closure).
|
||||
- **Hyperhive.** The commit pins one hyperhive rev for the swarm, so an agent
|
||||
runs the hyperhive of the controller's deploy, not its hive's.
|
||||
- **Marker.** After a successful swap or create `hive-c0re` writes the closure's
|
||||
store path to `applied/.<n>.closure`. The autoupdate pass
|
||||
([below](#autoupdate-on-startup)) compares it with the record.
|
||||
|
||||
## Deploy markers per agent
|
||||
|
||||
```
|
||||
/var/lib/hyperhive/applied/ deploy markers - core-only
|
||||
└── .<name>.closure # store path of the last closure applied
|
||||
|
||||
/var/lib/hyperhive/meta/ plain hive state - core
|
||||
├── topology.json # the live agent set
|
||||
└── swarm-resource-limits.json # the swarm's per-agent caps
|
||||
```
|
||||
|
||||
hive-c0re keeps no copy of an agent's config: the forge's `agent-configs/<name>`
|
||||
`main` is the only one, and a config change is a PR against it. The marker
|
||||
records the closure last applied, and nothing else. Hives that ran before the
|
||||
marker existed have `.<name>.hyperhive-rev`, `.<name>.config-rev` and
|
||||
`meta-rev/<name>` files instead. Nothing reads them any more, and you can
|
||||
delete them.
|
||||
|
||||
Older hives also kept an *applied* repo at `applied/<name>` and a *proposed*
|
||||
clone at `agents/<name>/config`. Nothing reads either any more, and you can
|
||||
delete them.
|
||||
|
||||
The container's `--flake` ref is `internal/meta` (see "Meta flake" above).
|
||||
The agent's `agent-configs/<n>/flake.nix`
|
||||
is a fixed boilerplate that exports `nixosModules.default =
|
||||
import ./agent.nix`; the meta flake imports that module and
|
||||
wraps it with identity + `HIVE_PORT` / `HIVE_LABEL`.
|
||||
|
||||
### Forge config repos
|
||||
|
||||
The bundled `hive-forge` container runs on the swarm's forge host
|
||||
(`deploy.forgejo.enable`, see [`2-handbook/swarm/services.md`](../swarm/services.md)).
|
||||
Every agent created through the swarm has a repo in the private
|
||||
`agent-configs` Forgejo org, created and seeded by swarm-controller.
|
||||
hive-c0re pushes nothing to it, keeps no clone and reads nothing from it: forge
|
||||
`main` advances only by an operator's merge.
|
||||
|
||||
Each agent is a **write collaborator on its own** `agent-configs/<name>`
|
||||
repo — so it can push a branch and open a config PR — but not a member
|
||||
of any other agent's, so it can't reach another agent's config through
|
||||
the forge. Branch protection keeps the agent off the `main` push
|
||||
allowlist and allowlists merging to the `operators` team only, so an
|
||||
agent can't fast-forward its own config or self-merge its PR (see
|
||||
[Config changes](#config-changes) above). hive-c0re passes the credential to
|
||||
each `git` call inline, never writing it into a repo's config, so
|
||||
no stored token sits in a repo an agent might one day read.
|
||||
|
||||
The dashboard deep-links into this org — a `config repo` link
|
||||
per container row. See `docs/2-handbook/hives/dashboard.md`.
|
||||
|
||||
### Submitting agent's view of config repos
|
||||
|
||||
No agent has a config repo mounted, not even one holding
|
||||
`ManageRootAgent`: reading another agent's config means cloning its repo
|
||||
from the forge, subject to the forge's own access rules.
|
||||
|
||||
An agent submits a change to another agent's config the same way it
|
||||
makes any other change: **clone the target agent's config repo from the
|
||||
forge into its own state dir, commit on a branch, open a PR**, and let
|
||||
the operator review and merge it. By design, no second, mount-shaped
|
||||
path reaches the same file without the review.
|
||||
|
||||
Agents holding the `manage_root_agent` permission (granted per agent in
|
||||
the swarm's `agent-permissions` record; see
|
||||
`docs/5-project/process/conventions.md#permissions`) get additional host-side bind
|
||||
mounts via `set_nspawn_flags`:
|
||||
|
||||
- `/var/lib/hyperhive/agents/<n>/state` → `/agents/<n>/state` (RW) for
|
||||
**every** agent. The permission means "may manage any agent," so the mounted
|
||||
set is every agent.
|
||||
- `/var/lib/hyperhive/meta/` → `/meta/` (RO) — the hive's plain state
|
||||
(topology, swarm limits).
|
||||
|
||||
An agent without the permission only has its direct children's config
|
||||
dirs.
|
||||
|
||||
⚠️ nspawn bakes bind flags at container start, so hive-c0re restarts
|
||||
an agent, after its current turn, when its `manage_root_agent` changes.
|
||||
|
||||
No agent holds the permission by default: an agent whose record leaves
|
||||
`permissions` unset gets the default, which doesn't include
|
||||
`manage_root_agent`, whatever the agent's name. The swarm UI's setup
|
||||
assistant creates ruth with it. For a ruth that predates that, whose
|
||||
permissions were unset, swarm-controller writes the default plus
|
||||
`manage_root_agent` into her key once at start, so she keeps her mounts;
|
||||
it never touches a key that's already set.
|
||||
|
||||
## Startup migrations (older hosts)
|
||||
|
||||
hive-c0re runs a couple of idempotent migrations on every startup so a
|
||||
host set up before the current deploy scheme (described above)
|
||||
converges to it automatically. Each phase is a no-op
|
||||
once already applied:
|
||||
|
||||
- **Agent files**: refreshes the files that follow the agent set
|
||||
(`agent_set::refresh`) from the current agent list. Set
|
||||
`HIVE_SKIP_META_MIGRATION=1` on the service to defer this phase.
|
||||
- **Legacy meta repo**: deletes the leftover `.git` in
|
||||
`/var/lib/hyperhive/meta/` and the
|
||||
unread `/var/lib/hyperhive/forge-core-token`. The files beside the `.git`
|
||||
stay. The forge-side `core/meta` repo and `core` account aren't touched.
|
||||
|
||||
No state loss in either migration: claude creds, `/state/` notes, the
|
||||
events DB all survive. The root
|
||||
agent keeps its session; sub-agents stay logged in.
|
||||
|
||||
## The root/bootstrap container
|
||||
|
||||
The root agent container runs through the **same lifecycle as
|
||||
sub-agents**. Nothing creates it at hive boot: you create ruth from the
|
||||
swarm dashboard like any agent, and an existing ruth container starts with
|
||||
the hive like every other. The root agent can clone
|
||||
`agent-configs/ruth` to read its own `agent.nix` and open a config PR there
|
||||
for operator review, same as any other agent.
|
||||
|
||||
Differences from sub-agents:
|
||||
|
||||
- Reference docs: her `agent.nix` turns on `services.hyperhive.agent.docs.enable`.
|
||||
Her container builds from `agent-base` like every agent, and the seeded
|
||||
`agent.nix` is the same for every name, so a newly created ruth has no docs
|
||||
until a config PR adds that line.
|
||||
- Web UI port via `lifecycle::agent_web_port("ruth")` — same
|
||||
FNV-1a hash as every other agent (8100..8999 range).
|
||||
- The root agent holds `manage_root_agent` (the swarm UI's setup
|
||||
assistant creates her with it), for which `set_nspawn_flags` adds extra
|
||||
binds: every agent's `state` dir (RW) under `/agents/<n>/state`, and the
|
||||
hive's plain state under `/meta` (RO).
|
||||
- Its socket comes from the generic per-agent registration, in the same
|
||||
per-agent runtime dir as any other agent (`/run/hyperhive/agents/ruth/`),
|
||||
not a separate manager-only path.
|
||||
|
||||
**Migration note** (for older hosts): drop any `containers.root =
|
||||
{ ... }` block from your host NixOS config, and any `services.hyperhive.ruthless`
|
||||
definition, then create ruth from the swarm dashboard.
|
||||
|
||||
## Root-agent policy
|
||||
|
||||
The system prompt (`hive-agent/prompts/system.md`, rendered by
|
||||
`hive-agent/src/prompt.rs`) is the **same for every agent**; what
|
||||
varies is which MCP tools it surfaces (gated by the agent's permissions,
|
||||
which the swarm sets). No `role:manager` block renders only
|
||||
for the root agent. The root agent's role-specific
|
||||
behaviour comes from its AGENTS.md / agent-specific instructions, not
|
||||
the system prompt template.
|
||||
|
||||
## System messages
|
||||
|
||||
Hive-c0re has no typed event that drives an immediate claude turn. A
|
||||
message from sender `system` is plain inbox text, and the harness logs it
|
||||
(`log_system_event` in `hive-agent/src/main.rs`) without parsing the body.
|
||||
Today hive-c0re sends two: the `/knowledge updated` broadcast
|
||||
(`workers/knowledge.rs`) and the wake message for a graceful stop
|
||||
(`Coordinator::kick_agent`).
|
||||
|
||||
**A previously running container crashing** drives no agent turn.
|
||||
`crash_watch` only `tracing::warn!`s `"container crash
|
||||
detected"`; a swarm-level grafana alert matches that log line directly
|
||||
and pages the operator, with no agent (and no per-hive dashboard
|
||||
banner) in the loop. See
|
||||
[`docs/4-implementation/scheduler/observability.md`](../../4-implementation/scheduler/observability.md#container-crash-alerting-hive-c0re).
|
||||
|
||||
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
||||
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
||||
when convenient" events with no reason to drive an immediate turn, so
|
||||
they go out via `swarm_notices::notify` instead: `subsystem = "core"`,
|
||||
`key = "<event>:<agent>"`, and a single free-text `summary`
|
||||
(`rebuilt_todo_summary` renders `Rebuilt`'s `ok`/`note`/`sha`/`tag`
|
||||
fields into that string).
|
||||
|
||||
To add a new lifecycle notice: if it's "FYI, check when convenient,"
|
||||
call `swarm_notices::notify` — no new wire type needed. If it needs to
|
||||
drive an immediate turn, decide first whether it actually needs an
|
||||
*agent* in the loop at all (the crash case above didn't, once a
|
||||
swarm-level alert could carry the same signal).
|
||||
|
||||
## Autoupdate on startup
|
||||
|
||||
`hive-c0re serve` runs `auto_update::run` in a background task right
|
||||
after opening the coordinator. It enumerates managed containers and
|
||||
rebuilds any whose closure marker differs from the store path the agent's
|
||||
record in `agent-closures` declares. That's how a hive that missed the
|
||||
swarm's deploy request catches up, and it repeats every five minutes, which
|
||||
covers a request lost while the hive is up. Sub-agents and the root agent go
|
||||
through the same graceful rebuild subgraph.
|
||||
|
||||
The comparison reads the record and the marker
|
||||
(`/var/lib/hyperhive/applied/.<name>.closure`, written after a successful
|
||||
swap or create) and nothing on the forge. Where the marker is missing, as for
|
||||
every agent on the first boot after the marker arrived, the hive reads the
|
||||
closure the container's profile
|
||||
(`/nix/var/nix/profiles/per-container/h-<name>/system`) points at and writes
|
||||
that as the marker, so the first pass catches an agent that doesn't run its declared
|
||||
closure. Those first-boot rebuilds go through the same job queue as
|
||||
any other: each holds one of the `buildSlots`, so they don't swap all at once.
|
||||
It rebuilds nothing when the hive can't tell: no connection to the swarm queue,
|
||||
no record, a closure the builder didn't push, or a profile it can't read or that
|
||||
leaves the nix store. The periodic pass
|
||||
attempts a declared store path once per process, and skips an agent while a
|
||||
rebuild of it waits or runs, so a closure that fails to deploy doesn't
|
||||
restart its agent every five minutes. Restarting `hive-c0re`, or a rebuild from
|
||||
the agent's menu, tries again.
|
||||
|
||||
A stopped agent gets its rebuild at its next start: the start path upgrades
|
||||
the start of a stale, stopped agent to a rebuild followed by the start. In
|
||||
the window after a config merge, before the swarm's builder has recorded the
|
||||
new closure, the rebuild deploys the previous one.
|
||||
|
||||
The dashboard shows no pending-update state: the hive applies a changed
|
||||
declaration itself. The agent menu's rebuild queues the same
|
||||
`job_queue::templates::rebuild` DAG, which downloads and applies the declared
|
||||
closure again.
|
||||
842
docs/2-handbook/agents/persistence.md
Normal file
842
docs/2-handbook/agents/persistence.md
Normal file
|
|
@ -0,0 +1,842 @@
|
|||
# Persistence + retention
|
||||
|
||||
Where state lives, what survives what, and how it's bounded.
|
||||
|
||||
## For operators
|
||||
|
||||
The short answer to "does anything disappear": **destroying an agent
|
||||
keeps its state, purging it doesn't.**
|
||||
|
||||
- **`DESTR0Y`** (the default action) stops and removes the container
|
||||
but keeps everything on disk — config history, claude login, `/state/`
|
||||
notes, harness data. The agent shows up as a tombstone (K3PT ST4T3 on
|
||||
the C0R3 page); `swarmctl agent create` with the same name and hive
|
||||
recreates it from the kept state, **no re-login needed**.
|
||||
- **`PURG3`** (opt-in, from the dashboard or `hivectl agent <name>
|
||||
destroy --purge`) is `DESTR0Y` plus wiping all of it — config
|
||||
history, claude credentials, `/state/` notes, everything. **No
|
||||
undo.** Only reach for this when you actually want the agent gone
|
||||
for good. The agent's matrix encryption keys go with the state dir,
|
||||
but an account whose key backup is on (see
|
||||
[matrix](../swarm/matrix.md)) gets them back from its
|
||||
homeserver when it next comes up, so its encrypted history stays
|
||||
readable.
|
||||
|
||||
Beyond that:
|
||||
|
||||
- **Broker messages**: acked ones vacuum after 30 days; anything
|
||||
undelivered or delivered-but-not-yet-acked is always kept, however
|
||||
old.
|
||||
- **An agent's own `/state/` notes and claude login survive every
|
||||
restart and rebuild** — only an explicit purge (or a hive
|
||||
`--purge`-style host operation, or the agent's own choices) touches
|
||||
them.
|
||||
- The **root/bootstrap agent has no guard**: destroying ruth is as final as
|
||||
for any agent, and she stays gone until you create her again from the
|
||||
swarm dashboard.
|
||||
|
||||
Everything below this point is implementation detail: exact table
|
||||
schemas, file layouts, and internal migration mechanics.
|
||||
|
||||
## Sqlite databases
|
||||
|
||||
### `/var/lib/hyperhive/db/broker.sqlite` (host)
|
||||
|
||||
One table, the message queue:
|
||||
|
||||
- `messages` — every inter-agent / operator-authored message.
|
||||
`sender / recipient / body / sent_at / delivered_at / acked_at /
|
||||
in_reply_to / priority`. `in_reply_to` links a reply to its parent
|
||||
row id; the dashboard and per-agent inbox render these as threaded
|
||||
rows.
|
||||
|
||||
⚠️ The `mcp__hyperhive__remind` queue lives in a harness-local,
|
||||
per-agent store — see [`/harness/` contents
|
||||
below](#state-dirs-per-agent) for where reminders (and todos)
|
||||
live.
|
||||
|
||||
The swarm owns power intent, so hive-c0re keeps none on disk. An older build
|
||||
also kept a `kv` table (the agents running at the last `hivectl stop`) and an
|
||||
`agent_power` table (the durable `wanted` intent) here; opening the file drops
|
||||
both.
|
||||
|
||||
On a host whose hive-c0re once kept scheduled prompts, the file also
|
||||
holds `scheduled_prompts` and `scheduled_prompt_targets`. Nothing opens
|
||||
them; scheduled prompts live at swarm level
|
||||
([`2-handbook/swarm/schedules.md`](../swarm/schedules.md)). They're safe to drop by
|
||||
hand.
|
||||
|
||||
Retention:
|
||||
|
||||
- `Broker::vacuum_delivered` runs hourly via a tokio task in
|
||||
`hive-c0re::main`. Drops acked message rows older than 30 days
|
||||
(`acked_at IS NOT NULL`). Undelivered + delivered-but-not-acked
|
||||
rows are always kept — the harness `ack_turn`s only after a
|
||||
successful turn, so `requeue_inflight` can still requeue
|
||||
an unacked row on a crash.
|
||||
|
||||
### `/harness/hyperhive-events.sqlite` (per agent)
|
||||
|
||||
Lives inside each container's bind-mounted `/harness/` dir (host
|
||||
path: `/var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite`).
|
||||
One table:
|
||||
|
||||
- `events(id, ts, kind, payload_json)` — every `LiveEvent` the
|
||||
harness emits during turn loop execution.
|
||||
|
||||
The db runs in WAL mode, so `-wal` and `-shm` files sit beside it while
|
||||
the harness is up. Copy or back up the three together, or the recent rows
|
||||
sit in the `-wal` file until a checkpoint.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
The harness both writes and vacuums it. Cleanup runs in-container
|
||||
because hive-c0re runs as the unprivileged `hive-core` user under
|
||||
privsep and can't delete agent-owned files (host-side deletes hit
|
||||
`PermissionDenied` on the bash-task trio and a readonly database error
|
||||
here). `hive-agent`'s `vacuum::run`
|
||||
(`hive-agent/src/vacuum.rs`) sweeps hourly. Retention is
|
||||
**type-scoped**: it deletes only the verbose `stream` rows (the raw
|
||||
claude `stream-json` deltas — one per text chunk / tool use, the bulk
|
||||
of the file's size) older than 14 days, and keeps every other kind
|
||||
(`turn_start`, `turn_end`, `note`, `status_changed`, `model_changed`,
|
||||
`token_usage_changed`, `turn_state_changed`) indefinitely — those are
|
||||
small and carry the semantic per-turn history the operator scrolls
|
||||
back through when debugging a regression. Age-only within the
|
||||
`stream` kind — no row cap — so a chatty turn doesn't lose its stream
|
||||
history sooner than a quiet one. The trade-off (accepted): a
|
||||
misbehaving harness can skip its own cleanup — but a compromised harness is
|
||||
already inside the container trust boundary
|
||||
([`docs/2-handbook/agents/security.md`](security.md)), and these are ephemeral local
|
||||
artifacts, so cleaning them up where they live is the honest fix.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
Path overridable via `HYPERHIVE_EVENTS_DB` (for dev / no-`/harness`
|
||||
setups). On open failure the `Bus` falls back to no-store mode
|
||||
rather than crashing the harness — events still broadcast over SSE,
|
||||
just nothing persisted.
|
||||
|
||||
### `/harness/hyperhive-turn-stats.sqlite` (per agent)
|
||||
|
||||
Per-turn analytics sink. One row per claude turn captures
|
||||
identity (`model`, `wake_from`, `result_kind`), timing
|
||||
(`started_at`, `ended_at`, `duration_ms`), cost (input / output /
|
||||
cache_read / cache_creation token counts), behaviour
|
||||
(`tool_call_count` + `tool_call_breakdown_json`), and post-turn
|
||||
snapshot metrics (`open_threads_count`,
|
||||
`open_reminders_count` — fetched via the same socket the harness
|
||||
already uses for `GetOpenThreads` + `CountPendingReminders`).
|
||||
Bin-loop helpers `build_row` + `record` land each row at
|
||||
`turn_end`; writes are best-effort, a sqlite hiccup logs + lets
|
||||
the turn loop continue.
|
||||
|
||||
The `hive-bash-daemon` (not the harness) writes a sibling
|
||||
`bash_commands(ts INTEGER, head TEXT)` table in the same
|
||||
file: one row per executed bash task recording the normalised command head -
|
||||
the basename of the first real command, looking past `cd repo &&`
|
||||
prefixes, env-assignments, and prefix-runners like `sudo`/`env`. It
|
||||
backs the "favorite tools" view on the /stats page. Best-effort and created on first write
|
||||
(`CREATE TABLE IF NOT EXISTS`), so it's absent until a bash
|
||||
task runs.
|
||||
|
||||
turn-stats.sqlite has **no vacuum** — it's one small row per turn
|
||||
(~hundreds of KB even over months), read directly by the `/stats` page,
|
||||
so pruning it would only lose trend
|
||||
history for no space gain.
|
||||
|
||||
### `/state/hyperhive-harness.json` (per agent)
|
||||
|
||||
Consolidated harness state file written atomically (`.tmp` + rename) by
|
||||
`Bus::emit_status` whenever rate-limited or login-failed flags change.
|
||||
Shape:
|
||||
|
||||
```json
|
||||
{ "rate_limited": false, "needs_login": false, "active_model": "…" }
|
||||
```
|
||||
|
||||
- `rate_limited` — set when the harness detects a 429 from the Claude
|
||||
API; cleared by any subsequent status emit. Drives
|
||||
`ContainerView.rate_limited` on the dashboard.
|
||||
- `needs_login` — set when a turn hits 401 (expired OAuth credentials);
|
||||
cleared by `"online"` status (re-auth completed). Drives the
|
||||
`needs_login` flag alongside the `claude_has_session` check.
|
||||
- `active_model` — the resolved Claude model for the dashboard badge.
|
||||
|
||||
The turn loop is the only writer today, but it still goes
|
||||
read-modify-write under a shared in-process lock and merges into the
|
||||
existing object rather than reconstructing it — so a second writer
|
||||
would preserve fields it doesn't own, and the lock closes the
|
||||
lost-update window between a writer's read and its rename. The lock
|
||||
is in-process only, so it wouldn't serialise a writer running as a
|
||||
separate process; none of today's writers are.
|
||||
|
||||
hive-c0re reads this file on each `build_all` sweep (~10s) via
|
||||
`container_view::read_harness_flags`. Falls back to the individual
|
||||
sentinel files (`hyperhive-rate-limited`, `hyperhive-needs-login`) when the
|
||||
JSON is absent.
|
||||
|
||||
### `/var/lib/hyperhive/db/build_logs.sqlite` (host)
|
||||
|
||||
Full stdout + stderr capture for every `nixos-container` / `nix
|
||||
build` invocation the lifecycle layer fires. One row per invocation;
|
||||
the row accumulates lines as the child runs.
|
||||
|
||||
Capturing the full stream (rather than a short tail buffer) matters
|
||||
because real eval errors routinely run long — "tried alternatives"
|
||||
blocks alone are often 30+ lines — so a truncated tail would cut off
|
||||
the actual failure and leave only the host journal holding the
|
||||
complete output. With this table the dashboard can surface the entire
|
||||
log.
|
||||
|
||||
Three indices:
|
||||
- `(agent, started_at)` — backs the per-agent latest-N lookup used
|
||||
by the agent card chip.
|
||||
- `(status, finished_at)` — backs the retention sweep that runs
|
||||
as part of the existing hourly vacuum.
|
||||
- `(node_id)` — added by a later migration so a build log row can be
|
||||
looked up by the job-queue node it belongs to (a `hive_jobq` node is
|
||||
immutable after insert, so hive-c0re records the link on the log row
|
||||
instead); `node_id` is nullable.
|
||||
|
||||
Writes are best-effort: `append_stdout` / `append_stderr` / `finish`
|
||||
log a warning on sqlite error and let the build continue. A failed
|
||||
log row never blocks a rebuild.
|
||||
|
||||
Each operation also logs to the host journal, target `nixos-container`, and the
|
||||
swarm log store collects those lines. Every line carries `build_id` (this
|
||||
table's row id; empty without the store), `agent`, `kind` and `cmdline`. A
|
||||
`start` line (`event=start`) opens the build, one line follows per line of
|
||||
output, and a `finish` line (`event=finish`) closes it with `status` (`ok` or
|
||||
`fail`) and `runtime_secs`. hive-c0re logs a nix warning or error at warn, and a failed
|
||||
finish too.
|
||||
|
||||
A row is open (`finished_at IS NULL`) only while its child runs. At startup
|
||||
`Coordinator::open` closes every row still open as `fail` and appends a
|
||||
line saying hive-c0re stopped: the previous process died mid-build, and
|
||||
the retention sweep never deletes open rows.
|
||||
|
||||
### `/harness/hyperhive-model` (per agent)
|
||||
|
||||
Single-line text file holding a claude model name. The harness only
|
||||
reads it, at boot in `Bus::new` and again when the swarm clears the
|
||||
model it set. It holds the model the agent runs when neither the swarm nor
|
||||
`HIVE_DEFAULT_MODEL` (agent.nix) names one. Nothing in the harness
|
||||
writes it. Path overridable via `HYPERHIVE_MODEL_FILE`. Survives
|
||||
destroy/recreate, gone on `--purge`.
|
||||
|
||||
### `/harness/paused` (per agent)
|
||||
|
||||
Empty marker file. Its presence parks the agent's turn loop: the
|
||||
harness keeps serving its web UI and MCP daemons but drives no turns,
|
||||
and inbox messages queue unacked until it's removed (see
|
||||
[turn loop](../../4-implementation/turn-loop/README.md#the-loop)).
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
Unusually, it's read and written from **both** sides of the harness
|
||||
bind-mount, and that's the whole design: the harness stats it
|
||||
in-container via `hive-agent`'s `paths::paused_marker`, while hive-c0re
|
||||
stats it on the host (`Coordinator::is_paused`) to populate the
|
||||
`paused` field on the agent card, and creates/removes it
|
||||
(`Coordinator::set_paused`) for `hivectl agent <name> pause|resume` and the
|
||||
dashboard toggle. Because the file itself is the only shared state
|
||||
there's no protocol between them, no round-trip into the container, and
|
||||
pause keeps working when the harness is wedged or the container is
|
||||
stopped.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
It lives in `/harness/` rather than `/state/` deliberately: `/state/`
|
||||
is the agent's own space to fill, and this is harness control state.
|
||||
Survives destroy/recreate, gone on `--purge` — so a paused agent comes
|
||||
back paused after a restart, which is the intended behaviour rather
|
||||
than an accident of storage.
|
||||
|
||||
## State dirs (per agent)
|
||||
|
||||
Under `/var/lib/hyperhive/agents/<name>/`:
|
||||
|
||||
- `config/` — unused. Older hives kept a proposed nix repo here; nothing
|
||||
reads it any more, and you can delete it. An agent reads its config by
|
||||
cloning `agent-configs/<name>` from the forge.
|
||||
- `claude/` — claude OAuth credentials, bind-mounted RW to
|
||||
`/home/<name>/.claude` inside the container.
|
||||
- `state/` — durable notes and `hyperhive-harness.json`. Bind-mounted
|
||||
to `/agents/<name>/state` inside the container (uniform for
|
||||
all agents). The `$HYPERHIVE_STATE_DIR` env var exposes
|
||||
the same path to in-container scripts. Notable files written here
|
||||
by the harness:
|
||||
- `hyperhive-status` — single-line free-text status string written
|
||||
by `set_status`; cleared on explicit `set_status("")`. Read by
|
||||
hive-c0re and the per-agent `/api/dashboard-state` endpoint to
|
||||
surface the status chip on the dashboard. Absent when no status
|
||||
has a value.
|
||||
- `hyperhive-harness.json` — rate-limited / needs-login flags read
|
||||
by the dashboard's async container-state fetch. See
|
||||
`docs/2-handbook/hives/dashboard.md::Container row`.
|
||||
|
||||
The agent writes its own instructions to `AGENTS.md` here; claude and
|
||||
opencode load it automatically from their cwd, `state/`. At boot,
|
||||
`agents_md_migrate::run` renames a `state/CLAUDE.md` to `AGENTS.md`
|
||||
when no `AGENTS.md` exists. When both exist it renames neither and
|
||||
logs an error naming both paths: claude loads only `CLAUDE.md` while
|
||||
one exists in its cwd or above. The harness also publishes the file
|
||||
to the swarm, which reads that copy, and applies the git diffs the
|
||||
swarm sends to edit it. A new agent's first start writes the text given at
|
||||
create, only when no file exists. This file stays the only copy that counts:
|
||||
[`docs/2-handbook/swarm/README.md`](../swarm/README.md#an-agents-agentsmd).
|
||||
- `harness/` — harness-internal ephemeral state; not intended for
|
||||
agent consumption. Bind-mounted to `/agents/<name>/harness`
|
||||
inside the container (`$HYPERHIVE_HARNESS_DIR`). Contents:
|
||||
- `bash-tasks/` — task JSON + stdout/stderr files for
|
||||
background `mcp__bash__run` jobs. JSON files are
|
||||
`<id>.json` (status + tails), `<id>.out` / `<id>.err`
|
||||
(full captured output). The harness's own hourly sweep
|
||||
(`hive-agent`'s `vacuum::run`, same one that ages out `stream`
|
||||
event rows above) deletes terminal task trios older than 48
|
||||
hours; non-terminal (still-running) tasks are never deleted. This
|
||||
sweep runs in-container, for the same privsep-ownership reason as
|
||||
the events vacuum above.
|
||||
- `hyperhive-state.sqlite` — consolidated loose-ends-v2 store: todos
|
||||
and reminders, one small table each in a single file (in-container
|
||||
daemons — `hive-bash-daemon`, `hive-matrix-daemon`,
|
||||
`hive-forge-notify` — upsert keyed todos here over the harness's
|
||||
in-agent socket, `HIVE_AGENT_SOCKET`; the harness merges them into
|
||||
`get_loose_ends` output and clears a row on `mark_todo_done`).
|
||||
At boot, `db_migrate::run` creates this file from
|
||||
`hyperhive-todos.sqlite` and `hyperhive-reminders.sqlite` when those
|
||||
are present. Also backs the `mcp__hyperhive__remind` queue.
|
||||
|
||||
The harness itself is also a producer, not just the socket server:
|
||||
boot wiring's `spawn_todo_socket` starts `todo_server::run` (the
|
||||
socket the out-of-process daemons above dial) alongside
|
||||
`disk_watch::run` — an *in-process* todo producer that shares the
|
||||
store + wake `Notify` directly rather than dialling its own socket.
|
||||
`disk_watch` raises a keyed `disk` todo when the filesystem backing
|
||||
this agent's state gets tight, naming the agent's own biggest
|
||||
directories; it buckets the summary and carries no raw byte
|
||||
counts, so an unchanged situation re-upserts as `changed == false`
|
||||
and never re-wakes.
|
||||
|
||||
Retention, same hourly `hive-agent::vacuum::run` sweep as
|
||||
`hyperhive-events.sqlite` below: it reaps delivered (soft-deleted) reminder
|
||||
rows 14 days after delivery, kept that long only to serve
|
||||
the trailing-window `ReminderRollup` stats, and reaps acked todo rows
|
||||
30 days after acking (long enough that only a genuinely quiet
|
||||
month triggers the "one spurious re-announcement" fallback a
|
||||
reconciled producer like `disk_watch` relies on — see `todos.rs`'s
|
||||
module doc). Un-acked todos and undelivered reminders are never
|
||||
swept — same "audit trail, not cache" treatment as the c0re-side
|
||||
tables above.
|
||||
- `hyperhive-events.sqlite` — turn-loop event log.
|
||||
- `hyperhive-turn-stats.sqlite` — per-turn timing stats.
|
||||
- `hyperhive-model` / `hyperhive-effort` — single-line model name and
|
||||
effort level files, read-only fallbacks.
|
||||
- `work/` — the agent's working space for git worktrees, build dirs
|
||||
and other large working data. Bind-mounted RW to
|
||||
`/agents/<name>/work` inside the container and owned by the agent
|
||||
user. It sits outside `state/` so a claude or subagent started under
|
||||
`work/` doesn't walk up into `state/AGENTS.md` and load the agent's
|
||||
own identity. Survives `destroy`/recreate, gone on `--purge`. Counted
|
||||
in the per-agent disk metric (`measure_agent_disk`) together with
|
||||
`state/` and the container rootfs. The dashboard doesn't serve it.
|
||||
|
||||
### The container root (per agent)
|
||||
|
||||
An agent container starts with `systemd-nspawn --ephemeral`
|
||||
(`hive-priv`'s `WriteNspawnFlags` passes the flag). nspawn runs it on a
|
||||
snapshot of `/var/lib/nixos-containers/h-<name>`, and the snapshot vanishes
|
||||
at stop. The container loses anything the agent writes outside a bind: `/home/<name>`
|
||||
apart from `.claude/`, `/etc`, `/var`, `/root`, `/tmp`. What persists is
|
||||
`state/`, `work/`, `harness/`, `claude/` (at `~/.claude`), the per-container
|
||||
`/nix/var/nix/profiles` and `/nix/var/nix/gcroots` binds, and the sockets dir.
|
||||
`/home/<name>` holds caches and nothing the agent should keep: worktrees,
|
||||
clones and builds go in `work/`.
|
||||
|
||||
The nix profile needs one extra step. nix keeps a non-root user's profile in
|
||||
`~/.local/state/nix`, so the agent's tmpfiles rules
|
||||
(`nix/agent-modules/user.nix`) point that directory into
|
||||
`/nix/var/nix/profiles/per-user/<name>` (the persistent bind) and recreate
|
||||
`~/.nix-profile` at every boot. Packages the agent installs with `nix-env` or
|
||||
`nix profile` survive a restart, and go with the container on `destroy`.
|
||||
|
||||
**The one-time salvage.** The first start of an agent whose root has content
|
||||
runs `hive-priv/src/rootfs.rs` before hive-priv passes `--ephemeral`, from the
|
||||
`WriteNspawnFlags` that precedes every start, while the container is down:
|
||||
|
||||
1. Everything in the root except the cache paths (`tmp`, `var/cache`,
|
||||
`var/log/journal`, `var/lib/weston`, `var/lib/opentelemetry-collector`) is
|
||||
copied to `work/salvaged-rootfs-<YYYYMMDD>/`, owned by the agent, with
|
||||
modes and mtimes kept. The agent finds its old `/home/<name>` there.
|
||||
2. hive-priv copies the agent's nix state in the old home to the persistent profile bind, following no link in the path (the container can write there).
|
||||
3. hive-priv renames the old root to `h-<name>.pre-ephemeral-<YYYYMMDD>` and
|
||||
logs the path. Nothing deletes it; the operator removes it when ready.
|
||||
4. A fresh template root takes its place, carrying `etc/machine-id` (the old
|
||||
id, so the journal directory and DHCP identity hold) and the marker
|
||||
`etc/hyperhive-ephemeral-root`.
|
||||
|
||||
The salvage runs again only if the template holds content outside the cache
|
||||
paths, which happens when a hive without this change started the container
|
||||
persistently in between; the new copy gets a numbered directory beside the
|
||||
first. A failed salvage leaves the root where it was and the container starts
|
||||
without the flag (hive-priv logs the error), so a failure costs the switch,
|
||||
never data.
|
||||
|
||||
**Checked before every start.** nixpkgs' start script runs as host root on
|
||||
the root template, and its `mkdir -p`, `chmod`, `touch` and
|
||||
`cp --remove-destination` follow a link the container planted at the paths
|
||||
they touch. The container's limits drop-in
|
||||
(`container@h-<name>.service.d/hyperhive-limits.conf`) therefore carries an
|
||||
`ExecStartPre=` that runs `hive-priv check-rootfs h-<name>` ahead of that
|
||||
script, on systemd's own restarts too. It looks at each path with `lstat`,
|
||||
never following a link, and fails the start with one line on stderr naming the
|
||||
path and its type when:
|
||||
|
||||
- `usr`, `usr/bin`, `etc`, `var`, `var/lib`, `var/lib/private` or `root`
|
||||
exists and isn't a real directory;
|
||||
- `etc/machine-id` or `etc/resolv.conf` exists and isn't a regular file
|
||||
(`etc/os-release` may also be a link, which the script skips and a booted
|
||||
NixOS guest creates);
|
||||
- the profile's `system` (`/nix/var/nix/profiles/per-container/h-<name>/system`)
|
||||
exists and isn't a link that resolves into `/nix/store`, or the `system`
|
||||
file inside the resolved directory isn't a regular file.
|
||||
|
||||
Absent paths pass, so a first start works. The check neither deletes nor
|
||||
repairs: the operator removes the offending entry, and the next start goes
|
||||
through.
|
||||
The drop-in names the `hive-priv` binary that wrote it, so a drop-in never
|
||||
points at a build without the sub-command. A drop-in written by an older
|
||||
points at a build without the sub-command. That build's store path has a GC
|
||||
root, `/nix/var/nix/gcroots/hyperhive-check-rootfs/h-<name>`, written before the
|
||||
drop-in and replaced on each write, so a garbage collection can't remove the
|
||||
binary a restart would run. The directory sits beside `gcroots/per-container`,
|
||||
which is the only gcroots path bound into a container, so only root writes it.
|
||||
Destroying the container removes the drop-in and then its root; the build stays
|
||||
until the next collection. A drop-in written by an older hive-priv has no
|
||||
`ExecStartPre=` and no root, and the container starts as before.
|
||||
|
||||
**Hyperhive's platform containers: warn only.** The forge, the matrix server and
|
||||
the swarm services (15 containers, declared by their own modules) run the same
|
||||
check as `hive-priv check-rootfs --warn-only <name>`, from `ExecStartPre=` on
|
||||
their own `container@<name>` unit, through the symlink `/run/hive-rootfs-check`
|
||||
that `hive-priv.nix` points at the current build, so a new hyperhive build does
|
||||
not change the unit or restart the container. It evaluates every rule, logs each
|
||||
violation
|
||||
as a warning, and exits 0, so the container starts either way. The line reads
|
||||
`rootfs check (warn-only): <name>: <message>` with journald's warning priority.
|
||||
Strict refusal needs the live roots of all 15 to pass first, and the warnings
|
||||
show which ones would not. A container the operator runs that hyperhive doesn't
|
||||
declare gets no check at all: the modules name their own containers, and
|
||||
nothing enumerates the host's `containers`.
|
||||
|
||||
**Disk use.** The copy shares blocks with the old root where it can: hive-priv
|
||||
clones each file (`FICLONE`), which on btrfs or xfs allocates metadata only, and
|
||||
copies the data of a file the filesystem refuses to clone. Before it creates
|
||||
anything, hive-priv sizes the root and compares it with the free space under
|
||||
`work/`. A salvage whose files clone (tested with one file of the root) must
|
||||
leave 2 GiB free. One that has to copy data must leave 15 % of the filesystem
|
||||
free. When the free space is short, hive-priv skips the salvage, deletes
|
||||
nothing, logs `salvage skipped: <container> need <size> free <size>`, and the
|
||||
container starts persistent; the next start tries again. hive-priv checks a file
|
||||
that fails to clone against the 2 GiB reserve again before it copies the data. A
|
||||
salvage that fails partway removes the directory it created, and leaves the root
|
||||
and any earlier salvage as they were. After a salvage, hive-priv logs `salvage: <container> <size> reflinked=<n>/<m> files`.
|
||||
|
||||
**Concurrency.** hive-priv holds an `flock` on
|
||||
`/run/hive-priv/salvage-<container>.lock` (the unit's `RuntimeDirectory`, which
|
||||
no container can write) for the whole `WriteNspawnFlags`, so a second request for
|
||||
the same container waits and then finds the finished template. Across
|
||||
containers, one salvage copies at a time, so the free space each one checks
|
||||
excludes any copy still in flight. The request runs on the blocking
|
||||
thread pool, so a long copy holds no async worker. hive-c0re's per-agent lease
|
||||
already serialises the requests it sends; the lock covers hive-priv's own
|
||||
guarantee.
|
||||
|
||||
The journal still ships. The agent's collector (`nix/agent-modules/otel.nix`)
|
||||
reads the guest's own `/var/log/journal` and pushes it over OTLP, so no
|
||||
journal link to the host takes part; nspawn skips `--link-journal=try-guest`
|
||||
in ephemeral mode without failing. The journal vanishes at stop, and with it
|
||||
the lines written after the last push.
|
||||
|
||||
### User namespace (opt-in, per agent)
|
||||
|
||||
An agent whose config sets `services.hyperhive.agent.user.privateUsers = true`
|
||||
runs in its own user namespace. Every other agent keeps the host's uids.
|
||||
|
||||
**Approving the config PR that sets it authorises a host-level change:** the
|
||||
first start afterwards has hive-priv, as root, chown the agent's `state`,
|
||||
`harness`, `work` and `claude` dirs once to the host's data owner. It deletes nothing.
|
||||
|
||||
**How the hive learns the flag.** It never evaluates the agent's nix. The swarm
|
||||
builder reads `config.services.hyperhive.agent.user.privateUsers` from the
|
||||
config it evaluates in its sandbox and puts it in the agent's closure record
|
||||
(`private_users`). hive-c0re reads the record when it writes the container's
|
||||
flags and sends the value to hive-priv with each start. An agent whose module
|
||||
doesn't declare the option reads false. When the builder can't read the
|
||||
option (its eval fails or times out), the record carries no flag, and so does a
|
||||
record or builder from before the field. A record without a flag, a record
|
||||
hive-c0re can't read, or none, leaves the agent as it is.
|
||||
|
||||
- **On disk the data belongs to one host uid:gid**,
|
||||
`services.hyperhive.deploy.hive-controller.userns.dataOwner`
|
||||
(default `60000:60000`). Inside the container that owner shows as the agent
|
||||
user. hive-priv mounts the agent's `state/`, `harness/`, `work/` and
|
||||
`claude/` dirs and its socket dir with `owneridmap`, and every other bind too
|
||||
(config, `/knowledge`, the runtime dir), whose owner shows as root inside.
|
||||
- **`/nix/store` has to be on ext4, btrfs, xfs or erofs.** nspawn mounts the
|
||||
store into a namespaced container idmapped, and the kernel refuses that on
|
||||
overlayfs and on some other filesystems. `WriteNspawnFlags` for an opted-in
|
||||
agent checks the `statfs` type of `/nix/store` against an allowlist of those
|
||||
four; the VM test's erofs store idmaps on the hive's kernel. The check
|
||||
refuses every other filesystem, zfs and tmpfs included, even where a newer
|
||||
kernel could idmap it, because the gate exists to stop a container that would
|
||||
not boot. The error names the type and the fix, and
|
||||
comes before anything moves. Check a host with
|
||||
`findmnt -no FSTYPE /nix/store`.
|
||||
- **The container starts with `--private-users=pick --private-users-ownership=map`.**
|
||||
nspawn idmaps the root at mount time, so a different range for `pick` never
|
||||
walks or chowns it.
|
||||
- **The move happens once.** The first `WriteNspawnFlags` that finds the
|
||||
agent's container stopped chowns those four dirs to `dataOwner` (ownership
|
||||
only; it removes or copies nothing), writes
|
||||
`/var/lib/hive-priv/userns/<name>`, and creates the mount points in the root
|
||||
template owned by the agent user. A later request with the marker present
|
||||
chowns nothing. A running container waits for its next stop. The socket dir
|
||||
is on a tmpfs, so `EnsureAgentSocketDir` hands it to `dataOwner` at every
|
||||
start. The walk works by open directory fd and follows no link, so an agent
|
||||
or manager that can write the dirs can't redirect it to a host path.
|
||||
- **A move that fails starts over.** hive-priv opens all four dirs and
|
||||
checks the agent's ids before the first chown, so a link or file where a dir
|
||||
goes refuses with nothing changed. It then writes the ids to
|
||||
`/var/lib/hive-priv/userns/<name>.pending`. If the walk fails after that (a
|
||||
chown error, a tree nested too deep), the next start repeats it from those
|
||||
ids, then writes the marker and removes the pending file. hive-priv refuses a
|
||||
pending file that doesn't hold `<owner> <uid> <gid>`, and names it.
|
||||
- **The agent's gid must equal its uid.** `owneridmap` maps the gid of a bind
|
||||
with the uid map, so a group other than the uid's has no mapping and writes
|
||||
fail. The agent module asserts that `user.uid` and `user.gid` hold the
|
||||
same number when `privateUsers` is on. Start the agent once with the pin
|
||||
before turning it on: the move refuses an agent whose `state` dir has another
|
||||
gid than uid, naming the agent.
|
||||
- **Turning it off is manual.** hive-priv refuses to start an agent whose
|
||||
marker exists while its record says false, and one whose `dataOwner`
|
||||
changed. Chown the four dirs back, then delete the marker and any
|
||||
`<name>.pending` beside it.
|
||||
- **Keep a `ManageRootAgent` holder off it.** The other agents' dirs it mounts
|
||||
aren't remapped.
|
||||
|
||||
The pause marker needs nothing extra: hive-priv gives it the owner of the
|
||||
`harness/` dir. hive-c0re still creates agent dirs and new subvolumes as
|
||||
`hive-core`, and only the first namespaced start chowns them. A dir created
|
||||
after the move, such as a recreated subvolume, stays `hive-core`'s and the
|
||||
agent can't write to it.
|
||||
|
||||
### Cross-agent access to state
|
||||
|
||||
An agent holding the `manage_root_agent` permission gets every other
|
||||
agent's `state` dir bind-mounted **read-write** (`bind_child_agent_dirs` in
|
||||
`lifecycle/host_config.rs`). The RW is deliberate, not an oversight: the
|
||||
holder recovers other agents, which includes writing into their state (for
|
||||
example seeding notes, clearing a stuck sentinel) as well as reading it.
|
||||
|
||||
This is the **only** cross-agent mount. An agent without that permission
|
||||
sees its own dirs and nothing else.
|
||||
|
||||
**`harness` isn't mounted at all.** It holds that agent's own runtime
|
||||
material - `bash-tasks/`, the turn-stats and event sqlite dbs - and
|
||||
nothing argues for anyone else reading it, let alone writing it.
|
||||
hive-c0re reads a harness dir **directly on the host** when it wants
|
||||
those stats, which needs no mount into another container.
|
||||
|
||||
**`config` isn't mounted either.** A config change is a PR on that agent's
|
||||
config repo, made from a clone and merged after review. Anyone who wants to
|
||||
read the config clones `agent-configs/<name>` from the forge. A mount, even a
|
||||
read-only one, would show an agent a tree that's not the one that governs it
|
||||
whenever the forge has moved on.
|
||||
|
||||
Isolation still holds by default: a container has its *own* dirs and,
|
||||
unless it holds the permission, nothing else.
|
||||
|
||||
Under `/var/lib/hyperhive/applied/` — the hive-c0re-only deploy markers
|
||||
`.<name>.closure`: the store path of the closure the agent was last swapped to
|
||||
or created from, or, where the file was missing, the closure the container's
|
||||
profile pointed at when the hive first read it. No config lives here; the
|
||||
forge's `agent-configs/<name>` is the only copy. A hive that predates this
|
||||
finds per-agent `applied/<name>` repos here. Nothing reads them any more, and
|
||||
you can delete them.
|
||||
|
||||
Under `/var/lib/hyperhive/meta/` — system-level config files, plain state
|
||||
with no git history. The flake agents build from is the
|
||||
swarm's `internal/meta`, not here. A hive that predates this finds a
|
||||
leftover `.git` here; `migrate::run` deletes it at startup. Contents:
|
||||
|
||||
- `topology.json` — the agent roster (`["alice", "bob", "ruth"]`).
|
||||
Written by `topology::reconcile` from `agent_set::refresh`; read by
|
||||
`topology::all_agents`, which is the set the `manage_root_agent`
|
||||
permission grants mounts over. The reader also accepts a map-shaped
|
||||
file carrying a `parent` per agent, and keeps its keys.
|
||||
- `swarm-resource-limits.json` — the caps the swarm sets for this
|
||||
hive's agents
|
||||
(`{ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }`).
|
||||
Written only by `workers::swarm_limits`, which copies each agent's key of
|
||||
the swarm's `agent-limits` bucket here every 30 seconds. Read where
|
||||
`lifecycle::write_dropins` generates the systemd drop-in, **not** injected
|
||||
into the container — these are host-side caps on the container, so
|
||||
the capped party never sets them. It sees only the resulting memory cap,
|
||||
as a byte count or `infinity` in its runtime dir (below). Fallback is per
|
||||
*field*: an absent file, absent agent, or absent field falls back to the
|
||||
hive-wide `services.hyperhive.deploy.hive-controller.agentCpuQuota` / `agentMemoryMax`,
|
||||
so an agent can have only its memory capped and still track the hive
|
||||
default for CPU. Every value passes the validators first, and one bad
|
||||
value makes the worker change nothing here. The `CPUWeight=` / `IOWeight=`
|
||||
shares in the same drop-in have **no** per-agent setting — they're
|
||||
hive-wide only and come straight off `HiveEnv`, so this file has no field
|
||||
for them.
|
||||
- `resource-limits.json` — retired. Per-agent overrides set on the hive used
|
||||
to live here; nothing reads or writes it now. An old copy in this directory
|
||||
is harmless and you can delete it.
|
||||
|
||||
An agent holding `manage_root_agent`, normally ruth, has the meta dir
|
||||
RO-mounted at `/meta/`.
|
||||
|
||||
Outside this directory, `/var/lib/hyperhive/swarm-agent-permissions.json`
|
||||
holds the permissions the swarm grants this hive's agents, as
|
||||
`workers::swarm_permissions` last read them from the swarm's
|
||||
`agent-permissions` bucket (every 30 seconds): a copy, not the
|
||||
record, which `swarmctl agent permissions` sets. `set_nspawn_flags`
|
||||
reads it at every container start for `manage_root_agent`.
|
||||
|
||||
`/var/lib/hyperhive/swarm-agent-archetypes.json` holds the archetype each of
|
||||
this hive's archetype agents runs, as last read from the swarm's
|
||||
`agent-archetype` bucket at the agent's provision and each rebuild. An agent
|
||||
with an entry has no config repo and no `agent-<n>` meta input; see
|
||||
[archetypes](../swarm/archetypes.md).
|
||||
|
||||
hive-c0re creates containers from the closure the swarm builds from `internal/meta` (`nixosConfigurations.<n>`).
|
||||
|
||||
## Destroy vs purge
|
||||
|
||||
See [For operators](#for-operators) above for what each action does to
|
||||
an agent's state. The mechanics, for completeness:
|
||||
|
||||
- `DESTR0Y` also drops the systemd drop-in; reviving the agent is
|
||||
`swarmctl agent create` with the same name and hive, which reuses the
|
||||
kept state.
|
||||
- `PURG3` wipes `/var/lib/hyperhive/agents/<name>/` — the
|
||||
union of everything `DESTR0Y` left behind.
|
||||
|
||||
`actions::destroy` has no guard for the root/bootstrap agent: it destroys ruth
|
||||
like any other agent, and she stays gone until you create her again from the
|
||||
swarm dashboard.
|
||||
|
||||
### btrfs subvolumes for `/var/lib/hyperhive/agents/<name>`
|
||||
|
||||
On a btrfs host, `lifecycle::ensure_agent_state_subvolume` creates a brand-new agent's state root as a
|
||||
**btrfs subvolume** instead of a plain directory (progressive
|
||||
enhancement). This is a no-op fallback on
|
||||
non-btrfs hosts and for any agent whose root already exists, so
|
||||
hive-c0re automigrates nothing: existing agents keep their plain dirs
|
||||
until an explicit opt-in upgrade.
|
||||
|
||||
- **Creation:** `lifecycle::ensure_agent_state_subvolume` runs before
|
||||
hive-c0re creates the per-agent subdirs (spawn / rebuild).
|
||||
It skips the work when the root already exists; otherwise it asks
|
||||
hive-priv (`EnsureAgentSubvolume`) to `btrfs subvolume create` the
|
||||
root when the FS is btrfs (`statfs` magic gate) and chown it to the
|
||||
`hive-core` user so the normal `state/` `claude/` `harness/` mkdirs
|
||||
succeed inside it.
|
||||
- **DESTR0Y keeps the subvolume** exactly like a plain dir — revival
|
||||
reuses it untouched.
|
||||
- **PURG3 deletes it correctly:** `rmdir`/`remove_dir_all` can't remove
|
||||
a subvolume root, so purge first calls hive-priv
|
||||
(`DeleteAgentSubvolume`) which `btrfs subvolume delete`s it iff it's
|
||||
actually a subvolume, then the normal `remove_dir_all` sweep covers
|
||||
plain-dir agents.
|
||||
|
||||
Per-subvolume disk-usage accounting and optional quotas have since
|
||||
landed as the qgroup work: `hivectl quota-enable` turns on btrfs
|
||||
qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and
|
||||
`hivectl agent <name> quota show|set` reads/limits one agent's
|
||||
subvolume usage through the same hive-priv-mediated path as
|
||||
subvolume creation/deletion above.
|
||||
|
||||
This is the same subvolume `hivectl agent <name> subvol snapshot push`
|
||||
sends to the swarm's snapshot store — see
|
||||
[`docs/4-implementation/networking/snapshot-store.md`](../../4-implementation/networking/snapshot-store.md) for what a pushed
|
||||
snapshot contains and how the store authenticates a sender.
|
||||
|
||||
The subvolume is the whole agent state root, so `work/` is part of every
|
||||
snapshot and migration of it and can make them much larger.
|
||||
|
||||
## `/var/lib/swarm-controller/` (swarm-controller host only)
|
||||
|
||||
Only present on the one host running
|
||||
`services.hyperhive.deploy.swarm-controller.enable`. systemd `StateDirectory=`,
|
||||
so it survives restarts and redeploys.
|
||||
|
||||
- `webhook-secret` — the HMAC key Forgejo signs the swarm's forge webhooks
|
||||
with, as it was before the swarm secret store held it. **Keep it.** The
|
||||
controller stores its value in the swarm secret store at
|
||||
`swarm/controller/swarm-controller/forge/webhook-secret` the first time it
|
||||
starts with that path empty, and reads the store from then on. The file is the
|
||||
fallback while the store is unreachable or its policy doesn't grant the path
|
||||
yet, and nothing else reads or removes it. A host with neither the file nor a
|
||||
store value gets a new secret in the store, with nothing to configure.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
If the store can't answer and the file is unreadable or missing at startup, the daemon still starts and logs
|
||||
`webhook secret unavailable`; the webhook endpoint then answers 503
|
||||
rather than accepting deliveries it can't verify. Everything else the
|
||||
controller serves is unaffected. The daemon keeps reading the store in the
|
||||
background, at most five minutes apart, and the endpoint and the forge hook
|
||||
registration start working when it answers. It writes a new secret only when
|
||||
the store answers that it holds none, never because a read failed.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
- `operator-matrix-admins.json` — the matrix user ids the operator admin
|
||||
sync has made homeserver admin, a JSON array. The sync revokes admin
|
||||
only from these, so deleting the file leaves every current admin in
|
||||
place; a missing file is an empty set. A file the controller can't
|
||||
read or parse stops the sync, grants included, and the controller logs
|
||||
`operator matrix admin: pass failed` on every pass. See
|
||||
[`docs/2-handbook/swarm/matrix.md`](../swarm/matrix.md#operators-are-homeserver-admins).
|
||||
|
||||
## Run-time dirs
|
||||
|
||||
`/run/hyperhive/` is tmpfs-backed (systemd `RuntimeDirectory=`) but
|
||||
preserved across hive-c0re restarts via `RuntimeDirectoryPreserve=yes`.
|
||||
Without that, every restart wipes bind sources and existing
|
||||
containers can't start.
|
||||
|
||||
- `/run/hyperhive/host.sock` — admin socket (host-side CLI).
|
||||
- `/run/hyperhive/agents/<name>/mcp.sock` — per-agent socket
|
||||
(bind-mounted into the container as `/run/hive/mcp.sock`).
|
||||
- `/run/hyperhive/agents/<name>/memory-max-bytes` — the agent's effective
|
||||
`MemoryMax=` as a byte count (a percentage resolved against the host's
|
||||
physical memory) or `infinity`, written by `lifecycle::write_dropins`
|
||||
before every start. Absent only when hive-c0re can't tell the cap. The
|
||||
container reads it at boot as `/run/hive/memory-max-bytes`
|
||||
(`nix/agent-modules/memory-cap.nix`).
|
||||
|
||||
On startup, `Coordinator::register_agent` drops any prior socket
|
||||
task before rebinding — idempotent so a hive-c0re restart followed
|
||||
by `rebuild alice` recreates the agent's socket without a clean
|
||||
reinstall.
|
||||
|
||||
## First-boot agent-user migration
|
||||
|
||||
The harness runs as a per-agent unix user inside the container
|
||||
(`services.hyperhive.agent.user.name`, defaults to the agent's logical label so each
|
||||
container has a uniquely named user). The activation script below
|
||||
moves root-owned state to the agent user so the claude session survives.
|
||||
|
||||
`system.activationScripts.hive-agent-user-migrate` (in
|
||||
`nix/agent-modules/user.nix`) runs on every activation,
|
||||
marker-guarded so the substantive moves only happen once per
|
||||
container lifetime:
|
||||
|
||||
1. **`${homeDir}` exists with the right ownership** — covers the
|
||||
the first boot before `useradd`'s `createHome` has had a
|
||||
chance to chown. Also re-applies on every rebuild in case the
|
||||
meta-flake's per-agent name evolves (rare).
|
||||
2. **Migrate any leftover `/root/.claude` content into
|
||||
`${homeDir}/.claude`.** Marker
|
||||
(`/var/lib/hive-agent-user-migrated`) guards single-shot; it's
|
||||
only written once there's nothing left to migrate, so a `cp`
|
||||
failure leaves it absent and the next boot retries. The root is
|
||||
ephemeral, so the marker vanishes at every stop and the step
|
||||
re-runs; with no `/root/.claude` it only writes the marker again.
|
||||
`cp -an` (no-clobber) so any pre-existing files at the new
|
||||
location win — never blow over data already there, and so a
|
||||
retry after a partial copy is as safe as the first attempt.
|
||||
3. **Chown the bind-mounted state dir** (`/agents/*/state`)
|
||||
recursively so the agent user can read/write it. Wildcard
|
||||
matches the single agent that container sees; `-h` skips
|
||||
symlinks the agent might have planted.
|
||||
4. **Chown the `~/.claude/` bind-mount** recursively. The harness reads
|
||||
`~/.claude/` as the agent user to decide Online vs NeedsLogin in
|
||||
`login::has_session`. Without the chown, credentials owned by
|
||||
another uid get silently treated as "no session" and the operator
|
||||
re-prompts every boot.
|
||||
5. **Hand the socket dir `/run/hive-agent/<name>` to the agent
|
||||
user**, `0751`, not recursive. hive-priv creates it `0751 root`
|
||||
on the host before every start; the harness binds its sockets
|
||||
there as the agent user. Why the mode matters:
|
||||
[`boundary-mechanism.md`](../../4-implementation/trust-boundary/boundary-mechanism.md#the-per-agent-socket-dir).
|
||||
A namespaced agent has the dir bound at `/agents/<name>/sockets` and
|
||||
linked from `/run/hive-agent/<name>`; the activation chowns nothing, because
|
||||
a mount point under `/run` would belong to root and the id mapping would
|
||||
give the agent user no ids there. A later in-container activation sees the
|
||||
link and logs `is a symlink or not a directory; not handing it`. The log
|
||||
is harmless.
|
||||
|
||||
## Matrix per-agent daemon + token-arrival trigger
|
||||
|
||||
`hive-matrix-daemon` is a long-running matrix-sdk Client + sync
|
||||
process per agent. Serves its MCP tools directly over
|
||||
streamable-http (`services.hyperhive.agent.mcp.matrixHttpPort`, no stdio bridge —
|
||||
same shape as `hive-bash-daemon`), emits hyperhive wake signals
|
||||
on incoming room events via `/run/hive/mcp.sock`. Conditional on the
|
||||
agent having at least one `services.hyperhive.agent.matrixAccounts`
|
||||
entry — there is no separate enable switch, and the same condition gates
|
||||
the daemon, its path watcher AND the autoinjected
|
||||
`extraMcpServers.matrix` entry. The module declares the hive-internal
|
||||
`main` entry whenever `services.hyperhive.agent.matrix.url` is non-null,
|
||||
so on a real hive (where hive-c0re renders that URL per agent) every
|
||||
agent has one; a `null` URL with no operator-declared account is the
|
||||
"this agent has no matrix" state.
|
||||
|
||||
**First-boot ordering**: a token can arrive after the container comes
|
||||
up — a declared token file, or a token the swarm mints into the store.
|
||||
Without the path-trigger sibling
|
||||
(`systemd.paths.hive-matrix-daemon`, `PathExistsGlob =
|
||||
<this agent's state dir>/matrix-token*` — the trailing `*` also catches
|
||||
a declared secondary token file like `matrix-token-ccc`), the daemon
|
||||
would exit 0 quietly the first time it ran and the MCP would have no
|
||||
daemon until the next restart. The `.path` unit makes the appearance
|
||||
of the token re-fire the service so the daemon comes alive in the
|
||||
same boot cycle as
|
||||
provisioning. A token in the store changes no file, so a store-backed agent
|
||||
also gets a timer that restarts the daemon five minutes after it last
|
||||
exited. A running daemon re-lists its linked accounts in the store
|
||||
every two minutes and, when the set changed, exits with status
|
||||
75, which the unit's `RestartForceExitStatus` restarts. On the same
|
||||
two-minute tick it tries again each secondary account that didn't come
|
||||
up — a declared account with no token yet, a token the homeserver
|
||||
rejected, or a homeserver that
|
||||
was down — waiting two minutes after its first failure and twice as long
|
||||
after each further one, up to sixteen minutes. It tries one account at a
|
||||
time and counts a try that hasn't finished after two minutes as a failure,
|
||||
so one hung account doesn't hold the others back. One that comes up joins
|
||||
the running daemon; the primary and the other accounts aren't
|
||||
restarted. The same token watcher also drives avatar setting: on a
|
||||
restart the daemon re-runs each account's bring-up, which sets the
|
||||
avatar (see below).
|
||||
|
||||
### matrix avatar (set by the daemon over the live Client)
|
||||
|
||||
`hive-matrix-daemon` itself publishes the agent icon (`services.hyperhive.agent.icon`, an SVG) as each matrix
|
||||
account's profile avatar
|
||||
(`hive-matrix-mcp::client::sync_avatar`), not a separate oneshot. After
|
||||
the daemon builds + restores an account's `Client` (authenticated,
|
||||
pointed at that account's resolved homeserver), it calls matrix-sdk's
|
||||
`account().upload_avatar()` — one call that uploads the media and sets
|
||||
`avatar_url`. Because it reuses the live Client, there is no hardcoded
|
||||
homeserver URL, no token re-read, and no token-file globbing: the daemon
|
||||
already iterates every configured + dashboard-discovered account in its
|
||||
bring-up loop, so it sets the avatar for **every** account.
|
||||
|
||||
Nix rasterizes the SVG to a 512x512 PNG at build time (`iconPng`, via
|
||||
librsvg) and forwards its store path as `HIVE_ICON_PNG` on the daemon
|
||||
unit, gated on `services.hyperhive.agent.icon != null`. No icon configured → the env is
|
||||
unset → `sync_avatar` returns early and sets no avatar.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
Idempotency is **per-account**: an `avatar-icon-hash` file in each
|
||||
account's matrix-sdk `state_dir`. The daemon hashes the PNG bytes and
|
||||
skips the upload when unchanged, because every upload mints a fresh
|
||||
`mxc://` URI that emits a profile state event in every joined room —
|
||||
re-uploading identical bytes is timeline spam. A swarm-UI-linked
|
||||
account gets its avatar when the daemon restarts on a changed account
|
||||
listing (exit 75, which re-runs the per-account bring-up), so no separate
|
||||
avatar trigger is needed. The daemon swallows avatar failures
|
||||
(logged, non-fatal) so they never break account bring-up or sync.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
178
docs/2-handbook/agents/security.md
Normal file
178
docs/2-handbook/agents/security.md
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
# Security model
|
||||
|
||||
## Agent trust model
|
||||
|
||||
The implementation docs document the specific mechanisms (nixbld isolation,
|
||||
privilege separation; see the list at the end). This section frames the model they
|
||||
serve: **what hyperhive defends, what it deliberately doesn't, and where the
|
||||
operator is accepting risk.** It's the reference for "is it safe to give an
|
||||
agent capability X?."
|
||||
|
||||
### The trust boundary is the container, not credential storage
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
An agent is **trusted code running inside its own nspawn container**. The
|
||||
boundary that matters is the container: a sub-agent can't see the host
|
||||
netns, another agent's container, or another agent's state dir. Within its
|
||||
own container the agent is privileged — it has **passwordless `sudo` by
|
||||
default**. Isolating credentials _from the agent itself_ is therefore **not a
|
||||
goal**: an agent can read its own tokens, its own `/home/<name>/.claude`, and
|
||||
run arbitrary commands as root inside its container. (The narrow exception is
|
||||
_cross-tenant_ leakage — for example the unsandboxed-nix-build `0600` token policy
|
||||
(see [nix builds](../../4-implementation/trust-boundary/nix-builds.md)) stops a build's nixbld user reading the agent's own forge token. That hardens
|
||||
the boundary; it doesn't sandbox the agent from itself.)
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
The corollary: **don't reason about security as "can the agent be stopped from
|
||||
touching its credentials." Reason about it as "what's the blast radius if this
|
||||
agent does the worst possible thing with everything it can reach."**
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
### Scoped tokens bound the blast radius
|
||||
|
||||
Each agent gets its own scoped credentials, never shared:
|
||||
|
||||
- **forge token** → that agent's Forgejo account only (its own repos +
|
||||
collaborator grants; can't act as another agent or as `core`).
|
||||
- **matrix token** → that agent's matrix account only.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
_Its own_ account's scope bounds a compromised/confused agent's reach on the
|
||||
forge or matrix, not the swarm's. This is the main thing standing
|
||||
between "one agent does something unwise" and "the whole hive is affected."
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
**Identity vs. secret (matrix).** The scoping is on the _secret_, not the
|
||||
_identity_: an agent's matrix **token** is private to its own account, but its
|
||||
matrix **identities** — the public handles (`name`, `user_id` `@user:server`,
|
||||
`homeserver`) — are intentionally readable by any agent via `GetAgentMeta`, so
|
||||
peers can find and address one another on a shared matrix instance. Only the
|
||||
public handle crosses that boundary; the token never does.
|
||||
|
||||
### The swarm secret store isn't a boundary between hives
|
||||
|
||||
A hive reads its agents' credentials out of the swarm's secret store with its
|
||||
own certificate. **Every hive's policy grants read on every agent's
|
||||
credentials**, not only on the agents it hosts — so a compromised hive can read
|
||||
the matrix token of an agent running on a different hive. That includes each
|
||||
account's key-backup recovery passphrase, which with the token decrypts the
|
||||
agent's whole encrypted history, not only messages from then on.
|
||||
|
||||
That's deliberate and it's the interim state, not the intent. An agent's
|
||||
credential path doesn't name the hive hosting it (agents move between hives),
|
||||
so a per-hive grant has to be an enumeration the controller re-emits whenever
|
||||
the roster changes — and an enumeration that can drift or land out of order
|
||||
advertises a boundary it doesn't actually hold. A wide grant that says what it
|
||||
is beats a narrow one that only looks narrow.
|
||||
|
||||
What still holds: the grant is **read-only** (a hive can't write an agent's
|
||||
credential, so it can't hand itself an agent's identity), and it reaches two
|
||||
prefixes and nothing else in the store — every agent's credentials, and the
|
||||
reader's _own_ entry under the hive namespace, which names the hive asking and
|
||||
so widens nothing between them.
|
||||
|
||||
A hive reads no **swarm service's** OIDC client secret. Every reader of
|
||||
`swarm/services/` presents a certificate that isn't a hive's: Grafana, the swarm
|
||||
collector and the store forwarder each fetch theirs with a leaf of their own,
|
||||
and the store's granter fetches the bao UI's.
|
||||
|
||||
Three services read **every agent's queue secret**
|
||||
(`swarm/agents/<agent>/queue`), each under a leaf of its own whose policy
|
||||
reaches that one leaf per agent: the queue's auth callout (`swarm-nats-auth`),
|
||||
the log store's agent check (`swarm-logs-auth`, on the log store's host), and
|
||||
the metrics store's (`swarm-metrics-agent-auth`, under the `swarm-metrics-auth`
|
||||
leaf on the metrics store's host). Any one compromised can present any agent's
|
||||
queue secret, at the queue and at both stores' read routes alike. The two store checks also read the
|
||||
agent's `swarm/agents/<agent>/observability` leaf, which only the controller writes, and let
|
||||
the agent through only with the `logs` or `metrics` permission it names.
|
||||
|
||||
### Per-hive credentials live under the hive kind
|
||||
|
||||
`swarm_secret_client::policy` scopes the **hive** stanza narrowly (unlike the
|
||||
agent stanza above) because a hive's own path names it, so scoping to the
|
||||
reader's own name costs nothing and drifts nowhere. That only holds if a
|
||||
credential that must be one-per-hive is actually stored under the hive kind.
|
||||
**A credential that must be one-per-hive goes under `Kind::Hive`**; putting it
|
||||
under the agent prefix and expecting the grant to scope it — that's the mistake
|
||||
this note exists to stop.
|
||||
|
||||
### Threat model: prompt injection → confused deputy
|
||||
|
||||
The realistic adversary **never needs to breach the container**. They supply
|
||||
**untrusted input the agent reads and acts on**: a poisoned issue or PR
|
||||
comment, a cloned repo's README/CI, a scraped webpage, a crafted matrix
|
||||
message. The agent is the trusted, capable party; the _input_ is the
|
||||
untrusted part. A successful injection turns the agent into a **confused
|
||||
deputy** — it uses its legitimate capabilities (push, comment, deploy, run
|
||||
shell) on the attacker's behalf.
|
||||
|
||||
Mitigations are therefore about **bounding capability and inserting human
|
||||
checkpoints**, not about sandboxing the agent from its own tools:
|
||||
|
||||
- **Operator merges, not the agent** — an agent may _push_ branches, but a
|
||||
**human (the operator) merges the PR**, keeping a person in the loop on the
|
||||
highest-value action. On the **internal forge this is technically enforced,
|
||||
not just convention**: agents can't create repos (`max_repo_creation = 0`),
|
||||
and `main` on an `agent-configs/<name>` repo carries swarm-controller's
|
||||
branch protection: merge and approval allowlisted to the `operators` team,
|
||||
which swarm-controller converges on every config repo. An operator's merge
|
||||
there is also what deploys the config. Existing `agents/<repo>` repos carry
|
||||
the same merge gate.
|
||||
An agent (a write collaborator, not a repo admin) can neither change those
|
||||
settings nor merge its own PR. It's **not** set up for external VCS (GitHub
|
||||
etc.), though — there, operator-merge is process + accepted risk, not a
|
||||
technical control.
|
||||
- **Operator sign-off** — config changes are config PRs an operator merges
|
||||
on the forge (see
|
||||
[`config-changes.md`](config-changes.md)), and
|
||||
scheduled prompts are operator-only, set at swarm level
|
||||
(see [`schedules.md`](../swarm/schedules.md)).
|
||||
Before that merge, the swarm builder evaluates an unmerged config PR
|
||||
off-hive, on any host that sets `deploy.builder.enable` (the
|
||||
all-swarm-services switch defaults it on): it builds the
|
||||
PR's head with a read-only forge token, in its own netns, as an untrusted
|
||||
client of its host's nix daemon (see [`builder.md`](../../4-implementation/swarm/builder.md)).
|
||||
|
||||
### Capability = accepted risk
|
||||
|
||||
Every capability granted to an agent is a risk the operator is **explicitly
|
||||
accepting**. The rule of thumb:
|
||||
|
||||
> **Don't give an agent access to something you can't afford to lose.**
|
||||
|
||||
If an agent can deploy to prod, you are accepting the risk of a dropped
|
||||
production database (via injection or plain error). If that's unacceptable,
|
||||
the answer is _don't grant the capability_ — not "grant it and hope the
|
||||
sandbox holds," because there is no sandbox between an agent and the tools you
|
||||
handed it.
|
||||
|
||||
### No autosandboxing of external tokens
|
||||
|
||||
hyperhive provisions and scopes its **own** per-agent forge + matrix tokens.
|
||||
It does **not** automatically sandbox or scope **external** credentials
|
||||
(GitHub PATs, cloud keys, third-party API tokens). The scope of an external
|
||||
token is **operator-accepted risk**: if you drop a broadly scoped GitHub token
|
||||
into an agent's config, that agent has exactly that reach, with no hyperhive
|
||||
layer narrowing it. Scope external tokens tightly at the source (the external
|
||||
provider) before handing them over.
|
||||
|
||||
## Agent-supplied names in paths
|
||||
|
||||
Wherever an agent-supplied name reaches a filesystem path, hyperhive validates
|
||||
it first. The agent socket's `GetAgentMeta` takes `name` as a serde-validated
|
||||
`hive_types::Ident` (or falls back to `Ident::parse` for the "self" case)
|
||||
before building `agent_notes_dir(name)`, so a `..` component can't traverse.
|
||||
|
||||
## Implementation detail
|
||||
|
||||
- How in-container `nix build` stays apart from the credentials, and its
|
||||
threat model: [nix builds](../../4-implementation/trust-boundary/nix-builds.md).
|
||||
- How `hive-c0re` runs unprivileged and delegates root operations:
|
||||
[privilege separation](../../4-implementation/trust-boundary/hive-c0re-privilege.md).
|
||||
423
docs/2-handbook/agents/web-page.md
Normal file
423
docs/2-handbook/agents/web-page.md
Normal file
|
|
@ -0,0 +1,423 @@
|
|||
# Per-agent page
|
||||
|
||||
> Part of [Agents](README.md). See also:
|
||||
> [Shape (shared)](../../4-implementation/web-ui/shape.md) · [Dashboard layout](../hives/dashboard.md)
|
||||
|
||||
|
||||
Two fixed-position layers frame a full-viewport terminal: a header and
|
||||
scrollable main content — plus a slide-in side panel for flyouts and long
|
||||
content. The page has no message box: to write to an agent, use the swarm
|
||||
UI's terminal.
|
||||
|
||||
## Header
|
||||
|
||||
Preact component tree (`Header.tsx` + `StatusChips.tsx` +
|
||||
`MetaNav.tsx` + `HeaderPill.tsx`, wired together in `Root.tsx`) — see
|
||||
`frontend/packages/agent/src/components/`. This section describes the
|
||||
rendered component tree: every element is Preact component output.
|
||||
|
||||
**Fixed-overlay header** (`<header class="agent-header">`): frosted
|
||||
glass — `backdrop-filter: blur` lets scrolled terminal rows show
|
||||
through. Measures its own rendered height via `ResizeObserver`
|
||||
(`Header.tsx`) and writes it to a CSS custom property the content
|
||||
below reads for its offset, so content never overlaps the header
|
||||
regardless of how many rows of badges/pills wrap onto an extra line
|
||||
at a given viewport width. Two columns:
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
- **Agent icon** (`<img class="agent-icon">`): fixed-size square
|
||||
identity anchor — `width: 5em; height: 5em` with explicit pixel
|
||||
sizing so the `<img>`'s intrinsic (large) dimensions don't push
|
||||
the parent flex container open via `align-items: stretch`-driven
|
||||
height feedback. Falls back to the dimmed hyperhive mark
|
||||
(`/favicon.svg`) on load error (`/icon` 404s when the agent has no
|
||||
`services.hyperhive.agent.icon` override — there is no server-side default image).
|
||||
- **Main column** (`.agent-header-main`, `Header.tsx`): two rows —
|
||||
title (`◆ <label> ◆`) and, when set, the "swarm / hive" identity
|
||||
line. Nothing else lives here; both rows are short, fixed-shape text
|
||||
that doesn't wrap in practice, so this column's height stays stable
|
||||
regardless of how many badges or pills are showing (a design
|
||||
correction mid-review — the first pass put the variable-width
|
||||
content here instead, which is exactly the kind of thing the
|
||||
`ResizeObserver` measurement above exists to catch even when a
|
||||
layout choice reintroduces it).
|
||||
- **Pills cluster** (`.agent-header-pills`, right-aligned): status
|
||||
badges (`StatusChips.tsx`) and flyout triggers together in one row,
|
||||
so the header reads as one identity zone + one status/actions zone:
|
||||
- **Alive badge**: `● alive` (green) / `⊘ rate limited` (red) /
|
||||
`◌ needs login` / `◌ logging in` / `○ offline` / `… connecting`.
|
||||
- **State badge**: `💤 idle` / `🧠 thinking` / `📦 compacting` /
|
||||
`○ offline` / `… booting` + age suffix.
|
||||
- **Model and effort badges** (`model · <name>`, `effort · <level>`):
|
||||
display only. Each tooltip says where the value comes from
|
||||
(`state.model_source` / `state.effort_source`: set on the swarm,
|
||||
agent.nix, an existing `hyperhive-model` / `hyperhive-effort` file, or
|
||||
the built-in default); the model's also says what the last turn
|
||||
resolved it to. Both are set on the swarm: the swarm UI's agents page,
|
||||
or `swarmctl agent inference set`
|
||||
([`inference.md`](../../4-implementation/swarm/inference.md)).
|
||||
- On an ACP agent each badge shows the session's own value when it
|
||||
doesn't offer the requested one. The effort badge is hidden
|
||||
(`state.effort` is `null`) until the first turn has attached a
|
||||
session, when the model offers no effort levels, and while a newly
|
||||
set model waits for the next turn, since the levels come with the
|
||||
model.
|
||||
- **Ctx / cost badges**: `ctx · 142k` (last inference's prompt size,
|
||||
tooltip shows % of context window) / `cost · 1.3M` (cumulative
|
||||
tokens billed across every inference in the last turn).
|
||||
- **Pause badge** (`⏸ pause` / `▶ resume`): toggles via the same
|
||||
hive-c0re-owned `/api/pause/<name>` / `/api/resume/<name>`
|
||||
endpoints the dashboard uses (pausing needs a write this
|
||||
unprivileged process can't make directly).
|
||||
- **Inbox / todos pills** (`📬 inbox · N` / `📋 todos · N`): hidden
|
||||
when empty; selecting it opens the matching flyout in the side panel.
|
||||
- **Profile avatar**: the signed-in user's initial, opening a popover with
|
||||
their name and links to authelia's settings and sign-out pages, as on the
|
||||
dashboard (`AgentUserMenu.tsx`). It shows only behind the gateway, where
|
||||
the page shares the dashboard's origin and session; opened on the agent's
|
||||
own port there is no sign-in and no avatar. The name comes from the
|
||||
dashboard's `/api/whoami`, authelia's URL from the dashboard's
|
||||
`/api/state`.
|
||||
- **Links badge** (`🔗`): opens a popover listing `↑ dashboard`
|
||||
(back-link to the host dashboard, `${dashboardBase}`)
|
||||
followed by this agent's `agent_links()`-sourced navigation (stats,
|
||||
screen when VNC is enabled, forge profile + config-repo mirror when
|
||||
the agent has a forge account, any `services.hyperhive.agent.dashboardLinks`
|
||||
extras) — all as real `<a>` elements, not a `Dropdown`-style
|
||||
command list, so ctrl/middle-click and "copy link address" keep
|
||||
working. Each `AgentLink.kind` resolves differently: `container` →
|
||||
same-origin path; `forge` → `state.forge_public_url + url`, and
|
||||
hive-agent drops the link entirely when that's unset (never guessed from
|
||||
`<host>:3000`); `external` → already absolute. Same source
|
||||
(`GET /api/state`'s `links` field) also feeds
|
||||
`DashboardState.links` for the dashboard card's icon strip —
|
||||
`agent_links()` in hive-agent is the single source of truth for
|
||||
both. The dashboard link lives in this 🔗 popover. `cancel turn`,
|
||||
`new session` and `log out` are rows of the state badge's menu, each
|
||||
with a click-again confirm (see Actions below).
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
Values throughout come from `GET /api/state`'s cold-load snapshot,
|
||||
kept in sync afterwards by the SSE stream (see Live view below) —
|
||||
`context_window_tokens` for the ctx badge tooltip, `qualified_label`
|
||||
(the hive-qualified `name@domain` form, used for the browser tab title
|
||||
so two tabs from different hives are distinguishable; the header's own
|
||||
`◆ <label> ◆` stays short).
|
||||
|
||||
## Main content
|
||||
|
||||
**Main content** (`<main class="agent-main">`): fills the viewport
|
||||
and scrolls behind the fixed header + footer.
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
- `#status` overlay: empty when online; shows the login form / OAuth
|
||||
URL when `status` is `needs_login_*`. The OAuth code input is
|
||||
`type="password"` with a `👁 reveal` toggle that flips it back to
|
||||
`text` on press so the operator can sanity-check the paste before
|
||||
submit — avoids accidental on-screen token exposure to
|
||||
shoulder-surfers or screenshots. `autocomplete="one-time-code"`
|
||||
is the semantic value for OAuth codes (per WHATWG): browsers may
|
||||
silently ignore `autocomplete="off"` on `type="password"`, but
|
||||
browsers honour `one-time-code` and suppress the "save password
|
||||
for this site?" prompt that would otherwise fire on submit.
|
||||
- Terminal-wrap: live event tail (sticky-bottom autoscroll +
|
||||
`↓ N new` pill when not at bottom). The pill is **anchored in
|
||||
`.agent-main`**, not in `log.parentElement = .terminal-wrap`:
|
||||
`.terminal-wrap` applies `backdrop-filter: blur` for the frost
|
||||
effect, which creates a CSS stacking context — anchoring the
|
||||
pill inside that context would confine its `z-index` below the
|
||||
fixed header in the root stacking context, and it'd never
|
||||
float. `.agent-main` has no backdrop-filter (no stacking-context
|
||||
creators), so the pill's `z-index` reaches the root and properly
|
||||
composites above the header. Geometry is unchanged —
|
||||
`.agent-main` and `.terminal-wrap` both `inset: 0` fill the same
|
||||
area.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
## Side panel
|
||||
|
||||
**Side panel** (slide-in from right): singleton shared with the
|
||||
dashboard's side panel shape. Carries inbox and todos flyouts (opened
|
||||
via the header pills) as well as long content (file previews, diffs,
|
||||
journald logs). Inbox flyout: unread messages addressed to this agent
|
||||
(`acked_at IS NULL`, newest-first, up to 30); reply messages indented
|
||||
with `↳ reply ·` in amber. A `✓ mark all read` button appears in the
|
||||
flyout header when the inbox is non-empty; selecting it confirms then
|
||||
POSTs cross-origin to the core dashboard's `POST
|
||||
/api/agent/{name}/mark-all-read`, which acks all pending messages for
|
||||
this agent so the harness won't receive wake-prompts for them. A `{
|
||||
marked: N }` pill surfaces the count. After the drain the inbox list
|
||||
empties on reload (the filter is `acked_at IS NULL`, so drained
|
||||
messages disappear). Todos flyout ("loose-ends v2"): the harness-local
|
||||
todos other subsystems push at this agent (`GET /api/todos`) — matrix,
|
||||
forge, and bash are the built-in producers, but any user-configured MCP
|
||||
server can push its own via the same in-agent socket. Each row shows the
|
||||
producing subsystem, an
|
||||
optional source label, a summary, and age. A checkbox per row plus a
|
||||
select-all / select-none / `✓ mark done` bulk row above the list POSTs
|
||||
the checked ids to `POST /api/todos/mark-done`, which dismisses them
|
||||
from the harness-local store (same effect as `cancel_loose_end(kind:
|
||||
"todo")`, just from the web UI instead of the agent's own tool calls).
|
||||
If the todo store errors or doesn't answer for any id, the endpoint returns
|
||||
500 naming how many ids failed, and the panel shows `failed: …` instead of
|
||||
`✓ marked done`.
|
||||
|
||||
## Live view
|
||||
|
||||
Each agent runs an `events::Bus`: a `tokio::sync::broadcast<LiveEvent>`
|
||||
plus a sqlite-backed history at `/state/hyperhive-events.sqlite`.
|
||||
The harness emits `TurnStart { from, body, unread }`,
|
||||
`Stream(value)` (one per parsed stream-json line), `Note`,
|
||||
`TurnEnd { ok, note }`. Each event also carries a `ts` (unix
|
||||
seconds) — a flattened sibling of the event tag on both the live
|
||||
SSE frame and the replayed history rows. The web UI:
|
||||
|
||||
- fetches `GET /events/history` on page load and replays the last
|
||||
2000 events (oldest first);
|
||||
- then subscribes to `GET /events/stream` (SSE) for live tail;
|
||||
- shows a granular state badge above the terminal, driven
|
||||
authoritatively from `/api/state.turn_state`. SSE turn_start /
|
||||
turn_end still flip the badge instantly between renders;
|
||||
- terminal-themed: phosphor mauve glow, Crust bg,
|
||||
backdrop-filter blur, row fade-in slide-up.
|
||||
|
||||
The backfill/live-tail dedupe, sticky-bottom autoscroll, and "↓ N
|
||||
new" pill are the shared terminal-pane mechanics described in
|
||||
[Shape](../../4-implementation/web-ui/shape.md#shared-terminal-pane) — this page's log is one
|
||||
instance of that same factory.
|
||||
|
||||
Per-stream rendering (see [`docs/4-implementation/web-ui/terminal-rendering.md`](../../4-implementation/web-ui/terminal-rendering.md) for
|
||||
the full row taxonomy and dispatch logic):
|
||||
|
||||
- `Stream` `tool_use` →
|
||||
- `Write` / `Edit`: collapsed `<details>` with a +/- diff body
|
||||
(`-` lines from `input.old_string`, `+` lines from
|
||||
`input.new_string` or every line of `input.content`).
|
||||
Summary carries the path + line counts.
|
||||
- others (`Read /path`, `Bash $ cmd`, `mcp__hyperhive__send →
|
||||
operator: "..."`, etc.): flat one-line per-tool format with
|
||||
per-tool salient-arg extraction (`fmtToolUse`).
|
||||
- `Stream` `tool_result` short → flat `← ...`; long → collapsed
|
||||
`<details>` `▸ ← Nl · headline` (select to expand full body).
|
||||
- `Stream` `thinking` → `.thinking` row with a `💭 thinking …`
|
||||
indicator.
|
||||
- `Stream` `system` → handled by subtype: `plugin_install` and
|
||||
`compact_boundary` emit muted notes; `commands_changed` emits an
|
||||
expandable details row listing slash commands; `thinking_tokens`
|
||||
updates a single in-place `🧠` counter; `init`, `result`, and
|
||||
`rate_limit_event` drop silently (noise / used elsewhere); other
|
||||
subtypes → muted `⚙ <subtype>` note.
|
||||
- `Note` → `· text`.
|
||||
- `TurnStart` → `◆ TURN ← <from>` with the wake-prompt body; a
|
||||
muted `· HH:MM:SS` time suffix from the event `ts`.
|
||||
- `TurnEnd` → `✓ turn ok` / `✗ turn fail — note`, with a
|
||||
`· HH:MM:SS · <duration>` suffix (duration = end − the paired
|
||||
turn-start), triggers a `refreshState()`. The time suffix is
|
||||
guarded on a numeric `ts`, so rows degrade cleanly when absent.
|
||||
|
||||
## Actions
|
||||
|
||||
The state badge's menu carries the actions on the agent's claude:
|
||||
|
||||
- `cancel turn` — `POST /api/cancel` → SIGINT to the turn's claude and every
|
||||
command it started (its process group), emits a Note. On an ACP agent
|
||||
it sends the agent `session/cancel` instead, and SIGINT if the turn
|
||||
hasn't ended 10s later. Offered while the state is `thinking`.
|
||||
- `new session` — `POST /api/new-session` (confirms first).
|
||||
Arms a one-shot on the Bus; next turn runs without
|
||||
`--continue`, dropping the resume session entirely.
|
||||
- `log out` — `POST /api/logout` (confirms first). Wipes OAuth
|
||||
credential files, parks the agent in `needs_login`, and preserves
|
||||
session history (`~/.claude/projects/`).
|
||||
|
||||
`POST /api/compact` sets the deferred `Bus::request_compact()` flag and
|
||||
returns immediately. The harness runs the `/compact` at the next turn
|
||||
boundary (end of the in-flight turn, or `turn::run_pending_compact` when
|
||||
idle); output streams into the live panel, which ends with a `/compact done`
|
||||
or `/compact failed: …` note. It works mid-turn, not only when idle. Nothing
|
||||
on this page calls it now.
|
||||
|
||||
## Per-agent endpoints
|
||||
|
||||
Successful POSTs return 200 (no 303 redirects). Error responses
|
||||
use semantic status codes: **400** for missing/malformed input
|
||||
(`body` required, unknown model name, unrecognised effort level),
|
||||
**409** for retryable state conflicts (turn in flight when
|
||||
`/compact` fires), **500** only for genuine server/transport
|
||||
failures. The matching
|
||||
mutations fire `LiveEvent` variants on the per-agent bus, so the
|
||||
client doesn't refetch `/api/state` on submit — the SSE stream
|
||||
delivers the new state faster anyway.
|
||||
|
||||
The agent has no login flow of its own. An agent without credentials
|
||||
reports `needs_login_idle`, the operator logs in from the swarm UI, and the
|
||||
credentials arrive from the swarm secret store.
|
||||
|
||||
- `POST /api/cancel` — SIGINT the in-flight claude turn and the
|
||||
commands it started. Emits a `LiveEvent::Note`.
|
||||
- `POST /api/compact` — run `/compact` on the persistent session
|
||||
(same MCP config + system prompt + allowed tools as a normal
|
||||
turn — only the stdin payload differs). Flips state to
|
||||
`Compacting` via `Bus::set_state`, which emits
|
||||
`TurnStateChanged`.
|
||||
- `POST /api/new-session` — arm a one-shot for the next turn to
|
||||
drop `--continue`. Emits a `LiveEvent::Note`.
|
||||
- `POST /api/logout` — three-step teardown that re-uses the
|
||||
existing `wait_for_login` resumption path:
|
||||
1. SIGINT any running claude (matches `/api/cancel`'s pattern —
|
||||
idempotent no-op when nothing is running) so the credential
|
||||
wipe doesn't race a mid-API-call turn.
|
||||
2. Delete only the OAuth credential files
|
||||
(`.credentials.json` + `mcp-needs-auth-cache.json` under
|
||||
`~/.claude/`). **Preserves** session history files
|
||||
(`projects/<hash>/*.jsonl`), sessions, shell-snapshots,
|
||||
plans, settings, telemetry, and the dir itself, so
|
||||
`claude --continue` keeps working after a fresh login.
|
||||
⚠️ Deleting the whole `~/.claude/` directory instead breaks
|
||||
session continuity — logout must stay narrowed to just the
|
||||
credential files.
|
||||
3. Flip `LoginState::NeedsLogin` + emit a `LiveEvent::Note`
|
||||
naming the files it just deleted, then emit
|
||||
`needs_login_idle`. The turn-loop's next iteration parks
|
||||
into `wait_for_login`, which snapshots the credential dir
|
||||
(now missing the wiped files) and resumes when a fresh
|
||||
credentials file appears (a login from the swarm UI stores one,
|
||||
and the agent installs it).
|
||||
|
||||
Always returns 200 with a body describing what happened — it
|
||||
folds per-file errors into the response + the Note so the
|
||||
operator sees them in the live panel rather than as an HTTP
|
||||
error. Missing files (already logged out) count as
|
||||
idempotent.
|
||||
- `GET /api/state` — cold-load snapshot (`StateSnapshot`) consumed by
|
||||
the frontend (`useAgentState.ts`) on page load. Includes `turn_state`,
|
||||
`context_window_tokens`, `qualified_label`,
|
||||
`model`, `model_source`, `effort`, `effort_source`, `links`, and
|
||||
other fields described inline throughout this document. All
|
||||
subsequent state updates arrive via SSE.
|
||||
- `GET /api/dashboard-state` — lean snapshot of agent-owned fields
|
||||
fetched once per running agent by the dashboard's container row to
|
||||
get fresh values without relying on hive-c0re's periodic file-reads.
|
||||
Returns `{ status_text?, status_set_at?, ctx_tokens?, context_window_tokens,
|
||||
rate_limited, links }`. Only called when the container is running;
|
||||
skipped (muted badges) when stopped. Also accessible via the gateway
|
||||
at `/agent/<name>/api/dashboard-state`.
|
||||
- `GET /api/todos` — harness-local todos snapshot (loose-ends v2)
|
||||
consumed by the todos flyout (`refreshTodos` / `buildTodosList`).
|
||||
See the todos-flyout paragraph above for the producer/subsystem
|
||||
model and the bulk mark-done row.
|
||||
- `GET /api/stats?window=24h|7d|30d|all` — time-bucketed turn analytics
|
||||
`Snapshot` consumed by the `/stats` page. `all` ranges from the
|
||||
earliest recorded turn with an adaptive bucket width.
|
||||
- `GET /icon` — agent's icon as `image/svg+xml`. Returns
|
||||
`/etc/hyperhive/icon.svg` (set via `services.hyperhive.agent.icon` in `agent.nix`)
|
||||
when present, otherwise **404** — there is no server-side default.
|
||||
Consumers (dashboard container row, this page's own header icon)
|
||||
hit `/icon` optimistically and fall back client-side on load failure
|
||||
to the frontend-bundled `/favicon.svg` rather than probing first.
|
||||
- `GET /events/history` — replay buffer for the terminal.
|
||||
- `GET /screen` — VNC viewer page (minimal RFB-over-WebSocket
|
||||
renderer — deliberately thin, just enough to display the
|
||||
desktop + forward pointer + keyboard. A production-grade viewer
|
||||
would vendor noVNC; this file ships the minimal in-tree variant).
|
||||
Only accessible when `services.hyperhive.agent.gui.enable = true` in the agent's
|
||||
`agent.nix`; the harness shows a 🖥 screen link in the state row
|
||||
when `gui_vnc_port` is present. A `← agent` back-link in the page
|
||||
header returns to the main per-agent page. Controls: `⤢ fit` CSS-downscales
|
||||
the canvas to the window via `relayoutCanvas()` setting explicit
|
||||
pixel dimensions on the canvas — *not* CSS `max-width/max-height`,
|
||||
because a flex item's automatic minimum size (`min-width: auto`
|
||||
resolves to the canvas's intrinsic framebuffer resolution) silently
|
||||
clamps `max-*` back up, making fit mode a no-op that just centred +
|
||||
clipped the oversized canvas. The fit-mode rules pin the canvas
|
||||
with `flex: none; min-width: 0; min-height: 0` so the JS-set size
|
||||
sticks. `⤡ match size` sends an RFB `SetDesktopSize` request so the
|
||||
server (weston) changes its real output resolution to the window
|
||||
dimensions; enabled once the server advertises the
|
||||
`ExtendedDesktopSize` pseudo-encoding (`-308` rect in the header).
|
||||
Fit-mode state persists in `localStorage` (`screen-fit`); default
|
||||
is on. `sendPointer` rescales pointer coordinates so presses
|
||||
land on the right pixel regardless of CSS scale.
|
||||
- `GET /screen/ws` — raw RFB byte relay: proxies WebSocket
|
||||
frames to the weston VNC server at `127.0.0.1:<vnc_port>`.
|
||||
Transparent to any RFB variant. VNC port comes from the
|
||||
`HIVE_GUI_VNC_PORT` env var (a fixed port set on the harness
|
||||
service when `services.hyperhive.agent.gui.enable`; see `weston-vnc.nix`).
|
||||
- `GET|POST /extra/<name>/…` — **extra web proxies** declared via
|
||||
`services.hyperhive.agent.extraWebProxies` in `agent.nix` (serialised to the
|
||||
`HIVE_EXTRA_WEB_PROXIES` env var as a JSON object
|
||||
`{"<name>": "<upstream_url>"}`). Each entry mounts a transparent
|
||||
reverse-proxy at `/extra/<name>/` that forwards every request (method,
|
||||
headers, body) to the configured upstream, strips hop-by-hop headers
|
||||
on both sides (repeated response headers such as two `Set-Cookie` lines
|
||||
are all forwarded), and buffers the full response body (MVP — SSE
|
||||
connections will appear as one large response rather than streaming).
|
||||
The `/extra/` namespace ensures user-declared proxies can never
|
||||
conflict with native agent endpoints. Upstream values are either an
|
||||
`http(s)://` URL (forwarded via `reqwest`) or a Unix domain socket,
|
||||
spelled `unix:<path>` (for example `unix:/run/myapp/http.sock`) — dialed
|
||||
directly with a raw HTTP/1.1 client per request, since `reqwest` has
|
||||
no UDS transport. Implemented in `web_ui/proxy.rs`.
|
||||
|
||||
Bus events (new vocabulary on `/events/stream`):
|
||||
|
||||
- `status_changed { status }` — `online` / `rate_limited` /
|
||||
`needs_login_idle`. Drives the
|
||||
alive-badge. The harness sets `rate_limited` when it detects a
|
||||
429 response and clears it when the retry sleep expires.
|
||||
- `model_changed { model }` — drives the model chip. The harness emits it
|
||||
when it applies or clears a swarm value.
|
||||
- `effort_changed { effort }` — drives the effort chip. The harness
|
||||
emits it when it applies or clears a swarm value.
|
||||
- `token_usage_changed { ctx: TokenUsage, cost: TokenUsage }`
|
||||
— drives the ctx + cost badges. Emitted from
|
||||
`Bus::record_turn_usage` at turn-end; `ctx` is the last
|
||||
inference's usage (current context size), `cost` is the
|
||||
cumulative across every inference (the `result` event's
|
||||
totals).
|
||||
- `turn_state_changed { state, since_unix }` — drives the
|
||||
state badge (`idle`/`thinking`/`compacting`).
|
||||
|
||||
## Stats page
|
||||
|
||||
`GET /stats` is a separate per-agent page (served by the
|
||||
harness, linked from the per-agent page's `📊 stats →` and from
|
||||
each dashboard container row). A `← live` back-link in the page
|
||||
header returns to the main per-agent page. Turn analytics, read-only, from
|
||||
`/state/hyperhive-turn-stats.sqlite`. `GET /api/stats?window=
|
||||
24h|7d|30d|all` returns a time-bucketed `Snapshot`; the page renders
|
||||
it with Chart.js (bundled into `stats.js` via esbuild — no CDN
|
||||
dependency). Charts: turns, duration (p50 · p95 · avg), context
|
||||
tokens, token cost per bucket, a **turns-by-model** stacked bar
|
||||
(model choice drives token cost, so it sits directly under the
|
||||
cost chart), doughnuts for tool / wake-source / result mix, and a
|
||||
**result-trend** stacked bar — per-bucket `result_counts` so
|
||||
error / rate-limit / compaction outcomes are visible over time
|
||||
(the doughnut shows only the window total).
|
||||
A **favorite tools** doughnut shows the most-run shell commands —
|
||||
normalised `bash_commands` heads written per bash task by the
|
||||
hive-bash-daemon capture: the basename of the *first real command*,
|
||||
looking past `cd repo &&` prefixes, env-assignments, and
|
||||
prefix-runners like `sudo` / `env` (so `cd /repo && cargo build`
|
||||
records `cargo`, not `cd`). Read via `bash_breakdown`; the card
|
||||
stays hidden until the agent has run a bash command (a missing
|
||||
`bash_commands` table degrades to an empty list), so it never
|
||||
renders an empty chart.
|
||||
A summary chip row carries window totals, plus two
|
||||
token-efficiency chips derived from the bucket sums: **cache
|
||||
hit-rate** (`cache_read` over all input-side tokens) and
|
||||
**tokens/turn**. When `reminder_stats` is present (fetched via
|
||||
`ReminderRollup` RPC and merged into the snapshot in
|
||||
`web_ui/stats.rs::api_stats`) three more chips appear: **reminders
|
||||
scheduled / delivered / pending** for the window. When the
|
||||
per-session capture has data, a **first-turn ctx** chip shows the
|
||||
input tokens of the most recent fresh claude session's first turn —
|
||||
a proxy for system-prompt + AGENTS.md sprawl (a fresh session's first
|
||||
turn pays the full static prefix uncached). It's derived in `stats.rs`
|
||||
(`first_turn_ctx`: the first turn, `ORDER BY started_at LIMIT 1`, of
|
||||
the latest `sessions` row in the window) and stays out of the JSON
|
||||
until the `sessions` / `turn_stats.session_id` capture has rows — so
|
||||
the chip stays hidden on older dbs (inert-until-capture).
|
||||
`stats.rs` opens the sqlite db read-only and degrades to an
|
||||
empty snapshot on any error — the page is decorative, never
|
||||
authoritative.
|
||||
12
docs/2-handbook/external-accounts/README.md
Normal file
12
docs/2-handbook/external-accounts/README.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# External accounts
|
||||
|
||||
An agent can hold an account on a service outside the swarm, so it can push
|
||||
code or read notifications there as itself. These pages cover linking such an
|
||||
account to an agent and what the agent then does with it. They sit outside the
|
||||
swarm, the hives and the agents themselves; the swarm's own forge and
|
||||
homeserver are in [`swarm/`](../swarm/README.md).
|
||||
|
||||
- **[GitHub](github.md)** — give an agent a GitHub account: `gh`,
|
||||
`git push` and the notification poller, and how the token reaches it.
|
||||
- **[GitLab](gitlab.md)** — the same for one or more GitLab instances
|
||||
(`glab` and `git push`).
|
||||
153
docs/2-handbook/external-accounts/github.md
Normal file
153
docs/2-handbook/external-accounts/github.md
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
# GitHub accounts
|
||||
|
||||
Give an agent a managed GitHub identity — a `gh` CLI and `git push` over
|
||||
HTTPS, both authenticated by an operator-supplied personal access token
|
||||
(PAT) — so it can run GitHub API calls and push commits without any manual
|
||||
`gh auth login`.
|
||||
|
||||
Provisioning is UI-driven: link a PAT to the agent in the swarm UI and it
|
||||
works. No per-agent nix declaration, no rebuild — the agent fetches the token
|
||||
from the swarm secret store itself.
|
||||
|
||||
## Enabling
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
The integration is **on by default** for every agent (`services.hyperhive.agent.github.enable
|
||||
= true`), inert until the operator provisions a PAT. No per-agent declaration is
|
||||
needed — an agent gains GitHub by having a PAT stored for it.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
To turn it off for the whole swarm, set the swarm option:
|
||||
|
||||
```nix
|
||||
services.hyperhive.swarm.github.enable = false;
|
||||
```
|
||||
|
||||
hive-c0re's meta-flake renderer then injects `services.hyperhive.agent.github.enable = false`
|
||||
into every agent, so no agent ships the `gh` wrapper, the credential helper
|
||||
or the token fetch.
|
||||
(`services.hyperhive.agent.github.enable` also exists per-agent for completeness, but the
|
||||
swarm-wide switch is the intended control.)
|
||||
|
||||
github.com only. The token **value** never touches nix — the agent writes it
|
||||
to `<state>/github-token` at runtime (see [Provisioning](#provisioning)).
|
||||
|
||||
## Provisioning
|
||||
|
||||
The PAT is operator-supplied. In the [swarm UI](../swarm/ui.md#linking-external-accounts),
|
||||
open the agent on `/agents`, choose **link github account** and paste the PAT
|
||||
(`PUT /api/hives/{hive}/agents/{agent}/github-account`). swarm-controller
|
||||
stores it at `swarm/agents/<agent>/github-token` in the swarm secret store;
|
||||
no hive writes it. One token per agent: swarm-controller refuses to link
|
||||
another until you delete the stored one, and no route hands it back.
|
||||
|
||||
The agent's `hive-agent-github-token` unit reads that path under the
|
||||
agent's own store certificate and writes `<state>/github-token` (`0600`,
|
||||
owned by the agent), on boot and every two minutes, replacing the file only
|
||||
when the token changed. It needs a store identity
|
||||
(`services.hyperhive.agent.bao.addr`); an agent without one gets no token.
|
||||
The unit never deletes the file: a `github-token` already in place stays
|
||||
when the store holds none or doesn't answer. Where the token lives and who
|
||||
reads it: [credentials-reference.md](../swarm/credentials-reference.md).
|
||||
|
||||
## Security
|
||||
|
||||
- Use a **dedicated bot account**, never a human's.
|
||||
- Mint a **minimally scoped PAT** — only the repos/scopes the agent's
|
||||
workflow needs. Agents have passwordless sudo, so a compromised or
|
||||
hallucinating agent can act as the account within the token's scopes;
|
||||
scope is the real blast-radius limiter, and the container boundary is
|
||||
the enforcement. See [security.md](../agents/security.md).
|
||||
|
||||
## Implementation
|
||||
|
||||
<details>
|
||||
<summary>How the agent's `gh`/`git push` actually authenticate, and how
|
||||
the notification poller works</summary>
|
||||
|
||||
### How the agent uses it
|
||||
|
||||
When enabled, the container gets:
|
||||
|
||||
- **A `gh` wrapper** on `PATH` (shadowing the raw `gh`) that exports
|
||||
`GH_TOKEN` from the token file at invocation, then execs real `gh`. So
|
||||
`gh pr create`, `gh api …`, etc. just work — `gh` derives the identity
|
||||
from the token.
|
||||
- **A git credential helper** (`git-credential-hive-github`), wired via a
|
||||
host-scoped `/etc/gitconfig` entry for `https://github.com`, so
|
||||
`git push https://github.com/<owner>/<repo>` authenticates as
|
||||
`x-access-token` + the PAT (GitHub ignores the username for PAT auth).
|
||||
Host-scoped, so it never touches the forge (`localhost:3000`) or any
|
||||
other remote.
|
||||
|
||||
Both scripts read the token from `<state>/github-token` **at invocation
|
||||
time**, so a PAT written (or rotated) mid-session takes effect immediately
|
||||
— no container rebuild or restart. Until the file exists, `gh` / `git push`
|
||||
fail unauthenticated. The build bakes the token path into the scripts
|
||||
(rather than reading it from an env var), because claude's Bash tool runs in a
|
||||
minimal environment that wouldn't carry one.
|
||||
|
||||
### Notifications
|
||||
|
||||
`hive-github-notify` polls github.com for the agent, turning each unread
|
||||
notification thread into a todo. it's a **separate binary and a
|
||||
separate systemd unit** from the internal forge's poller
|
||||
(`hive-forge-notify`, see [forge.md](../swarm/forge.md#notification-poller-hive-forge-notifysrcnotifyrs)),
|
||||
installed by `nix/agent-modules/github.nix` under
|
||||
`services.hyperhive.agent.github.enable`. Both binaries ship from the one
|
||||
`hive-forge-notify` derivation, so the unit is a second `ExecStart`
|
||||
path, not a new package.
|
||||
|
||||
Two units rather than one daemon with two loops, because it puts the
|
||||
decision in nix: a hive built without this module has **no github poller
|
||||
in its closure at all**, which is what makes GitHub access separable
|
||||
rather than merely switched off. it's also why this isn't a cargo
|
||||
feature — a feature would unify across the workspace and cost every
|
||||
crate its build cache.
|
||||
|
||||
At runtime the poller needs the PAT above. No PAT, no polling: the unit
|
||||
logs why and exits 0, which is why it's `Restart = on-failure` and
|
||||
never `always` — a clean exit on a PAT-less agent must not become a
|
||||
restart loop. The poller waits about ten minutes for the PAT to appear
|
||||
before it gives up. A path unit (`hive-github-notify.path`) starts the poller
|
||||
again when `<state>/github-token` changes, so a PAT that lands later still
|
||||
enables polling. Once polling, it re-reads the file on every
|
||||
tick, so it picks up a rotated PAT on the next tick without a restart.
|
||||
|
||||
Forgejo models its notifications API on GitHub's, so one tolerant
|
||||
parse serves both: `id`, `repository.full_name`,
|
||||
`subject {title,url,latest_comment_url}` and `updated_at` line up field
|
||||
for field. Lenient deserializers absorb the two real differences —
|
||||
GitHub sends the thread id as a _string_ where Forgejo
|
||||
sends a number, and says `PullRequest` where Forgejo says `Pull`.
|
||||
`hive-github-notify` prefixes todo keys `gh:<id>` so a github thread id can't collide with a
|
||||
forge one.
|
||||
|
||||
Two host differences worth knowing before touching this code:
|
||||
|
||||
- **Auth scheme, not just value.** Forgejo takes
|
||||
`Authorization: token <t>`; GitHub wants `Bearer <t>` plus `Accept:
|
||||
application/vnd.github+json`, `X-GitHub-Api-Version` and a
|
||||
`User-Agent`. Sending Forgejo's form to GitHub doesn't error — it
|
||||
authenticates as _nobody_ and silently drops to the unauthenticated
|
||||
rate limit. The cheap way to tell the two apart is the rate-limit
|
||||
header: `x-ratelimit-remaining` near 5000 is an authenticated user,
|
||||
near 60 is anonymous.
|
||||
- **GitHub sets the cadence.** It returns `X-Poll-Interval` (60s in
|
||||
practice, slower than the poller's own tick) and rate-limits callers who
|
||||
ignore it, so the loop re-arms to the server's interval whenever that's
|
||||
_slower_ than the poller's. A hint faster than the poller's own tick
|
||||
isn't a reason to poll harder.
|
||||
|
||||
⚠️ **This needs the `notifications` scope on the PAT.** A token minted
|
||||
for `gh` + `git push` typically carries `repo` only, which is enough to
|
||||
push and open PRs but **not** to read the notification stream (nor to
|
||||
mark a thread read, which is the same scope). A PAT without it doesn't
|
||||
break anything: the poller logs the refusal and stays quiet, and the
|
||||
agent never gets GitHub wakes. If an agent's GitHub
|
||||
notifications never arrive, check the token's scopes first — the
|
||||
symptom is silence, not an error.
|
||||
|
||||
</details>
|
||||
91
docs/2-handbook/external-accounts/gitlab.md
Normal file
91
docs/2-handbook/external-accounts/gitlab.md
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
# GitLab accounts
|
||||
|
||||
Give an agent managed accounts on one or more GitLab instances: a `glab` CLI
|
||||
and `git push` over HTTPS, both authenticated by operator-supplied access
|
||||
tokens, with no manual `glab auth login`.
|
||||
|
||||
An agent can hold any number of accounts, each under a label you choose. Two labels
|
||||
can name two instances, such as `gitlab.com` and a self-hosted one. Provisioning
|
||||
needs no per-agent nix declaration and no rebuild: the agent fetches its tokens
|
||||
from the swarm secret store itself.
|
||||
|
||||
## Linking an account
|
||||
|
||||
In the swarm UI, open an agent, choose **link account**, then **gitlab
|
||||
account**. Fill in a label, the instance URL (it starts as `https://gitlab.com`)
|
||||
and the token. The form shows the recommended scopes and the controller's
|
||||
answer if it refuses the input. The linked account appears in the agent's list
|
||||
of [linked accounts](../swarm/ui.md#linking-external-accounts) with a delete
|
||||
action. The UI calls the controller's API, which you can also call directly:
|
||||
|
||||
```text
|
||||
PUT /api/hives/{hive}/agents/{agent}/gitlab-accounts/{label}
|
||||
{"url": "https://gitlab.example.org", "token": "<access token>"}
|
||||
```
|
||||
|
||||
- `label` is lowercase letters, digits and hyphens. It names the account in the
|
||||
store and in the account list.
|
||||
- `url` is the instance's base URL. The host must be a DNS name: an IP
|
||||
address, `localhost` and a single-label name get a 400, because the agent
|
||||
reaches the instance through its hive's egress proxy, which takes names
|
||||
only. A port and a subfolder (`https://example.org:8443/gitlab`) are part of
|
||||
the address. The path may hold letters, digits and `. _ ~ / -`; a URL with
|
||||
user information, a query, a fragment or any other character gets a 400, so
|
||||
every stored account is one the agent uses.
|
||||
- A plain `http://` URL works, and then the token crosses the network in
|
||||
cleartext. Use `https://` unless the instance sits on a trusted network.
|
||||
- `token` is a GitLab personal, group or project access token. Scope it
|
||||
`api` and `write_repository`: `glab` needs the first, and `git push` over
|
||||
HTTPS needs the second. Use a dedicated bot account and the narrowest token
|
||||
that does the agent's work, because the agent has passwordless sudo and can
|
||||
act within the token's scopes.
|
||||
|
||||
A label that holds an account already answers 409; delete it first
|
||||
(`DELETE .../gitlab-accounts/{label}`), then link again. No route hands a
|
||||
token back. `GET /api/hives/{hive}/agents/{agent}/linked-accounts` lists the
|
||||
account as kind `gitlab`, with its label and URL.
|
||||
|
||||
Linking also allows the instance's host in the agent's egress, as a
|
||||
`link:gitlab:<label>` entry on the swarm's allowed-domains list. swarm-controller
|
||||
derives the host from the URL it stored, so nothing the agent writes can widen
|
||||
its egress. Deleting the account removes the entry.
|
||||
|
||||
## What the agent gets
|
||||
|
||||
An agent with a store identity (`services.hyperhive.agent.bao.addr`) has:
|
||||
|
||||
- the `glab` command, which reads its configuration from `<state>/glab/`;
|
||||
- a git credential helper, `git-credential-hive-gitlab`, for each linked
|
||||
instance;
|
||||
- the `hive-agent-gitlab-accounts` unit and its timer.
|
||||
|
||||
The unit logs in to the store with the agent's own certificate, lists
|
||||
`swarm/agents/<agent>/gitlab/`, reads each account, and writes two files in
|
||||
`<state>/glab/` (mode `0600`) on boot and every two minutes, replacing a file
|
||||
only when its content changed:
|
||||
|
||||
- `config.yml`, `glab`'s own configuration with one `hosts` entry per instance,
|
||||
keyed by host and port, holding the token, the API protocol and the
|
||||
subfolder. It sets `git_protocol: https`. The `host` default is the first
|
||||
label's instance; inside a repository `glab` uses the host of its remote.
|
||||
- `gitconfig`, one `[credential "<scheme>://<host>"]` block per instance, which
|
||||
`/etc/gitconfig` includes. The helper answers only for a host that has an
|
||||
account, so it never touches GitHub or the swarm's forge.
|
||||
|
||||
The unit generates both files. A deleted account leaves them on the next run, and
|
||||
a hand edit doesn't survive. `glab` keeps one token per host: when two labels
|
||||
name the same host, the first label in alphabetical order wins. If the unit
|
||||
can't read one of the accounts, it replaces nothing and retries, so the files
|
||||
keep the last complete set.
|
||||
|
||||
The unit fails fast and restarts itself every 30 seconds for up to a day, as
|
||||
the other store fetches do, so a store that's down or sealed never holds the
|
||||
container's boot open.
|
||||
|
||||
## Rolling out
|
||||
|
||||
The endpoint, the store path and the agent unit are new, and each end ignores
|
||||
what it doesn't know. An old controller has no GitLab route, so no account gets
|
||||
linked. An old agent has no unit, so a stored account sits unused. A new agent
|
||||
under an old controller lists a store directory that doesn't exist and writes an
|
||||
empty `glab` configuration.
|
||||
113
docs/2-handbook/hives/README.md
Normal file
113
docs/2-handbook/hives/README.md
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
# Hives
|
||||
|
||||
A **hive** is one host running `hive-c0re` and its agent containers. This
|
||||
section is for the operator of that host: the per-hive dashboard, the
|
||||
`hivectl` CLI and the operator/agent trust boundary on the host. For what
|
||||
the whole swarm shares, see [the swarm UI](../swarm/ui.md) instead.
|
||||
|
||||
- **[Dashboard](dashboard.md)** — every tab and standalone page of the
|
||||
per-hive dashboard, in full detail: endpoint shapes, event wiring, badge
|
||||
rules.
|
||||
- **[hivectl](hivectl.md)** — the curated guide to the host CLI:
|
||||
provisioning forge and matrix accounts, container lifecycle shortcuts,
|
||||
interactive agent shell access.
|
||||
- **[hivectl-cli](hivectl-cli.md)** — the flag-by-flag reference,
|
||||
generated from the binary and kept in lockstep by CI.
|
||||
- **[The operator/agent boundary](boundary.md)** — what the operator may do
|
||||
that an agent may not, and why the line sits where it does.
|
||||
|
||||
For `swarmctl`, the swarm operator's CLI, see
|
||||
[`swarmctl-cli.md`](../swarm/swarmctl-cli.md). The tool surface agents get
|
||||
inside their containers is in
|
||||
[`4-implementation/agent-tools/`](../../4-implementation/agent-tools/README.md).
|
||||
|
||||
## Where things are
|
||||
|
||||
Everything starts at the **dashboard**, served at `/`. A pages menu (the
|
||||
link icon beside the profile avatar, at the right of the tab strip) links to
|
||||
the other surfaces: the API docs and the swarm UI.
|
||||
|
||||
The dashboard is a single page with these tabs:
|
||||
|
||||
- **4G3NTS** — every agent, live. This is the default tab and the one you
|
||||
check most.
|
||||
- **R3BU1LD QU3U3** — the rebuild queue. A build's output and history are on
|
||||
the swarm's `builds` Grafana board.
|
||||
- **K3PT ST4T3** — kept state from agents with no container (tombstones),
|
||||
with a purge button per entry.
|
||||
|
||||
Each agent also has its own page, a full terminal view of that agent's
|
||||
session. Select its name anywhere in the dashboard, or open `/agent/<name>/`
|
||||
(or `http://<host>:<port>/` if the gateway isn't in front). [The agent's web page](../agents/web-page.md)
|
||||
covers it.
|
||||
|
||||
## The things you do here
|
||||
|
||||
**Check on an agent.** 4G3NTS shows every container as a row: name, whether
|
||||
it's running, what it's doing (a live status pill such as `rebuilding…` or
|
||||
`starting…` while something's in flight), and quick links (stats, screen,
|
||||
forge profile). Select the name to open its terminal and watch it work.
|
||||
|
||||
**Review a config change.** Agent config changes (new packages, env vars, MCP
|
||||
servers) are pull requests on the agent's `agent-configs` repo. Review and
|
||||
merge them on the forge; the merge deploys the change. See
|
||||
[`config-changes.md`](../agents/config-changes.md#config-changes).
|
||||
|
||||
**Start, stop, restart or rebuild an agent.** Select one or more agents on
|
||||
4G3NTS (select the icon) and use the selection bar, or use the per-agent `⋮`
|
||||
menu on a single row. Rebuilding downloads and applies the closure the swarm
|
||||
declares for the agent again. The hive applies a changed declaration itself,
|
||||
so use it to retry one that failed.
|
||||
|
||||
**Watch a build.** The R3BU1LD QU3U3 tab shows the rebuild queue live, plus a
|
||||
streaming log of whatever's building. Useful right after merging a config
|
||||
change or bumping a flake input.
|
||||
|
||||
**Grant or revoke a permission.** Not on the hive dashboard: an agent's
|
||||
permissions live in the swarm. Set them with `swarmctl agent permissions`, or
|
||||
on the swarm UI's create-agent screen when creating the agent; see
|
||||
[`5-project/process/conventions.md`](../../5-project/process/conventions.md#permissions).
|
||||
|
||||
**Read an agent's logs.** Logs aren't in the hive dashboard. Every hive ships
|
||||
its agents', daemons' and infra containers' journals to the swarm's
|
||||
VictoriaLogs; read them in Grafana's **Explore**, as
|
||||
[`swarm/services.md`](../swarm/services.md#logs-victorialogs) says.
|
||||
|
||||
**Set up a schedule.** Scheduled prompts are swarm-level: set them with
|
||||
`swarmctl schedule` ([`swarm/schedules.md`](../swarm/schedules.md)) or in the
|
||||
schedules card on the swarm UI's messages tab ([`swarm/ui.md`](../swarm/ui.md)).
|
||||
|
||||
## More depth
|
||||
|
||||
The dashboard and the per-agent pages are single-page apps that share one
|
||||
skeleton: `GET /` returns a static shell, `/api/state` returns JSON, and
|
||||
JavaScript renders, with no full-page reloads.
|
||||
|
||||
- **[Shape (shared by both)](../../4-implementation/web-ui/shape.md)** — the
|
||||
SPA skeleton, SSE multiplexing, terminal pane, listener bind, per-agent
|
||||
relative paths, the `data-async` form pattern and atomic repaint.
|
||||
- **[CSS theme variables](../../4-implementation/web-ui/css-vars.md)** — the
|
||||
Catppuccin Mocha custom properties declared once in `base.css`, and the
|
||||
rule that per-page stylesheets reference them and never redeclare them.
|
||||
- **[swarm-ui design guide](../../5-project/web-ui/design-guide.md)** — the
|
||||
visual language, motion, theming, error UX and component-first principles
|
||||
for the swarm-level Preact app. `/components` on a running swarm UI is the
|
||||
living demo of every building block it names.
|
||||
|
||||
For the code, these jump straight to the relevant section:
|
||||
|
||||
- **"How does the dashboard SPA stay live without polling?"** →
|
||||
[`shape.md`](../../4-implementation/web-ui/shape.md) (SSE multiplexing,
|
||||
Worker-death self-heal, atomic repaint).
|
||||
- **"What does a container row contain?"** → [`dashboard.md`](dashboard.md)
|
||||
(Container row, Selection bar).
|
||||
- **"What endpoints does the dashboard expose?"** →
|
||||
[`dashboard.md`](dashboard.md) (Dashboard endpoints, Dashboard event
|
||||
channel).
|
||||
- **"How does the per-agent terminal render tool calls?"** →
|
||||
[`terminal-rendering.md`](../../4-implementation/web-ui/terminal-rendering.md)
|
||||
(the full row taxonomy and dispatch walkthrough); for a summary see
|
||||
[`web-page.md`](../agents/web-page.md) (Per-stream rendering).
|
||||
- **"Which CSS variable to use, and where are the colours defined?"** →
|
||||
[`css-vars.md`](../../4-implementation/web-ui/css-vars.md) (Palette, the
|
||||
single-source `base.css` rule).
|
||||
109
docs/2-handbook/hives/boundary.md
Normal file
109
docs/2-handbook/hives/boundary.md
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
# The operator/agent boundary
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
||||
Design rationale for hyperhive's two-principal trust model. The
|
||||
_implementation_ work — container network isolation, the unifying
|
||||
gateway, core-daemon privsep — is tracked as `area:ops` issues on
|
||||
the forge.
|
||||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
The operator/agent boundary is technically enforced, not just a
|
||||
convention: containers run in private netns (network isolation is
|
||||
always on), the gateway proxies all operator-facing traffic, and
|
||||
`hive-c0re` runs as the unprivileged `hive-core` user.
|
||||
|
||||
## Two principals, two paths
|
||||
|
||||
- **Operator** — reaches every UI (the dashboard + every
|
||||
per-agent page) through the gateway, on one origin.
|
||||
The core daemon serves operator-authority actions
|
||||
(answer-as-operator, lifecycle POSTs), reachable only via the
|
||||
gateway.
|
||||
- **Agent** — speaks only for itself, only over its per-agent
|
||||
unix socket. The socket's identity _is_ the agent (see
|
||||
`docs/5-project/process/conventions.md`, "identity = socket"). An agent must not
|
||||
be able to reach the core daemon's HTTP surface, another
|
||||
agent's socket, or another agent's web UI.
|
||||
|
||||
## Design rule
|
||||
|
||||
**Operator-authority actions never get a per-agent-socket entry
|
||||
point.** They live in hive-c0re.
|
||||
|
||||
Worked example — destroying or rebuilding a container is a
|
||||
`POST /api/{destroy,rebuild}/{name}` on the core dashboard, _never_ a
|
||||
per-agent-socket `Request` variant. If it were a per-agent-socket
|
||||
request, a compromised agent could `curl` its own socket and destroy
|
||||
or rebuild itself (or, if the variant took an arbitrary target, another
|
||||
agent) without ever touching the core's own authenticated surface.
|
||||
|
||||
## Why network isolation is the load-bearing step
|
||||
|
||||
Without network isolation, containers share the host network namespace
|
||||
and can reach `localhost:<core-port>`, the dashboard, and every other
|
||||
agent's web port — the operator/agent split is on the honour system and
|
||||
every boundary claim above is aspirational. Network isolation is what
|
||||
makes the boundary _real_; the gateway and privsep are ergonomics and
|
||||
defence-in-depth layered on top.
|
||||
|
||||
Network isolation is complete and always on: every agent container
|
||||
runs in a private netns behind the hive bridge, and there is no
|
||||
shared-netns mode. See `docs/4-implementation/networking/network.md`.
|
||||
|
||||
Concretely, the core daemon's dashboard `/api` carries **no
|
||||
application-layer authentication** — the core daemon serves operator-authority
|
||||
routes unauthenticated at the HTTP layer. Their protection is entirely (a) the
|
||||
gateway, which fronts all operator traffic and is where operator auth lives,
|
||||
and (b) network isolation, which keeps agents — and the CI runner's untrusted
|
||||
PR builds — off host-loopback so nothing can reach `127.0.0.1:<dashboard_port>`
|
||||
directly. This is deliberate given the load-bearing role of network isolation
|
||||
above, but it's a standing invariant: the `/api` must never bind to a
|
||||
non-loopback address or get exposed outside the gateway, and every new
|
||||
operator-authority route inherits that assumption. hyperhive treats
|
||||
`swarm-ci-runner` like an agent for this purpose — it runs untrusted PR code and is
|
||||
netns-isolated for the same reason.
|
||||
|
||||
The boundary rests on three layers:
|
||||
|
||||
1. **Gateway** — fronts all surfaces (dashboard + every per-agent UI)
|
||||
on one origin. An nginx nixos-container proxies per-agent UIs under
|
||||
`/agent/<name>/`, which is what lets each agent page's inbox panel
|
||||
POST `mark-all-read` to the core dashboard's
|
||||
`/api/agent/{name}/mark-all-read` go same-origin instead of needing
|
||||
a cross-origin CORS shim. Pure ergonomics — no behavioural risk on
|
||||
its own.
|
||||
2. **Network isolation** — the load-bearing layer: every agent
|
||||
container runs in a private netns behind the hive bridge, always
|
||||
on and unconditional. This is what turns the operator/agent split
|
||||
from an honour-system convention into an enforced boundary.
|
||||
3. **Privsep** — defence in depth on the core process; `hive-c0re`
|
||||
runs as the unprivileged `hive-core` user and delegates root
|
||||
operations to `hive-priv`, a narrow socket-activated helper. See
|
||||
[`docs/4-implementation/trust-boundary/hive-c0re-privilege.md`](../../4-implementation/trust-boundary/hive-c0re-privilege.md) for the privilege boundary table.
|
||||
|
||||
### host admin socket access (`hivectl`)
|
||||
|
||||
`hivectl` drives the whole hive — spawn / kill / destroy / rebuild /
|
||||
deploy — over the **host admin socket** `/run/hyperhive/host.sock`,
|
||||
socket-activated by the `hive-c0re.socket` unit. That socket _is_ the
|
||||
full-control surface, so who can connect to it's a real trust
|
||||
boundary.
|
||||
|
||||
By default the socket is `0660` group-owned by **`hive-admin`**, an
|
||||
empty group — so it's effectively **root-only** until an operator is
|
||||
explicitly granted access. Grant sudoless `hivectl` by listing login
|
||||
users in `services.hyperhive.deploy.hive-controller.adminUsers`; the option adds each to
|
||||
`hive-admin`, and members connect without `sudo`. The runtime dir
|
||||
`/run/hyperhive` is `0751` (traverse-only, no listing) so the group can
|
||||
reach the socket path; the socket's own `0660 hive-admin` mode gates
|
||||
the connection, and the per-agent subdirs under it keep their own
|
||||
restrictive perms. Keep `adminUsers` to trusted operators — membership
|
||||
is equivalent to root over the hive.
|
||||
|
||||
## Implementation detail
|
||||
|
||||
How `hive-priv` is socket-activated and how the per-agent socket directory
|
||||
gets its owner and mode:
|
||||
[socket mechanics](../../4-implementation/trust-boundary/boundary-mechanism.md).
|
||||
528
docs/2-handbook/hives/dashboard.md
Normal file
528
docs/2-handbook/hives/dashboard.md
Normal file
|
|
@ -0,0 +1,528 @@
|
|||
# Dashboard layout
|
||||
|
||||
> Part of [Hives](README.md). See also:
|
||||
> [Shape (shared)](../../4-implementation/web-ui/shape.md) · [Per-agent page](../agents/web-page.md)
|
||||
|
||||
This page covers the per-hive UI that hive-c0re serves. Swarm-wide
|
||||
surfaces — the agent roster, agent creation, linking external forge and
|
||||
matrix accounts, the controller's job graph — are on the
|
||||
[swarm UI](../swarm/ui.md).
|
||||
|
||||
hive-c0re serves the dashboard at `/`.
|
||||
It has a fixed chrome header at the top and a `<main>` that shows exactly
|
||||
one tab pane at a time. The URL hash (`#agents`, `#rebuild`, `#meta`,
|
||||
`#kept`) drives which pane is
|
||||
active; hash changes don't reload the page.
|
||||
|
||||
**Chrome header** (fixed, overlays the active tab pane):
|
||||
- **Pages menu**: a link icon at the right end of the chrome, beside the
|
||||
profile avatar (`links-menu.js`, `@hive/shared`'s `LinksMenu`, the same
|
||||
component the swarm UI uses for its service links), opening a popover with links to **api docs**
|
||||
(`/api/docs/`) and **swarm** (the swarm
|
||||
UI at the swarm apex domain, opened in a new tab). The swarm entry
|
||||
comes from `state.swarm_public_url`, which hive-c0re reads from
|
||||
`HIVE_SWARM_PUBLIC_URL` (set from `services.hyperhive.swarm.domain`);
|
||||
with no domain set there is no swarm entry. The standalone pages link back to `/`.
|
||||
- **Profile menu**: the signed-in user's avatar at the right end of the
|
||||
chrome (`user-menu.js`, the same component as the swarm UI's and the agent
|
||||
page's). It opens a popover with the user's name and links to authelia's
|
||||
settings and sign-out pages. The name comes from `/api/whoami`, which the
|
||||
dashboard's nginx vhost proxies to authelia, and the links from
|
||||
`state.authelia_public_url`, which hive-c0re reads from
|
||||
`HIVE_AUTHELIA_PUBLIC_URL` (set from `services.hyperhive.swarm.authelia.url`).
|
||||
If the name doesn't load the avatar shows a question mark, and without the URL the
|
||||
popover has no links.
|
||||
- **Tab strip**: `◆ 4G3NTS ◆`, `◆ R3BU1LD QU3U3 ◆`,
|
||||
`◆ K3PT ST4T3 ◆`.
|
||||
- **Banner-thin** (`░▒▓█▓▒░ HYPERHIVE / HIVE-C0RE / WE ARE THE WIRED ░▒▓█▓▒░`)
|
||||
— sits below the tab strip.
|
||||
- **No warning banner.** No page shows hive-level warnings (host disk
|
||||
pressure, failing background sweeps, forge boot failures, agents that
|
||||
need a claude login). They live in a push-based registry
|
||||
(`hive-c0re`'s `warnings.rs`): any subsystem raises or clears its own
|
||||
entry through an RAII guard, and each raise logs one
|
||||
`hive warning raised` line. The operator sees the registry as metrics and
|
||||
Grafana alerts: see
|
||||
[Hive-scoped metrics](../../4-implementation/scheduler/observability.md#hive-scoped-metrics-hive-c0re).
|
||||
A container crash is a swarm-level Grafana alert on the crash watcher's
|
||||
own log line, so a hive with no otel export has no operator-visible
|
||||
warning signal beyond `/health/ready`.
|
||||
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
|
||||
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
|
||||
arrive in the state snapshot.
|
||||
|
||||
## 4G3NTS tab
|
||||
|
||||
**C0NTAINERS** — live containers as a flat list, one row per
|
||||
container, sorted alphabetically by agent name; every container the
|
||||
rescan reports gets a row. Pulsing red banner at the top
|
||||
of this section if any two sub-agents hash to the same port
|
||||
(`port_conflicts` from `/api/state`): the operator must rename
|
||||
one of them and rebuild. `lifecycle::{spawn,rebuild}` also
|
||||
preflight this and refuse with a clear error message naming the
|
||||
conflicting agent.
|
||||
|
||||
## K3PT ST4T3 tab
|
||||
|
||||
Passive / rare-interaction state. `kept.js` renders it from the
|
||||
`/api/state` snapshot and keeps it live from the `tombstones_changed`
|
||||
stream event. (The rebuild queue and build logs are dashboard tabs too —
|
||||
see below.)
|
||||
|
||||
Per-container CPU, memory and disk load isn't shown in the hive UI. The
|
||||
same figures go out as OTEL metrics and appear in Grafana (see
|
||||
[observability](../../4-implementation/scheduler/observability.md)). An agent's CPU and memory
|
||||
caps live per agent in the swarm UI.
|
||||
|
||||
**K3PT ST4T3** — agents with kept state and **no container** (size + age +
|
||||
claude-creds badge). One action: `PURG3` (wipes the state
|
||||
dir; `POST /api/purge-tombstone/{name}`).
|
||||
|
||||
⚠️ **Not only *destroyed* agents.** Nothing records a destroy: `Provision`
|
||||
writes every definition-side artifact (state subvolume,
|
||||
meta registration, topology entry) *before* the container exists,
|
||||
and that artifact set survives `lifecycle::destroy`. An agent
|
||||
part-way through a spawn is byte-identical on disk to a tombstone, and
|
||||
the pane lists both. The pane carries a standing warning to that effect, and
|
||||
the row badge says `offline` rather than `destroyed` — the absence of a
|
||||
container is all hive-c0re can actually prove. Fixing it properly
|
||||
needs a recorded destroy — **deferred** to the swarm-controller /
|
||||
snapshot-storage rework where the problem changes shape.
|
||||
|
||||
## Build tab (`#rebuild`)
|
||||
|
||||
The rebuild queue is the **R3BU1LD QU3U3** tab (`#rebuild`). It lives in `builds.js`, which `tabs.js` imports: the
|
||||
dashboard's one tab strip, `/api/state` snapshot and
|
||||
`/api/dashboard/stream` subscription serve them, with
|
||||
`rebuild_queue_changed` on the stream alongside the dashboard's other
|
||||
kinds.
|
||||
The 4G3NTS tab does **not** read the graph endpoint, by design — the swarm
|
||||
view stays independent of job-queue internals. Its per-agent pending
|
||||
badges are transient-only, and its queue-summary banner reads the much
|
||||
narrower `GET /api/jobq/rollup` instead (see Container row, below) — a
|
||||
handful of pre-tallied counts, not the graph. The banner's "view queue →"
|
||||
link goes to the `#rebuild` tab.
|
||||
|
||||
**R3BU1LD QU3U3** — pending, in-flight, and recently settled container
|
||||
operations: rebuilds and first-spawns. One
|
||||
operation runs at a time; the worker drains FIFO. **Is a mounted
|
||||
`JobqGraph`** (the shared generic graph-viewer Preact component,
|
||||
`@hive/shared/jobq-graph.js` — the same one swarm-ui uses) —
|
||||
`builds.js` doesn't render the queue itself; it just mounts the component
|
||||
(`mountJobqGraph(container, props)`, since this page has no JSX
|
||||
pipeline) with an `onUpdate` callback to drive the two things below it
|
||||
that the generic view doesn't show. The component owns fetching, cold
|
||||
and live: `GET /api/jobq/graph` on mount, and a refetch via the mount
|
||||
handle's `.refresh()` on every `rebuild_queue_changed` SSE tick (that
|
||||
event is a bare `{ seq }` trigger carrying no queue payload; both
|
||||
dashboard pages treat the tick as a pure refetch trigger).
|
||||
|
||||
Each row is one root graph node (`parent: null`); a multi-step op's
|
||||
per-agent subgraphs and sub-steps render as nodes within that one
|
||||
entry (structural `parent` edges define the tree; sibling order is
|
||||
array order, unchanged from the wire). A row shows a state glyph (`⏸`
|
||||
pending / `▶` running / `◐` finishing — own work done, a sub-node
|
||||
still running / `✔` done / `✖` failed / `⊘` cancelled / `·` skipped)
|
||||
and each step's own label/agent. A `Node`-kind dep on a sibling shows
|
||||
as a plain "waits on: `<label>`" text line under the row rather than a
|
||||
gutter rail — a rail breaks visually whenever a nested subtree sits
|
||||
between the two related rows, since a text line needs no continuous
|
||||
vertical space to draw.
|
||||
`builds.js` mounts the component with `cancellable` set, which turns on
|
||||
a per-node cancel button (`✕`) on any non-terminal row — the button
|
||||
calls the mount's `onCancel(id)` callback, and the page does the actual
|
||||
`POST /api/rebuild-queue/{id}/cancel`, matching that callback to its
|
||||
own domain concept (the component knows nothing about that endpoint).
|
||||
**Rows carry no source chip, kind label, timing, or build-log
|
||||
deep-link** — the generic graph wire doesn't carry those fields, and
|
||||
rows deliberately present exactly what the endpoint provides rather
|
||||
than reconstructing chrome hive-c0re doesn't send. Settled entries
|
||||
render their **full step tree**, not just a bare summary — the wire
|
||||
doesn't filter `Done` nodes out.
|
||||
|
||||
**State filter.** A row of per-state checkboxes above
|
||||
the tree — one per lifecycle state, matching the row glyphs — lets the
|
||||
operator narrow which root groups render; unchecking a state re-fetches
|
||||
`GET /api/jobq/graph?states=<checked, comma-joined>` rather than
|
||||
hiding rows client-side, so the `onUpdate` callback's node list (and
|
||||
everything downstream of it) only
|
||||
ever sees what's actually shown. Filtering is by a **root's own**
|
||||
state, which is already its subtree's rolled-up answer, so the filter
|
||||
keeps or drops a whole group, never splitting mid-tree. Default selection is
|
||||
every state **except** `Done`/`Skipped` — a fresh queue view leads
|
||||
with what's still moving or needs attention, not the settled tail;
|
||||
builds.js omits the `states` param entirely (identical request to
|
||||
before this filter existed) when the operator has checked every state. Server-side: the query
|
||||
narrows [`Queue::graph_snapshot`]'s already-bounded (`MAX_HISTORY_DAGS`)
|
||||
root set — the history cap and the state filter are independent
|
||||
concerns, so a narrow filter never reaches further back in time to
|
||||
compensate.
|
||||
|
||||
A build's output and history aren't on this page. hive-c0re logs every container build
|
||||
to the journal, and the swarm's `builds` Grafana board reads those lines: the builds with
|
||||
their outcome and runtime, and the output of the one you pick.
|
||||
|
||||
## Container row
|
||||
|
||||
A full-height **square agent icon** (5em, capped) on the left. The
|
||||
icon is the **selection toggle**: select (or Enter/Space) adds/removes
|
||||
the agent from the selection set; `aria-pressed` reflects the state;
|
||||
the tooltip says "select … for bulk actions" or "deselect … (or press
|
||||
Esc to clear all)." The `<img>` points at `<url>/icon`; load failure
|
||||
falls back to the dimmed hyperhive mark (`/favicon.svg`). The card
|
||||
body sits to the right with three stacked lines
|
||||
(`assets/agents.js::renderContainers`).
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
**Icon layout + load strategy:** the `<img>` is absolutely
|
||||
positioned (`inset: 0`) inside the `.container-icon` wrapper —
|
||||
the wrapper is the flex child and sizes itself via `width: 5em` +
|
||||
`aspect-ratio: 1`, the `<img>` is out of flow so its load state
|
||||
(pending, loaded, broken) can never contribute intrinsic size or
|
||||
reflow the row. Without that, the row would briefly grow as the
|
||||
image's natural dimensions arrived, then snap back on
|
||||
`object-fit: contain`. The load itself is fire-and-forget: the
|
||||
dashboard doesn't pre-check whether the agent is reachable, it
|
||||
just lets the `<img>` try and listens for an `error` event. On
|
||||
failure the handler swaps the `src` to `/favicon.svg` (served by
|
||||
the dashboard itself, always reachable) and adds the
|
||||
`icon-unreachable` class for the dimmed look. When the dashboard
|
||||
already knows the container is stopped (`ContainerView.running = false`)
|
||||
the fallback fires immediately, skipping the doomed `<url>/icon`
|
||||
fetch entirely.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
**Line 1** — agent name (link → new tab), m1nd/ag3nt chip, an
|
||||
**icon-only nav strip** plus live agent-owned state, all populated
|
||||
async from a single `GET /api/dashboard-state` call to the agent's
|
||||
own hive-agent. The response (`DashboardState`) carries: `links` (nav
|
||||
strip entries — `📊 stats`, `🖥 screen` when GUI is enabled, `⬡ forge
|
||||
profile`, `↳ agent-configs mirror`, plus any agent-declared
|
||||
`dashboardLinks` extras), `status_text` / `status_set_at` (agent
|
||||
self-reported status — the `(set N ago)` chip carries a `data-set-at`
|
||||
stamp and ticks every 30s to stay fresh across the long-lived keyed row
|
||||
cache), `rate_limited`, `ctx_tokens` / `context_window_tokens`
|
||||
(context-window badge data). hive-agent is the single source
|
||||
of truth for all of these. The dashboard resolves each `AgentLink.kind`
|
||||
against a per-agent base URL depending on whether hive-gateway is in
|
||||
front (`StateSnapshot.gateway_enabled`, sourced from the
|
||||
`HIVE_GATEWAY_ENABLED` env). The c0re NixOS module sets that env
|
||||
unconditionally — the gateway always runs; the flag stays so the
|
||||
dashboard doesn't have to learn that. Gateway-on: base URL is
|
||||
`/agent/<name>` (same origin, gateway proxies to the
|
||||
per-agent harness — TCP or unix-domain depending on the agent's
|
||||
`HIVE_WEB_SOCKET` opt-in, see
|
||||
`docs/4-implementation/networking/gateway.md::Per-agent unix-socket upstream`).
|
||||
Gateway-off is the flag-absent fallback (a dashboard served outside a
|
||||
hyperhive deploy): base URL is
|
||||
`http://<host>:<container.port>` (direct TCP). Forge
|
||||
links resolve against `state.forge_public_url` (sourced from
|
||||
`services.hyperhive.swarm.forge.publicUrl`) and disappear entirely when
|
||||
that's unset — never guessed from `<host>:3000`. External links are
|
||||
already absolute. The same base URL drives the primary agent-name
|
||||
link + favicon fetch, so the whole row routes through the gateway
|
||||
as a unit.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
**When the container is stopped** (`ContainerView.running = false`),
|
||||
the dashboard skips the async `dashboard-state` fetch entirely (the agent
|
||||
web server is down), so a single badge replaces the badge chain,
|
||||
the nav strip is empty, and status text / rate-limited / ctx
|
||||
badges disappear. The agent icon goes straight to the dimmed
|
||||
`/favicon.svg` fallback instead of attempting a doomed load from the
|
||||
container's URL. Static fields — `pending_reminders`, `parent`, `config` link — remain visible
|
||||
regardless of run state.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
**Line 2** — status badges only (no per-card action buttons — actions
|
||||
live in the **selection bar** or the **per-agent `⋮` menu**, see
|
||||
below).
|
||||
|
||||
### Status badges
|
||||
|
||||
**Which single badge:** `ContainerView.failed`
|
||||
(systemd `ActiveState=failed` — the unit exhausted its bounded
|
||||
restarts and gave up on its own) draws a red `✖ gave up` badge;
|
||||
otherwise a plain muted `■ not running` — a container an operator
|
||||
stopped deliberately. Both states read `running: false`; `failed` is
|
||||
the orthogonal fact (a fourth one alongside `paused`/
|
||||
`needs_login`, same "independent flags, no state machine" shape —
|
||||
see `ContainerView`'s own doc comment) that tells them apart. An
|
||||
older hive-c0re without the field serves `failed: undefined`, which
|
||||
reads falsy — degrades cleanly to the single `not running` badge.
|
||||
|
||||
When the container is running, status badges follow — `⊘ rate
|
||||
limited` (red, while the harness parks itself after a 429), `needs
|
||||
login` — plus **one `◐ pending-state…` pill per
|
||||
active transient** (replaces buttons during operator-initiated
|
||||
start / stop / restart / rebuild / destroy). An agent can carry
|
||||
**multiple transients at once** — for example a lease-exempt `prebuild`
|
||||
running alongside a `stop_for_update` on the same agent — and each
|
||||
renders as its own independent badge instead of collapsing
|
||||
into one label, matching the existing multi-badge convention this
|
||||
line already uses for `paused`/model/ctx.
|
||||
|
||||
Any pending badge means the row is actually **running** something
|
||||
right now (see **Pending-badge derivation** below); every row
|
||||
carrying ≥1 badge keeps the amber row tint AND draws a **rotating
|
||||
amber ring** around the agent icon.
|
||||
|
||||
**Pending-badge derivation:** transients only (`transientsState`,
|
||||
keyed `agent -> Map<kind, since_unix>`) — a transient is **derived
|
||||
from a job-queue node currently `Running`** against that agent, not
|
||||
declared per request, so its label follows the operation as it
|
||||
progresses (a rebuild reads `stop_for_update`, then `swap`, then
|
||||
`reconcile` rather than one constant `rebuilding` for its whole
|
||||
life). Two consequences for anything rendering it:
|
||||
|
||||
- The label vocabulary is **open** — it's the node's own wire tag
|
||||
(`NodeKind`'s `&str` conversion, the same strings the graph wire's node
|
||||
labels carry), not a fixed set. Treat it as an opaque display
|
||||
string; don't switch on specific values.
|
||||
- It's **not** exclusively operator-initiated, and **not** limited
|
||||
to rebuild-shaped work — `running_transients()` in hive-c0re is
|
||||
a status-only test (any `Running` node whose payload names a
|
||||
non-empty agent lights a pill), so work the operator never
|
||||
triggered (a crash-recover rebuild, a boot-sweep rebuild, a
|
||||
lease-exempt `prebuild`) lights the same mechanism.
|
||||
|
||||
Ops with no queue node behind them (destroy, migration) supply
|
||||
their own label directly via `TransientSet`/`TransientCleared`
|
||||
events carrying no backing node at all.
|
||||
|
||||
**Queued (not-yet-started) work shows nothing on the card.** Only
|
||||
running work gets a per-agent pending badge, by design — the swarm
|
||||
view stays independent of job-queue internals. The queue-summary
|
||||
banner below is the only queued-work indicator on this tab, and it
|
||||
reads the narrow `/api/jobq/rollup` endpoint rather than the full
|
||||
graph.
|
||||
|
||||
**Active model badge** (`model · <name>`, blue) appears when the
|
||||
container is running and the harness has persisted a model name (the
|
||||
`active_model` field of `hyperhive-harness.json`, the consolidated
|
||||
harness state file in the agent's state dir). Read by hive-c0re's
|
||||
`ContainerView` (`read_active_model`); absent until the agent has
|
||||
completed at least one turn, and hive-c0re suppresses stale values
|
||||
for stopped containers.
|
||||
|
||||
**`ctx · Nk` chip** shows the agent's last-turn context size,
|
||||
populated from `DashboardState.ctx_tokens` (absent until the
|
||||
agent has completed at least one turn). The chip colour (green /
|
||||
yellow / red) keys off `DashboardState.context_window_tokens`
|
||||
(the real context window for the model the agent last ran on,
|
||||
authoritative from the agent side); the badge goes yellow ≥ 50%
|
||||
and red ≥ 75% of that window, matching the harness compaction
|
||||
watermarks. When the window value is absent the badge falls back
|
||||
to fixed 100k / 150k thresholds.
|
||||
|
||||
**Per-agent `⋮` overflow menu** — a `⋮` button appears on the right
|
||||
edge of each container row. Selecting it opens a small dropdown with
|
||||
per-agent actions and navigation links. Contents:
|
||||
|
||||
- `↺ R3ST4RT` (running agents only) / `■ ST0P` (running only) /
|
||||
`▶ ST4RT` (stopped only) — single-agent run-state toggles.
|
||||
Identical to the bulk actions on the selection bar but operate
|
||||
on one agent without requiring a selection.
|
||||
- `↻ R3BU1LD` — always available; queues a rebuild for this agent.
|
||||
- `DESTR0Y` / `PURG3` — destructive, each prompts for confirmation
|
||||
via the themed dialog (see **Themed dialogs** below).
|
||||
|
||||
**Build-queue summary banner** — when the job queue has any active
|
||||
work, a compact amber banner sits above the container list: `◐ build
|
||||
queue — N running · M queued — view queue →` (the link goes to the
|
||||
R3BU1LD QU3U3 tab). The shared `JobqRollup` Preact component
|
||||
(`@hive/shared/jobq-rollup.js` — the same one swarm-ui's `/jobs` page
|
||||
mounts, pointed at swarm-controller's own rollup endpoint instead),
|
||||
mounted once into `#jobq-rollup-section` by `agents.js::initJobqRollup`
|
||||
and refreshed via its own handle rather than being re-rendered by
|
||||
`renderContainers`. Reads `GET /api/jobq/rollup`
|
||||
(`hive-jobq-wire::state_rollup`) — `Vec<{ state, nodes, roots }>`,
|
||||
every lifecycle state present in a fixed order, zero counts included —
|
||||
rather than the full `/api/jobq/graph` tree: `running` sums the
|
||||
`Running` and `Finishing` entries' `roots` (`Finishing` = own work
|
||||
done, subtree still going, still in flight), `queued` reads the
|
||||
`Pending` entry's `roots`. `roots` specifically, not `nodes` — the
|
||||
banner means *N whole operations*, not raw steps (one rebuild is ~7
|
||||
nodes but 1 root); `nodes` exists on the same endpoint for a consumer
|
||||
that wants step-level counts instead, unused here.
|
||||
|
||||
### Themed dialogs
|
||||
|
||||
All confirmations and transient error notices use an in-app themed
|
||||
dialog system (`@hive/shared/modal.js`) rather than the browser's
|
||||
native `confirm()` / `alert()` chrome, so they match the Catppuccin
|
||||
palette and the OS can't style them away. Two primitives, both built
|
||||
on the `openDialog` core:
|
||||
|
||||
- `themedConfirm({ message, danger, confirmLabel, checkboxes })`
|
||||
— a modal confirm that resolves to `null` on cancel or an object
|
||||
of checkbox states on confirm. Destructive actions pass
|
||||
`danger: true` (the confirm button turns red and the cancel
|
||||
button takes focus). Selecting the backdrop and `Esc` both cancel.
|
||||
- `themedToast(message, { type, ... })` — a non-blocking toast (top-right,
|
||||
`info` / `error` / `ok`) for transient validation + action
|
||||
failures, so an error doesn't lock the operator behind a modal.
|
||||
Single-action errors autodismiss; bulk / partial-failure
|
||||
summaries are sticky (select to dismiss) so they aren't missed.
|
||||
|
||||
Every destructive run-state action (`ST0P`, `R3ST4RT`, `R3BU1LD`,
|
||||
`DESTR0Y`, `PURG3`) routes through `themedConfirm`, on both
|
||||
the per-agent `⋮` menu and the bulk selection bar.
|
||||
|
||||
**Graceful stop** — the `■ ST0P` confirm dialog (per-agent and
|
||||
bulk) carries a `stop gracefully — let the agent finish its turn
|
||||
and flush state before the container stops` checkbox. When ticked,
|
||||
the action POSTs `/api/kill/<name>?graceful=true` (the bulk path
|
||||
appends the flag per-agent); unticked is the instant hard stop
|
||||
(`/api/kill/<name>` with no query). hive-c0re enqueues a
|
||||
`Signal` → `Drain` job-queue node pair (`NodeKind::Signal` /
|
||||
`NodeKind::Drain`): `Signal` sets the graceful-stop fence and kicks
|
||||
the harness so it runs one stop-checkpoint turn (so the agent can
|
||||
flush `/state`); `Drain` awaits the harness clearing that fence,
|
||||
bounded by a 3-minute timeout (`GRACEFUL_STOP_TIMEOUT`) that
|
||||
resolves either way and falls back to the downstream mechanical
|
||||
stop. The quiescing progress surfaces through the same
|
||||
rebuild-queue pending-badge mechanism the card already reads for a
|
||||
rebuild, driven by the `Signal`/`Drain` pair rather than a nix
|
||||
build. (The `hivectl stop --graceful` CLI flag enqueues the same
|
||||
`Signal`/`Drain` pair, so the dashboard and CLI paths behave
|
||||
identically.)
|
||||
|
||||
## Selection bar
|
||||
|
||||
Bulk actions (`R3ST4RT` / `ST0P` / `ST4RT` / `P4US3` / `R3SUM3` /
|
||||
`R3BU1LD` / `DESTR0Y` / `PURG3`) live here rather than as per-card
|
||||
buttons — see **Container row** above. Selecting an agent's icon
|
||||
toggles its selection (an in-memory `Set<name>`); `Esc` or the bar's
|
||||
`✕ clear` button drops everything. The selection persists across tab
|
||||
switches in-memory — the bar just hides on non-4G3NTS tabs since other
|
||||
tabs don't show the agent cards needed to cross-reference.
|
||||
|
||||
When the operator selects one or more agents (via the icon), a sticky
|
||||
frosted-mauve bar slides up from the bottom of the viewport
|
||||
(`#selection-bar`, `position: fixed; bottom: 0`). It shows:
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
- **Count + names** — "N agents selected · name1, name2, …"
|
||||
- **Bulk action buttons** — only enabled when ALL selected agents
|
||||
support the action; greyed out with a tooltip naming the blockers
|
||||
when the selection is mixed:
|
||||
- `↺ R3ST4RT` — running agents only
|
||||
- `■ ST0P` — running agents only
|
||||
- `▶ ST4RT` — stopped agents only
|
||||
- `⏸ P4US3` — unpaused agents only (orthogonal to running/stopped —
|
||||
see **Container row**'s own pause/resume note)
|
||||
- `▶ R3SUM3` — paused agents only
|
||||
- `↻ R3BU1LD` — always available
|
||||
- `DESTR0Y` / `PURG3` — always available
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
That list is the whole set — the bar offers no other bulk action.
|
||||
- **`✕ clear`** button + `Esc` key clear the entire selection.
|
||||
|
||||
Every render prunes stale selections (agents destroyed while
|
||||
selected) before the bar appears.
|
||||
|
||||
## Dashboard endpoints
|
||||
|
||||
Also browsable interactively at `/api/docs` (a Swagger UI, linked from
|
||||
the pages menu's **api docs** entry), with the raw spec at `/api/openapi.json`.
|
||||
It's a growing supplement, not yet a full replacement for the list
|
||||
below — some endpoints aren't in it yet.
|
||||
|
||||
- `POST /api/{rebuild,kill,restart,start,destroy}/{name}` — lifecycle.
|
||||
`destroy` accepts `purge=on` to also wipe state dirs.
|
||||
- `POST /api/purge-tombstone/{name}` — wipe a tombstone's state dirs.
|
||||
- `POST /api/rebuild-queue/{id}/cancel` — drop a `Queued` entry.
|
||||
Refuses `Running` / terminal-state entries (in-flight
|
||||
rebuilds can't be safely interrupted). Always 200; body is
|
||||
`{"cancelled": true}` on a successful flip or
|
||||
`{"cancelled": false}` when the entry wasn't in `Queued` state.
|
||||
- `POST /api/agent/{name}/mark-all-read` — ack all pending broker
|
||||
messages for `{name}`. Backfills `delivered_at` for rows
|
||||
not yet delivered and sets `acked_at = now`. Returns
|
||||
`{ "marked": N }`. Agent name validated against
|
||||
`[a-z0-9_-]`, 1-63 chars; 400 on bad input.
|
||||
- `GET /api/build-logs/{agent}?limit=N` — most-recent build log
|
||||
headers for one agent, newest first. Returns
|
||||
`Vec<BuildLogHeader>` (JSON): `id`, `agent`, `kind`, `cmdline`,
|
||||
`started_at`, `finished_at`, `status` (`"ok"` / `"fail"` /
|
||||
`null` while in-progress). `limit` defaults to 10, server-side
|
||||
cap at 50. Agent name validated (`[a-z0-9_-]`, 1-63 chars).
|
||||
- `GET /api/build-logs/id/{id}` — full build log by id. Returns
|
||||
`BuildLogFull` (JSON): all header fields plus `stdout` and
|
||||
`stderr` as plain text (newline-terminated lines, utf-8). HTTP
|
||||
404 when the row is missing (vacuum-reaped or stale id).
|
||||
- `GET /api/build-logs/id/{id}/stream` — live SSE stream of
|
||||
stdout+stderr for an in-progress build (newline-delimited JSON
|
||||
frames). The
|
||||
stream closes when the build finishes or errors.
|
||||
- `GET /api/build-logs/id/{id}/raw` — raw concatenated
|
||||
stdout+stderr as `text/plain`. Works on both finished and in-progress
|
||||
build logs.
|
||||
- `GET /api/reminders` — list pending reminders for the
|
||||
dashboard's queued-reminders panel.
|
||||
- `POST /api/cancel-reminder/{id}` — hard-delete a pending reminder.
|
||||
- `POST /api/retry-reminder/{id}` — re-arm a reminder whose delivery
|
||||
failed (clears the failure state so the scheduler retries).
|
||||
|
||||
- `GET /api/dashboard/stream` — unified live event channel:
|
||||
broker `sent` / `delivered`, plus the mutation events listed
|
||||
below. Each frame carries `seq`.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
## Dashboard event channel
|
||||
|
||||
Wire vocabulary on `/api/dashboard/stream` (kind tag is in the JSON
|
||||
payload):
|
||||
|
||||
- `sent` / `delivered` — broker traffic, mirrored from the
|
||||
intra-process channel by a forwarder task. Both carry `id: i64`
|
||||
(the broker row id) and `in_reply_to: Option<i64>` for thread
|
||||
rendering.
|
||||
- `transient_set` (name, transient_kind, since_unix) /
|
||||
`transient_cleared` (name) — lifecycle action spinners. The
|
||||
client ticks the elapsed-seconds badge off `since_unix`
|
||||
client-side, no polling. `transient_kind` is an **open**
|
||||
display string (the running node's own tag), not a fixed
|
||||
enum — render it, don't branch on it.
|
||||
- `container_state_changed` (container: ContainerView) /
|
||||
`container_removed` (name) — per-row container mutations,
|
||||
emitted by `Coordinator::rescan_containers_and_emit` from
|
||||
the job queue's own node execution (`job_queue::exec`, for example
|
||||
after a rebuild's stop/swap/start steps or a destroy's teardown
|
||||
step) — and from the 10s
|
||||
`crash_watch` poll. Client upserts/removes by name and
|
||||
reads the pending overlay from `transientsState` since the
|
||||
payload doesn't carry it.
|
||||
- `rebuild_queue_changed` (seq) — **payload-free trigger**, fired on
|
||||
every queue mutation (enqueue, state transition, dedup collapse,
|
||||
terminal-history trim). Unlike the snapshot events below it ships
|
||||
no state at all: the client re-fetches `GET /api/jobq/graph`, which
|
||||
is where it reads the queue from cold too.
|
||||
- `reminders_changed` (seq, reminders: `Vec<PendingReminder>`) —
|
||||
full snapshot of all pending reminders. Emitted after every
|
||||
reminder mutation: agent `remind` calls (`agent_server`),
|
||||
operator cancel / retry (`/api/system/reminders/*`), `cancel_loose_end`
|
||||
with Reminder kind, and the scheduler tick after each delivery
|
||||
batch (`reminder_scheduler`).
|
||||
- `tombstones_changed` (seq, tombstones: `Vec<TombstoneView>`) —
|
||||
full snapshot of destroyed-but-state-kept agents. Emitted on
|
||||
destroy, purge, and revive. The K3PT ST4T3 tab subscribes.
|
||||
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
`/api/state` is **only fetched on cold-load and on the few
|
||||
forms that mutate non-event-derived state** (PURG3, since
|
||||
tombstones aren't event-shaped yet). Every other section — transients,
|
||||
containers —
|
||||
derives from `/api/dashboard/stream` after the initial snapshot,
|
||||
maintaining its own client-side store and applying events on
|
||||
top.
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
Generalised form helper: `form[data-confirm="…"]` pops
|
||||
`confirm()` before submit.
|
||||
|
||||
503
docs/2-handbook/hives/hivectl-cli.md
Normal file
503
docs/2-handbook/hives/hivectl-cli.md
Normal file
|
|
@ -0,0 +1,503 @@
|
|||
# Command-Line Help for `hivectl`
|
||||
|
||||
This document contains the help content for the `hivectl` command-line program.
|
||||
|
||||
**Command Overview:**
|
||||
|
||||
* [`hivectl`↴](#hivectl)
|
||||
* [`hivectl agent`↴](#hivectl-agent)
|
||||
* [`hivectl agent restart`↴](#hivectl-agent-restart)
|
||||
* [`hivectl agent pause`↴](#hivectl-agent-pause)
|
||||
* [`hivectl agent resume`↴](#hivectl-agent-resume)
|
||||
* [`hivectl agent start`↴](#hivectl-agent-start)
|
||||
* [`hivectl agent stop`↴](#hivectl-agent-stop)
|
||||
* [`hivectl agent kill`↴](#hivectl-agent-kill)
|
||||
* [`hivectl agent destroy`↴](#hivectl-agent-destroy)
|
||||
* [`hivectl agent rebuild`↴](#hivectl-agent-rebuild)
|
||||
* [`hivectl agent choom`↴](#hivectl-agent-choom)
|
||||
* [`hivectl agent watch`↴](#hivectl-agent-watch)
|
||||
* [`hivectl agent quota`↴](#hivectl-agent-quota)
|
||||
* [`hivectl agent quota show`↴](#hivectl-agent-quota-show)
|
||||
* [`hivectl agent quota set`↴](#hivectl-agent-quota-set)
|
||||
* [`hivectl agent subvol`↴](#hivectl-agent-subvol)
|
||||
* [`hivectl agent subvol upgrade`↴](#hivectl-agent-subvol-upgrade)
|
||||
* [`hivectl agent subvol snapshot`↴](#hivectl-agent-subvol-snapshot)
|
||||
* [`hivectl agent subvol snapshot create`↴](#hivectl-agent-subvol-snapshot-create)
|
||||
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
|
||||
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
|
||||
* [`hivectl agent subvol snapshot push`↴](#hivectl-agent-subvol-snapshot-push)
|
||||
* [`hivectl list-agents`↴](#hivectl-list-agents)
|
||||
* [`hivectl quota-enable`↴](#hivectl-quota-enable)
|
||||
* [`hivectl wg`↴](#hivectl-wg)
|
||||
* [`hivectl wg init`↴](#hivectl-wg-init)
|
||||
* [`hivectl wg peer`↴](#hivectl-wg-peer)
|
||||
* [`hivectl wg status`↴](#hivectl-wg-status)
|
||||
* [`hivectl peer-config`↴](#hivectl-peer-config)
|
||||
* [`hivectl stop`↴](#hivectl-stop)
|
||||
* [`hivectl start`↴](#hivectl-start)
|
||||
* [`hivectl restart`↴](#hivectl-restart)
|
||||
* [`hivectl completions`↴](#hivectl-completions)
|
||||
|
||||
## `hivectl`
|
||||
|
||||
Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that don't go through the broker, plus future recovery / debugging verbs.
|
||||
|
||||
**Usage:** `hivectl [OPTIONS] <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
|
||||
* `list-agents` — Show all managed agents with their status and technical state
|
||||
* `quota-enable` — Enable btrfs qgroup accounting on the agent-state filesystem
|
||||
* `wg` — WireGuard inter-hive mesh setup helpers
|
||||
* `peer-config` — Generate the federation peer-config block for THIS hive
|
||||
* `stop` — Stop containers hive-wide in one operator action
|
||||
* `start` — Start containers hive-wide — the inverse of `hivectl stop`
|
||||
* `restart` — Restart containers hive-wide — `stop` then `start` over one scope
|
||||
* `completions` — Generate a shell completion script for `hivectl` and print it to stdout
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--socket <SOCKET>` — Path to the hive-c0re host admin socket, used by the daemon-assisted verbs (`agent`, `list-agents`, `stop`, `start`). Global: accepted before or after the subcommand. Verbs that don't talk to the daemon ignore it
|
||||
|
||||
Default value: `/run/hyperhive/host.sock`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent`
|
||||
|
||||
Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running.
|
||||
|
||||
Everything here targets a single named agent (`hivectl agent foo restart`, `hivectl agent foo choom`, …) — anything that acts hive-wide lives at the top level instead (`list-agents`, `restart`/`stop`/`start` with a scope, `quota-enable`).
|
||||
|
||||
**Usage:** `hivectl agent <NAME> <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `restart` — Stop and start this agent container without rebuilding config
|
||||
* `pause` — Park this agent's turn loop, leaving the container running
|
||||
* `resume` — Resume this paused agent — it drains whatever queued up while parked
|
||||
* `start` — Start this EXISTING agent container. Fails immediately if `name` has no config/topology entry at all — it never attempts first-time creation, which is swarm-level (`swarmctl agent create`)
|
||||
* `stop` — Gracefully stop this agent container: signal → drain → reconcile. Never escalates to a hard kill — use `kill` for that
|
||||
* `kill` — Hard-stop this managed container
|
||||
* `destroy` — Tear down this sub-agent container, keeping its state by default. No undo
|
||||
* `rebuild` — Apply pending config to this managed container
|
||||
* `choom` — Open an interactive Claude session inside this agent's container
|
||||
* `watch` — Follow this agent's live turn/tool-call event stream from the CLI
|
||||
* `quota` — This agent's disk accounting + optional quota via btrfs qgroups
|
||||
* `subvol` — btrfs subvolume management for this agent's state dir
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<NAME>` — Agent name (for example `damocles`, `iris`)
|
||||
|
||||
|
||||
|
||||
## `hivectl agent restart`
|
||||
|
||||
Stop and start this agent container without rebuilding config
|
||||
|
||||
**Usage:** `hivectl agent restart [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--no-wait` — Return immediately after hivectl queues the restart DAG
|
||||
|
||||
|
||||
|
||||
## `hivectl agent pause`
|
||||
|
||||
Park this agent's turn loop, leaving the container running.
|
||||
|
||||
The harness stops driving turns but keeps serving its web UI and MCP daemons, so the container, its mounts and its warm caches stay up while it burns no tokens. Inbox messages queue unacked and the backlog drains on `resume`. Sticky: it survives a restart, and pausing a stopped agent makes it come up paused.
|
||||
|
||||
**Usage:** `hivectl agent pause [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--no-wait` — Return immediately after hivectl queues the pause DAG instead of waiting for the harness to acknowledge it
|
||||
|
||||
|
||||
|
||||
## `hivectl agent resume`
|
||||
|
||||
Resume this paused agent — it drains whatever queued up while parked
|
||||
|
||||
**Usage:** `hivectl agent resume`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent start`
|
||||
|
||||
Start this EXISTING agent container. Fails immediately if `name` has no config/topology entry at all — it never attempts first-time creation, which is swarm-level (`swarmctl agent create`)
|
||||
|
||||
**Usage:** `hivectl agent start [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--paused` — Start (or leave) the agent paused: if it's currently down, hivectl writes the pause marker before the container boots, so it comes up paused instead of driving turns immediately. If it's already running, this pauses it in place and doesn't attempt a (re)start
|
||||
|
||||
|
||||
|
||||
## `hivectl agent stop`
|
||||
|
||||
Gracefully stop this agent container: signal → drain → reconcile. Never escalates to a hard kill — use `kill` for that
|
||||
|
||||
**Usage:** `hivectl agent stop`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent kill`
|
||||
|
||||
Hard-stop this managed container
|
||||
|
||||
**Usage:** `hivectl agent kill`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent destroy`
|
||||
|
||||
Tear down this sub-agent container, keeping its state by default. No undo
|
||||
|
||||
**Usage:** `hivectl agent destroy [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--purge` — Also wipe the agent's state dirs (config + creds + notes)
|
||||
|
||||
|
||||
|
||||
## `hivectl agent rebuild`
|
||||
|
||||
Apply pending config to this managed container
|
||||
|
||||
**Usage:** `hivectl agent rebuild`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent choom`
|
||||
|
||||
Open an interactive Claude session inside this agent's container.
|
||||
|
||||
A fresh session by default, or resume a prior one. Requires root or `hive-admin` group membership, plus a running container.
|
||||
|
||||
**Usage:** `hivectl agent choom [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--resume <SESSION>` — Resume a prior claude session by its session id, passed through as `claude --resume <value>` (claude's `--continue` takes no value — it resumes the cwd's latest session, which is the harness's, so choom never uses it; this flag matches the claude flag it maps to). Omit for a fresh blank session. Requires a value
|
||||
|
||||
|
||||
|
||||
## `hivectl agent watch`
|
||||
|
||||
Follow this agent's live turn/tool-call event stream from the CLI.
|
||||
|
||||
Prints one compact line per event as they happen. `Ctrl-C` to stop. Requires the agent to be running.
|
||||
|
||||
**Usage:** `hivectl agent watch`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent quota`
|
||||
|
||||
This agent's disk accounting + optional quota via btrfs qgroups.
|
||||
|
||||
Needs `hivectl quota-enable` run once hive-wide first. No-op on non-btrfs hosts.
|
||||
|
||||
**Usage:** `hivectl agent quota <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `show` — Report this agent's disk usage from btrfs qgroups
|
||||
* `set` — Set or clear this agent's disk-usage quota
|
||||
|
||||
|
||||
|
||||
## `hivectl agent quota show`
|
||||
|
||||
Report this agent's disk usage from btrfs qgroups
|
||||
|
||||
**Usage:** `hivectl agent quota show`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent quota set`
|
||||
|
||||
Set or clear this agent's disk-usage quota
|
||||
|
||||
**Usage:** `hivectl agent quota set <SIZE>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<SIZE>` — Size cap (`5G`, `500M`, `1073741824`) or `none` to clear
|
||||
|
||||
|
||||
|
||||
## `hivectl agent subvol`
|
||||
|
||||
btrfs subvolume management for this agent's state dir.
|
||||
|
||||
Upgrade an existing plain-dir agent's state into a btrfs subvolume so it gains snapshots and per-subvol usage/quota.
|
||||
|
||||
**Usage:** `hivectl agent subvol <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `upgrade` — Convert a plain-dir agent state root into a btrfs subvolume in place, so it gains snapshots and per-subvol usage/quota
|
||||
* `snapshot` — Read-only snapshots of this agent's state subvolume
|
||||
|
||||
|
||||
|
||||
## `hivectl agent subvol upgrade`
|
||||
|
||||
Convert a plain-dir agent state root into a btrfs subvolume in place, so it gains snapshots and per-subvol usage/quota.
|
||||
|
||||
Bounces the agent to migrate its state, so it requires `--yes`.
|
||||
|
||||
**Usage:** `hivectl agent subvol upgrade [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--yes` — Confirm: this stops the agent, migrates its state dir, and restarts it. Required — the command refuses without it
|
||||
|
||||
|
||||
|
||||
## `hivectl agent subvol snapshot`
|
||||
|
||||
Read-only snapshots of this agent's state subvolume
|
||||
|
||||
**Usage:** `hivectl agent subvol snapshot <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `create` — Create a read-only snapshot (agent must already be a subvolume)
|
||||
* `delete` — Delete a snapshot created by `subvol snapshot create`
|
||||
* `send` — Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
|
||||
* `push` — Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport
|
||||
|
||||
|
||||
|
||||
## `hivectl agent subvol snapshot create`
|
||||
|
||||
Create a read-only snapshot (agent must already be a subvolume)
|
||||
|
||||
**Usage:** `hivectl agent subvol snapshot create --label <LABEL>`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--label <LABEL>` — Snapshot label. Mandatory, and must start with `hive-` — the prefix doubles as an allow-list hive-priv checks so only hivectl-issued snapshot names can reach the `btrfs subvolume snapshot` shellout
|
||||
|
||||
|
||||
|
||||
## `hivectl agent subvol snapshot delete`
|
||||
|
||||
Delete a snapshot created by `subvol snapshot create`
|
||||
|
||||
**Usage:** `hivectl agent subvol snapshot delete <LABEL>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
|
||||
|
||||
|
||||
|
||||
## `hivectl agent subvol snapshot send`
|
||||
|
||||
Export a snapshot to a local file via `btrfs send` — the local-file half of the inter-hive migration transport (`push` is the network half). Also useful standalone as a point-in-time backup: a full send with no `--parent` produces a self-contained archive of the snapshot
|
||||
|
||||
**Usage:** `hivectl agent subvol snapshot send [OPTIONS] --dest <DEST> <LABEL>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--parent <PARENT>` — Optional parent snapshot label for an incremental send (`btrfs send -p`) — must be an existing, older snapshot of the same agent. Omit for a full send
|
||||
* `--dest <DEST>` — Destination filename (not a path) under the migrate-staging dir. Refused if it already exists
|
||||
|
||||
|
||||
|
||||
## `hivectl agent subvol snapshot push`
|
||||
|
||||
Stream a snapshot to the swarm's snapshot store over the WireGuard mesh — the network half of the migration transport.
|
||||
|
||||
Stages nothing locally: `btrfs send` writes straight into the connection, so a multi-gigabyte agent needs no scratch space on this host. The mesh is the authentication (cryptokey routing binds the sender's address to its key), so there is no credential to pass here.
|
||||
|
||||
No destination argument exists: a swarm has one store, read from `services.hyperhive.swarm.snapshotStore`.
|
||||
|
||||
**Usage:** `hivectl agent subvol snapshot push [OPTIONS] <LABEL>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<LABEL>` — Snapshot label passed to `subvol snapshot create --label`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--parent <PARENT>` — Optional parent snapshot label for an incremental send (`btrfs send -p`) — must be an existing, older snapshot of the same agent, and must already be present on the receiver. Omit for a full send
|
||||
|
||||
|
||||
|
||||
## `hivectl list-agents`
|
||||
|
||||
Show all managed agents with their status and technical state.
|
||||
|
||||
Global — not scoped to one agent, so it lives at the top level rather than under `hivectl agent <name>`. Needs the hive-c0re daemon running.
|
||||
|
||||
**Usage:** `hivectl list-agents [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--json` — Emit the raw JSON rows instead of the padded table (for scripting). The table is the default human-readable shape
|
||||
|
||||
|
||||
|
||||
## `hivectl quota-enable`
|
||||
|
||||
Enable btrfs qgroup accounting on the agent-state filesystem.
|
||||
|
||||
Global one-shot toggle (not per-agent), hence top-level rather than under `hivectl agent <name>`. Idempotent — safe to re-run. Once enabled, `hivectl agent <name> quota show`/`quota set` work.
|
||||
|
||||
**Usage:** `hivectl quota-enable`
|
||||
|
||||
|
||||
|
||||
## `hivectl wg`
|
||||
|
||||
WireGuard inter-hive mesh setup helpers.
|
||||
|
||||
Generate this hive's mesh key and print the nix to enable the mesh, add a peer, or inspect live interface state.
|
||||
|
||||
**Usage:** `hivectl wg <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `init` — Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
|
||||
* `peer` — Print the nix to add a peer hive to the mesh
|
||||
* `status` — Show the live mesh interface state
|
||||
|
||||
|
||||
|
||||
## `hivectl wg init`
|
||||
|
||||
Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
|
||||
|
||||
**Usage:** `hivectl wg init [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--address <ADDRESS>` — This hive's mesh address (for example `10.42.0.1/32`) to bake into the printed snippet. Omit to get a placeholder you fill in
|
||||
|
||||
|
||||
|
||||
## `hivectl wg peer`
|
||||
|
||||
Print the nix to add a peer hive to the mesh
|
||||
|
||||
**Usage:** `hivectl wg peer [OPTIONS] --pubkey <PUBKEY> --address <ADDRESS> <DOMAIN>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<DOMAIN>` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--pubkey <PUBKEY>` — Peer's WireGuard public key (from its `hivectl wg init`)
|
||||
* `--address <ADDRESS>` — Peer's mesh address (for example `10.42.0.2/32`)
|
||||
* `--endpoint <ENDPOINT>` — Peer's `host:port` endpoint (omit for a peer that only dials out, for example one behind NAT — it must set an endpoint pointing back here)
|
||||
|
||||
|
||||
|
||||
## `hivectl wg status`
|
||||
|
||||
Show the live mesh interface state
|
||||
|
||||
**Usage:** `hivectl wg status`
|
||||
|
||||
|
||||
|
||||
## `hivectl peer-config`
|
||||
|
||||
Generate the federation peer-config block for THIS hive.
|
||||
|
||||
Prints the nix a peer operator pastes into their swarm config to trust and reach this hive.
|
||||
|
||||
**Usage:** `hivectl peer-config [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--wg-address <WG_ADDRESS>` — This hive's WireGuard mesh address (for example `10.42.0.1/32`), emitted as `wireguardAddress`. Omit when not running the mesh
|
||||
* `--wg-endpoint <WG_ENDPOINT>` — This hive's public WireGuard endpoint (`host:port`), emitted as `wireguardEndpoint`. Omit when peers dial in / no mesh
|
||||
|
||||
|
||||
|
||||
## `hivectl stop`
|
||||
|
||||
Stop containers hive-wide in one operator action.
|
||||
|
||||
Bare `hivectl stop` stops every agent and the gateway; scope flags narrow it to specific sub-agents or the gateway. The swarm's forge and matrix are never in scope: systemd on the services host runs them.
|
||||
|
||||
**Usage:** `hivectl stop [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--agents` — All sub-agent containers
|
||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||
* `--graceful` — Gracefully quiesce each agent before stopping, instead of a hard stop. Each agent gets a graceful-stop DAG on the job queue: it signals the harness, which runs one stop-checkpoint turn to flush durable `/state` and drains, then the DAG stops the container (bounded by a 3-min timeout that falls back to a hard stop). All drains overlap. Applies to agents only
|
||||
* `--no-wait` — Return immediately after hivectl queues the stop DAGs instead of waiting for them with live per-node progress
|
||||
|
||||
|
||||
|
||||
## `hivectl start`
|
||||
|
||||
Start containers hive-wide — the inverse of `hivectl stop`.
|
||||
|
||||
Bare `hivectl start` starts every agent; scope flags narrow it. The swarm's forge and matrix are never in scope.
|
||||
|
||||
**Usage:** `hivectl start [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--agents` — All sub-agent containers
|
||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||
* `--no-wait` — Return immediately after hivectl queues the start DAGs instead of waiting for them with live per-node progress
|
||||
|
||||
|
||||
|
||||
## `hivectl restart`
|
||||
|
||||
Restart containers hive-wide — `stop` then `start` over one scope.
|
||||
|
||||
Bare `hivectl restart` restarts everything; scope flags narrow it. The swarm's forge and matrix are never in scope.
|
||||
|
||||
**Usage:** `hivectl restart [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--agents` — All sub-agent containers
|
||||
* `--agent <NAME>` — A specific sub-agent by name. Repeatable: `--agent a --agent b`
|
||||
* `--gateway` — The gateway (`hive-gateway`) — nginx on the host, not a container
|
||||
* `--graceful` — Gracefully quiesce each agent on the stop half (see `stop --graceful`). Applies to agents only
|
||||
|
||||
|
||||
|
||||
## `hivectl completions`
|
||||
|
||||
Generate a shell completion script for `hivectl` and print it to stdout.
|
||||
|
||||
Supports bash, zsh, fish, elvish, and powershell. The NixOS module already installs the zsh script system-wide; this is for ad-hoc or other-shell use.
|
||||
|
||||
**Usage:** `hivectl completions <SHELL>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<SHELL>` — Shell to emit completions for
|
||||
|
||||
Possible values: `bash`, `elvish`, `fish`, `powershell`, `zsh`
|
||||
|
||||
|
||||
|
||||
|
||||
159
docs/2-handbook/hives/hivectl.md
Normal file
159
docs/2-handbook/hives/hivectl.md
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
# hivectl
|
||||
|
||||
`hivectl` is the operator-facing host CLI for hyperhive. It lives on the
|
||||
host (not inside any container). It talks to `hive-c0re` over the host
|
||||
admin socket `/run/hyperhive/host.sock`, which is root-only by default —
|
||||
so it needs `sudo` unless you grant sudoless access by listing your
|
||||
login user in `services.hyperhive.deploy.hive-controller.adminUsers` (adds you to the
|
||||
`hive-admin` group that owns the socket; see
|
||||
[`docs/2-handbook/hives/boundary.md`](boundary.md#host-admin-socket-access-hivectl)).
|
||||
Available via the `hive-c0re` package in the host NixOS config.
|
||||
|
||||
Unlike the `hive-c0re` daemon subcommands (which go through the broker),
|
||||
`hivectl` covers direct host-side administration: container lifecycle
|
||||
shortcuts and interactive agent shell access.
|
||||
|
||||
This page is the curated guide. For the exhaustive flag-by-flag
|
||||
reference autogenerated from the binary's own command tree, see
|
||||
[`hivectl-cli.md`](hivectl-cli.md) — emitted by the hidden `hivectl
|
||||
markdown-docs` subcommand and kept in lockstep with the code by the
|
||||
`hivectl-docs` flake check (CI fails if the committed copy drifts).
|
||||
Regenerate with `nix build .#default && ./result/bin/hivectl
|
||||
markdown-docs > docs/2-handbook/hives/hivectl-cli.md`.
|
||||
|
||||
## Agents
|
||||
|
||||
Container lifecycle shortcuts that go through the host admin socket.
|
||||
Requires the `hive-c0re` daemon to be running. Everything scoped to a
|
||||
single agent lives under `hivectl agent <name> <verb>` — `hivectl` hoists
|
||||
the name onto the parent command, so none of the verbs below repeat it.
|
||||
|
||||
```bash
|
||||
hivectl list-agents # roster: every agent's status + technical state
|
||||
hivectl list-agents --json # same data as raw JSON rows (for scripting)
|
||||
hivectl agent iris restart # stop + start the `iris` container (no rebuild)
|
||||
hivectl agent iris pause # park iris's turn loop, leave the container running
|
||||
hivectl agent iris resume # let it drive turns again, draining what queued up
|
||||
```
|
||||
|
||||
`list-agents` prints a padded table with one row per managed agent —
|
||||
`NAME STATUS REMIND`. STATUS collapses the health flags
|
||||
(`running` / `stopped`, plus ` paused` / ` needs-login`
|
||||
when set — `paused` is orthogonal to running, see below);
|
||||
REMIND is the count of pending reminders. It reuses the same per-agent aggregation the dashboard
|
||||
renders, so the CLI roster and the web UI never drift. `--json` emits the
|
||||
raw rows instead of the table.
|
||||
|
||||
`agent <name> restart` is the manual equivalent of the MCP `restart`
|
||||
tool — useful when you need to kick a container from the host without
|
||||
going through an agent at all. For more than one agent at once, use
|
||||
the top-level `hivectl restart --agents` (or `--agent <name>` repeated) —
|
||||
it rides a single DAG and reports per-target failures at the end rather
|
||||
than aborting mid-run, instead of shelling out to `agent <name> restart`
|
||||
in a loop.
|
||||
|
||||
`pause` / `resume` are the "stop burning tokens without losing the
|
||||
container" pair. Pausing writes a marker file into the agent's harness
|
||||
dir (`<state>/<name>/harness/paused`) which the harness re-stats every
|
||||
5 s at the top of its serve loop; while it's there the agent drives no
|
||||
turns, but the container, its mounts, its warm caches, its web UI and
|
||||
its MCP daemons all stay up. Inbox messages queue **unacked**, so a
|
||||
resume drains the backlog rather than dropping it. Points worth knowing:
|
||||
|
||||
- **Sticky.** The marker lives on the persistent harness mount, so a
|
||||
paused agent stays paused across a container restart — and pausing a
|
||||
_stopped_ agent makes it come up parked.
|
||||
- **Not a DAG.** Unlike `restart`/`stop`, there's no container operation
|
||||
to sequence, so it applies immediately with nothing to wait on.
|
||||
- **Stopping a paused agent is still fast.** hivectl skips the graceful-stop
|
||||
handshake for a paused agent (it would never answer), which
|
||||
is safe precisely because the pause check sits at the top of the loop:
|
||||
a paused agent has no turn in flight to checkpoint.
|
||||
- Visible as ` paused` in `list-agents`'s STATUS column, as a `paused`
|
||||
field on the JSON rows, and as a badge on the dashboard card.
|
||||
|
||||
## Choom
|
||||
|
||||
Drop into an interactive Claude session inside an agent container.
|
||||
Replaces the current process with `machinectl shell <name>@h-<name>`
|
||||
running claude from the agent's state dir. Requires root, or membership in
|
||||
`hive-admin` (the same sudoless-`hivectl` group as the [host admin
|
||||
socket](boundary.md) — hive-c0re ships a polkit rule
|
||||
granting that group the one action `machinectl shell` needs); `choom`
|
||||
checks this itself and refuses up front with a message naming the
|
||||
requirement rather than letting systemd reject the exec later with an
|
||||
opaque polkit prompt. Same caveat as the socket grant: a shell opened
|
||||
before the operator added you to `hive-admin` doesn't see it until you log
|
||||
back in — secondary group membership applies at login.
|
||||
|
||||
It also needs the daemon socket, unlike the other exec-into-a-container
|
||||
paths: the "is this actually an agent?" pre-flight reads the agents root,
|
||||
which the daemon's user owns and which isn't group-readable, so the check
|
||||
is a `HostRequest` rather than a local `stat`. An unprivileged `choom`
|
||||
therefore names the grant it wants, instead of reporting a permission
|
||||
problem with the state dir.
|
||||
|
||||
```bash
|
||||
hivectl agent iris choom # fresh blank Claude session in iris's container
|
||||
hivectl agent iris choom --resume <session-id> # rejoin a prior session by id
|
||||
```
|
||||
|
||||
Bare `choom` starts a fresh blank session. `--resume <value>` passes
|
||||
through as `claude --resume <value>` to rejoin a prior session by its
|
||||
session id — the flag name deliberately matches the claude flag it maps
|
||||
to. (choom never uses claude's `--continue`: that's a bare flag that
|
||||
takes no argument and resumes the cwd's _latest_ session, that is, the
|
||||
harness's; claude would consume a value after it as the first prompt,
|
||||
silently poking the live harness session.) Passing the flag requires a
|
||||
value. Either way choom never collides with the harness's live
|
||||
session in the same project dir: the harness pins its own id via
|
||||
`--resume`, so a blank choom session is invisible to it. The container
|
||||
must be running.
|
||||
|
||||
choom reproduces the harness's own claude invocation so the operator
|
||||
lands in a faithful copy of the agent's environment:
|
||||
|
||||
- **as the agent user**, not root: `machinectl shell` defaults to root,
|
||||
which would make claude read `/root/.claude` (empty) instead of the
|
||||
agent's `/home/<name>/.claude` where its OAuth credentials live. choom
|
||||
prefixes the machine with `<name>@` (the meta-flake sets the agent's
|
||||
unix user name to its label).
|
||||
- **from `/agents/<name>/state`**: the session and `AGENTS.md` (the
|
||||
persona) resolve against the right project dir.
|
||||
- **with the harness flags**: `--settings`, `--mcp-config`, and
|
||||
`--system-prompt-file` from `/run/hive-config/` — the same files the
|
||||
harness writes each turn — so the operator gets the agent's settings,
|
||||
the hyperhive/matrix MCP tools, and the role prompt. Each flag is
|
||||
included only when its file exists.
|
||||
- **without the onboarding walkthrough**: a boot-time oneshot
|
||||
(`hive-claude-onboarding`) seeds `hasCompletedOnboarding` and the
|
||||
project trust flags into `~/.claude.json` once before the harness
|
||||
starts, so the first interactive choom lands straight in a session
|
||||
instead of the onboarding/trust dialog the headless harness never
|
||||
completes. It's the single place hyperhive touches that file.
|
||||
|
||||
## Watch
|
||||
|
||||
Follow an agent's live turn/tool-call event stream from the CLI:
|
||||
|
||||
```bash
|
||||
hivectl agent iris watch # tail iris's live events; Ctrl-C to stop
|
||||
```
|
||||
|
||||
Dials the same unix socket the gateway's nginx `proxy_pass`es through
|
||||
(`/run/hive-agent/<name>/web.sock` — see [Per-agent unix-socket
|
||||
upstream](../../4-implementation/networking/gateway.md#per-agent-unix-socket-upstream)) directly and
|
||||
speaks a bare HTTP/1.1 request for the agent's existing `/events/stream`
|
||||
SSE endpoint over it. No gateway hop, no daemon round-trip for the
|
||||
stream itself — the daemon socket is only used for the "does this agent
|
||||
exist" pre-flight, same reasoning as `choom` above. Requires the agent
|
||||
to be running with its web UI socket bound; a fresh spawn/rebuild that
|
||||
hasn't come up yet gets a clear connection-refused hint rather than a
|
||||
raw OS error.
|
||||
|
||||
Prints one compact line per event — reuses the `_icon`/`_summary` fields
|
||||
the harness already stamps onto stream-json events for the web UI
|
||||
(`stream_enrich.rs`), so tool calls and turn markers read as short
|
||||
glyph-prefixed lines instead of raw JSON. Not an attempt at the web
|
||||
UI's full collapsible-details rendering (`docs/4-implementation/web-ui/terminal-rendering.md`)
|
||||
— that's presentation for a browser, this is a `tail -f`.
|
||||
|
|
@ -39,6 +39,13 @@ grafana, the controller, authelia, …) →
|
|||
`admins` only. → [`ui.md`](ui.md)
|
||||
- **shared services** — one forge, homeserver, SSO, queue and metrics/logs
|
||||
stack, each on whichever host you put it. → [`services.md`](services.md)
|
||||
- **forge** — the swarm's Forgejo: per-agent accounts, notifications and the
|
||||
CLI agents use. → [`forge.md`](forge.md)
|
||||
- **matrix** — the swarm's homeserver, several accounts per agent, and
|
||||
federation. → [`matrix.md`](matrix.md)
|
||||
- **knowledge repo** — the reference docs every agent reads, and how to add
|
||||
one. → [`knowledge.md`](knowledge.md)
|
||||
- **`swarmctl`** — the swarm operator's CLI. → [`swarmctl-cli.md`](swarmctl-cli.md)
|
||||
- **SSO** — the secrets authelia generates and how each reaches its reader.
|
||||
→ [`sso.md`](sso.md)
|
||||
- **secrets** — the rules a credential follows → [`credentials.md`](credentials.md);
|
||||
|
|
@ -453,13 +460,13 @@ them, and the notification router, not the controller, decides what they mean.
|
|||
A `config-pr` delivery reporting a PR merged into `main` queues a deploy
|
||||
of its `merge_commit_sha` on the one hive whose wanted state places the
|
||||
agent; with no such hive, or more than one, the controller deploys nothing. See
|
||||
[config-changes.md § Config changes](../agent-lifecycle/config-changes.md#config-changes).
|
||||
[config-changes.md § Config changes](../agents/config-changes.md#config-changes).
|
||||
A merge whose `closed` delivery never arrives deploys nothing either; the
|
||||
operator recovers by redelivering that delivery from the forge hook page,
|
||||
which re-enters the same handler and re-queues the deploy.
|
||||
|
||||
**`internal/knowledge` is on that path.** The controller's is the only
|
||||
hook on it ([`knowledge.md`](../integrations/knowledge.md) covers clearing a
|
||||
hook on it ([`knowledge.md`](knowledge.md) covers clearing a
|
||||
leftover). A webhook has exactly one target URL, so a second registration
|
||||
would take delivery away from the first rather than add a recipient.
|
||||
|
||||
|
|
@ -479,7 +486,7 @@ host also serves the swarm UI vhost — that's what publishes the
|
|||
endpoint, and a hook the forge can't reach would collect failed
|
||||
deliveries while looking healthy. The controller generates the HMAC
|
||||
secret on first start and keeps it
|
||||
(see [`docs/2-handbook/agent-lifecycle/persistence.md`](../agent-lifecycle/persistence.md)).
|
||||
(see [`docs/2-handbook/agents/persistence.md`](../agents/persistence.md)).
|
||||
|
||||
To check it's working, push to `internal/knowledge` and look for
|
||||
`webhook: verified delivery` in `journalctl -u swarm-controller`. A
|
||||
|
|
@ -588,7 +595,7 @@ mechanism.
|
|||
added to the public CAs; it holds the store's `pki` root, so a peer whose
|
||||
certificate chains to the same root validates. A peer outside this
|
||||
swarm's root needs a CA-issued certificate (ACME). Federation firewall
|
||||
and TLS requirements: [`2-handbook/integrations/matrix.md`](../integrations/matrix.md).
|
||||
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
|
||||
|
||||
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
|
||||
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
|
||||
|
|
@ -683,7 +690,7 @@ If both hosts are behind NAT, you need a STUN relay or a third host
|
|||
swarm's `btrfs receive` endpoint and the `swarm.snapshotStore` option
|
||||
- [`5-project/process/conventions.md`](../../5-project/process/conventions.md) § Hive identity —
|
||||
env vars, qualified labels
|
||||
- [`2-handbook/integrations/matrix.md`](../integrations/matrix.md) — matrix
|
||||
- [`2-handbook/swarm/matrix.md`](matrix.md) — matrix
|
||||
federation, TLS cert autogeneration, firewall posture
|
||||
- [`4-implementation/networking/gateway.md`](../../4-implementation/networking/gateway.md) — nginx vhosts and
|
||||
the `.well-known/matrix/` autodiscovery scheme
|
||||
|
|
|
|||
401
docs/2-handbook/swarm/forge.md
Normal file
401
docs/2-handbook/swarm/forge.md
Normal file
|
|
@ -0,0 +1,401 @@
|
|||
# Swarm forge (Forgejo)
|
||||
|
||||
Private Forgejo instance running in a nixos-container (`hive-forge`, the
|
||||
container and CLI name), used as the
|
||||
swarm's persistent code-collaboration surface (issues, PRs, reviews,
|
||||
attachments). Configured via `services.hyperhive.swarm.forge.*` (what the
|
||||
forge IS to every hive — names, ports, URLs) plus
|
||||
`services.hyperhive.deploy.forgejo.*` (what the host running it decides —
|
||||
which build it runs, how it's served, what it mirrors, where its
|
||||
host-local secrets sit).
|
||||
Container
|
||||
shape, ROOT_URL / sub-domain routing, and operator-vs-in-cluster URL
|
||||
handling live in [`docs/4-implementation/networking/gateway.md`](../../4-implementation/networking/gateway.md); this file owns the
|
||||
per-agent integration story and the notification pump that wakes
|
||||
each agent on relevant activity.
|
||||
|
||||
## Token scopes
|
||||
|
||||
Two scope sets:
|
||||
|
||||
**`AGENT_TOKEN_SCOPES`** (swarm-controller's `forge::agent_token`, pinned
|
||||
by a test). swarm-controller mints every agent token with it:
|
||||
|
||||
| Scope | Why |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| `write:repository` | Create, clone, push, delete repos; merge PRs. |
|
||||
| `write:issue` | Open / comment / review issues **and** pull requests (Forgejo namespaces PR conversation under issues). |
|
||||
| `write:user` | Edit own profile, create repos under own user. |
|
||||
| `write:organization` | Create + manage orgs (lets agents share a forge namespace). |
|
||||
| `read:user` | Token-owner endpoint used for self-identification at harness startup. |
|
||||
| `write:misc` | Hooks, attachments, the rest of the long tail. |
|
||||
| `read:notification` | Poll `GET /notifications` for unread events. |
|
||||
| `write:notification` | Mark notifications read via `PATCH /notifications/threads/{id}`. |
|
||||
|
||||
**`HIVE_TOKEN_SCOPES`** (swarm-controller's `forge::hive_token`, pinned by a
|
||||
test): `read:repository` alone, for each hive's read-only account (next
|
||||
section).
|
||||
|
||||
---
|
||||
|
||||
## How a hive reads the forge
|
||||
|
||||
Each hive in swarm-controller's hive directory has its own forge account,
|
||||
`hive-<hive>`. swarm-controller creates it, makes it a member of the `hives`
|
||||
team in `agent-configs`, and mints one token named `swarm-hive` into the
|
||||
swarm secret store at `swarm/hives/<hive>/forge/token`. The team has read
|
||||
permission on `repo.code` and `repo.pulls` of every repo in the org,
|
||||
including repos created later. A pass at start and every five minutes
|
||||
creates what's missing and re-mints a token that's missing or no longer
|
||||
matches the forge. A pass whose store or forge reads fail queues nothing, so
|
||||
an outage never rotates every hive's token.
|
||||
|
||||
swarm-controller refuses to create an agent whose name starts with `hive-`:
|
||||
these accounts own that prefix. An agent that already had such a
|
||||
name keeps it, so the accounts are also told apart by email. swarm-controller
|
||||
creates the hive's account with the email `hive-<hive>@hives.hyperhive.local`
|
||||
and refuses an account by that name with any other email: it never adds an
|
||||
agent's account to the team or mints a token for it. An agent's own forge
|
||||
account setup refuses the hive's account the same way.
|
||||
|
||||
hive-c0re reads the token with `forge::reader_token()`, under the hive's own
|
||||
store certificate, every time it reads the forge. A hive whose store holds
|
||||
none reads nothing. These reads go through it, and none of them needs a local
|
||||
forge container:
|
||||
|
||||
- cloning and pulling `/knowledge`
|
||||
|
||||
The hive builds nothing, so no nix process reads the forge as the hive's
|
||||
account. hive-c0re hands the token to each `git` command it runs
|
||||
(`http.extraHeader`) and keeps no copy on disk. A hive whose store holds no token
|
||||
reads nothing private from the forge. This also holds on a hive with no forge
|
||||
container.
|
||||
|
||||
Roll a change to any of this out store host first (its granter writes the
|
||||
controller's grant on `swarm/hives/+/forge/token`), then the controller host,
|
||||
then the hives. A hive that starts before its token exists reads nothing, and
|
||||
builds nothing, until the controller's next pass.
|
||||
|
||||
---
|
||||
|
||||
## Per-agent forge accounts
|
||||
|
||||
Each agent gets its own Forgejo user and access token. swarm-controller
|
||||
creates the user when it creates the agent, and mints one token named
|
||||
`swarm-agent` with the admin API into the swarm secret store at
|
||||
`swarm/agents/<agent>/forge-token`. A pass at start and every five minutes
|
||||
covers every agent with a store identity: it creates a missing forge user and
|
||||
re-mints any token that's missing or no longer matches the forge;
|
||||
a rotation deletes the old `swarm-agent` token first, so each agent holds at
|
||||
most one. The agent fetches the token under its own store certificate into
|
||||
`/run/hive-agent-forge-token/token` (`nix/agent-modules/forge-token.nix`),
|
||||
and `hive-forge`, the git credential helper, the `forge_notify` poller and
|
||||
the avatar sync read it from there, falling back to `<state>/forge-token`,
|
||||
the file hive-c0re wrote before. hive-c0re no longer creates agent users or
|
||||
mints agent tokens. swarm-controller deletes the `hyperhive-<unix-seconds>`
|
||||
tokens it left behind, for each agent whose current `swarm-agent` token is
|
||||
live; tokens of other names are never touched.
|
||||
|
||||
Two things live in the `agent-configs` Forgejo organization:
|
||||
|
||||
- A config repo per agent (`agent-configs/<name>`). The
|
||||
agent is a **write collaborator on its own** repo — it can push
|
||||
config-change branches and open config PRs — but `main` is
|
||||
branch-protected by swarm-controller: the merge and approval
|
||||
allowlists are the `operators` team, and the agent can neither push
|
||||
`main` directly nor self-merge. An operator's merge in the Forgejo UI
|
||||
deploys the merged commit (see
|
||||
[config-changes.md § Config changes](../agents/config-changes.md#config-changes)).
|
||||
hive-c0re pushes nothing to this org and reads nothing from it.
|
||||
Repos stay private, so an agent can't read another
|
||||
agent's config. The agent's **persistent meta flake input** is this repo's `main` head,
|
||||
pinned by sha in `internal/meta` (`agent-<n>.url =
|
||||
git+<forge>/agent-configs/<n>.git?rev=<sha>`; see
|
||||
[config-changes.md § Meta flake](../agents/config-changes.md#meta-flake)).
|
||||
|
||||
The `hive-forge` CLI (separate workspace crate, see
|
||||
[`README.md`](../../README.md) file map) wraps the Forgejo REST API
|
||||
with the per-agent token; agents call it for issue / PR / comment
|
||||
ops as if it were a peer. All REST calls across the workspace
|
||||
(`hive-forge` verbs, hive-c0re provisioning, this poller) go through
|
||||
the typed `forgejo-api` crate; only non-`/api/v1` web-router routes
|
||||
(attachment / artifact downloads, log streaming) and the poller's
|
||||
enrichment fetches of server-provided subject URLs stay on raw
|
||||
reqwest.
|
||||
|
||||
## Notification poller (`hive-forge-notify/src/notify.rs`)
|
||||
|
||||
Its own long-running per-agent daemon (`hive-forge-notify`, a sibling of
|
||||
`hive-bash-daemon` / `hive-matrix-daemon`) — it used
|
||||
to be a background task inside the `hive-agent` serve loop. Polls
|
||||
`GET /api/v1/notifications?all=false` every 30 seconds (Forgejo's
|
||||
unread-only filter), formats each notification as a broker
|
||||
`Wake { from: "forge" }` message, and delivers it to the agent's own
|
||||
inbox so claude's normal turn loop picks it up.
|
||||
|
||||
The daemon waits about ten minutes for the agent's forge token, then exits 0,
|
||||
which its `Restart=on-failure` doesn't restart. A path unit
|
||||
(`hive-forge-notify.path`) starts it again when the fetched token file or the
|
||||
state-dir `forge-token` changes, so a token minted later still starts the
|
||||
polling.
|
||||
|
||||
The crate builds a second, independent binary for a different host — see
|
||||
[github.md](../external-accounts/github.md#notifications).
|
||||
|
||||
The host-specific calls — list unread, mark read, resolve own login —
|
||||
live behind `Source` in `hive-forge-notify/src/source.rs`;
|
||||
classification, formatting, dedupe and todo delivery live in `notify.rs`,
|
||||
common to both binaries. Todo
|
||||
keys here are the bare thread ids, and must stay that way: renaming them
|
||||
would orphan every in-flight forge todo on the first restart after a
|
||||
deploy.
|
||||
|
||||
### Mark-read on delivery
|
||||
|
||||
On a **successful** broker delivery, `forge_notify` marks the thread
|
||||
read on forge straight away (`PATCH /notifications/threads/{id}`). The
|
||||
broker inbox is the durable work queue now — each delivered wake is a
|
||||
sqlite row with its own ack lifecycle — so the forge unread flag no
|
||||
longer needs to track whether the agent has _processed_ a
|
||||
notification. Clearing it on delivery keeps forge's unread set **small
|
||||
by construction**: at rest it holds only threads that failed to
|
||||
deliver plus whatever arrived since the last 30s poll.
|
||||
|
||||
That size property is the whole point. A container rebuild starts the
|
||||
poller with no memory of what it delivered, re-scans `?all=false`, and
|
||||
finds nothing stale — the delivered threads are already read on forge.
|
||||
Forge's own read-state is thus the durable, cross-rebuild record of
|
||||
what `forge_notify` has delivered; there is **no persisted cursor**.
|
||||
|
||||
**Gotcha:** don't reintroduce a persisted dedup cursor here. A design
|
||||
that leaves threads unread and tracks delivery via a separately persisted
|
||||
cursor is fragile — losing that cursor across a rebuild re-delivers the
|
||||
agent's entire still-unread backlog as a flood of fresh wakes. Forge's
|
||||
own read-state is the only durable record this design needs.
|
||||
|
||||
**Read-before-comment guard doesn't block a fresh wake.** hive-forge's
|
||||
read-before-comment guard (which keys off forge unread-state) doesn't
|
||||
force the agent to view a thread before commenting on it: the broker
|
||||
wake already carries the notification body, so _delivery is the read_.
|
||||
An agent that wants the full thread still runs `hive-forge comments` /
|
||||
`view`.
|
||||
|
||||
**In-process dedupe (small, ephemeral).** A single-process map (thread
|
||||
id → last-delivered `updated_at`) guards the narrow window where a
|
||||
mark-read call _transiently fails_ and the thread reappears unread in
|
||||
the next poll before its `updated_at` bumps — so a flaky PATCH doesn't
|
||||
re-fire the wake. it's **not persisted** and resets on restart (forge
|
||||
read-state covers the durable case). Each poll prunes it to the ids in
|
||||
the single `limit=UNREAD_FETCH_LIMIT` (50) fetch page, so it can never
|
||||
exceed that many entries (a debug assertion pins the invariant; the
|
||||
fetch limit and the bound are the same constant). `forge_notify` leaves
|
||||
a failed _delivery_ unread and out of the map, so it resurfaces next tick.
|
||||
|
||||
`forge_notify` marks self-echo notifications (the agent's own writes, see below)
|
||||
read directly without a delivery — same `mark_read` call, no wake.
|
||||
|
||||
### Activation gates (graceful no-ops)
|
||||
|
||||
The poller starts switched off and stays that way for any of:
|
||||
|
||||
- `HIVE_FORGE_URL` not set or not parseable as a URL (a hand-run
|
||||
harness; hive-c0re sets it from `swarm.forge.domain` on every hive).
|
||||
- no token in `$HIVE_FORGE_TOKEN_FILE` or `<state>/forge-token` (agent has no forge
|
||||
account — pre-provisioning or destroy-without-purge race).
|
||||
- Initial client construction fails (the typed `forgejo-api` client
|
||||
for the API calls, or the plain reqwest client kept for the
|
||||
best-effort enrichment fetches of server-provided subject URLs;
|
||||
both unlikely; treated as fatal-to-the-task only).
|
||||
|
||||
Switched off = the spawned task returns immediately. All other failure
|
||||
modes (HTTP errors, parse errors, mark-read failures) are
|
||||
best-effort: logged at debug/warn and retried next tick.
|
||||
|
||||
### Self-notification filtering
|
||||
|
||||
Forgejo fires notifications for the agent's own actions (it opened a
|
||||
PR, posted a comment, submitted a review). Surfacing those would
|
||||
loop claude on its own writes. `forge_notify` drops the comment/review case
|
||||
silently (mark-read without delivery):
|
||||
|
||||
- **Self-authored comments / reviews** — comment payload's
|
||||
`user.login` matches `own_login`.
|
||||
- **Self-authored creations** (an agent opening its own PR/issue) — the
|
||||
already-fetched subject payload's poster `user.login` matches
|
||||
`own_login`. `forge_notify` drops only _creations_ this way; a later state
|
||||
change on the agent's own subject comes from someone else, so it still surfaces.
|
||||
|
||||
`forge_notify` fetches `own_login` at startup via `GET /api/v1/user`. On fetch
|
||||
failure the filter degrades open (no filtering) rather than crashing
|
||||
the task — a noisy inbox beats a silently stuck poller — but the fetch
|
||||
is **re-attempted on each poll tick** until it succeeds, so a boot-time
|
||||
failure (the forge not yet reachable) self-heals instead of leaving
|
||||
self-echo filtering off for the whole process lifetime.
|
||||
|
||||
### Body excerpt + truncation + heading escape
|
||||
|
||||
The wake message embeds the comment / review body so the agent sees
|
||||
actual content without a follow-up fetch — **not** a new issue/PR's own
|
||||
description, which the wrapper table below deliberately omits. Three
|
||||
pipeline steps in order:
|
||||
|
||||
1. **Truncate** to `BODY_TRUNCATE = 500` chars at a char-boundary;
|
||||
appends `…` when cut. Truncation happens BEFORE escape so the
|
||||
mention-overflow diff (next step) compares like-for-like against
|
||||
the raw body.
|
||||
2. **Mention overflow extraction** — when truncation actually
|
||||
trimmed content, walk the full body line-by-line and surface any
|
||||
`@username` lines that fell outside the embed window. Rendered as
|
||||
a trailing `mentions (truncated from body):\n > <line>` block.
|
||||
Mention detection requires the `@` to be at line start or
|
||||
following a non-username byte, so email-style `foo@bar.com` does
|
||||
NOT count.
|
||||
3. **ATX heading escape** — for each line that's a strict
|
||||
`CommonMark` ATX heading (1-6 leading `#`s followed by a space,
|
||||
tab, or end-of-line), prepend `\` so the embedded body doesn't
|
||||
blow into a top-level h1/h2 inside the wrapper message when the
|
||||
dashboard renders it. Lines like `#tag`, `#123`, `#!/bin/bash` <!-- lint:allow: syntax examples, not real tags -->
|
||||
**aren't** headings — no escape, no cosmetic noise. Indented
|
||||
"headings" inside lists / nested quotes keep their leading
|
||||
whitespace.
|
||||
|
||||
The strict ATX rule is deliberate: `\#tag` and `#tag` render
|
||||
identically, so an over-eager escape just adds visual clutter
|
||||
without changing behavior. Setext-style headings (`title\n====`)
|
||||
aren't handled — rarer in practice, would need multi-line
|
||||
lookahead.
|
||||
|
||||
### Wrapper format
|
||||
|
||||
Five shapes, distinguished by the notification's classification:
|
||||
|
||||
| Trigger | Wrapper |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------ |
|
||||
| Comment on issue / PR | `[comment on PR #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
|
||||
| Review submission | `[PR approved #N owner/repo] title\nurl: ...\n\nauthor: body\nassignee: ...` |
|
||||
| New issue / PR | `[new PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| Later activity (open, not creation) | `[activity on PR #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
| State change | `[PR merged #N owner/repo] title\nurl: ...\nassignee: ...` |
|
||||
|
||||
The issue/PR's own description is deliberately never embedded here (only a
|
||||
comment/review body is — that's the actual new content a notification
|
||||
exists to surface): repeating a subject's own description on every "new"
|
||||
or "activity on" wake for it's noise the agent already has via the URL,
|
||||
not context (mara, triage: "just the issue title should be enough").
|
||||
|
||||
Review labels come from the Forgejo `state` field: `APPROVED` →
|
||||
`approved`, `REQUEST_CHANGES` → `changes requested`, `COMMENT` →
|
||||
`review comment`. `forge_notify` drops `PENDING` (review saved but not
|
||||
submitted yet — no peer-visible event). Unknown states fall back to
|
||||
the generic comment wrapper.
|
||||
|
||||
A review submitted with **no body** renders `reviewed by: <author>` in
|
||||
place of the `<author>: <body>` line — deliberately worded to not collide
|
||||
with the meta-suffix `reviewer:` line (requested reviewers, below).
|
||||
|
||||
### Inline review comments
|
||||
|
||||
A review's line comments belong to neither the review body nor the
|
||||
`latest_comment_url`. A review with only inline comments has no such url
|
||||
and would otherwise land as a bare `activity on PR`. Every PR wake therefore
|
||||
fetches the PR's reviews (`/pulls/N/reviews`, up to four pages of 50) and,
|
||||
for the reviews new to the agent, appends an
|
||||
`inline review comments (read them with `hive-forge pr reviews N`):` block
|
||||
saying where they sit: one `author: path:line, path, ...` line per review
|
||||
author (bare `path` when Forgejo dropped the position because the line left
|
||||
the diff; control characters in a path become spaces). The comment text stays
|
||||
out of the wake, and the agent reads the full review with the CLI.
|
||||
|
||||
A review counts as new when it landed strictly after the thread's previously
|
||||
delivered `updated_at`. Without a baseline every review counts, and so does
|
||||
one with no submit time, so nothing disappears. It doesn't count when the
|
||||
agent wrote it, when it's a `PENDING` draft, or when `comments_count` is 0.
|
||||
The notifier fetches the newest 5 such reviews and lists at most 10 locations
|
||||
(`(+N more)` for the rest). Best effort: a failed fetch yields less text and
|
||||
never drops the wake. Cost: one reviews fetch per PR wake, plus one per new
|
||||
commented review.
|
||||
|
||||
### Merge/close vs a later comment
|
||||
|
||||
A notification carrying a `latest_comment_url` normally takes the comment
|
||||
path. But a merged/closed subject **keeps** its `latest_comment_url` set,
|
||||
so a just-merged PR that had any prior discussion would route to the
|
||||
comment path and render `[comment on PR]` (with a stale pre-merge comment
|
||||
body) instead of `[PR merged]` — the agent never learns its PR merged.
|
||||
When the notification IS the merge/close transition — its
|
||||
event time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` of the
|
||||
subject's `closed_at` (set for both `merged` and `closed`) — the
|
||||
state-change path wins even with a comment url present
|
||||
(`state_change_is_current`). A genuine **later** comment on an
|
||||
already-closed subject bumps `updated_at` well past `closed_at`, so it
|
||||
stays on the comment path and keeps its comment body. Missing/unparseable
|
||||
timestamps default to the state-change path, so a merge is never silently
|
||||
hidden behind a stale comment.
|
||||
|
||||
#### Merge racing a comment
|
||||
|
||||
The one gap the timestamp cut leaves: a genuine comment posted **within
|
||||
`NEW_ITEM_TOLERANCE_SECS` of the merge** bumps `updated_at` close enough
|
||||
to `closed_at` that `state_change_is_current` returns `true` — so it takes
|
||||
the state-change path, dropping its body. Best of both worlds:
|
||||
on the merge/close path `forge_notify` fetches the `latest_comment_url` comment and, when
|
||||
its `created_at` is strictly **after** the subject's `closed_at`
|
||||
(`comment_is_after_close`) — in other words, it raced the merge rather than being the
|
||||
pre-merge last comment the subject keeps — append it as a
|
||||
`comment by <author>: <excerpt>` block before the meta suffix
|
||||
(`fresh_post_close_comment_tail`). The wake carries **both** `[PR merged]`
|
||||
and the racing comment. `forge_notify` leaves the kept pre-merge comment (created before
|
||||
`closed_at`) off, drops a self-authored racing comment (don't
|
||||
echo the agent's own write), and appends nothing for a missing/unparseable `created_at`/
|
||||
`closed_at` (conservative — only surface a comment it can
|
||||
positively place after the close). Cost: one extra comment fetch on
|
||||
merge/close notifications, acceptable given how rare they are.
|
||||
|
||||
### "new" vs "activity on"
|
||||
|
||||
A review submitted with **no body** carries no `latest_comment_url`,
|
||||
so it misses the comment path and lands on the state-change path with
|
||||
`state == "open"` — exactly like a freshly opened PR. Labeling that
|
||||
`new PR` is misleading: agents dismiss it as a duplicate of the
|
||||
original open notification and miss the review. The `open`
|
||||
state only earns the `new <kind>` label when the notification's event
|
||||
time (`updated_at`) is within `NEW_ITEM_TOLERANCE_SECS` (120s) of the
|
||||
subject's `created_at`. `forge_notify` labels anything later `activity on <kind>`
|
||||
— neutral and non-misleading, since it can't cheaply say _what_ the
|
||||
activity was without an extra reviews fetch. Missing/unparseable
|
||||
timestamps default to `new` (preserve prior behavior rather than mask a
|
||||
genuine new item). A small dependency-free
|
||||
RFC 3339 helper (`parse_rfc3339`) parses timestamps.
|
||||
|
||||
`forge_notify` extracts the number from `subject.html_url`'s last path segment
|
||||
(strips `#anchor` first), and the repo slug from `repository.full_name`.
|
||||
Both degrade gracefully when absent (number → blank, repo → blank)
|
||||
so unexpected Forgejo shapes don't crash the formatter.
|
||||
|
||||
### Meta suffix
|
||||
|
||||
Every wrapper ends with one or more of:
|
||||
|
||||
- `assignee: <list>` — always present; `unassigned` when empty so
|
||||
the line shape is stable.
|
||||
- `reviewer: <list>` — PR notifications only, present only when
|
||||
`requested_reviewers` is non-empty.
|
||||
|
||||
### Review-request override
|
||||
|
||||
For new PRs, the kind label flips to `[review requested #N
|
||||
owner/repo]` when `own_login` appears in `requested_reviewers`,
|
||||
regardless of the Forgejo `reason` field. Forgejo doesn't reliably
|
||||
set `reason == "review_requested"` (often null instead), so the
|
||||
fallback checks the subject payload directly. `is_new` gates detection
|
||||
so the label only fires once on PR creation, not on every
|
||||
subsequent comment.
|
||||
|
||||
### Subscription management
|
||||
|
||||
The poller does **not** autounsubscribe from repo watches — it
|
||||
delivers every unread notification it's handed. A hive-forge CLI
|
||||
subscription verb bounds the firehose explicitly (dropping broad repo
|
||||
watches an agent doesn't need) — the poller never guesses which
|
||||
watches to drop. See the `subscription` verb in
|
||||
[`docs/4-implementation/agent-tools/forge.md`](../../4-implementation/agent-tools/forge.md).
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue