mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

docs: sort the top-level directories into five sections by what the reader wants

Browse source 
Every docs directory moves under 1-getting-started, 2-handbook,
4-implementation or 5-project; the options reference (section 3) comes from
the website and has no directory here. trust-boundary stays whole under the
handbook for now. web-ui, swarm and tools split between handbook and
implementation, and the command-line references move to
2-handbook/reference. Links, path references in code and nix, and the docs
index follow the moves; no page content changes.
This commit is contained in:
iris 2026-11-06 18:30:12 +01:00 • committed by mara
commit 6739bb903d
4 changed files with 736 additions and 701 deletions
Download patch file Download diff file Expand all files Collapse all files

363
docs/4-implementation/scheduler/ci.md Normal file
View file

@ -0,0 +1,363 @@
# swarm-ci-runner: Forgejo Actions Runner
The `swarm-ci-runner` module runs the swarm's Forgejo Actions runner in a
`swarm-ci-runner` nixos-container, executing CI jobs from
`.forgejo/workflows/ci.yml` on every PR. It's a swarm service: it runs on
whichever host sets it, and needs neither the forge nor a hive on that host.
## For operators
**Enabling it's one line**: `services.hyperhive.deploy.ci-runner.enable = true`
in the NixOS config of the host that should run it. No manual token
provisioning: swarm-controller publishes the forge's runner registration token
to the secret store, and the runner reads it with a store identity of its own.
On the store's host `glue-bao-tls.nix` mints that identity; on any other host, copy
`ci-runner.pem` and `ci-runner-key.pem` from the store host's
`/var/lib/swarm-bao-pki/` and name them in
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
`ciRunnerClientKeyFile`.
Two things worth knowing before you rely on it:
- **Only run CI for contributors you trust.** Builds run **unsandboxed**
(an nspawn limitation, not a choice) — a malicious build script in a
PR could make network requests or write to the container filesystem
during the build. Fine for a small trusted-team hive where everyone
already has forge access; if you take outside/fork contributions,
gate CI behind Forgejo's fork-PR-approval setting or restrict the
workflow to push-only triggers (forks can't push to your branches).
See _Security_ below for the full threat model.
- **Watch your disk.** CI builds through the host's nix store with no
automatic garbage collection of its own — a busy CI day can fill
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
disk-pressure GC config from _Host store maintenance_ below to your
host's NixOS config (not optional if you plan to actually use this).
Everything below this point is implementation detail: registration
internals, container design, and the full security writeup.
## CI checks
<!-- vale write-good.Passive = NO -->
The jobs run on every PR (and on `workflow_dispatch` for manual
re-triggers), defined in
[`.forgejo/workflows/ci.yml`](../../../.forgejo/workflows/ci.yml). The first four
below are required checks (forge branch protection) — a hit on any of them
blocks merge. The two vale jobs split what used to be a single check: the
error-only one exists so it _can_ become a required check without also
gating on the full job's standing warning/suggestion backlog, but flipping
that switch on the forge is a separate operator action, not something this
table can assert ahead of it happening.
<!-- vale write-good.Passive = YES -->
| Job | What it runs | Required? |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| **nix flake check** | `scripts/ci-flake-check.sh`: nix-eval-jobs, then builds each check (treefmt, clippy, tests, module eval); no VM tests | yes |
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) | yes |
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) | yes |
| **doc-pointer lint** | flags `docs/…md` paths and relative markdown links that no longer resolve (`scripts/check-doc-refs.sh`) | yes |
| **attribution-trailer lint** | flags `Co-Authored-By` trailers with Claude/Anthropic attribution in PR commits (`scripts/check-attribution-trailers.sh`) | not yet — intended to be |
| **prose lint (vale, errors)** | vale, `docs/` tree, error-severity rule hits only | not yet — intended to be |
| **prose lint (vale)** | vale, `docs/` tree, full severity range (warnings + suggestions too) | no — advisory, standing backlog |
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
without an empty commit.
<!-- vale write-good.Passive = NO -->
**`ci-rerun --pr` verifies the code but doesn't reliably move the PR's own
status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
JSON): a genuine PR-triggered run carries `event: pull_request` with the
`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
carries `event: workflow_dispatch` with the real branch name instead.
Forgejo's PR commit-status tracking appears scoped to `pull_request`-event
runs specifically, so a dispatched run — even a genuinely successful one —
never writes to that status slot. Reproduced with three dispatches on one
PR over 20+ minutes, all genuinely green, the PR's own status unmoved
throughout. **Always re-check `pr status` after a dispatch and believe
what it says — don't push a commit just to unwedge it, that's the
empty-commit anti-pattern this verb exists to avoid.** If `hive-forge
ci-log --run <n> --job 0` shows the code green (verdict is the
last line) but the status is stuck, ask the operator to select "rerun" in
the forge web UI — it's CSRF-gated, so only they can do it.
<!-- vale write-good.Passive = YES -->
### Running checks locally
Don't run `nix flake check` directly — it dispatches to the shared build farm and
wastes a remote-builder slot. Use the devshell equivalents instead:
```sh
nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix fmt # authoritative; the formatter set is nix/treefmt.nix
sh scripts/check-issue-refs.sh # tracker-tag lint
sh scripts/check-comment-blocks.sh # comment-block lint
```
`scripts/pre-push` provides a git pre-push hook that automates the two
lint checks. Install it once per clone:
```sh
ln -sf ../../scripts/pre-push .git/hooks/pre-push
```
After that, any `git push` automatically runs both lints and aborts with a
diagnostic if either fails — catching the issue locally before CI sees it.
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
slow); run those manually before pushing Rust changes.
### VM tests
Tests that boot NixOS VMs are the `vmTests.<system>.<name>` flake output
(`nix/vm-tests.nix`), not `checks`, so the per-PR **nix flake check** job never
builds them. [`.forgejo/workflows/vm-nightly.yml`](../../../.forgejo/workflows/vm-nightly.yml)
builds all of them nightly at 02:30; a failing test fails that run. To run one
by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
workflow, optional `test` input), or run
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives
peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that
must not handshake).
## Configuration reference
`services.hyperhive.deploy.ci-runner.enable = true` on the host that runs it
(see _For operators_ above). The runner reaches the forge at
`https://<swarm.forge.domain>`, so that name has to resolve from the host: the
gateway's dnsmasq answers it locally where the forge runs, and forwards it to
the operator's DNS everywhere else.
Optional tuning under `services.hyperhive.deploy.ci-runner`: `name` (runner
name in the forge admin panel), `concurrency` (parallel job capacity), `labels`
(workflow targeting; the default `hive-ci:host` is what this repo's workflows
name), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration string
for example `"3h"` — a job that exceeds it's killed so a hung or runaway build
can't hold the runner's single slot indefinitely).
The old `services.hyperhive.deploy.forgejo.ci.{enable,name,concurrency,labels,package,jobTimeout}`
and `services.hyperhive.swarm.forge.ci.{enable,name,concurrency,labels,package}`
are still accepted: each forwards to the same-named option under
`deploy.ci-runner`, and evaluation warns naming both paths. Set the new name to
silence the warning. The two `name` defaults differ (`hive-ci` before,
`swarm-ci-runner` now), so a host that never set the old `name` shows up under
the new default.
The forwarded `enable` carries the new module's requirements, not the old
module's: evaluation refuses a host that sets only the old `enable` unless
`services.hyperhive.deploy.bao.ciRunnerClientCertFile` and
`ciRunnerClientKeyFile` are both set. They default to the identity
`glue-bao-tls.nix` mints when `services.hyperhive.deploy.bao.enable` is true;
elsewhere place them as described under _For operators_. The runner no longer
needs `deploy.forgejo.enable` or a hive on its host.
## Container design
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`), takes its address from the gateway's DHCP, and resolves through the gateway's dnsmasq on the bridge IP. It can't reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI.
- **Non-ephemeral**: runner credentials (`.runner`) persist across restarts, written to the container's state dir on registration and reused after.
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/5-project/process/gotchas.md`.
- **Credential isolation**: no forge admin credential enters the container. swarm-controller holds the forge admin token; the runner holds only the registration token and its own `.runner`, and the host unit that fetches the token holds a store identity that reads that one path. Only `swarm-ci-runner-register.service` loads the registration token; the daemon unit and the jobs it runs have it neither in their environment nor among their credentials.
## Registration
1. **swarm-controller** (`swarm-controller/src/forge/runner_token.rs`) reads
the forge's instance-wide token from
`GET /api/v1/admin/runners/registration-token` at start and every five
minutes, and writes it to the secret store at
`swarm/services/ci-runner/registration-token` when it differs from the
value it last wrote. Its store grant on that path is write-only.
2. **`swarm-bao-ci-runner.service`** on the runner's host logs in to the store
with the `ci-runner` leaf, whose `swarm-ci-runner` role reads that one path
and nothing else, retrying every 30 seconds until the token is there. It
writes the token to a root-only file in `/run/swarm-ci-runner` on the host,
which the container sees read-only, and restarts the runner when the value
changes. `/run` is tmpfs, so no copy reaches persistent disk; after a reboot
the unit fetches the token again, and until the store answers only a
runner that has no `.runner` yet waits. It runs at boot, on deploy, and every
ten minutes on a timer.
3. **`swarm-ci-runner-register.service`** in the container is the only unit
that loads that file, with `LoadCredential=`. systemd hands a unit's
credentials to all of its processes, and for a `host` label act_runner
runs every job step as a process of the daemon with the daemon's whole
environment, so the daemon unit carries the token in neither. The
register unit is a oneshot under the daemon's dynamic user and state
directory. It reads the token from `$CREDENTIALS_DIRECTORY` and runs
`act_runner register` only when `.runner` is missing, so the `.runner` it
writes is the daemon's own. **The runner unit** (`gitea-runner-swarm`)
starts it in a pre-start step and runs the daemon once it succeeds.
A root step of the register unit, ahead of registering, removes `.runner`
in three cases:
- The forge has no record of the runner. Forgejo answers every call from an
unknown runner with `unregistered runner`, and the daemon exits 1 when its
startup `Declare` gets that answer. On the next start the step finds the
message in the daemon's journal since its last start, and registers again.
A daemon that's already running when an admin deletes its runner keeps polling
and logging the error without exiting; the next restart of the unit or
container registers it again.
- `.runner` names another forge address than the configured one.
- `.runner` names other labels than the configured ones, compared in the
form the daemon writes back: `<name>:<scheme>`, with `host` for a
label that names no scheme.
A changed token alone doesn't register again: the daemon authenticates with
the per-runner token in `.runner`. Without `.runner` and with no token yet
the register unit fails and the runner unit's pre-start step with it;
`Restart=on-failure` on the runner unit starts both again every 15 seconds,
with no start limit.
Host jobs run as the runner's own user, so they can still read `.runner`
and act as this runner; see the security section below.
A new registration is a new runner on the forge: the previous entry stays in
the forge admin panel as offline until an admin deletes it. The runner the old
`hive-ci` module registered is one such entry, since this runner registers
under its own name and token. Delete it under Site administration, Actions,
Runners.
## Actions mirrors
<!-- vale write-good.Passive = NO -->
With `services.hyperhive.deploy.forgejo.actionsFromSelf` enabled, the swarm
controller seeds `actions/checkout` and `actions/upload-artifact` pull-mirrors
(from `github.com/actions/*`) on the forge and Forgejo's `DEFAULT_ACTIONS_URL`
points at the forge itself. CI `uses: actions/<name>@vN` steps then resolve on
the forge, with no external DNS on the CI critical path. A workflow that names
any other bare `actions/*` action fails at job setup with "repository not
found" until that action joins the list in `nix/host-modules/hive-forge/default.nix`.
A forge host that runs the swarm controller therefore resolves those actions
from itself, with or without a runner on that host.
Set `actionsFromSelf = false` there to resolve actions from `data.forgejo.org`
and seed no mirror.
The option defaults to whether the forge's host also runs the swarm controller
(`services.hyperhive.deploy.swarm-controller.enable`; set by `glue-forge-controller.nix`): on when it does, off
otherwise. Setting it `true` on a forge host without the controller leaves
the actions unresolved, because nothing seeds the mirrors; evaluation
warns about it.
<!-- vale write-good.Passive = YES -->
The **swarm controller** seeds the mirror in its swarm-wide forge-objects
pass (`swarm-controller/src/forge/objects.rs`), beside the orgs it ensures.
`glue-forge-controller.nix` forwards the effective mirror list as
`SWARM_CONTROLLER_FORGE_MIRRORS` in the `swarm-controller` service
environment (JSON-encoded `[{upstream, dest}]` list). It reads the list of
the host the controller runs on: mirrors declared on any other host aren't
seeded, and evaluation warns about them.
**General-purpose mirrors**: you can pre-seed any external repo as a
pull-mirror via `services.hyperhive.deploy.forgejo.mirrors`:
```nix
services.hyperhive.deploy.forgejo.mirrors = [
{ upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
{ upstream = "https://github.com/example/tool"; dest = "mirrors/tool"; }
];
```
The controller creates each entry as a real Forgejo pull-mirror — not a one-off
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
request), so a DNS blip during that sync will propagate back to the
runner as a hard `git clone` failure. The controller
autocreates the `<owner>` org in `dest`. Keep mirror dests out of the
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
to avoid provisioning collisions.
## Security: unsandboxed builds and trusted contributors
**The runner should only run CI for trusted contributors.** The security boundary is weaker than it looks:
### What unsandboxed builds mean
nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
A malicious `default.nix` or build script in a PR can therefore:
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge by name at `https://<forge.domain>` (no admin credentials) and **can't** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (a firewall rule drops bridge→127.0.0.0/8 traffic).
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only `swarm-ci-runner-register.service` loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read `.runner` and act as this runner; it can't issue admin API calls.
Note: `nix flake check --no-build` (eval-only) reduces the exposure but doesn't eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The CI workflow's `nix flake check` job builds every check derivation, which is the higher-risk path.
### Mitigation
For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.
For repos with external contributors or fork PRs:
- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.
## Host store maintenance (recommended)
The CI runner builds derivations through the **host** nix-daemon — the
`swarm-ci-runner` container shares the host store and has no daemon of its own. Build
outputs accumulate in `/nix/store` with no automatic collection, and a busy
CI day can fill the disk until every job fails fast with `ENOSPC`.
Store GC is a **host-level** concern, so it belongs in the host's own NixOS
configuration, not in the hyperhive service modules — a single service should
not reach out and change the host's global nix-daemon options. Add the
following to your host config:
```nix
{
# Daily GC: delete store paths not referenced by a live root and older
# than a day. Keeps the store bounded between builds.
#
# `randomizedDelaySec` because a fixed daily time lands on the same wall
# clock as whatever else runs nightly. A collection that overlaps a CI
# job doesn't fail it — the runner builds through the host nix-daemon,
# so an in-flight build's temp roots are protected — but the first build
# after a collection rebuilds what the collection deleted, so a run that
# lands just behind one pays for it in wall time.
nix.gc = {
automatic = true;
dates = "daily";
randomizedDelaySec = "45min";
options = "--delete-older-than 1d";
};
# Disk-pressure GC: when free space drops below min-free mid-build, the
# daemon collects garbage up to max-free before continuing. This is the
# real-time net the daily timer can't provide — a same-day build burst is
# what fills the disk. Tune to your disk size.
nix.settings.min-free = 20 * 1024 * 1024 * 1024; # 20 GiB
nix.settings.max-free = 50 * 1024 * 1024 * 1024; # 50 GiB
}
```
**Measure a collection on your own hosts before relying on that stagger.**
`randomizedDelaySec` spreads the _start_, not the run, and the sweep over
unused `.links` entries dominates the run — which scales with the size of the
store, not with what this collection deletes. On a large store it can take most
of an hour, so two hosts both set to `daily` may overlap however much jitter you
give them. `journalctl -u nix-gc.service` reports each run's wall clock.
**Remote builders:** if CI dispatches builds to a remote builder (for example via
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
store, so apply the same GC config wherever the builder runs —
GC on the coordinator host won't reclaim space on the builder.
⚠️ Two stores also means the temp-root protection above **doesn't span
them**. Each daemon holds roots only for the builds it runs, so a
collection on the _builder_ knows nothing about a coordinator build
waiting on a path there. Stagger the two hosts' GC windows rather than
giving them the same `dates`, and prefer `min-free`/`max-free` on the
builder — disk-pressure collection at least runs while nothing is mid-copy.
## References
- `nix/host-modules/swarm-ci-runner.nix`: runner configuration, token fetch, re-registration step, container setup.
- `swarm-controller/src/forge/runner_token.rs`: publishes the registration token.
- `.forgejo/workflows/ci.yml`: workflow definition.
- `docs/5-project/process/gotchas.md`: nix sandboxing limitations in containers.

535
docs/4-implementation/scheduler/coordinator.md Normal file
View file

@ -0,0 +1,535 @@
# hive-c0re coordinator internals
Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems.
For the public API surface (dashboard, socket protocol) see `docs/5-project/process/conventions.md`
and `docs/2-handbook/agent-lifecycle/persistence.md`.
---
## Job queue
**For the job queue as a general idea — graph of steps, shared resource
slots, no hive-c0re specifics — see [`jobq.md`](jobq.md) instead.** This
section covers the concrete node inventory hive-c0re builds on top of that
engine, plus the internals (module layout, scheduler mechanics,
resource/lease semantics).
hive-c0re submits every container operation (rebuild, first-spawn, power
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of basic nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers.
Exceptions like graceful stop, deferred starts, and the boot sweep
need no bespoke code paths — each takes the shape of a DAG built
from the same basic nodes.
### Two levels: DAG and node
The **DAG** is the unit of cancel and the
dashboard group; the **node** is the unit of scheduling / execution /
build-log. Deps are intra-DAG edges only (`AfterOk` by default:
the dep must succeed, a failed/cancelled dep cancels the dependent —
cancel-downstream). Cross-DAG ordering comes from the per-agent lease,
never from edges between DAGs.
A DAG is **declared, not described**: a template builds it through
`hive_jobq::JobBuilder`, naming each node it depends on via the handle
`b.node(kind)` handed back, and the builder inserts the nodes itself. A handle
only exists for a node already declared, so every edge points backwards and
nothing can write a cycle down — there is no submit-time validation pass,
because there is no malformed spec to reject.
### Node inventory (primitives)
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
| Node | Wraps |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Prebuild` | download the swarm's closure for the agent into the store while the container keeps serving (its meta preamble is the upstream `MetaSync` node). With no closure to deploy it fails here, before the stop. Skipped when the container is already down; `Swap` downloads it instead |
| `Swap` | drop-in rewrite + profile-swap onto the swarm's closure (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
| `Create` | first-spawn `nixos-container create` proper; assumes the upstream `Provision` node already registered the agent; downloads the closure the swarm built for the agent and creates the container from it (with none to deploy it fails and creates nothing), then marks the closure applied |
Cheap — no build slot:
<!-- vale write-good.Passive = NO -->
| Node | Behavior |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MetaSync` | the rebuild's preamble — rebuild-dir prep, the archetype refresh and the agent-set refresh (`agent_set::refresh`), then reads the agent's closure record once and pins its store path for `Prebuild` and `Swap`; fails when the swarm has no closure to deploy; reads nothing from the forge. Declares no resources |
| `Provision` | first-spawn pre-create provisioning — the state subvolume, agent-set refresh (`agent_set::refresh`); runs ahead of `Create`. Store-only, no container yet |
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev markers, forge/matrix sync, manager kick, rescan; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
| `PauseSignal` | write the pause marker + mark `pause_pending`. No kick, unlike `Signal` — the harness's between-turns poll is already responsive enough, and `Signal`'s kick-message body ("you were just (re)started") would be actively misleading here |
| `PauseDrain` | await the harness reporting `PauseAcknowledged`, bounded timeout; best-effort like `Drain` |
| `DestroyContainer` | `nixos-container destroy` + un-registration (drop from the roster, clear the ephemeral runtime dir). Runs downstream of a `Stop`, so deliberately excluded from `takes_container_down` — the container is already down by the time it claims |
| `PurgeState` | the `purge = true` half of a destroy: delete the agent's state subvolume (via hive-priv) plus its state dir. Own node because it's conditional and the irreversible step |
| `DestroyBookkeeping` | the post-destroy tail — meta sync, drop the power intent, notify the manager, rescan, re-emit the tombstone. Same split rationale as `RebuildBookkeeping`/`Swap`. The destructive work is `PurgeState`'s |
| `SetWanted` | write the durable power intent (`wanted = Up`/`Offline`) as the head node of a power-op DAG. Takes the agent lease even though it's a store write, so the intent write and the tail `Reconcile` are atomic per-agent — two racing power ops can't clobber each other's intent before either reconciles |
| `EmitRebuilt` | tail of a rebuild — emit the agent's `Rebuilt` manager event (ok/fail per outcome, nothing on cancel). One node per agent _and_ per outcome |
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
| `KnowledgePull` | `/knowledge` pull (`knowledge::pull`): at boot (commits that landed while `hive-c0re` was down), on the swarm knowledge-changed event, and hourly as a fallback. Holds `Resource::KnowledgeTree` (capacity 1), so two pulls never overlap on the working tree; each trigger queues its own pass, which waits for the resource if one is already live. Agentless |
| `WantedPull` | pull of the agent set the swarm controller declares for this hive (`wanted::pull`), converging the agents it names. Runs at boot, waiting up to a minute in total for the queue connection, and again after every attach of the wanted-state watch (which retries with a backoff when the attach fails or the stream ends); the deploy event (`swarm_status`) and the watch are the fast path, this repairs a missed one. A first deploy already queued or running for an agent isn't queued again. Agentless |
<!-- vale write-good.Passive = YES -->
Nothing in the queue commits to a meta repo: the hive has none to commit to,
and `agent_set::refresh` serializes itself with its own mutex.
### Every operation as a DAG
The `stop` / `start` power ops write the durable `wanted` intent via a head
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
it writes _no_ intent (no `SetWanted` head) — it bounces the container and
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
deliberately stopped agent isn't forced back up by a hive-wide restart. The
hive-wide power ops — `restart`, `stop`, and `start` — take an agent _list_:
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
subgraph each (independent roots, run concurrently on their own leases), not
N separate DAGs.
**`job_queue/power.rs` builds these dynamically from each agent's live
running state** (an async `lifecycle::is_running` read) — that's why they
live there rather than in the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile`
(convergence guarantee — cheap, noops when already converged) is ALWAYS
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
state-conditional — skipped for a _down_ agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
closes the TOCTOU window: if an agent flips state between the `is_running`
read and node exec, the tail `Reconcile` still converges it in-DAG (with
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
sweep. `start` folds the per-agent declared-closure upgrade in (a _down + stale_
agent's subgraph is a rebuild-then-start).
```text
rebuild(a): MetaSync(a) → AgentWindow(a){ Prebuild(a) ∥ [Signal(a)→Drain(a) if graceful]; both →(after-ok) StopForUpdate(a) → Swap(a) →(after-ok) RebuildBookkeeping(a) } →(after-any) Reconcile(a)
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
spawn(a): [wanted=Up at deploy] Provision(a) → Create(a) → WriteDropin(a) → Reconcile(a)
boot: «graceful rebuild subgraph» per stale agent (closure marker ≠ the record's store path);
plus Reconcile(a) for every drifted agent (all ONE DAG)
```
Notable collapses:
- **`rebuild` is one uniform shape** — no `was_running` branch.
`StopForUpdate` noops when already down; the tail `Reconcile` autonoops the
start when `wanted = Offline` (a rebuild of a deliberately stopped agent
leaves it stopped).
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
terminal ok _or_ fail, bringing a wanted-up agent back on its old config.
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
whole-hive graceful stop fires every agent's signal immediately and all
drains overlap; each DAG's tail `Reconcile` does the actual stop.
- **The boot sweep is one DAG**: it declares one graceful `rebuild` subgraph
per stale agent, each an independent root, beside a `Reconcile` per drifted
agent. Not child DAGs — one DAG, no `parent_id`.
### Desired-state (spec vs status)
Per-agent power _intent_ — `wanted: Up | Offline` — is durable as the
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
`container_view` remains the observed _status_; `Reconcile` nodes converge the
two. Setting `wanted` is never a queued node: the power layer
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
Every operator power surface —
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
rides the queue through that power layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts
left are the gateway's (no lease, no
harness). Cancelling a still-queued power DAG reverts `wanted` to the
observed state — a cancel means "don't do it," not "do it later." hive-c0re
seeds agents without a row from observed state on first touch (running ⇒
`Up`); destroy removes the row.
The admin-socket responses carry the submitted DAG ids; `hivectl` polls
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
glyph, template, agent, node chain — so
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
non-zero). Nodes appended in-DAG (a `Reconcile` fanning its `Start`/`Stop`)
join the same DAG, so they surface under that DAG's id in the same loop — no
separate child DAGs.
### Scheduler semantics
<!-- vale write-good.Passive = NO -->
A node is **ready** when it's `Queued`, every dep is satisfied, and its
resources are free. Resources:
<!-- vale write-good.Passive = YES -->
1. **Build slots** — `services.hyperhive.deploy.hive-controller.buildSlots` permits (default 1),
held by nix-heavy nodes for the node's duration.
2. **Per-agent lifecycle lease** — keyed on the **node's** agent (agent is
per-node; a DAG can span agents) and globally exclusive per agent across
all DAGs: acquired either at a container-affecting node (`SetWanted`,
`Reconcile`, `WriteDropin`, `Create`) or at a **brace** (`AgentWindow`) on
behalf of a whole coordinated subtree; held by the owning
DAG until it's terminal, so two DAGs never interleave container ops on the
same agent. A DAG touching multiple agents holds one lease per agent.
(`SetWanted` is a store write, not a container op, but takes the lease anyway
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
`Prebuild`, `Provision` — they
touch the store, not the running container, which is exactly why a
stop can land while another DAG's prebuild is still downloading. Also exempt, for a different
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
container, but their `AgentWindow` brace holds the lease above them — see
_Braces_ below.
#### Braces
Templates otherwise declare a resource on **every** node that needs it, even
when a parent already holds it, so the requirement belongs to the node rather
than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
exception: a pure-resource-holder root that declares on behalf of a subtree
coordinated with itself, whose members then declare nothing.
It's forced rather than stylistic. Declaring a resource means _"I need this
exclusively"_, and the agent lease is single-unit — so **two siblings that both
declared it could never run concurrently.** For a subtree whose whole point is
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
the requirement truthfully on every node and running those nodes in parallel are
mutually exclusive. One holder above them speaks for the subtree.
This is the opposite of the failure the declare-your-own rule exists to prevent,
not a relapse into it: there the requirement was _implicit_, inferred from a
node's kind and true only by accident of placement. Here it's explicit, on one
node, with the omission below it documented on the brace itself.
Two consequences worth knowing:
<!-- vale write-good.Passive = NO -->
- **Flattening a chain under a brace is safe.** A brace (`AgentWindow`)
holds the lease continuity for the stop chain's subtree: as independent
siblings, `Signal`, `Drain`, and `StopForUpdate` would each acquire the
lease separately and leave a gap another DAG could claim the agent in,
mid-bounce.
- **Observability is unaffected.** `running_transients` keys off a node's
_payload_ agent, not off a declared lease edge, so every child still lights its
own dashboard pill and still reports its own `takes_container_down` to the
crash watcher. A brace itself reports `false`: it parents the stopping nodes
but doesn't stop anything, and claiming otherwise would widen crash
suppression across the build and tail, where a vanished container is still a
real crash.
<!-- vale write-good.Passive = YES -->
Among simultaneously ready nodes competing for a resource, DAG-submit order
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
created on lease acquisition and dropped when the DAG settles terminal.
The queue is in-memory only and lost on hive-c0re restart — deliberate:
desired state is re-derived at boot from the DB + rev markers (see _Boot
reconcile_), so there is no durable-recovery machinery to go wrong.
### Cancel, history
Cancel only applies to DAGs that are still fully queued (an in-flight nix build isn't
interruptible).
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
`Cancelled` / `Done`. The snapshot retains the 50 most recent terminal
DAGs — a flat cap over the whole sorted list, not per template, since
the dashboard renders one recent-builds list and one number bounds it.
### Wire shape
`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
payload.** It says _the queue changed_; a client that wants to know how
re-fetches `GET /api/jobq/graph`.
That endpoint serves the graph generically (`hive-jobq-wire`): every node
carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
string — the kind _is_ the phase label, there is no separate sub-step
string) and free-form `data` for what only some kinds have (`agent`,
`inputs`, `build_log_id`). Group roots ride as ordinary
nodes, so a group's state is just the root's own `state`.
No group-level `agent` exists — agent is per-node, so one group can
span agents; consumers derive a group's agents from its nodes. Build logs
are likewise **per-node**: the dashboard renders the node tree and keys the
live-log panel off the running node.
The event carries no payload by design: shipping a typed projection of
the whole queue in the event itself would be a second rendering of the
same graph that someone has to keep in agreement by hand with the
endpoint every consumer actually reads. Telling a client _when_ to refetch is
the event's whole job.
---
## Container view
`container_view.rs` maintains an in-memory snapshot of every nixos-container's
systemd service state. It's polled on coordinator startup and re-scanned after
every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects
the actual container status without a live `nixos-container list` call on each
render.
---
## Boot reconcile
On startup, `auto_update::run` classifies every agent by closure freshness (the
per-agent `.{name}.closure` marker under `/var/lib/hyperhive/applied/` vs the
store path the agent's `agent-closures` record declares) and persisted
`wanted` intent, then:
1. **Config path** — submit one `Boot` DAG holding a graceful `Rebuild`
subgraph for each stale agent whose `wanted = Up` (name-sorted). This is how
a hive that missed the swarm's deploy request catches up. Stale but wanted-offline
agents get no boot-time nix work — their rebuild happens on their next start
(the start submit path upgrades a stale start to rebuild+start). Each
rebuild's tail `Reconcile` brings the agent (back) up, covering both the
running-stale and stopped-but-wanted-up cases. An agent whose declaration
the hive can't read or compare (no queue connection, no record, a closure the
builder didn't push, a running closure it can't read) rebuilds nothing. An
agent with no marker takes the closure its container profile points at as the
marker, so a first boot after the marker arrived rebuilds exactly the
agents that run something else than declared, each holding a build slot.
2. **Power path** — every agent whose observed state drifted from `wanted`
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
Booting with every agent on its declared closure performs no rebuild — only
reconciles. After the boot pass, `auto_update::run` repeats the rebuild half
every five minutes. It skips an agent with a rebuild waiting or running
(`JobQueue::rebuild_live`) and one whose declared store path this process
already attempted, so a closure that fails to deploy gets one try per process.
The boot log line records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. `auto_update::run`
seeds agents without an `agent_power` row from observed state during
classification (the one-time migration; thereafter the DB is authoritative).
## Meta flake
Agents build from the swarm's `internal/meta` flake, which swarm-controller
commits and the swarm builder builds from (see
[Config changes](../../2-handbook/agent-lifecycle/config-changes.md#the-swarms-meta-flake)).
The hive keeps no flake of its own and deploys the closure the swarm builds
from it, as the agent's `agent-closures` record declares (`attic/closure.rs`).
`agent_set.rs` keeps the
host-side files that follow the agent set (`topology.json`, the gateway's
agent-socket map and nginx include) in line with the live agents, and
`meta.rs` removes the permission seed files once the swarm has them, and
`migrate.rs` deletes the legacy `meta/` repo's leftover `.git`.
---
## Container lifecycle (`lifecycle.rs`)
Every container operation ultimately calls into `lifecycle.rs`. Two paths exist:
**rebuild** (existing container) and **spawn** (first-time creation).
### Rebuild path (existing container)
Goal: apply the new system profile and any `EXTRA_NSPAWN_FLAGS` / drop-in changes
in a single start, with minimum downtime.
`nixos-container update` only runs `systemctl reload container@<c>` when the
container is already up (per `isContainerRunning` in `nixos-container.pl`). Stopping
first turns `update` into a boot-style operation: it builds + `nix-env --set`s the
new profile and skips the in-container `switch-to-configuration`. The subsequent
`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
one go, rather than the double-bounce a live `update` would trigger.
Sequence for a rebuild DAG (each step is its own queue node):
1. `MetaSync` — rebuild-dir prep, the archetype refresh, the agent-set
refresh, and the pin of the swarm's declared closure (one read of the record;
`Prebuild` and `Swap` use the pin). Short, and it declares no
resources.
2. `Prebuild` — download the swarm's closure for the agent **before** stopping.
The container keeps serving the previous generation while the download
happens out-of-band. A missing or unpullable closure fails the DAG here,
before `Swap` touches the running container, and the container keeps
running. Skipped when the container is already down — a stopped agent has
no uptime to preserve, so `Swap` downloads it.
3. `StopForUpdate` — bring the container down (noop when already stopped).
4. `Swap` — profile-swap onto the swarm's closure (near-instant after the
prebuild), then it writes the closure marker.
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
in-container activation script transitions old → new. Holds no build
slot, so the next DAG's `Prebuild` overlaps the container boot.
### Cold-start fallback
`start` after `update` can exit non-zero when packages are **removed** between
generations: the old-generation activation script references units that no longer
exist in the new closure, causing systemd to exit non-zero. The container may be
half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
`lifecycle::start_with_fallback` preserves and surfaces both errors if the
cold-start also fails. Every `Reconcile` node's start action uses this
fallback.
### Spawn path (new container)
For a first-time `create`, `nixos-container create` is atomic: it leaves no container
record to clean up if it fails. The `Create` node downloads the swarm's closure first,
so a spawn with no closure to deploy fails before it creates the container. Sequence:
download the closure → `create` from it → write nspawn flags → `systemctl daemon-reload` → `start`.
---
## Host-level resource + performance options
A handful of `services.hyperhive.deploy.hive-controller.*` options tune container resource
limits, build parallelism, and first-spawn latency.
### Build slots
`buildSlots` (default `1`) sets how many closure-fetching job-queue nodes
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
concurrently. The default serializes all of them; raise it on hosts with
the bandwidth and disk to take more agents at once. Per-agent correctness is independent of the count —
each agent's container-affecting ops serialize on its lifecycle lease
regardless.
### Container resource limits
`agentCpuQuota` and `agentMemoryMax` map directly to systemd
`CPUQuota=` and `MemoryMax=`. hive-c0re writes a
`container@h-<name>.service.d/` drop-in file on each spawn and
rebuild, so changes take effect on the next lifecycle op without
requiring a host rebuild.
The answer per agent and per field is, first match wins:
1. the swarm's cap for the agent (see below);
2. `agentCpuQuota` / `agentMemoryMax`.
A swarm cap replaces the hive's answer, so a hive can't raise an agent
above it. Like the rest of the drop-in, a changed swarm cap reaches a
running agent at its next spawn or rebuild. A hive has no way to set a cap
for one agent: you set caps per agent in the swarm UI.
The container can't read its own cap, so `write_dropins` also writes the
effective `MemoryMax=` into the agent's runtime dir
(`/run/hive/memory-max-bytes` inside) before every start: a byte count, or
`infinity` for no cap. A percentage becomes bytes the way systemd resolves
it, as that share of the host's physical memory rounded down to a page. A
value systemd ignores (zero, over 100%) is `infinity`. When hive-c0re keeps
the last drop-in because it can't read the swarm caps file,
the file carries that drop-in's `MemoryMax=`.
At boot the container derives two values from a byte count:
`BUN_JSC_forceRAMSize` (75%, claude-code's heap ceiling) and `MemoryHigh=` on
`hive-subagent-daemon` (two thirds). Neither is in the agent's build, so a
changed cap reaches them at the next container start without a rebuild.
`infinity` sets neither. A missing or malformed file sets neither as well, and
`hyperhive-memory-cap-check.service` logs a warning naming the path in the
container's journal. The container's `MemoryMax=` still applies.
When the container reaches its cap, the kernel kills in this order:
1. a subagent run (`oom_score_adj` +1000);
2. a bash task (0);
3. the harness and the agent's own `claude` (-500).
Each subagent run and each bash task has its own cgroup with
`memory.oom.group=1`, so the kernel kills all of a run or task and nothing
outside it. Inside the container these are
`/sys/fs/cgroup/system.slice/hive-subagent-daemon.service/runs/<name>-<n>/`
and `/sys/fs/cgroup/system.slice/hive-bash-daemon.service/tasks/<id>/`; the
daemon itself sits in `daemon/` beside them. A daemon that can't set these
up logs one warning at start and runs everything in its own cgroup; subagent
runs still get +1000.
<details><summary>Implementation: swarm caps</summary>
swarm-controller keeps the caps in the `agent-limits` KV bucket, one key per
agent and no hive in it, so an agent that moves to another hive keeps its
caps. swarm-controller alone writes it:
| method | path | body |
| -------- | ---------------------------- | --------------------------------------------------------------------------------------------- |
| `GET` | `/api/agents/<agent>/limits` | —; answers the stored caps, `{}` when there are none |
| `PUT` | `/api/agents/<agent>/limits` | `{"cpu_quota": "400%", "memory_max": "8G"}`, every field optional, replacing the stored value |
| `DELETE` | `/api/agents/<agent>/limits` | —; answers 204 |
A stored value the controller can't decode, such as one a newer build wrote,
answers `{}`, with a warning in its log, until the next `PUT` replaces it.
`PUT` refuses an agent no hive's wanted state declares, unknown fields, and
any value systemd would ignore, which would leave the agent uncapped. A
percentage has at most two decimals, no leading zero in its whole part (it
refuses `08%`), and is above zero, up to `100%` for
`memory_max` and `21474836.47%` for `cpu_quota`. A size is digits with an
optional fraction and an optional `B`, `K`, `M`, `G` or `T` (upper case), at
least one byte and below 2^64 - 1 bytes. The controller stores each value as
given.
Every 30 seconds hive-c0re (`workers::swarm_limits`) reads the key of each
agent it has and stores the result in `meta/swarm-resource-limits.json`. Each
value passes the validators first. One bad
value or a failed read leaves the file unchanged, and repeated failures raise a hive
warning (the `hyperhive.hive.warning` metric and the `hh-hive-warning` alert).
A hive's queue grant reads every key of the bucket
(`$JS.API.DIRECT.GET.KV_agent-limits.>`) and writes none. Scoping it to the
hive's own agents would need a roster in the auth responder.
</details>
The same drop-in carries `CPUWeight=` / `IOWeight=` from
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
setting: the quota and the memory max are **hard caps** that throttle
an agent even on an idle host, while the weights are cgroup
v2 **relative shares** that only decide who yields _under contention_.
A low-weight container still gets the whole machine when nothing else
wants it.
| Option | Default | Description |
| ---------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `services.hyperhive.deploy.hive-controller.agentCpuQuota` | `"200%"` | CPU cap per agent, as a percentage of one core (`"200%"` = 2 cores). Raise if agents hit CPU limits during builds or heavy tool use. |
| `services.hyperhive.deploy.hive-controller.agentMemoryMax` | `"4G"` | Memory cap per agent. Raise for agents that run large nix builds or hold big in-memory data. |
| `services.hyperhive.deploy.hive-controller.agentCpuWeight` | `80` | `cpu.weight` share per agent, `1`–`10000` or `null` to omit the setting. Kernel default is `100`, so `80` makes agents yield. |
| `services.hyperhive.deploy.hive-controller.agentIoWeight` | `80` | `io.weight` share per agent, same range and `null` handling. See the caveat below — it's a no-op on many hosts. |
Two things to know about the weights:
- They're **hive-wide** — unlike the caps there is no per-agent
setting, so every agent carries the same value and the weight does _not_
rank agents against each other.
What `80` buys is that agents yield to everything **not** on this
drop-in path: host services (nginx and dnsmasq among them) and the
infra containers (`hive-forge`, `hive-gateway`, `hive-matrix`), which
stay at the kernel default of `100`.
- `IOWeight=` is only honoured when the backing device runs the BFQ
scheduler or has blk-iocost QoS enabled. On a host using
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
and the kernel ignores it — harmless, but it will measure as nothing.
Check with `cat /sys/fs/cgroup/io.cost.qos`, and set the option to
`null` if you would rather not write a setting nothing reads.
For a hive-wide cap across all containers together, set
`systemd.slices.machine.serviceConfig.CPUQuota` in your NixOS
config — all nspawn containers live in `machine.slice`.
---
## See also
- `docs/2-handbook/agent-lifecycle/config-changes.md` — config PRs, meta flake, system messages
- `docs/2-handbook/agent-lifecycle/persistence.md` — SQLite schema, state-dir layout
- `docs/5-project/process/conventions.md` — wire protocol, recipient sentinels
- `docs/2-handbook/agent-lifecycle/agent-roster.md` — the agent roster and privilege

61
docs/4-implementation/scheduler/jobq.md Normal file
View file

@ -0,0 +1,61 @@
# The job queue, for operators
Long-running work runs through a job graph. The swarm controller keeps one
for swarm-level work — creating an agent's identity, forge user and config
repo. Each hive's hive-c0re keeps its own for container operations —
rebuild, first-spawn, power changes. This page explains
what the job queue _is_, as a general idea, independent of what either uses
it for. For the hive-c0re step catalogue and the engineering internals
(scheduler, leases, resource windows) see [`coordinator.md`](coordinator.md)
instead.
## What the job queue is, in the abstract
"jobq" is a generic engine for running many interdependent jobs under
limited concurrency — it has no idea what a "container" or a "rebuild" is.
Two ideas are all there is to it:
- **A job is a small graph of steps**, not one opaque blob. Steps can
depend on each other (this step only starts once that one finishes), so
a big operation is really a short, ordered sequence — not a single
black box that's either "done" or "not done."
- **A step can need a shared resource**, which only so many steps can hold
at once (a "slot"). If every currently running step already holds the
slots it needs, a new step that wants the same one waits its turn —
that's the whole reason things queue instead of all firing at once.
The engine's whole job is: whenever a step's ordering and resource needs
are both satisfied, run it. It has no opinion on what the steps _do_ —
that's supplied by whoever builds the graph. The engine is generic: the
swarm controller and hive-c0re each build their own graph on it.
## Watching it happen
Two views, one per graph, drawn by the same component: the swarm UI's
`/jobs` page shows the swarm controller's graph, and the hive dashboard's
**R3BU1LD QU3U3** tab (see
[`web-ui/dashboard.md`](../../2-handbook/web-ui/dashboard.md)) shows that hive's. Each
**row** is one job; the rows nested under it are that job's steps, in
order (occasionally a couple run side by side). A step shows one of:
<!-- vale write-good.Passive = NO -->
| Glyph | Meaning |
| ----- | ------------------------------------------------------- |
| `⏸` | queued, waiting its turn |
| `▶` | running |
| `◐` | its own work is done, waiting on a step nested under it |
| `✔` | finished successfully |
| `✖` | failed |
| `⊘` | cancelled |
| `·` | skipped (not needed for this run) |
<!-- vale write-good.Passive = YES -->
A step that isn't needed for a given run stays in the graph as `·` rather
than being absent from it, so the same kind of operation keeps a
recognizable shape run to run, whichever steps it actually needed.
⚠️ **The view hides `✔` and `·` by default**, so that full shape isn't what
you see first — tick them back on in the state filter. The selection is
part of the request, not a display toggle: hidden states are never fetched.

756
docs/4-implementation/scheduler/observability.md Normal file
View file

@ -0,0 +1,756 @@
# Observability (OpenTelemetry)
The swarm has one telemetry pipeline. The swarm's collector writes the
swarm's metrics and log stores and, if you name one, exports upstream; each
hive runs a collector of its own that forwards its agents' telemetry there.
What flows through it: each agent's Claude Code statistics (token usage,
cost, tool call counts) via Claude Code's built-in OpenTelemetry
integration, each agent container's journal, the hive tier's own
`hive-c0re.service` journal line for a detected container crash (see
[Container-crash alerting](#container-crash-alerting-hive-c0re)), and the
hyperhive metrics catalogued below.
## The two collectors
Telemetry crosses two collectors, and which one you configure depends on what
the host is:
| | runs where | receives from | does |
| ------------------------------------ | -------------- | --------------------------------- | ----------------------------------------------- |
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's stores and exports upstream |
| **hive tier** — always on | every hive | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential |
An all-local host runs both; the hop between them configures itself. A host that
runs the swarm's services without a hive runs only the swarm tier: the hive tier
labels a hive's telemetry outside its agents' control, and that host has no hive.
The swarm services there push to the swarm collector directly, each as its own
OAuth2 client.
```nix
services.hyperhive.deploy.swarm-otel.upstream = {
endpoint = "https://collector.example.com/otel"; # the upstream
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
};
```
## Enabling export
Every hive runs its collector and every agent exports to it, with nothing to
switch on: there is no per-hive or per-agent opt-in.
`upstream.endpoint` is where telemetry ends up after it leaves the swarm — optional,
because the swarm's own metrics store (`deploy.victoriametrics`) is a
destination in its own right. With both, telemetry goes to both. See
[`swarm/services.md`](../../2-handbook/swarm/services.md#metrics-victoriametrics--grafana).
**Telemetry leaves a hive exactly one way: through the collector that
runs on the host.** Agents never talk to `upstream.endpoint` themselves —
they export unauthenticated to a collector address only their own containers
can reach. They dial it as
`http://hive-collector.hive:4318`, a name identical on
every hive and every swarm, which each hive's own dnsmasq answers with its bridge IP, so an
agent's build carries no hive-specific address and no swarm domain. The otel module opens that
port on the bridge itself. That collector forwards to the swarm's
([`swarm/services.md`](../../2-handbook/swarm/services.md#telemetry-collector-otel)), the
single process holding the upstream credential and the only writer to the
swarm's stores. No agent holds a copy, and neither does the hive.
The hive collector reaches the swarm collector by its gateway name
(`swarm.otel.domain`, default `otel.<swarm domain>`) — the same DNS-and-CA-trust
shape every hive-to-swarm-service hop uses. On the host running the swarm
collector the hive's dnsmasq answers that name; elsewhere it resolves through
ordinary DNS.
⚠️ **The hive collector carries every bit of that hive's telemetry.** It
runs on the same host as the agents and restarts on failure. Telemetry isn't
the control plane, so degraded telemetry isn't degraded operation.
Both collector tiers queue outgoing batches on disk (`file_storage`) and retry
a failing batch until the receiving end accepts it, so a restart or a short
outage of the next hop delays telemetry instead of dropping it. The queue holds
at most 5000 batches. Two limits remain:
- A full queue makes the exporter refuse new batches and log the drop. The
journald receiver has already moved its cursor past a refused batch, so an
outage longer than the queue can hold still loses data, later and a bounded
amount.
- Retry has no time limit, so the exporter keeps resending a batch that the
store rejects with a retryable error, and enough of those fill the queue.
### Why two tiers
**Agents export only to the hive tier.** Exporting straight to `upstream.endpoint` would mean
every agent needs the credential — and the only place to hand it to an agent
container is somewhere the agent itself can read, its own claude settings
among them. `0600` protects a secret from other containers, not from the
agent it belongs to.
**The tiers stay separate on one box.** An all-local hive is a statement about
_where_ processes run, not about the shape of the deployment. A boundary that
disappears locally is one the local deployment stops testing.
### What the agent→collector hop is and isn't
**It has no application-level auth.** The receiver takes any OTLP that reaches
it; what bounds who can reach it's the firewall — `exposeHostPorts` opens the
port on the bridge interface only — "unauthenticated to a bridge address"
means _reachable from an agent container_, not _presents a credential_.
The consequence, stated because it's a choice rather than an oversight: **any
agent can push arbitrary OTLP, and it's forwarded on under the operator's
credential.** Neither tier can tell a container's genuine Claude Code stats
from anything else shaped like OTLP arriving on that port — including data
smuggled out in resource attributes on an otherwise-legitimate export.
That's a **different risk from the one the collector fixes**: an agent that
held the upstream credential could do everything above _and_ use the token
anywhere else. The collector removes the token and keeps the pipe. Agents are inside the trust
boundary (`docs/2-handbook/trust-boundary/security.md`: capability = accepted risk), so an agent being
able to _send_ is an accepted extension of that boundary — but it's not
closed by this design, and don't read anything here as closing it.
**The `agent` label is self-reported, and the hive→swarm authentication below
doesn't change that.** Treat it as a convenience for grouping dashboards, never
as evidence of which container produced a sample: any agent that can reach this
hive's collector can label its data as any other agent.
<!-- vale write-good.Passive = NO -->
**Logs ride the same hop, and a journal carries more than a counter does.** Each
agent container forwards its own journal through this port — every unit in it at
`info` and above, not an allowlist. That's the harness, the MCP daemons and
whatever a tool call spawned, so command lines and error text leave the
container, not just counts. Destination, credential and trust boundary are the
same as for any OTLP the agent sends.
<!-- vale write-good.Passive = YES -->
Worth spelling out, because two different hops are in play and only one of them
carries a credential:
- **agent→collector** (this section's hop) stays open on the bridge. Nothing
downstream can tell one agent's export from another's.
- **hive→swarm** authenticates. The swarm tier gives each hive its own receiver
and stamps `hive=` from whichever one accepted the sample, so _that_ label is
unforgeable — see [Authenticated ingest](#authenticated-ingest) below.
A verified `hive` is reachable and a verified `agent` isn't — and that falls
out of the topology rather than being a gap someone forgot to close. The swarm
runs one collector, and the mechanism gives it no finer grain: a bearer-token
check never reveals _which_ token matched, and a receiver reads request metadata
rather than the claims it authenticated with.
If you need per-agent numbers you can act on, take them from the agent's own
turn-stats rather than from a metric label.
### Authenticated ingest
The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
from whichever receiver accepted a sample. A hive therefore can't report
metrics as another hive, and can't relabel its own by editing what it sends —
the label isn't taken from the payload at all.
**Nothing to set where the hive holds its store identity.** The hive's collector
reads its secret from the swarm's secret store under the hive's own leaf
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client`, which authelia's mint, in its container,
writes, into `/run/hive-otel-oidc/` (tmpfs, so never on persistent disk). It
runs at boot and retries every 30s until the store answers; until then the
collector runs on an empty file and the swarm refuses its exports. When a
run lands a changed value it restarts the collector. A timer runs it again
every ten minutes, so a secret rotated later reaches the collector within
about ten minutes. On a hive that
runs the store the leaf is already there; elsewhere it's the credential you
place — see [secret placement](../../2-handbook/swarm/credentials.md).
To deliver the secret some other way, name the file. The fetch unit isn't
rendered, and the collector waits up to 120s for the file to appear:
```nix
services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";
```
Setting the option to `/var/lib/hive-otel-oidc/hive-<name>.secret`, the path
the fetch used to write, now counts as delivering the file yourself: nothing
refreshes it, and the build warns. Remove the setting to fetch from the store.
**No unauthenticated mode exists.** A hive always presents an identity, so a
missing credential is a build error rather than a quieter fallback — the
collector has no anonymous route to accept samples on, and every path it serves
belongs to exactly one hive.
Getting the secret wrong shows up as the hive's collector logging 401s from the
swarm tier and no metrics appearing for that hive.
⚠️ **`upstream.endpoint` must be valid for `upstream.protocol`.** The upstream exporter follows
`upstream.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an _address_: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
address"_. nix validates the collector's config at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere.
## Network access
Agent containers reach the host only on ports 80 and 443 (plus DNS). The
collector port needs nothing from you: the hive collector opens it on the bridge.
To let agents reach some other host-local service of your own →
[`networking/network.md`](../networking/network.md#reaching-host-services-exposehostports).
## Options reference
The nix modules (`nix/host-modules/otel.nix`, and `nix/host-modules/swarm-otel.nix`
for the `services.hyperhive.deploy.swarm-otel.upstream.*` export) generate every
`services.hyperhive.otel.*` option's full type, default, description and
example into the [options reference](https://hyperhive.darkest.space/options/)
(host options — `nix build .#docs-host` for a local render). The same module
fixes the collector port every hive and its agents share,
`services.hyperhive.swarm.hiveOtel.collectorPort`, and each hive's own
`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`.
That's the reference; this page covers what a per-option listing can't: the
two-tier architecture, the security model, and how the options interact.
## Built-in resource labels
The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container
wide**, for every unit and in `/etc/profile`, so every
process in an agent container exports to the hive's collector without any
per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
yourself from a tool call or a shell you open in the container.
Every agent's export therefore includes these resource attributes
automatically:
| Attribute | Value |
| -------------- | ------------------------------------------------------------ |
| `service.name` | `hyperhive-agent` (constant) |
| `agent` | agent logical name (for example `iris`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
The swarm's collector adds `hive`, the name of the hive whose receiver accepted
the push (see [Authenticated ingest](#authenticated-ingest)). The container
doesn't set it.
Append additional labels for one hive via `extraResourceAttributes` (see
option reference above). Agents get them at container start, so a change
reaches an agent at its next start, not through a rebuild. Pass custom
per-data-point labels with `hive-metric --labels` (see below).
## Log severity
Every journald receiver in the pipeline maps the journal's `PRIORITY` onto an
OpenTelemetry severity, so a line arrives in the log store already carrying its
level and `severity_text:ERROR` is a query you can write:
| `PRIORITY` | syslog | severity |
| ---------- | ------------------ | -------- |
| 0, 1, 2 | emerg, alert, crit | `FATAL` |
| 3 | err | `ERROR` |
| 4 | warning | `WARN` |
| 5, 6 | notice, info | `INFO` |
| 7 | debug | `DEBUG` |
Both tiers that read a journal do this — the agent container's forwarder over
its own journal, and the swarm collector over the host's — and both import the
same mapping from `nix/journald-severity.nix`, which holds the table once.
Note the direction. Syslog's priority counts **down** in urgency, where
OpenTelemetry's severity number counts **up**. Neither is a passthrough of the
other, and a pipeline that hands one straight to the other renders debug lines
as critical without erroring anywhere.
A record that reaches the store from somewhere other than a journal has no
priority to map. Claude Code's own telemetry is the live example: it exports
OTLP log records directly (`OTEL_LOGS_EXPORTER`, see
`nix/agent-modules/claude-settings.nix`) and sets no severity on them, so they
store as `Unspecified`. The **Log rows with no severity** panel on the
`hyperhive · logs` dashboard counts both cases and keeps them apart: lines that
had a priority and arrived without a severity anyway signal a broken mapping,
while lines that never had one signal nothing of the sort.
### Why Explore's level buttons need the datasource told
The stored field is `severity_text`, and Grafana's log-level buttons filter on
a field called `level` — a name no row here carries. That's not something the
pipeline can fix: VictoriaLogs' OTLP ingester names the stored field
`severity_text` unconditionally (v1.52.0, `app/vlinsert/opentelemetry/pb.go`),
and its ingest parameters have no `_level_field` sibling to rename it with.
`nix/host-modules/swarm-grafana.nix` tells the datasource instead,
provisioning it with a `logLevelRules` entry per severity in the table
above, each matching `severity_text` against one of
those values — which is the plugin's way of saying "the level lives in this
field." `logLevelRules` is the datasource plugin's own jsonData key and the
only level-related one it has — there is no single "the level lives in field
X" string to set, so mapping every level takes one rule each. A level button
then filters on `level` **or** the matching `severity_text`, and the store
answers the second half. Each enabled rule appends an `OR
<field>:<op>"<value>"` term to the query a level button emits, beside the
`level:…` term that matches nothing here. Selecting "info" goes from
```
level:contains_common_case("info","information","informational","notice")
```
to that OR `severity_text:="INFO"`, which the rows here do match.
`Unspecified` is deliberately left out of those rules. It's VictoriaLogs' own
rendering of an absent severity, and it's what the **Log rows with no
severity** panel counts — giving it a level would dress the missing data up
as a colour and retire the instrument that measures it.
#### Three ways to get a `logLevelRules` entry wrong, all of them silent
Grafana accepts any jsonData it doesn't recognise, so a bad rule provisions
cleanly and the affected level button goes on returning zero rows:
- `enabled` must be literally `true`, not merely not-false. The query
builder keeps rules on `rule.enabled` being truthy while the row colouring
path keeps them on `!== false`, so an omitted flag colours rows correctly
and leaves the buttons broken — working in the half nobody is looking at.
- `level` must be a canonical Grafana `LogLevel` value. The builder groups
rules by it and only ever looks up `critical`, `error`, `warning`, `info`,
`debug`, `trace`; `warn` and `fatal` are enum aliases that resolve to
other spellings and match no group.
- the builder compares `value` with `===`, so it must be the severity text
exactly as stored: the uppercase OpenTelemetry short names `overwrite_text`
writes in `nix/journald-severity.nix`, not that table's lowercase keys.
## Host-emitted container-resource metrics (hive-c0re)
<!-- vale write-good.Passive = NO -->
**hive-c0re itself** also exports each agent
container's resource load — the same cgroup gauges —
to this hive's own collector, exactly like an agent does and with
no separate toggle. These come from the host, not the in-container Claude SDK,
so they cover containers even when their agent is idle.
<!-- vale write-good.Passive = YES -->
Emitted via the OpenTelemetry Rust SDK, using the
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
metric names + the standard `container.name` attribute where a spec metric
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
(= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source |
| --------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
| `container.memory.usage` | `By` | gauge | `memory.current` |
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
| `hyperhive.container.storage.usage` | `By` | gauge | `state/` + `work/` + the root template (the live ephemeral layer isn't counted), measured by `hive-priv` every 30 min (custom — semconv only has `disk.io`; omitted while unknown: not measured yet, or the last measurement failed) |
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — no `rate()` needed) |
| `hyperhive.agent.unrestricted_internet` | `1` | gauge | the agent's `unrestricted_internet` flag in the swarm permissions this hive last read: `1` on, `0` off; a flag never set reads `1`. Omitted while the stored permissions file is unreadable |
| `hyperhive.agent.closure_pullable` | `1` | gauge | `1` when this hive can fetch the agent's latest closure from the swarm's binary cache and verify its signature, else `0` with a `reason` label (`no-record`, `not-cached`, `no-credentials`, `fetch-failed`, `bad-signature`; see [the builder docs](../swarm/builder.md#what-hives-do-with-the-cache)). Omitted before the first probe pass, every 5 minutes |
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
alongside the spec `container.cpu.time` counter for convenience). The `swarm`
label rides on the resource, and the swarm collector adds `hive` from the
receiver that accepted the sample. Any other label goes in the hive's
`extraResourceAttributes`.
<!-- vale write-good.Passive = NO -->
**An absent `hyperhive.agent.unrestricted_internet` series doesn't mean off.**
A hive that predates the metric exports no series at all, and neither does a
hive that can't read its stored permissions. Only a `0` means the flag is off.
Count the agents still on with
`sum({"hyperhive.agent.unrestricted_internet"} == 1)`; dotted names need the
quoted-name selector.
The export cadence is a fixed 60s. Transport is OTLP/HTTP
(JSON) to the hive collector's bridge address, with no auth header — that
first hop is unauthenticated for every producer on this host, and the upstream
credential stays on the swarm tier.
## Container-crash alerting (hive-c0re)
The crash watcher (`crash_watch.rs`, a 10s poll of every managed container)
classifies a container that stopped without an operator-initiated action as
a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
no broker event, no per-hive dashboard banner. Both of those used to exist;
this change removed them in favour of the log line.
The hive tier's own collector runs a `journald` receiver scoped to the units
in `services.hyperhive.otel.journalUnits`: `hive-c0re.service`, the `container@*.service`
units (the start-time rootfs check lines), plus
`squid.service` and `squid-selfcheck.service` (see [Egress proxy](../networking/network.md#egress-proxy-squid)).
A host that runs the cache without the swarm collector adds `swarm-bao-attic-secret.service`
and `swarm-bao-attic-tokens.service`.
A host that also runs the swarm collector names none of these: its whole-host journal receiver
ships them already, and the hive collector then has no `journald` receiver.
No bind mount is needed. Unlike an agent's journal, this collector already
shares a filesystem with the units it reads, because it's a host systemd
service rather than a container. It forwards the crash log line alongside the
rest of the hive's telemetry. A
swarm-level grafana alert, `hh-container-crashed` in
[`nix/host-modules/swarm-grafana.nix`](../../../nix/host-modules/swarm-grafana.nix),
matches it directly and pages the operator — no agent, and no per-hive
dashboard, in the loop.
Requires `services.journald.storage = "persistent"` (the NixOS default) on
every hive host — asserted at eval time.
## Agent-emitted per-turn metrics (`hive-agent`)
<!-- vale write-good.Passive = NO -->
The harness itself (`hive-agent`) exports one small set
of metrics per claude turn, recorded the moment the turn ends (not polled).
These are deliberately the fields Claude Code's own built-in export (see
above) can't know about — the harness's own wall-clock timing, what woke the
turn, its own outcome classification, the loose-ends backlog, session
boundaries, how full the context window is, the tools a turn called, the shell
commands bash tasks ran, and reminders. Token usage is **not** duplicated here;
that's already covered by Claude's own export, which has no tool-call counter.
Cost is here only for agents on an ACP runtime, which have no export
of their own — see [ACP-reported cost](#acp-reported-cost).
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Attributes |
| ---------------------------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------- |
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
| `hyperhive.agent.todos.open` | — | gauge | none — un-acked todos in the harness-local todo store (reminders excluded), recorded every 30 seconds |
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `AGENTS.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
| `hyperhive.agent.subagent_role.lines` | — | gauge | `role` (the file stem) — one series per role file in `subagent_roles/`, on the same ~15-minute tick |
| `hyperhive.agent.claude_usage.percent` | `%` | gauge | `window` (`five_hour`, `seven_day`, … as the usage endpoint names them) — polled every 5 minutes, **not** per turn |
| `hyperhive.agent.claude_usage.resets_at` | `s` | gauge | `window` — unix seconds at which that window resets; same 5-minute poll |
| `hyperhive.agent.context.used` | `{token}` | gauge | none — tokens in the context window at turn end, the numerator of the web UI's ctx% |
| `hyperhive.agent.context.size` | `{token}` | gauge | none — the context window those tokens fill, ctx%'s denominator |
| `hyperhive.agent.cost.usage` | — | counter | `model`, `currency` — ACP agents only, see below |
| `hyperhive.agent.tool.calls` | — | counter | `tool` — the tool's name, or `Skill:<skill>` for a skill; counted when a turn ends |
| `hyperhive.agent.bash.commands` | — | counter | `command` — the command name a bash task ran (the first real command, past `cd` and `sudo`), counted when a turn ends |
| `hyperhive.agent.reminders` | — | counter | `event` (`scheduled` or `delivered`) |
| `hyperhive.agent.first_turn.context` | `{token}` | gauge | none — input tokens of the first turn of a fresh session, the cold cost of the system prompt and `AGENTS.md` |
An agent with no `subagent_roles/` directory exports no
`subagent_role.lines` series. A role file deleted while the harness runs
keeps its last value until the harness restarts.
For the two `claude_usage` gauges the harness polls the Claude subscription
usage endpoint (`GET /api/oauth/usage`, the one behind claude's own `/usage`)
with the OAuth access token from the agent's own
`~/.claude/.credentials.json`. The harness only reads that token and never
refreshes it — claude owns refresh-token rotation. An agent with no OAuth
session (API-key backends, not yet logged in) or an expired token skips the
poll, so a gauge keeps its last value until the next successful poll — a
`resets_at` in the past means the paired `percent` is stale.
The two `context` gauges hold the two numbers the web UI divides for its ctx%:
the last turn's context tokens (input, cache read and cache creation), and the
API-reported window, else the model's default. A turn that parsed no usage
leaves both where the previous turn put them. The dashboard's **Context
window used by agent** panel (health tab) divides one by the other.
Resource attributes (`service.name`, `agent`, `swarm`) come from the
same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
section, and the swarm's collector adds `hive` — nothing extra to configure.
The export cadence is a fixed 60s, the same
as `hive-c0re`'s container-resource export above. That only controls how often
the harness flushes the batched points to the collector, not how often it
records them (every turn, always).
### ACP-reported cost
An ACP agent reports what its session has cost so far in the `cost` field of
its `usage_update` notifications — a running total, not a per-turn figure
(opencode sums every assistant message in the session). The harness counts
what each report adds to the last one and exports that as
`hyperhive.agent.cost.usage`, so `increase()` over it gives the money spent
in a range. It follows a few rules:
- A new session's first report counts in full, since the session started at
zero.
- A session loaded into a freshly started agent (after a harness restart or
an agent crash) has an unknown total until it reports one, so that first
report only sets the baseline — the turn it covers goes uncounted.
- A total lower than the previous one means the agent dropped part of the
session's history; it adds nothing and becomes the new baseline.
- Cost from a compaction the harness runs between turns counts toward the
next turn.
The value is in whatever `currency` the agent names; opencode always sends
`USD`, and the `hyperhive · agents` dashboard's ACP cost panels (cost tab)
filter on it. A claude agent never records this metric: claude's own export
already carries its cost as `claude_code.cost.usage`, and keeping the two
apart means no turn is ever counted twice. Add both names together for a
whole-swarm spend figure.
## Hive-scoped metrics (hive-c0re)
<!-- vale write-good.Passive = NO -->
hive-c0re measures everything above **per agent**, tagged with the hive it
runs in. It measures these five per **hive** instead, and they carry no
`agent` label — so a hive
that hosts no agents still reports, and "this hive is quiet" is
distinguishable from "this hive is gone." Select them with
`{hive!="",agent=""}`.
<!-- vale write-good.Passive = YES -->
| Metric | Unit | Kind | Meaning |
| -------------------------------------- | ---- | ----- | -------------------------------------------------------------------------------------------------------------------- |
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
| `hyperhive.hive.warning` | `1` | gauge | `1` per raised warning, with `kind` and `level` attributes; a cleared warning stops reporting |
| `hyperhive.hive.nix_store.utilization` | `1` | gauge | used fraction (0–1) of the filesystem that holds the nix store; no sample when the probe fails |
hive-c0re reports both levels of `hyperhive.hive.warnings` every cycle, `0`
included, so a healthy hive is visible as zeros rather than as missing
series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
detail behind it — `warn`-level entries mean "an operator should look" and do
**not** set `degraded`. `hyperhive.hive.warning` names the cause: the `kind`
label is one of a fixed set in the source (for example `disk_pressure`, a
failing sweep such as `swarm_status_publish`, `pending_logins`), and the message lives in the
`hive warning raised` line hive-c0re logs when it raises the warning.
The swarm Grafana (`nix/host-modules/swarm-grafana.nix`) alerts on these:
| Alert | Fires when |
| ----------------------- | ----------------------------------------------------------------------------------- |
| `hh-hive-degraded` | `hyperhive.hive.degraded` is `1` for 2 minutes |
| `hh-hive-warning` | any `hyperhive.hive.warning` series other than `disk_pressure` exists for 2 minutes |
| `hh-host-disk-high` | `hyperhive.hive.nix_store.utilization` stays above 0.85 for 10 minutes |
| `hh-host-disk-critical` | `hyperhive.hive.nix_store.utilization` stays above 0.95 for 2 minutes |
The disk alerts replace the old dashboard banner for the nix store, and
`hh-hive-warning` replaces it for every other warning. Disk pressure has its
own pair so the thresholds sit on the metric rather than on the registry
entry.
Same cadence, transport and resource labels as the container metrics above.
## VCS activity metrics (`swarm-controller`)
`swarm-controller` registers a single instance-wide Forgejo webhook (a
"global/system" hook, not scoped to any one org or repo) and counts commit
and push activity as deliveries arrive — occurrence-driven, not polled.
Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
counts of durable rows (issues, comments, repos), and Forgejo doesn't
store either a commit or a push anywhere as a row to count.
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ------- | ------------------- |
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
A push with zero commits (a branch delete, or a force-push that doesn't add
new commits) still increments `push.count`; `commit.count` only advances
when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_OTLP_ENDPOINT`), fixed 60s cadence
and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`
resource-attribute channel as `swarm-controller`'s other OTEL exporter (its
`hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that
crate's own doc comment) — `swarm-controller` sets `service.name =
swarm-controller` directly rather than reading it from the container
environment, since it's a standalone daemon, not a per-agent harness process.
Both exporters push through `swarm-controller`'s own authenticated client,
and the collector's `oidc/swarm` receiver checks the bearer. When a push
fails, `swarm-controller` logs `otel export: push to the collector failed`
at warn, with the kind of failure (`token`, `transport` or `status <code>`)
and the start of the response body. It repeats a persisting failure every
10 minutes with a count, and logs once at info when pushes succeed again.
## Review classifier result metrics (`swarm-controller`)
`swarm-controller` counts what review requests and review runs come to, from
the same exporter as the VCS metrics above. The classifier service adds the
retry counter beside its cost counters
([`review-classifier.md`](../swarm/review-classifier.md#observability-and-cost)).
| Metric | Unit | Kind | Attributes |
| ------------------------------------------ | ---- | --------- | ------------------------------------------------------------------------------------ |
| `hyperhive.review_classifier.requests` | — | counter | `outcome`: `queued`, `queue_failed`, `unparsable`, `deferred`, `started`, `reposted` |
| `hyperhive.review_classifier.verdicts` | — | counter | `outcome`: `approve`, `request_changes`, `no_verdict` |
| `hyperhive.review_classifier.checks` | — | counter | `check` (the rule's name), `result`: `match`, `no_match`, `unsure`, `no_answer` |
| `hyperhive.review_classifier.run.duration` | `s` | histogram | `outcome`: the verdict names, or `post_failed` |
| `hyperhive.review_classifier.ask.retries` | — | counter | `runtime`: `api_key`, `acp` |
A verdict means the classifier decided, before it posts the review: a run
whose post fails still adds to `verdicts`, and shows as `post_failed` in
`run.duration`. A repo with no rules file gets an approval with no model call and
counts as an `approve` verdict and run, with no `checks`.
`deferred` counts each time the queue hands over a request for a PR whose
review is still going, so a long review adds one per retry wait. `reposted`
is a request for a head already reviewed: it posts the earlier verdict and
adds to `verdicts` but runs no checks, so it adds nothing to `checks` or
`run.duration`. A request the queue refuses posts a comment and counts
as `queue_failed` and a `no_verdict`. `no_answer` is a classification that
failed, so a `no_verdict` run with a `no_answer` check is the usual cause of
a comment instead of a review. The `check` values are the names in a repo's
rules file, at most 10 per repo.
The `review-classifier` dashboard in Grafana charts these.
## Swarm builder metrics
`swarm-builder` exports what it builds through the OTLP receiver of its
container's own forwarder, which pushes to the swarm collector as the builder's
client. `nix/host-modules/swarm-builder.nix` sets
`OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` for the daemon. The exporter is the same setup as the review
classifier's: HTTP/JSON, cumulative, one export a minute, and nothing exported
without `OTEL_EXPORTER_OTLP_ENDPOINT`.
| Metric | Unit | Kind | Attributes |
| ------------------------------------------ | ---- | --------- | ---------------------------------------------------------------- |
| `hyperhive.swarm_builder.builds` | — | counter | `kind`: `pr`, `meta`; `outcome`: `success`, `failure`, `refused` |
| `hyperhive.swarm_builder.build.duration` | `s` | histogram | `kind`, `outcome`: `success`, `failure` |
| `hyperhive.swarm_builder.queue.depth` | — | gauge | none |
| `hyperhive.swarm_builder.queue.wait` | `s` | histogram | `kind` |
| `hyperhive.swarm_builder.publish.failures` | — | counter | none |
`kind` `pr` is an agent config PR's head and `meta` an agent built from the
swarm meta flake. Lock and archetype requests aren't counted. `refused` is a
request that waited out the one-build slot, which the builder then answers busy
without building, so it has a `builds` point and no `build.duration` point. A build that
can't start (no forge token) counts as a `failure` with a duration; a payload
that doesn't decode counts as a `failure` in `builds` only.
`build.duration` runs from holding the slot to the reply, including the cache
push of a meta build. `queue.wait` runs from the request's arrival to the slot,
or to the refusal. `queue.depth` is the requests waiting for the slot now,
exported every minute, 0 when idle, so it doubles as the liveness signal; the
`bao.client_cert.*` gauges below ride the same exporter.
`publish.failures` counts replies the queue refused to take, which the
controller sees as a missing answer.
## Store client-certificate metrics
Every service that logs in to the secret store with a client certificate can
report when that certificate stops being valid. Two observable gauges, one
data point each, unix seconds:
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ----- | ---------- |
| `bao.client_cert.not_before` | `s` | gauge | `cert` |
| `bao.client_cert.not_after` | `s` | gauge | `cert` |
`cert` names the identity (`client`, `controller`, `agent`, `review-classifier`,
`ci-metrics`, `builder`); the resource labels (`service.name`, `hive`, `agent`) say who
holds it. `swarm_secret_client::cert_metrics` (cargo feature `otel`) registers both gauges on the service's
existing exporter and reads `BAO_CLIENT_CERT` at every collection, so a
renewed leaf shows up on the next export. A file that's missing, or that isn't a
certificate yields no data point and a debug log line. The parser is
`swarm_secret_client::cert`.
| Service | `cert` | Exporter it rides |
| ------------------------- | ------------------- | --------------------------------------------------- |
| `hive-c0re` | `client` | the container-stats exporter |
| `hive-agent` | `agent` | the per-turn exporter, built on its first recording |
| `swarm-controller` | `controller` | the job-graph exporter |
| `swarm-review-classifier` | `review-classifier` | the cost exporter, built on its first recording |
| `swarm-ci-metrics` | `ci-metrics` | a push of its own at the start of each timer pass |
| `swarm-builder` | `builder` | the build exporter, started with the daemon |
Not covered yet: the other six Rust services that hold a store certificate
(`swarm-claude-login`, `swarm-notify-router`,
`swarm-nats-auth`, `swarm-logs-auth`, `swarm-metrics-auth`,
`swarm-matrix-ctl`) have no exporter, and the ten identities that only shell
units use report nothing. Neither does any CA certificate. Each leaf's own
window is all the gauges show, and a CA that expires before its leaves is
invisible to them.
The swarm Grafana alerts on the lifetime fraction:
| Alert | Fires when |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `hh-bao-client-cert-lifetime` | `(time() - not_before) / (not_after - not_before)` stays above 0.8 for 10 minutes, per `service.name`, `cert`, `hive` and `agent` |
The two gauges of a series carry identical labels, so MetricsQL divides them
one-to-one without `on()` or `group_left`. The query wraps each in
`last_over_time(...[15m])` so `swarm-ci-metrics`, which reports every five
minutes, keeps a series. A series that stops reporting resolves the alert
rather than firing it (no data counts as OK).
An agent leaf lives 7 days and renews at half its life, so a fraction over 0.8
means the renewal stalled about two days ago. The host leaves (`client`,
`controller`, and the others minted by `swarm-bao-pki`) live 3650 days and never
renew; for them 0.8 comes about eight years after minting. A guard on the days
left would suit them better and doesn't exist yet.
## Agent-emitted custom metrics (`hive-metric`)
Agents can push arbitrary labeled metrics to the same OTEL collector via the
`hive-metric` CLI tool, available in every agent container.
### Usage
```text
hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...]
```
- `<name>` — metric name (for example `tasks_completed`, `latency_ms`).
- `<value>` — numeric value (f64; integers and floats both accepted).
- `--type counter|gauge` — metric kind: `counter` (increasing sum, default) or
`gauge` (instantaneous point-in-time value).
- `--temporality delta|cumulative` — counter reporting mode (`counter` only,
ignored for `gauge`): `delta` (this call's own contribution, default — send
`1` each time and the collector accumulates) or `cumulative` (this call
reports the running total, which a stateless one-shot CLI can't track
itself).
- `--labels key=value` — extra per-data-point labels. Repeat the flag for
more than one.
`hive-metric` inherits the resource labels (agent, swarm, service.name)
automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them.
### Examples
```text
# Counter: one more task finished (delta is the default — no flag needed)
hive-metric tasks_completed 1 --labels phase=scan
# Gauge: current queue depth (absolute value — must use --type gauge)
hive-metric queue_depth 17 --type gauge
# Float gauge with multiple labels (instantaneous measurement)
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
```
### Error when an agent turns OTEL off
An agent whose own config sets `services.hyperhive.agent.otel.enable = false`
has no `OTEL_EXPORTER_OTLP_ENDPOINT` env var, and `hive-metric` exits with an
informative error message. No silently dropped metrics.
### Wire format
`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
`OTEL_EXPORTER_OTLP_PROTOCOL` setting. `hive-metric` forwards auth headers from
`OTEL_EXPORTER_OTLP_HEADERS` verbatim.
## Metrics temporality
The harness configures OTEL export with **cumulative** temporality by default
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
overriding Claude Code's default of DELTA. This avoids silent metric drops in
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a
delta-to-cumulative processor.
<!-- vale write-good.Passive = NO -->
**`hive-metric` counters are the one exception**, reporting delta by default
(see above) — programmatically set on the exporter, which overrides this
container-wide env var for that tool specifically. `--type gauge` is
unaffected either way; gauges have no temporality. The hive-tier collector
runs a `deltatocumulative` processor ahead of export, so a delta
`hive-metric` counter still lands in VictoriaMetrics as a cumulative
series — the standard `rate()`/`increase()` idioms work on it exactly like
any other counter in this system, no extra query needed.
<!-- vale write-good.Passive = YES -->