docs: restructure into topic subdirectories, collapse duplicated index
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
This commit is contained in:
iris
mara
parent
e4a22b4190
commit
07b62612b0
124 changed files with 301 additions and 377 deletions
238
docs/scheduler/ci.md
Normal file
238
docs/scheduler/ci.md
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
# hive-ci: Forgejo Actions Runner
|
||||
|
||||
The `hive-ci` module runs a Forgejo Actions runner in a `hive-ci` nixos-container,
|
||||
executing CI jobs from `.forgejo/workflows/ci.yml` on every PR.
|
||||
|
||||
## For operators
|
||||
|
||||
**Enabling it is one line**: `services.hyperhive.deploy.forgejo.ci.enable = true`
|
||||
in the host NixOS config. No manual token provisioning — hive-c0re
|
||||
registers the runner with the forge automatically.
|
||||
|
||||
Two things worth knowing before you rely on it:
|
||||
|
||||
- **Only run CI for contributors you trust.** Builds run **unsandboxed**
|
||||
(an nspawn limitation, not a choice) — a malicious build script in a
|
||||
PR could make network requests or write to the container filesystem
|
||||
during the build. Fine for a small trusted-team hive where everyone
|
||||
already has forge access; if you take outside/fork contributions,
|
||||
gate CI behind Forgejo's fork-PR-approval setting or restrict the
|
||||
workflow to push-only triggers (forks can't push to your branches).
|
||||
See *Security* below for the full threat model.
|
||||
- **Watch your disk.** CI builds through the host's nix store with no
|
||||
automatic garbage collection of its own — a busy CI day can fill
|
||||
`/nix/store` until every job fails with `ENOSPC`. Add the daily +
|
||||
disk-pressure GC config from *Host store maintenance* below to your
|
||||
host's NixOS config (not optional if you plan to actually use this).
|
||||
|
||||
Everything below this point is implementation detail: exact
|
||||
auto-registration internals, container design, and the full security
|
||||
writeup.
|
||||
|
||||
## CI checks
|
||||
|
||||
Three jobs run on every PR (and on `workflow_dispatch` for manual re-triggers),
|
||||
defined in [`.forgejo/workflows/ci.yml`](../.forgejo/workflows/ci.yml). All
|
||||
three are required checks (forge branch protection) — a hit on any of them
|
||||
blocks merge.
|
||||
|
||||
| Job | What it runs |
|
||||
| --- | --- |
|
||||
| **nix flake check** | treefmt + rustfmt formatting, `cargo clippy -D warnings`, `cargo test`, module evaluation |
|
||||
| **tracker-tag lint** | flags `#NNN` issue tags in source and comments (`scripts/check-issue-refs.sh`) |
|
||||
| **comment-block lint** | flags contiguous comment blocks over 30 lines (`scripts/check-comment-blocks.sh`) |
|
||||
|
||||
`hive-forge ci-rerun --pr N` dispatches a `workflow_dispatch` retrigger
|
||||
without an empty commit.
|
||||
|
||||
**`ci-rerun --pr` verifies the code but does not reliably move the PR's own
|
||||
status checks.** Measured directly (raw `GET /repos/.../actions/tasks`
|
||||
JSON): a genuine PR-triggered run carries `event: pull_request` with the
|
||||
`#<n>` pseudo-ref as its `head_branch`; a `ci-rerun --pr`-dispatched run
|
||||
carries `event: workflow_dispatch` with the real branch name instead.
|
||||
Forgejo's PR commit-status tracking appears scoped to `pull_request`-event
|
||||
runs specifically, so a dispatched run — even a genuinely successful one —
|
||||
never writes to that status slot. Reproduced with three dispatches on one
|
||||
PR over 20+ minutes, all genuinely green, the PR's own status unmoved
|
||||
throughout. **Always re-check `pr-status` after a dispatch and believe
|
||||
what it says — don't push a commit just to unwedge it, that's the
|
||||
empty-commit anti-pattern this verb exists to avoid.** If the code is
|
||||
verified green (`hive-forge ci-log --run <n> --job 0`, verdict is the
|
||||
last line) but the status is stuck, ask the operator to click "rerun" in
|
||||
the forge web UI — it's CSRF-gated, so only they can do it.
|
||||
|
||||
### Running checks locally
|
||||
|
||||
Don't run `nix flake check` directly — it dispatches to the shared build farm and
|
||||
wastes a remote-builder slot. Use the devshell equivalents instead:
|
||||
|
||||
```sh
|
||||
nix develop -c cargo clippy --all-targets -- -D warnings
|
||||
nix develop -c cargo test
|
||||
nix develop -c treefmt # same as nix fmt; treefmt covers rustfmt + nixfmt + taplo
|
||||
sh scripts/check-issue-refs.sh # tracker-tag lint
|
||||
sh scripts/check-comment-blocks.sh # comment-block lint
|
||||
```
|
||||
|
||||
A git pre-push hook that automates the two lint checks is provided at
|
||||
`scripts/pre-push`. Install it once per clone:
|
||||
|
||||
```sh
|
||||
ln -sf ../../scripts/pre-push .git/hooks/pre-push
|
||||
```
|
||||
|
||||
After that, any `git push` automatically runs both lints and aborts with a
|
||||
diagnostic if either fails — catching the issue locally before CI sees it.
|
||||
Note that the hook does **not** run `cargo clippy` or `cargo test` (those are
|
||||
slow); run those manually before pushing Rust changes.
|
||||
|
||||
## Configuration reference
|
||||
|
||||
The internal forge is always present (mandatory), so the runner always has a
|
||||
hive-forge instance to register against — nothing extra to enable beyond
|
||||
`services.hyperhive.deploy.forgejo.ci.enable = true` (see *For operators* above).
|
||||
|
||||
Optional tuning: `services.hyperhive.deploy.forgejo.ci.name` (runner name in forge
|
||||
admin panel), `concurrency` (parallel job capacity), `labels` (workflow
|
||||
targeting), `jobTimeout` (per-job wall-clock cap, default `"1h"`, Go duration
|
||||
string e.g. `"3h"` — a job that exceeds it is killed so a hung or runaway
|
||||
build can't hold the runner's single slot indefinitely).
|
||||
|
||||
## Container design
|
||||
|
||||
- **Private netns, bridge-attached**: the container runs in its own network namespace (`privateNetwork = true`, `hostBridge`) and reaches hive-forge through the gateway at `http://<forge.domain>` (resolved to the bridge IP via `networking.extraHosts`). It cannot reach host-loopback services — the core dashboard at `127.0.0.1:7000` and the raw forge port are unreachable from CI. Requires `forge.behindGateway = true`.
|
||||
- **Non-ephemeral**: runner credentials persist across restarts (written to container's stateDir on first registration, reused thereafter).
|
||||
- **Sandbox fallback**: nspawn containers can't create user-namespaces, so nix's sandboxing would always fail. Module sets `nix.settings.sandbox-fallback = true` in the container — nix builds run unsandboxed (safe because the container is already isolated). See `docs/gotchas.md`.
|
||||
- **Credential isolation**: the forge admin token (`forge-core-token`) never enters the container. hive-c0re holds it and performs all forge API calls (runner validation + registration-token mint, in `forge/ci_runner.rs`); via hive-priv it writes only the runner registration token to the host env-file `/run/hive-ci/runner-token`, which the container bind-mounts read-only.
|
||||
|
||||
## Auto-registration flow
|
||||
|
||||
Registration is **off the container's boot-critical path** — hive-c0re owns
|
||||
it and runs it out of band, so a slow forge or core-token never delays the
|
||||
container's start. Gotcha: don't gate the container's own start on a forge
|
||||
round-trip (a host-side unit that did this could exceed the nspawn start
|
||||
timeout and trip a restart loop) — registration must stay something
|
||||
hive-c0re drives after the container is already up. The core admin token is
|
||||
held only by hive-c0re on the host; only the runner registration token
|
||||
reaches the container.
|
||||
|
||||
### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)
|
||||
|
||||
Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — failures are logged and never abort the sweep; a healthy runner is never restarted.
|
||||
|
||||
1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
|
||||
- **200**: still registered — done, no restart.
|
||||
- **404 / other non-200 / malformed**: stale — re-register (below).
|
||||
- **transport error** (forge unreachable): keep the existing creds; a network blip must not wipe a valid runner.
|
||||
2. If absent or stale: mint a fresh token from `GET /api/v1/admin/runners/registration-token`, then hand it to hive-priv's `RegisterCiRunner`, which (as root) writes `TOKEN=<real>` **in place** to the host env-file `/run/hive-ci/runner-token` (preserving the inode nspawn pinned into the container at start) and restarts `gitea-runner-hive.service` inside the container so it picks up the credential and registers.
|
||||
|
||||
### Container side
|
||||
|
||||
- The container boots immediately — nothing gates its start on registration.
|
||||
- tmpfiles seeds `/run/hive-ci/runner-token` with `TOKEN=placeholder` so the runner's `EnvironmentFile` always exists.
|
||||
- `gitea-runner-hive.service` has an `ExecStartPre` precond (ahead of the nix-daemon wait) that **fails fast** unless it is already registered (`.runner` present) or a real, non-placeholder token is in place. `Restart=on-failure` (no start-limit cap) self-heals it: a runner that precond-fails at boot keeps retrying until hive-c0re writes the token (c0re's explicit restart is the primary path; the retry is the safety net).
|
||||
- **Convergence**: because the token write targets the *host* file, even if c0re's restart races the container being down, the container later starts, reads the now-real token, passes the precond, and registers on its own.
|
||||
|
||||
## Actions checkout mirror
|
||||
|
||||
When `deploy.forgejo.ci.enable` is set, hive-c0re auto-seeds an
|
||||
`actions/checkout` pull-mirror on the local forge and sets Forgejo's
|
||||
`DEFAULT_ACTIONS_URL` to point at the local instance. This means CI
|
||||
`uses: actions/checkout@vN` steps resolve entirely on loopback — no
|
||||
external DNS on the CI critical path.
|
||||
|
||||
The mirror is seeded by **hive-c0re** itself during its forge
|
||||
provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module
|
||||
forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the
|
||||
`hive-c0re` service environment (JSON-encoded `[{upstream, dest}]`
|
||||
list). hive-c0re already holds the admin token for the rest of the
|
||||
forge provisioning sweep (orgs, agent accounts, etc.), so mirror
|
||||
seeding lives in the same place rather than a separate host-side unit.
|
||||
|
||||
**General-purpose mirrors**: you can pre-seed any external repo as a
|
||||
pull-mirror via `services.hyperhive.swarm.forge.mirrors`:
|
||||
|
||||
```nix
|
||||
services.hyperhive.swarm.forge.mirrors = [
|
||||
{ upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
|
||||
{ upstream = "https://github.com/example/tool"; dest = "mirrors/tool"; }
|
||||
];
|
||||
```
|
||||
|
||||
Each entry is created as a real Forgejo pull-mirror — not a one-off
|
||||
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
|
||||
request), so a DNS blip during that sync will propagate back to the
|
||||
runner as a hard `git clone` failure. The
|
||||
`<owner>` org in `dest` is auto-created. Keep mirror dests out of the
|
||||
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
|
||||
to avoid provisioning collisions.
|
||||
|
||||
## Security: unsandboxed builds and trusted contributors
|
||||
|
||||
**hive-ci should only run CI for trusted contributors.** The security boundary is weaker than it looks:
|
||||
|
||||
### What unsandboxed builds mean
|
||||
|
||||
nspawn containers cannot create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
|
||||
|
||||
A malicious `default.nix` or build script in a PR can therefore:
|
||||
|
||||
- **Make arbitrary network requests** to any address reachable from the container. The container runs in its own netns behind the hive bridge, so it reaches the forge only through the gateway (`http://<forge.domain>`, public/read endpoints — no admin credentials) and **cannot** reach host-loopback services: the unauthenticated core dashboard at `127.0.0.1:7000` and the raw forge port are off-limits (bridge→127.0.0.0/8 is dropped).
|
||||
- **Write to the container filesystem**, including corrupting the runner's state dir or `.runner` credentials.
|
||||
|
||||
The core admin token (`forge-core-token`) is **not** bind-mounted into the container. It is held and used only by hive-c0re on the host (`forge/ci_runner.rs`), which mints per-runner registration tokens; only that registration token reaches the container's env-file. A build process can still reach forge over the network, but cannot use the admin token to issue privileged API calls.
|
||||
|
||||
Note: `nix flake check --no-build` (eval-only) reduces the attack surface but does not eliminate it — `builtins.fetchGit`, `builtins.fetchurl`, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full `nix flake check` (builds derivations), which is the higher-risk path.
|
||||
|
||||
### Mitigation
|
||||
|
||||
For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.
|
||||
|
||||
For repos with external contributors or fork PRs:
|
||||
|
||||
- Use Forgejo's **fork PR approval workflow** (`repository.settings` → "Require approval for fork PRs from first-time contributors") to gate CI until a maintainer approves the first PR.
|
||||
- Or restrict the CI workflow trigger to push events on branches (not `pull_request` from forks) — forks can't push to upstream branches.
|
||||
|
||||
The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.
|
||||
|
||||
## Host store maintenance (recommended)
|
||||
|
||||
The CI runner builds derivations through the **host** nix-daemon — the
|
||||
hive-ci container shares the host store and has no daemon of its own. Build
|
||||
outputs accumulate in `/nix/store` with no automatic collection, and a busy
|
||||
CI day can fill the disk until every job fails fast with `ENOSPC`.
|
||||
|
||||
Store GC is a **host-level** concern, so it belongs in the host's own NixOS
|
||||
configuration, not in the hyperhive service modules — a single service should
|
||||
not reach out and change the host's global nix-daemon options. Add the
|
||||
following to your host config:
|
||||
|
||||
```nix
|
||||
{
|
||||
# Daily GC: delete store paths not referenced by a live root and older
|
||||
# than a day. Keeps the store bounded between builds.
|
||||
nix.gc = {
|
||||
automatic = true;
|
||||
dates = "daily";
|
||||
options = "--delete-older-than 1d";
|
||||
};
|
||||
|
||||
# Disk-pressure GC: when free space drops below min-free mid-build, the
|
||||
# daemon collects garbage up to max-free before continuing. This is the
|
||||
# real-time net the daily timer can't provide — a same-day build burst is
|
||||
# what fills the disk. Tune to your disk size.
|
||||
nix.settings.min-free = 20 * 1024 * 1024 * 1024; # 20 GiB
|
||||
nix.settings.max-free = 50 * 1024 * 1024 * 1024; # 50 GiB
|
||||
}
|
||||
```
|
||||
|
||||
**Remote builders:** if CI dispatches builds to a remote builder (e.g. via
|
||||
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
|
||||
store, so the same GC config should be applied wherever the builder runs —
|
||||
GC on the coordinator host won't reclaim space on the builder.
|
||||
|
||||
## References
|
||||
|
||||
- `nix/host-modules/hive-ci.nix`: runner configuration, auto-registration script, container setup.
|
||||
- `.forgejo/workflows/ci.yml`: workflow definition.
|
||||
- `docs/gotchas.md`: nix sandboxing limitations in containers.
|
||||
552
docs/scheduler/coordinator.md
Normal file
552
docs/scheduler/coordinator.md
Normal file
|
|
@ -0,0 +1,552 @@
|
|||
# hive-c0re coordinator internals
|
||||
|
||||
Architecture notes for the `hive-c0re` coordinator daemon's internal subsystems.
|
||||
For the public API surface (dashboard, socket protocol) see `docs/conventions.md`
|
||||
and `docs/persistence.md`.
|
||||
|
||||
---
|
||||
|
||||
## Job queue
|
||||
|
||||
**For the job queue as a general idea — graph of steps, shared resource
|
||||
slots, no hive-c0re specifics — see [`jobq.md`](jobq.md) instead.** This
|
||||
section covers the concrete node inventory hive-c0re builds on top of that
|
||||
engine, plus the internals (module layout, scheduler mechanics,
|
||||
resource/lease semantics).
|
||||
|
||||
Every container/meta operation (rebuild, meta-update, first-spawn, power
|
||||
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
|
||||
as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
|
||||
concurrency comes from the resource classes below, not from multiple workers.
|
||||
Special cases like graceful stop, deferred starts, and the meta-update
|
||||
cascade need no bespoke code paths — each is expressed as a DAG *shape*
|
||||
built from the same primitive nodes.
|
||||
|
||||
### Two levels: DAG and node
|
||||
|
||||
The **DAG** is the unit of cancel / approval-resolution and the
|
||||
dashboard group; the **node** is the unit of scheduling / execution /
|
||||
build-log. Deps are intra-DAG edges only (`AfterOk` by default:
|
||||
the dep must succeed, a failed/cancelled dep cancels the dependent —
|
||||
cancel-downstream). Cross-DAG ordering comes from the per-agent lease,
|
||||
never from edges between DAGs.
|
||||
|
||||
A DAG is **declared, not described**: a template builds it through
|
||||
`hive_jobq::JobBuilder`, naming each node it depends on via the handle
|
||||
`b.node(kind)` handed back, and the builder inserts the nodes itself. A handle
|
||||
only exists for a node already declared, so every edge points backwards and a
|
||||
cycle cannot be written down — there is no submit-time validation pass, because
|
||||
there is no malformed spec to reject.
|
||||
|
||||
### Node inventory (primitives)
|
||||
|
||||
Nix-heavy — hold one of the `buildSlots` permits for the node's duration:
|
||||
|
||||
| Node | Wraps |
|
||||
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Prebuild` | `lifecycle::prebuild_toplevel` — build the toplevel out-of-band while the container keeps serving (its meta preamble is the upstream `MetaSync` node) |
|
||||
| `Swap` | drop-in rewrite + `nixos-container update` profile-swap (requires the container stopped); the post-swap bookkeeping tail lives in the sibling `RebuildBookkeeping` node |
|
||||
| `Create` | first-spawn provisioning + `nixos-container create` (atomic build+create) |
|
||||
| `MetaLock` | meta flake lock bump (`lock_update` / boot-sweep `lock_update_hyperhive`, commit fused — see below); fans out child `Rebuild` DAGs on completion |
|
||||
| `DeployWindow` | resource-holding root of the merge-config-PR deploy subtree — declares the build slot, the lease and the meta window, then completes immediately so its children run under them (see _Approvals_ below) |
|
||||
| `DeployApply` | the deploy's irreversible half: ff-merge the reviewed PR head, two-phase meta deploy, container rebuild |
|
||||
|
||||
Cheap — no build slot:
|
||||
|
||||
| Node | Behavior |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `MergeVerify` | the deploy's pre-merge gate — PR-head drift check, fetch, `verify_commit` eval. Mutates nothing, so a rejection here needs no compensation |
|
||||
| `DeployTail` | the deploy's `AfterAny` compensation + bookkeeping tail — rolls `applied/main` back from the parked `refs/hyperhive/rollback/<id>` and aborts the staged meta lock when the deploy never confirmed good, then mirrors the config repo to the forge. Infallible by construction |
|
||||
| `MetaSync` | the rebuild's meta preamble — rebuild-dir prep, idempotent meta `sync_agents`, optional per-agent relock. Holds the `MetaWindow` resource (below); deliberately its own node so the window never covers `Prebuild`'s multi-minute build |
|
||||
| `Reconcile` | idempotent power converge: read `wanted` (below) + observed state; start if `Up` & down (cold-start fallback included), stop if `Offline` & up, else noop |
|
||||
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
|
||||
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here) |
|
||||
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
|
||||
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
|
||||
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
|
||||
| `WriteDropin` | `set_nspawn_flags` + `set_resource_limits` + daemon-reload |
|
||||
| `WritePermFile` | commit `tool-groups.json` / `capabilities.json` (single git commit under `META_LOCK`) + emit the P3RM1SS10NS snapshots |
|
||||
| `Reparent` | `set-parent` / `set-parent-bulk`: apply every `(child, new_parent)` move under one `META_LOCK` commit (`meta::bulk_commit_topology`), send the per-agent move notifications, rescan + diff-emit. Agentless like `MetaLock` — a bulk move can span multiple agents, and a reparent touches the meta repo, not any one container. `moves` is typed `(Ident, Option<Ident>)` pairs, not raw strings. Rides the existing `Template::MetaUpdate` variant rather than a dedicated one — it's internal-only (never reaches the graph wire), so the stand-in only affects `terminal_hook` dispatch (resolves to no hook either way) and history-retention bucketing |
|
||||
|
||||
There is deliberately **no `GitCommit` node**: `meta.rs` fuses each mutation
|
||||
with its commit under its internal `META_LOCK` mutex, so a standalone commit
|
||||
node would open a dirty-working-tree window between nodes.
|
||||
|
||||
Two further layers protect the meta repo across *windows* that span multiple
|
||||
`META_LOCK` acquisitions — above all the approval deploy's prepare→finalize
|
||||
span, which keeps a bumped `flake.lock` **staged uncommitted** for the whole
|
||||
container build:
|
||||
|
||||
- **The deploy window** (`Resource::MetaWindow`): a global, capacity-1 queue
|
||||
resource declared by every node kind that mutates the meta repo — `MetaSync`,
|
||||
`MetaLock`, `WritePermFile`, `Reparent`, `Provision`'s agent registration, and
|
||||
`DeployWindow` — the deploy subtree's root, which holds it across every
|
||||
phase below it (it declares `Resource::MetaWindow`). Two meta
|
||||
mutations can therefore never interleave, so no commit lands inside another
|
||||
node's staged window. It is a queue resource rather than a runtime mutex
|
||||
because a resource is held by a subtree root across its whole subtree, which
|
||||
a `MutexGuard` (bounded by one executor fn) cannot — that is what lets a
|
||||
multi-node deploy own one window. For the same reason the window must stay
|
||||
*off* long store-only work: the rebuild's meta preamble is its own
|
||||
`MetaSync` node, a sibling of (never a parent of) `Prebuild`, so the
|
||||
toplevel build runs outside the window and `buildSlots > 1` still gives
|
||||
concurrent rebuilds across agents.
|
||||
- **Path-limited commits**: the targeted meta committers (perm files,
|
||||
topology, lock bumps, finalize) commit `-- <their paths>` with path-scoped
|
||||
dirty checks, so even a non-queue caller (boot migration, destroy's
|
||||
`sync_agents`) can never sweep someone else's staged content into its
|
||||
commit.
|
||||
|
||||
### Every operation as a DAG
|
||||
|
||||
The `stop` / `start` power ops write the durable `wanted` intent via a head
|
||||
`SetWanted` node (not a pre-submit side effect) — it holds the agent lease,
|
||||
so intent-write + reconcile is atomic per-agent. `restart` is the exception:
|
||||
it writes *no* intent (no `SetWanted` head) — it bounces the container and
|
||||
lets the tail `Reconcile` converge to the agent's existing `wanted`, so a
|
||||
deliberately-stopped agent is not forced back up by a hive-wide restart. The
|
||||
hive-wide power ops — `restart`, `stop`, and `start` — take an agent *list*:
|
||||
a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
|
||||
subgraph each (independent roots, run concurrently on their own leases), not
|
||||
N separate DAGs.
|
||||
|
||||
**These are built dynamically from each agent's live running state** (an
|
||||
async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
|
||||
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
|
||||
a head `SetWanted` (intent) — `restart` does not; the tail `Reconcile`
|
||||
(convergence guarantee — cheap, noops when already converged) is ALWAYS
|
||||
present; only the *mechanical* nodes (`Signal`/`Drain`/`StopForUpdate`) are
|
||||
state-conditional — skipped for a *down* agent (nothing to quiesce/stop). Keeping `Reconcile` in every shape
|
||||
closes the TOCTOU window: if an agent flips state between the `is_running`
|
||||
read and node exec, the tail `Reconcile` still converges it in-DAG (with
|
||||
`StopForUpdate`-noop as the backstop) — no reliance on an external reconcile
|
||||
sweep. `start` folds the per-agent stale-rev upgrade in (a *down + stale*
|
||||
agent's subgraph is a rebuild-then-start).
|
||||
|
||||
```text
|
||||
rebuild(a): MetaSync(a) → AgentWindow(a){ Prebuild(a) ∥ [Signal(a)→Drain(a) if graceful]; both →(after-ok) StopForUpdate(a) → Swap(a) →(after-ok) RebuildBookkeeping(a) } →(after-any) Reconcile(a)
|
||||
stop(a..): online a: SetWanted(a,Off) → [Signal→Drain→ if graceful] Reconcile(a)
|
||||
offline a: SetWanted(a,Off) → Reconcile(a) (N subgraphs, 1 DAG)
|
||||
restart(a..): online a: [Signal→Drain→ if graceful] StopForUpdate(a) → Reconcile(a) (no SetWanted)
|
||||
offline a: Reconcile(a) (nothing to stop; Reconcile converges to existing wanted)
|
||||
start(a..): a: SetWanted(a,Up) → Reconcile(a) (down+stale ⇒ SetWanted(a,Up) → «rebuild subgraph»)
|
||||
spawn(a): [wanted=Up at approve] Create(a) → WriteDropin(a) → Reconcile(a)
|
||||
perm-change(a): WritePermFile(a) → «rebuild subgraph»
|
||||
meta-update(inp): MetaLock(inp) →(in-DAG) «rebuild subgraph» per affected agent
|
||||
boot: (if any rev marker stale) MetaLock(hyperhive) →(in-DAG) «rebuild subgraph» per stale agent;
|
||||
plus Reconcile(a) for every drifted agent (all ONE DAG)
|
||||
reparent(moves): Reparent(moves) (no rebuild — topology.json is read live)
|
||||
```
|
||||
|
||||
Notable collapses:
|
||||
|
||||
- **`rebuild` is one uniform shape** — no `was_running` branch.
|
||||
`StopForUpdate` noops when already down; the tail `Reconcile` auto-noops the
|
||||
start when `wanted = Offline` (a rebuild of a deliberately-stopped agent
|
||||
leaves it stopped).
|
||||
- **The swap-failure recovery-start is structural**: `Reconcile` deps on
|
||||
`Swap` with the one `AfterAny` edge in the system — it runs after `Swap`
|
||||
terminal ok *or* fail, bringing a wanted-up agent back on its old config.
|
||||
- **Deferred start is automatic**: `Reconcile` holds no build slot, so the
|
||||
next DAG's `Prebuild` starts as soon as `Swap` frees the slot.
|
||||
- **Graceful stop needs no watcher thread**: `Signal`/`Drain` are cheap, so a
|
||||
whole-hive graceful stop fires every agent's signal immediately and all
|
||||
drains overlap; each DAG's tail `Reconcile` does the actual stop.
|
||||
- **The meta-update cascade grows in the same DAG on completion**:
|
||||
`MetaLock`'s executor computes the affected agent set after the bump lands
|
||||
and grows one `rebuild` subgraph per agent into its *own* DAG via
|
||||
`append_subgraph` (rooted on the `MetaLock`, `relock = false` so the cascade
|
||||
doesn't revert the bump). Not child DAGs — one DAG, no `parent_id`. A failed
|
||||
bump appends nothing (no cancel-children dance). Same shape as the startup
|
||||
sweep; the meta-update DAG carries the `Rebuilding` transient so each cascade
|
||||
agent keeps crash-watch suppression during its `Swap`.
|
||||
|
||||
### Desired-state (spec vs status)
|
||||
|
||||
Per-agent power *intent* — `wanted: Up | Offline` — is durable as the
|
||||
`agent_power` table in the coordinator DB (`hive-c0re/src/stores/power.rs`).
|
||||
`container_view` remains the observed *status*; `Reconcile` nodes converge the
|
||||
two. Setting `wanted` is never a queued node: the power layer
|
||||
(`job_queue/power.rs`) writes the row synchronously, then inserts the DAG
|
||||
whose `Reconcile` reads the fresh value — rapid toggles are last-writer-wins.
|
||||
Power toggles never commit to the meta repo. Every operator power surface —
|
||||
dashboard buttons, the MCP tools, and `hivectl stop/start/restart/kill` —
|
||||
rides the queue through that power layer, so intent, lease serialization,
|
||||
and crash-watch suppression can't drift per surface; the only direct starts
|
||||
left are the root-agent bootstrap and infra containers (no lease, no
|
||||
harness). Cancelling a still-queued power DAG reverts `wanted` to the
|
||||
observed state — a cancel means "don't do it", not "do it later". Agents
|
||||
without a row are seeded from observed state on first touch (running ⇒
|
||||
`Up`); destroy removes the row.
|
||||
|
||||
The admin-socket responses carry the submitted DAG ids; `hivectl` polls
|
||||
`HostRequest::QueueNodes` (~1s) and prints a progress line per DAG — roll-up
|
||||
glyph, template, agent, node chain — so
|
||||
CLI verbs block until their jobs finish (`--no-wait` opts out; failures exit
|
||||
non-zero). Nodes appended in-DAG (a `MetaLock` growing per-agent rebuild
|
||||
subgraphs, a `Reconcile` fanning its `Start`/`Stop`) join the same DAG, so
|
||||
they surface under that DAG's id in the same loop — no separate child DAGs.
|
||||
|
||||
### Scheduler semantics
|
||||
|
||||
A node is **ready** when it's `Queued`, every dep is satisfied, and its
|
||||
resources are free. Resources:
|
||||
|
||||
1. **Build slots** — `services.hyperhive.c0re.buildSlots` permits (default 1),
|
||||
held by nix-heavy nodes for the node's duration.
|
||||
2. **Per-agent lifecycle lease** — keyed on the **node's** agent (agent is
|
||||
per-node; a DAG can span agents) and globally exclusive per agent across
|
||||
all DAGs: acquired either at a container-affecting node (`SetWanted`,
|
||||
`Reconcile`, `WriteDropin`, `Create`) or at a **brace** (`AgentWindow`,
|
||||
`DeployWindow`) on behalf of a whole coordinated subtree; held by the owning
|
||||
DAG until it's terminal, so two DAGs never interleave container ops on the
|
||||
same agent. A DAG touching several agents holds one lease per agent.
|
||||
(`SetWanted` is a store write, not a container op, but takes the lease anyway
|
||||
so a power-op DAG's intent write + reconcile is atomic — two racing ops can't
|
||||
clobber intent before either reconciles.) **Lease-exempt**: `MetaSync`,
|
||||
`Prebuild`, `MetaLock`, `WritePermFile`, `Reparent` — they touch the store /
|
||||
meta, not the running container, which is exactly why a stop can land while
|
||||
another DAG's prebuild is still building. Also exempt, for a different
|
||||
reason, are the rebuild subtree's own members (`StopForUpdate`, `Swap`,
|
||||
`Signal`, `Drain`, `RebuildBookkeeping`): they genuinely do touch the
|
||||
container, but their `AgentWindow` brace holds the lease above them — see
|
||||
_Braces_ below.
|
||||
|
||||
#### Braces
|
||||
|
||||
Templates otherwise declare a resource on **every** node that needs it, even
|
||||
when a parent already holds it, so the requirement belongs to the node rather
|
||||
than to one DAG shape it happens to appear in. A **brace** is the one sanctioned
|
||||
exception: a pure-resource-holder root that declares on behalf of a subtree
|
||||
coordinated with itself, whose members then declare nothing.
|
||||
|
||||
It is forced rather than stylistic. Declaring a resource means *"I need this
|
||||
exclusively"*, and the agent lease is single-unit — so **two siblings that both
|
||||
declared it could never run concurrently.** For a subtree whose whole point is
|
||||
concurrency (`Prebuild` beside the `Signal` → `Drain` quiesce window), declaring
|
||||
the requirement truthfully on every node and running those nodes in parallel are
|
||||
mutually exclusive. One holder above them speaks for the subtree.
|
||||
|
||||
This is the opposite of the failure the declare-your-own rule exists to prevent,
|
||||
not a relapse into it: there the requirement was *implicit*, inferred from a
|
||||
node's kind and true only by accident of placement. Here it is explicit, on one
|
||||
node, with the omission below it documented on the brace itself.
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- **Flattening a chain under a brace is safe.** The stop chain used to nest
|
||||
`Signal` over `Drain` over `StopForUpdate` specifically so the lease stayed
|
||||
continuous — as independent siblings each would acquire it separately and
|
||||
leave a gap another DAG could claim the agent in, mid-bounce. A brace supplies
|
||||
that continuity directly, so the nesting is no longer load-bearing.
|
||||
- **Observability is unaffected.** `running_transients` keys off a node's
|
||||
*payload* agent, not off a declared lease edge, so every child still lights its
|
||||
own dashboard pill and still reports its own `takes_container_down` to the
|
||||
crash watcher. A brace itself reports `false`: it parents the stopping nodes
|
||||
but does not stop anything, and claiming otherwise would widen crash
|
||||
suppression across the build and tail, where a vanished container is still a
|
||||
real crash.
|
||||
|
||||
Among simultaneously-ready nodes competing for a resource, DAG-submit order
|
||||
wins (FIFO) so bulk operations drain predictably. The scheduler also owns the
|
||||
DAG-lifetime transient guard (dashboard pill + crash-watch suppression),
|
||||
created on lease acquisition and dropped when the DAG settles terminal.
|
||||
|
||||
The queue is in-memory only and lost on hive-c0re restart — deliberate:
|
||||
desired state is re-derived at boot from the DB + rev markers (see _Boot
|
||||
reconcile_), so there is no durable-recovery machinery to go wrong.
|
||||
|
||||
### Cancel, history
|
||||
|
||||
Submit-time dedup was removed with the agent-per-node move (a multi-agent DAG
|
||||
has no single agent to key a dedup on), so every submit enqueues a fresh DAG;
|
||||
whether any dedup needs reintroducing is tracked as a follow-up.
|
||||
|
||||
Cancel only applies to still-fully-queued DAGs (an in-flight nix build isn't
|
||||
interruptible) — each op is one DAG now, so there are no child DAGs to cascade to.
|
||||
Roll-up state: `Failed` if any node failed, else `Running` / `Queued` /
|
||||
`Cancelled` / `Done`. The snapshot retains the 50 most recent terminal
|
||||
DAGs — a flat cap over the whole sorted list, not per template, since
|
||||
the dashboard renders one recent-builds list and one number bounds it.
|
||||
|
||||
### Approvals
|
||||
|
||||
`MergeConfigPr` approvals ride as a four-node deploy subtree:
|
||||
|
||||
```
|
||||
DeployWindow (root — build slot + lease + meta window, no work of its own)
|
||||
├── MergeVerify drift gate, fetch, verify_commit
|
||||
├── DeployApply AfterOk(verify) park rollback ref, ff-merge, deploy
|
||||
└── DeployTail AfterAny(apply) compensate, mirror to forge
|
||||
```
|
||||
|
||||
The root's resources are held across the whole subtree, so the two-phase
|
||||
`prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock`
|
||||
protected even though the phases are separate nodes. Splitting them buys
|
||||
three things a single opaque node couldn't have: per-phase visibility on the
|
||||
dashboard, a `MergeVerify` failure that provably mutated nothing, and a
|
||||
compensation step that survives a hive-c0re restart — the pre-merge
|
||||
`applied/main` is parked in `refs/hyperhive/rollback/<approval-id>`, not in a
|
||||
local variable, so `DeployTail` can still undo a half-finished deploy after a
|
||||
crash.
|
||||
|
||||
`Spawn` and `UpdateMetaInputs` approvals map onto the ordinary `spawn` /
|
||||
`meta-update` shapes. The scheduler fires `actions::resolve_approval_dag`
|
||||
exactly once when **any** approval-carrying DAG settles terminal — deploys
|
||||
included, since their outcome is now the DAG's own state (including
|
||||
cancelled-while-queued, which fails the approval instead of dangling it).
|
||||
|
||||
### Wire shape
|
||||
|
||||
`RebuildQueueChanged { seq }` (event name kept) — **a bare trigger, no
|
||||
payload.** It says *the queue changed*; a client that wants to know how
|
||||
re-fetches `GET /api/jobq/graph`.
|
||||
|
||||
That endpoint serves the graph generically (`hive-jobq-wire`): every node
|
||||
carries `id`, `parent`, `deps`, `state`, `label` (the node kind's own wire
|
||||
string — the kind *is* the phase label, there is no separate sub-step
|
||||
string) and free-form `data` for what only some kinds have (`agent`,
|
||||
`approval_id`, `inputs`, `build_log_id`). Group roots ride as ordinary
|
||||
nodes, so a group's state is just the root's own `state`.
|
||||
|
||||
There is **no group-level `agent`** — agent is per-node, so one group can
|
||||
span agents; consumers derive a group's agent(s) from its nodes. Build logs
|
||||
are likewise **per-node**: the dashboard renders the node tree and keys the
|
||||
live-log panel off the running node.
|
||||
|
||||
The event carries no payload by design: shipping a typed projection of
|
||||
the whole queue in the event itself would be a second rendering of the
|
||||
same graph that has to be kept in agreement by hand with the endpoint
|
||||
every consumer actually reads. Telling a client *when* to refetch is
|
||||
the event's whole job.
|
||||
|
||||
---
|
||||
|
||||
## Container view
|
||||
|
||||
`container_view.rs` maintains an in-memory snapshot of every nixos-container's
|
||||
systemd service state. It is polled on coordinator startup and re-scanned after
|
||||
every lifecycle operation (spawn, rebuild, kill) so the dashboard always reflects
|
||||
the actual container status without a live `nixos-container list` call on each
|
||||
render.
|
||||
|
||||
---
|
||||
|
||||
## Boot reconcile
|
||||
|
||||
On startup, `auto_update::run` classifies every agent by rev freshness (the
|
||||
per-agent `.{name}.hyperhive-rev` marker under `/var/lib/hyperhive/applied/`
|
||||
vs the current flake path) and persisted `wanted` intent, then:
|
||||
|
||||
1. **Config path** — when *any* marker is stale, submit one `Boot`
|
||||
DAG: a `MetaLock` (hyperhive input bump, non-fatal) that grows an in-DAG
|
||||
`Rebuild` subgraph for each stale agent whose `wanted = Up` (topology-sorted,
|
||||
parents first). Stale but wanted-offline agents get no boot-time nix work — their
|
||||
rebuild happens on their next start (the start submit path upgrades a
|
||||
stale start to rebuild+start), which is also why the lock bump runs even
|
||||
when every stale agent is offline: those later start-upgrades must build
|
||||
against the bumped lock. Each child rebuild's tail `Reconcile` brings the
|
||||
agent (back) up, covering both the running-stale and stopped-but-wanted-up
|
||||
cases.
|
||||
|
||||
2. **Power path** — every agent whose observed state drifted from `wanted`
|
||||
gets a plain `Reconcile` DAG (`kind = reconcile`, source `auto_update`).
|
||||
|
||||
Booting with no config change performs **no meta commit** — only reconciles.
|
||||
The sweep reason records the rebuild / deferred / up-to-date counts so the
|
||||
operator sees at a glance how much work the boot triggered. Agents without an
|
||||
`agent_power` row are seeded from observed state during classification (the
|
||||
one-time migration; thereafter the DB is authoritative).
|
||||
|
||||
## Meta flake
|
||||
|
||||
`meta.rs` owns the single coordinator-managed flake at `/var/lib/hyperhive/meta/`.
|
||||
This flake consumes every agent's applied config repo as a flake input and exports
|
||||
one `nixosConfiguration` per agent. Container lifecycle ops drive the lock file so
|
||||
meta's git log is the system-wide deploy audit trail.
|
||||
|
||||
Key operations:
|
||||
|
||||
- **`sync_agents`** (idempotent) — render `flake.nix` for the current agent set,
|
||||
init the repo on first call, relock if the rendered contents changed, commit.
|
||||
Called by spawn / destroy / startup migration.
|
||||
- **`prepare_deploy` + `finalize_deploy` / `abort_deploy`** — two-phase for the
|
||||
`MergeConfigPr` deploy path so a failed `nixos-container update` leaves no orphan
|
||||
commit in meta. Prepare writes the new lock without committing; finalize commits
|
||||
with the deploy message; abort restores the lock.
|
||||
- **`lock_update_hyperhive`** — one-shot for the boot-reconcile path (the
|
||||
sweep DAG's `MetaLock` node): bumps the `hyperhive` input lock and commits;
|
||||
the scheduler fans out the agent rebuilds on completion.
|
||||
|
||||
Every public `meta.rs` operation takes the module's internal `META_LOCK`
|
||||
mutex, so concurrent job-queue nodes (and the approval deploy pipeline) never
|
||||
race on the repo's `.git/index.lock`.
|
||||
|
||||
---
|
||||
|
||||
## Container lifecycle (`lifecycle.rs`)
|
||||
|
||||
Every container operation ultimately calls into `lifecycle.rs`. Two paths exist:
|
||||
**rebuild** (existing container) and **spawn** (first-time creation).
|
||||
|
||||
### Rebuild path (existing container)
|
||||
|
||||
Goal: apply the new system profile and any `EXTRA_NSPAWN_FLAGS` / drop-in changes
|
||||
in a single start, with minimum downtime.
|
||||
|
||||
`nixos-container update` only runs `systemctl reload container@<c>` when the
|
||||
container is already up (per `isContainerRunning` in `nixos-container.pl`). Stopping
|
||||
first turns `update` into a boot-style operation: it builds + `nix-env --set`s the
|
||||
new profile and skips the in-container `switch-to-configuration`. The subsequent
|
||||
`start` then applies both the new profile and any `EXTRA_NSPAWN_FLAGS` changes in
|
||||
one go, rather than the double-bounce a live `update` would trigger.
|
||||
|
||||
Sequence for a rebuild DAG (each step is its own queue node):
|
||||
|
||||
1. `MetaSync` — rebuild-dir prep, meta `sync_agents`, and (unless this is a
|
||||
meta-update cascade child) the per-agent relock. Short, and the only step
|
||||
that mutates the meta repo, so it is the only one holding the global deploy
|
||||
window.
|
||||
2. `Prebuild` — build the new `system.build.toplevel` **before** stopping.
|
||||
The container keeps serving the previous generation while eval + fetch +
|
||||
build happen out-of-band. `nixos-container update` then finds the result
|
||||
cached and skips straight to the profile-swap. Build failures surface
|
||||
here, before the running container is touched. (Runs even for a stopped
|
||||
container — same total nix work, one uniform DAG shape.)
|
||||
3. `StopForUpdate` — bring the container down (noop when already stopped).
|
||||
4. `Swap` — `nixos-container update --flake meta#<name>` profile-swap
|
||||
(near-instant after the prebuild).
|
||||
5. `Reconcile` — boot into the new generation when `wanted = Up`; the
|
||||
in-container activation script transitions old → new. Holds no build
|
||||
slot, so the next DAG's `Prebuild` overlaps the container boot.
|
||||
|
||||
The approval deploy uses this same chain rather than a rebuild path of its
|
||||
own. Its `DeployApply` node does not build: it merges, opens the two-phase
|
||||
meta deploy, and returns the chain above as a subgraph the scheduler grafts
|
||||
into the live DAG under that node. A `FinalizeDeploy` node gated on the
|
||||
graft's completion then plants the deploy tag — so "did the agent come back
|
||||
up?" is answered by `Reconcile` succeeding, the same way it is for every
|
||||
other rebuild, instead of by a fused inline start.
|
||||
|
||||
The grafted nodes land *inside* `DeployWindow`'s subtree, so they re-enter
|
||||
the meta window and build slot it already holds rather than deadlocking
|
||||
against it.
|
||||
|
||||
### Cold-start fallback
|
||||
|
||||
`start` after `update` can exit non-zero when packages are **removed** between
|
||||
generations: the old-generation activation script references units that no longer
|
||||
exist in the new closure, causing systemd to exit non-zero. The container may be
|
||||
half-started at that point.
|
||||
|
||||
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
|
||||
→ `start` (clean cold-start, no generation transition, new activation runs cleanly).
|
||||
Both errors are preserved and surfaced if the cold-start also fails. The fallback
|
||||
lives in `lifecycle::start_with_fallback`, used by every `Reconcile` node's
|
||||
start action.
|
||||
|
||||
### Spawn path (new container)
|
||||
|
||||
For a first-time `create`, `nixos-container create` is atomic: if the build fails,
|
||||
no container record is left to clean up. A separate prebuild would just duplicate
|
||||
the eval, so it's skipped. Sequence: `create --flake meta#<name>` → write nspawn
|
||||
flags → `systemctl daemon-reload` → `start`.
|
||||
|
||||
### Prebuild attr path
|
||||
|
||||
`nix build` does not auto-resolve `meta#<name>` against `nixosConfigurations` the
|
||||
way `nixos-container` does internally. The explicit attr path
|
||||
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel` is required;
|
||||
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
|
||||
or the flake root directly — none of which exist in the rendered meta flake.
|
||||
|
||||
---
|
||||
|
||||
## Host-level resource + performance options
|
||||
|
||||
A handful of `services.hyperhive.c0re.*` options tune container resource
|
||||
limits, build parallelism, and first-spawn latency.
|
||||
|
||||
### Build slots
|
||||
|
||||
`buildSlots` (default `1`) sets how many nix-heavy job-queue nodes
|
||||
(prebuilds, profile swaps, first-spawn creates, meta lock bumps) run
|
||||
concurrently. The default serializes all heavy nix work like the pre-DAG
|
||||
rebuild queue did; raise it on hosts with the cores/RAM to build several
|
||||
agent toplevels at once. Per-agent correctness is independent of the count —
|
||||
each agent's container-affecting ops serialize on its lifecycle lease
|
||||
regardless.
|
||||
|
||||
### Container resource limits
|
||||
|
||||
`agentCpuQuota` and `agentMemoryMax` map directly to systemd
|
||||
`CPUQuota=` and `MemoryMax=`. hive-c0re writes a
|
||||
`container@h-<name>.service.d/` drop-in file on each spawn and
|
||||
rebuild, so changes take effect on the next lifecycle op without
|
||||
requiring a host rebuild.
|
||||
|
||||
The same drop-in carries `CPUWeight=` / `IOWeight=` from
|
||||
`agentCpuWeight` / `agentIoWeight`. Those are a different kind of
|
||||
setting: the quota and the memory max are **hard caps** that throttle
|
||||
an agent even on a completely idle host, while the weights are cgroup
|
||||
v2 **relative shares** that only decide who yields *under contention*.
|
||||
A low-weight container still gets the whole machine when nothing else
|
||||
wants it.
|
||||
|
||||
| Option | Default | Description |
|
||||
| ---------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `services.hyperhive.c0re.agentCpuQuota` | `"200%"` | CPU cap per agent, as a percentage of one core (`"200%"` = 2 cores). Raise if agents hit CPU limits during builds or heavy tool use. |
|
||||
| `services.hyperhive.c0re.agentMemoryMax` | `"4G"` | Memory cap per agent. Raise for agents that run large nix builds or hold big in-memory data. |
|
||||
| `services.hyperhive.c0re.agentCpuWeight` | `80` | `cpu.weight` share per agent, `1`–`10000` or `null` to omit the setting. Kernel default is `100`, so `80` makes agents yield. |
|
||||
| `services.hyperhive.c0re.agentIoWeight` | `80` | `io.weight` share per agent, same range and `null` handling. See the caveat below — it is a no-op on many hosts. |
|
||||
|
||||
Two things to know about the weights:
|
||||
|
||||
- They are **hive-wide** — unlike the caps there is no per-agent
|
||||
override in `meta/resource-limits.json`, so every agent carries the
|
||||
same value and the weight does *not* rank agents against each other.
|
||||
What `80` buys is that agents yield to everything **not** on this
|
||||
drop-in path: host services (nginx and dnsmasq among them) and the
|
||||
infra containers (`hive-ci`, `hive-forge`, `hive-gateway`,
|
||||
`hive-matrix`), which stay at the kernel default of `100`.
|
||||
- `IOWeight=` is only honoured when the backing device runs the BFQ
|
||||
scheduler or has blk-iocost QoS enabled. On a host using
|
||||
`none`/`mq-deadline`/`kyber` without iocost, systemd writes the value
|
||||
and the kernel ignores it — harmless, but it will measure as nothing.
|
||||
Check with `cat /sys/fs/cgroup/io.cost.qos`, and set the option to
|
||||
`null` if you would rather not write a setting nothing reads.
|
||||
|
||||
For a hive-wide cap across all containers together, set
|
||||
`systemd.slices.machine.serviceConfig.CPUQuota` in your NixOS
|
||||
config — all nspawn containers live in `machine.slice`.
|
||||
|
||||
### Pre-building agent templates
|
||||
|
||||
`preBuildAgentTemplates` (default `false`) causes the host NixOS
|
||||
build to pre-fetch the per-container system closures
|
||||
(`agent-base` + manager toplevels) into `/nix/store`, instead of
|
||||
leaving that work to the first `nixos-container start`. The
|
||||
trade-off:
|
||||
|
||||
- **On** (recommended for x86_64 hosts that care about first-spawn
|
||||
latency): the first `nixos-container start` for any new agent
|
||||
completes in seconds because nothing is left to fetch. Cost:
|
||||
the full nixpkgs runtime closure + claude-code + the harness
|
||||
binary are added to the host system closure (low single-digit GB
|
||||
additional).
|
||||
- **Off** (default): the host closure stays lean; the first spawn
|
||||
does all the eval + fetch work at runtime (can take several
|
||||
minutes on a fresh store).
|
||||
|
||||
**Note**: toplevels are pinned to `x86_64-linux`. Enabling on an
|
||||
`aarch64` host forces a cross-compilation or remote-builder build,
|
||||
which is almost never desired. Leave off on non-x86 hosts.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/approvals.md` — approval flow + scheduled prompts
|
||||
- `docs/persistence.md` — SQLite schema, state-dir layout
|
||||
- `docs/conventions.md` — wire protocol, recipient sentinels
|
||||
- `docs/agent-hierarchy.md` — topology and parent/child relations
|
||||
52
docs/scheduler/jobq.md
Normal file
52
docs/scheduler/jobq.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# The job queue, for operators
|
||||
|
||||
Every container operation — rebuild, first-spawn, a config-PR deploy,
|
||||
power changes — runs through one shared job queue. This page explains
|
||||
what the job queue *is*, as a general idea, independent of what any one
|
||||
subsystem uses it for. For the hive-c0re-specific step catalogue and the
|
||||
engineering internals (scheduler, leases, resource windows) see
|
||||
[`coordinator.md`](coordinator.md) instead.
|
||||
|
||||
## What the job queue is, in the abstract
|
||||
|
||||
"jobq" is a generic engine for running many interdependent jobs under
|
||||
limited concurrency — it has no idea what a "container" or a "rebuild" is.
|
||||
Two ideas are all there is to it:
|
||||
|
||||
- **A job is a small graph of steps**, not one opaque blob. Steps can
|
||||
depend on each other (this step only starts once that one finishes), so
|
||||
a big operation is really a short, ordered sequence — not a single
|
||||
black box that's either "done" or "not done."
|
||||
- **A step can need a shared resource**, which only so many steps can hold
|
||||
at once (a "slot"). If every currently-running step already holds the
|
||||
slots it needs, a new step that wants the same one waits its turn —
|
||||
that's the whole reason things queue instead of all firing at once.
|
||||
|
||||
The engine's whole job is: whenever a step's ordering and resource needs
|
||||
are both satisfied, run it. It has no opinion on what the steps *do* —
|
||||
that's supplied by whoever builds the graph. hive-c0re is the one thing
|
||||
building graphs on it today, but nothing about the engine is specific to
|
||||
containers or rebuilds; there's nothing stopping another subsystem from
|
||||
using the same engine for its own unrelated queue.
|
||||
|
||||
## Watching it happen
|
||||
|
||||
Each **row** you see in a queue view (the **BU1LDS** page's R3BU1LD QU3U3
|
||||
— see [`web-ui/dashboard.md`](../web-ui/dashboard.md) — and swarm-ui's
|
||||
`/jobs` page both render the same underlying graph) is one job; the rows
|
||||
nested under it are that job's steps, in order (occasionally a couple run
|
||||
side by side). A step shows one of:
|
||||
|
||||
| Glyph | Meaning |
|
||||
| ----- | -------------------------------------------------------- |
|
||||
| `⏸` | queued, waiting its turn |
|
||||
| `▶` | running |
|
||||
| `◐` | its own work is done, waiting on a step nested under it |
|
||||
| `✔` | finished successfully |
|
||||
| `✖` | failed |
|
||||
| `⊘` | cancelled |
|
||||
| `·` | skipped (not needed for this run) |
|
||||
|
||||
A step that isn't needed for a given run shows as `·` rather than being
|
||||
left out of the tree entirely, so the same kind of operation keeps a
|
||||
recognizable shape run to run, whichever steps it actually needed.
|
||||
387
docs/scheduler/observability.md
Normal file
387
docs/scheduler/observability.md
Normal file
|
|
@ -0,0 +1,387 @@
|
|||
# Observability (OpenTelemetry)
|
||||
|
||||
hyperhive has built-in support for exporting per-agent Claude Code statistics —
|
||||
token usage, cost, tool call counts — to any OTLP-compatible collector via
|
||||
Claude Code's built-in OpenTelemetry integration.
|
||||
|
||||
This is a **hive-wide** setting: one switch in the host NixOS config enables it
|
||||
for every agent container simultaneously. There is no per-agent opt-in or opt-out.
|
||||
|
||||
## Enabling export
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel = {
|
||||
enable = true;
|
||||
endpoint = "https://collector.example.com/otel";
|
||||
};
|
||||
```
|
||||
|
||||
`enable` is the single gate. `endpoint` is where telemetry ends up after it
|
||||
leaves the swarm — optional, because the swarm's own metrics store
|
||||
(`deploy.victoriametrics`) is a destination in its own right. With both,
|
||||
telemetry goes to both. See
|
||||
[`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana).
|
||||
|
||||
**There is exactly one way telemetry leaves a hive: through the collector that
|
||||
`enable` starts on the host.** Agents never talk to `endpoint` themselves —
|
||||
they export unauthenticated to a bridge address only their own containers can
|
||||
reach. That collector forwards to the swarm's
|
||||
([`swarm/services.md`](../swarm/services.md#telemetry-collector-otel)), which is
|
||||
the single process holding the upstream credential and the only writer to the
|
||||
swarm's store. No agent holds a copy, and neither does this hive.
|
||||
|
||||
The hive collector reaches the swarm collector by its gateway name
|
||||
(`swarm.otel.domain`, default `otel.<swarm domain>`) — the same DNS-and-CA-trust
|
||||
shape every hive-to-swarm-service hop uses, not a URL an operator has to point
|
||||
anywhere. A hive that does not run the swarm's services still resolves that
|
||||
name through the gateway; nothing here needs setting for the split-host case.
|
||||
|
||||
⚠️ **The collector is therefore in the path of all telemetry.** It runs on the
|
||||
same host as the agents and restarts on failure, and telemetry is not the
|
||||
control plane — degraded telemetry is not degraded operation — but the export
|
||||
no longer survives independently of anything host-side.
|
||||
|
||||
### what the agent→collector hop is and isn't
|
||||
|
||||
**It has no application-level auth.** The receiver takes any OTLP that reaches
|
||||
it; what bounds who can reach it is the firewall — `exposeHostPorts` opens the
|
||||
port on the bridge interface only. So "unauthenticated to a bridge address"
|
||||
means *reachable from an agent container*, not *presents a credential*.
|
||||
|
||||
The consequence, stated because it is a choice rather than an oversight: **any
|
||||
agent can push arbitrary OTLP, and it is forwarded on under the operator's
|
||||
credential.** Neither tier can tell a container's genuine Claude Code stats
|
||||
from anything else shaped like OTLP arriving on that port — including data
|
||||
smuggled out in resource attributes on an otherwise-legitimate export.
|
||||
|
||||
That is a **different risk from the one the collector fixes**, and strictly
|
||||
smaller than what preceded it: before, every agent held the upstream credential
|
||||
itself, so it could do all of the above *and* use the token anywhere else. The
|
||||
collector removes the token and keeps the pipe. Agents are inside the trust
|
||||
boundary (`docs/security.md`: capability = accepted risk), so an agent being
|
||||
able to *send* is an accepted extension of that boundary — but it is not
|
||||
closed by this design, and nothing here should be read as closing it.
|
||||
|
||||
**The `agent` label is self-reported, and no planned authentication changes
|
||||
that.** Treat it as a convenience for grouping dashboards, never as evidence of
|
||||
which container produced a sample: any agent that can reach this hive's
|
||||
collector can label its data as any other agent.
|
||||
|
||||
Worth spelling out, because two different hops are in play and only one of them
|
||||
is getting a credential:
|
||||
|
||||
- **agent→collector** (this section's hop) stays open on the bridge. Nothing
|
||||
downstream can tell one agent's export from another's.
|
||||
- **hive→swarm** is where the planned ingest auth goes. The swarm tier stamps
|
||||
`hive=` from the connection it authenticated, so *that* label becomes
|
||||
unforgeable.
|
||||
|
||||
So a verified `hive` is reachable and a verified `agent` is not — and that falls
|
||||
out of the topology rather than being a gap someone forgot to close. The swarm
|
||||
runs one collector, and the mechanism gives it no finer grain: a bearer-token
|
||||
check never reveals *which* token matched, and a receiver reads request metadata
|
||||
rather than the claims it authenticated with.
|
||||
|
||||
If you need per-agent numbers you can act on, take them from the agent's own
|
||||
turn-stats rather than from a metric label.
|
||||
|
||||
## Options reference
|
||||
|
||||
Every `services.hyperhive.otel.*` option's full type/default/description/
|
||||
example is generated straight from the nix module (`nix/host-modules/
|
||||
otel.nix`) into [`/options/`](/options/) (host options — `nix build
|
||||
.#docs-host` for a local render). That page is kept honest by the build in a
|
||||
way a hand-copied version here cannot be, so it is the reference, not this
|
||||
doc. What follows is what a flat per-option listing can't express: the
|
||||
two-tier architecture, the security model, and how the options interact.
|
||||
|
||||
## The two collectors
|
||||
|
||||
Telemetry crosses two collectors, and which one you configure depends on what
|
||||
the host is:
|
||||
|
||||
| | runs where | receives from | does |
|
||||
|---|---|---|---|
|
||||
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential, picks no destination |
|
||||
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's store and exports upstream |
|
||||
|
||||
An all-local host runs both, and needs nothing said about the hop between them.
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel = {
|
||||
enable = true;
|
||||
endpoint = "https://collector.example.com/otel"; # the upstream
|
||||
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
|
||||
};
|
||||
```
|
||||
|
||||
**Why the hive tier isn't optional.** Exporting straight to `endpoint` means
|
||||
every agent needs the credential to authenticate — and the harness delivers
|
||||
that token into the agent's own `~/.claude/settings.json`, a file the agent can
|
||||
read. `0600` protects it from other containers, not from the agent itself. As
|
||||
long as the direct path stays *selectable*, that hole stays selectable; an
|
||||
option that can reintroduce it is a hole with extra steps.
|
||||
|
||||
**Why the tiers stay separate on one box.** They are not collapsed when
|
||||
co-located: an all-local hive is a statement about *where* processes run, not
|
||||
about the shape of the deployment. A boundary that disappears locally is one
|
||||
the local deployment stops testing.
|
||||
|
||||
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
|
||||
redefines it — the agent-facing value is *derived*
|
||||
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
|
||||
keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
|
||||
automatically; there is nothing to open by hand.
|
||||
|
||||
### Authenticated ingest
|
||||
|
||||
The swarm tier gives **each hive its own receiver**, and stamps the `hive` label
|
||||
from whichever receiver accepted a sample. A hive therefore cannot report
|
||||
metrics as another hive, and cannot relabel its own by editing what it sends —
|
||||
the label is not taken from the payload at all.
|
||||
|
||||
**On an all-local swarm there is nothing to set.** Each hive already has an
|
||||
identity, and its collector reads the secret that host's own authelia minted.
|
||||
|
||||
**On a hive that does not host the swarm's services**, the secret has to arrive
|
||||
somehow — copy it across and name it:
|
||||
|
||||
```nix
|
||||
services.hyperhive.otel.clientSecretFile = "/run/secrets/hive-telemetry.secret";
|
||||
```
|
||||
|
||||
**There is no unauthenticated mode.** A hive always presents an identity, so a
|
||||
missing credential is a build error rather than a quieter fallback — the
|
||||
collector has no anonymous route to accept samples on, and every path it serves
|
||||
belongs to exactly one hive.
|
||||
|
||||
Getting the secret wrong shows up as the hive's collector logging 401s from the
|
||||
swarm tier and no metrics appearing for that hive.
|
||||
|
||||
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
|
||||
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
|
||||
exporter takes an *address*: `https://host/path` is a legal
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with *"missing port in
|
||||
address"*. The collector's config is validated at build time, so a mismatch is
|
||||
a build error naming the reason rather than telemetry silently going nowhere.
|
||||
|
||||
## Network access
|
||||
|
||||
Agent containers can only reach the host on ports 80 and 443 by default. To let
|
||||
them reach some other host-local service you run yourself — a database, a
|
||||
scratch HTTP endpoint — open its port on the bridge:
|
||||
|
||||
```nix
|
||||
services.hyperhive.network.exposeHostPorts = [ 5432 ];
|
||||
```
|
||||
|
||||
and point whatever consumes it at `10.42.0.1:5432` rather than loopback: inside
|
||||
a container, loopback is the *container*. The bridge IP is the host's address on
|
||||
the `hive-br0` bridge. The service must also bind an address the bridge can
|
||||
reach — a `127.0.0.1`-only listener stays unreachable no matter what the
|
||||
firewall allows. See `docs/network.md::Reaching host services` for details.
|
||||
|
||||
⚠️ **None of this is needed for hyperhive's own telemetry** — `otel.enable`
|
||||
contributes the collector's port and derives the agent-facing endpoint itself.
|
||||
|
||||
## Built-in resource labels
|
||||
|
||||
The OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
|
||||
`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) are set **container
|
||||
wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every
|
||||
process in an agent container exports to the hive's collector without any
|
||||
per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
|
||||
yourself from a tool call or `hivectl shell`.
|
||||
|
||||
Every agent's export therefore includes these resource attributes
|
||||
automatically:
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| `service.name` | `hyperhive-agent` (constant) |
|
||||
| `agent` | agent logical name (e.g. `iris`) |
|
||||
| `hive` | hive display name (`services.hyperhive.hiveName`) |
|
||||
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
|
||||
|
||||
Additional labels can be appended via `extraResourceAttributes` (see option
|
||||
reference above); custom per-data-point labels can be passed with
|
||||
`hive-metric --labels` (see below).
|
||||
|
||||
## Host-emitted container-resource metrics (hive-c0re)
|
||||
|
||||
When OTEL is enabled, **hive-c0re itself** also exports each agent
|
||||
container's resource load — the same cgroup gauges shown on the dashboard
|
||||
LOAD tab — to this hive's own collector, exactly like an agent does and with
|
||||
no separate toggle. These come from the host, not the in-container Claude SDK,
|
||||
so they cover containers even when their agent is idle.
|
||||
|
||||
Emitted via the OpenTelemetry Rust SDK, using the
|
||||
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
|
||||
metric names + the standard `container.name` attribute where a spec metric
|
||||
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
|
||||
`service.name = hyperhive-c0re`; each data point is tagged `container.name`
|
||||
(= the `h-<agent>` machine) and the hive `agent` label:
|
||||
|
||||
| Metric | Unit | Kind | Source |
|
||||
|--------|------|------|--------|
|
||||
| `container.cpu.time` | `s` | counter | cumulative `cpu.stat` `usage_usec` → seconds |
|
||||
| `container.memory.usage` | `By` | gauge | `memory.current` |
|
||||
| `hyperhive.container.memory.limit` | `By` | gauge | `memory.max` (custom — semconv has no `.limit` metric; omitted when unlimited) |
|
||||
| `hyperhive.container.memory.peak` | `By` | gauge | `memory.peak` (custom — no semconv metric; omitted if unavailable) |
|
||||
| `hyperhive.container.storage.usage` | `By` | gauge | state dir + writable rootfs (custom — semconv only has `disk.io`; omitted until the slow disk sampler runs) |
|
||||
| `hyperhive.container.cpu.percent` | `%` | gauge | host-normalised percent (custom — the value the dashboard LOAD tab shows, no `rate()` needed) |
|
||||
|
||||
The `hyperhive.`-prefixed metrics have no semconv equivalent (memory
|
||||
limit + peak, on-disk footprint, and an instantaneous cpu percent kept
|
||||
alongside the spec `container.cpu.time` counter for convenience). Hive
|
||||
labels (`hive`, `swarm`, …) ride on the resource via
|
||||
`extraResourceAttributes`.
|
||||
|
||||
Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP
|
||||
(JSON) to the hive collector's bridge address, with no auth header — that
|
||||
first hop is unauthenticated for every producer on this host, and the upstream
|
||||
credential stays on the swarm tier.
|
||||
|
||||
## Agent-emitted per-turn metrics (`hive-agent`)
|
||||
|
||||
When OTEL is enabled, the harness itself (`hive-agent`) exports one small set
|
||||
of metrics per claude turn, recorded the moment the turn ends (not polled).
|
||||
These are deliberately the fields Claude Code's own built-in export (see
|
||||
above) can't know about — the harness's own wall-clock timing, what woke the
|
||||
turn, its own outcome classification, the loose-ends backlog, and session
|
||||
boundaries. Token usage, cost, and tool-call counts are **not** duplicated
|
||||
here; that's already covered by Claude's own export.
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
|--------|------|------|------------|
|
||||
| `hyperhive.agent.turn.duration` | `ms` | histogram | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.turn.count` | — | counter | `wake_from`, `result_kind`, `model` |
|
||||
| `hyperhive.agent.session.count` | — | counter | `model` (incremented once per fresh, non-`--continue`'d session) |
|
||||
| `hyperhive.agent.loose_ends.threads` | — | gauge | none |
|
||||
| `hyperhive.agent.loose_ends.reminders` | — | gauge | none |
|
||||
| `hyperhive.agent.claude_md.lines` | — | gauge | none — recorded from the `CLAUDE.md`-size watch's own ~15-minute tick, **not** per turn like the rows above |
|
||||
|
||||
Resource attributes (`service.name`, `agent`, `hive`, `swarm`) come from the
|
||||
same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
|
||||
section — nothing extra to configure. Cadence follows
|
||||
`HYPERHIVE_OTEL_METRIC_INTERVAL_MS` (default 60s, same variable + default as
|
||||
`hive-c0re`'s container-resource export above) — that only controls how often
|
||||
the batched points are flushed to the collector, not how often they're
|
||||
recorded (every turn, always).
|
||||
|
||||
## Hive-scoped metrics (hive-c0re)
|
||||
|
||||
Everything above is measured **per agent**, tagged with the hive it runs in.
|
||||
These three are measured per **hive**, and carry no `agent` label — so a hive
|
||||
that hosts no agents still reports, and "this hive is quiet" is
|
||||
distinguishable from "this hive is gone". Select them with
|
||||
`{hive!="",agent=""}`.
|
||||
|
||||
| Metric | Unit | Kind | Meaning |
|
||||
|--------|------|------|---------|
|
||||
| `process.uptime` | `s` | gauge | seconds since this hive's `hive-c0re` started exporting; a restart reads as a drop to ~0 |
|
||||
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
|
||||
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
|
||||
|
||||
Both levels are reported every cycle, `0` included, so a healthy hive is
|
||||
visible as zeros rather than as missing series.
|
||||
|
||||
`hyperhive.hive.degraded` is what a dashboard should alert on: it is
|
||||
`hive-c0re`'s own readiness verdict, so it stays in step with `/health/ready`
|
||||
and with what the swarm controller sees. `hyperhive.hive.warnings` is the
|
||||
detail behind it — `warn`-level entries mean "an operator should look" and do
|
||||
**not** set `degraded`.
|
||||
|
||||
Same cadence, transport and resource labels as the container metrics above.
|
||||
|
||||
## VCS activity metrics (`swarm-controller`)
|
||||
|
||||
`swarm-controller` registers a single instance-wide Forgejo webhook (a
|
||||
"global/system" hook, not scoped to any one org or repo) and counts commit
|
||||
and push activity as deliveries arrive — occurrence-driven, not polled.
|
||||
Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
|
||||
counts of durable rows (issues, comments, repos), and neither a commit nor a
|
||||
push is stored anywhere as a row to count.
|
||||
|
||||
| Metric | Unit | Kind | Attributes |
|
||||
|--------|------|------|------------|
|
||||
| `hyperhive.vcs.commit.count` | — | counter | `repo` (`org/repo`) |
|
||||
| `hyperhive.vcs.push.count` | — | counter | `repo` (`org/repo`) |
|
||||
|
||||
A push with zero commits (a branch delete, or a force-push that doesn't add
|
||||
new commits) still increments `push.count`; `commit.count` only advances
|
||||
when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_OTLP_ENDPOINT`), cadence variable
|
||||
(`HYPERHIVE_OTEL_METRIC_INTERVAL_MS`) and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`
|
||||
resource-attribute channel as `swarm-controller`'s other OTEL exporter (its
|
||||
`hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that
|
||||
crate's own doc comment) — `service.name = swarm-controller` is set
|
||||
directly rather than read from the container environment, since
|
||||
`swarm-controller` is a standalone daemon, not a per-agent harness process.
|
||||
|
||||
## Agent-emitted custom metrics (`hive-metric`)
|
||||
|
||||
Agents can push arbitrary labeled metrics to the same OTEL collector via the
|
||||
`hive-metric` CLI tool, available in every agent container when
|
||||
`services.hyperhive.otel.enable = true`.
|
||||
|
||||
### Usage
|
||||
|
||||
```text
|
||||
hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulative] [--labels key=value...]
|
||||
```
|
||||
|
||||
- `<name>` — metric name (e.g. `tasks_completed`, `latency_ms`).
|
||||
- `<value>` — numeric value (f64; integers and floats both accepted).
|
||||
- `--type counter|gauge` — metric kind: `counter` (increasing sum, default) or
|
||||
`gauge` (instantaneous point-in-time value).
|
||||
- `--temporality delta|cumulative` — counter reporting mode (`counter` only,
|
||||
ignored for `gauge`): `delta` (this call's own contribution, default — send
|
||||
`1` each time and the collector accumulates) or `cumulative` (this call
|
||||
reports the running total, which a stateless one-shot CLI can't track
|
||||
itself).
|
||||
- `--labels key=value` — extra per-data-point labels. May be repeated.
|
||||
The resource labels (agent, hive, swarm, service.name) are inherited
|
||||
automatically from `OTEL_RESOURCE_ATTRIBUTES` — do not re-specify them.
|
||||
|
||||
### Examples
|
||||
|
||||
```text
|
||||
# Counter: one more task finished (delta is the default — no flag needed)
|
||||
hive-metric tasks_completed 1 --labels phase=scan
|
||||
|
||||
# Gauge: current queue depth (absolute value — must use --type gauge)
|
||||
hive-metric queue_depth 17 --type gauge
|
||||
|
||||
# Float gauge with multiple labels (instantaneous measurement)
|
||||
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
|
||||
```
|
||||
|
||||
### Error when OTEL is not configured
|
||||
|
||||
When `services.hyperhive.otel.enable = false` (the default), the
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` env var is not set and `hive-metric` exits
|
||||
with an informative error message. No silently-dropped metrics.
|
||||
|
||||
### Wire format
|
||||
|
||||
`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
|
||||
`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
|
||||
`OTEL_EXPORTER_OTLP_PROTOCOL` setting. Auth headers from
|
||||
`OTEL_EXPORTER_OTLP_HEADERS` are forwarded verbatim.
|
||||
|
||||
## Metrics temporality
|
||||
|
||||
OTEL export is configured with **cumulative** temporality by default
|
||||
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
|
||||
overriding Claude Code's default of DELTA. This avoids silent metric drops in
|
||||
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a
|
||||
delta-to-cumulative processor.
|
||||
|
||||
**`hive-metric` counters are the one exception**, reporting delta by default
|
||||
(see above) — programmatically set on the exporter, which overrides this
|
||||
container-wide env var for that tool specifically. `--type gauge` is
|
||||
unaffected either way; gauges have no temporality. The hive-tier collector
|
||||
runs a `deltatocumulative` processor ahead of export, so a delta
|
||||
`hive-metric` counter still lands in VictoriaMetrics as a cumulative
|
||||
series — the standard `rate()`/`increase()` idioms work on it exactly like
|
||||
any other counter in this system, no special query needed.
|
||||
Loading…
Reference in a new issue