docs: fix genuine passive-voice hits in docs/scheduler

Seventh batch of the ongoing write-good.Passive pass (hyperhive#4042):
read all 65 hits across jobq.md/ci.md/observability.md/coordinator.md
in context and rewrote 41 with a clearly nameable actor -- mostly
hive-c0re, nix/the nix module, the harness, or a specific fn/type
named right there or nearby (coordinator.md's node-inventory table
and DAG-shape descriptions name concrete Rust items constantly, so
the actor is almost always sitting in the same sentence).

Left 24 alone: predicate-adjective-copula state descriptions ("is
stuck", "is gone", "is unaffected", "is done", etc. -- the largest
recurring bucket this batch, especially in observability.md's
scope/status descriptions), negative-capability idioms ("no X is
needed/left", "X can't be written down"), the established "is
tracked as a follow-up" idiom, a firewall-shorthand notation
("bridge->127.0.0.0/8 is dropped") where rewriting would break the
compact rule-like format, a CLI-flag "(repeatable)" annotation ("May
be repeated"), a Rust type-signature fact ("`moves` is typed ..."),
a hypothetical/counterfactual maintenance-burden clause, a
readiness-condition list ("a node is ready when ... every dep is
satisfied"), and one deliberately-parallel idiom pair
("When OTEL is enabled" used identically twice as a section-opening
convention -- fixing one would break the parallelism, not the
opposite).

One self-caught regression: an early attempt to fix "used by every
`Reconcile` node's start action" (a reduced participial clause, not
flagged) into "is used by every `Reconcile` node's start action"
introduced a brand-new flagged passive. Caught by the post-edit vale
count (expected 65->24, got 65->25) not matching, same discipline as
the docs/turn-loop batch's tail-truncation catch -- re-ran with
active voice instead ("Every `Reconcile` node's start action uses
this fallback").

Verified via vale before/after: 65 -> 24 write-good.Passive hits,
exactly the 24 left alone above; error count and other warning
categories unchanged. Re-read every changed line in full surrounding
context after editing before running the final vale check.
This commit is contained in:
iris 2026-09-08 12:56:50 +02:00 committed by mara
commit 09e4e2f5e9
4 changed files with 68 additions and 68 deletions

View file

@ -56,8 +56,8 @@ never writes to that status slot. Reproduced with three dispatches on one
PR over 20+ minutes, all genuinely green, the PR's own status unmoved PR over 20+ minutes, all genuinely green, the PR's own status unmoved
throughout. **Always re-check `pr-status` after a dispatch and believe throughout. **Always re-check `pr-status` after a dispatch and believe
what it says — don't push a commit just to unwedge it, that's the what it says — don't push a commit just to unwedge it, that's the
empty-commit anti-pattern this verb exists to avoid.** If the code is empty-commit anti-pattern this verb exists to avoid.** If `hive-forge
verified green (`hive-forge ci-log --run <n> --job 0`, verdict is the ci-log --run <n> --job 0` shows the code green (verdict is the
last line) but the status is stuck, ask the operator to select "rerun" in last line) but the status is stuck, ask the operator to select "rerun" in
the forge web UI — it's CSRF-gated, so only they can do it. the forge web UI — it's CSRF-gated, so only they can do it.
@ -74,8 +74,8 @@ sh scripts/check-issue-refs.sh # tracker-tag lint
sh scripts/check-comment-blocks.sh # comment-block lint sh scripts/check-comment-blocks.sh # comment-block lint
``` ```
A git pre-push hook that automates the two lint checks is provided at `scripts/pre-push` provides a git pre-push hook that automates the two
`scripts/pre-push`. Install it once per clone: lint checks. Install it once per clone:
```sh ```sh
ln -sf ../../scripts/pre-push .git/hooks/pre-push ln -sf ../../scripts/pre-push .git/hooks/pre-push
@ -112,13 +112,13 @@ it and runs it out of band, so a slow forge or core-token never delays the
container's start. Gotcha: don't gate the container's own start on a forge container's start. Gotcha: don't gate the container's own start on a forge
round-trip (a host-side unit that did this could exceed the nspawn start round-trip (a host-side unit that did this could exceed the nspawn start
timeout and trip a restart loop) — registration must stay something timeout and trip a restart loop) — registration must stay something
hive-c0re drives after the container is already up. The core admin token is hive-c0re drives after the container is already up. hive-c0re holds the
held only by hive-c0re on the host; only the runner registration token core admin token only on the host; only the runner registration token
reaches the container. reaches the container.
### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep) ### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)
Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — failures are logged and never abort the sweep; a healthy runner is never restarted. Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — hive-c0re logs failures and never aborts the sweep; a healthy runner is never restarted.
1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token: 1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
- **200**: still registered — done, no restart. - **200**: still registered — done, no restart.
@ -141,7 +141,7 @@ When `deploy.forgejo.ci.enable` is set, hive-c0re autoseeds an
`uses: actions/checkout@vN` steps resolve entirely on loopback — no `uses: actions/checkout@vN` steps resolve entirely on loopback — no
external DNS on the CI critical path. external DNS on the CI critical path.
The mirror is seeded by **hive-c0re** itself during its forge **hive-c0re** itself seeds the mirror during its forge
provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module
forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the
`hive-c0re` service environment (JSON-encoded `[{upstream, dest}]` `hive-c0re` service environment (JSON-encoded `[{upstream, dest}]`
@ -159,11 +159,11 @@ services.hyperhive.deploy.forgejo.mirrors = [
]; ];
``` ```
Each entry is created as a real Forgejo pull-mirror — not a one-off hive-c0re creates each entry as a real Forgejo pull-mirror — not a one-off
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack` clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
request), so a DNS blip during that sync will propagate back to the request), so a DNS blip during that sync will propagate back to the
runner as a hard `git clone` failure. The runner as a hard `git clone` failure. hive-c0re
`<owner>` org in `dest` is autocreated. Keep mirror dests out of the autocreates the `<owner>` org in `dest`. Keep mirror dests out of the
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`) hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
to avoid provisioning collisions. to avoid provisioning collisions.
@ -173,7 +173,7 @@ to avoid provisioning collisions.
### What unsandboxed builds mean ### What unsandboxed builds mean
nspawn containers can't create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase. nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
A malicious `default.nix` or build script in a PR can therefore: A malicious `default.nix` or build script in a PR can therefore:
@ -235,15 +235,15 @@ following to your host config:
``` ```
**Measure a collection on your own hosts before relying on that stagger.** **Measure a collection on your own hosts before relying on that stagger.**
`randomizedDelaySec` spreads the _start_, not the run, and the run is dominated `randomizedDelaySec` spreads the _start_, not the run, and the sweep over
by the sweep over unused `.links` entries — which scales with the size of the unused `.links` entries dominates the run — which scales with the size of the
store, not with what this collection deletes. On a large store it can take most store, not with what this collection deletes. On a large store it can take most
of an hour, so two hosts both set to `daily` may overlap however much jitter you of an hour, so two hosts both set to `daily` may overlap however much jitter you
give them. `journalctl -u nix-gc.service` reports each run's wall clock. give them. `journalctl -u nix-gc.service` reports each run's wall clock.
**Remote builders:** if CI dispatches builds to a remote builder (for example via **Remote builders:** if CI dispatches builds to a remote builder (for example via
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_ `nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
store, so the same GC config should be applied wherever the builder runs — store, so apply the same GC config wherever the builder runs —
GC on the coordinator host won't reclaim space on the builder. GC on the coordinator host won't reclaim space on the builder.
⚠️ Two stores also means the temp-root protection above **doesn't span ⚠️ Two stores also means the temp-root protection above **doesn't span

View file

@ -14,8 +14,8 @@ section covers the concrete node inventory hive-c0re builds on top of that
engine, plus the internals (module layout, scheduler mechanics, engine, plus the internals (module layout, scheduler mechanics,
resource/lease semantics). resource/lease semantics).
Every container/meta operation (rebuild, meta-update, first-spawn, power hive-c0re submits every container/meta operation (rebuild, meta-update, first-spawn, power
changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`) changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of primitive nodes**. One scheduler task drives all DAGs; as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers. concurrency comes from the resource classes below, not from multiple workers.
Special cases like graceful stop, deferred starts, and the meta-update Special cases like graceful stop, deferred starts, and the meta-update
@ -63,7 +63,7 @@ Cheap — no build slot:
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down | | `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up | | `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace | | `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal``Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ | | `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal``Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn | | `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way | | `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
@ -101,7 +101,7 @@ container build:
phase below it (it declares `Resource::MetaWindow`). Two meta phase below it (it declares `Resource::MetaWindow`). Two meta
mutations can therefore never interleave, so no commit lands inside another mutations can therefore never interleave, so no commit lands inside another
node's staged window. It's a queue resource rather than a runtime mutex node's staged window. It's a queue resource rather than a runtime mutex
because a resource is held by a subtree root across its whole subtree, which because a subtree root holds a resource across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) can't — that's what lets a a `MutexGuard` (bounded by one executor fn) can't — that's what lets a
multi-node deploy own one window. For the same reason the window must stay multi-node deploy own one window. For the same reason the window must stay
_off_ long store-only work: the rebuild's meta preamble is its own _off_ long store-only work: the rebuild's meta preamble is its own
@ -127,9 +127,9 @@ a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
subgraph each (independent roots, run concurrently on their own leases), not subgraph each (independent roots, run concurrently on their own leases), not
N separate DAGs. N separate DAGs.
**These are built dynamically from each agent's live running state** (an **`job_queue/power.rs` builds these dynamically from each agent's live
async `lifecycle::is_running` read), so they live in `job_queue/power.rs`, running state** (an async `lifecycle::is_running` read) — that's why they
not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry live there rather than in the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile` a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile`
(convergence guarantee — cheap, noops when already converged) is ALWAYS (convergence guarantee — cheap, noops when already converged) is ALWAYS
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
@ -192,8 +192,8 @@ rides the queue through that power layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts and crash-watch suppression can't drift per surface; the only direct starts
left are the root-agent bootstrap and infra containers (no lease, no left are the root-agent bootstrap and infra containers (no lease, no
harness). Cancelling a still-queued power DAG reverts `wanted` to the harness). Cancelling a still-queued power DAG reverts `wanted` to the
observed state — a cancel means "don't do it," not "do it later." Agents observed state — a cancel means "don't do it," not "do it later." hive-c0re
without a row are seeded from observed state on first touch (running ⇒ seeds agents without a row from observed state on first touch (running ⇒
`Up`); destroy removes the row. `Up`); destroy removes the row.
The admin-socket responses carry the submitted DAG ids; `hivectl` polls The admin-socket responses carry the submitted DAG ids; `hivectl` polls
@ -275,7 +275,7 @@ reconcile_), so there is no durable-recovery machinery to go wrong.
### Cancel, history ### Cancel, history
Submit-time dedup was removed with the agent-per-node move (a multi-agent DAG The agent-per-node move removed submit-time dedup (a multi-agent DAG
has no single agent to key a dedup on), so every submit enqueues a fresh DAG; has no single agent to key a dedup on), so every submit enqueues a fresh DAG;
whether any dedup needs reintroducing is tracked as a follow-up. whether any dedup needs reintroducing is tracked as a follow-up.
@ -297,13 +297,13 @@ DeployWindow (root — build slot + lease + meta window, no work of its own)
└── DeployTail AfterAny(apply) compensate, mirror to forge └── DeployTail AfterAny(apply) compensate, mirror to forge
``` ```
The root's resources are held across the whole subtree, so the two-phase The root holds its resources across the whole subtree, so the two-phase
`prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock` `prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock`
protected even though the phases are separate nodes. Splitting them buys protected even though the phases are separate nodes. Splitting them buys
three things a single opaque node couldn't have: per-phase visibility on the three things a single opaque node couldn't have: per-phase visibility on the
dashboard, a `MergeVerify` failure that provably mutated nothing, and a dashboard, a `MergeVerify` failure that provably mutated nothing, and a
compensation step that survives a hive-c0re restart — the pre-merge compensation step that survives a hive-c0re restart — `DeployApply` parks the pre-merge
`applied/main` is parked in `refs/hyperhive/rollback/<approval-id>`, not in a `applied/main` in `refs/hyperhive/rollback/<approval-id>`, not in a
local variable, so `DeployTail` can still undo a half-finished deploy after a local variable, so `DeployTail` can still undo a half-finished deploy after a
crash. crash.
@ -378,9 +378,9 @@ vs the current flake path) and persisted `wanted` intent, then:
Booting with no config change performs **no meta commit** — only reconciles. Booting with no config change performs **no meta commit** — only reconciles.
The sweep reason records the rebuild / deferred / up-to-date counts so the The sweep reason records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. Agents without an operator sees at a glance how much work the boot triggered. `auto_update::run`
`agent_power` row are seeded from observed state during classification (the seeds agents without an `agent_power` row from observed state during
one-time migration; thereafter the DB is authoritative). classification (the one-time migration; thereafter the DB is authoritative).
## Meta flake ## Meta flake
@ -435,7 +435,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
The container keeps serving the previous generation while eval + fetch + The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface cached and skips straight to the profile-swap. Build failures surface
here, before the running container is touched. Skipped when the here, before `Swap` touches the running container. Skipped when the
container is already down — a stopped agent has no uptime to preserve, container is already down — a stopped agent has no uptime to preserve,
so `Swap` builds inline instead of paying the double eval. so `Swap` builds inline instead of paying the double eval.
3. `StopForUpdate` — bring the container down (noop when already stopped). 3. `StopForUpdate` — bring the container down (noop when already stopped).
@ -449,9 +449,9 @@ The approval deploy uses this same chain rather than a rebuild path of its
own. Its `DeployApply` node doesn't build: it merges, opens the two-phase own. Its `DeployApply` node doesn't build: it merges, opens the two-phase
meta deploy, and returns the chain above as a subgraph the scheduler grafts meta deploy, and returns the chain above as a subgraph the scheduler grafts
into the live DAG under that node. A `FinalizeDeploy` node gated on the into the live DAG under that node. A `FinalizeDeploy` node gated on the
graft's completion then plants the deploy tag — so "did the agent come back graft's completion then plants the deploy tag — so `Reconcile`'s success
up?" is answered by `Reconcile` succeeding, the same way it's for every answers "did the agent come back up?", the same way it does for every
other rebuild, instead of by a fused inline start. other rebuild, instead of a fused inline start.
The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter
the meta window and build slot it already holds rather than deadlocking the meta window and build slot it already holds rather than deadlocking
@ -466,9 +466,9 @@ half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes) Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
`start` (clean cold-start, no generation transition, new activation runs cleanly). `start` (clean cold-start, no generation transition, new activation runs cleanly).
Both errors are preserved and surfaced if the cold-start also fails. The fallback `lifecycle::start_with_fallback` preserves and surfaces both errors if the
lives in `lifecycle::start_with_fallback`, used by every `Reconcile` node's cold-start also fails. Every `Reconcile` node's start action uses this
start action. fallback.
### Spawn path (new container) ### Spawn path (new container)
@ -480,8 +480,8 @@ flags → `systemctl daemon-reload` → `start`.
### Prebuild attr path ### Prebuild attr path
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the `nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the
way `nixos-container` does internally. The explicit attr path way `nixos-container` does internally. nix requires the explicit attr path
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel` is required; `<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`;
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`, using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
or the flake root directly — none of which exist in the rendered meta flake. or the flake root directly — none of which exist in the rendered meta flake.
@ -556,14 +556,14 @@ trade-off:
- **On** (recommended for x86_64 hosts that care about first-spawn - **On** (recommended for x86_64 hosts that care about first-spawn
latency): the first `nixos-container start` for any new agent latency): the first `nixos-container start` for any new agent
completes in seconds because nothing is left to fetch. Cost: completes in seconds because nothing is left to fetch. Cost:
the full nixpkgs runtime closure + claude-code + the harness `preBuildAgentTemplates` adds the full nixpkgs runtime closure + claude-code + the harness
binary are added to the host system closure (low single-digit GB binary to the host system closure (low single-digit GB
additional). additional).
- **Off** (default): the host closure stays lean; the first spawn - **Off** (default): the host closure stays lean; the first spawn
does all the eval + fetch work at runtime (can take a few does all the eval + fetch work at runtime (can take a few
minutes on a fresh store). minutes on a fresh store).
**Note**: toplevels are pinned to `x86_64-linux`. Enabling on an **Note**: the meta flake pins toplevels to `x86_64-linux`. Enabling on an
`aarch64` host forces a cross-compilation or remote-builder build, `aarch64` host forces a cross-compilation or remote-builder build,
which is almost never desired. Leave off on non-x86 hosts. which is almost never desired. Leave off on non-x86 hosts.

View file

@ -47,6 +47,6 @@ side by side). A step shows one of:
| `⊘` | cancelled | | `⊘` | cancelled |
| `·` | skipped (not needed for this run) | | `·` | skipped (not needed for this run) |
A step that isn't needed for a given run shows as `·` rather than being A step that isn't needed for a given run shows as `·` rather than
left out of the tree entirely, so the same kind of operation keeps a dropping out of the tree entirely, so the same kind of operation keeps a
recognizable shape run to run, whichever steps it actually needed. recognizable shape run to run, whichever steps it actually needed.

View file

@ -87,10 +87,10 @@ turn-stats rather than from a metric label.
## Options reference ## Options reference
Every `services.hyperhive.otel.*` option's full type/default/description/ The nix module (`nix/host-modules/otel.nix`) generates every
example is generated straight from the nix module (`nix/host-modules/ `services.hyperhive.otel.*` option's full type/default/description/
otel.nix`) into [`/options/`](/options/) (host options — `nix build example straight into [`/options/`](/options/) (host options — `nix build
.#docs-host` for a local render). That page is kept honest by the build in a .#docs-host` for a local render). The build keeps that page honest in a
way a hand-copied version here can't be, so it's the reference, not this way a hand-copied version here can't be, so it's the reference, not this
doc. What follows is what a flat per-option listing can't express: the doc. What follows is what a flat per-option listing can't express: the
two-tier architecture, the security model, and how the options interact. two-tier architecture, the security model, and how the options interact.
@ -130,7 +130,7 @@ the local deployment stops testing.
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier **`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
redefines it — the agent-facing value is _derived_ redefines it — the agent-facing value is _derived_
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint` (`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The bridge port is contributed to `exposeHostPorts` keeps working unchanged. The otel module contributes the bridge port to `exposeHostPorts`
automatically; there is nothing to open by hand. automatically; there is nothing to open by hand.
### Authenticated ingest ### Authenticated ingest
@ -162,7 +162,7 @@ swarm tier and no metrics appearing for that hive.
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC `otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an _address_: `https://host/path` is a legal exporter takes an _address_: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in `OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
address"_. The collector's config is validated at build time, so a mismatch is address"_. nix validates the collector's config at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere. a build error naming the reason rather than telemetry silently going nowhere.
## Network access ## Network access
@ -186,8 +186,8 @@ contributes the collector's port and derives the agent-facing endpoint itself.
## Built-in resource labels ## Built-in resource labels
The OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`, The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) are set **container `OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container
wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every
process in an agent container exports to the hive's collector without any process in an agent container exports to the hive's collector without any
per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
@ -203,8 +203,8 @@ automatically:
| `hive` | hive display name (`services.hyperhive.hiveName`) | | `hive` | hive display name (`services.hyperhive.hiveName`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) | | `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
Additional labels can be appended via `extraResourceAttributes` (see option Append additional labels via `extraResourceAttributes` (see option
reference above); custom per-data-point labels can be passed with reference above); pass custom per-data-point labels with
`hive-metric --labels` (see below). `hive-metric --labels` (see below).
## Host-emitted container-resource metrics (hive-c0re) ## Host-emitted container-resource metrics (hive-c0re)
@ -219,7 +219,7 @@ Emitted via the OpenTelemetry Rust SDK, using the
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/) [semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
metric names + the standard `container.name` attribute where a spec metric metric names + the standard `container.name` attribute where a spec metric
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; each data point is tagged `container.name` `service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name`
(= the `h-<agent>` machine) and the hive `agent` label: (= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source | | Metric | Unit | Kind | Source |
@ -266,8 +266,8 @@ same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
section — nothing extra to configure. Cadence follows section — nothing extra to configure. Cadence follows
`HYPERHIVE_OTEL_METRIC_INTERVAL_MS` (default 60s, same variable + default as `HYPERHIVE_OTEL_METRIC_INTERVAL_MS` (default 60s, same variable + default as
`hive-c0re`'s container-resource export above) — that only controls how often `hive-c0re`'s container-resource export above) — that only controls how often
the batched points are flushed to the collector, not how often they're the harness flushes the batched points to the collector, not how often it
recorded (every turn, always). records them (every turn, always).
## Hive-scoped metrics (hive-c0re) ## Hive-scoped metrics (hive-c0re)
@ -283,7 +283,7 @@ distinguishable from "this hive is gone." Select them with
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows | | `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) | | `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
Both levels are reported every cycle, `0` included, so a healthy hive is hive-c0re reports both levels every cycle, `0` included, so a healthy hive is
visible as zeros rather than as missing series. visible as zeros rather than as missing series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's `hyperhive.hive.degraded` is what a dashboard should alert on: it's
@ -300,8 +300,8 @@ Same cadence, transport and resource labels as the container metrics above.
"global/system" hook, not scoped to any one org or repo) and counts commit "global/system" hook, not scoped to any one org or repo) and counts commit
and push activity as deliveries arrive — occurrence-driven, not polled. and push activity as deliveries arrive — occurrence-driven, not polled.
Forgejo's own native `/metrics` endpoint has no equivalent: it exposes Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
counts of durable rows (issues, comments, repos), and neither a commit nor a counts of durable rows (issues, comments, repos), and Forgejo doesn't
push is stored anywhere as a row to count. store either a commit or a push anywhere as a row to count.
| Metric | Unit | Kind | Attributes | | Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ------- | ------------------- | | ---------------------------- | ---- | ------- | ------------------- |
@ -314,9 +314,9 @@ when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_O
(`HYPERHIVE_OTEL_METRIC_INTERVAL_MS`) and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES` (`HYPERHIVE_OTEL_METRIC_INTERVAL_MS`) and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`
resource-attribute channel as `swarm-controller`'s other OTEL exporter (its resource-attribute channel as `swarm-controller`'s other OTEL exporter (its
`hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that `hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that
crate's own doc comment) — `service.name = swarm-controller` is set crate's own doc comment) — `swarm-controller` sets `service.name =
directly rather than read from the container environment, since swarm-controller` directly rather than reading it from the container
`swarm-controller` is a standalone daemon, not a per-agent harness process. environment, since it's a standalone daemon, not a per-agent harness process.
## Agent-emitted custom metrics (`hive-metric`) ## Agent-emitted custom metrics (`hive-metric`)
@ -340,7 +340,7 @@ hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulativ
reports the running total, which a stateless one-shot CLI can't track reports the running total, which a stateless one-shot CLI can't track
itself). itself).
- `--labels key=value` — extra per-data-point labels. May be repeated. - `--labels key=value` — extra per-data-point labels. May be repeated.
The resource labels (agent, hive, swarm, service.name) are inherited `hive-metric` inherits the resource labels (agent, hive, swarm, service.name)
automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them. automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them.
### Examples ### Examples
@ -366,12 +366,12 @@ with an informative error message. No silently dropped metrics.
`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to `hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the `$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
`OTEL_EXPORTER_OTLP_PROTOCOL` setting. Auth headers from `OTEL_EXPORTER_OTLP_PROTOCOL` setting. `hive-metric` forwards auth headers from
`OTEL_EXPORTER_OTLP_HEADERS` are forwarded verbatim. `OTEL_EXPORTER_OTLP_HEADERS` verbatim.
## Metrics temporality ## Metrics temporality
OTEL export is configured with **cumulative** temporality by default The harness configures OTEL export with **cumulative** temporality by default
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`), (`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
overriding Claude Code's default of DELTA. This avoids silent metric drops in overriding Claude Code's default of DELTA. This avoids silent metric drops in
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a