Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
4 changed files with 68 additions and 68 deletions

View file

@ -56,8 +56,8 @@ never writes to that status slot. Reproduced with three dispatches on one
PR over 20+ minutes, all genuinely green, the PR's own status unmoved PR over 20+ minutes, all genuinely green, the PR's own status unmoved
throughout. **Always re-check `pr-status` after a dispatch and believe throughout. **Always re-check `pr-status` after a dispatch and believe
what it says — don't push a commit just to unwedge it, that's the what it says — don't push a commit just to unwedge it, that's the
empty-commit anti-pattern this verb exists to avoid.** If `hive-forge empty-commit anti-pattern this verb exists to avoid.** If the code is
ci-log --run <n> --job 0` shows the code green (verdict is the verified green (`hive-forge ci-log --run <n> --job 0`, verdict is the
last line) but the status is stuck, ask the operator to select "rerun" in last line) but the status is stuck, ask the operator to select "rerun" in
the forge web UI — it's CSRF-gated, so only they can do it. the forge web UI — it's CSRF-gated, so only they can do it.
@ -74,8 +74,8 @@ sh scripts/check-issue-refs.sh # tracker-tag lint
sh scripts/check-comment-blocks.sh # comment-block lint sh scripts/check-comment-blocks.sh # comment-block lint
``` ```
`scripts/pre-push` provides a git pre-push hook that automates the two A git pre-push hook that automates the two lint checks is provided at
lint checks. Install it once per clone: `scripts/pre-push`. Install it once per clone:
```sh ```sh
ln -sf ../../scripts/pre-push .git/hooks/pre-push ln -sf ../../scripts/pre-push .git/hooks/pre-push
@ -112,13 +112,13 @@ it and runs it out of band, so a slow forge or core-token never delays the
container's start. Gotcha: don't gate the container's own start on a forge container's start. Gotcha: don't gate the container's own start on a forge
round-trip (a host-side unit that did this could exceed the nspawn start round-trip (a host-side unit that did this could exceed the nspawn start
timeout and trip a restart loop) — registration must stay something timeout and trip a restart loop) — registration must stay something
hive-c0re drives after the container is already up. hive-c0re holds the hive-c0re drives after the container is already up. The core admin token is
core admin token only on the host; only the runner registration token held only by hive-c0re on the host; only the runner registration token
reaches the container. reaches the container.
### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep) ### hive-c0re side (`forge/ci_runner.rs`, run during the startup sweep)
Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — hive-c0re logs failures and never aborts the sweep; a healthy runner is never restarted. Gated on `HYPERHIVE_FORGE_CI_ENABLED` (the nix module sets it on `hive-c0re.service` when `deploy.forgejo.ci.enable`). Best-effort — failures are logged and never abort the sweep; a healthy runner is never restarted.
1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token: 1. If `.runner` exists at `/var/lib/nixos-containers/hive-ci/var/lib/gitea-runner/hive/.runner`, validate its id against `GET /api/v1/admin/runners/{id}` with the core admin token:
- **200**: still registered — done, no restart. - **200**: still registered — done, no restart.
@ -141,7 +141,7 @@ When `deploy.forgejo.ci.enable` is set, hive-c0re autoseeds an
`uses: actions/checkout@vN` steps resolve entirely on loopback — no `uses: actions/checkout@vN` steps resolve entirely on loopback — no
external DNS on the CI critical path. external DNS on the CI critical path.
**hive-c0re** itself seeds the mirror during its forge The mirror is seeded by **hive-c0re** itself during its forge
provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module provisioning sweep (`forge/repos.rs::ensure_mirrors`). The nix module
forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the forwards the effective mirror list as `HYPERHIVE_FORGE_MIRRORS` in the
`hive-c0re` service environment (JSON-encoded `[{upstream, dest}]` `hive-c0re` service environment (JSON-encoded `[{upstream, dest}]`
@ -159,11 +159,11 @@ services.hyperhive.deploy.forgejo.mirrors = [
]; ];
``` ```
hive-c0re creates each entry as a real Forgejo pull-mirror — not a one-off Each entry is created as a real Forgejo pull-mirror — not a one-off
clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack` clone. Forgejo re-syncs the mirror on every pull (`git-upload-pack`
request), so a DNS blip during that sync will propagate back to the request), so a DNS blip during that sync will propagate back to the
runner as a hard `git clone` failure. hive-c0re runner as a hard `git clone` failure. The
autocreates the `<owner>` org in `dest`. Keep mirror dests out of the `<owner>` org in `dest` is autocreated. Keep mirror dests out of the
hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`) hive-c0re-managed namespaces (`config/`, `shared/`, `agents/`, `core/`)
to avoid provisioning collisions. to avoid provisioning collisions.
@ -173,7 +173,7 @@ to avoid provisioning collisions.
### What unsandboxed builds mean ### What unsandboxed builds mean
nspawn containers can't create user-namespaces, so the module sets `nix.settings.sandbox-fallback = true` in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase. nspawn containers can't create user-namespaces, so `nix.settings.sandbox-fallback = true` is set in the container. This means every `nix build` (and `nix flake check`) runs **without a build sandbox** — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.
A malicious `default.nix` or build script in a PR can therefore: A malicious `default.nix` or build script in a PR can therefore:
@ -235,15 +235,15 @@ following to your host config:
``` ```
**Measure a collection on your own hosts before relying on that stagger.** **Measure a collection on your own hosts before relying on that stagger.**
`randomizedDelaySec` spreads the _start_, not the run, and the sweep over `randomizedDelaySec` spreads the _start_, not the run, and the run is dominated
unused `.links` entries dominates the run — which scales with the size of the by the sweep over unused `.links` entries — which scales with the size of the
store, not with what this collection deletes. On a large store it can take most store, not with what this collection deletes. On a large store it can take most
of an hour, so two hosts both set to `daily` may overlap however much jitter you of an hour, so two hosts both set to `daily` may overlap however much jitter you
give them. `journalctl -u nix-gc.service` reports each run's wall clock. give them. `journalctl -u nix-gc.service` reports each run's wall clock.
**Remote builders:** if CI dispatches builds to a remote builder (for example via **Remote builders:** if CI dispatches builds to a remote builder (for example via
`nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_ `nix.buildMachines` / `ssh-ng://`), the build outputs land in _that host's_
store, so apply the same GC config wherever the builder runs — store, so the same GC config should be applied wherever the builder runs —
GC on the coordinator host won't reclaim space on the builder. GC on the coordinator host won't reclaim space on the builder.
⚠️ Two stores also means the temp-root protection above **doesn't span ⚠️ Two stores also means the temp-root protection above **doesn't span

View file

@ -14,8 +14,8 @@ section covers the concrete node inventory hive-c0re builds on top of that
engine, plus the internals (module layout, scheduler mechanics, engine, plus the internals (module layout, scheduler mechanics,
resource/lease semantics). resource/lease semantics).
hive-c0re submits every container/meta operation (rebuild, meta-update, first-spawn, power Every container/meta operation (rebuild, meta-update, first-spawn, power
changes) to the global job-DAG queue (`hive-c0re/src/job_queue/`) changes) is submitted to the global job-DAG queue (`hive-c0re/src/job_queue/`)
as a **DAG of primitive nodes**. One scheduler task drives all DAGs; as a **DAG of primitive nodes**. One scheduler task drives all DAGs;
concurrency comes from the resource classes below, not from multiple workers. concurrency comes from the resource classes below, not from multiple workers.
Special cases like graceful stop, deferred starts, and the meta-update Special cases like graceful stop, deferred starts, and the meta-update
@ -63,7 +63,7 @@ Cheap — no build slot:
| `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down | | `Start` | mechanical container start — runtime dir + drop-ins, `start_with_fallback`, MCP listener registration, the manager kick. Fanned out by a `Reconcile` that observed `wanted = Up` and the container down |
| `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up | | `Stop` | mechanical container stop — `nixos-container` kill, MCP listener unregister, the `Killed` manager notify. Fanned out by a `Reconcile` that observed `wanted = Offline` and up |
| `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped | | `StopForUpdate` | mechanical `nixos-container stop` for the profile swap; never touches `wanted`; noop if already stopped |
| `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the DAG's `EmitRebuilt` tail node emits the `Rebuilt` manager event, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace | | `RebuildBookkeeping` | the swap's Ok-only bookkeeping tail — rev marker, forge/matrix sync, manager kick, rescan, meta-inputs snapshot; `AfterOk(Swap)` so it runs only on a successful swap (the `Rebuilt` manager event is emitted by the DAG's `EmitRebuilt` tail node, not here). Split out of `Swap` for dashboard visibility + retry granularity, declares no resources of its own — a coordinated child of the `AgentWindow` brace |
| `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ | | `AgentWindow` | pure resource holder — the brace for one agent's rebuild. Declares the build slot + agent lease atomically and holds both for its whole subtree, so `Prebuild` and the `Signal`→`Drain` quiesce window run concurrently instead of one nested under the other. Performs no work; see _Braces_ |
| `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn | | `Signal` | set the graceful fence + kick, so the harness runs one stop-checkpoint turn |
| `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way | | `Drain` | await the harness clearing the fence, bounded by the 3-min graceful-stop timeout; resolves ok either way |
@ -101,7 +101,7 @@ container build:
phase below it (it declares `Resource::MetaWindow`). Two meta phase below it (it declares `Resource::MetaWindow`). Two meta
mutations can therefore never interleave, so no commit lands inside another mutations can therefore never interleave, so no commit lands inside another
node's staged window. It's a queue resource rather than a runtime mutex node's staged window. It's a queue resource rather than a runtime mutex
because a subtree root holds a resource across its whole subtree, which because a resource is held by a subtree root across its whole subtree, which
a `MutexGuard` (bounded by one executor fn) can't — that's what lets a a `MutexGuard` (bounded by one executor fn) can't — that's what lets a
multi-node deploy own one window. For the same reason the window must stay multi-node deploy own one window. For the same reason the window must stay
_off_ long store-only work: the rebuild's meta preamble is its own _off_ long store-only work: the rebuild's meta preamble is its own
@ -127,9 +127,9 @@ a hive-wide `hivectl restart` / `stop` / `start` is ONE DAG with a per-agent
subgraph each (independent roots, run concurrently on their own leases), not subgraph each (independent roots, run concurrently on their own leases), not
N separate DAGs. N separate DAGs.
**`job_queue/power.rs` builds these dynamically from each agent's live **These are built dynamically from each agent's live running state** (an
running state** (an async `lifecycle::is_running` read) — that's why they async `lifecycle::is_running` read), so they live in `job_queue/power.rs`,
live there rather than in the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry not the pure/sync `templates.rs`. Per-agent shape rule: `stop`/`start` carry
a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile` a head `SetWanted` (intent) — `restart` doesn't; the tail `Reconcile`
(convergence guarantee — cheap, noops when already converged) is ALWAYS (convergence guarantee — cheap, noops when already converged) is ALWAYS
present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are present; only the _mechanical_ nodes (`Signal`/`Drain`/`StopForUpdate`) are
@ -192,8 +192,8 @@ rides the queue through that power layer, so intent, lease serialization,
and crash-watch suppression can't drift per surface; the only direct starts and crash-watch suppression can't drift per surface; the only direct starts
left are the root-agent bootstrap and infra containers (no lease, no left are the root-agent bootstrap and infra containers (no lease, no
harness). Cancelling a still-queued power DAG reverts `wanted` to the harness). Cancelling a still-queued power DAG reverts `wanted` to the
observed state — a cancel means "don't do it," not "do it later." hive-c0re observed state — a cancel means "don't do it," not "do it later." Agents
seeds agents without a row from observed state on first touch (running ⇒ without a row are seeded from observed state on first touch (running ⇒
`Up`); destroy removes the row. `Up`); destroy removes the row.
The admin-socket responses carry the submitted DAG ids; `hivectl` polls The admin-socket responses carry the submitted DAG ids; `hivectl` polls
@ -275,7 +275,7 @@ reconcile_), so there is no durable-recovery machinery to go wrong.
### Cancel, history ### Cancel, history
The agent-per-node move removed submit-time dedup (a multi-agent DAG Submit-time dedup was removed with the agent-per-node move (a multi-agent DAG
has no single agent to key a dedup on), so every submit enqueues a fresh DAG; has no single agent to key a dedup on), so every submit enqueues a fresh DAG;
whether any dedup needs reintroducing is tracked as a follow-up. whether any dedup needs reintroducing is tracked as a follow-up.
@ -297,13 +297,13 @@ DeployWindow (root — build slot + lease + meta window, no work of its own)
└── DeployTail AfterAny(apply) compensate, mirror to forge └── DeployTail AfterAny(apply) compensate, mirror to forge
``` ```
The root holds its resources across the whole subtree, so the two-phase The root's resources are held across the whole subtree, so the two-phase
`prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock` `prepare_deploy` / `finalize_deploy` span keeps its staged `flake.lock`
protected even though the phases are separate nodes. Splitting them buys protected even though the phases are separate nodes. Splitting them buys
three things a single opaque node couldn't have: per-phase visibility on the three things a single opaque node couldn't have: per-phase visibility on the
dashboard, a `MergeVerify` failure that provably mutated nothing, and a dashboard, a `MergeVerify` failure that provably mutated nothing, and a
compensation step that survives a hive-c0re restart — `DeployApply` parks the pre-merge compensation step that survives a hive-c0re restart — the pre-merge
`applied/main` in `refs/hyperhive/rollback/<approval-id>`, not in a `applied/main` is parked in `refs/hyperhive/rollback/<approval-id>`, not in a
local variable, so `DeployTail` can still undo a half-finished deploy after a local variable, so `DeployTail` can still undo a half-finished deploy after a
crash. crash.
@ -378,9 +378,9 @@ vs the current flake path) and persisted `wanted` intent, then:
Booting with no config change performs **no meta commit** — only reconciles. Booting with no config change performs **no meta commit** — only reconciles.
The sweep reason records the rebuild / deferred / up-to-date counts so the The sweep reason records the rebuild / deferred / up-to-date counts so the
operator sees at a glance how much work the boot triggered. `auto_update::run` operator sees at a glance how much work the boot triggered. Agents without an
seeds agents without an `agent_power` row from observed state during `agent_power` row are seeded from observed state during classification (the
classification (the one-time migration; thereafter the DB is authoritative). one-time migration; thereafter the DB is authoritative).
## Meta flake ## Meta flake
@ -435,7 +435,7 @@ Sequence for a rebuild DAG (each step is its own queue node):
The container keeps serving the previous generation while eval + fetch + The container keeps serving the previous generation while eval + fetch +
build happen out-of-band. `nixos-container update` then finds the result build happen out-of-band. `nixos-container update` then finds the result
cached and skips straight to the profile-swap. Build failures surface cached and skips straight to the profile-swap. Build failures surface
here, before `Swap` touches the running container. Skipped when the here, before the running container is touched. Skipped when the
container is already down — a stopped agent has no uptime to preserve, container is already down — a stopped agent has no uptime to preserve,
so `Swap` builds inline instead of paying the double eval. so `Swap` builds inline instead of paying the double eval.
3. `StopForUpdate` — bring the container down (noop when already stopped). 3. `StopForUpdate` — bring the container down (noop when already stopped).
@ -449,9 +449,9 @@ The approval deploy uses this same chain rather than a rebuild path of its
own. Its `DeployApply` node doesn't build: it merges, opens the two-phase own. Its `DeployApply` node doesn't build: it merges, opens the two-phase
meta deploy, and returns the chain above as a subgraph the scheduler grafts meta deploy, and returns the chain above as a subgraph the scheduler grafts
into the live DAG under that node. A `FinalizeDeploy` node gated on the into the live DAG under that node. A `FinalizeDeploy` node gated on the
graft's completion then plants the deploy tag — so `Reconcile`'s success graft's completion then plants the deploy tag — so "did the agent come back
answers "did the agent come back up?" the same way it does for every up?" is answered by `Reconcile` succeeding, the same way it's for every
other rebuild, instead of a fused inline start. other rebuild, instead of by a fused inline start.
The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter The grafted nodes land _inside_ `DeployWindow`'s subtree, so they re-enter
the meta window and build slot it already holds rather than deadlocking the meta window and build slot it already holds rather than deadlocking
@ -466,9 +466,9 @@ half-started at that point.
Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes) Fallback: `stop` (graceful SIGTERM drain) → `kill` (SIGKILL any lingering processes)
→ `start` (clean cold-start, no generation transition, new activation runs cleanly). → `start` (clean cold-start, no generation transition, new activation runs cleanly).
`lifecycle::start_with_fallback` preserves and surfaces both errors if the Both errors are preserved and surfaced if the cold-start also fails. The fallback
cold-start also fails. Every `Reconcile` node's start action uses this lives in `lifecycle::start_with_fallback`, used by every `Reconcile` node's
fallback. start action.
### Spawn path (new container) ### Spawn path (new container)
@ -480,8 +480,8 @@ flags → `systemctl daemon-reload` → `start`.
### Prebuild attr path ### Prebuild attr path
`nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the `nix build` doesn't autoresolve `meta#<name>` against `nixosConfigurations` the
way `nixos-container` does internally. nix requires the explicit attr path way `nixos-container` does internally. The explicit attr path
`<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel`; `<flake-root>#nixosConfigurations.<name>.config.system.build.toplevel` is required;
using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`, using the bare `meta#<name>` ref would make nix look in `packages`, `legacyPackages`,
or the flake root directly — none of which exist in the rendered meta flake. or the flake root directly — none of which exist in the rendered meta flake.
@ -556,14 +556,14 @@ trade-off:
- **On** (recommended for x86_64 hosts that care about first-spawn - **On** (recommended for x86_64 hosts that care about first-spawn
latency): the first `nixos-container start` for any new agent latency): the first `nixos-container start` for any new agent
completes in seconds because nothing is left to fetch. Cost: completes in seconds because nothing is left to fetch. Cost:
`preBuildAgentTemplates` adds the full nixpkgs runtime closure + claude-code + the harness the full nixpkgs runtime closure + claude-code + the harness
binary to the host system closure (low single-digit GB binary are added to the host system closure (low single-digit GB
additional). additional).
- **Off** (default): the host closure stays lean; the first spawn - **Off** (default): the host closure stays lean; the first spawn
does all the eval + fetch work at runtime (can take a few does all the eval + fetch work at runtime (can take a few
minutes on a fresh store). minutes on a fresh store).
**Note**: the meta flake pins toplevels to `x86_64-linux`. Enabling on an **Note**: toplevels are pinned to `x86_64-linux`. Enabling on an
`aarch64` host forces a cross-compilation or remote-builder build, `aarch64` host forces a cross-compilation or remote-builder build,
which is almost never desired. Leave off on non-x86 hosts. which is almost never desired. Leave off on non-x86 hosts.

View file

@ -47,6 +47,6 @@ side by side). A step shows one of:
| `⊘` | cancelled | | `⊘` | cancelled |
| `·` | skipped (not needed for this run) | | `·` | skipped (not needed for this run) |
A step that isn't needed for a given run shows as `·` rather than A step that isn't needed for a given run shows as `·` rather than being
dropping out of the tree entirely, so the same kind of operation keeps a left out of the tree entirely, so the same kind of operation keeps a
recognizable shape run to run, whichever steps it actually needed. recognizable shape run to run, whichever steps it actually needed.

View file

@ -87,10 +87,10 @@ turn-stats rather than from a metric label.
## Options reference ## Options reference
The nix module (`nix/host-modules/otel.nix`) generates every Every `services.hyperhive.otel.*` option's full type/default/description/
`services.hyperhive.otel.*` option's full type/default/description/ example is generated straight from the nix module (`nix/host-modules/
example straight into [`/options/`](/options/) (host options — `nix build otel.nix`) into [`/options/`](/options/) (host options — `nix build
.#docs-host` for a local render). The build keeps that page honest in a .#docs-host` for a local render). That page is kept honest by the build in a
way a hand-copied version here can't be, so it's the reference, not this way a hand-copied version here can't be, so it's the reference, not this
doc. What follows is what a flat per-option listing can't express: the doc. What follows is what a flat per-option listing can't express: the
two-tier architecture, the security model, and how the options interact. two-tier architecture, the security model, and how the options interact.
@ -130,7 +130,7 @@ the local deployment stops testing.
**`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier **`endpoint` keeps meaning "where telemetry goes upstream."** Neither tier
redefines it — the agent-facing value is _derived_ redefines it — the agent-facing value is _derived_
(`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint` (`http://<bridgeIp>:<collector.port>`), so an existing deployment's `endpoint`
keeps working unchanged. The otel module contributes the bridge port to `exposeHostPorts` keeps working unchanged. The bridge port is contributed to `exposeHostPorts`
automatically; there is nothing to open by hand. automatically; there is nothing to open by hand.
### Authenticated ingest ### Authenticated ingest
@ -162,7 +162,7 @@ swarm tier and no metrics appearing for that hive.
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC `otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
exporter takes an _address_: `https://host/path` is a legal exporter takes an _address_: `https://host/path` is a legal
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in `OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
address"_. nix validates the collector's config at build time, so a mismatch is address"_. The collector's config is validated at build time, so a mismatch is
a build error naming the reason rather than telemetry silently going nowhere. a build error naming the reason rather than telemetry silently going nowhere.
## Network access ## Network access
@ -186,8 +186,8 @@ contributes the collector's port and derives the agent-facing endpoint itself.
## Built-in resource labels ## Built-in resource labels
The harness sets the OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`, The OTLP variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `_PROTOCOL`,
`OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) **container `OTEL_RESOURCE_ATTRIBUTES`, the temporality preference) are set **container
wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every wide** — in systemd's `DefaultEnvironment` and in `/etc/profile` — so every
process in an agent container exports to the hive's collector without any process in an agent container exports to the hive's collector without any
per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run per-tool wiring. That covers Claude Code, `hive-metric`, and anything you run
@ -203,8 +203,8 @@ automatically:
| `hive` | hive display name (`services.hyperhive.hiveName`) | | `hive` | hive display name (`services.hyperhive.hiveName`) |
| `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) | | `swarm` | swarm display name (`services.hyperhive.swarm.name`, if set) |
Append additional labels via `extraResourceAttributes` (see option Additional labels can be appended via `extraResourceAttributes` (see option
reference above); pass custom per-data-point labels with reference above); custom per-data-point labels can be passed with
`hive-metric --labels` (see below). `hive-metric --labels` (see below).
## Host-emitted container-resource metrics (hive-c0re) ## Host-emitted container-resource metrics (hive-c0re)
@ -219,7 +219,7 @@ Emitted via the OpenTelemetry Rust SDK, using the
[semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/) [semconv `container.*`](https://opentelemetry.io/docs/specs/semconv/system/container-metrics/)
metric names + the standard `container.name` attribute where a spec metric metric names + the standard `container.name` attribute where a spec metric
exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource exists, so off-the-shelf OTEL/Grafana container dashboards work. Resource
`service.name = hyperhive-c0re`; hive-c0re tags each data point `container.name` `service.name = hyperhive-c0re`; each data point is tagged `container.name`
(= the `h-<agent>` machine) and the hive `agent` label: (= the `h-<agent>` machine) and the hive `agent` label:
| Metric | Unit | Kind | Source | | Metric | Unit | Kind | Source |
@ -266,8 +266,8 @@ same container-wide `OTEL_RESOURCE_ATTRIBUTES` as everything else in this
section — nothing extra to configure. Cadence follows section — nothing extra to configure. Cadence follows
`HYPERHIVE_OTEL_METRIC_INTERVAL_MS` (default 60s, same variable + default as `HYPERHIVE_OTEL_METRIC_INTERVAL_MS` (default 60s, same variable + default as
`hive-c0re`'s container-resource export above) — that only controls how often `hive-c0re`'s container-resource export above) — that only controls how often
the harness flushes the batched points to the collector, not how often it the batched points are flushed to the collector, not how often they're
records them (every turn, always). recorded (every turn, always).
## Hive-scoped metrics (hive-c0re) ## Hive-scoped metrics (hive-c0re)
@ -283,7 +283,7 @@ distinguishable from "this hive is gone." Select them with
| `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows | | `hyperhive.hive.degraded` | `1` | gauge | `1` while the hive reports itself unhealthy — the same verdict `/health/ready` gives and the swarm status view shows |
| `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) | | `hyperhive.hive.warnings` | `1` | gauge | how many warnings are currently raised, split by a `level` attribute (`warn`, `crit`) |
hive-c0re reports both levels every cycle, `0` included, so a healthy hive is Both levels are reported every cycle, `0` included, so a healthy hive is
visible as zeros rather than as missing series. visible as zeros rather than as missing series.
`hyperhive.hive.degraded` is what a dashboard should alert on: it's `hyperhive.hive.degraded` is what a dashboard should alert on: it's
@ -300,8 +300,8 @@ Same cadence, transport and resource labels as the container metrics above.
"global/system" hook, not scoped to any one org or repo) and counts commit "global/system" hook, not scoped to any one org or repo) and counts commit
and push activity as deliveries arrive — occurrence-driven, not polled. and push activity as deliveries arrive — occurrence-driven, not polled.
Forgejo's own native `/metrics` endpoint has no equivalent: it exposes Forgejo's own native `/metrics` endpoint has no equivalent: it exposes
counts of durable rows (issues, comments, repos), and Forgejo doesn't counts of durable rows (issues, comments, repos), and neither a commit nor a
store either a commit or a push anywhere as a row to count. push is stored anywhere as a row to count.
| Metric | Unit | Kind | Attributes | | Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ------- | ------------------- | | ---------------------------- | ---- | ------- | ------------------- |
@ -314,9 +314,9 @@ when the delivery actually carries commits. Same enable signal (`OTEL_EXPORTER_O
(`HYPERHIVE_OTEL_METRIC_INTERVAL_MS`) and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES` (`HYPERHIVE_OTEL_METRIC_INTERVAL_MS`) and `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`
resource-attribute channel as `swarm-controller`'s other OTEL exporter (its resource-attribute channel as `swarm-controller`'s other OTEL exporter (its
`hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that `hive-jobq-metrics`-backed job-graph rollup, undocumented here — see that
crate's own doc comment) — `swarm-controller` sets `service.name = crate's own doc comment) — `service.name = swarm-controller` is set
swarm-controller` directly rather than reading it from the container directly rather than read from the container environment, since
environment, since it's a standalone daemon, not a per-agent harness process. `swarm-controller` is a standalone daemon, not a per-agent harness process.
## Agent-emitted custom metrics (`hive-metric`) ## Agent-emitted custom metrics (`hive-metric`)
@ -340,7 +340,7 @@ hive-metric <name> <value> [--type counter|gauge] [--temporality delta|cumulativ
reports the running total, which a stateless one-shot CLI can't track reports the running total, which a stateless one-shot CLI can't track
itself). itself).
- `--labels key=value` — extra per-data-point labels. May be repeated. - `--labels key=value` — extra per-data-point labels. May be repeated.
`hive-metric` inherits the resource labels (agent, hive, swarm, service.name) The resource labels (agent, hive, swarm, service.name) are inherited
automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them. automatically from `OTEL_RESOURCE_ATTRIBUTES` — don't re-specify them.
### Examples ### Examples
@ -366,12 +366,12 @@ with an informative error message. No silently dropped metrics.
`hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to `hive-metric` always uses **OTLP HTTP/JSON** (`application/json` POST to
`$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the `$OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`), regardless of the
`OTEL_EXPORTER_OTLP_PROTOCOL` setting. `hive-metric` forwards auth headers from `OTEL_EXPORTER_OTLP_PROTOCOL` setting. Auth headers from
`OTEL_EXPORTER_OTLP_HEADERS` verbatim. `OTEL_EXPORTER_OTLP_HEADERS` are forwarded verbatim.
## Metrics temporality ## Metrics temporality
The harness configures OTEL export with **cumulative** temporality by default OTEL export is configured with **cumulative** temporality by default
(`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`), (`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`),
overriding Claude Code's default of DELTA. This avoids silent metric drops in overriding Claude Code's default of DELTA. This avoids silent metric drops in
Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a Prometheus-family backends (including Grafana LGTM / Mimir) that don't ship a