mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-builder: export build counts, durations, queue wait and publish failures through the host's collector

Browse source 
The builder pushes its build metrics (builds by kind and outcome, build
duration, queue depth and wait, reply publish failures) to its forwarder's
OTLP receiver, which relays them to the swarm collector.

It also registers the standard store client-certificate gauges
(bao.client_cert.*, cert=builder) on the same exporter. They are the
builder's liveness signal: exported every minute whether or not a build
runs. The cert-metrics table in docs/scheduler/observability.md gains the
builder row.
This commit is contained in:
atlas 2026-11-04 23:13:53 +01:00 • committed by mara
commit e30f5849d5
2 changed files with 740 additions and 32 deletions
Download patch file Download diff file Expand all files Collapse all files

39
docs/scheduler/observability.md
View file

@ -597,6 +597,38 @@ rules file, at most 10 per repo.
The `review-classifier` dashboard in Grafana charts these.
## Swarm builder metrics
`swarm-builder` exports what it builds through the OTLP receiver of its
container's own forwarder, which pushes to the swarm collector as the builder's
client. `nix/host-modules/swarm-builder.nix` sets
`OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` for the daemon. The exporter is the same setup as the review
classifier's: HTTP/JSON, cumulative, one export a minute, and nothing exported
without `OTEL_EXPORTER_OTLP_ENDPOINT`.
| Metric | Unit | Kind | Attributes |
| ------------------------------------------ | ---- | --------- | ---------------------------------------------------------------- |
| `hyperhive.swarm_builder.builds` | — | counter | `kind`: `pr`, `meta`; `outcome`: `success`, `failure`, `refused` |
| `hyperhive.swarm_builder.build.duration` | `s` | histogram | `kind`, `outcome`: `success`, `failure` |
| `hyperhive.swarm_builder.queue.depth` | — | gauge | none |
| `hyperhive.swarm_builder.queue.wait` | `s` | histogram | `kind` |
| `hyperhive.swarm_builder.publish.failures` | — | counter | none |
`kind` `pr` is an agent config PR's head and `meta` an agent built from the
swarm meta flake. Lock and archetype requests aren't counted. `refused` is a
request that waited out the one-build slot, which the builder then answers busy
without building, so it has a `builds` point and no `build.duration` point. A build that
can't start (no forge token) counts as a `failure` with a duration; a payload
that doesn't decode counts as a `failure` in `builds` only.
`build.duration` runs from holding the slot to the reply, including the cache
push of a meta build. `queue.wait` runs from the request's arrival to the slot,
or to the refusal. `queue.depth` is the requests waiting for the slot now,
exported every minute, 0 when idle, so it doubles as the liveness signal; the
`bao.client_cert.*` gauges below ride the same exporter.
`publish.failures` counts replies the queue refused to take, which the
controller sees as a missing answer.
## Store client-certificate metrics
Every service that logs in to the secret store with a client certificate can
@ -609,7 +641,7 @@ data point each, unix seconds:
| `bao.client_cert.not_after` | `s` | gauge | `cert` |
`cert` names the identity (`client`, `controller`, `agent`, `review-classifier`,
`ci-metrics`); the resource labels (`service.name`, `hive`, `agent`) say who
`ci-metrics`, `builder`); the resource labels (`service.name`, `hive`, `agent`) say who
holds it. `swarm_secret_client::cert_metrics` (cargo feature `otel`) registers both gauges on the service's
existing exporter and reads `BAO_CLIENT_CERT` at every collection, so a
renewed leaf shows up on the next export. A file that's missing, or that isn't a
@ -623,9 +655,10 @@ certificate yields no data point and a debug log line. The parser is
| `swarm-controller` | `controller` | the job-graph exporter |
| `swarm-review-classifier` | `review-classifier` | the cost exporter, built on its first recording |
| `swarm-ci-metrics` | `ci-metrics` | a push of its own at the start of each timer pass |
| `swarm-builder` | `builder` | the build exporter, started with the daemon |
Not covered yet: the other seven Rust services that hold a store certificate
(`swarm-builder`, `swarm-claude-login`, `swarm-notify-router`,
Not covered yet: the other six Rust services that hold a store certificate
(`swarm-claude-login`, `swarm-notify-router`,
`swarm-nats-auth`, `swarm-logs-auth`, `swarm-metrics-auth`,
`swarm-matrix-ctl`) have no exporter, and the ten identities that only shell
units use report nothing. Neither does any CA certificate. Each leaf's own

3
docs/swarm/builder.md
View file

@ -563,6 +563,9 @@ request for the forge's host: any other host gets nothing. The builder holds no
status with its own forge token. Details per path:
[`credentials-reference.md`](credentials-reference.md).
The builder exports build counts, durations and queue wait; see
[Swarm builder metrics](../scheduler/observability.md#swarm-builder-metrics).
## Isolation
The container has its own network namespace on the bridge, as the CI runner