vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

export bao client-cert not_before/not_after gauges and alert past 80% of lifetime

Browse source
This commit is contained in:
atlas 2026-10-25 17:41:47 +01:00 • committed by mara
commit 078696aaee
23 changed files with 604 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

53
docs/scheduler/observability.md
View file

@ -559,6 +559,59 @@ rules file, at most 10 per repo.
The `review-classifier` dashboard in Grafana charts these.
## Store client-certificate metrics
Every service that logs in to the secret store with a client certificate can
report when that certificate stops being valid. Two observable gauges, one
data point each, unix seconds:
| Metric | Unit | Kind | Attributes |
| ---------------------------- | ---- | ----- | ---------- |
| `bao.client_cert.not_before` | `s` | gauge | `cert` |
| `bao.client_cert.not_after` | `s` | gauge | `cert` |
`cert` names the identity (`client`, `controller`, `agent`, `review-classifier`,
`ci-metrics`); the resource labels (`service.name`, `hive`, `agent`) say who
holds it. The `swarm-cert-metrics` crate registers both gauges on the service's
existing exporter and reads `BAO_CLIENT_CERT` at every collection, so a
renewed leaf shows up on the next export. A file that's missing, or that isn't a
certificate yields no data point and a debug log line. The parser is
`swarm_secret_client::cert`.
| Service | `cert` | Exporter it rides |
| ------------------------- | ------------------- | --------------------------------------------------- |
| `hive-c0re` | `client` | the container-stats exporter |
| `hive-agent` | `agent` | the per-turn exporter, built on its first recording |
| `swarm-controller` | `controller` | the job-graph exporter |
| `swarm-review-classifier` | `review-classifier` | the cost exporter, built on its first recording |
| `swarm-ci-metrics` | `ci-metrics` | a push of its own at the start of each timer pass |
Not covered yet: the other seven Rust services that hold a store certificate
(`swarm-builder`, `swarm-claude-login`, `swarm-notify-router`,
`swarm-nats-auth`, `swarm-logs-auth`, `swarm-metrics-auth`,
`swarm-matrix-ctl`) have no exporter, and the ten identities that only shell
units use report nothing. Neither does any CA certificate. Each leaf's own
window is all the gauges show, and a CA that expires before its leaves is
invisible to them.
The swarm Grafana alerts on the lifetime fraction:
| Alert | Fires when |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `hh-bao-client-cert-lifetime` | `(time() - not_before) / (not_after - not_before)` stays above 0.8 for 10 minutes, per `service.name`, `cert`, `hive` and `agent` |
The two gauges of a series carry identical labels, so MetricsQL divides them
one-to-one without `on()` or `group_left`. The query wraps each in
`last_over_time(...[15m])` so `swarm-ci-metrics`, which reports every five
minutes, keeps a series. A series that stops reporting resolves the alert
rather than firing it (no data counts as OK).
An agent leaf lives 7 days and renews at half its life, so a fraction over 0.8
means the renewal stalled about two days ago. The host leaves (`client`,
`controller`, and the others minted by `swarm-bao-pki`) live 3650 days and never
renew; for them 0.8 comes about eight years after minting. A guard on the days
left would suit them better and doesn't exist yet.
## Agent-emitted custom metrics (`hive-metric`)
Agents can push arbitrary labeled metrics to the same OTEL collector via the