b072fbf4d6
A swarm-ci-metrics host unit on a 5-minute timer lists every forge repo's Actions runs as the new read-only ci-metrics account and pushes each finished run's duration and queue wait over OTLP, stamped at the run's stop time. The first pass backfills from run 1. swarm-controller mints the account's token into bao and keeps a ci-metrics team with read on the Actions unit in every organisation. Closes #5376
96 lines
5.1 KiB
Markdown
96 lines
5.1 KiB
Markdown
# CI metrics
|
||
|
||
`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
|
||
store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
|
||
can show run time and queue wait per repo, workflow and PR, over any range back
|
||
to a repo's first run.
|
||
|
||
## Where it runs
|
||
|
||
A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
|
||
`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
|
||
has the poller's store identity, which the host that runs the store has. Run it
|
||
on one host per swarm: each poller keeps its own cursor, so two of them push
|
||
every run twice.
|
||
|
||
It reaches the forge, authelia, the store and the collector by name, so it needs
|
||
none of them on its host. On a host without the store, its leaf
|
||
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
|
||
file to copy, as for the CI runner.
|
||
|
||
## What it reads, and as whom
|
||
|
||
| what | where | identity |
|
||
| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
|
||
| its forge token | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
|
||
| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass | the same leaf |
|
||
| the repos | the forge's repo search: every repo with Actions on, neither archived nor a mirror | the forge account `ci-metrics`, with that token |
|
||
| each repo's runs | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account |
|
||
|
||
swarm-controller creates the `ci-metrics` account and mints its token with the
|
||
scope `read:repository`, the way it does for the builder. It also keeps a
|
||
`ci-metrics` team in every organisation, with read on the Actions unit of every
|
||
repo in it, and the account as its one member. The poller therefore covers:
|
||
|
||
- every public repo, which needs no grant;
|
||
- every private repo an organisation owns, through that team;
|
||
- **no** private repo a user owns. The forge grants a user's repo only to a
|
||
collaborator, and a collaborator reads every unit, code included.
|
||
|
||
## What it pushes
|
||
|
||
Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
|
||
as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
|
||
forwards them to the metrics store. Each sample carries the run's `stopped`
|
||
time as its timestamp, which the store keeps, so a run from last month lands
|
||
last month.
|
||
|
||
| metric | value, in seconds |
|
||
| ----------------------------- | ------------------- |
|
||
| `hyperhive.ci.run.duration` | `stopped − started` |
|
||
| `hyperhive.ci.run.queue_wait` | `started − created` |
|
||
|
||
Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
|
||
(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
|
||
`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
|
||
`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
|
||
them, every run is a series of its own, and the board reads each with
|
||
`last_over_time`.
|
||
|
||
A run becomes no sample when its status is `cancelled` or `skipped`, it has no
|
||
start (or the epoch as its start, which the forge writes for a run cancelled
|
||
before it started), or it stops no later than it starts. A run still going a day
|
||
after its `created` time counts as abandoned. The journal line for each repo counts
|
||
these by reason.
|
||
|
||
## The cursor and the backfill
|
||
|
||
The poller keeps one file, `cursor.json`, in its state directory
|
||
(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
|
||
poller has pushed or skipped every run, and the run numbers above it already
|
||
handled. A run still going holds that number, so a run that finishes after a
|
||
newer one still gets pushed.
|
||
|
||
With no file, the cursor of each repo is at run 0, so the first pass is the
|
||
backfill: it walks every run back to the first, a page of 50 a second. For a
|
||
repo with 5000 runs that takes a few minutes.
|
||
|
||
The poller writes the file only after the collector accepted a push, never
|
||
before. A crash between the two pushes those runs again on the next pass. The
|
||
store then holds the same point twice in the same series, which changes no
|
||
panel. Deleting the file re-runs the backfill the same way.
|
||
|
||
A failed repo leaves its cursor where it was, the pass goes on with the next
|
||
repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
|
||
timer runs it again 5 minutes later.
|
||
|
||
## The board
|
||
|
||
`nix/host-modules/swarm-grafana/dashboards/ci.json`, with a repo and a workflow
|
||
variable:
|
||
|
||
- mean and 90th percentile run time of successful runs, over time;
|
||
- queue wait over time, mean and longest;
|
||
- finished runs per day, by outcome;
|
||
- a table per PR over the time range: mean and longest successful run time,
|
||
runs, failed runs.
|