# CI metrics

`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
can show run time and queue wait per repo, workflow and PR, over any range back
to a repo's first run.

## Where it runs

A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
has the poller's store identity, which the host that runs the store has. Run it
on one host per swarm: each poller keeps its own cursor, so two of them push
every run twice.

It reaches the forge, authelia, the store and the collector by name, so it needs
none of them on its host. On a host without the store, its leaf
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
file to copy, as for the CI runner.

With a self-signed gateway, those names serve leaves the swarm's own CA issued.
The poller reads them through `swarm-ci-metrics-ca-bundle.service`, which joins
the system CAs and the host's `trust-bundle.pem` into
`/run/swarm-ci-metrics-ca/trust-bundle.pem` and sets that file as the poller's
`SSL_CERT_FILE` ([CA](ca.md#the-trust-bundle)). Without it, every pass fails
with `invalid peer certificate: UnknownIssuer`.

## What it reads, and as whom

| what                   | where                                                                                   | identity                                                 |
| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| its forge token        | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass              | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass        | the same leaf                                            |
| the repos              | the forge's repo search: every repo with Actions on, neither archived nor a mirror      | the forge account `ci-metrics`, with that token          |
| each repo's runs       | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account                                         |

swarm-controller creates the `ci-metrics` account and mints its token with the
scope `read:repository`, the way it does for the builder. It also keeps a
`ci-metrics` team in every organisation, with read on the Actions unit of every
repo in it, and the account as its one member. The poller therefore covers:

- every public repo, which needs no grant;
- every private repo an organisation owns, through that team;
- **no** private repo a user owns. The forge grants a user's repo only to a
  collaborator, and a collaborator reads every unit, code included.

## What it pushes

Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
forwards them to the metrics store. Each sample carries the run's `stopped`
time as its timestamp, which the store keeps, so a run from last month lands
last month.

| metric                        | value, in seconds   |
| ----------------------------- | ------------------- |
| `hyperhive.ci.run.duration`   | `stopped − started` |
| `hyperhive.ci.run.queue_wait` | `started − created` |

Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
them, every run is a series of its own, and the board reads each with
`last_over_time`.

A run becomes no sample when its status is `cancelled` or `skipped`, it has no
start (or the epoch as its start, which the forge writes for a run cancelled
before it started), or it stops no later than it starts. A run still going a day
after its `created` time counts as abandoned. The journal line for each repo counts
these by reason.

## The cursor and the backfill

The poller keeps one file, `cursor.json`, in its state directory
(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
poller has pushed or skipped every run, and the run numbers above it already
handled. A run still going holds that number, so a run that finishes after a
newer one still gets pushed.

With no file, the cursor of each repo is at run 0, so the first pass is the
backfill: it walks every run back to the first, a page of 50 a second. For a
repo with 5000 runs that takes a few minutes.

The poller writes the file only after the collector accepted a push, never
before. A crash between the two pushes those runs again on the next pass. The
store then holds the same point twice in the same series, which changes no
panel. Deleting the file re-runs the backfill the same way.

A failed repo leaves its cursor where it was, the pass goes on with the next
repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
timer runs it again 5 minutes later.

## The board

`nix/host-modules/swarm-grafana/dashboards/ci.json`, with a repo and a workflow
variable:

- mean and 90th percentile run time of successful runs, over time;
- queue wait over time, mean and longest;
- finished runs per day, by outcome;
- a table per PR over the time range: mean and longest successful run time,
  runs, failed runs.