814edd1d67
Each service section lost its container, credential and metrics detail to a Host setup section on its own implementation page, unchanged apart from links. The four agentAuth option rows that sat above the page title moved into the telemetry page's metrics option table.
118 lines
6.3 KiB
Markdown
118 lines
6.3 KiB
Markdown
# CI metrics
|
||
|
||
`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
|
||
store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
|
||
can show run time and queue wait per repo, workflow and PR, over any range back
|
||
to a repo's first run.
|
||
|
||
## Where it runs
|
||
|
||
A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
|
||
`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
|
||
has the poller's store identity, which the host that runs the store has. Run it
|
||
on one host per swarm: each poller keeps its own cursor, so two of them push
|
||
every run twice.
|
||
|
||
It reaches the forge, authelia, the store and the collector by name, so it needs
|
||
none of them on its host. On a host without the store, its leaf
|
||
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
|
||
file to copy, as for the CI runner.
|
||
|
||
With a self-signed gateway, those names serve leaves the swarm's own CA issued.
|
||
The poller reads them through `swarm-ci-metrics-ca-bundle.service`, which joins
|
||
the system CAs and the host's `trust-bundle.pem` into
|
||
`/run/swarm-ci-metrics-ca/trust-bundle.pem` and sets that file as the poller's
|
||
`SSL_CERT_FILE` ([CA](ca.md#the-trust-bundle)). Without it, every pass fails
|
||
with `invalid peer certificate: UnknownIssuer`.
|
||
|
||
## What it reads, and as whom
|
||
|
||
| what | where | identity |
|
||
| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
|
||
| its forge token | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
|
||
| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass | the same leaf |
|
||
| the repos | the forge's repo search: every repo with Actions on, neither archived nor a mirror | the forge account `ci-metrics`, with that token |
|
||
| each repo's runs | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account |
|
||
|
||
swarm-controller creates the `ci-metrics` account and mints its token with the
|
||
scope `read:repository`, the way it does for the builder. It also keeps a
|
||
`ci-metrics` team in every organisation, with read on the Actions unit of every
|
||
repo in it, and the account as its one member. The poller therefore covers:
|
||
|
||
- every public repo, which needs no grant;
|
||
- every private repo an organisation owns, through that team;
|
||
- **no** private repo a user owns. The forge grants a user's repo only to a
|
||
collaborator, and a collaborator reads every unit, code included.
|
||
|
||
## What it pushes
|
||
|
||
Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
|
||
as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
|
||
forwards them to the metrics store. Each sample carries the run's `stopped`
|
||
time as its timestamp, which the store keeps, so a run from last month lands
|
||
last month.
|
||
|
||
| metric | value, in seconds |
|
||
| ----------------------------- | ------------------- |
|
||
| `hyperhive.ci.run.duration` | `stopped − started` |
|
||
| `hyperhive.ci.run.queue_wait` | `started − created` |
|
||
|
||
Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
|
||
(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
|
||
`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
|
||
`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
|
||
them, every run is a series of its own, and the board reads each with
|
||
`last_over_time`.
|
||
|
||
A run becomes no sample when its status is `cancelled` or `skipped`, it has no
|
||
start (or the epoch as its start, which the forge writes for a run cancelled
|
||
before it started), or it stops no later than it starts. A run still going a day
|
||
after its `created` time counts as abandoned. The journal line for each repo counts
|
||
these by reason.
|
||
|
||
## The cursor and the backfill
|
||
|
||
The poller keeps one file, `cursor.json`, in its state directory
|
||
(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
|
||
poller has pushed or skipped every run, and the run numbers above it already
|
||
handled. A run still going holds that number, so a run that finishes after a
|
||
newer one still gets pushed.
|
||
|
||
With no file, the cursor of each repo is at run 0, so the first pass is the
|
||
backfill: it walks every run back to the first, a page of 50 a second. For a
|
||
repo with 5000 runs that takes a few minutes.
|
||
|
||
The poller writes the file only after the collector accepted a push, never
|
||
before. A crash between the two pushes those runs again on the next pass. The
|
||
store then holds the same point twice in the same series, which changes no
|
||
panel. Deleting the file re-runs the backfill the same way.
|
||
|
||
A failed repo leaves its cursor where it was, the pass goes on with the next
|
||
repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
|
||
timer runs it again 5 minutes later.
|
||
|
||
## The board
|
||
|
||
The **CI runs** tab of the `forge` board (`nix/host-modules/swarm-grafana/dashboards/forge.json`), with a repo and a workflow
|
||
variable:
|
||
|
||
- mean and 90th percentile run time of successful runs, over time;
|
||
- queue wait over time, mean and longest;
|
||
- finished runs per day, by outcome;
|
||
- a table per PR over the time range: mean and longest successful run time,
|
||
runs, failed runs.
|
||
|
||
## Host setup
|
||
|
||
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host
|
||
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container.
|
||
`allSwarmServices` defaults it on wherever the host has the poller's store
|
||
identity, as for the other swarm services. Set `deploy.ci-metrics.enable = false` to
|
||
opt out.
|
||
|
||
Each pass pushes the run time and queue wait of every newly finished forge
|
||
Actions run to the metrics store, through its own route on the swarm collector;
|
||
the first pass pushes every run since the first. It reads the forge with the
|
||
read-only token of its own `ci-metrics` forge account, under its own
|
||
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
|
||
copy.
|