push ci run metrics to victoriametrics, with a grafana board

A swarm-ci-metrics host unit on a 5-minute timer lists every forge
repo's Actions runs as the new read-only ci-metrics account and pushes
each finished run's duration and queue wait over OTLP, stamped at the
run's stop time. The first pass backfills from run 1. swarm-controller
mints the account's token into bao and keeps a ci-metrics team with
read on the Actions unit in every organisation.

Closes #5376

docs/swarm/ci-metrics.md

@@ -0,0 +1,96 @@
+# CI metrics
+
+`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
+store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
+can show run time and queue wait per repo, workflow and PR, over any range back
+to a repo's first run.
+
+## Where it runs
+
+A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
+`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
+has the poller's store identity, which the host that runs the store has. Run it
+on one host per swarm: each poller keeps its own cursor, so two of them push
+every run twice.
+
+It reaches the forge, authelia, the store and the collector by name, so it needs
+none of them on its host. On a host without the store, its leaf
+(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
+file to copy, as for the CI runner.
+
+## What it reads, and as whom
+
+| what                   | where                                                                                   | identity                                                 |
+| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
+| its forge token        | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass              | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
+| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass        | the same leaf                                            |
+| the repos              | the forge's repo search: every repo with Actions on, neither archived nor a mirror      | the forge account `ci-metrics`, with that token          |
+| each repo's runs       | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account                                         |
+
+swarm-controller creates the `ci-metrics` account and mints its token with the
+scope `read:repository`, the way it does for the builder. It also keeps a
+`ci-metrics` team in every organisation, with read on the Actions unit of every
+repo in it, and the account as its one member. The poller therefore covers:
+
+- every public repo, which needs no grant;
+- every private repo an organisation owns, through that team;
+- **no** private repo a user owns. The forge grants a user's repo only to a
+  collaborator, and a collaborator reads every unit, code included.
+
+## What it pushes
+
+Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
+as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
+forwards them to the metrics store. Each sample carries the run's `stopped`
+time as its timestamp, which the store keeps, so a run from last month lands
+last month.
+
+| metric                        | value, in seconds   |
+| ----------------------------- | ------------------- |
+| `hyperhive.ci.run.duration`   | `stopped − started` |
+| `hyperhive.ci.run.queue_wait` | `started − created` |
+
+Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
+(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
+`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
+`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
+them, every run is a series of its own, and the board reads each with
+`last_over_time`.
+
+A run becomes no sample when its status is `cancelled` or `skipped`, it has no
+start (or the epoch as its start, which the forge writes for a run cancelled
+before it started), or it stops no later than it starts. A run still going a day
+after its `created` time counts as abandoned. The journal line for each repo counts
+these by reason.
+
+## The cursor and the backfill
+
+The poller keeps one file, `cursor.json`, in its state directory
+(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
+poller has pushed or skipped every run, and the run numbers above it already
+handled. A run still going holds that number, so a run that finishes after a
+newer one still gets pushed.
+
+With no file, the cursor of each repo is at run 0, so the first pass is the
+backfill: it walks every run back to the first, a page of 50 a second. For a
+repo with 5000 runs that takes a few minutes.
+
+The poller writes the file only after the collector accepted a push, never
+before. A crash between the two pushes those runs again on the next pass. The
+store then holds the same point twice in the same series, which changes no
+panel. Deleting the file re-runs the backfill the same way.
+
+A failed repo leaves its cursor where it was, the pass goes on with the next
+repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
+timer runs it again 5 minutes later.
+
+## The board
+
+`nix/host-modules/swarm-grafana/dashboards/ci.json`, with a repo and a workflow
+variable:
+
+- mean and 90th percentile run time of successful runs, over time;
+- queue wait over time, mean and longest;
+- finished runs per day, by outcome;
+- a table per PR over the time range: mean and longest successful run time,
+  runs, failed runs.