push ci run metrics to victoriametrics, with a grafana board
A swarm-ci-metrics host unit on a 5-minute timer lists every forge repo's Actions runs as the new read-only ci-metrics account and pushes each finished run's duration and queue wait over OTLP, stamped at the run's stop time. The first pass backfills from run 1. swarm-controller mints the account's token into bao and keeps a ci-metrics team with read on the Actions unit in every organisation. Closes #5376
This commit is contained in:
atlas
mara
parent
8b7a4e7955
commit
b072fbf4d6
36 changed files with 2849 additions and 56 deletions
|
|
@ -64,6 +64,8 @@ neither is a renaming of the other.
|
|||
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the CI runner, Grafana, log store, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
|
||||
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
|
||||
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
|
||||
| the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
|
|
@ -408,10 +410,10 @@ next controller restart.
|
|||
### Per-principal identities
|
||||
|
||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||
identity and sharing one merges the identities. Each of these nine leaves has
|
||||
identity and sharing one merges the identities. Each of these ten leaves has
|
||||
a role and a policy of its own, and reaches one path, except the review
|
||||
classifier's, the CI runner's, the builder's, the claude login minter's, the
|
||||
forge host's and Grafana's, which reach more than one:
|
||||
classifier's, the CI runner's, the builder's, the CI run poller's, the claude
|
||||
login minter's, the forge host's and Grafana's, which reach more than one:
|
||||
|
||||
| unit | option pair under `deploy.bao.` | reads, or writes |
|
||||
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
|
@ -423,38 +425,40 @@ forge host's and Grafana's, which reach more than one:
|
|||
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
|
||||
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
|
||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||
|
||||
The first exists **per hive**, because the path it reads carries a hive name
|
||||
and every hive runs its own reader. Its subject is
|
||||
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
||||
composed spelling as a hive name, so nobody can name a hive into another hive's
|
||||
role. The other eight reach a path that names a swarm service rather than a hive,
|
||||
role. The other nine reach a path that names a swarm service rather than a hive,
|
||||
so one role each is enough and their subjects are the flat
|
||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
||||
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
|
||||
`deploy.bao.ciMetricsCommonName`, `deploy.bao.claudeLoginCommonName` and
|
||||
`deploy.bao.forgePublisherCommonName`.
|
||||
The last two are the writers, and the last is the one leaf two units share:
|
||||
the forge's host holds one store identity for the token it publishes and the
|
||||
OIDC client secret it reads.
|
||||
|
||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all eight and defaults
|
||||
all sixteen options, and there is nothing to do. Elsewhere you copy each leaf from
|
||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
|
||||
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
|
||||
the store's host and name it here: a service principal's from the top of
|
||||
`/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from
|
||||
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
|
||||
directory). That's one file per principal rather than one file
|
||||
shared by eight, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||
refuses the build wherever the runner runs without its pair, as
|
||||
`swarm-review-classifier.nix` does for the classifier and `swarm-grafana.nix`
|
||||
for Grafana. The forge's pair refuses nothing: without
|
||||
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
|
||||
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
|
||||
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
|
||||
naming both options, and the controller runs without forge access and the forge
|
||||
without a login source until an operator places the leaf.
|
||||
|
||||
On a host without the store, where the forwarder never runs, forgetting one of
|
||||
the other eight isn't a quiet degrade. A hand-configured remote-store hive that
|
||||
the other nine isn't a quiet degrade. A hand-configured remote-store hive that
|
||||
names the hive's own `clientCertFile` and misses a principal's pair has the
|
||||
build refused, by the module of the reader it missed, naming that pair — see
|
||||
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
||||
|
|
|
|||
Loading…
Reference in a new issue