Compare commits
37 changed files with 56 additions and 2854 deletions
|
|
@ -137,8 +137,6 @@ Swarm-level:
|
||||||
`swarm-logs-auth`.
|
`swarm-logs-auth`.
|
||||||
- **`swarm-logs-auth/`** — nginx `auth_request` responder for agent access
|
- **`swarm-logs-auth/`** — nginx `auth_request` responder for agent access
|
||||||
to the log store.
|
to the log store.
|
||||||
- **`swarm-metrics-auth/`** — nginx `auth_request` responder for Grafana's
|
|
||||||
bearer on the metrics store's query API.
|
|
||||||
- **`swarm-logs/`** — agent CLI: a LogsQL query in, log lines out.
|
- **`swarm-logs/`** — agent CLI: a LogsQL query in, log lines out.
|
||||||
- **`swarm-matrix-client/`** — appservice calls against the swarm
|
- **`swarm-matrix-client/`** — appservice calls against the swarm
|
||||||
homeserver, shared by `swarm-controller` and `swarm-matrix-ctl`.
|
homeserver, shared by `swarm-controller` and `swarm-matrix-ctl`.
|
||||||
|
|
@ -147,9 +145,6 @@ Swarm-level:
|
||||||
crates.
|
crates.
|
||||||
- **`swarm-builder/`** — builds agent config PR heads for commit status.
|
- **`swarm-builder/`** — builds agent config PR heads for commit status.
|
||||||
Design: [`docs/swarm/builder.md`](docs/swarm/builder.md).
|
Design: [`docs/swarm/builder.md`](docs/swarm/builder.md).
|
||||||
- **`swarm-ci-metrics/`** — polls the forge's Actions runs and pushes each
|
|
||||||
finished run's duration and queue wait to the metrics store. Design:
|
|
||||||
[`docs/swarm/ci-metrics.md`](docs/swarm/ci-metrics.md).
|
|
||||||
- **`swarm-claude-login/`** — runs `claude auth login` for one agent and
|
- **`swarm-claude-login/`** — runs `claude auth login` for one agent and
|
||||||
stores the credentials at `swarm/agents/<agent>/claude-credentials`.
|
stores the credentials at `swarm/agents/<agent>/claude-credentials`.
|
||||||
Design: [`docs/swarm/claude-login.md`](docs/swarm/claude-login.md).
|
Design: [`docs/swarm/claude-login.md`](docs/swarm/claude-login.md).
|
||||||
|
|
|
||||||
19
Cargo.lock
generated
19
Cargo.lock
generated
|
|
@ -4807,25 +4807,6 @@ dependencies = [
|
||||||
"tracing",
|
"tracing",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "swarm-ci-metrics"
|
|
||||||
version = "0.1.0"
|
|
||||||
dependencies = [
|
|
||||||
"anyhow",
|
|
||||||
"forgejo-api",
|
|
||||||
"hive-log",
|
|
||||||
"reqwest",
|
|
||||||
"serde",
|
|
||||||
"serde_json",
|
|
||||||
"swarm-queue-client",
|
|
||||||
"swarm-secret-client",
|
|
||||||
"tempfile",
|
|
||||||
"time",
|
|
||||||
"tokio",
|
|
||||||
"tracing",
|
|
||||||
"url",
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "swarm-claude-login"
|
name = "swarm-claude-login"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
|
|
|
||||||
|
|
@ -28,7 +28,6 @@ members = [
|
||||||
"swarm-agent-verify",
|
"swarm-agent-verify",
|
||||||
"swarm-authelia-bridge",
|
"swarm-authelia-bridge",
|
||||||
"swarm-builder",
|
"swarm-builder",
|
||||||
"swarm-ci-metrics",
|
|
||||||
"swarm-claude-login",
|
"swarm-claude-login",
|
||||||
"swarm-authelia-bridge-sock",
|
"swarm-authelia-bridge-sock",
|
||||||
"swarm-controller",
|
"swarm-controller",
|
||||||
|
|
|
||||||
|
|
@ -91,8 +91,6 @@ declarations.
|
||||||
(planned, not implemented yet).
|
(planned, not implemented yet).
|
||||||
- **Does an agent config PR build before it's merged?** →
|
- **Does an agent config PR build before it's merged?** →
|
||||||
[`swarm/builder.md`](swarm/builder.md).
|
[`swarm/builder.md`](swarm/builder.md).
|
||||||
- **How long do CI runs take, per repo and PR, and where does the board get
|
|
||||||
it?** → [`swarm/ci-metrics.md`](swarm/ci-metrics.md).
|
|
||||||
- **How do I log an agent in to claude from the swarm UI, and why doesn't a
|
- **How do I log an agent in to claude from the swarm UI, and why doesn't a
|
||||||
stale stored copy log it out again?** →
|
stale stored copy log it out again?** →
|
||||||
[`swarm/claude-login.md`](swarm/claude-login.md).
|
[`swarm/claude-login.md`](swarm/claude-login.md).
|
||||||
|
|
|
||||||
|
|
@ -1,96 +0,0 @@
|
||||||
# CI metrics
|
|
||||||
|
|
||||||
`swarm-ci-metrics` puts the forge's Actions run times into the swarm's metrics
|
|
||||||
store, one sample per finished run, so Grafana's **hyperhive · CI runs** board
|
|
||||||
can show run time and queue wait per repo, workflow and PR, over any range back
|
|
||||||
to a repo's first run.
|
|
||||||
|
|
||||||
## Where it runs
|
|
||||||
|
|
||||||
A oneshot `swarm-ci-metrics.service` on a 5-minute timer, on the host that sets
|
|
||||||
`deploy.ci-metrics.enable`. `allSwarmServices` defaults it on wherever the host
|
|
||||||
has the poller's store identity, which the host that runs the store has. Run it
|
|
||||||
on one host per swarm: each poller keeps its own cursor, so two of them push
|
|
||||||
every run twice.
|
|
||||||
|
|
||||||
It reaches the forge, authelia, the store and the collector by name, so it needs
|
|
||||||
none of them on its host. On a host without the store, its leaf
|
|
||||||
(`deploy.bao.ciMetricsClientCertFile` and `ciMetricsClientKeyFile`) is the one
|
|
||||||
file to copy, as for the CI runner.
|
|
||||||
|
|
||||||
## What it reads, and as whom
|
|
||||||
|
|
||||||
| what | where | identity |
|
|
||||||
| ---------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
|
|
||||||
| its forge token | `swarm/services/ci-metrics/forge/token` in the secret store, once per pass | its store leaf, subject `deploy.bao.ciMetricsCommonName` |
|
|
||||||
| its OIDC client secret | `swarm/services/swarm-ci-metrics/oidc/client` in the secret store, once per pass | the same leaf |
|
|
||||||
| the repos | the forge's repo search: every repo with Actions on, neither archived nor a mirror | the forge account `ci-metrics`, with that token |
|
|
||||||
| each repo's runs | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account |
|
|
||||||
|
|
||||||
swarm-controller creates the `ci-metrics` account and mints its token with the
|
|
||||||
scope `read:repository`, the way it does for the builder. It also keeps a
|
|
||||||
`ci-metrics` team in every organisation, with read on the Actions unit of every
|
|
||||||
repo in it, and the account as its one member. The poller therefore covers:
|
|
||||||
|
|
||||||
- every public repo, which needs no grant;
|
|
||||||
- every private repo an organisation owns, through that team;
|
|
||||||
- **no** private repo a user owns. The forge grants a user's repo only to a
|
|
||||||
collaborator, and a collaborator reads every unit, code included.
|
|
||||||
|
|
||||||
## What it pushes
|
|
||||||
|
|
||||||
Two gauges, pushed over OTLP to `https://<swarm.otel.domain>/swarm-ci-metrics`
|
|
||||||
as its own OIDC client (`swarm.ci-metrics.clientId`). The swarm collector
|
|
||||||
forwards them to the metrics store. Each sample carries the run's `stopped`
|
|
||||||
time as its timestamp, which the store keeps, so a run from last month lands
|
|
||||||
last month.
|
|
||||||
|
|
||||||
| metric | value, in seconds |
|
|
||||||
| ----------------------------- | ------------------- |
|
|
||||||
| `hyperhive.ci.run.duration` | `stopped − started` |
|
|
||||||
| `hyperhive.ci.run.queue_wait` | `started − created` |
|
|
||||||
|
|
||||||
Both carry the labels `repo` (`owner/name`), `run` (the run number), `workflow`
|
|
||||||
(the workflow file), `pr` (the PR number, or the ref for a run outside a PR),
|
|
||||||
`sha` (12 characters), `outcome` (`success` or `failure`) and `event`, plus
|
|
||||||
`service.name="swarm-ci-metrics"` and the collector's `swarm`. With `run` among
|
|
||||||
them, every run is a series of its own, and the board reads each with
|
|
||||||
`last_over_time`.
|
|
||||||
|
|
||||||
A run becomes no sample when its status is `cancelled` or `skipped`, it has no
|
|
||||||
start (or the epoch as its start, which the forge writes for a run cancelled
|
|
||||||
before it started), or it stops no later than it starts. A run still going a day
|
|
||||||
after its `created` time counts as abandoned. The journal line for each repo counts
|
|
||||||
these by reason.
|
|
||||||
|
|
||||||
## The cursor and the backfill
|
|
||||||
|
|
||||||
The poller keeps one file, `cursor.json`, in its state directory
|
|
||||||
(`/var/lib/swarm-ci-metrics`). Per repo, it holds a run number below which the
|
|
||||||
poller has pushed or skipped every run, and the run numbers above it already
|
|
||||||
handled. A run still going holds that number, so a run that finishes after a
|
|
||||||
newer one still gets pushed.
|
|
||||||
|
|
||||||
With no file, the cursor of each repo is at run 0, so the first pass is the
|
|
||||||
backfill: it walks every run back to the first, a page of 50 a second. For a
|
|
||||||
repo with 5000 runs that takes a few minutes.
|
|
||||||
|
|
||||||
The poller writes the file only after the collector accepted a push, never
|
|
||||||
before. A crash between the two pushes those runs again on the next pass. The
|
|
||||||
store then holds the same point twice in the same series, which changes no
|
|
||||||
panel. Deleting the file re-runs the backfill the same way.
|
|
||||||
|
|
||||||
A failed repo leaves its cursor where it was, the pass goes on with the next
|
|
||||||
repo, and the unit fails, so `systemctl status swarm-ci-metrics` shows it. The
|
|
||||||
timer runs it again 5 minutes later.
|
|
||||||
|
|
||||||
## The board
|
|
||||||
|
|
||||||
`nix/host-modules/swarm-grafana/dashboards/ci.json`, with a repo and a workflow
|
|
||||||
variable:
|
|
||||||
|
|
||||||
- mean and 90th percentile run time of successful runs, over time;
|
|
||||||
- queue wait over time, mean and longest;
|
|
||||||
- finished runs per day, by outcome;
|
|
||||||
- a table per PR over the time range: mean and longest successful run time,
|
|
||||||
runs, failed runs.
|
|
||||||
|
|
@ -55,34 +55,33 @@ of the cell says how.
|
||||||
|
|
||||||
<!-- vale write-good.Passive = NO -->
|
<!-- vale write-good.Passive = NO -->
|
||||||
|
|
||||||
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
|
||||||
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
|
||||||
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
|
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
|
||||||
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
|
||||||
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
|
||||||
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||||
| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
|
||||||
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
|
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
|
||||||
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
|
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
|
||||||
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
|
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
|
||||||
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
|
||||||
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
|
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
|
||||||
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
|
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
|
||||||
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds |
|
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds |
|
||||||
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
|
||||||
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
|
||||||
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
|
||||||
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
|
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
|
||||||
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
|
||||||
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
|
||||||
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
|
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
|
||||||
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||||
| `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
| `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
|
||||||
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
|
||||||
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
|
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
|
||||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account (read on the Actions unit of every organisation's repos through a `ci-metrics` team in each), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-ci-metrics` on its host, under its own certificate, once per pass; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the poller reads the store on every pass |
|
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
||||||
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
|
|
||||||
|
|
||||||
<!-- vale write-good.Passive = YES -->
|
<!-- vale write-good.Passive = YES -->
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -64,8 +64,6 @@ neither is a renaming of the other.
|
||||||
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||||
| the CI runner, Grafana, log store, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
|
| the CI runner, Grafana, log store, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
|
||||||
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
|
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
|
||||||
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
|
|
||||||
| the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
|
|
||||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||||
|
|
@ -410,10 +408,10 @@ next controller restart.
|
||||||
### Per-principal identities
|
### Per-principal identities
|
||||||
|
|
||||||
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
|
||||||
identity and sharing one merges the identities. Each of these ten leaves has
|
identity and sharing one merges the identities. Each of these nine leaves has
|
||||||
a role and a policy of its own, and reaches one path, except the review
|
a role and a policy of its own, and reaches one path, except the review
|
||||||
classifier's, the CI runner's, the builder's, the CI run poller's, the claude
|
classifier's, the CI runner's, the builder's, the claude login minter's, the
|
||||||
login minter's, the forge host's and Grafana's, which reach more than one:
|
forge host's and Grafana's, which reach more than one:
|
||||||
|
|
||||||
| unit | option pair under `deploy.bao.` | reads, or writes |
|
| unit | option pair under `deploy.bao.` | reads, or writes |
|
||||||
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
|
@ -425,40 +423,38 @@ login minter's, the forge host's and Grafana's, which reach more than one:
|
||||||
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
|
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
|
||||||
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
|
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
|
||||||
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |
|
||||||
| `swarm-ci-metrics` | `ciMetricsClientCertFile` / `ciMetricsClientKeyFile` | `swarm/services/ci-metrics/forge/token` and `swarm/services/<poller client id>/oidc/client` |
|
|
||||||
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
| `swarm-bao-forge-oidc` and the forge's controller-token mint and its journal forwarder in its container | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` and `swarm/services/<forge forwarder client id>/oidc/client` |
|
||||||
|
|
||||||
The first exists **per hive**, because the path it reads carries a hive name
|
The first exists **per hive**, because the path it reads carries a hive name
|
||||||
and every hive runs its own reader. Its subject is
|
and every hive runs its own reader. Its subject is
|
||||||
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
|
||||||
composed spelling as a hive name, so nobody can name a hive into another hive's
|
composed spelling as a hive name, so nobody can name a hive into another hive's
|
||||||
role. The other nine reach a path that names a swarm service rather than a hive,
|
role. The other eight reach a path that names a swarm service rather than a hive,
|
||||||
so one role each is enough and their subjects are the flat
|
so one role each is enough and their subjects are the flat
|
||||||
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
|
||||||
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName`,
|
||||||
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
`deploy.bao.reviewClassifierCommonName`, `deploy.bao.builderCommonName`,
|
||||||
`deploy.bao.ciMetricsCommonName`, `deploy.bao.claudeLoginCommonName` and
|
`deploy.bao.claudeLoginCommonName` and `deploy.bao.forgePublisherCommonName`.
|
||||||
`deploy.bao.forgePublisherCommonName`.
|
|
||||||
The last two are the writers, and the last is the one leaf two units share:
|
The last two are the writers, and the last is the one leaf two units share:
|
||||||
the forge's host holds one store identity for the token it publishes and the
|
the forge's host holds one store identity for the token it publishes and the
|
||||||
OIDC client secret it reads.
|
OIDC client secret it reads.
|
||||||
|
|
||||||
On a host that mints its own PKI, `glue-bao-tls.nix` signs all nine and defaults
|
On a host that mints its own PKI, `glue-bao-tls.nix` signs all eight and defaults
|
||||||
all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from
|
all sixteen options, and there is nothing to do. Elsewhere you copy each leaf from
|
||||||
the store's host and name it here: a service principal's from the top of
|
the store's host and name it here: a service principal's from the top of
|
||||||
`/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from
|
`/var/lib/swarm-bao-pki/`, another hive's queue-agent pair from
|
||||||
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
|
`/var/lib/swarm-bao-pki/hives/<hive>/` ([bao.md § TLS](bao.md#tls) lists that
|
||||||
directory). That's one file per principal rather than one file
|
directory). That's one file per principal rather than one file
|
||||||
shared by nine, which is the whole of what this buys. `swarm-ci-runner.nix`
|
shared by eight, which is the whole of what this buys. `swarm-ci-runner.nix`
|
||||||
refuses the build wherever the runner runs without its pair, as
|
refuses the build wherever the runner runs without its pair, as
|
||||||
`swarm-review-classifier.nix` does for the classifier, `swarm-ci-metrics.nix`
|
`swarm-review-classifier.nix` does for the classifier and `swarm-grafana.nix`
|
||||||
for the CI run poller and `swarm-grafana.nix` for Grafana. The forge's pair refuses nothing: without
|
for Grafana. The forge's pair refuses nothing: without
|
||||||
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
|
it `forgejo-swarm-controller-account` and `swarm-bao-forge-oidc` render, fail at start
|
||||||
naming both options, and the controller runs without forge access and the forge
|
naming both options, and the controller runs without forge access and the forge
|
||||||
without a login source until an operator places the leaf.
|
without a login source until an operator places the leaf.
|
||||||
|
|
||||||
On a host without the store, where the forwarder never runs, forgetting one of
|
On a host without the store, where the forwarder never runs, forgetting one of
|
||||||
the other nine isn't a quiet degrade. A hand-configured remote-store hive that
|
the other eight isn't a quiet degrade. A hand-configured remote-store hive that
|
||||||
names the hive's own `clientCertFile` and misses a principal's pair has the
|
names the hive's own `clientCertFile` and misses a principal's pair has the
|
||||||
build refused, by the module of the reader it missed, naming that pair — see
|
build refused, by the module of the reader it missed, naming that pair — see
|
||||||
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
|
||||||
|
|
|
||||||
|
|
@ -357,21 +357,6 @@ the private config repo with the read-only token of its own `builder` forge
|
||||||
account. On a host without the store, that leaf is the one file to copy, as
|
account. On a host without the store, that leaf is the one file to copy, as
|
||||||
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
|
for the CI runner. What it builds and why are in [`builder.md`](builder.md).
|
||||||
|
|
||||||
### CI metrics
|
|
||||||
|
|
||||||
The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host
|
|
||||||
that sets `deploy.ci-metrics.enable`. It runs on the host, not in a container.
|
|
||||||
`allSwarmServices` defaults it on wherever the host has the poller's store
|
|
||||||
identity, as for the builder above. Set `deploy.ci-metrics.enable = false` to
|
|
||||||
opt out.
|
|
||||||
|
|
||||||
Each pass pushes the run time and queue wait of every newly finished forge
|
|
||||||
Actions run to the metrics store, through its own route on the swarm collector;
|
|
||||||
the first pass pushes every run since the first. It reads the forge with the
|
|
||||||
read-only token of its own `ci-metrics` forge account, under its own
|
|
||||||
`ci-metrics` leaf. On a host without the store, that leaf is the one file to
|
|
||||||
copy. What it reads and pushes are in [`ci-metrics.md`](ci-metrics.md).
|
|
||||||
|
|
||||||
### Review classifier
|
### Review classifier
|
||||||
|
|
||||||
The review classifier's try-out service, in a `swarm-review-classifier`
|
The review classifier's try-out service, in a `swarm-review-classifier`
|
||||||
|
|
|
||||||
|
|
@ -172,9 +172,6 @@
|
||||||
services.hyperhive.deploy.builder.package =
|
services.hyperhive.deploy.builder.package =
|
||||||
lib.mkDefault
|
lib.mkDefault
|
||||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-builder;
|
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-builder;
|
||||||
services.hyperhive.deploy.ci-metrics.package =
|
|
||||||
lib.mkDefault
|
|
||||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-ci-metrics;
|
|
||||||
services.hyperhive.deploy.claude-login.package =
|
services.hyperhive.deploy.claude-login.package =
|
||||||
lib.mkDefault
|
lib.mkDefault
|
||||||
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-claude-login;
|
self.packages.${pkgs.stdenv.hostPlatform.system}.swarm-claude-login;
|
||||||
|
|
|
||||||
|
|
@ -49,11 +49,6 @@ in
|
||||||
inherit (pkgs) lib;
|
inherit (pkgs) lib;
|
||||||
shared = moduleEvalShared;
|
shared = moduleEvalShared;
|
||||||
};
|
};
|
||||||
module-eval-ci-metrics = import ./module-eval/ci-metrics.nix {
|
|
||||||
inherit pkgs self nixosSystem;
|
|
||||||
inherit (pkgs) lib;
|
|
||||||
shared = moduleEvalShared;
|
|
||||||
};
|
|
||||||
module-eval-nats-authelia = import ./module-eval/nats-authelia.nix {
|
module-eval-nats-authelia = import ./module-eval/nats-authelia.nix {
|
||||||
inherit pkgs self nixosSystem;
|
inherit pkgs self nixosSystem;
|
||||||
inherit (pkgs) lib;
|
inherit (pkgs) lib;
|
||||||
|
|
|
||||||
|
|
@ -47,7 +47,6 @@
|
||||||
./glue-nats-wireguard.nix
|
./glue-nats-wireguard.nix
|
||||||
./glue-queue-agent-credential.nix
|
./glue-queue-agent-credential.nix
|
||||||
./glue-builder-oidc-client.nix
|
./glue-builder-oidc-client.nix
|
||||||
./glue-ci-metrics-otel-oidc-client.nix
|
|
||||||
./glue-ci-runner-otel-oidc-client.nix
|
./glue-ci-runner-otel-oidc-client.nix
|
||||||
./glue-claude-login-oidc-client.nix
|
./glue-claude-login-oidc-client.nix
|
||||||
./glue-forge-otel-oidc-client.nix
|
./glue-forge-otel-oidc-client.nix
|
||||||
|
|
@ -66,7 +65,6 @@
|
||||||
./swarm-bao.nix
|
./swarm-bao.nix
|
||||||
./swarm-builder.nix
|
./swarm-builder.nix
|
||||||
./swarm-claude-login.nix
|
./swarm-claude-login.nix
|
||||||
./swarm-ci-metrics.nix
|
|
||||||
./swarm-ci-runner.nix
|
./swarm-ci-runner.nix
|
||||||
./swarm-review-classifier.nix
|
./swarm-review-classifier.nix
|
||||||
./swarm-secret-publisher.nix
|
./swarm-secret-publisher.nix
|
||||||
|
|
|
||||||
|
|
@ -117,8 +117,6 @@ in
|
||||||
reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem";
|
reviewClassifierClientKeyFile = lib.mkDefault "${pkiDir}/review-classifier-key.pem";
|
||||||
builderClientCertFile = lib.mkDefault "${pkiDir}/builder.pem";
|
builderClientCertFile = lib.mkDefault "${pkiDir}/builder.pem";
|
||||||
builderClientKeyFile = lib.mkDefault "${pkiDir}/builder-key.pem";
|
builderClientKeyFile = lib.mkDefault "${pkiDir}/builder-key.pem";
|
||||||
ciMetricsClientCertFile = lib.mkDefault "${pkiDir}/ci-metrics.pem";
|
|
||||||
ciMetricsClientKeyFile = lib.mkDefault "${pkiDir}/ci-metrics-key.pem";
|
|
||||||
claudeLoginClientCertFile = lib.mkDefault "${pkiDir}/claude-login.pem";
|
claudeLoginClientCertFile = lib.mkDefault "${pkiDir}/claude-login.pem";
|
||||||
claudeLoginClientKeyFile = lib.mkDefault "${pkiDir}/claude-login-key.pem";
|
claudeLoginClientKeyFile = lib.mkDefault "${pkiDir}/claude-login-key.pem";
|
||||||
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
|
forgePublisherClientCertFile = lib.mkDefault "${pkiDir}/forge-publisher.pem";
|
||||||
|
|
@ -312,13 +310,6 @@ in
|
||||||
[ -s ${pkiDir}/builder.pem ] || ${signLeaf} ${pkiDir} builder \
|
[ -s ${pkiDir}/builder.pem ] || ${signLeaf} ${pkiDir} builder \
|
||||||
${lib.escapeShellArg deployCfg.bao.builderCommonName} "" clientAuth
|
${lib.escapeShellArg deployCfg.bao.builderCommonName} "" clientAuth
|
||||||
|
|
||||||
# The CI run poller's, which reads its forge token and OIDC client
|
|
||||||
# secret. Minted here for the matrix-ctl leaf's reason: the poller is a
|
|
||||||
# swarm singleton, so on a poller host without the store this is the
|
|
||||||
# file an operator copies.
|
|
||||||
[ -s ${pkiDir}/ci-metrics.pem ] || ${signLeaf} ${pkiDir} ci-metrics \
|
|
||||||
${lib.escapeShellArg deployCfg.bao.ciMetricsCommonName} "" clientAuth
|
|
||||||
|
|
||||||
# The claude login minter's, which writes agents' claude credentials
|
# The claude login minter's, which writes agents' claude credentials
|
||||||
# and reads its queue client secret. Minted here for the matrix-ctl
|
# and reads its queue client secret. Minted here for the matrix-ctl
|
||||||
# leaf's reason: the minter is a swarm service, so on a minter host
|
# leaf's reason: the minter is a swarm service, so on a minter host
|
||||||
|
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
# Glue: register the CI run poller as an OIDC client wherever authelia runs.
|
|
||||||
#
|
|
||||||
# ONE PAIRING PER FILE: that poller ← authelia, and nothing else.
|
|
||||||
# Without it authelia mints no secret, the publisher has nothing to copy into
|
|
||||||
# the store, and the poller's push is a 401.
|
|
||||||
#
|
|
||||||
# Gated on authelia being HERE, not on this host running the poller: a
|
|
||||||
# client is a row in THIS host's provider config, the split
|
|
||||||
# ./glue-ci-runner-otel-oidc-client.nix makes for the same reason.
|
|
||||||
{
|
|
||||||
lib,
|
|
||||||
config,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
hyperhiveCfg = config.services.hyperhive;
|
|
||||||
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
config = lib.mkIf hyperhiveCfg.deploy.authelia.enable {
|
|
||||||
services.hyperhive.swarm.authelia.oidc.clients = [
|
|
||||||
{
|
|
||||||
id = clientId;
|
|
||||||
description = "HyperHive CI run poller";
|
|
||||||
kind = "machine";
|
|
||||||
redirectUris = [ ];
|
|
||||||
# Its own id as its own permitted audience, the one the swarm
|
|
||||||
# collector's receiver for the poller checks.
|
|
||||||
audience = [ clientId ];
|
|
||||||
# The receiver verifies the token offline against `/jwks.json`;
|
|
||||||
# authelia's default opaque handle fails there.
|
|
||||||
accessTokenSignedResponseAlg = "RS256";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
@ -449,10 +449,6 @@ let
|
||||||
capabilities = ["create", "read", "update"]
|
capabilities = ["create", "read", "update"]
|
||||||
}
|
}
|
||||||
|
|
||||||
path "${credentialMountPath}/data/${ciMetricsForgeTokenLeaf}" {
|
|
||||||
capabilities = ["create", "read", "update"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
|
||||||
capabilities = ["read"]
|
capabilities = ["read"]
|
||||||
}
|
}
|
||||||
|
|
@ -508,8 +504,8 @@ let
|
||||||
# The review classifier's forge token is denied too: the controller posts
|
# The review classifier's forge token is denied too: the controller posts
|
||||||
# the classifier's reviews with whatever token is there, so a write to it
|
# the classifier's reviews with whatever token is there, so a write to it
|
||||||
# picks the account those reviews come from.
|
# picks the account those reviews come from.
|
||||||
# So are the builder's and the CI run poller's: a write to either picks the
|
# So is the builder's: a write to it picks the account the builder fetches
|
||||||
# account that service reads the forge as.
|
# config repos as.
|
||||||
secretPublisherPolicyText = ''
|
secretPublisherPolicyText = ''
|
||||||
path "${credentialMountPath}/data/swarm/hives/*" {
|
path "${credentialMountPath}/data/swarm/hives/*" {
|
||||||
capabilities = ["create", "update"]
|
capabilities = ["create", "update"]
|
||||||
|
|
@ -543,10 +539,6 @@ let
|
||||||
capabilities = ["deny"]
|
capabilities = ["deny"]
|
||||||
}
|
}
|
||||||
|
|
||||||
path "${credentialMountPath}/data/${ciMetricsForgeTokenLeaf}" {
|
|
||||||
capabilities = ["deny"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "${credentialMountPath}/data/${controllerClientLeaf}" {
|
path "${credentialMountPath}/data/${controllerClientLeaf}" {
|
||||||
capabilities = ["create", "update"]
|
capabilities = ["create", "update"]
|
||||||
}
|
}
|
||||||
|
|
@ -628,12 +620,6 @@ let
|
||||||
# to fetch config repos.
|
# to fetch config repos.
|
||||||
builderForgeTokenLeaf = "swarm/services/builder/forge/token";
|
builderForgeTokenLeaf = "swarm/services/builder/forge/token";
|
||||||
|
|
||||||
# The CI run poller's forge token, the nix half of
|
|
||||||
# `swarm_secret_client::ci_metrics::forge_token_path`. The controller mints it
|
|
||||||
# and reads it back before deciding whether to re-mint; the poller reads it
|
|
||||||
# to list runs.
|
|
||||||
ciMetricsForgeTokenLeaf = "swarm/services/ci-metrics/forge/token";
|
|
||||||
|
|
||||||
# The review classifier's ACP provider key, the nix half of
|
# The review classifier's ACP provider key, the nix half of
|
||||||
# `swarm_secret_client::review_classifier::acp_provider_key_path`. Written
|
# `swarm_secret_client::review_classifier::acp_provider_key_path`. Written
|
||||||
# and read like the API key above.
|
# and read like the API key above.
|
||||||
|
|
@ -1109,18 +1095,6 @@ let
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
|
|
||||||
# The CI run poller: its forge token and its own OIDC client secret, both
|
|
||||||
# read once per pass by the host unit.
|
|
||||||
ciMetricsReaders = [
|
|
||||||
{
|
|
||||||
name = "swarm-ci-metrics";
|
|
||||||
cn = baoDeploy.ciMetricsCommonName;
|
|
||||||
policyText =
|
|
||||||
readStanza "${credentialMountPath}/data/${ciMetricsForgeTokenLeaf}"
|
|
||||||
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.ci-metrics.clientId}/oidc/client";
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# The claude login minter: every agent's `claude-credentials` leaf,
|
# The claude login minter: every agent's `claude-credentials` leaf,
|
||||||
# create and update so a relogin replaces the last, and nothing else under
|
# create and update so a relogin replaces the last, and nothing else under
|
||||||
# an agent; and its own queue client secret, read at start.
|
# an agent; and its own queue client secret, read at start.
|
||||||
|
|
@ -2029,19 +2003,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
ciMetricsCommonName = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "swarm-ci-metrics";
|
|
||||||
description = ''
|
|
||||||
Subject the store's `swarm-ci-metrics` cert-auth role accepts: the
|
|
||||||
identity the CI run poller presents from its host unit. Its grant is
|
|
||||||
two paths, read only: `swarm/services/ci-metrics/forge/token` and its
|
|
||||||
own OIDC client secret.
|
|
||||||
|
|
||||||
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
claudeLoginCommonName = lib.mkOption {
|
claudeLoginCommonName = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "swarm-claude-login";
|
default = "swarm-claude-login";
|
||||||
|
|
@ -2357,31 +2318,6 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
ciMetricsClientCertFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "/var/lib/swarm-bao-pki/ci-metrics.pem";
|
|
||||||
description = ''
|
|
||||||
Certificate the CI run poller presents to the store. Its subject must
|
|
||||||
be {option}`services.hyperhive.deploy.bao.ciMetricsCommonName`.
|
|
||||||
|
|
||||||
./glue-bao-tls.nix points it at the leaf it mints on the store's host.
|
|
||||||
On a poller host without the store, it is a file the operator copies
|
|
||||||
from the store's host. A path, never a value.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
ciMetricsClientKeyFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "/var/lib/swarm-bao-pki/ci-metrics-key.pem";
|
|
||||||
description = ''
|
|
||||||
Private key for
|
|
||||||
{option}`services.hyperhive.deploy.bao.ciMetricsClientCertFile`.
|
|
||||||
A path, never a value.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
claudeLoginClientCertFile = lib.mkOption {
|
claudeLoginClientCertFile = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.str;
|
type = lib.types.nullOr lib.types.str;
|
||||||
default = null;
|
default = null;
|
||||||
|
|
@ -3499,8 +3435,6 @@ in
|
||||||
|
|
||||||
systemd.services.swarm-bao-builder-policy = readerPolicyUnit "write the swarm builder's forge-token-reader bao policy and cert-auth role" builderReaders;
|
systemd.services.swarm-bao-builder-policy = readerPolicyUnit "write the swarm builder's forge-token-reader bao policy and cert-auth role" builderReaders;
|
||||||
|
|
||||||
systemd.services.swarm-bao-ci-metrics-policy = readerPolicyUnit "write the CI run poller's forge-token-reader bao policy and cert-auth role" ciMetricsReaders;
|
|
||||||
|
|
||||||
systemd.services.swarm-bao-claude-login-policy = readerPolicyUnit "write the claude login minter's credentials-writer bao policy and cert-auth role" claudeLoginWriters;
|
systemd.services.swarm-bao-claude-login-policy = readerPolicyUnit "write the claude login minter's credentials-writer bao policy and cert-auth role" claudeLoginWriters;
|
||||||
|
|
||||||
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
|
systemd.services.swarm-bao-forge-publisher-policy = readerPolicyUnit "write the forge's controller-token-publisher bao policy and cert-auth role" forgePublisherWriters;
|
||||||
|
|
|
||||||
|
|
@ -1,166 +0,0 @@
|
||||||
# The swarm's CI run poller: a oneshot `swarm-ci-metrics` on a 5-minute timer,
|
|
||||||
# on the host that sets `deploy.ci-metrics.enable`, which
|
|
||||||
# `deploy.allSwarmServices` defaults on where the host has the poller's store
|
|
||||||
# identity (./swarm-required-services.nix).
|
|
||||||
#
|
|
||||||
# Each pass lists the Actions runs of every repo the `ci-metrics` forge account
|
|
||||||
# sees and pushes each newly finished run's duration and queue wait to the
|
|
||||||
# swarm collector, stamped at the run's end. The first pass, with no cursor in
|
|
||||||
# the state directory, walks every run back to the first. docs/swarm/ci-metrics.md
|
|
||||||
# has the design.
|
|
||||||
#
|
|
||||||
# It reaches the forge, authelia, the store and the collector by name, so
|
|
||||||
# nothing here asks which host runs any of them. Its one credential on disk is
|
|
||||||
# its store leaf, handed in by `LoadCredential=`; the forge token and its OIDC
|
|
||||||
# client secret it reads from the store under that leaf on every pass.
|
|
||||||
{
|
|
||||||
lib,
|
|
||||||
config,
|
|
||||||
...
|
|
||||||
}:
|
|
||||||
let
|
|
||||||
hyperhiveCfg = config.services.hyperhive;
|
|
||||||
cfg = hyperhiveCfg.deploy.ci-metrics;
|
|
||||||
baoCfg = hyperhiveCfg.swarm.bao;
|
|
||||||
baoDeploy = hyperhiveCfg.deploy.bao;
|
|
||||||
gatewayCfg = hyperhiveCfg.gateway;
|
|
||||||
swarmOtelCfg = hyperhiveCfg.swarm.otel;
|
|
||||||
|
|
||||||
unit = "swarm-ci-metrics";
|
|
||||||
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
|
|
||||||
|
|
||||||
# The route `serviceProducers.swarm-ci-metrics` in ./swarm-otel-service.nix
|
|
||||||
# declares, by the collector's gateway name. The port suffix is elided on
|
|
||||||
# 443, as in ./swarm-builder.nix.
|
|
||||||
gatewayPortSuffix = lib.optionalString (
|
|
||||||
gatewayCfg.httpsPort != 443
|
|
||||||
) ":${toString gatewayCfg.httpsPort}";
|
|
||||||
otlpEndpoint = "https://${swarmOtelCfg.domain}${gatewayPortSuffix}/${unit}";
|
|
||||||
|
|
||||||
haveClientIdentity =
|
|
||||||
baoDeploy.ciMetricsClientCertFile != null && baoDeploy.ciMetricsClientKeyFile != null;
|
|
||||||
|
|
||||||
credential = id: "/run/credentials/${unit}.service/${id}";
|
|
||||||
# Credential id → host path of the poller's store identity.
|
|
||||||
storeFiles = {
|
|
||||||
"bao-client.pem" = baoDeploy.ciMetricsClientCertFile;
|
|
||||||
"bao-client-key.pem" = baoDeploy.ciMetricsClientKeyFile;
|
|
||||||
}
|
|
||||||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
|
||||||
"bao-ca.pem" = baoDeploy.serverCaFile;
|
|
||||||
};
|
|
||||||
in
|
|
||||||
{
|
|
||||||
options.services.hyperhive.swarm.ci-metrics.clientId = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
readOnly = true;
|
|
||||||
default = "swarm-ci-metrics";
|
|
||||||
description = ''
|
|
||||||
OAuth2 client id the CI run poller authenticates to the swarm
|
|
||||||
collector as, and the audience it asks its token for.
|
|
||||||
|
|
||||||
Read-only because three hosts have to agree on it: authelia registers
|
|
||||||
the client (`glue-ci-metrics-otel-oidc-client.nix`), the swarm collector
|
|
||||||
admits it on the poller's receiver (`swarm.otel.serviceProducers`), and
|
|
||||||
the secret publisher carries its secret to the store.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
options.services.hyperhive.deploy.ci-metrics = {
|
|
||||||
enable = lib.mkOption {
|
|
||||||
type = lib.types.bool;
|
|
||||||
default = false;
|
|
||||||
description = ''
|
|
||||||
Run the CI run poller on this host: every 5 minutes it pushes each
|
|
||||||
newly finished forge Actions run's duration and queue wait to the
|
|
||||||
swarm collector, for the Grafana CI dashboard. Its first run pushes
|
|
||||||
every run since the first.
|
|
||||||
|
|
||||||
Requires the poller's store identity,
|
|
||||||
{option}`services.hyperhive.deploy.bao.ciMetricsClientCertFile` and
|
|
||||||
`ciMetricsClientKeyFile`. Run it on one host per swarm: two pollers
|
|
||||||
keep two cursors and push every run twice.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
package = lib.mkOption {
|
|
||||||
type = lib.types.package;
|
|
||||||
defaultText = lib.literalExpression "hyperhive.packages.\${system}.swarm-ci-metrics";
|
|
||||||
description = "The CI run poller package.";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
config = lib.mkIf cfg.enable {
|
|
||||||
assertions = [
|
|
||||||
{
|
|
||||||
assertion = haveClientIdentity;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.deploy.ci-metrics.enable requires the poller's
|
|
||||||
swarm-secret-store client identity: set both
|
|
||||||
|
|
||||||
services.hyperhive.deploy.bao.ciMetricsClientCertFile
|
|
||||||
services.hyperhive.deploy.bao.ciMetricsClientKeyFile
|
|
||||||
|
|
||||||
swarm-ci-metrics.service reads its forge token and its OIDC client
|
|
||||||
secret with it.
|
|
||||||
|
|
||||||
Both default to the pair glue-bao-tls.nix mints when
|
|
||||||
services.hyperhive.deploy.bao.enable is true (the host that runs
|
|
||||||
the store). Otherwise copy ci-metrics.pem and ci-metrics-key.pem
|
|
||||||
from the store host's /var/lib/swarm-bao-pki/ and name them in
|
|
||||||
those two options.
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
systemd.services.${unit} = {
|
|
||||||
description = "push finished forge CI runs to the swarm metrics store";
|
|
||||||
after = [ "network-online.target" ];
|
|
||||||
wants = [ "network-online.target" ];
|
|
||||||
# No `RemainAfterExit`: the timer starts this again, and starting an
|
|
||||||
# active unit is a no-op.
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "oneshot";
|
|
||||||
ExecStart = "${cfg.package}/bin/swarm-ci-metrics";
|
|
||||||
# The first pass reads every run the forge has, a page a second.
|
|
||||||
TimeoutStartSec = "30min";
|
|
||||||
# Paths only: `argv` is world-readable via /proc/<pid>/cmdline.
|
|
||||||
LoadCredential = lib.mapAttrsToList (id: path: "${id}:${path}") storeFiles;
|
|
||||||
DynamicUser = true;
|
|
||||||
# The cursor. Losing it re-pushes every run.
|
|
||||||
StateDirectory = unit;
|
|
||||||
StateDirectoryMode = "0700";
|
|
||||||
SyslogIdentifier = unit;
|
|
||||||
ProtectSystem = "strict";
|
|
||||||
ProtectHome = true;
|
|
||||||
PrivateTmp = true;
|
|
||||||
PrivateDevices = true;
|
|
||||||
NoNewPrivileges = true;
|
|
||||||
RestrictSUIDSGID = true;
|
|
||||||
RemoveIPC = true;
|
|
||||||
};
|
|
||||||
environment = {
|
|
||||||
SWARM_CI_METRICS_FORGE_URL = "https://${hyperhiveCfg.swarm.forge.domain}";
|
|
||||||
SWARM_CI_METRICS_OTLP_ENDPOINT = otlpEndpoint;
|
|
||||||
SWARM_CI_METRICS_OIDC_TOKEN_ENDPOINT = "${hyperhiveCfg.swarm.authelia.url}/api/oidc/token";
|
|
||||||
SWARM_CI_METRICS_OIDC_CLIENT_ID = clientId;
|
|
||||||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
|
||||||
BAO_CLIENT_CERT = credential "bao-client.pem";
|
|
||||||
BAO_CLIENT_KEY = credential "bao-client-key.pem";
|
|
||||||
}
|
|
||||||
// lib.optionalAttrs (storeFiles ? "bao-ca.pem") {
|
|
||||||
BAO_CACERT = credential "bao-ca.pem";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.timers.${unit} = {
|
|
||||||
description = "run the CI run poller every 5 minutes";
|
|
||||||
wantedBy = [ "timers.target" ];
|
|
||||||
timerConfig = {
|
|
||||||
OnBootSec = "2min";
|
|
||||||
OnUnitInactiveSec = "5min";
|
|
||||||
RandomizedDelaySec = "30s";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
@ -197,7 +197,6 @@ let
|
||||||
"agents.json"
|
"agents.json"
|
||||||
"authelia.json"
|
"authelia.json"
|
||||||
"bao.json"
|
"bao.json"
|
||||||
"ci.json"
|
|
||||||
"claude-usage.json"
|
"claude-usage.json"
|
||||||
"collector.json"
|
"collector.json"
|
||||||
"controller.json"
|
"controller.json"
|
||||||
|
|
|
||||||
|
|
@ -1,410 +0,0 @@
|
||||||
{
|
|
||||||
"uid": "hyperhive-swarm-ci",
|
|
||||||
"title": "hyperhive · CI runs",
|
|
||||||
"description": "Forge Actions run time and queue wait, one sample per finished run, per repo and workflow.",
|
|
||||||
"tags": ["hyperhive", "ci"],
|
|
||||||
"timezone": "utc",
|
|
||||||
"schemaVersion": 39,
|
|
||||||
"version": 1,
|
|
||||||
"editable": true,
|
|
||||||
"refresh": "5m",
|
|
||||||
"time": {
|
|
||||||
"from": "now-30d",
|
|
||||||
"to": "now"
|
|
||||||
},
|
|
||||||
"templating": {
|
|
||||||
"list": [
|
|
||||||
{
|
|
||||||
"name": "repo",
|
|
||||||
"label": "Repo",
|
|
||||||
"type": "query",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"query": "label_values({__name__=\"hyperhive.ci.run.duration\"}, repo)",
|
|
||||||
"refresh": 2,
|
|
||||||
"includeAll": true,
|
|
||||||
"allValue": ".*",
|
|
||||||
"multi": true,
|
|
||||||
"current": {
|
|
||||||
"text": "All",
|
|
||||||
"value": "$__all"
|
|
||||||
},
|
|
||||||
"sort": 1
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "workflow",
|
|
||||||
"label": "Workflow",
|
|
||||||
"type": "query",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"query": "label_values({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\"}, workflow)",
|
|
||||||
"refresh": 2,
|
|
||||||
"includeAll": true,
|
|
||||||
"allValue": ".*",
|
|
||||||
"multi": true,
|
|
||||||
"current": {
|
|
||||||
"text": "All",
|
|
||||||
"value": "$__all"
|
|
||||||
},
|
|
||||||
"sort": 1
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"panels": [
|
|
||||||
{
|
|
||||||
"id": 1,
|
|
||||||
"type": "timeseries",
|
|
||||||
"title": "Run duration, successful runs",
|
|
||||||
"description": "Mean and 90th percentile of stopped minus started, over the runs that ended in each interval.",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"interval": "1h",
|
|
||||||
"gridPos": {
|
|
||||||
"h": 9,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"refId": "A",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "avg(last_over_time({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\", workflow=~\"$workflow\", outcome=\"success\"}[$__interval]))",
|
|
||||||
"legendFormat": "mean"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"refId": "B",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "quantile(0.9, last_over_time({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\", workflow=~\"$workflow\", outcome=\"success\"}[$__interval]))",
|
|
||||||
"legendFormat": "p90"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "s",
|
|
||||||
"color": {
|
|
||||||
"mode": "palette-classic"
|
|
||||||
},
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineWidth": 2,
|
|
||||||
"fillOpacity": 0,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"axisSoftMin": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "list",
|
|
||||||
"placement": "bottom",
|
|
||||||
"showLegend": true,
|
|
||||||
"calcs": []
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": 2,
|
|
||||||
"type": "timeseries",
|
|
||||||
"title": "Queue wait",
|
|
||||||
"description": "Started minus created: how long each run waited for a runner, mean and longest per interval.",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"interval": "1h",
|
|
||||||
"gridPos": {
|
|
||||||
"h": 9,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"refId": "A",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "avg(last_over_time({__name__=\"hyperhive.ci.run.queue_wait\", repo=~\"$repo\", workflow=~\"$workflow\"}[$__interval]))",
|
|
||||||
"legendFormat": "mean"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"refId": "B",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "max(last_over_time({__name__=\"hyperhive.ci.run.queue_wait\", repo=~\"$repo\", workflow=~\"$workflow\"}[$__interval]))",
|
|
||||||
"legendFormat": "longest"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "s",
|
|
||||||
"color": {
|
|
||||||
"mode": "palette-classic"
|
|
||||||
},
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineWidth": 2,
|
|
||||||
"fillOpacity": 0,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"axisSoftMin": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "list",
|
|
||||||
"placement": "bottom",
|
|
||||||
"showLegend": true,
|
|
||||||
"calcs": []
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": 3,
|
|
||||||
"type": "timeseries",
|
|
||||||
"title": "Runs per day by outcome",
|
|
||||||
"description": "Finished runs per UTC day. Cancelled runs are not counted.",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"interval": "1d",
|
|
||||||
"gridPos": {
|
|
||||||
"h": 9,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 9
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"refId": "A",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "count by (outcome) (last_over_time({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\", workflow=~\"$workflow\"}[$__interval]))",
|
|
||||||
"legendFormat": "{{outcome}}"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "short",
|
|
||||||
"decimals": 0,
|
|
||||||
"color": {
|
|
||||||
"mode": "palette-classic"
|
|
||||||
},
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "bars",
|
|
||||||
"lineWidth": 1,
|
|
||||||
"fillOpacity": 80,
|
|
||||||
"showPoints": "never",
|
|
||||||
"stacking": {
|
|
||||||
"mode": "normal",
|
|
||||||
"group": "A"
|
|
||||||
},
|
|
||||||
"axisSoftMin": 0
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": [
|
|
||||||
{
|
|
||||||
"matcher": {
|
|
||||||
"id": "byName",
|
|
||||||
"options": "success"
|
|
||||||
},
|
|
||||||
"properties": [
|
|
||||||
{
|
|
||||||
"id": "color",
|
|
||||||
"value": {
|
|
||||||
"mode": "fixed",
|
|
||||||
"fixedColor": "green"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"matcher": {
|
|
||||||
"id": "byName",
|
|
||||||
"options": "failure"
|
|
||||||
},
|
|
||||||
"properties": [
|
|
||||||
{
|
|
||||||
"id": "color",
|
|
||||||
"value": {
|
|
||||||
"mode": "fixed",
|
|
||||||
"fixedColor": "red"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "list",
|
|
||||||
"placement": "bottom",
|
|
||||||
"showLegend": true,
|
|
||||||
"calcs": ["sum"]
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": 4,
|
|
||||||
"type": "table",
|
|
||||||
"title": "Duration per PR",
|
|
||||||
"description": "Per PR (or ref, for runs not on a PR) over the time range: run time of its successful runs.",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 9,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 9
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"refId": "A",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "avg by (pr) (last_over_time({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\", workflow=~\"$workflow\", outcome=\"success\"}[$__range]))",
|
|
||||||
"format": "table",
|
|
||||||
"instant": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"refId": "B",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "max by (pr) (last_over_time({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\", workflow=~\"$workflow\", outcome=\"success\"}[$__range]))",
|
|
||||||
"format": "table",
|
|
||||||
"instant": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"refId": "C",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "count by (pr) (last_over_time({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\", workflow=~\"$workflow\"}[$__range]))",
|
|
||||||
"format": "table",
|
|
||||||
"instant": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"refId": "D",
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "@datasourceUid@"
|
|
||||||
},
|
|
||||||
"expr": "count by (pr) (last_over_time({__name__=\"hyperhive.ci.run.duration\", repo=~\"$repo\", workflow=~\"$workflow\", outcome=\"failure\"}[$__range]))",
|
|
||||||
"format": "table",
|
|
||||||
"instant": true
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"transformations": [
|
|
||||||
{
|
|
||||||
"id": "merge",
|
|
||||||
"options": {}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "organize",
|
|
||||||
"options": {
|
|
||||||
"excludeByName": {
|
|
||||||
"Time": true
|
|
||||||
},
|
|
||||||
"renameByName": {
|
|
||||||
"pr": "PR",
|
|
||||||
"Value #A": "Mean (success)",
|
|
||||||
"Value #B": "Longest (success)",
|
|
||||||
"Value #C": "Runs",
|
|
||||||
"Value #D": "Failed runs"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "sortBy",
|
|
||||||
"options": {
|
|
||||||
"sort": [
|
|
||||||
{
|
|
||||||
"field": "PR",
|
|
||||||
"desc": true
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"custom": {
|
|
||||||
"align": "auto"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": [
|
|
||||||
{
|
|
||||||
"matcher": {
|
|
||||||
"id": "byRegexp",
|
|
||||||
"options": "^(Mean|Longest).*"
|
|
||||||
},
|
|
||||||
"properties": [
|
|
||||||
{
|
|
||||||
"id": "unit",
|
|
||||||
"value": "s"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"matcher": {
|
|
||||||
"id": "byRegexp",
|
|
||||||
"options": "^(Runs|Failed runs)$"
|
|
||||||
},
|
|
||||||
"properties": [
|
|
||||||
{
|
|
||||||
"id": "decimals",
|
|
||||||
"value": 0
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"options": {
|
|
||||||
"showHeader": true,
|
|
||||||
"cellHeight": "sm"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
@ -445,12 +445,6 @@ in
|
||||||
clientId = hyperhiveCfg.swarm.victoriametrics.otel.clientId;
|
clientId = hyperhiveCfg.swarm.victoriametrics.otel.clientId;
|
||||||
port = lib.mkDefault 4401;
|
port = lib.mkDefault 4401;
|
||||||
};
|
};
|
||||||
# The CI run poller in ./swarm-ci-metrics.nix, a host unit that pushes
|
|
||||||
# its run samples here directly.
|
|
||||||
swarm-ci-metrics = {
|
|
||||||
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
|
|
||||||
port = lib.mkDefault 4403;
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
# Not under any `enable`: see `publishedScrapeTargets` for why every host
|
# Not under any `enable`: see `publishedScrapeTargets` for why every host
|
||||||
|
|
|
||||||
|
|
@ -127,13 +127,6 @@ in
|
||||||
&& deployCfg.bao.builderClientKeyFile != null
|
&& deployCfg.bao.builderClientKeyFile != null
|
||||||
);
|
);
|
||||||
|
|
||||||
# The CI run poller, by the same rule: its module asserts its store identity.
|
|
||||||
config.services.hyperhive.deploy.ci-metrics.enable = lib.mkDefault (
|
|
||||||
deployCfg.allSwarmServices
|
|
||||||
&& deployCfg.bao.ciMetricsClientCertFile != null
|
|
||||||
&& deployCfg.bao.ciMetricsClientKeyFile != null
|
|
||||||
);
|
|
||||||
|
|
||||||
# The claude login minter, defaulted on the same terms as the review
|
# The claude login minter, defaulted on the same terms as the review
|
||||||
# classifier.
|
# classifier.
|
||||||
config.services.hyperhive.deploy.claude-login.enable = lib.mkDefault (
|
config.services.hyperhive.deploy.claude-login.enable = lib.mkDefault (
|
||||||
|
|
|
||||||
|
|
@ -108,8 +108,6 @@ let
|
||||||
hyperhiveCfg.swarm.matrix.otel.clientId
|
hyperhiveCfg.swarm.matrix.otel.clientId
|
||||||
# The metrics store container's journal forwarder, whose host reads it.
|
# The metrics store container's journal forwarder, whose host reads it.
|
||||||
hyperhiveCfg.swarm.victoriametrics.otel.clientId
|
hyperhiveCfg.swarm.victoriametrics.otel.clientId
|
||||||
# The CI run poller, which reads it itself on its host.
|
|
||||||
hyperhiveCfg.swarm.ci-metrics.clientId
|
|
||||||
];
|
];
|
||||||
|
|
||||||
# The swarm controller's OIDC client, which it reads back from the store
|
# The swarm controller's OIDC client, which it reads back from the store
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue