vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-victoriametrics: forward the container's journal to the swarm collector

Browse source 
The metrics store's container now runs the journal forwarder the other
service containers run, pushing to a receiver of its own on the swarm
collector (serviceProducers swarm-victoriametrics) as a new machine OIDC
client (swarm.victoriametrics.otel.clientId), registered by a glue file
wherever authelia runs and published to the store by the secret publisher.

The container keeps its loopback-only netns. The collector cannot dial a
unix socket, so swarm-victoriametrics-egress-relay, a socket-activated
systemd-socket-proxyd inside the container, takes its connections on
127.0.0.1:4380 and passes them to /run/swarm-victoriametrics-egress/egress.sock.
The host's nginx listens there and sends /swarm-victoriametrics/ to the swarm
collector and /api/oidc/token to authelia, both by name and verified, and
refuses every other path. nginx gets ReadWritePaths on that directory and
removes a stale socket before it starts.

The container cannot reach the store, so the client secret is fetched on the
host by swarm-victoriametrics-forwarder-oidc under swarm-metrics-auth's leaf,
the shape swarm-bao-forwarder-oidc has, and bound into the container
read-only. The forwarder module gains hostSecretFile for that: set, it renders
no fetch unit. swarm-metrics-auth's policy gains one read stanza on the
forwarder's client path.

The container drops --link-journal=host, and the host collector stops reading
it.

Also lists the forge publisher's and matrix-ctl's leaves in the forwarder
secrets row of docs/swarm/secrets.md, which the hive-forge/hive-matrix change
missed.
This commit is contained in:
atlas 2026-10-16 12:05:10 +02:00 • committed by mara
commit 7294707d7d
15 changed files with 454 additions and 62 deletions
Download patch file Download diff file Expand all files Collapse all files

54
docs/swarm/credentials.md
View file

@ -55,33 +55,33 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds | | `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes | | `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build | | `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

5
docs/swarm/secrets.md
View file

@ -53,7 +53,7 @@ neither is a renaming of the other.
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | | Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | | Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | | the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
@ -62,7 +62,8 @@ neither is a renaming of the other.
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the CI runner, Grafana, log store, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`) | | the CI runner, Grafana, log store, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) |
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |

27
docs/swarm/services.md
View file

@ -213,6 +213,16 @@ mints. On any other host you place a leaf whose CN is
you set both. It reads the stored bearer at most once a minute, so a you set both. It reads the stored bearer at most once a minute, so a
replaced bearer starts working up to a minute later. replaced bearer starts working up to a minute later.
The store's container has no network, so its journal forwarder leaves through
a relay. Inside the container, `swarm-victoriametrics-egress-relay` listens on
loopback and passes each connection to a socket the host's nginx listens on,
`/run/swarm-victoriametrics-egress/egress.sock`. nginx sends the forwarder's
route to the swarm collector and its token request to authelia, both by name
and verified, and refuses every other path. On the host,
`swarm-victoriametrics-forwarder-oidc.service` fetches the forwarder's OIDC
secret under the same leaf as `swarm-metrics-auth`, and the container sees it
read-only through a bind mount.
The collector scrapes Grafana's own metrics at The collector scrapes Grafana's own metrics at
`https://<swarm.grafana.domain>/metrics`, so a collector on a host without `https://<swarm.grafana.domain>/metrics`, so a collector on a host without
Grafana still gets them. The gateway admits only the collector's client to Grafana still gets them. The gateway admits only the collector's client to
@ -291,16 +301,17 @@ gateway either way.
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`, The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `hive-forge` and `hive-matrix` containers each run a `swarm-victorialogs`, `swarm-victoriametrics`, `hive-forge` and `hive-matrix`
collector of their own that forwards the container's journal here, on a route containers each run a collector of their own that forwards the container's
and receiver per service (`https://<swarm.otel.domain>/<container>/`), journal here, on a route and receiver per service
authenticated as an OIDC client of that service's. The queue, builder, login (`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
minter and classifier reuse the service's own client; the store, CI runner, of that service's. The queue, builder, login minter and classifier reuse the
Grafana, log store, forge and homeserver have a forwarder client of their own service's own client; the store, CI runner, Grafana, log store, metrics store,
forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`, (`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `review-classifier`, `ci-runner`, `nats`, `builder`, `claude-login`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `forgejo`, `tuwunel`) and no `hive` label. The `grafana`, `victorialogs`, `victoriametrics`, `forgejo`, `tuwunel`) and no
forwarder of all but the store's is `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`. `nix/container-modules/swarm-journal-forwarder.nix`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable` **Both store exporters are unconditional**, and `deploy.victoriametrics.enable`

31
nix/container-modules/swarm-journal-forwarder.nix
View file

@ -10,7 +10,9 @@
# The client secret is fetched inside the container, under the container's own # The client secret is fetched inside the container, under the container's own
# store identity, by `swarm-journal-forwarder-oidc`. That needs the container to # store identity, by `swarm-journal-forwarder-oidc`. That needs the container to
# reach the store, authelia and the swarm collector by name: the container needs # reach the store, authelia and the swarm collector by name: the container needs
# a network and a resolver. # a network and a resolver. A container with neither gets the secret from the
# host instead (`hostSecretFile`), and reaches the collector and authelia
# through whatever `endpoint` and `tokenUrl` name.
# #
# ⚠️ The host must NOT start this container with `--link-journal=host`. That # ⚠️ The host must NOT start this container with `--link-journal=host`. That
# bind-mounts a host directory over the guest's /var/log/journal which journald # bind-mounts a host directory over the guest's /var/log/journal which journald
@ -38,6 +40,8 @@ let
# then resumes from its saved journal cursor. # then resumes from its saved journal cursor.
secretDir = "/run/${fetchUnit}"; secretDir = "/run/${fetchUnit}";
secretFile = "${secretDir}/client-secret"; secretFile = "${secretDir}/client-secret";
fetchesHere = cfg.hostSecretFile == null;
credentialSource = if fetchesHere then secretFile else cfg.hostSecretFile;
# The collector runs under `DynamicUser` and opens `client_secret_file` # The collector runs under `DynamicUser` and opens `client_secret_file`
# itself. `LoadCredential` reads the file as root and re-exposes it under # itself. `LoadCredential` reads the file as root and re-exposes it under
@ -67,7 +71,8 @@ in
Base URL of this service's own receiver on the swarm collector: Base URL of this service's own receiver on the swarm collector:
`https://''${swarm.otel.domain}/<producer>`, a route `https://''${swarm.otel.domain}/<producer>`, a route
{option}`services.hyperhive.swarm.otel.serviceProducers` declares. {option}`services.hyperhive.swarm.otel.serviceProducers` declares.
The exporter appends `/v1/logs`. The exporter appends `/v1/logs`. In a container with no network, the
same path on a loopback relay that leads there.
''; '';
}; };
@ -96,6 +101,19 @@ in
''; '';
}; };
hostSecretFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = ''
The client secret as a file the host fetched and bound in, as a path
inside the container. Set, the container fetches nothing and the
`store` options are unused: the shape for a container that cannot
reach the store. The host writes the file only once fetched, and
restarts the collector when it changes. Null, the fetch unit reads it
with `store.*`.
'';
};
store = { store = {
package = lib.mkOption { package = lib.mkOption {
type = lib.types.package; type = lib.types.package;
@ -235,14 +253,14 @@ in
# the secret is still on its way, leaving the unit failed instead of # the secret is still on its way, leaving the unit failed instead of
# retrying. # retrying.
systemd.services.opentelemetry-collector.serviceConfig = { systemd.services.opentelemetry-collector.serviceConfig = {
LoadCredential = [ "${credentialId}:${secretFile}" ]; LoadCredential = [ "${credentialId}:${credentialSource}" ];
RestartSec = 15; RestartSec = 15;
}; };
# Reads the client secret with the container's own store identity. A store # Reads the client secret with the container's own store identity. A store
# or authelia that is not up yet is a failed run that retries # or authelia that is not up yet is a failed run that retries
# (../host-modules/lib/store-retry.nix), and a collector that restarts until the file lands. # (../host-modules/lib/store-retry.nix), and a collector that restarts until the file lands.
systemd.services.${fetchUnit} = { systemd.services.${fetchUnit} = lib.mkIf fetchesHere {
description = "fetch this container's journal forwarder OIDC client secret from the swarm secret store"; description = "fetch this container's journal forwarder OIDC client secret from the swarm secret store";
wantedBy = [ "multi-user.target" ]; wantedBy = [ "multi-user.target" ];
before = [ "opentelemetry-collector.service" ]; before = [ "opentelemetry-collector.service" ];
@ -309,7 +327,8 @@ in
''; '';
}; };
systemd.timers.${fetchUnit} = systemd.timers.${fetchUnit} = lib.mkIf fetchesHere (
storeRefreshTimer "re-read this container's journal forwarder OIDC client secret from the swarm secret store"; storeRefreshTimer "re-read this container's journal forwarder OIDC client secret from the swarm secret store"
);
}; };
} }

1
nix/host-modules/default.nix
View file

@ -58,6 +58,7 @@
./glue-swarm-bao-otel-oidc-client.nix ./glue-swarm-bao-otel-oidc-client.nix
./glue-swarm-otel-oidc-client.nix ./glue-swarm-otel-oidc-client.nix
./glue-victorialogs-otel-oidc-client.nix ./glue-victorialogs-otel-oidc-client.nix
./glue-victoriametrics-otel-oidc-client.nix
./swarm-authelia-service.nix ./swarm-authelia-service.nix
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao-service.nix ./swarm-bao-service.nix

37
nix/host-modules/glue-victoriametrics-otel-oidc-client.nix Normal file
View file

@ -0,0 +1,37 @@
# Glue: register the metrics store container's journal forwarder as an OIDC
# client wherever authelia runs.
#
# ONE PAIRING PER FILE: that forwarder ← authelia, and nothing else.
# Without it authelia mints no secret, the publisher has nothing to copy into
# the store, and the forwarder's export is a 401.
#
# Gated on authelia being HERE, not on this host running the metrics store: a
# client is a row in THIS host's provider config, the split
# ./glue-swarm-bao-otel-oidc-client.nix makes for the same reason.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
clientId = hyperhiveCfg.swarm.victoriametrics.otel.clientId;
in
{
config = lib.mkIf hyperhiveCfg.deploy.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = clientId;
description = "HyperHive metrics store journal forwarder";
kind = "machine";
redirectUris = [ ];
# Its own id as its own permitted audience, the one the swarm
# collector's receiver for this container checks.
audience = [ clientId ];
# The receiver verifies the token offline against `/jwks.json`;
# authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
}
];
};
}

13
nix/host-modules/swarm-bao.nix
View file

@ -1038,12 +1038,16 @@ let
]; ];
# The metrics store's responder, which checks the bearer Grafana presents # The metrics store's responder, which checks the bearer Grafana presents
# against the one stored for it, and reads nothing else. # against the one stored for it; and the host unit that fetches the store
# container's journal forwarder secret under the same leaf. Nothing else.
metricsAuthReaders = [ metricsAuthReaders = [
{ {
name = "swarm-metrics-auth"; name = "swarm-metrics-auth";
cn = baoDeploy.metricsAuthCommonName; cn = baoDeploy.metricsAuthCommonName;
policyText = readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"; policyText =
readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
# The metrics store container's journal forwarder, fetched on its host.
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.victoriametrics.otel.clientId}/oidc/client";
} }
]; ];
@ -1944,7 +1948,10 @@ in
Subject the store's `swarm-metrics-auth` cert-auth role accepts: the Subject the store's `swarm-metrics-auth` cert-auth role accepts: the
identity the metrics store's Grafana auth responder presents to read identity the metrics store's Grafana auth responder presents to read
the bearer Grafana's datasource sends, the bearer Grafana's datasource sends,
`swarm/services/<grafana client id>/metrics/bearer`, and nothing else. `swarm/services/<grafana client id>/metrics/bearer`, and the one
`swarm-victoriametrics-forwarder-oidc` presents to read
`swarm/services/<metrics store forwarder client id>/oidc/client`.
Nothing else.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings. ⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
''; '';

6
nix/host-modules/swarm-otel-service.nix
View file

@ -439,6 +439,12 @@ in
clientId = hyperhiveCfg.swarm.matrix.otel.clientId; clientId = hyperhiveCfg.swarm.matrix.otel.clientId;
port = lib.mkDefault 4400; port = lib.mkDefault 4400;
}; };
# The forwarder in ./swarm-victoriametrics.nix, which reaches this route
# through its host's nginx.
swarm-victoriametrics = {
clientId = hyperhiveCfg.swarm.victoriametrics.otel.clientId;
port = lib.mkDefault 4401;
};
}; };
# Not under any `enable`: see `publishedScrapeTargets` for why every host # Not under any `enable`: see `publishedScrapeTargets` for why every host

6
nix/host-modules/swarm-otel.nix
View file

@ -216,9 +216,9 @@ let
# The `serviceProducers` containers (`swarm-bao`, `swarm-nats`, # The `serviceProducers` containers (`swarm-bao`, `swarm-nats`,
# `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`, # `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`,
# `swarm-ci-runner`, `swarm-grafana`, `swarm-victorialogs`, `hive-forge`, # `swarm-ci-runner`, `swarm-grafana`, `swarm-victorialogs`, `hive-forge`,
# `hive-matrix`) are the exceptions on purpose: their journals stay inside # `hive-matrix`, `swarm-victoriametrics`) are the exceptions on purpose: their
# the container for its own collector instead, so this receiver never sees # journals stay inside the container for its own collector instead, so this
# their units. # receiver never sees their units.
hostJournalDir = "/var/log/journal"; hostJournalDir = "/var/log/journal";
# Each store's OTLP route, by domain. One binding because the same string is # Each store's OTLP route, by domain. One binding because the same string is

2
nix/host-modules/swarm-secret-publisher.nix
View file

@ -106,6 +106,8 @@ let
hyperhiveCfg.swarm.victorialogs.otel.clientId hyperhiveCfg.swarm.victorialogs.otel.clientId
hyperhiveCfg.swarm.forge.otel.clientId hyperhiveCfg.swarm.forge.otel.clientId
hyperhiveCfg.swarm.matrix.otel.clientId hyperhiveCfg.swarm.matrix.otel.clientId
# The metrics store container's journal forwarder, whose host reads it.
hyperhiveCfg.swarm.victoriametrics.otel.clientId
]; ];
# The swarm controller's OIDC client, which it reads back from the store # The swarm controller's OIDC client, which it reads back from the store

16
nix/host-modules/swarm-victoriametrics-service.nix
View file

@ -31,6 +31,22 @@ in
''; '';
}; };
otel.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-victoriametrics-collector";
description = ''
OAuth2 client id the journal forwarder inside the metrics store's
container authenticates as, and the audience it asks its token for.
Read-only because three hosts have to agree on it: authelia
registers the client (`glue-victoriametrics-otel-oidc-client.nix`),
the swarm collector admits it on the container's receiver
(`swarm.otel.serviceProducers`), and the secret publisher carries
its secret to the store.
'';
};
domain = lib.mkOption { domain = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "metrics.${domainBase}"; default = "metrics.${domainBase}";

231
nix/host-modules/swarm-victoriametrics.nix
View file

@ -22,8 +22,9 @@ let
gatewayCfg = hyperhiveCfg.gateway; gatewayCfg = hyperhiveCfg.gateway;
# A netns of its own with no interface but loopback: the store makes no # A netns of its own with no interface but loopback: the store makes no
# outbound calls, and the gateway reaches it over `socketPath` below, so # outbound calls, the gateway reaches it over `socketPath` below, and the
# nothing on the host or the hive bridge can dial its listener. # journal forwarder leaves only through `egressSocket`, so nothing on the host
# or the hive bridge can dial its listener.
privateNetwork = true; privateNetwork = true;
# Fixed, not an option: the directory's mode is what keeps the socket # Fixed, not an option: the directory's mode is what keeps the socket
@ -126,6 +127,72 @@ let
# The path Grafana's host creates the bearer at, under the KV mount. The # The path Grafana's host creates the bearer at, under the KV mount. The
# same leaf as `grafanaMetricsBearerLeaf` in ./swarm-bao.nix. # same leaf as `grafanaMetricsBearerLeaf` in ./swarm-bao.nix.
grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer"; grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer";
# ── the container's journal forwarder ───────────────────────────────
#
# With no interface but loopback, the forwarder reaches nothing by name. A
# relay in the container listens on `relayPort` and forwards each connection
# to `egressSocket`, a socket this host's nginx listens on and binds into
# the container. nginx sends `/<machine>/` to the swarm collector's route
# (`serviceProducers.swarm-victoriametrics` in ./swarm-otel-service.nix) and
# the token endpoint to authelia, both by name and verified. The collector
# library dials no unix socket itself, so the relay is needed.
#
# The client secret is fetched on the host, under the responder's store
# identity, and bound in read-only: the container cannot reach the store.
forwarderClientId = cfg.otel.clientId;
gatewayPortSuffix = lib.optionalString (
gatewayCfg.httpsPort != 443
) ":${toString gatewayCfg.httpsPort}";
otelDomain = hyperhiveCfg.swarm.otel.domain;
autheliaDomain = hyperhiveCfg.swarm.authelia.domain;
# Its own directory, not `socketDir`: that one's mode admits nginx to the
# store's listener, and this one admits the container to nginx's.
egressDir = "/run/swarm-victoriametrics-egress";
egressSocket = "${egressDir}/egress.sock";
relayUnit = "swarm-victoriametrics-egress-relay";
# Loopback in the container's own netns.
relayPort = 4380;
relayUrl = "http://127.0.0.1:${toString relayPort}";
forwarderTelemetryPort = 8900;
# The forwarder's way out, see `egressSocket`. Every other path is refused,
# so the container reaches the two routes its forwarder uses and nothing
# else on the network.
egressServer = ''
server {
listen unix:${egressSocket};
location / {
return 403;
}
location ^~ /${cfg.machine}/ {
proxy_pass https://${otelDomain}${gatewayPortSuffix};
${gatewayCfg.lib.verifiedProxyTo otelDomain}
}
location = /api/oidc/token {
proxy_pass https://${autheliaDomain};
${gatewayCfg.lib.verifiedProxyTo autheliaDomain}
}
}
'';
forwarderFetchUnit = "swarm-victoriametrics-forwarder-oidc";
forwarderStoreSecretPath = "secret/swarm/services/${forwarderClientId}/oidc/client";
# On the host's /run (tmpfs), and seen read-only in the container at the
# same path. The file is not created ahead of the fetch: `LoadCredential`
# refuses a missing source, so the forwarder stays down until the store
# answers and then resumes from its saved read cursor.
forwarderSecretDir = "/run/${forwarderFetchUnit}";
forwarderSecret = "${forwarderSecretDir}/${forwarderClientId}.secret";
storeRetry = import ./lib/store-retry.nix { };
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
in in
{ {
# What the store IS from any hive's point of view, the name it answers on, # What the store IS from any hive's point of view, the name it answers on,
@ -228,6 +295,16 @@ in
both options. both options.
''; '';
} }
{
assertion =
(hyperhiveCfg.swarm.otel.serviceProducers.${cfg.machine}.clientId or null) == forwarderClientId;
message = ''
services.hyperhive.swarm.otel.serviceProducers has no entry
"${cfg.machine}" with clientId "${forwarderClientId}", so the swarm
collector has no receiver for the metrics store's journal forwarder.
Declare it in swarm-otel-service.nix.
'';
}
]; ];
# The gateway name, inside `deployCfg.victoriametrics.enable` — that # The gateway name, inside `deployCfg.victoriametrics.enable` — that
@ -244,8 +321,9 @@ in
# Which check a read location's `auth_request` gets. `auth_request` takes a # Which check a read location's `auth_request` gets. `auth_request` takes a
# fixed URI, so it names `/__metrics_read_auth`, which rewrites to the # fixed URI, so it names `/__metrics_read_auth`, which rewrites to the
# target this map picks: a bearer goes to Grafana's check, anything else # target this map picks: a bearer goes to Grafana's check, anything else
# (an operator's session cookie) to authelia. # (an operator's session cookie) to authelia. After the forwarder's egress
services.nginx.appendHttpConfig = '' # listener, `egressServer`.
services.nginx.appendHttpConfig = egressServer + ''
map $http_authorization $swarm_metrics_read_auth_target { map $http_authorization $swarm_metrics_read_auth_target {
"~^Bearer " /__metrics_grafana_auth; "~^Bearer " /__metrics_grafana_auth;
default /__metrics_authelia; default /__metrics_authelia;
@ -359,6 +437,12 @@ in
# Everything else is kept out. # Everything else is kept out.
systemd.tmpfiles.rules = [ systemd.tmpfiles.rules = [
"d ${socketDir} 0770 root ${toString nginxGid} - -" "d ${socketDir} 0770 root ${toString nginxGid} - -"
# The relay in the container connects as root; nothing else on the host
# but nginx may.
"d ${egressDir} 0750 ${config.services.nginx.user} ${config.services.nginx.group} - -"
# The forwarder secret's bind source, so it exists when nspawn sets the
# mount up. Not the file: see `forwarderSecretDir`.
"d ${forwarderSecretDir} 0700 root root -"
]; ];
# Grafana's check on the read API. On the host rather than in the store's # Grafana's check on the read API. On the host rather than in the store's
@ -422,12 +506,100 @@ in
}; };
}; };
# nginx runs as its own user under `ProtectSystem=strict`, so it can only
# create the socket where it may write. It removes the socket on a clean
# exit only; a stale one after a crash fails every later `bind`, and with
# it the whole gateway, so it goes before each start. `-`: the directory
# is tmpfiles', and absent until that has run.
systemd.services.nginx = {
serviceConfig.ReadWritePaths = [ "-${egressDir}" ];
preStart = lib.mkBefore "rm -f ${egressSocket}";
};
# Reads the forwarder's client secret with the responder's store identity,
# the shape ./swarm-bao.nix's `swarm-bao-forwarder-oidc` has. The store
# and authelia may be a host away and this boot waits on neither, so a run
# that fails retries (./lib/store-retry.nix), and the collector restarts
# until the file lands.
systemd.services.${forwarderFetchUnit} = {
description = "fetch the metrics store forwarder's OIDC client secret from the swarm secret store";
after = [
"swarm-bao-pki.service"
"container@${cfg.machine}.service"
];
wants = [ "container@${cfg.machine}.service" ];
wantedBy = [ "multi-user.target" ];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
# No `RemainAfterExit`: the timer below starts this again, and
# starting an active unit is a no-op.
Type = "oneshot";
SyslogIdentifier = forwarderFetchUnit;
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.victoriametrics.authBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.victoriametrics.authBaoClientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
${refreshConsumer}
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, which
# needs a `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to the swarm secret store with the metrics store responder's certificate." >&2
cat "$err" >&2
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg forwarderStoreSecretPath} 2>"$err")"; then
echo "the store did not return ${forwarderStoreSecretPath}: the forwarder has no client secret yet." >&2
cat "$err" >&2
exit 1
fi
if [ -z "$secret" ]; then
echo "the store returned an empty ${forwarderStoreSecretPath}." >&2
exit 1
fi
# `printf` inside `atomic_write_secret` is bash's own, so the value
# never becomes an argument in /proc.
install -d -m 0700 ${lib.escapeShellArg forwarderSecretDir}
if secret_differs ${lib.escapeShellArg forwarderSecret} "$secret"; then
atomic_write_secret 0400 root:root ${lib.escapeShellArg forwarderSecret} "$secret"
fi
# `LoadCredential` copies the file at start only: a rotated secret
# reaches the forwarder by restarting it.
refresh_consumer ${lib.escapeShellArg cfg.machine} opentelemetry-collector.service ${lib.escapeShellArg forwarderSecret}
'';
};
systemd.timers.${forwarderFetchUnit} =
storeRefreshTimer "re-read the metrics store forwarder's OIDC client secret from the swarm secret store";
containers.${cfg.machine} = { containers.${cfg.machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes # No `--link-journal=host`: the journal stays in the guest, where the
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = [ "--link-journal=host" ];
inherit privateNetwork; inherit privateNetwork;
# A bind mount rather than a path both sides happen to know: a # A bind mount rather than a path both sides happen to know: a
@ -436,11 +608,54 @@ in
hostPath = socketDir; hostPath = socketDir;
isReadOnly = false; isReadOnly = false;
}; };
# Connecting to a socket is a write to it.
bindMounts.${egressDir} = {
hostPath = egressDir;
isReadOnly = false;
};
bindMounts.${forwarderSecretDir} = {
hostPath = forwarderSecretDir;
isReadOnly = true;
};
config = config =
{ config, pkgs, ... }: { config, pkgs, ... }:
{ {
imports = [ ../container-modules/swarm-container.nix ]; imports = [
../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
];
# Authenticated as the forwarder's own OIDC client, its secret
# fetched on the host. Both URLs are the relay's.
services.hyperhive.swarmJournalForwarder = {
enable = true;
serviceName = "victoriametrics";
endpoint = "${relayUrl}/${cfg.machine}";
clientId = forwarderClientId;
tokenUrl = "${relayUrl}/api/oidc/token";
telemetryPort = forwarderTelemetryPort;
hostSecretFile = forwarderSecret;
};
# The relay out of this netns, started by its first connection.
# systemd-socket-proxyd as root: the egress directory admits root
# and nginx only.
systemd.sockets.${relayUnit} = {
description = "loopback listener for the journal forwarder's way out of this container";
wantedBy = [ "sockets.target" ];
listenStreams = [ "127.0.0.1:${toString relayPort}" ];
};
systemd.services.${relayUnit} = {
description = "relay the journal forwarder's connections to the host's egress socket";
requires = [ "${relayUnit}.socket" ];
after = [ "${relayUnit}.socket" ];
serviceConfig = {
ExecStart = "${pkgs.systemd}/lib/systemd/systemd-socket-proxyd ${egressSocket}";
PrivateTmp = true;
NoNewPrivileges = true;
};
};
# No resolver unit: with no interface but loopback there is no # No resolver unit: with no interface but loopback there is no
# nameserver to reach, and the store resolves no names. # nameserver to reach, and the store resolves no names.

12
nix/module-eval/bao-grants.nix
View file

@ -781,9 +781,11 @@ let
lib.hasInfix "/logs-auth.pem ]" s && lib.hasInfix "swarm-logs-auth \"\" clientAuth" s; lib.hasInfix "/logs-auth.pem ]" s && lib.hasInfix "swarm-logs-auth \"\" clientAuth" s;
} }
{ {
# Read-only on the one leaf Grafana's host creates, and nothing else: # Read-only on the one leaf Grafana's host creates and on the store
# this principal checks a presented bearer and does nothing with it. # container forwarder's client secret, and nothing else: this principal
name = "the metrics store responder's grant is Grafana's metrics bearer, read-only, and nothing else"; # checks a presented bearer, and its host fetches one secret for the
# container.
name = "the metrics store responder's grant is Grafana's metrics bearer and its container forwarder's client secret, read-only, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-metrics-auth-policy.script; s = baoGrantHere.systemd.services.swarm-bao-metrics-auth-policy.script;
@ -793,6 +795,10 @@ let
path = "secret/data/swarm/services/swarm-grafana/metrics/bearer"; path = "secret/data/swarm/services/swarm-grafana/metrics/bearer";
caps = [ "read" ]; caps = [ "read" ];
} }
{
path = "secret/data/swarm/services/swarm-victoriametrics-collector/oidc/client";
caps = [ "read" ];
}
] ]
&& lib.hasInfix "auth/cert/certs/swarm-metrics-auth" s && lib.hasInfix "auth/cert/certs/swarm-metrics-auth" s
&& lib.hasInfix "allowed_common_names=swarm-metrics-auth" s && lib.hasInfix "allowed_common_names=swarm-metrics-auth" s

70
nix/module-eval/nats-authelia.nix
View file

@ -205,6 +205,10 @@ let
machine = "hive-matrix"; machine = "hive-matrix";
option = "matrix"; option = "matrix";
} }
{
machine = "swarm-victoriametrics";
option = "victoriametrics";
}
]; ];
ownClientIdOf = h: svc: h.services.hyperhive.swarm.${svc.option}.otel.clientId; ownClientIdOf = h: svc: h.services.hyperhive.swarm.${svc.option}.otel.clientId;
@ -954,6 +958,72 @@ let
ok = ok =
!shared.matrixNoBaoIdentity.containers.hive-matrix.config.services.opentelemetry-collector.enable; !shared.matrixNoBaoIdentity.containers.hive-matrix.config.services.opentelemetry-collector.enable;
} }
{
# The metrics store's container has no network: the forwarder dials the
# relay on loopback and reads a secret the host fetched and bound in.
name = "the metrics store container forwards its journal through the relay as its own client";
ok =
let
h = servicesHost;
container = h.containers.swarm-victoriametrics;
c = container.config;
s = c.services.opentelemetry-collector.settings;
otel = h.services.hyperhive.swarm.otel;
clientId = h.services.hyperhive.swarm.victoriametrics.otel.clientId;
secret = "/run/swarm-victoriametrics-forwarder-oidc/${clientId}.secret";
fetch = h.systemd.services.swarm-victoriametrics-forwarder-oidc;
in
c.services.opentelemetry-collector.enable
&& s.exporters.otlphttp.endpoint == "http://127.0.0.1:4380/swarm-victoriametrics"
&& s.extensions."oauth2client/swarm".token_url == "http://127.0.0.1:4380/api/oidc/token"
&& s.extensions."oauth2client/swarm".client_id == clientId
&& s.extensions."oauth2client/swarm".endpoint_params.audience == clientId
&& otel.serviceProducers.swarm-victoriametrics.clientId == clientId
&&
(lib.findFirst (a: a.key == "service.name") null s.processors.resource.attributes).value
== "victoriametrics"
&& lib.elem "oidc-client-secret:${secret}" c.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential
&& !(c.systemd.services ? swarm-journal-forwarder-oidc)
&& container.bindMounts."/run/swarm-victoriametrics-forwarder-oidc".isReadOnly
&& !(lib.elem "--link-journal=host" (container.extraFlags or [ ]))
&&
fetch.environment.BAO_CLIENT_CERT
== h.services.hyperhive.deploy.victoriametrics.authBaoClientCertFile
&&
fetch.environment.BAO_CLIENT_KEY == h.services.hyperhive.deploy.victoriametrics.authBaoClientKeyFile
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/${clientId}/oidc/client" fetch.script
&& lib.hasInfix "refresh_consumer swarm-victoriametrics opentelemetry-collector.service" fetch.script;
}
{
# The relay: a loopback listener in the container, a proxy to the bound
# socket, and the host nginx server on that socket, which opens the
# forwarder's two routes and nothing else.
name = "the metrics store's egress relay leads loopback to the collector's route and authelia's token endpoint only";
ok =
let
h = servicesHost;
container = h.containers.swarm-victoriametrics;
c = container.config;
sock = "/run/swarm-victoriametrics-egress/egress.sock";
relay = "swarm-victoriametrics-egress-relay";
nginx = h.services.nginx.appendHttpConfig;
otelDomain = h.services.hyperhive.swarm.otel.domain;
autheliaDomain = h.services.hyperhive.swarm.authelia.domain;
in
c.systemd.sockets.${relay}.listenStreams == [ "127.0.0.1:4380" ]
&&
lib.hasSuffix "/lib/systemd/systemd-socket-proxyd ${sock}"
c.systemd.services.${relay}.serviceConfig.ExecStart
&& !container.bindMounts."/run/swarm-victoriametrics-egress".isReadOnly
&& lib.hasInfix "listen unix:${sock};" nginx
&& lib.hasInfix "location / {\n return 403;" nginx
&& lib.hasInfix "location ^~ /swarm-victoriametrics/ {\n proxy_pass https://${otelDomain}" nginx
&& lib.hasInfix "proxy_ssl_name ${otelDomain};" nginx
&& lib.hasInfix "location = /api/oidc/token {\n proxy_pass https://${autheliaDomain};" nginx
&& lib.hasInfix "proxy_ssl_name ${autheliaDomain};" nginx
&& lib.elem "-/run/swarm-victoriametrics-egress" h.systemd.services.nginx.serviceConfig.ReadWritePaths
&& lib.hasInfix "rm -f ${sock}" h.systemd.services.nginx.preStart;
}
] ]
++ forwarderCases ++ forwarderCases
++ hostNetnsForwarderCases; ++ hostNetnsForwarderCases;

5
nix/module-eval/secret-publisher.nix
View file

@ -219,8 +219,8 @@ let
{ {
# The forwarders in the CI runner's, Grafana's, the log store's, the # The forwarders in the CI runner's, Grafana's, the log store's, the
# forge's and the homeserver's containers read their secrets from these # forge's and the homeserver's containers read their secrets from these
# paths. # paths, and the metrics store's host reads its container's.
name = "the publisher carries the CI runner, Grafana, log store, forge and homeserver forwarders' secrets"; name = "the publisher carries the CI runner, Grafana, log store, forge, homeserver and metrics store forwarders' secrets";
ok = ok =
lib.all lib.all
( (
@ -235,6 +235,7 @@ let
"swarm-victorialogs-collector" "swarm-victorialogs-collector"
"swarm-forge-collector" "swarm-forge-collector"
"swarm-matrix-collector" "swarm-matrix-collector"
"swarm-victoriametrics-collector"
]; ];
} }
{ {