vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-ci-runner, swarm-grafana, swarm-victorialogs: forward each container's journal to the swarm collector

Browse source 
Each container now runs the journal forwarder the queue's and the builder's
containers run, pushing to a receiver of its own on the swarm collector. Each
authenticates as a new machine OIDC client of its own
(swarm.<service>.otel.clientId), registered by a glue file wherever authelia
runs, minted by authelia and published to the store by the secret publisher.
Grafana's browser login client is unchanged.

The fetch unit inside each container reads the client secret with the store
identity the host already holds for that service (the runner's, Grafana's,
and the log store responder's), now also handed into the container as an
nspawn system credential. Each of those three policies gains one read stanza
on its forwarder's client path. The host units that use those identities are
unchanged.

The three containers drop --link-journal=host, and the host collector stops
reading them.
This commit is contained in:
atlas 2026-10-16 10:14:57 +02:00 • committed by mara
commit c6ca9ab845
20 changed files with 810 additions and 107 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/swarm/credentials.md
View file

@ -56,7 +56,7 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
@ -74,7 +74,7 @@ of the cell says how.
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's and the log store's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s and `swarm-logs-auth`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes | | `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |

13
docs/swarm/secrets.md
View file

@ -42,7 +42,7 @@ neither is a renaming of the other.
## Swarm-level — one of each per swarm ## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
@ -62,6 +62,7 @@ neither is a renaming of the other.
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the CI runner, Grafana and log store containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm.<service>.otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`) |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
@ -408,16 +409,16 @@ next controller restart.
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. Each of these nine leaves has identity and sharing one merges the identities. Each of these nine leaves has
a role and a policy of its own, and reaches one path, except the review a role and a policy of its own, and reaches one path, except the review
classifier's, the builder's, the claude login minter's, the forge host's and classifier's, the CI runner's, the builder's, the claude login minter's, the
Grafana's, which reach two each: forge host's and Grafana's, which reach more than one:
| unit | option pair under `deploy.bao.` | reads, or writes | | unit | option pair under `deploy.bao.` | reads, or writes |
| --------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` | | `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` | | `swarm-bao-grafana-oidc`, `swarm-bao-grafana-admin`, and Grafana's journal forwarder in its container | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | reads `swarm/services/<grafana client id>/oidc/client` and `swarm/services/<grafana forwarder client id>/oidc/client`; creates and reads `swarm/services/<grafana client id>/admin/password` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` | | `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` | | `swarm-bao-ci-runner`, and the runner's journal forwarder in its container | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` and `swarm/services/<runner forwarder client id>/oidc/client` |
| `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` | | `swarm-review-classifier`, in its container | `reviewClassifierClientCertFile` / `reviewClassifierClientKeyFile` | `swarm/services/review-classifier/anthropic-api-key`, `swarm/services/review-classifier/acp-provider` and `swarm/services/<classifier client id>/oidc/client` |
| `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` | | `swarm-claude-login`, in its container | `claudeLoginClientCertFile` / `claudeLoginClientKeyFile` | creates and updates `swarm/agents/+/claude-credentials`; reads `swarm/services/<minter client id>/oidc/client` |
| `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` | | `swarm-builder`, in its container | `builderClientCertFile` / `builderClientKeyFile` | `swarm/services/builder/forge/token` and `swarm/services/<builder client id>/oidc/client` |

19
docs/swarm/services.md
View file

@ -289,13 +289,18 @@ gateway either way.
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. | | `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. | | `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login` and The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-review-classifier` containers each run a collector of their own that `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana` and
forwards the container's journal here, on a route and receiver per service `swarm-victorialogs` containers each run a collector of their own that forwards
(`https://<swarm.otel.domain>/<container>/`), authenticated as that service's the container's journal here, on a route and receiver per service
own OIDC client. Their records carry `service.name` (`bao`, `nats`, `builder`, (`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
`claude-login`, `review-classifier`) and no `hive` label. The forwarder of all of that service's. The queue, builder, login minter and classifier reuse the
but the store's is `nix/container-modules/swarm-journal-forwarder.nix`. service's own client; the store, CI runner, Grafana and log store have a
forwarder client of their own (`swarm.<service>.otel.clientId`). Their records
carry `service.name` (`bao`, `nats`, `builder`, `claude-login`,
`review-classifier`, `ci-runner`, `grafana`, `victorialogs`) and no `hive`
label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable` **Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm doesn't gate them: that option says this host _runs_ the store, while the swarm

3
nix/host-modules/default.nix
View file

@ -47,12 +47,15 @@
./glue-nats-wireguard.nix ./glue-nats-wireguard.nix
./glue-queue-agent-credential.nix ./glue-queue-agent-credential.nix
./glue-builder-oidc-client.nix ./glue-builder-oidc-client.nix
./glue-ci-runner-otel-oidc-client.nix
./glue-claude-login-oidc-client.nix ./glue-claude-login-oidc-client.nix
./glue-grafana-otel-oidc-client.nix
./glue-review-classifier-oidc-client.nix ./glue-review-classifier-oidc-client.nix
./glue-secret-publisher-authelia.nix ./glue-secret-publisher-authelia.nix
./glue-secret-publisher-bao-identity.nix ./glue-secret-publisher-bao-identity.nix
./glue-swarm-bao-otel-oidc-client.nix ./glue-swarm-bao-otel-oidc-client.nix
./glue-swarm-otel-oidc-client.nix ./glue-swarm-otel-oidc-client.nix
./glue-victorialogs-otel-oidc-client.nix
./swarm-authelia-service.nix ./swarm-authelia-service.nix
./swarm-authelia.nix ./swarm-authelia.nix
./swarm-bao-service.nix ./swarm-bao-service.nix

37
nix/host-modules/glue-ci-runner-otel-oidc-client.nix Normal file
View file

@ -0,0 +1,37 @@
# Glue: register the CI runner container's journal forwarder as an OIDC
# client wherever authelia runs.
#
# ONE PAIRING PER FILE: that forwarder ← authelia, and nothing else.
# Without it authelia mints no secret, the publisher has nothing to copy into
# the store, and the forwarder's export is a 401.
#
# Gated on authelia being HERE, not on this host running the runner: a
# client is a row in THIS host's provider config, the split
# ./glue-swarm-bao-otel-oidc-client.nix makes for the same reason.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
clientId = hyperhiveCfg.swarm.ci-runner.otel.clientId;
in
{
config = lib.mkIf hyperhiveCfg.deploy.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = clientId;
description = "HyperHive CI runner journal forwarder";
kind = "machine";
redirectUris = [ ];
# Its own id as its own permitted audience, the one the swarm
# collector's receiver for this container checks.
audience = [ clientId ];
# The receiver verifies the token offline against `/jwks.json`;
# authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
}
];
};
}

37
nix/host-modules/glue-grafana-otel-oidc-client.nix Normal file
View file

@ -0,0 +1,37 @@
# Glue: register the Grafana container's journal forwarder as an OIDC
# client wherever authelia runs.
#
# ONE PAIRING PER FILE: that forwarder ← authelia, and nothing else.
# Without it authelia mints no secret, the publisher has nothing to copy into
# the store, and the forwarder's export is a 401.
#
# Gated on authelia being HERE, not on this host running Grafana: a
# client is a row in THIS host's provider config, the split
# ./glue-swarm-bao-otel-oidc-client.nix makes for the same reason.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
clientId = hyperhiveCfg.swarm.grafana.otel.clientId;
in
{
config = lib.mkIf hyperhiveCfg.deploy.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = clientId;
description = "HyperHive Grafana journal forwarder";
kind = "machine";
redirectUris = [ ];
# Its own id as its own permitted audience, the one the swarm
# collector's receiver for this container checks.
audience = [ clientId ];
# The receiver verifies the token offline against `/jwks.json`;
# authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
}
];
};
}

37
nix/host-modules/glue-victorialogs-otel-oidc-client.nix Normal file
View file

@ -0,0 +1,37 @@
# Glue: register the log store container's journal forwarder as an OIDC
# client wherever authelia runs.
#
# ONE PAIRING PER FILE: that forwarder ← authelia, and nothing else.
# Without it authelia mints no secret, the publisher has nothing to copy into
# the store, and the forwarder's export is a 401.
#
# Gated on authelia being HERE, not on this host running the log store: a
# client is a row in THIS host's provider config, the split
# ./glue-swarm-bao-otel-oidc-client.nix makes for the same reason.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
clientId = hyperhiveCfg.swarm.victorialogs.otel.clientId;
in
{
config = lib.mkIf hyperhiveCfg.deploy.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = clientId;
description = "HyperHive log store journal forwarder";
kind = "machine";
redirectUris = [ ];
# Its own id as its own permitted audience, the one the swarm
# collector's receiver for this container checks.
audience = [ clientId ];
# The receiver verifies the token offline against `/jwks.json`;
# authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
}
];
};
}

14
nix/host-modules/swarm-bao.nix
View file

@ -931,6 +931,8 @@ let
cn = baoDeploy.grafanaOidcCommonName; cn = baoDeploy.grafanaOidcCommonName;
policyText = policyText =
readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client" readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/oidc/client"
# The journal forwarder in its container, under this same leaf.
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.grafana.otel.clientId}/oidc/client"
+ createStanza "${credentialMountPath}/data/${grafanaAdminLeaf}" + createStanza "${credentialMountPath}/data/${grafanaAdminLeaf}"
+ createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}" + createStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
+ '' + ''
@ -1018,14 +1020,17 @@ let
# The log store's auth responders: the agent one checks the secret an agent # The log store's auth responders: the agent one checks the secret an agent
# presents against the one stored for it (the same `+` leaf as the queue # presents against the one stored for it (the same `+` leaf as the queue
# responder's), the other checks the bearer Grafana presents against # responder's), the other checks the bearer Grafana presents against
# Grafana's metrics bearer. Nothing else. # Grafana's metrics bearer; and the store container's journal forwarder
# reads its own client secret under the same leaf. Nothing else.
logsAuthReaders = [ logsAuthReaders = [
{ {
name = "swarm-logs-auth"; name = "swarm-logs-auth";
cn = baoDeploy.logsAuthCommonName; cn = baoDeploy.logsAuthCommonName;
policyText = policyText =
readStanza "${credentialMountPath}/data/swarm/agents/+/queue" readStanza "${credentialMountPath}/data/swarm/agents/+/queue"
+ readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"; + readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"
# The journal forwarder in the store's container, under this same leaf.
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.victorialogs.otel.clientId}/oidc/client";
} }
]; ];
@ -1045,7 +1050,10 @@ let
{ {
name = "swarm-ci-runner"; name = "swarm-ci-runner";
cn = baoDeploy.ciRunnerCommonName; cn = baoDeploy.ciRunnerCommonName;
policyText = readStanza "${credentialMountPath}/data/${ciRunnerTokenLeaf}"; policyText =
readStanza "${credentialMountPath}/data/${ciRunnerTokenLeaf}"
# The journal forwarder in the runner's container, under this same leaf.
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.ci-runner.otel.clientId}/oidc/client";
} }
]; ];

97
nix/host-modules/swarm-ci-runner.nix
View file

@ -55,9 +55,40 @@ let
consumers = [ consumers = [
runnerUnit runnerUnit
registerUnit registerUnit
# Go: `SSL_CERT_FILE` replaces its trust store, for the swarm
# collector's and authelia's names.
"opentelemetry-collector"
]; ];
}; };
# The container's journal forwarder pushes to its own receiver on the swarm
# collector, the route `serviceProducers.swarm-ci-runner` in
# ./swarm-otel-service.nix declares. The port suffix is elided on 443, as in
# ./swarm-bao.nix.
swarmOtelCfg = config.services.hyperhive.swarm.otel;
forwarderClientId = config.services.hyperhive.swarm.ci-runner.otel.clientId;
gatewayPortSuffix = lib.optionalString (
gatewayCfg.httpsPort != 443
) ":${toString gatewayCfg.httpsPort}";
forwarderEndpoint = "https://${swarmOtelCfg.domain}${gatewayPortSuffix}/${machine}";
forwarderTokenUrl = "${toString config.services.hyperhive.swarm.authelia.url}/api/oidc/token";
forwarderUnit = "swarm-journal-forwarder-oidc";
# Loopback in the container's own netns, and distinct from the other service
# containers' forwarders all the same.
forwarderTelemetryPort = 8895;
# The runner's store identity, handed into the container as system
# credentials (credential id → host source) for the forwarder's fetch unit.
storeFiles = lib.filterAttrs (_: source: source != null) {
ci-runner-bao-cert = baoDeploy.ciRunnerClientCertFile;
ci-runner-bao-key = baoDeploy.ciRunnerClientKeyFile;
ci-runner-bao-server-ca = baoDeploy.serverCaFile;
};
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}";
# The upstream unit is `DynamicUser`, so its state dir is # The upstream unit is `DynamicUser`, so its state dir is
# `/var/lib/private/gitea-runner`; `/var/lib/gitea-runner` is a symlink to # `/var/lib/private/gitea-runner`; `/var/lib/gitea-runner` is a symlink to
# it, which is the spelling a step run as root (`+`) can follow. # it, which is the spelling a step run as root (`+`) can follow.
@ -99,6 +130,21 @@ let
privateNetwork = true; privateNetwork = true;
in in
{ {
options.services.hyperhive.swarm.ci-runner.otel.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-ci-runner-collector";
description = ''
OAuth2 client id the journal forwarder inside the CI runner's container
authenticates as, and the audience it asks its token for.
Read-only because three hosts have to agree on it: authelia registers
the client (`glue-ci-runner-otel-oidc-client.nix`), the swarm collector
admits it on the container's receiver (`swarm.otel.serviceProducers`),
and the secret publisher carries its secret to the store.
'';
};
options.services.hyperhive.deploy.ci-runner = { options.services.hyperhive.deploy.ci-runner = {
# `enable` is in ./deploy.nix. # `enable` is in ./deploy.nix.
@ -178,6 +224,14 @@ in
options. options.
''; '';
} }
{
assertion = lib.all oneWord (lib.attrValues storeFiles);
message = ''
The CI runner's store identity paths must not contain whitespace:
nspawn's --load-credential receives them through nixos-container's
unquoted extra flags.
'';
}
]; ];
# The container hangs off the bridge, takes its address from the # The container hangs off the bridge, takes its address from the
@ -274,7 +328,16 @@ in
systemd.timers.swarm-bao-ci-runner = storeRefreshTimer "re-read the CI runner's registration token from the swarm secret store"; systemd.timers.swarm-bao-ci-runner = storeRefreshTimer "re-read the CI runner's registration token from the swarm secret store";
systemd.services."container@${machine}" = caTrust.containerOrdering; # nspawn fails to start the container when a credential source is
# missing, so where this host mints the leaf the container pulls the
# minting unit in and waits for it, as ./swarm-builder.nix's does.
systemd.services."container@${machine}" = lib.mkMerge [
caTrust.containerOrdering
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];
}
];
systemd.tmpfiles.rules = [ systemd.tmpfiles.rules = [
# The bind source must exist when the container starts. # The bind source must exist when the container starts.
@ -287,9 +350,9 @@ in
containers.${machine} = { containers.${machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes # No `--link-journal=host`: the journal stays in the guest, where the
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = [ "--link-journal=host" ]; extraFlags = lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
inherit privateNetwork; inherit privateNetwork;
hostBridge = networkCfg.bridgeName; hostBridge = networkCfg.bridgeName;
@ -338,9 +401,35 @@ in
{ {
imports = [ imports = [
../container-modules/swarm-container.nix ../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
caBundleModule caBundleModule
]; ];
# Authenticated as the forwarder's own OIDC client, with the runner's
# store identity: one identity for the whole container. It is a
# system credential, so the fetch unit loads it.
services.hyperhive.swarmJournalForwarder = {
enable = true;
serviceName = "ci-runner";
endpoint = forwarderEndpoint;
clientId = forwarderClientId;
tokenUrl = forwarderTokenUrl;
telemetryPort = forwarderTelemetryPort;
store = {
package = baoDeploy.package;
addr = "https://${baoCfg.domain}:${toString baoCfg.port}";
secretPath = "secret/swarm/services/${forwarderClientId}/oidc/client";
clientCertFile = forwarderCredential "ci-runner-bao-cert";
clientKeyFile = forwarderCredential "ci-runner-bao-key";
caFile =
if storeFiles ? ci-runner-bao-server-ca then
forwarderCredential "ci-runner-bao-server-ca"
else
null;
};
};
systemd.services.${forwarderUnit}.serviceConfig.LoadCredential = lib.attrNames storeFiles;
# Its own netns, so it keeps its own firewall; resolvconf stays on # Its own netns, so it keeps its own firewall; resolvconf stays on
# and serves `networking.nameservers` below. # and serves `networking.nameservers` below.
services.hyperhive.swarmContainer = { services.hyperhive.swarmContainer = {

18
nix/host-modules/swarm-grafana-service.nix
View file

@ -37,6 +37,24 @@ in
''; '';
}; };
otel.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-grafana-collector";
description = ''
OAuth2 client id the journal forwarder inside Grafana's container
authenticates as, and the audience it asks its token for. Its own,
not {option}`services.hyperhive.swarm.grafana.oidc.clientId`, which
is the browser login's.
Read-only because three hosts have to agree on it: authelia
registers the client (`glue-grafana-otel-oidc-client.nix`), the
swarm collector admits it on the container's receiver
(`swarm.otel.serviceProducers`), and the secret publisher carries
its secret to the store.
'';
};
domain = lib.mkOption { domain = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "grafana.${domainBase}"; default = "grafana.${domainBase}";

116
nix/host-modules/swarm-grafana.nix
View file

@ -229,6 +229,33 @@ let
autheliaUrl = toString hyperhiveCfg.swarm.authelia.url; autheliaUrl = toString hyperhiveCfg.swarm.authelia.url;
# The container's journal forwarder pushes to its own receiver on the swarm
# collector, the route `serviceProducers.swarm-grafana` in
# ./swarm-otel-service.nix declares, as a client of its own rather than the
# browser login's. The port suffix is elided on 443, as in ./swarm-bao.nix.
forwarderClientId = cfg.otel.clientId;
gatewayPortSuffix = lib.optionalString (
gatewayCfg.httpsPort != 443
) ":${toString gatewayCfg.httpsPort}";
forwarderEndpoint = "https://${hyperhiveCfg.swarm.otel.domain}${gatewayPortSuffix}/${cfg.machine}";
forwarderUnit = "swarm-journal-forwarder-oidc";
# In the host netns: distinct from every other collector's there, which the
# assertion below checks.
forwarderTelemetryPort = 8896;
# Grafana's store identity, handed into the container as system credentials
# (credential id → host source) for the forwarder's fetch unit. The host units
# below keep reading the same files directly.
storeFiles = lib.filterAttrs (_: source: source != null) {
grafana-bao-cert = baoDeploy.grafanaOidcClientCertFile;
grafana-bao-key = baoDeploy.grafanaOidcClientKeyFile;
grafana-bao-server-ca = baoDeploy.serverCaFile;
};
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}";
# The three secrets below land on the host's /run (tmpfs), which the # The three secrets below land on the host's /run (tmpfs), which the
# container bind-mounts read-only at the same path: none reaches persistent # container bind-mounts read-only at the same path: none reaches persistent
# disk, and a reboot fetches them again. Each file is absent until its # disk, and a reboot fetches them again. Each file is absent until its
@ -493,6 +520,39 @@ in
from that CA out of band and named here — see docs/swarm/secrets.md. from that CA out of band and named here — see docs/swarm/secrets.md.
''; '';
} }
{
assertion = lib.all oneWord (lib.attrValues storeFiles);
message = ''
Grafana's store identity paths must not contain whitespace: nspawn's
--load-credential receives them through nixos-container's unquoted
extra flags.
'';
}
{
assertion =
(hyperhiveCfg.swarm.otel.serviceProducers.${cfg.machine}.clientId or null) == forwarderClientId;
message = ''
services.hyperhive.swarm.otel.serviceProducers has no entry
"${cfg.machine}" with clientId "${forwarderClientId}", so the swarm
collector has no receiver for Grafana's journal forwarder. Declare
it in swarm-otel-service.nix.
'';
}
{
assertion =
!(lib.elem forwarderTelemetryPort [
hyperhiveCfg.otel.telemetryPort
hyperhiveCfg.swarm.otel.telemetryPort
hyperhiveCfg.swarm.bao.otel.telemetryPort
hyperhiveCfg.swarm.nats.otel.telemetryPort
]);
message = ''
Grafana's journal forwarder's telemetry port (${toString forwarderTelemetryPort})
collides with another collector's in the host netns: the hive
collector's, the swarm collector's, the store forwarder's or the
queue forwarder's.
'';
}
]; ];
# Websockets: Grafana Live streams panel updates over one, and without # Websockets: Grafana Live streams panel updates over one, and without
@ -555,8 +615,17 @@ in
}; };
# Order the container after the host CA service so the bind source below # Order the container after the host CA service so the bind source below
# exists before nspawn sets the mount up. # exists before nspawn sets the mount up. nspawn also fails to start it
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; # when a credential source is missing, so where this host mints the leaf
# the container pulls the minting unit in and waits for it, as
# ./swarm-builder.nix's does.
systemd.services."container@${cfg.machine}" = lib.mkMerge [
caTrust.containerOrdering
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];
}
];
# The socket directory, created host-side before the container starts. # The socket directory, created host-side before the container starts.
# nixos-container refuses to start when a bind source is missing, so this # nixos-container refuses to start when a bind source is missing, so this
@ -1036,9 +1105,9 @@ in
containers.${cfg.machine} = { containers.${cfg.machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes # No `--link-journal=host`: the journal stays in the guest, where the
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = [ "--link-journal=host" ]; extraFlags = lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
inherit privateNetwork; inherit privateNetwork;
# The socket directory, shared with the host so nginx can reach in. # The socket directory, shared with the host so nginx can reach in.
@ -1068,9 +1137,16 @@ in
{ {
imports = [ imports = [
../container-modules/swarm-container.nix ../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
(import ../container-modules/swarm-container-resolver.nix { (import ../container-modules/swarm-container-resolver.nix {
inherit (networkCfg) bridgeIp; inherit (networkCfg) bridgeIp;
dnsConsumers = [ "grafana.service" ]; # The forwarder and its secret fetch dial the swarm collector,
# authelia and the store by name too.
dnsConsumers = [
"grafana.service"
"opentelemetry-collector.service"
"${forwarderUnit}.service"
];
}) })
# Grafana is Go, and `SSL_CERT_FILE` *replaces* the default store # Grafana is Go, and `SSL_CERT_FILE` *replaces* the default store
# rather than adding to it — so it needs the system CAs and the # rather than adding to it — so it needs the system CAs and the
@ -1080,12 +1156,38 @@ in
(caTrust.trustBundle { (caTrust.trustBundle {
inherit pkgs; inherit pkgs;
name = cfg.machine; name = cfg.machine;
consumers = [ "grafana" ]; consumers = [
"grafana"
# Go too, for the swarm collector's and authelia's names.
"opentelemetry-collector"
];
}) })
]; ];
services.hyperhive.swarmContainer = { inherit privateNetwork; }; services.hyperhive.swarmContainer = { inherit privateNetwork; };
# Authenticated as the forwarder's own OIDC client, with Grafana's
# store identity: one identity for the whole container. It is a
# system credential, so the fetch unit loads it.
services.hyperhive.swarmJournalForwarder = {
enable = true;
serviceName = "grafana";
endpoint = forwarderEndpoint;
clientId = forwarderClientId;
tokenUrl = "${autheliaUrl}/api/oidc/token";
telemetryPort = forwarderTelemetryPort;
store = {
package = baoDeploy.package;
addr = "https://${baoCfg.domain}:${toString baoCfg.port}";
secretPath = "secret/swarm/services/${forwarderClientId}/oidc/client";
clientCertFile = forwarderCredential "grafana-bao-cert";
clientKeyFile = forwarderCredential "grafana-bao-key";
caFile =
if storeFiles ? grafana-bao-server-ca then forwarderCredential "grafana-bao-server-ca" else null;
};
};
systemd.services.${forwarderUnit}.serviceConfig.LoadCredential = lib.attrNames storeFiles;
# Grafana's `secret_key` has **no default in nixpkgs** and an # Grafana's `secret_key` has **no default in nixpkgs** and an
# assertion refuses the build without one — which is how the first # assertion refuses the build without one — which is how the first
# version of this module broke a deploy. It signs the datasource # version of this module broke a deploy. It signs the datasource

14
nix/host-modules/swarm-otel-service.nix
View file

@ -415,6 +415,20 @@ in
clientId = hyperhiveCfg.swarm.review-classifier.clientId; clientId = hyperhiveCfg.swarm.review-classifier.clientId;
port = lib.mkDefault 4395; port = lib.mkDefault 4395;
}; };
# The forwarders in ./swarm-ci-runner.nix, ./swarm-grafana.nix and
# ./swarm-victorialogs.nix, each authenticated as a client of its own.
swarm-ci-runner = {
clientId = hyperhiveCfg.swarm.ci-runner.otel.clientId;
port = lib.mkDefault 4396;
};
swarm-grafana = {
clientId = hyperhiveCfg.swarm.grafana.otel.clientId;
port = lib.mkDefault 4397;
};
swarm-victorialogs = {
clientId = hyperhiveCfg.swarm.victorialogs.otel.clientId;
port = lib.mkDefault 4398;
};
}; };
# Not under any `enable`: see `publishedScrapeTargets` for why every host # Not under any `enable`: see `publishedScrapeTargets` for why every host

3
nix/host-modules/swarm-otel.nix
View file

@ -214,7 +214,8 @@ let
# ⚠️ Drop that flag from one of THOSE containers and this receiver silently # ⚠️ Drop that flag from one of THOSE containers and this receiver silently
# stops seeing it — no error, just a unit that never appears in the store. # stops seeing it — no error, just a unit that never appears in the store.
# The `serviceProducers` containers (`swarm-bao`, `swarm-nats`, # The `serviceProducers` containers (`swarm-bao`, `swarm-nats`,
# `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`) are the # `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`,
# `swarm-ci-runner`, `swarm-grafana`, `swarm-victorialogs`) are the
# exceptions on purpose: their journals stay inside the container for its own # exceptions on purpose: their journals stay inside the container for its own
# collector instead, so this receiver never sees their units. # collector instead, so this receiver never sees their units.
hostJournalDir = "/var/log/journal"; hostJournalDir = "/var/log/journal";

5
nix/host-modules/swarm-secret-publisher.nix
View file

@ -98,6 +98,11 @@ let
hyperhiveCfg.swarm.builder.clientId hyperhiveCfg.swarm.builder.clientId
# The claude login minter, which reads it itself inside its container. # The claude login minter, which reads it itself inside its container.
hyperhiveCfg.swarm.claude-login.clientId hyperhiveCfg.swarm.claude-login.clientId
# The journal forwarders in the CI runner's, Grafana's and the log store's
# containers, each reading it inside its container.
hyperhiveCfg.swarm.ci-runner.otel.clientId
hyperhiveCfg.swarm.grafana.otel.clientId
hyperhiveCfg.swarm.victorialogs.otel.clientId
]; ];
# The swarm controller's OIDC client, which it reads back from the store # The swarm controller's OIDC client, which it reads back from the store

16
nix/host-modules/swarm-victorialogs-service.nix
View file

@ -31,6 +31,22 @@ in
''; '';
}; };
otel.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-victorialogs-collector";
description = ''
OAuth2 client id the journal forwarder inside the log store's
container authenticates as, and the audience it asks its token for.
Read-only because three hosts have to agree on it: authelia
registers the client (`glue-victorialogs-otel-oidc-client.nix`),
the swarm collector admits it on the container's receiver
(`swarm.otel.serviceProducers`), and the secret publisher carries
its secret to the store.
'';
};
domain = lib.mkOption { domain = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "logs.${domainBase}"; default = "logs.${domainBase}";

120
nix/host-modules/swarm-victorialogs.nix
View file

@ -80,6 +80,35 @@ let
# Shared host netns, like every sibling swarm container: Grafana reaches # Shared host netns, like every sibling swarm container: Grafana reaches
# this at 127.0.0.1:<port>. # this at 127.0.0.1:<port>.
privateNetwork = false; privateNetwork = false;
# The container's journal forwarder pushes to its own receiver on the swarm
# collector, the route `serviceProducers.swarm-victorialogs` in
# ./swarm-otel-service.nix declares. The port suffix is elided on 443, as in
# ./swarm-bao.nix.
forwarderClientId = cfg.otel.clientId;
gatewayPortSuffix = lib.optionalString (
gatewayCfg.httpsPort != 443
) ":${toString gatewayCfg.httpsPort}";
forwarderEndpoint = "https://${hyperhiveCfg.swarm.otel.domain}${gatewayPortSuffix}/${cfg.machine}";
forwarderUnit = "swarm-journal-forwarder-oidc";
# In the host netns: distinct from every other collector's there, which the
# assertion below checks.
forwarderTelemetryPort = 8897;
forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}";
# The responder's store identity, handed into the container as system
# credentials for the forwarder's fetch unit. The host units keep reading
# the same files.
forwarderStoreFiles = lib.filterAttrs (_: source: source != null) logsAuthStoreFiles;
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
# The forwarder reaches the swarm collector and authelia by name over TLS; in
# self-signed mode their leaves chain to the hive CA.
caTrust = import ./lib/hive-ca-trust.nix {
inherit lib gatewayCfg;
tlsCfg = deployCfg.hive-controller.tls;
};
in in
{ {
# What the store IS from any hive's point of view, the name it answers on and # What the store IS from any hive's point of view, the name it answers on and
@ -211,6 +240,51 @@ in
both options. both options.
''; '';
} }
{
assertion = lib.all oneWord (lib.attrValues forwarderStoreFiles);
message = ''
The log store responder's store identity paths must not contain
whitespace: nspawn's --load-credential receives them through
nixos-container's unquoted extra flags.
'';
}
{
assertion =
(hyperhiveCfg.swarm.otel.serviceProducers.${cfg.machine}.clientId or null) == forwarderClientId;
message = ''
services.hyperhive.swarm.otel.serviceProducers has no entry
"${cfg.machine}" with clientId "${forwarderClientId}", so the swarm
collector has no receiver for the log store's journal forwarder.
Declare it in swarm-otel-service.nix.
'';
}
{
assertion =
!(lib.elem forwarderTelemetryPort [
hyperhiveCfg.otel.telemetryPort
hyperhiveCfg.swarm.otel.telemetryPort
hyperhiveCfg.swarm.bao.otel.telemetryPort
hyperhiveCfg.swarm.nats.otel.telemetryPort
]);
message = ''
The log store's journal forwarder's telemetry port (${toString forwarderTelemetryPort})
collides with another collector's in the host netns: the hive
collector's, the swarm collector's, the store forwarder's or the
queue forwarder's.
'';
}
];
# nspawn fails to start the container when a credential source is
# missing, so where this host mints the leaf the container pulls the
# minting unit in and waits for it, as ./swarm-builder.nix's does. The CA
# ordering keeps the trust bundle's bind source there first.
systemd.services."container@${cfg.machine}" = lib.mkMerge [
caTrust.containerOrdering
{
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];
}
]; ];
# Which check a `/select/logsql/` request gets. `auth_request` takes a # Which check a `/select/logsql/` request gets. `auth_request` takes a
@ -506,24 +580,62 @@ in
containers.${cfg.machine} = { containers.${cfg.machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes # No `--link-journal=host`: the journal stays in the guest, where the
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = [ "--link-journal=host" ]; extraFlags = lib.mapAttrsToList (
id: source: "--load-credential=${id}:${source}"
) forwarderStoreFiles;
inherit privateNetwork; inherit privateNetwork;
bindMounts = caTrust.bindMount;
config = config =
{ ... }: { ... }:
{ {
imports = [ imports = [
../container-modules/swarm-container.nix ../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
(import ../container-modules/swarm-container-resolver.nix { (import ../container-modules/swarm-container-resolver.nix {
inherit (networkCfg) bridgeIp; inherit (networkCfg) bridgeIp;
dnsConsumers = [ "victorialogs.service" ]; # The forwarder and its secret fetch dial the swarm collector,
# authelia and the store by name too.
dnsConsumers = [
"victorialogs.service"
"opentelemetry-collector.service"
"${forwarderUnit}.service"
];
})
# Go: `SSL_CERT_FILE` replaces its trust store, for the swarm
# collector's and authelia's names.
(caTrust.trustBundle {
inherit pkgs;
name = cfg.machine;
consumers = [ "opentelemetry-collector" ];
}) })
]; ];
services.hyperhive.swarmContainer = { inherit privateNetwork; }; services.hyperhive.swarmContainer = { inherit privateNetwork; };
# Authenticated as the forwarder's own OIDC client, with the
# responder's store identity: one identity for the whole container.
# It is a system credential, so the fetch unit loads it.
services.hyperhive.swarmJournalForwarder = {
enable = true;
serviceName = "victorialogs";
endpoint = forwarderEndpoint;
clientId = forwarderClientId;
tokenUrl = "${toString hyperhiveCfg.swarm.authelia.url}/api/oidc/token";
telemetryPort = forwarderTelemetryPort;
store = {
package = deployCfg.bao.package;
addr = "https://${baoCfg.domain}:${toString baoCfg.port}";
secretPath = "secret/swarm/services/${forwarderClientId}/oidc/client";
clientCertFile = forwarderCredential "bao-client.pem";
clientKeyFile = forwarderCredential "bao-client-key.pem";
caFile = if forwarderStoreFiles ? "bao-ca.pem" then forwarderCredential "bao-ca.pem" else null;
};
};
systemd.services.${forwarderUnit}.serviceConfig.LoadCredential = lib.attrNames forwarderStoreFiles;
services.victorialogs = { services.victorialogs = {
enable = true; enable = true;
package = deployCfg.victorialogs.package; package = deployCfg.victorialogs.package;

38
nix/module-eval/bao-grants.nix
View file

@ -600,7 +600,7 @@ let
# run rotate a value Grafana or the metrics store already holds. The # run rotate a value Grafana or the metrics store already holds. The
# controller's Grafana token is the one leaf it may replace, which it # controller's Grafana token is the one leaf it may replace, which it
# does when Grafana refuses the stored token. # does when Grafana refuses the stored token.
name = "the Grafana reader reads its own client secret, creates its admin password and metrics bearer, and writes the controller's Grafana token, and nothing else"; name = "the Grafana reader reads its own and its forwarder's client secrets, creates its admin password and metrics bearer, and writes the controller's Grafana token, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script; s = baoGrantHere.systemd.services.swarm-bao-grafana-oidc-policy.script;
@ -610,6 +610,10 @@ let
path = "secret/data/swarm/services/swarm-grafana/oidc/client"; path = "secret/data/swarm/services/swarm-grafana/oidc/client";
caps = [ "read" ]; caps = [ "read" ];
} }
{
path = "secret/data/swarm/services/swarm-grafana-collector/oidc/client";
caps = [ "read" ];
}
{ {
path = "secret/data/swarm/services/swarm-grafana/admin/password"; path = "secret/data/swarm/services/swarm-grafana/admin/password";
caps = [ caps = [
@ -726,10 +730,10 @@ let
lib.hasInfix "/nats-auth.pem ]" s && lib.hasInfix "swarm-nats-auth \"\" clientAuth" s; lib.hasInfix "/nats-auth.pem ]" s && lib.hasInfix "swarm-nats-auth \"\" clientAuth" s;
} }
{ {
# Every agent's queue secret and Grafana's metrics bearer, both # Every agent's queue secret, Grafana's metrics bearer and the store
# read-only, and nothing else: this principal checks a presented secret # container forwarder's client secret, all read-only, and nothing else:
# and does nothing with it. # this principal checks a presented secret and does nothing with it.
name = "the log store responder's grant is every agent's queue secret and Grafana's bearer, read-only, and nothing else"; name = "the log store responder's grant is every agent's queue secret, Grafana's bearer and its container forwarder's client secret, read-only, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-logs-auth-policy.script; s = baoGrantHere.systemd.services.swarm-bao-logs-auth-policy.script;
@ -743,6 +747,10 @@ let
path = "secret/data/swarm/services/swarm-grafana/metrics/bearer"; path = "secret/data/swarm/services/swarm-grafana/metrics/bearer";
caps = [ "read" ]; caps = [ "read" ];
} }
{
path = "secret/data/swarm/services/swarm-victorialogs-collector/oidc/client";
caps = [ "read" ];
}
] ]
&& lib.hasInfix "auth/cert/certs/swarm-logs-auth" s && lib.hasInfix "auth/cert/certs/swarm-logs-auth" s
&& lib.hasInfix "allowed_common_names=swarm-logs-auth" s && lib.hasInfix "allowed_common_names=swarm-logs-auth" s
@ -1536,16 +1544,24 @@ let
&& !(lib.hasInfix "secret/data/swarm/services/review-classifier/acp-provider" s); && !(lib.hasInfix "secret/data/swarm/services/review-classifier/acp-provider" s);
} }
{ {
# The runner's role is the token's only reader. Its policy is the one # The runner's role is the token's only reader. Its policy is that leaf
# leaf, read-only, under the runner's own subject. # and its container forwarder's client secret, each read-only, under the
name = "the CI runner's grant is its registration token and nothing else"; # runner's own subject.
name = "the CI runner's grant is its registration token and its forwarder's client secret, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-ci-runner-policy.script; s = baoGrantHere.systemd.services.swarm-bao-ci-runner-policy.script;
in in
lib.hasInfix "path \"secret/data/swarm/services/ci-runner/registration-token\" {" s grantsIn s == [
&& lib.hasInfix "capabilities = [\"read\"]" s {
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1 path = "secret/data/swarm/services/ci-runner/registration-token";
caps = [ "read" ];
}
{
path = "secret/data/swarm/services/swarm-ci-runner-collector/oidc/client";
caps = [ "read" ];
}
]
&& !(lib.hasInfix "secret/data/swarm/services/*" s) && !(lib.hasInfix "secret/data/swarm/services/*" s)
&& !(lib.hasInfix "secret/data/swarm/agents" s) && !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/hives" s) && !(lib.hasInfix "secret/data/swarm/hives" s)

45
nix/module-eval/forge-placement.nix
View file

@ -492,6 +492,51 @@ let
forgeSettings.service.DISABLE_REGISTRATION == true forgeSettings.service.DISABLE_REGISTRATION == true
&& !(forgeSettings.service.ALLOW_ONLY_INTERNAL_REGISTRATION or false); && !(forgeSettings.service.ALLOW_ONLY_INTERNAL_REGISTRATION or false);
} }
{
# The runner's own store leaf, handed in as system credentials, reads the
# forwarder's client secret; the host's token fetch keeps the same files.
name = "the CI runner container forwards its journal to its own route as its own client";
ok =
let
h = ciAlone;
container = h.containers.swarm-ci-runner;
c = container.config;
s = c.services.opentelemetry-collector.settings;
fetch = c.systemd.services.swarm-journal-forwarder-oidc;
clientId = h.services.hyperhive.swarm.ci-runner.otel.clientId;
dir = "/run/credentials/swarm-journal-forwarder-oidc.service";
in
c.services.opentelemetry-collector.enable
&&
s.exporters.otlphttp.endpoint == "https://${h.services.hyperhive.swarm.otel.domain}/swarm-ci-runner"
&& s.extensions."oauth2client/swarm".client_id == clientId
&& s.extensions."oauth2client/swarm".endpoint_params.audience == clientId
&& h.services.hyperhive.swarm.otel.serviceProducers.swarm-ci-runner.clientId == clientId
&&
(lib.findFirst (a: a.key == "service.name") null s.processors.resource.attributes).value
== "ci-runner"
&& fetch.environment.BAO_CLIENT_CERT == "${dir}/ci-runner-bao-cert"
&& fetch.environment.BAO_CLIENT_KEY == "${dir}/ci-runner-bao-key"
&& lib.elem "ci-runner-bao-cert" fetch.serviceConfig.LoadCredential
&& lib.elem "ci-runner-bao-key" fetch.serviceConfig.LoadCredential
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/${clientId}/oidc/client" fetch.script
&& !(lib.elem "--link-journal=host" container.extraFlags)
&& lib.elem "--load-credential=ci-runner-bao-cert:/etc/pki/bao-ci-runner.pem" container.extraFlags
&& lib.elem "--load-credential=ci-runner-bao-key:/etc/pki/bao-ci-runner-key.pem" container.extraFlags
&&
h.systemd.services.swarm-bao-ci-runner.environment.BAO_CLIENT_CERT == "/etc/pki/bao-ci-runner.pem";
}
{
name = "the CI runner forwarder's metrics port is loopback";
ok =
let
c = ciAlone.containers.swarm-ci-runner.config;
reader = lib.head c.services.opentelemetry-collector.settings.service.telemetry.metrics.readers;
in
c.services.hyperhive.swarmJournalForwarder.telemetryPort == 8895
&& reader.pull.exporter.prometheus.host == "127.0.0.1"
&& reader.pull.exporter.prometheus.port == 8895;
}
]; ];
in in
runGroup "forge-placement" cases runGroup "forge-placement" cases

141
nix/module-eval/nats-authelia.nix
View file

@ -180,6 +180,98 @@ let
] ]
) forwarderServices; ) forwarderServices;
# The forwarders that authenticate as a client of their own rather than the
# service's, each with the store identity the host already holds. The CI
# runner's container cases are in ./forge-placement.nix, beside its fixture.
ownClientForwarders = [
{
machine = "swarm-ci-runner";
option = "ci-runner";
}
{
machine = "swarm-grafana";
option = "grafana";
}
{
machine = "swarm-victorialogs";
option = "victorialogs";
}
];
ownClientIdOf = h: svc: h.services.hyperhive.swarm.${svc.option}.otel.clientId;
# The two of them that share the host netns, on the shared services host.
hostNetnsForwarders = [
{
machine = "swarm-grafana";
option = "grafana";
name = "grafana";
cert = "grafana-bao-cert";
key = "grafana-bao-key";
port = 8896;
}
{
machine = "swarm-victorialogs";
option = "victorialogs";
name = "victorialogs";
cert = "bao-client.pem";
key = "bao-client-key.pem";
port = 8897;
}
];
hostNetnsForwarderCases = lib.concatMap (
svc:
let
h = servicesHost;
container = h.containers.${svc.machine};
c = container.config;
s = c.services.opentelemetry-collector.settings;
otel = h.services.hyperhive.swarm.otel;
fetch = c.systemd.services.swarm-journal-forwarder-oidc;
clientId = ownClientIdOf h svc;
in
[
{
name = "the ${svc.name} container forwards its journal to its own route as its own client";
ok =
c.services.opentelemetry-collector.enable
&& s.exporters.otlphttp.endpoint == "https://${otel.domain}/${svc.machine}"
&& s.extensions."oauth2client/swarm".client_id == clientId
&& s.extensions."oauth2client/swarm".endpoint_params.audience == clientId
&& otel.serviceProducers.${svc.machine}.clientId == clientId
&&
(lib.findFirst (a: a.key == "service.name") null s.processors.resource.attributes).value == svc.name
&& fetch.environment.BAO_CLIENT_CERT == "${forwarderFetchDir}/${svc.cert}"
&& fetch.environment.BAO_CLIENT_KEY == "${forwarderFetchDir}/${svc.key}"
&& lib.elem svc.cert fetch.serviceConfig.LoadCredential
&& lib.elem svc.key fetch.serviceConfig.LoadCredential
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/${clientId}/oidc/client" fetch.script
&& !(lib.elem "--link-journal=host" container.extraFlags)
&& lib.any (lib.hasPrefix "--load-credential=${svc.cert}:") container.extraFlags
&& lib.any (lib.hasPrefix "--load-credential=${svc.key}:") container.extraFlags;
}
{
# Shared host netns: one port space with every other collector there.
name = "the ${svc.name} forwarder's metrics port is free in the host netns and loopback";
ok =
let
taken = [
8888
8889
8890
h.services.hyperhive.swarm.nats.otel.telemetryPort
]
++ map (o: o.port) (lib.filter (o: o.machine != svc.machine) hostNetnsForwarders);
reader = lib.head s.service.telemetry.metrics.readers;
in
c.services.hyperhive.swarmJournalForwarder.telemetryPort == svc.port
&& !(lib.elem svc.port taken)
&& reader.pull.exporter.prometheus.host == "127.0.0.1"
&& reader.pull.exporter.prometheus.port == svc.port;
}
]
) hostNetnsForwarders;
cases = [ cases = [
{ {
name = "authelia and nats declare no scrape target on a host with no collector"; name = "authelia and nats declare no scrape target on a host with no collector";
@ -763,7 +855,54 @@ let
in in
lib.all ok1 (map (svc: svc.machine) forwarderServices); lib.all ok1 (map (svc: svc.machine) forwarderServices);
} }
{
name = "the swarm collector admits the CI runner, Grafana and log store forwarders on receivers of their own";
ok =
let
h = shared.baoWithCollector;
otel = h.services.hyperhive.swarm.otel;
s = h.containers.swarm-otel.config.services.opentelemetry-collector.settings;
ports = map (p: p.port) (lib.attrValues otel.serviceProducers);
in
lib.all (
svc:
s.receivers."otlp/${svc.machine}".protocols.http.auth.authenticator == "oidc/${svc.machine}"
&& s.extensions."oidc/${svc.machine}".audience == ownClientIdOf h svc
&& lib.elem "oidc/${svc.machine}" s.service.extensions
&& lib.elem "otlp/${svc.machine}" s.service.pipelines."logs/${otel.producerName}".receivers
&& h.services.nginx.virtualHosts.${otel.domain}.locations ? "/${svc.machine}/"
) ownClientForwarders
&& lib.length (lib.unique ports) == lib.length ports;
}
{
# Grafana's forwarder is a second client beside its browser login, which
# stays an interactive client.
name = "the CI runner, Grafana and log store forwarder clients are registered as machine clients with their own audience and a signed access token";
ok =
let
h = shared.autheliaOnly;
clients = h.services.hyperhive.swarm.authelia.oidc.clients;
find = id: lib.findFirst (c: c.id == id) null clients;
ok1 =
svc:
let
id = ownClientIdOf h svc;
c = find id;
in
c != null
&& c.audience == [ id ]
&& c.accessTokenSignedResponseAlg == "RS256"
&& c.kind == "machine";
browser = find h.services.hyperhive.swarm.grafana.oidc.clientId;
in
lib.all ok1 ownClientForwarders
&&
h.services.hyperhive.swarm.grafana.otel.clientId != h.services.hyperhive.swarm.grafana.oidc.clientId
&& browser != null
&& browser.kind != "machine";
}
] ]
++ forwarderCases; ++ forwarderCases
++ hostNetnsForwarderCases;
in in
runGroup "nats-authelia" cases runGroup "nats-authelia" cases

18
nix/module-eval/secret-publisher.nix
View file

@ -216,6 +216,24 @@ let
secretPublisherHere.systemd.services.swarm-secret-publish.script secretPublisherHere.systemd.services.swarm-secret-publish.script
); );
} }
{
# The forwarders in the CI runner's, Grafana's and the log store's
# containers read their secrets from these paths.
name = "the publisher carries the CI runner, Grafana and log store forwarders' secrets";
ok =
lib.all
(
id:
lib.hasInfix "secret/swarm/services/${id}/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
)
)
[
"swarm-ci-runner-collector"
"swarm-grafana-collector"
"swarm-victorialogs-collector"
];
}
{ {
# The review classifier reads its queue client secret from the store # The review classifier reads its queue client secret from the store
# inside its container; authelia registers it on every IdP host. # inside its container; authelia registers it on every IdP host.