vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-nats: forward the queue container's journal to the swarm collector

Browse source 
The swarm-nats container now runs its own otel collector, which reads the
container's journal and pushes it to the swarm collector at
https://<swarm.otel.domain>/swarm-nats, authenticated as the queue's
existing OIDC client (swarm.nats.clientId). Records carry
service.name=nats and no hive label.

The collector lives in a reusable container module,
nix/container-modules/swarm-journal-forwarder.nix, which follows
swarm-bao's forwarder: a journald receiver over /var/log/journal with
merge and a file_storage cursor, an otlphttp exporter with an
oauth2client authenticator that requests its own client id as audience,
self-metrics on a loopback telemetry port, and the client secret handed
in through LoadCredential. Unlike swarm-bao, the secret is fetched inside
the container by swarm-journal-forwarder-oidc.service, under the
container's own store identity (the swarm-nats-auth leaf, which already
holds a read grant on secret/swarm/services/swarm-nats/oidc/client).
The fetch retries until the store answers, and the collector restarts
until the file exists. A 10-minute timer re-reads the secret and restarts
the collector when it changed.

The swarm collector's authenticated producer route was hard-wired to
swarm-bao. It is now rendered from swarm.otel.serviceProducers
(internal), one receiver, oidc authenticator, gateway location and
pipeline entry per service: swarm-bao (port 4391, unchanged names) and
swarm-nats (port 4392). The reserved-fragment and port-collision
assertions cover every entry.

The queue's OIDC client registration gains its own id as audience and
RS256-signed access tokens, which the receiver verifies offline.

swarm-nats drops --link-journal=host, so its journal stays in the guest
where the forwarder reads it, as swarm-bao did in 44009dc51. The new
swarm.nats.otel.telemetryPort (8891) keeps the forwarder off the
collectors' 8888-8890 in the shared host netns, with an assertion.

module-eval-nats-authelia gains four cases: the container's forwarder
and fetch unit on the store host and on a queue-only host, the
responder role's read grant, the swarm collector's swarm-nats receiver
on a host without the queue, and the client registration.

Part of #4499
Part of #5342
This commit is contained in:
atlas 2026-10-16 02:04:02 +02:00 • committed by mara
commit 40a2bd047b
9 changed files with 660 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

55
docs/swarm/secrets.md
View file

@ -41,33 +41,34 @@ neither is a renaming of the other.
## Swarm-level — one of each per swarm ## Swarm-level — one of each per swarm
| secret | generated by | lives at | hive elsewhere | | secret | generated by | lives at | hive elsewhere |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | | authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them |
| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | | authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same |
| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | | Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | | Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | | the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents/<agent>/claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) |
| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | | queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | | the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | | the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this |
| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything |
Authelia mints the three keys for itself, in-container, precisely Authelia mints the three keys for itself, in-container, precisely
because nothing outside that container ever reads them. **That's the test worth because nothing outside that container ever reads them. **That's the test worth

7
docs/swarm/services.md
View file

@ -288,6 +288,13 @@ gateway either way.
| `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. | | `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. |
| `swarm.otel.port` | Only if something else on the services host already claims `4319`. | | `swarm.otel.port` | Only if something else on the services host already claims `4319`. |
The `swarm-bao` and `swarm-nats` containers each run a collector of their own
that forwards the container's journal here, on a route and receiver per
service (`https://<swarm.otel.domain>/<container>/`), authenticated as that
service's own OIDC client. Their records carry `service.name` (`bao`,
`nats`) and no `hive` label. The queue's forwarder is
`nix/container-modules/swarm-journal-forwarder.nix`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable` **Both store exporters are unconditional**, and `deploy.victoriametrics.enable`
doesn't gate them: that option says this host _runs_ the store, while the swarm doesn't gate them: that option says this host _runs_ the store, while the swarm
has one either way, reached by its swarm name through the gateway. has one either way, reached by its swarm name through the gateway.

315
nix/container-modules/swarm-journal-forwarder.nix Normal file
View file

@ -0,0 +1,315 @@
# A swarm service container's own journal forwarder: an otel collector that
# reads this container's journal and pushes it to the swarm collector by its
# gateway name, authenticated as the service's own OIDC client.
#
# Imported inside the container's own `config`. The host module that declares
# `containers.<name>` sets the options below: the container's evaluation is
# nested inside the host's, and reading the host side back from in here
# recurses.
#
# The client secret is fetched inside the container, under the container's own
# store identity, by `swarm-journal-forwarder-oidc`. That needs the container to
# reach the store, authelia and the swarm collector by name: the container needs
# a network and a resolver.
#
# ⚠️ The host must NOT start this container with `--link-journal=host`. That
# bind-mounts a host directory over the guest's /var/log/journal which journald
# does not write into, and the receiver below reads nothing while the collector
# stays healthy. nixpkgs' default `--link-journal=try-guest` keeps the journal
# in the guest, where this reads it.
{
config,
lib,
pkgs,
...
}:
let
cfg = config.services.hyperhive.swarmJournalForwarder;
atomicWriteSecret = import ../host-modules/lib/atomic-write-secret.nix { };
refreshConsumer = import ../host-modules/lib/refresh-consumer.nix { };
storeRetry = import ../host-modules/lib/store-retry.nix { };
storeRefreshTimer = import ../host-modules/lib/store-refresh-timer.nix { };
fetchUnit = "swarm-journal-forwarder-oidc";
# On the container's /run (tmpfs), so the secret never reaches persistent
# disk. The file is not created ahead of the fetch: `LoadCredential` refuses a
# missing source, so the collector stays down until the store answers and
# then resumes from its saved journal cursor.
secretDir = "/run/${fetchUnit}";
secretFile = "${secretDir}/client-secret";
# The collector runs under `DynamicUser` and opens `client_secret_file`
# itself. `LoadCredential` reads the file as root and re-exposes it under
# `${CREDENTIALS_DIRECTORY}/<id>`, whichever uid the unit got.
credentialId = "oidc-client-secret";
authName = "oauth2client/swarm";
in
{
options.services.hyperhive.swarmJournalForwarder = {
enable = lib.mkEnableOption "this container's own journal forwarder to the swarm collector";
serviceName = lib.mkOption {
type = lib.types.str;
example = "nats";
description = ''
`service.name` stamped on every record. The display identity a
dashboard selects on, matching the service's scrape job name. No
`hive` label: the swarm collector stamps `swarm` from the receiver
that accepted the push.
'';
};
endpoint = lib.mkOption {
type = lib.types.str;
example = "https://otel.swarm.example/swarm-nats";
description = ''
Base URL of this service's own receiver on the swarm collector:
`https://''${swarm.otel.domain}/<producer>`, a route
{option}`services.hyperhive.swarm.otel.serviceProducers` declares.
The exporter appends `/v1/logs`.
'';
};
clientId = lib.mkOption {
type = lib.types.str;
description = ''
OAuth2 client the forwarder mints its token as, and the audience it
asks for. Must equal the `clientId` of this service's
{option}`services.hyperhive.swarm.otel.serviceProducers` entry, which
is the one audience that receiver accepts.
'';
};
tokenUrl = lib.mkOption {
type = lib.types.str;
example = "https://auth.swarm.example/api/oidc/token";
description = "Authelia's token endpoint.";
};
telemetryPort = lib.mkOption {
type = lib.types.port;
description = ''
Loopback port for the collector's own metrics (refused and dropped
records, exporter failures). Required because the container may share
the host netns, where the binary's default 8888 is the hive tier's.
'';
};
store = {
package = lib.mkOption {
type = lib.types.package;
description = "The `bao` CLI the fetch unit logs in and reads with.";
};
addr = lib.mkOption {
type = lib.types.str;
description = "The swarm secret store's address, by name.";
};
secretPath = lib.mkOption {
type = lib.types.str;
example = "secret/swarm/services/swarm-nats/oidc/client";
description = ''
KV path of the client secret. The container's store identity must
hold a read grant on it.
'';
};
clientCertFile = lib.mkOption {
type = lib.types.str;
description = ''
The container's store identity, as a path inside the container. A
path, never a value.
'';
};
clientKeyFile = lib.mkOption {
type = lib.types.str;
description = "Private key for `store.clientCertFile`. A path, never a value.";
};
caFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = "CA the store's server certificate chains to; null means the system trust store.";
};
};
};
config = lib.mkIf cfg.enable {
assertions = [
{
# With a volatile journal the receiver finds an empty directory and the
# collector starts clean and ships nothing.
assertion =
!(lib.elem config.services.journald.storage [
"volatile"
"none"
]);
message = ''
This container forwards its own journal to the swarm collector, but
services.journald.storage is "${config.services.journald.storage}".
The forwarder reads /var/log/journal, which journald only writes when
it stores persistently.
'';
}
];
services.opentelemetry-collector = {
enable = true;
# `journald` is a contrib receiver.
package = pkgs.opentelemetry-collector-contrib;
# A parser, not a wiring check: it accepts an extension that is never
# started.
validateConfigFile = true;
settings = {
# The receiver's read cursor, so a restart resumes rather than skipping
# what was written while it was down. `create_directory`, because
# `otelcol validate` runs in the build sandbox before systemd's
# `StateDirectory=` exists.
extensions.file_storage = {
directory = "/var/lib/opentelemetry-collector";
create_directory = true;
};
receivers.journald = {
# ⚠️ Stated: the receiver's default is the runtime journal under
# /run/log/journal, which is empty when journald stores persistently.
directory = "/var/log/journal";
# Without `--merge`, `journalctl --follow` over a journal with no
# entry for the current boot crash-loops the receiver.
merge = true;
storage = "file_storage";
};
processors.resource.attributes = [
{
key = "service.name";
value = cfg.serviceName;
action = "upsert";
}
];
exporters.otlphttp = {
endpoint = cfg.endpoint;
# The receiver accepts an OIDC token and nothing else.
auth.authenticator = authName;
};
extensions.${authName} = {
client_id = cfg.clientId;
# A path systemd resolves at runtime, so neither the secret nor the
# credentials directory is rendered into the nix store.
client_secret_file = "\${env:CREDENTIALS_DIRECTORY}/${credentialId}";
token_url = cfg.tokenUrl;
# ⚠️ Requested, not merely registered: a token minted without asking
# carries `aud: []` and the receiver refuses it.
endpoint_params.audience = cfg.clientId;
};
service.pipelines.logs = {
receivers = [ "journald" ];
processors = [ "resource" ];
exporters = [ "otlphttp" ];
};
# ⚠️ `readers`, not `metrics.address`: this collector version rejects
# `address` at startup.
service.telemetry.metrics.readers = [
{
pull.exporter.prometheus = {
host = "127.0.0.1";
port = cfg.telemetryPort;
};
}
];
# An extension not listed here is inert: the receiver's storage would
# name a component that never starts, and the exporter would push
# unauthenticated.
service.extensions = [
"file_storage"
authName
];
};
};
# `Restart` is nixpkgs' `"always"`; a second definition conflicts. Only
# `RestartSec` is ours: the 100ms default burns the start-rate limit while
# the secret is still on its way, leaving the unit failed instead of
# retrying.
systemd.services.opentelemetry-collector.serviceConfig = {
LoadCredential = [ "${credentialId}:${secretFile}" ];
RestartSec = 15;
};
# Reads the client secret with the container's own store identity. A store
# or authelia that is not up yet is a failed run that retries
# (../host-modules/lib/store-retry.nix), and a collector that restarts until the file lands.
systemd.services.${fetchUnit} = {
description = "fetch this container's journal forwarder OIDC client secret from the swarm secret store";
wantedBy = [ "multi-user.target" ];
before = [ "opentelemetry-collector.service" ];
path = [
cfg.store.package
pkgs.coreutils
pkgs.systemd
];
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
# No `RemainAfterExit`: the timer below starts this again, and starting
# an active unit is a no-op.
Type = "oneshot";
SyslogIdentifier = fetchUnit;
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = cfg.store.addr;
BAO_CLIENT_CERT = cfg.store.clientCertFile;
BAO_CLIENT_KEY = cfg.store.clientKeyFile;
}
// lib.optionalAttrs (cfg.store.caFile != null) {
BAO_CACERT = cfg.store.caFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
${refreshConsumer}
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, which
# needs a `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to the swarm secret store with this container's certificate." >&2
cat "$err" >&2
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg cfg.store.secretPath} 2>"$err")"; then
echo "the store did not return ${cfg.store.secretPath}: the forwarder has no client secret yet." >&2
cat "$err" >&2
exit 1
fi
if [ -z "$secret" ]; then
echo "the store returned an empty ${cfg.store.secretPath}." >&2
exit 1
fi
# `printf` inside `atomic_write_secret` is bash's own, so the value
# never becomes an argument in /proc.
install -d -m 0700 ${lib.escapeShellArg secretDir}
if secret_differs ${lib.escapeShellArg secretFile} "$secret"; then
atomic_write_secret 0400 root:root ${lib.escapeShellArg secretFile} "$secret"
fi
# `LoadCredential` copies the file at start only: a rotated secret
# reaches the collector by restarting it.
refresh_unit opentelemetry-collector.service ${lib.escapeShellArg secretFile}
'';
};
systemd.timers.${fetchUnit} =
storeRefreshTimer "re-read this container's journal forwarder OIDC client secret from the swarm secret store";
};
}

8
nix/host-modules/glue-nats-oidc-client.nix
View file

@ -39,6 +39,14 @@ in
# responder calls, works either way, so the token endpoint's # responder calls, works either way, so the token endpoint's
# `unauthorized_client` is the only place a wrong kind shows. # `unauthorized_client` is the only place a wrong kind shows.
kind = "machine"; kind = "machine";
# The queue container's journal forwarder mints its token as this
# client, for the swarm collector's `oidc/swarm-nats` receiver, which
# accepts this audience alone (`serviceProducers` in
# ./swarm-otel-service.nix).
audience = [ natsCfg.clientId ];
# That receiver verifies the token offline against `/jwks.json`;
# authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
} }
]; ];
}; };

18
nix/host-modules/swarm-nats-service.nix
View file

@ -107,6 +107,24 @@ in
''; '';
}; };
otel.telemetryPort = lib.mkOption {
type = lib.types.port;
default = 8891;
description = ''
Loopback port the journal forwarder inside the queue's container
serves its own metrics on (refused and dropped records, exporter
failures).
⚠️ Not 8888, 8889 or 8890: the hive collector, the swarm collector
and the store's forwarder bind those
({option}`services.hyperhive.otel.telemetryPort`,
{option}`services.hyperhive.swarm.otel.telemetryPort`,
{option}`services.hyperhive.swarm.bao.otel.telemetryPort`), and this
container shares the host netns with them when co-located. The second
collector to bind a port dies with `bind: address already in use`.
'';
};
clientId = lib.mkOption { clientId = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "swarm-nats"; default = "swarm-nats";

88
nix/host-modules/swarm-nats.nix
View file

@ -25,6 +25,16 @@ let
machine = "swarm-nats"; machine = "swarm-nats";
# The container's journal forwarder pushes to its own receiver on the swarm
# collector, the route `serviceProducers.swarm-nats` in
# ./swarm-otel-service.nix declares, by the collector's gateway name. The
# port suffix is elided on 443, as in ./swarm-bao.nix.
swarmOtelCfg = config.services.hyperhive.swarm.otel;
gatewayPortSuffix = lib.optionalString (
config.services.hyperhive.gateway.httpsPort != 443
) ":${toString config.services.hyperhive.gateway.httpsPort}";
forwarderEndpoint = "https://${swarmOtelCfg.domain}${gatewayPortSuffix}/${machine}";
tlsCfg = config.services.hyperhive.deploy.hive-controller.tls; tlsCfg = config.services.hyperhive.deploy.hive-controller.tls;
gatewayCfg = config.services.hyperhive.gateway; gatewayCfg = config.services.hyperhive.gateway;
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
@ -38,7 +48,12 @@ let
caBundleModule = caTrust.trustBundle { caBundleModule = caTrust.trustBundle {
inherit pkgs; inherit pkgs;
name = machine; name = machine;
consumers = [ "swarm-nats-auth" ]; consumers = [
"swarm-nats-auth"
# Go: `SSL_CERT_FILE` replaces its trust store, for the swarm
# collector's and authelia's names.
"opentelemetry-collector"
];
}; };
# Where the responder's credentials live *inside* the container, and the # Where the responder's credentials live *inside* the container, and the
# host path that resolves to. Two names for one location, because the # host path that resolves to. Two names for one location, because the
@ -575,6 +590,40 @@ in
both options. both options.
''; '';
} }
{
# The forwarder's route and the receiver behind it are two
# declarations of one fact; a mismatch is a 404 or a 401 on an export
# that looks healthy.
assertion = (swarmOtelCfg.serviceProducers.${machine}.clientId or null) == cfg.clientId;
message = ''
services.hyperhive.swarm.otel.serviceProducers has no entry
"${machine}" with clientId "${cfg.clientId}", so the swarm
collector has no receiver for the queue container's journal
forwarder. Declare it in swarm-otel-service.nix.
'';
}
{
assertion =
let
ports = [
cfg.port
cfg.monitorPort
cfg.metricsPort
cfg.otel.telemetryPort
config.services.hyperhive.otel.telemetryPort
swarmOtelCfg.telemetryPort
config.services.hyperhive.swarm.bao.otel.telemetryPort
];
in
lib.length (lib.unique ports) == lib.length ports;
message = ''
services.hyperhive.swarm.nats.otel.telemetryPort
(${toString cfg.otel.telemetryPort}) collides with another port in
the host netns: the queue's own ports, or the telemetry port of the
hive collector, the swarm collector or the store's forwarder. Pick
a free port.
'';
}
]; ];
# Open the client port on the bridge, so agent containers can reach # Open the client port on the bridge, so agent containers can reach
@ -601,9 +650,8 @@ in
containers.swarm-nats = { containers.swarm-nats = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes # No `--link-journal=host`: the journal stays in the guest, where the
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = [ "--link-journal=host" ];
inherit privateNetwork; inherit privateNetwork;
# The public trust bundle (empty when the gateway is not self-signed) # The public trust bundle (empty when the gateway is not self-signed)
# and the queue's own TLS leaf, both read-only. # and the queue's own TLS leaf, both read-only.
@ -618,19 +666,49 @@ in
{ {
imports = [ imports = [
../container-modules/swarm-container.nix ../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
(import ../container-modules/swarm-container-resolver.nix { (import ../container-modules/swarm-container-resolver.nix {
inherit (networkCfg) bridgeIp; inherit (networkCfg) bridgeIp;
# The responder introspects authelia BY NAME on every auth # The responder introspects authelia BY NAME on every auth
# request, and a responder that cannot resolve it denies # request, and a responder that cannot resolve it denies
# every client — so it must not start before the resolver # every client — so it must not start before the resolver
# file exists. # file exists.
dnsConsumers = [ "swarm-nats-auth.service" ]; #
# The forwarder and its secret fetch dial the swarm collector,
# authelia and the store by name too.
dnsConsumers = [
"swarm-nats-auth.service"
"opentelemetry-collector.service"
"swarm-journal-forwarder-oidc.service"
];
}) })
caBundleModule caBundleModule
]; ];
services.hyperhive.swarmContainer = { inherit privateNetwork; }; services.hyperhive.swarmContainer = { inherit privateNetwork; };
# Authenticated as the queue's own OIDC client, whose secret the
# responder's store identity already reads: one identity for the
# whole container.
services.hyperhive.swarmJournalForwarder = {
enable = true;
# The scrape job's name, so the queue's logs and metrics carry one
# `service.name`.
serviceName = "nats";
endpoint = forwarderEndpoint;
inherit (cfg) clientId;
tokenUrl = "${toString autheliaUrl}/api/oidc/token";
telemetryPort = cfg.otel.telemetryPort;
store = {
package = baoDeploy.package;
addr = "https://${baoCfg.domain}:${toString baoCfg.port}";
secretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
clientCertFile = inContainer "bao-client.pem";
clientKeyFile = inContainer "bao-client-key.pem";
caFile = if authStoreFiles ? "bao-ca.pem" then inContainer "bao-ca.pem" else null;
};
};
services.nats = { services.nats = {
enable = true; enable = true;

56
nix/host-modules/swarm-otel-service.nix
View file

@ -166,6 +166,48 @@ in
''; '';
}; };
serviceProducers = lib.mkOption {
internal = true;
type = lib.types.attrsOf (
lib.types.submodule {
options = {
clientId = lib.mkOption {
type = lib.types.str;
description = ''
The producer's OAuth2 client id: the one audience its
receiver's `oidc/<name>` authenticator accepts, and the one
the forwarder requests.
'';
};
port = lib.mkOption {
type = lib.types.port;
description = ''
Loopback port of the producer's receiver on the collector's
host. In the port-collision assertion with every other
receiver there.
'';
};
};
}
);
default = { };
description = ''
Swarm services whose own container forwards to this collector, keyed
by the route name: each gets a receiver `otlp/<name>`, an
authenticator `oidc/<name>` and a gateway location
`https://''${domain}/<name>/`, and its records land in the swarm
tier's pipelines with no `hive` label.
A receiver per producer because an `oidc` authenticator checks one
audience, and each service mints its token as its own client.
Declared below for every host, under no `enable`: the collector's
host renders the receiver whether or not the service runs there.
Every name must contain a fragment of
`nix/reserved-hive-fragments.nix`, so no hive can own its components.
'';
};
storeProducerName = lib.mkOption { storeProducerName = lib.mkOption {
type = lib.types.str; type = lib.types.str;
readOnly = true; readOnly = true;
@ -346,6 +388,20 @@ in
}; };
}; };
config.services.hyperhive.swarm.otel.serviceProducers = {
# The forwarder in ./swarm-bao.nix.
${cfg.storeProducerName} = {
clientId = baoCfg.otel.clientId;
port = cfg.storeProducerPort;
};
# The forwarder in ./swarm-nats.nix, which addresses this route by the
# queue container's name.
swarm-nats = {
clientId = hyperhiveCfg.swarm.nats.clientId;
port = lib.mkDefault 4392;
};
};
# Not under any `enable`: see `publishedScrapeTargets` for why every host # Not under any `enable`: see `publishedScrapeTargets` for why every host
# declares every entry. # declares every entry.
# #

145
nix/host-modules/swarm-otel.nix
View file

@ -41,21 +41,21 @@ let
# both sites) is the only way the two can't drift apart. # both sites) is the only way the two can't drift apart.
swarmTierName = cfg.producerName; swarmTierName = cfg.producerName;
# The SECOND swarm-tier producer: the collector inside the secret store's # The other swarm-tier producers: the forwarder inside each swarm service's
# container (`swarm-bao.nix`). It gets an owner of its own rather than # own container (`serviceProducers` in ./swarm-otel-service.nix). Each gets
# pushing into `swarmTierName`'s route, because the audience that route # an owner of its own rather than pushing into `swarmTierName`'s route,
# checks is `swarm-controller`'s own client id — one identity per principal, # because the audience that route checks is `swarm-controller`'s own client
# so a second principal brings its own client, its own audience and # id — one identity per principal, so each principal brings its own client,
# therefore its own authenticator and receiver. # its own audience and therefore its own authenticator and receiver.
# #
# Named after the store's container, so `otlp/swarm-bao` reads as the thing # Named after the service's container, so `otlp/swarm-bao` reads as the
# that pushes into it. No `reserved-names.nix` entry is needed for it the # thing that pushes into it. No `reserved-names.nix` entry is needed the way
# way `swarm` has one: the name CONTAINS `swarm`, which # `swarm` has one: each name CONTAINS `swarm`, which
# `nix/reserved-hive-fragments.nix` forbids as a substring of any hive name, # `nix/reserved-hive-fragments.nix` forbids as a substring of any hive name,
# so no hive can ever own these components. The assertion below checks that # so no hive can ever own these components. The assertion below checks that
# fragment is still listed rather than trusting it. # fragment is still listed rather than trusting it.
storeProducerName = baoCfg.machine; serviceProducers = cfg.serviceProducers;
storeClientId = baoCfg.otel.clientId; serviceProducerNames = lib.attrNames serviceProducers;
reservedHiveFragments = import ../reserved-hive-fragments.nix; reservedHiveFragments = import ../reserved-hive-fragments.nix;
# Every `<owner>` no hive may take. Read from `nix/reserved-names.nix`, the # Every `<owner>` no hive may take. Read from `nix/reserved-names.nix`, the
@ -206,16 +206,16 @@ let
# `--link-journal=try-guest`, which puts the journal inside the container and # `--link-journal=try-guest`, which puts the journal inside the container and
# leaves the host with a symlink into that container's transient root: a # leaves the host with a symlink into that container's transient root: a
# reader here cannot follow it, and it dangles the moment the container # reader here cannot follow it, and it dangles the moment the container
# stops. Every container block except `swarm-bao` therefore sets # stops. Every container block except `swarm-bao` and `swarm-nats` therefore sets
# `extraFlags = [ "--link-journal=host" ]`, which the invocation expands # `extraFlags = [ "--link-journal=host" ]`, which the invocation expands
# AFTER the hardcoded flag, so the files land here under their own # AFTER the hardcoded flag, so the files land here under their own
# machine-id subdirectory and are bind-mounted into the guest instead. # machine-id subdirectory and are bind-mounted into the guest instead.
# #
# ⚠️ Drop that flag from one of THOSE containers and this receiver silently # ⚠️ Drop that flag from one of THOSE containers and this receiver silently
# stops seeing it — no error, just a unit that never appears in the store. # stops seeing it — no error, just a unit that never appears in the store.
# `swarm-bao` is the one exception on purpose: its journal stays inside the # `swarm-bao` and `swarm-nats` are the exceptions on purpose: their journals
# container for its own collector instead (see `swarm-bao.nix`), so it was # stay inside the container for its own collector instead (see
# never one of the units this receiver could see either. # `serviceProducers`), so this receiver never sees their units.
hostJournalDir = "/var/log/journal"; hostJournalDir = "/var/log/journal";
# Each store's OTLP route, by domain. One binding because the same string is # Each store's OTLP route, by domain. One binding because the same string is
@ -380,14 +380,16 @@ in
# request reaches the receiver, which knows nothing about the # request reaches the receiver, which knows nothing about the
# path it was found at. # path it was found at.
"/${swarmTierName}/".proxyPass = "http://127.0.0.1:${toString cfg.producerPort}/"; "/${swarmTierName}/".proxyPass = "http://127.0.0.1:${toString cfg.producerPort}/";
# The secret store's forwarder, on its own route for its own
# audience — same trailing slashes, same reason.
"/${storeProducerName}/".proxyPass = "http://127.0.0.1:${toString cfg.storeProducerPort}/";
# There is no swarm-wide inbox, and a closed door is the honest # There is no swarm-wide inbox, and a closed door is the honest
# description of that. Every other route into this collector # description of that. Every other route into this collector
# belongs to exactly one hive or the swarm-tier producer above. # belongs to exactly one hive or one swarm-tier producer.
"/".return = "404"; "/".return = "404";
}; }
# Each service forwarder, on its own route for its own audience —
# same trailing slashes, same reason.
// lib.mapAttrs' (
name: p: lib.nameValuePair "/${name}/" { proxyPass = "http://127.0.0.1:${toString p.port}/"; }
) serviceProducers;
}; };
# The identities this tier authenticates against. Stated rather than # The identities this tier authenticates against. Stated rather than
@ -673,21 +675,21 @@ in
''; '';
} }
{ {
# The store forwarder's components are named after a container whose # The service forwarders' components are named after containers
# name merely CONTAINS the reserved word, so what protects them is # whose names merely CONTAIN the reserved word, so what protects them
# the substring list rather than the equality one. Same reason the # is the substring list rather than the equality one. Same reason the
# assertion above exists: the list lives in another file, and a guard # assertion above exists: the list lives in another file, and a guard
# whose subject can be edited elsewhere has to assert its own case is # whose subject can be edited elsewhere has to assert its own case is
# still covered. # still covered.
assertion = lib.any (f: lib.hasInfix f storeProducerName) reservedHiveFragments; assertion = lib.all (n: lib.any (f: lib.hasInfix f n) reservedHiveFragments) serviceProducerNames;
message = '' message = ''
nix/reserved-hive-fragments.nix forbids no substring of nix/reserved-hive-fragments.nix forbids no substring of one of
'${storeProducerName}', which the swarm collector needs kept out of ${lib.concatStringsSep ", " serviceProducerNames}, which the swarm
the hive namespace: it names components `<kind>/<owner>`, so a hive collector needs kept out of the hive namespace: it names components
called '${storeProducerName}' would replace the secret store `<kind>/<owner>`, so a hive with that name would replace a service
forwarder's own receiver and pipeline entry. forwarder's own receiver and pipeline entry.
Put the fragment back, or give the store's container a name one of Put the fragment back, or give the service's route a name one of
them covers. them covers.
''; '';
} }
@ -776,9 +778,9 @@ in
others = [ others = [
cfg.telemetryPort cfg.telemetryPort
cfg.producerPort cfg.producerPort
cfg.storeProducerPort
otelCfg.collector.port otelCfg.collector.port
]; ]
++ map (p: p.port) (lib.attrValues serviceProducers);
all = derived ++ others; all = derived ++ others;
in in
lib.length (lib.unique all) == lib.length all; lib.length (lib.unique all) == lib.length all;
@ -922,15 +924,18 @@ in
auth.authenticator = "oidc/${swarmTierName}"; auth.authenticator = "oidc/${swarmTierName}";
}; };
} }
# The secret store's forwarder, which is a swarm-tier # Each service forwarder, a swarm-tier producer with a
# producer with a principal of its own — a receiver per # principal of its own — a receiver per audience, because an
# audience, because an `oidc` authenticator checks one. # `oidc` authenticator checks one.
// { // lib.mapAttrs' (
"otlp/${storeProducerName}".protocols.http = { name: p:
endpoint = "127.0.0.1:${toString cfg.storeProducerPort}"; lib.nameValuePair "otlp/${name}" {
auth.authenticator = "oidc/${storeProducerName}"; protocols.http = {
}; endpoint = "127.0.0.1:${toString p.port}";
} auth.authenticator = "oidc/${name}";
};
}
) serviceProducers
# MERGED with the per-hive receivers, never assigned over # MERGED with the per-hive receivers, never assigned over
# them. A plain assignment here would drop every hive's # them. A plain assignment here would drop every hive's
# receiver and still render a valid config that starts # receiver and still render a valid config that starts
@ -1128,13 +1133,10 @@ in
# same reasoning as its receiver above: `otlp/${swarmTierName}` # same reasoning as its receiver above: `otlp/${swarmTierName}`
# always exists, so the authenticator it names must too, or an # always exists, so the authenticator it names must too, or an
# extension-not-listed startup failure follows every deploy. # extension-not-listed startup failure follows every deploy.
++ [ ++ [ "oidc/${swarmTierName}" ]
"oidc/${swarmTierName}" # Unconditional for the same reason: a service forwarder's
# Unconditional for the same reason, and one level sharper: # receiver exists whether or not the service runs on this host.
# the store runs in every swarm, so its forwarder's ++ map (n: "oidc/${n}") serviceProducerNames
# receiver is as permanent as the swarm tier's own.
"oidc/${storeProducerName}"
]
# The push side's authenticators. Same rule as above: one an # The push side's authenticators. Same rule as above: one an
# exporter names but this list omits is INERT — the collector # exporter names but this list omits is INERT — the collector
# starts clean and pushes unauthenticated. # starts clean and pushes unauthenticated.
@ -1178,17 +1180,16 @@ in
"metrics/${swarmTierName}" = { "metrics/${swarmTierName}" = {
receivers = [ receivers = [
"otlp/${swarmTierName}" "otlp/${swarmTierName}"
# The store forwarder's receiver joins the swarm
# tier's pipeline rather than bringing one of its own:
# the stamp it needs is `resource/${swarmTierName}`'s
# exact stamp — the swarm's name and deliberately no
# `hive` — and a second pipeline would be that same
# processor under a second name to keep in step. What
# a producer is separated FOR is the credential it
# proves, and that is the receiver's job, not the
# pipeline's.
"otlp/${storeProducerName}"
] ]
# The service forwarders' receivers join the swarm
# tier's pipeline rather than bringing their own: the
# stamp they need is `resource/${swarmTierName}`'s exact
# stamp — the swarm's name and deliberately no `hive` —
# and a second pipeline would be that same processor
# under a second name to keep in step. What a producer is
# separated FOR is the credential it proves, and that is
# the receiver's job, not the pipeline's.
++ map (n: "otlp/${n}") serviceProducerNames
++ lib.optional (cfg.scrapeTargets != { } || cfg.publishedScrapeTargets != { }) "prometheus"; ++ lib.optional (cfg.scrapeTargets != { } || cfg.publishedScrapeTargets != { }) "prometheus";
processors = [ "resource/${swarmTierName}" ]; processors = [ "resource/${swarmTierName}" ];
exporters = exporterNames; exporters = exporterNames;
@ -1227,14 +1228,11 @@ in
# `hive` value to put on them. # `hive` value to put on them.
// lib.optionalAttrs collectLogs { // lib.optionalAttrs collectLogs {
"logs/${swarmTierName}" = { "logs/${swarmTierName}" = {
# This host's own journal, and the store container's, # This host's own journal, and each service container's,
# which arrives over the receiver above rather than off # which arrives over its forwarder's receiver above
# a disk — the store's forwarder reads a journal no # rather than off a disk — each forwarder reads a journal
# reader on another host can see. # no reader on another host can see.
receivers = [ receivers = [ "journald" ] ++ map (n: "otlp/${n}") serviceProducerNames;
"journald"
"otlp/${storeProducerName}"
];
processors = [ processors = [
"filter/exclude-user-sessions" "filter/exclude-user-sessions"
"resource/${swarmTierName}" "resource/${swarmTierName}"
@ -1323,18 +1321,19 @@ in
audience = config.services.hyperhive.swarm.controller.queueClientId; audience = config.services.hyperhive.swarm.controller.queueClientId;
}; };
} }
# The secret store forwarder's own authenticator. Same shape # Each service forwarder's own authenticator. Same shape
# again, and the audience is that principal's own client id # again, and the audience is that principal's own client id
# — the self-referential form `swarm-authelia.nix` registers # — the self-referential form `swarm-authelia.nix` registers
# a hive client under and `swarm-controller` uses above, # a hive client under and `swarm-controller` uses above,
# read from the option `glue-swarm-bao-otel-oidc-client.nix` # read from the option the client's registration reads so
# registers so the two ends cannot spell it differently. # the two ends cannot spell it differently.
// { // lib.mapAttrs' (
"oidc/${storeProducerName}" = { name: p:
lib.nameValuePair "oidc/${name}" {
issuer_url = hyperhiveCfg.swarm.authelia.url; issuer_url = hyperhiveCfg.swarm.authelia.url;
audience = storeClientId; audience = p.clientId;
}; }
} ) serviceProducers
# The push side. Opposite direction to every `oidc/*` above — # The push side. Opposite direction to every `oidc/*` above —
# those VALIDATE a token arriving; these OBTAIN one to send. # those VALIDATE a token arriving; these OBTAIN one to send.
# Hence `oauth2client` rather than `oidc`, and hence a client # Hence `oauth2client` rather than `oidc`, and hence a client

73
nix/module-eval/nats-authelia.nix
View file

@ -560,6 +560,79 @@ let
in in
!(lib.hasInfix "127.0.0.1" url) && !(lib.hasInfix "localhost" url) && !(lib.hasInfix bridgeIp url); !(lib.hasInfix "127.0.0.1" url) && !(lib.hasInfix "localhost" url) && !(lib.hasInfix bridgeIp url);
} }
{
# The queue container's own journal forwarder, on the store's host and
# on a queue host with no store and no swarm collector: it exports to
# the swarm collector's `swarm-nats` route by name, authenticated as
# the queue's client, and its secret is read with the responder's
# store identity inside the container.
name = "the queue container forwards its journal to its own route on the swarm collector";
ok =
lib.all
(
h:
let
c = h.containers.swarm-nats.config;
s = c.services.opentelemetry-collector.settings;
otel = h.services.hyperhive.swarm.otel;
fetch = c.systemd.services.swarm-journal-forwarder-oidc;
clientId = h.services.hyperhive.swarm.nats.clientId;
in
c.services.opentelemetry-collector.enable
&& s.exporters.otlphttp.endpoint == "https://${otel.domain}/swarm-nats"
&& s.exporters.otlphttp.auth.authenticator == "oauth2client/swarm"
&& s.extensions."oauth2client/swarm".client_id == clientId
&& s.extensions."oauth2client/swarm".endpoint_params.audience == clientId
&& lib.elem "oauth2client/swarm" s.service.extensions
&& s.service.pipelines.logs.receivers == [ "journald" ]
&& lib.elem "oidc-client-secret:/run/swarm-journal-forwarder-oidc/client-secret" c.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential
&& fetch.environment.BAO_CLIENT_CERT == "/var/lib/swarm-nats-auth/bao-client.pem"
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/${clientId}/oidc/client" fetch.script
# The guest keeps its journal; a host-linked one is empty inside.
&& !(lib.elem "--link-journal=host" h.containers.swarm-nats.extraFlags)
)
[
natsOnStoreHost
natsWithStoreIdentity
];
}
{
# The read grant the fetch relies on: the responder's role, the identity
# the forwarder logs in with, reads the queue client's secret.
name = "the queue container's store identity may read the forwarder's client secret";
ok = lib.hasInfix "path \"secret/data/swarm/services/swarm-nats/oidc/client\" {" natsOnStoreHost.systemd.services.swarm-bao-nats-auth-policy.script;
}
{
# The far end: a receiver, authenticator, pipeline entry and gateway
# route of its own, checking the queue client as audience. Rendered on
# a collector host that runs no queue.
name = "the swarm collector admits the queue's forwarder on a receiver of its own";
ok =
let
h = shared.baoWithCollector;
otel = h.services.hyperhive.swarm.otel;
s = h.containers.swarm-otel.config.services.opentelemetry-collector.settings;
in
!h.services.hyperhive.deploy.nats.enable
&& s.receivers."otlp/swarm-nats".protocols.http.auth.authenticator == "oidc/swarm-nats"
&& s.extensions."oidc/swarm-nats".audience == h.services.hyperhive.swarm.nats.clientId
&& lib.elem "oidc/swarm-nats" s.service.extensions
&& lib.elem "otlp/swarm-nats" s.service.pipelines."logs/${otel.producerName}".receivers
&& h.services.nginx.virtualHosts.${otel.domain}.locations ? "/swarm-nats/"
# The store's forwarder keeps its own receiver beside it.
&& s.extensions."oidc/swarm-bao".audience == h.services.hyperhive.swarm.bao.otel.clientId;
}
{
# The token the forwarder mints must name its own client as audience
# and be a JWT, or the receiver refuses it.
name = "the queue's client is registered with its own audience and a signed access token";
ok =
let
clients = shared.autheliaOnly.services.hyperhive.swarm.authelia.oidc.clients;
c = lib.findFirst (c: c.id == "swarm-nats") null clients;
in
c != null && c.audience == [ "swarm-nats" ] && c.accessTokenSignedResponseAlg == "RS256";
}
]; ];
in in
runGroup "nats-authelia" cases runGroup "nats-authelia" cases