diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index e0475408..2295f432 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -41,33 +41,34 @@ neither is a renaming of the other. ## Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | -| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | -| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | -| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | -| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | -| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | -| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | -| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | -| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | -| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | -| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | -| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | +| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | +| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | +| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | +| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | +| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | +| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | +| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | +| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | +| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | +| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. **That's the test worth diff --git a/docs/swarm/services.md b/docs/swarm/services.md index 2be9643b..71bed5c2 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -288,6 +288,13 @@ gateway either way. | `swarm.otel.domain` | Only to rename it — the default already resolves correctly for every hive in the swarm. | | `swarm.otel.port` | Only if something else on the services host already claims `4319`. | +The `swarm-bao` and `swarm-nats` containers each run a collector of their own +that forwards the container's journal here, on a route and receiver per +service (`https:////`), authenticated as that +service's own OIDC client. Their records carry `service.name` (`bao`, +`nats`) and no `hive` label. The queue's forwarder is +`nix/container-modules/swarm-journal-forwarder.nix`. + **Both store exporters are unconditional**, and `deploy.victoriametrics.enable` doesn't gate them: that option says this host _runs_ the store, while the swarm has one either way, reached by its swarm name through the gateway. diff --git a/nix/container-modules/swarm-journal-forwarder.nix b/nix/container-modules/swarm-journal-forwarder.nix new file mode 100644 index 00000000..d43eaa52 --- /dev/null +++ b/nix/container-modules/swarm-journal-forwarder.nix @@ -0,0 +1,315 @@ +# A swarm service container's own journal forwarder: an otel collector that +# reads this container's journal and pushes it to the swarm collector by its +# gateway name, authenticated as the service's own OIDC client. +# +# Imported inside the container's own `config`. The host module that declares +# `containers.` sets the options below: the container's evaluation is +# nested inside the host's, and reading the host side back from in here +# recurses. +# +# The client secret is fetched inside the container, under the container's own +# store identity, by `swarm-journal-forwarder-oidc`. That needs the container to +# reach the store, authelia and the swarm collector by name: the container needs +# a network and a resolver. +# +# ⚠️ The host must NOT start this container with `--link-journal=host`. That +# bind-mounts a host directory over the guest's /var/log/journal which journald +# does not write into, and the receiver below reads nothing while the collector +# stays healthy. nixpkgs' default `--link-journal=try-guest` keeps the journal +# in the guest, where this reads it. +{ + config, + lib, + pkgs, + ... +}: +let + cfg = config.services.hyperhive.swarmJournalForwarder; + + atomicWriteSecret = import ../host-modules/lib/atomic-write-secret.nix { }; + refreshConsumer = import ../host-modules/lib/refresh-consumer.nix { }; + storeRetry = import ../host-modules/lib/store-retry.nix { }; + storeRefreshTimer = import ../host-modules/lib/store-refresh-timer.nix { }; + + fetchUnit = "swarm-journal-forwarder-oidc"; + # On the container's /run (tmpfs), so the secret never reaches persistent + # disk. The file is not created ahead of the fetch: `LoadCredential` refuses a + # missing source, so the collector stays down until the store answers and + # then resumes from its saved journal cursor. + secretDir = "/run/${fetchUnit}"; + secretFile = "${secretDir}/client-secret"; + + # The collector runs under `DynamicUser` and opens `client_secret_file` + # itself. `LoadCredential` reads the file as root and re-exposes it under + # `${CREDENTIALS_DIRECTORY}/`, whichever uid the unit got. + credentialId = "oidc-client-secret"; + authName = "oauth2client/swarm"; +in +{ + options.services.hyperhive.swarmJournalForwarder = { + enable = lib.mkEnableOption "this container's own journal forwarder to the swarm collector"; + + serviceName = lib.mkOption { + type = lib.types.str; + example = "nats"; + description = '' + `service.name` stamped on every record. The display identity a + dashboard selects on, matching the service's scrape job name. No + `hive` label: the swarm collector stamps `swarm` from the receiver + that accepted the push. + ''; + }; + + endpoint = lib.mkOption { + type = lib.types.str; + example = "https://otel.swarm.example/swarm-nats"; + description = '' + Base URL of this service's own receiver on the swarm collector: + `https://''${swarm.otel.domain}/`, a route + {option}`services.hyperhive.swarm.otel.serviceProducers` declares. + The exporter appends `/v1/logs`. + ''; + }; + + clientId = lib.mkOption { + type = lib.types.str; + description = '' + OAuth2 client the forwarder mints its token as, and the audience it + asks for. Must equal the `clientId` of this service's + {option}`services.hyperhive.swarm.otel.serviceProducers` entry, which + is the one audience that receiver accepts. + ''; + }; + + tokenUrl = lib.mkOption { + type = lib.types.str; + example = "https://auth.swarm.example/api/oidc/token"; + description = "Authelia's token endpoint."; + }; + + telemetryPort = lib.mkOption { + type = lib.types.port; + description = '' + Loopback port for the collector's own metrics (refused and dropped + records, exporter failures). Required because the container may share + the host netns, where the binary's default 8888 is the hive tier's. + ''; + }; + + store = { + package = lib.mkOption { + type = lib.types.package; + description = "The `bao` CLI the fetch unit logs in and reads with."; + }; + addr = lib.mkOption { + type = lib.types.str; + description = "The swarm secret store's address, by name."; + }; + secretPath = lib.mkOption { + type = lib.types.str; + example = "secret/swarm/services/swarm-nats/oidc/client"; + description = '' + KV path of the client secret. The container's store identity must + hold a read grant on it. + ''; + }; + clientCertFile = lib.mkOption { + type = lib.types.str; + description = '' + The container's store identity, as a path inside the container. A + path, never a value. + ''; + }; + clientKeyFile = lib.mkOption { + type = lib.types.str; + description = "Private key for `store.clientCertFile`. A path, never a value."; + }; + caFile = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + description = "CA the store's server certificate chains to; null means the system trust store."; + }; + }; + }; + + config = lib.mkIf cfg.enable { + assertions = [ + { + # With a volatile journal the receiver finds an empty directory and the + # collector starts clean and ships nothing. + assertion = + !(lib.elem config.services.journald.storage [ + "volatile" + "none" + ]); + message = '' + This container forwards its own journal to the swarm collector, but + services.journald.storage is "${config.services.journald.storage}". + The forwarder reads /var/log/journal, which journald only writes when + it stores persistently. + ''; + } + ]; + + services.opentelemetry-collector = { + enable = true; + # `journald` is a contrib receiver. + package = pkgs.opentelemetry-collector-contrib; + # A parser, not a wiring check: it accepts an extension that is never + # started. + validateConfigFile = true; + settings = { + # The receiver's read cursor, so a restart resumes rather than skipping + # what was written while it was down. `create_directory`, because + # `otelcol validate` runs in the build sandbox before systemd's + # `StateDirectory=` exists. + extensions.file_storage = { + directory = "/var/lib/opentelemetry-collector"; + create_directory = true; + }; + + receivers.journald = { + # ⚠️ Stated: the receiver's default is the runtime journal under + # /run/log/journal, which is empty when journald stores persistently. + directory = "/var/log/journal"; + # Without `--merge`, `journalctl --follow` over a journal with no + # entry for the current boot crash-loops the receiver. + merge = true; + storage = "file_storage"; + }; + + processors.resource.attributes = [ + { + key = "service.name"; + value = cfg.serviceName; + action = "upsert"; + } + ]; + + exporters.otlphttp = { + endpoint = cfg.endpoint; + # The receiver accepts an OIDC token and nothing else. + auth.authenticator = authName; + }; + + extensions.${authName} = { + client_id = cfg.clientId; + # A path systemd resolves at runtime, so neither the secret nor the + # credentials directory is rendered into the nix store. + client_secret_file = "\${env:CREDENTIALS_DIRECTORY}/${credentialId}"; + token_url = cfg.tokenUrl; + # ⚠️ Requested, not merely registered: a token minted without asking + # carries `aud: []` and the receiver refuses it. + endpoint_params.audience = cfg.clientId; + }; + + service.pipelines.logs = { + receivers = [ "journald" ]; + processors = [ "resource" ]; + exporters = [ "otlphttp" ]; + }; + + # ⚠️ `readers`, not `metrics.address`: this collector version rejects + # `address` at startup. + service.telemetry.metrics.readers = [ + { + pull.exporter.prometheus = { + host = "127.0.0.1"; + port = cfg.telemetryPort; + }; + } + ]; + + # An extension not listed here is inert: the receiver's storage would + # name a component that never starts, and the exporter would push + # unauthenticated. + service.extensions = [ + "file_storage" + authName + ]; + }; + }; + + # `Restart` is nixpkgs' `"always"`; a second definition conflicts. Only + # `RestartSec` is ours: the 100ms default burns the start-rate limit while + # the secret is still on its way, leaving the unit failed instead of + # retrying. + systemd.services.opentelemetry-collector.serviceConfig = { + LoadCredential = [ "${credentialId}:${secretFile}" ]; + RestartSec = 15; + }; + + # Reads the client secret with the container's own store identity. A store + # or authelia that is not up yet is a failed run that retries + # (../host-modules/lib/store-retry.nix), and a collector that restarts until the file lands. + systemd.services.${fetchUnit} = { + description = "fetch this container's journal forwarder OIDC client secret from the swarm secret store"; + wantedBy = [ "multi-user.target" ]; + before = [ "opentelemetry-collector.service" ]; + path = [ + cfg.store.package + pkgs.coreutils + pkgs.systemd + ]; + inherit (storeRetry) startLimitBurst startLimitIntervalSec; + serviceConfig = storeRetry.serviceConfig // { + # No `RemainAfterExit`: the timer below starts this again, and starting + # an active unit is a no-op. + Type = "oneshot"; + SyslogIdentifier = fetchUnit; + TimeoutStartSec = 30; + }; + environment = { + BAO_ADDR = cfg.store.addr; + BAO_CLIENT_CERT = cfg.store.clientCertFile; + BAO_CLIENT_KEY = cfg.store.clientKeyFile; + } + // lib.optionalAttrs (cfg.store.caFile != null) { + BAO_CACERT = cfg.store.caFile; + }; + script = '' + set -euo pipefail + + ${atomicWriteSecret} + ${refreshConsumer} + + err="$(mktemp)" + trap 'rm -f "$err"' EXIT + + # `-token-only` answers on stdout and skips the token helper, which + # needs a `sh` this unit's `path` does not carry. + if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then + echo "could not log in to the swarm secret store with this container's certificate." >&2 + cat "$err" >&2 + exit 1 + fi + export BAO_TOKEN + + if ! secret="$(bao kv get -field=value ${lib.escapeShellArg cfg.store.secretPath} 2>"$err")"; then + echo "the store did not return ${cfg.store.secretPath}: the forwarder has no client secret yet." >&2 + cat "$err" >&2 + exit 1 + fi + + if [ -z "$secret" ]; then + echo "the store returned an empty ${cfg.store.secretPath}." >&2 + exit 1 + fi + + # `printf` inside `atomic_write_secret` is bash's own, so the value + # never becomes an argument in /proc. + install -d -m 0700 ${lib.escapeShellArg secretDir} + if secret_differs ${lib.escapeShellArg secretFile} "$secret"; then + atomic_write_secret 0400 root:root ${lib.escapeShellArg secretFile} "$secret" + fi + + # `LoadCredential` copies the file at start only: a rotated secret + # reaches the collector by restarting it. + refresh_unit opentelemetry-collector.service ${lib.escapeShellArg secretFile} + ''; + }; + + systemd.timers.${fetchUnit} = + storeRefreshTimer "re-read this container's journal forwarder OIDC client secret from the swarm secret store"; + }; +} diff --git a/nix/host-modules/glue-nats-oidc-client.nix b/nix/host-modules/glue-nats-oidc-client.nix index a978093d..49b237f4 100644 --- a/nix/host-modules/glue-nats-oidc-client.nix +++ b/nix/host-modules/glue-nats-oidc-client.nix @@ -39,6 +39,14 @@ in # responder calls, works either way, so the token endpoint's # `unauthorized_client` is the only place a wrong kind shows. kind = "machine"; + # The queue container's journal forwarder mints its token as this + # client, for the swarm collector's `oidc/swarm-nats` receiver, which + # accepts this audience alone (`serviceProducers` in + # ./swarm-otel-service.nix). + audience = [ natsCfg.clientId ]; + # That receiver verifies the token offline against `/jwks.json`; + # authelia's default opaque handle fails there. + accessTokenSignedResponseAlg = "RS256"; } ]; }; diff --git a/nix/host-modules/swarm-nats-service.nix b/nix/host-modules/swarm-nats-service.nix index 6440f2bb..fe48c19b 100644 --- a/nix/host-modules/swarm-nats-service.nix +++ b/nix/host-modules/swarm-nats-service.nix @@ -107,6 +107,24 @@ in ''; }; + otel.telemetryPort = lib.mkOption { + type = lib.types.port; + default = 8891; + description = '' + Loopback port the journal forwarder inside the queue's container + serves its own metrics on (refused and dropped records, exporter + failures). + + ⚠️ Not 8888, 8889 or 8890: the hive collector, the swarm collector + and the store's forwarder bind those + ({option}`services.hyperhive.otel.telemetryPort`, + {option}`services.hyperhive.swarm.otel.telemetryPort`, + {option}`services.hyperhive.swarm.bao.otel.telemetryPort`), and this + container shares the host netns with them when co-located. The second + collector to bind a port dies with `bind: address already in use`. + ''; + }; + clientId = lib.mkOption { type = lib.types.str; default = "swarm-nats"; diff --git a/nix/host-modules/swarm-nats.nix b/nix/host-modules/swarm-nats.nix index aacc1fac..3859db5c 100644 --- a/nix/host-modules/swarm-nats.nix +++ b/nix/host-modules/swarm-nats.nix @@ -25,6 +25,16 @@ let machine = "swarm-nats"; + # The container's journal forwarder pushes to its own receiver on the swarm + # collector, the route `serviceProducers.swarm-nats` in + # ./swarm-otel-service.nix declares, by the collector's gateway name. The + # port suffix is elided on 443, as in ./swarm-bao.nix. + swarmOtelCfg = config.services.hyperhive.swarm.otel; + gatewayPortSuffix = lib.optionalString ( + config.services.hyperhive.gateway.httpsPort != 443 + ) ":${toString config.services.hyperhive.gateway.httpsPort}"; + forwarderEndpoint = "https://${swarmOtelCfg.domain}${gatewayPortSuffix}/${machine}"; + tlsCfg = config.services.hyperhive.deploy.hive-controller.tls; gatewayCfg = config.services.hyperhive.gateway; caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; }; @@ -38,7 +48,12 @@ let caBundleModule = caTrust.trustBundle { inherit pkgs; name = machine; - consumers = [ "swarm-nats-auth" ]; + consumers = [ + "swarm-nats-auth" + # Go: `SSL_CERT_FILE` replaces its trust store, for the swarm + # collector's and authelia's names. + "opentelemetry-collector" + ]; }; # Where the responder's credentials live *inside* the container, and the # host path that resolves to. Two names for one location, because the @@ -575,6 +590,40 @@ in both options. ''; } + { + # The forwarder's route and the receiver behind it are two + # declarations of one fact; a mismatch is a 404 or a 401 on an export + # that looks healthy. + assertion = (swarmOtelCfg.serviceProducers.${machine}.clientId or null) == cfg.clientId; + message = '' + services.hyperhive.swarm.otel.serviceProducers has no entry + "${machine}" with clientId "${cfg.clientId}", so the swarm + collector has no receiver for the queue container's journal + forwarder. Declare it in swarm-otel-service.nix. + ''; + } + { + assertion = + let + ports = [ + cfg.port + cfg.monitorPort + cfg.metricsPort + cfg.otel.telemetryPort + config.services.hyperhive.otel.telemetryPort + swarmOtelCfg.telemetryPort + config.services.hyperhive.swarm.bao.otel.telemetryPort + ]; + in + lib.length (lib.unique ports) == lib.length ports; + message = '' + services.hyperhive.swarm.nats.otel.telemetryPort + (${toString cfg.otel.telemetryPort}) collides with another port in + the host netns: the queue's own ports, or the telemetry port of the + hive collector, the swarm collector or the store's forwarder. Pick + a free port. + ''; + } ]; # Open the client port on the bridge, so agent containers can reach @@ -601,9 +650,8 @@ in containers.swarm-nats = { autoStart = true; ephemeral = false; - # Journal files on the host, not inside the container: nixpkgs hardcodes - # --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. - extraFlags = [ "--link-journal=host" ]; + # No `--link-journal=host`: the journal stays in the guest, where the + # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix). inherit privateNetwork; # The public trust bundle (empty when the gateway is not self-signed) # and the queue's own TLS leaf, both read-only. @@ -618,19 +666,49 @@ in { imports = [ ../container-modules/swarm-container.nix + ../container-modules/swarm-journal-forwarder.nix (import ../container-modules/swarm-container-resolver.nix { inherit (networkCfg) bridgeIp; # The responder introspects authelia BY NAME on every auth # request, and a responder that cannot resolve it denies # every client — so it must not start before the resolver # file exists. - dnsConsumers = [ "swarm-nats-auth.service" ]; + # + # The forwarder and its secret fetch dial the swarm collector, + # authelia and the store by name too. + dnsConsumers = [ + "swarm-nats-auth.service" + "opentelemetry-collector.service" + "swarm-journal-forwarder-oidc.service" + ]; }) caBundleModule ]; services.hyperhive.swarmContainer = { inherit privateNetwork; }; + # Authenticated as the queue's own OIDC client, whose secret the + # responder's store identity already reads: one identity for the + # whole container. + services.hyperhive.swarmJournalForwarder = { + enable = true; + # The scrape job's name, so the queue's logs and metrics carry one + # `service.name`. + serviceName = "nats"; + endpoint = forwarderEndpoint; + inherit (cfg) clientId; + tokenUrl = "${toString autheliaUrl}/api/oidc/token"; + telemetryPort = cfg.otel.telemetryPort; + store = { + package = baoDeploy.package; + addr = "https://${baoCfg.domain}:${toString baoCfg.port}"; + secretPath = "secret/swarm/services/${cfg.clientId}/oidc/client"; + clientCertFile = inContainer "bao-client.pem"; + clientKeyFile = inContainer "bao-client-key.pem"; + caFile = if authStoreFiles ? "bao-ca.pem" then inContainer "bao-ca.pem" else null; + }; + }; + services.nats = { enable = true; diff --git a/nix/host-modules/swarm-otel-service.nix b/nix/host-modules/swarm-otel-service.nix index 9b7edcf0..6f6dd8ac 100644 --- a/nix/host-modules/swarm-otel-service.nix +++ b/nix/host-modules/swarm-otel-service.nix @@ -166,6 +166,48 @@ in ''; }; + serviceProducers = lib.mkOption { + internal = true; + type = lib.types.attrsOf ( + lib.types.submodule { + options = { + clientId = lib.mkOption { + type = lib.types.str; + description = '' + The producer's OAuth2 client id: the one audience its + receiver's `oidc/` authenticator accepts, and the one + the forwarder requests. + ''; + }; + port = lib.mkOption { + type = lib.types.port; + description = '' + Loopback port of the producer's receiver on the collector's + host. In the port-collision assertion with every other + receiver there. + ''; + }; + }; + } + ); + default = { }; + description = '' + Swarm services whose own container forwards to this collector, keyed + by the route name: each gets a receiver `otlp/`, an + authenticator `oidc/` and a gateway location + `https://''${domain}//`, and its records land in the swarm + tier's pipelines with no `hive` label. + + A receiver per producer because an `oidc` authenticator checks one + audience, and each service mints its token as its own client. + + Declared below for every host, under no `enable`: the collector's + host renders the receiver whether or not the service runs there. + Every name must contain a fragment of + `nix/reserved-hive-fragments.nix`, so no hive can own its components. + ''; + }; + storeProducerName = lib.mkOption { type = lib.types.str; readOnly = true; @@ -346,6 +388,20 @@ in }; }; + config.services.hyperhive.swarm.otel.serviceProducers = { + # The forwarder in ./swarm-bao.nix. + ${cfg.storeProducerName} = { + clientId = baoCfg.otel.clientId; + port = cfg.storeProducerPort; + }; + # The forwarder in ./swarm-nats.nix, which addresses this route by the + # queue container's name. + swarm-nats = { + clientId = hyperhiveCfg.swarm.nats.clientId; + port = lib.mkDefault 4392; + }; + }; + # Not under any `enable`: see `publishedScrapeTargets` for why every host # declares every entry. # diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index b1cf1037..9e2f0f09 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -41,21 +41,21 @@ let # both sites) is the only way the two can't drift apart. swarmTierName = cfg.producerName; - # The SECOND swarm-tier producer: the collector inside the secret store's - # container (`swarm-bao.nix`). It gets an owner of its own rather than - # pushing into `swarmTierName`'s route, because the audience that route - # checks is `swarm-controller`'s own client id — one identity per principal, - # so a second principal brings its own client, its own audience and - # therefore its own authenticator and receiver. + # The other swarm-tier producers: the forwarder inside each swarm service's + # own container (`serviceProducers` in ./swarm-otel-service.nix). Each gets + # an owner of its own rather than pushing into `swarmTierName`'s route, + # because the audience that route checks is `swarm-controller`'s own client + # id — one identity per principal, so each principal brings its own client, + # its own audience and therefore its own authenticator and receiver. # - # Named after the store's container, so `otlp/swarm-bao` reads as the thing - # that pushes into it. No `reserved-names.nix` entry is needed for it the - # way `swarm` has one: the name CONTAINS `swarm`, which + # Named after the service's container, so `otlp/swarm-bao` reads as the + # thing that pushes into it. No `reserved-names.nix` entry is needed the way + # `swarm` has one: each name CONTAINS `swarm`, which # `nix/reserved-hive-fragments.nix` forbids as a substring of any hive name, # so no hive can ever own these components. The assertion below checks that # fragment is still listed rather than trusting it. - storeProducerName = baoCfg.machine; - storeClientId = baoCfg.otel.clientId; + serviceProducers = cfg.serviceProducers; + serviceProducerNames = lib.attrNames serviceProducers; reservedHiveFragments = import ../reserved-hive-fragments.nix; # Every `` no hive may take. Read from `nix/reserved-names.nix`, the @@ -206,16 +206,16 @@ let # `--link-journal=try-guest`, which puts the journal inside the container and # leaves the host with a symlink into that container's transient root: a # reader here cannot follow it, and it dangles the moment the container - # stops. Every container block except `swarm-bao` therefore sets + # stops. Every container block except `swarm-bao` and `swarm-nats` therefore sets # `extraFlags = [ "--link-journal=host" ]`, which the invocation expands # AFTER the hardcoded flag, so the files land here under their own # machine-id subdirectory and are bind-mounted into the guest instead. # # ⚠️ Drop that flag from one of THOSE containers and this receiver silently # stops seeing it — no error, just a unit that never appears in the store. - # `swarm-bao` is the one exception on purpose: its journal stays inside the - # container for its own collector instead (see `swarm-bao.nix`), so it was - # never one of the units this receiver could see either. + # `swarm-bao` and `swarm-nats` are the exceptions on purpose: their journals + # stay inside the container for its own collector instead (see + # `serviceProducers`), so this receiver never sees their units. hostJournalDir = "/var/log/journal"; # Each store's OTLP route, by domain. One binding because the same string is @@ -380,14 +380,16 @@ in # request reaches the receiver, which knows nothing about the # path it was found at. "/${swarmTierName}/".proxyPass = "http://127.0.0.1:${toString cfg.producerPort}/"; - # The secret store's forwarder, on its own route for its own - # audience — same trailing slashes, same reason. - "/${storeProducerName}/".proxyPass = "http://127.0.0.1:${toString cfg.storeProducerPort}/"; # There is no swarm-wide inbox, and a closed door is the honest # description of that. Every other route into this collector - # belongs to exactly one hive or the swarm-tier producer above. + # belongs to exactly one hive or one swarm-tier producer. "/".return = "404"; - }; + } + # Each service forwarder, on its own route for its own audience — + # same trailing slashes, same reason. + // lib.mapAttrs' ( + name: p: lib.nameValuePair "/${name}/" { proxyPass = "http://127.0.0.1:${toString p.port}/"; } + ) serviceProducers; }; # The identities this tier authenticates against. Stated rather than @@ -673,21 +675,21 @@ in ''; } { - # The store forwarder's components are named after a container whose - # name merely CONTAINS the reserved word, so what protects them is - # the substring list rather than the equality one. Same reason the + # The service forwarders' components are named after containers + # whose names merely CONTAIN the reserved word, so what protects them + # is the substring list rather than the equality one. Same reason the # assertion above exists: the list lives in another file, and a guard # whose subject can be edited elsewhere has to assert its own case is # still covered. - assertion = lib.any (f: lib.hasInfix f storeProducerName) reservedHiveFragments; + assertion = lib.all (n: lib.any (f: lib.hasInfix f n) reservedHiveFragments) serviceProducerNames; message = '' - nix/reserved-hive-fragments.nix forbids no substring of - '${storeProducerName}', which the swarm collector needs kept out of - the hive namespace: it names components `/`, so a hive - called '${storeProducerName}' would replace the secret store + nix/reserved-hive-fragments.nix forbids no substring of one of + ${lib.concatStringsSep ", " serviceProducerNames}, which the swarm + collector needs kept out of the hive namespace: it names components + `/`, so a hive with that name would replace a service forwarder's own receiver and pipeline entry. - Put the fragment back, or give the store's container a name one of + Put the fragment back, or give the service's route a name one of them covers. ''; } @@ -776,9 +778,9 @@ in others = [ cfg.telemetryPort cfg.producerPort - cfg.storeProducerPort otelCfg.collector.port - ]; + ] + ++ map (p: p.port) (lib.attrValues serviceProducers); all = derived ++ others; in lib.length (lib.unique all) == lib.length all; @@ -922,15 +924,18 @@ in auth.authenticator = "oidc/${swarmTierName}"; }; } - # The secret store's forwarder, which is a swarm-tier - # producer with a principal of its own — a receiver per - # audience, because an `oidc` authenticator checks one. - // { - "otlp/${storeProducerName}".protocols.http = { - endpoint = "127.0.0.1:${toString cfg.storeProducerPort}"; - auth.authenticator = "oidc/${storeProducerName}"; - }; - } + # Each service forwarder, a swarm-tier producer with a + # principal of its own — a receiver per audience, because an + # `oidc` authenticator checks one. + // lib.mapAttrs' ( + name: p: + lib.nameValuePair "otlp/${name}" { + protocols.http = { + endpoint = "127.0.0.1:${toString p.port}"; + auth.authenticator = "oidc/${name}"; + }; + } + ) serviceProducers # MERGED with the per-hive receivers, never assigned over # them. A plain assignment here would drop every hive's # receiver and still render a valid config that starts @@ -1128,13 +1133,10 @@ in # same reasoning as its receiver above: `otlp/${swarmTierName}` # always exists, so the authenticator it names must too, or an # extension-not-listed startup failure follows every deploy. - ++ [ - "oidc/${swarmTierName}" - # Unconditional for the same reason, and one level sharper: - # the store runs in every swarm, so its forwarder's - # receiver is as permanent as the swarm tier's own. - "oidc/${storeProducerName}" - ] + ++ [ "oidc/${swarmTierName}" ] + # Unconditional for the same reason: a service forwarder's + # receiver exists whether or not the service runs on this host. + ++ map (n: "oidc/${n}") serviceProducerNames # The push side's authenticators. Same rule as above: one an # exporter names but this list omits is INERT — the collector # starts clean and pushes unauthenticated. @@ -1178,17 +1180,16 @@ in "metrics/${swarmTierName}" = { receivers = [ "otlp/${swarmTierName}" - # The store forwarder's receiver joins the swarm - # tier's pipeline rather than bringing one of its own: - # the stamp it needs is `resource/${swarmTierName}`'s - # exact stamp — the swarm's name and deliberately no - # `hive` — and a second pipeline would be that same - # processor under a second name to keep in step. What - # a producer is separated FOR is the credential it - # proves, and that is the receiver's job, not the - # pipeline's. - "otlp/${storeProducerName}" ] + # The service forwarders' receivers join the swarm + # tier's pipeline rather than bringing their own: the + # stamp they need is `resource/${swarmTierName}`'s exact + # stamp — the swarm's name and deliberately no `hive` — + # and a second pipeline would be that same processor + # under a second name to keep in step. What a producer is + # separated FOR is the credential it proves, and that is + # the receiver's job, not the pipeline's. + ++ map (n: "otlp/${n}") serviceProducerNames ++ lib.optional (cfg.scrapeTargets != { } || cfg.publishedScrapeTargets != { }) "prometheus"; processors = [ "resource/${swarmTierName}" ]; exporters = exporterNames; @@ -1227,14 +1228,11 @@ in # `hive` value to put on them. // lib.optionalAttrs collectLogs { "logs/${swarmTierName}" = { - # This host's own journal, and the store container's, - # which arrives over the receiver above rather than off - # a disk — the store's forwarder reads a journal no - # reader on another host can see. - receivers = [ - "journald" - "otlp/${storeProducerName}" - ]; + # This host's own journal, and each service container's, + # which arrives over its forwarder's receiver above + # rather than off a disk — each forwarder reads a journal + # no reader on another host can see. + receivers = [ "journald" ] ++ map (n: "otlp/${n}") serviceProducerNames; processors = [ "filter/exclude-user-sessions" "resource/${swarmTierName}" @@ -1323,18 +1321,19 @@ in audience = config.services.hyperhive.swarm.controller.queueClientId; }; } - # The secret store forwarder's own authenticator. Same shape + # Each service forwarder's own authenticator. Same shape # again, and the audience is that principal's own client id # — the self-referential form `swarm-authelia.nix` registers # a hive client under and `swarm-controller` uses above, - # read from the option `glue-swarm-bao-otel-oidc-client.nix` - # registers so the two ends cannot spell it differently. - // { - "oidc/${storeProducerName}" = { + # read from the option the client's registration reads so + # the two ends cannot spell it differently. + // lib.mapAttrs' ( + name: p: + lib.nameValuePair "oidc/${name}" { issuer_url = hyperhiveCfg.swarm.authelia.url; - audience = storeClientId; - }; - } + audience = p.clientId; + } + ) serviceProducers # The push side. Opposite direction to every `oidc/*` above — # those VALIDATE a token arriving; these OBTAIN one to send. # Hence `oauth2client` rather than `oidc`, and hence a client diff --git a/nix/module-eval/nats-authelia.nix b/nix/module-eval/nats-authelia.nix index 496badbf..37b55033 100644 --- a/nix/module-eval/nats-authelia.nix +++ b/nix/module-eval/nats-authelia.nix @@ -560,6 +560,79 @@ let in !(lib.hasInfix "127.0.0.1" url) && !(lib.hasInfix "localhost" url) && !(lib.hasInfix bridgeIp url); } + { + # The queue container's own journal forwarder, on the store's host and + # on a queue host with no store and no swarm collector: it exports to + # the swarm collector's `swarm-nats` route by name, authenticated as + # the queue's client, and its secret is read with the responder's + # store identity inside the container. + name = "the queue container forwards its journal to its own route on the swarm collector"; + ok = + lib.all + ( + h: + let + c = h.containers.swarm-nats.config; + s = c.services.opentelemetry-collector.settings; + otel = h.services.hyperhive.swarm.otel; + fetch = c.systemd.services.swarm-journal-forwarder-oidc; + clientId = h.services.hyperhive.swarm.nats.clientId; + in + c.services.opentelemetry-collector.enable + && s.exporters.otlphttp.endpoint == "https://${otel.domain}/swarm-nats" + && s.exporters.otlphttp.auth.authenticator == "oauth2client/swarm" + && s.extensions."oauth2client/swarm".client_id == clientId + && s.extensions."oauth2client/swarm".endpoint_params.audience == clientId + && lib.elem "oauth2client/swarm" s.service.extensions + && s.service.pipelines.logs.receivers == [ "journald" ] + && lib.elem "oidc-client-secret:/run/swarm-journal-forwarder-oidc/client-secret" c.systemd.services.opentelemetry-collector.serviceConfig.LoadCredential + && fetch.environment.BAO_CLIENT_CERT == "/var/lib/swarm-nats-auth/bao-client.pem" + && lib.hasInfix "bao kv get -field=value secret/swarm/services/${clientId}/oidc/client" fetch.script + # The guest keeps its journal; a host-linked one is empty inside. + && !(lib.elem "--link-journal=host" h.containers.swarm-nats.extraFlags) + ) + [ + natsOnStoreHost + natsWithStoreIdentity + ]; + } + { + # The read grant the fetch relies on: the responder's role, the identity + # the forwarder logs in with, reads the queue client's secret. + name = "the queue container's store identity may read the forwarder's client secret"; + ok = lib.hasInfix "path \"secret/data/swarm/services/swarm-nats/oidc/client\" {" natsOnStoreHost.systemd.services.swarm-bao-nats-auth-policy.script; + } + { + # The far end: a receiver, authenticator, pipeline entry and gateway + # route of its own, checking the queue client as audience. Rendered on + # a collector host that runs no queue. + name = "the swarm collector admits the queue's forwarder on a receiver of its own"; + ok = + let + h = shared.baoWithCollector; + otel = h.services.hyperhive.swarm.otel; + s = h.containers.swarm-otel.config.services.opentelemetry-collector.settings; + in + !h.services.hyperhive.deploy.nats.enable + && s.receivers."otlp/swarm-nats".protocols.http.auth.authenticator == "oidc/swarm-nats" + && s.extensions."oidc/swarm-nats".audience == h.services.hyperhive.swarm.nats.clientId + && lib.elem "oidc/swarm-nats" s.service.extensions + && lib.elem "otlp/swarm-nats" s.service.pipelines."logs/${otel.producerName}".receivers + && h.services.nginx.virtualHosts.${otel.domain}.locations ? "/swarm-nats/" + # The store's forwarder keeps its own receiver beside it. + && s.extensions."oidc/swarm-bao".audience == h.services.hyperhive.swarm.bao.otel.clientId; + } + { + # The token the forwarder mints must name its own client as audience + # and be a JWT, or the receiver refuses it. + name = "the queue's client is registered with its own audience and a signed access token"; + ok = + let + clients = shared.autheliaOnly.services.hyperhive.swarm.authelia.oidc.clients; + c = lib.findFirst (c: c.id == "swarm-nats") null clients; + in + c != null && c.audience == [ "swarm-nats" ] && c.accessTokenSignedResponseAlg == "RS256"; + } ]; in runGroup "nats-authelia" cases