diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 7fb61edf..ec511e0f 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -1,4 +1,4 @@ -# Credentials: the target shape +, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API# Credentials: the target shape diff --git a/docs/swarm/secrets.md b/docs/swarm/secrets.md index 072605f9..f5cc487e 100644 --- a/docs/swarm/secrets.md +++ b/docs/swarm/secrets.md @@ -43,33 +43,33 @@ neither is a renaming of the other. ## Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | -| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | -| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | -| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | -| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | -| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. `swarm-metrics-auth` on the metrics store's host reads the stored value itself, under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | -| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | -| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | -| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | -| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | -| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | -| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | +| authelia session, JWT and storage-encryption keys | authelia's first-boot unit, in-container | `/var/lib/authelia-swarm/{session,jwt,storage-encryption}.key` | generated in place; nothing outside that container reads them | +| authelia OIDC HMAC key | same unit | `/var/lib/authelia-swarm/oidc-hmac.key` | same | +| authelia OIDC issuer key (RSA) | same unit | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store (below) | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | +| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | +| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | +| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grant on the bearer leaf, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | +| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | +| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | `/var/lib/forgejo/swarm-controller-token` inside the forge container, the minter's record that it has minted; `swarm-bao-forge-publisher` copies it into the swarm secret store at `swarm/controller/swarm-controller/forge/token` each time the forge container starts and each time the mint writes the file; the controller holds it in memory only | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | +| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | +| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which reloads nginx if it's active); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. **That's the test worth diff --git a/docs/swarm/services.md b/docs/swarm/services.md index 0ed83350..ab3eac61 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -1,3 +1,5 @@ +| `deploy.grafana.logsDatasourceUrl` | Only if you front VictoriaLogs with something else. It defaults to the log store's swarm name (`https://`), so Grafana and the store can run on different hosts. | + # Swarm-wide services Some things exist once per **swarm** rather than once per hive: the forge, @@ -247,6 +249,17 @@ swarm. It binds loopback and takes no credential of its own: the gateway vhost is the only way in, and the collector is the only intended writer. +**Who may query.** `/select/logsql/` takes one of three credentials. An +agent presents its `swarm-agent..` bearer, checked by +`swarm-logs-auth`. Grafana's datasource presents the same bearer as its +metrics datasource, which `swarm-bao-grafana-metrics-bearer` creates, and a +second `swarm-metrics-auth` instance (`swarm-logs-grafana-auth`) on the log +store's host checks it against the same store path; the gateway tells it from +the others by its shape, 64 hex digits. Anything else needs an operator's +authelia session. Because one bearer opens both read APIs, a rotation +(see [`secrets.md`](secrets.md)) restarts Grafana once, and the log store +accepts the new value within a minute. + The collector also scrapes this store's own metrics, at `https:///metrics`, so a collector on a host without the store still gets them. The gateway admits only the collector's diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 7c404943..92d19b07 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -1003,10 +1003,10 @@ let # of `adminPasswordStorePath` in ./swarm-grafana.nix. grafanaAdminLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/admin/password"; - # The bearer Grafana's metrics datasource presents at the metrics store. - # Grafana's leaf creates it, the store's responder reads it. The nix half of + # The bearer Grafana's datasources present at the metrics and log stores. + # Grafana's leaf creates it, the stores' responders read it. The nix half of # `metricsBearerStorePath` in ./swarm-grafana.nix and of `--store-path` in - # ./swarm-victoriametrics.nix. + # ./swarm-victoriametrics.nix and ./swarm-victorialogs.nix. grafanaMetricsBearerLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer"; createStanza = path: '' @@ -1015,14 +1015,17 @@ let } ''; - # The log store's agent auth responder, which checks the secret an agent - # presents against the one stored for it: the same `+` leaf as the queue - # responder's, and nothing else. + # The log store's auth responders: the agent one checks the secret an agent + # presents against the one stored for it (the same `+` leaf as the queue + # responder's), the other checks the bearer Grafana presents against + # Grafana's metrics bearer. Nothing else. logsAuthReaders = [ { name = "swarm-logs-auth"; cn = baoDeploy.logsAuthCommonName; - policyText = readStanza "${credentialMountPath}/data/swarm/agents/+/queue"; + policyText = + readStanza "${credentialMountPath}/data/swarm/agents/+/queue" + + readStanza "${credentialMountPath}/data/${grafanaMetricsBearerLeaf}"; } ]; diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index 979b3924..73c35e24 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -274,8 +274,10 @@ let # The bearer the metrics datasource presents at the metrics store's query # API, which `swarm-metrics-auth` on the store's host checks against the - # same store path. Minted once by `swarm-bao-grafana-metrics-bearer` below, - # under the same grant as the admin password. The file holds the whole + # same store path, and the logs datasource at the log store's query API, + # which a second `swarm-metrics-auth` instance in ./swarm-victorialogs.nix + # checks against it. Minted once by `swarm-bao-grafana-metrics-bearer` + # below, under the same grant as the admin password. The file holds the whole # header value, `Bearer `: nixpkgs accepts a `secureJsonData` entry # only when it is exactly one `$__file{}`. metricsBearerStorePath = "secret/swarm/services/${cfg.oidc.clientId}/metrics/bearer"; @@ -371,16 +373,20 @@ in logsDatasourceUrl = lib.mkOption { type = lib.types.str; - default = "http://127.0.0.1:${toString hyperhiveCfg.swarm.victorialogs.port}"; - defaultText = lib.literalExpression ''"http://127.0.0.1:''${toString services.hyperhive.swarm.victorialogs.port}"''; + default = "https://${hyperhiveCfg.swarm.victorialogs.domain}"; + defaultText = lib.literalExpression ''"https://''${services.hyperhive.swarm.victorialogs.domain}"''; description = '' Where the provisioned logs datasource points. Defaults to the log - store on this host, which binds loopback, so this default reaches - it only where Grafana and the store share a host. + store's swarm name, served by the gateway of whichever host runs the + store, so Grafana and the store need not share a host. Provisioned unconditionally, like the metrics datasource — the store being off is a deployment choice rather than a reason to withhold the connection, and an operator whose logs live elsewhere sets this. + + The datasource sends the same bearer as the metrics one, which the + store's `/select/logsql/` route checks. Whatever URL this names + receives that bearer. ''; }; @@ -1316,7 +1322,11 @@ in # not in the pipeline. jsonData = { inherit logLevelRules; + httpHeaderName1 = "Authorization"; }; + # The metrics datasource's bearer; the log store's vhost + # accepts it on `/select/logsql/`. + secureJsonData.httpHeaderValue1 = "$__file{${metricsBearerPath}}"; } ]; }; diff --git a/nix/host-modules/swarm-victorialogs.nix b/nix/host-modules/swarm-victorialogs.nix index 0d9a24f2..a59b6efb 100644 --- a/nix/host-modules/swarm-victorialogs.nix +++ b/nix/host-modules/swarm-victorialogs.nix @@ -68,6 +68,15 @@ let }; baoCfg = hyperhiveCfg.swarm.bao; + # Grafana's bearer, checked by a second `swarm-metrics-auth` instance. It + # reads the store with the same leaf as `swarm-logs-auth`, which is granted + # the one extra path below. The same leaf as `grafanaBearerStorePath` in + # ./swarm-victoriametrics.nix and `grafanaMetricsBearerLeaf` in ./swarm-bao.nix. + grafanaAuthUnit = "swarm-logs-grafana-auth"; + grafanaAuthSocket = "/run/${grafanaAuthUnit}/auth.sock"; + grafanaAuthCredential = id: "/run/credentials/${grafanaAuthUnit}.service/${id}"; + grafanaBearerStorePath = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/metrics/bearer"; + # Shared host netns, like every sibling swarm container: Grafana reaches # this at 127.0.0.1:. privateNetwork = false; @@ -210,6 +219,7 @@ in services.nginx.appendHttpConfig = '' map $http_authorization $swarm_logs_auth_target { "~^Bearer swarm-agent\." /__swarm_logs_agent_auth; + "~^Bearer [0-9a-f]{64}$" /__swarm_logs_grafana_auth; default /__hive_authelia; } limit_req_zone $binary_remote_addr zone=swarm_logs_agent_auth:1m rate=${deployCfg.victorialogs.authRate}; @@ -289,9 +299,9 @@ in # permissions are a later thing, and this location is where one # attaches when it exists. # - # Two checks: an agent's own `swarm-agent.` bearer goes to - # `swarm-logs-auth`, anything else to authelia (`appendHttpConfig` - # above picks). + # Three checks: an agent's own `swarm-agent.` bearer goes to + # `swarm-logs-auth`, Grafana's 64-hex bearer to `swarm-metrics-auth`, + # anything else to authelia (`appendHttpConfig` above picks). "^~ /select/logsql/" = { # No URI part, so the request path and its query string reach # the store as the caller sent them. @@ -345,6 +355,18 @@ in ''; }; + # Grafana's bearer, 64 hex digits (`openssl rand -hex 32`), checked by + # `swarm-metrics-auth`. The responder reads the store at most once per + # its cache TTL whatever the bearers, so no rate limit sits in front. + "= /__swarm_logs_grafana_auth" = { + proxyPass = "http://unix:${grafanaAuthSocket}:/"; + extraConfig = '' + internal; + proxy_pass_request_body off; + proxy_set_header Content-Length ""; + ''; + }; + # The subrequest itself — same target, same header set, same # reasoning as `swarm-ui.nix`'s own copy (measured against the # pinned authelia binary, not copied from an example). @@ -430,6 +452,57 @@ in }; }; + # Grafana's check on `/select/logsql/`: the metrics store's responder + # binary, pointed at the bearer Grafana's datasources share. + users.users.${grafanaAuthUnit} = { + isSystemUser = true; + group = grafanaAuthUnit; + description = "swarm log store auth_request responder for Grafana's bearer"; + }; + users.groups.${grafanaAuthUnit} = { }; + + systemd.services.${grafanaAuthUnit} = { + description = "swarm log store auth_request responder for Grafana's bearer"; + wantedBy = [ "multi-user.target" ]; + after = [ + "network.target" + "swarm-bao-pki.service" + ]; + serviceConfig = { + ExecStart = lib.concatStringsSep " " [ + "${deployCfg.victoriametrics.authPackage}/bin/swarm-metrics-auth" + "--socket ${grafanaAuthSocket}" + "--store-cert-role swarm-logs-auth" + "--store-path ${grafanaBearerStorePath}" + ]; + LoadCredential = lib.mapAttrsToList (id: path: "${id}:${path}") logsAuthStoreFiles; + # ⚠️ A static user, never `DynamicUser`: see the `swarm-logs-auth` unit above. + User = grafanaAuthUnit; + Group = grafanaAuthUnit; + RuntimeDirectory = grafanaAuthUnit; + RuntimeDirectoryMode = "0751"; + RuntimeDirectoryPreserve = "yes"; + Restart = "on-failure"; + RestartSec = "5s"; + SyslogIdentifier = grafanaAuthUnit; + ProtectSystem = "strict"; + ProtectHome = true; + PrivateTmp = "disconnected"; + PrivateDevices = true; + NoNewPrivileges = true; + RestrictSUIDSGID = true; + RemoveIPC = true; + }; + environment = { + BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; + BAO_CLIENT_CERT = grafanaAuthCredential "bao-client.pem"; + BAO_CLIENT_KEY = grafanaAuthCredential "bao-client-key.pem"; + } + // lib.optionalAttrs (logsAuthStoreFiles ? "bao-ca.pem") { + BAO_CACERT = grafanaAuthCredential "bao-ca.pem"; + }; + }; + containers.${cfg.machine} = { autoStart = true; ephemeral = false; diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index a171f7fc..49f16549 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -726,18 +726,24 @@ let lib.hasInfix "/nats-auth.pem ]" s && lib.hasInfix "swarm-nats-auth \"\" clientAuth" s; } { - # The same `+` leaf as the queue responder, read-only, and nothing else: - # this principal checks a presented secret and does nothing with it. - name = "the log store responder's grant is every agent's queue secret, read-only, and nothing else"; + # Every agent's queue secret and Grafana's metrics bearer, both + # read-only, and nothing else: this principal checks a presented secret + # and does nothing with it. + name = "the log store responder's grant is every agent's queue secret and Grafana's bearer, read-only, and nothing else"; ok = let s = baoGrantHere.systemd.services.swarm-bao-logs-auth-policy.script; in - lib.hasInfix "path \"secret/data/swarm/agents/+/queue\" {\n capabilities = [\"read\"]\n}" s - && lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1 - && !(lib.hasInfix "secret/data/swarm/agents/*" s) - && !(lib.hasInfix "secret/data/swarm/hives" s) - && !(lib.hasInfix "secret/data/swarm/services" s) + grantsIn s == [ + { + path = "secret/data/swarm/agents/+/queue"; + caps = [ "read" ]; + } + { + path = "secret/data/swarm/services/swarm-grafana/metrics/bearer"; + caps = [ "read" ]; + } + ] && lib.hasInfix "auth/cert/certs/swarm-logs-auth" s && lib.hasInfix "allowed_common_names=swarm-logs-auth" s && lib.hasInfix "token_policies=swarm-logs-auth" s; diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index df8c2608..ecc4f2ee 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -627,6 +627,22 @@ let && lib.hasInfix "path \"secret/data/${leaf}\" {\n capabilities = [\"read\"]\n}" s.swarm-bao-metrics-auth-policy.script && lib.hasInfix "path \"secret/data/${leaf}\" {\n capabilities = [\"create\", \"read\"]\n}" s.swarm-bao-grafana-oidc-policy.script; } + { + # The log store's second responder checks the very bearer the metrics + # datasource sends, at the same store path: a different file would be a + # bearer the responder never reads. + name = "Grafana's logs datasource sends the bearer the log store's responder checks, at one store path"; + ok = + let + datasources = + allLocal.containers.swarm-grafana.config.services.grafana.provision.datasources.settings.datasources; + metrics = lib.findFirst (d: d.uid == "swarm-victoriametrics") null datasources; + logs = lib.findFirst (d: d.uid == "swarm-victorialogs") null datasources; + leaf = "swarm/services/${allLocal.services.hyperhive.swarm.grafana.oidc.clientId}/metrics/bearer"; + in + logs.secureJsonData.httpHeaderValue1 == metrics.secureJsonData.httpHeaderValue1 + && lib.hasInfix "--store-path ${leaf}" allLocal.systemd.services.swarm-logs-grafana-auth.serviceConfig.ExecStart; + } { # The responder reads the store as itself, with the leaf the store's # host mints for it. @@ -709,15 +725,23 @@ let l = allLocal.services.nginx.virtualHosts."logs.t.local".locations; http = allLocal.services.nginx.appendHttpConfig; unit = allLocal.systemd.services.swarm-logs-auth; + grafanaUnit = allLocal.systemd.services.swarm-logs-grafana-auth; in lib.hasInfix "map \$http_authorization \$swarm_logs_auth_target" http && lib.hasInfix "\"~^Bearer swarm-agent\\.\" /__swarm_logs_agent_auth;" http + && lib.hasInfix "\"~^Bearer [0-9a-f]{64}$\" /__swarm_logs_grafana_auth;" http && lib.hasInfix "default /__hive_authelia;" http && lib.hasInfix "auth_request /__swarm_logs_auth;" l."^~ /select/logsql/".extraConfig && lib.hasInfix "rewrite ^ \$swarm_logs_auth_target last;" l."= /__swarm_logs_auth".extraConfig && lib.hasInfix "internal;" l."= /__swarm_logs_agent_auth".extraConfig && l."= /__swarm_logs_agent_auth".proxyPass == "http://unix:/run/swarm-logs-auth/auth.sock:/" && lib.hasInfix "--socket /run/swarm-logs-auth/auth.sock" unit.serviceConfig.ExecStart + && lib.hasInfix "internal;" l."= /__swarm_logs_grafana_auth".extraConfig + && + l."= /__swarm_logs_grafana_auth".proxyPass == "http://unix:/run/swarm-logs-grafana-auth/auth.sock:/" + && lib.hasInfix "--socket /run/swarm-logs-grafana-auth/auth.sock" grafanaUnit.serviceConfig.ExecStart + && lib.hasInfix "swarm-metrics-auth" grafanaUnit.serviceConfig.ExecStart + && lib.hasInfix "--store-cert-role swarm-logs-auth" grafanaUnit.serviceConfig.ExecStart && l ? "= /__hive_authelia"; } { @@ -760,6 +784,7 @@ let ]); in owners ? swarm-logs-auth + && owners ? swarm-logs-grafana-auth && owners ? swarm-metrics-auth && !(lib.any (u: hidden u.serviceConfig) (lib.attrValues owners)); } diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index 06d3d568..5113c324 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -378,6 +378,24 @@ let lib.any (e: lib.hasInfix "PRIORITY:*" e) counts && lib.any (e: lib.hasInfix "PRIORITY:\"\"" e) counts; } + { + # The logs datasource reaches the store through its swarm name — a + # loopback URL works only where Grafana and the store share a host — and + # sends the metrics datasource's bearer, which the store's `/select/logsql/` + # route checks. + name = "the logs datasource targets the log store's swarm name and sends the metrics bearer"; + ok = + let + datasources = + grafanaWithAuthelia.containers.swarm-grafana.config.services.grafana.provision.datasources.settings.datasources; + metrics = lib.findFirst (d: d.uid == "swarm-victoriametrics") null datasources; + logs = lib.findFirst (d: d.uid == "swarm-victorialogs") null datasources; + in + logs.url == "https://${grafanaWithAuthelia.services.hyperhive.swarm.victorialogs.domain}" + && !(lib.hasInfix "127.0.0.1" logs.url) + && logs.jsonData.httpHeaderName1 == "Authorization" + && logs.secureJsonData.httpHeaderValue1 == metrics.secureJsonData.httpHeaderValue1; + } { # The reader's half of the same mapping. The collector writes a severity # TEXT; nothing downstream reads it as a level unless the datasource is