From fcc9c8d0ba1764a1e238f70cd484dc13241abd6b Mon Sep 17 00:00:00 2001 From: atlas Date: Fri, 30 Oct 2026 16:12:37 +0100 Subject: [PATCH] drop the one-time migration: seed the store from a leftover file once, leave the file A leftover plaintext client secret is stored once with -cas=0 when the store holds none, so nothing rotates. A store value wins over a different file. The file is no longer read back, compared or removed, and the migration unit is gone. --- docs/swarm/credentials-reference.md | 76 +++++----- docs/swarm/credentials.md | 14 +- nix/checks.nix | 6 +- nix/host-modules/lib/oidc-client-secrets.nix | 68 +++------ nix/host-modules/swarm-authelia.nix | 87 +----------- nix/module-eval/secret-publisher.nix | 70 ++------- nix/script-tests/oidc-client-secrets.nix | 3 +- nix/script-tests/oidc-client-secrets.sh | 142 +++++-------------- 8 files changed, 115 insertions(+), 351 deletions(-) diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 9c69d516..03cac2f9 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -108,44 +108,44 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. ### Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | -| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | -| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice sync` in `hive-matrix`. The store's pair wins; a store holding none takes the container's own tokens, so the value doesn't change by moving. Only `appservice render` on a container with neither mints a new pair | `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` in `hive-matrix`, the last copy `sync` took, which tuwunel starts from; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down leaves the homeserver running on the files | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives//oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services//oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/.secret`; `authelia-swarm-client-secrets` moves the files an earlier release left into the store and removes them | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) | -| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | -| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | -| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services//secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs Grafana on that file | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it | -| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 | -| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | -| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on the `oidc/client` leaves it mints, the controller's client and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `authelia-swarm-secrets`, `authelia-swarm-client-secrets`, `swarm-authelia-bridge` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | -| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | -| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives//` and drop the setting | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | -| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | -| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents//backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file | -| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents//proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it | -| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | -| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | -| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | -| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | -| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | -| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services//oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | +| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | +| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice sync` in `hive-matrix`. The store's pair wins; a store holding none takes the container's own tokens, so the value doesn't change by moving. Only `appservice render` on a container with neither mints a new pair | `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` in `hive-matrix`, the last copy `sync` took, which tuwunel starts from; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down leaves the homeserver running on the files | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives//oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services//oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) | +| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | +| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | +| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services//secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs Grafana on that file | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it | +| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 | +| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | +| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on the `oidc/client` leaves it mints, the controller's client and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `authelia-swarm-secrets`, `swarm-authelia-bridge` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | +| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | +| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives//` and drop the setting | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | +| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents//backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file | +| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents//proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it | +| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | +| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | +| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | +| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | +| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | +| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services//oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | ### Hive-level — one of each per hive diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 94d31f4b..12a57d91 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -90,8 +90,7 @@ only when the store answers and holds none does it run `authelia crypto hash gen create the leaf and write the digest. Authelia keeps only the digest on disk, in `oidc-clients/.digest`, and starts from it while the store is down. A store that can't answer a read never mints, because a second secret would rotate the first. A client with a -secret but no digest, in the store or in a leftover plaintext file, gets the digest of -that secret and keeps the secret; `authelia crypto hash generate` computes the digest, +secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest, with the password typed into its prompt through a pty, never in a command line. `swarm-authelia-bridge` fetches its own secret from `swarm/services/swarm-authelia-bridge/oidc/client` into @@ -99,14 +98,9 @@ with the password typed into its prompt through a pty, never in a command line. answer keeps the bridge down while it retries. A container without the publisher's identity keeps the plaintext in `oidc-clients/.secret`, as before. -`authelia-swarm-client-secrets` moves the `oidc-clients/.secret` files an earlier -release left. For each, it stores the file's value when the store holds none, reads it -back, and removes the file only when the store holds the same value; it logs one line per -client containing `removed the file`. A store that holds a different value keeps both and -logs an `err` line, and the unit stays failed without retrying, because only an operator can -say which value is right; a store that doesn't answer keeps the file and retries. Once no -file remains the unit does nothing. A file for a client that's no longer registered stays -in place, with a log line. +The mint reads an `oidc-clients/.secret` file an earlier release left once: when the store holds +no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value. +A store that holds a value wins, whatever the file says, and the file stays on disk, unused. Three telemetry collectors exist, and each one fails that test: each reads its own secret out of the store, under its own identity. diff --git a/nix/checks.nix b/nix/checks.nix index 934eb4ae..a35ed7b2 100644 --- a/nix/checks.nix +++ b/nix/checks.nix @@ -88,9 +88,9 @@ in # Runs ./host-modules/lib/store-seeded-secret.nix's seed, store-wins and mint # decisions against a stub `bao` that keeps its leaves in files. script-test-store-seeded-secret = import ./script-tests/store-seeded-secret.nix { inherit pkgs; }; - # Runs ./host-modules/lib/oidc-client-secrets.nix's mint and migration - # decisions for authelia's client secrets against stub `bao` and `authelia`: - # what is minted, stored and removed, and what is left alone when the store + # Runs ./host-modules/lib/oidc-client-secrets.nix's mint decisions for + # authelia's client secrets against stub `bao` and `authelia`: + # what is minted, seeded from a leftover file or left alone when the store # cannot answer. script-test-oidc-client-secrets = import ./script-tests/oidc-client-secrets.nix { inherit pkgs; }; module-eval-nats-tls = import ./module-eval/nats-tls.nix { diff --git a/nix/host-modules/lib/oidc-client-secrets.nix b/nix/host-modules/lib/oidc-client-secrets.nix index a3af762e..57e8bdfe 100644 --- a/nix/host-modules/lib/oidc-client-secrets.nix +++ b/nix/host-modules/lib/oidc-client-secrets.nix @@ -4,21 +4,15 @@ # the store answers. # # `mint_client_secret` keeps every client with a digest. A client without one -# gets it from the secret that exists: the plaintext file an earlier release -# left, else the store's value. Only a client with none of the three is minted, +# gets it from the store's secret, which an earlier release's plaintext file +# seeds once when the store holds none. Only a client with neither is minted, # and then the store's leaf is created (`-cas=0`, so an existing value is never # replaced) before the digest is written, so a digest on disk always has a # secret behind it. A store that cannot be read never mints, because a second # secret would rotate the first. # -# `migrate_client_secret` moves the plaintext file an earlier release left in -# `$clients/.secret` into the store and removes it once a read-back shows -# the store holds the same value. It never overwrites a store value and never -# removes a file the store does not match. -# # Return statuses: 0 done, 1 retry later (the store could not answer, or a -# digest could not be computed), 3 (migration only) the file and the store hold -# different secrets, and only an operator can say which is right. +# digest could not be computed). # # Pure function — NOT a NixOS module. Call it after `./store-seeded-secret.nix` # and `./atomic-write-secret.nix`, with `store_init` and `store_login` run and @@ -72,14 +66,20 @@ mint_client_secret() { local id="$1" leaf="$2" dig="$clients/$1.digest" legacy="$clients/$1.secret" local state=0 out secret digest - if [ -s "$legacy" ]; then - if [ -s "$dig" ]; then - return 0 - fi - client_secret_derive_digest "$id" "$(< "$legacy")" "$legacy" - return - fi client_secret_read "$leaf" || state=$? + if [ "$state" = 1 ] && [ -s "$legacy" ]; then + # A file an earlier release left: its value goes into the store once, so + # the secret keeps its value. The file stays, unused. + printf '%s' "$(< "$legacy")" | store_bao kv put -cas=0 "$leaf" value=- >/dev/null 2>"$STORE_ERR" \ + || cat "$STORE_ERR" >&2 + state=0 + client_secret_read "$leaf" || state=$? + if [ "$state" = 1 ]; then + echo "oidc client $id: could not store the secret from $legacy in $leaf; not minting" >&2 + return 1 + fi + echo "oidc client $id: stored the secret from $legacy in the store" + fi if [ "$state" = 0 ]; then if [ -s "$dig" ]; then chmod 0600 "$dig" @@ -119,40 +119,4 @@ MINTED=1 echo "minted client secret for $id into the store" } - - # - migrate_client_secret() { - local id="$1" leaf="$2" legacy="$clients/$1.secret" value state=0 - [ -e "$legacy" ] || return 0 - value="$(< "$legacy")" - if [ -z "$value" ]; then - echo "<3>oidc client $id: $legacy is empty; keeping it" >&2 - return 3 - fi - client_secret_read "$leaf" || state=$? - if [ "$state" = 2 ]; then - echo "oidc client $id: the store cannot be read; keeping $legacy" >&2 - return 1 - fi - if [ "$state" = 0 ]; then - if [ "$CLIENT_SECRET_VALUE" != "$value" ]; then - echo "<3>oidc client $id: the store at $leaf and $legacy hold different secrets; keeping both" >&2 - return 3 - fi - rm -f "$legacy" || return 1 - echo "oidc client $id: already in the store, removed the file $legacy" - return 0 - fi - if ! printf '%s' "$value" | store_bao kv put -cas=0 "$leaf" value=- >/dev/null 2>"$STORE_ERR"; then - cat "$STORE_ERR" >&2 - echo "oidc client $id: could not create $leaf; reading it back in case another run created it" >&2 - fi - client_secret_read "$leaf" || return 1 - if [ "$CLIENT_SECRET_VALUE" != "$value" ]; then - echo "<3>oidc client $id: the store at $leaf and $legacy hold different secrets; keeping both" >&2 - return 3 - fi - rm -f "$legacy" || return 1 - echo "oidc client $id: stored in the store, removed the file $legacy" - } '' diff --git a/nix/host-modules/swarm-authelia.nix b/nix/host-modules/swarm-authelia.nix index 92e0128d..b52e878a 100644 --- a/nix/host-modules/swarm-authelia.nix +++ b/nix/host-modules/swarm-authelia.nix @@ -237,7 +237,6 @@ let publisherCfg.enable && publisherCfg.baoClientCertFile != null && publisherCfg.baoClientKeyFile != null; - migrateUnit = "${unitName}-client-secrets"; forwarderFetchUnit = "swarm-journal-forwarder-oidc"; # The publisher's store identity, handed into the container as system # credentials (credential id → host source). The forwarder's fetch presents @@ -580,9 +579,7 @@ let storeRetry = import ./lib/store-retry.nix { }; # The bridge's own client secret, in the memory-backed runtime directory of - # its unit. While a plaintext file from an earlier release exists the bridge - # keeps using it, since it is the value the digest was made from; the file - # goes once the store holds the same value, and the store's takes over. + # its unit, fetched from the store before each start. bridgeSecretFile = "/run/swarm-authelia-bridge/client-secret"; bridgeSecretFetch = '' set -euo pipefail @@ -591,12 +588,9 @@ let ${storeSeededSecret} ${oidcClientSecrets} - legacy=${lib.escapeShellArg "${clientsDir}/${bridgeClientId}.secret"} store_init store_login - if [ -s "$legacy" ]; then - value="$(< "$legacy")" - elif client_secret_read ${lib.escapeShellArg (clientLeaf bridgeClientId)}; then + if client_secret_read ${lib.escapeShellArg (clientLeaf bridgeClientId)}; then value="$CLIENT_SECRET_VALUE" else echo "the store holds no readable secret for ${bridgeClientId}" >&2 @@ -1094,7 +1088,6 @@ in ++ lib.optionals publishes [ "opentelemetry-collector.service" "${forwarderFetchUnit}.service" - "${migrateUnit}.service" "${unitName}-secrets.service" "swarm-authelia-bridge.service" ]; @@ -1133,79 +1126,6 @@ in serviceConfig.LoadCredential = lib.attrNames storeFiles; }; - # Moves the plaintext files an earlier release left in `clientsDir` - # into the store and removes each once the store holds the same - # value. With no file left it does nothing, store or no store. Nothing - # orders authelia after it: authelia starts from its digests. - systemd.services.${migrateUnit} = lib.mkIf publishes { - description = "move authelia's plaintext OIDC client secrets into the swarm secret store"; - wantedBy = [ "multi-user.target" ]; - after = [ "${unitName}-secrets.service" ]; - path = [ - pkgs.coreutils - pkgs.gnugrep - deployCfg.bao.package - ]; - startLimitBurst = storeRetry.startLimitBurst; - startLimitIntervalSec = storeRetry.startLimitIntervalSec; - serviceConfig = storeRetry.serviceConfig // { - Type = "oneshot"; - User = unitName; - Group = socketGroup; - UMask = "0077"; - LoadCredential = lib.attrNames storeFiles; - TimeoutStartSec = 120; - RestartPreventExitStatus = 3; - SyslogIdentifier = migrateUnit; - }; - environment = - let - store = storeIdentityIn migrateUnit; - in - { - BAO_ADDR = store.addr; - BAO_CLIENT_CERT = store.clientCertFile; - BAO_CLIENT_KEY = store.clientKeyFile; - } - // lib.optionalAttrs (store.caFile != null) { BAO_CACERT = store.caFile; }; - script = '' - set -uo pipefail - - ${storeSeededSecret} - ${oidcClientSecrets} - - clients=${lib.escapeShellArg clientsDir} - known=${lib.escapeShellArg (lib.concatMapStrings (c: " ${c.id}") secretClients + " ")} - - pending=0 - for f in "$clients"/*.secret; do - [ -e "$f" ] || continue - id="$(basename "$f" .secret)" - case "$known" in - *" $id "*) pending=1 ;; - *) echo "left $f in place: no client $id is registered here" >&2 ;; - esac - done - if [ "$pending" = 0 ]; then - echo "no plaintext client secret files to move" - exit 0 - fi - - store_init - store_login - - status=0 - ${lib.concatMapStrings (c: '' - rc=0 - migrate_client_secret ${lib.escapeShellArg c.id} ${lib.escapeShellArg (clientLeaf c.id)} || rc=$? - if [ "$rc" != 0 ] && [ "$status" != 1 ]; then - status="$rc" - fi - '') secretClients} - exit "$status" - ''; - }; - # The authelia binary itself, so an operator who gets a shell # in here can run `authelia crypto hash generate` to make a # password for the users file. Without it the container runs @@ -1323,8 +1243,7 @@ in after = [ "${unitName}-secrets.service" "${unitName}.service" - ] - ++ lib.optional publishes "${migrateUnit}.service"; + ]; wants = [ "${unitName}-secrets.service" "${unitName}.service" diff --git a/nix/module-eval/secret-publisher.nix b/nix/module-eval/secret-publisher.nix index 01884f72..115095c5 100644 --- a/nix/module-eval/secret-publisher.nix +++ b/nix/module-eval/secret-publisher.nix @@ -323,7 +323,6 @@ let onStdin = s: puts s != [ ] && lib.all (l: lib.hasInfix "value=-" l) (puts s); in onStdin (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script - && onStdin (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-client-secrets.script && !(onStdin "printf '%s' \"$secret\" | bao kv put -cas=0 x value=\"$secret\"") && !(onStdin "# kv put x value=-\nbao kv put x value=@/tmp/f") && onStdin "printf '%s' \"$secret\" | bao kv put -cas=0 x value=-"; @@ -369,8 +368,6 @@ let && !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c))); in holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script - && holdsTokenFirst (inAuthelia secretPublisherHere) - .systemd.services.authelia-swarm-client-secrets.script && holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-authelia-bridge.preStart && holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-matrix-oidc.script # Controls, so a clean verdict above means something. In order: a bare @@ -389,24 +386,24 @@ let # would have left the unit rendering only on the one deployment that has # no use for it. It runs beside the mint, in authelia's container, and # never on the host. - name = "the client secret migration holding an identity runs in authelia's container on a host with no store"; + name = "the mint holding an identity runs in authelia's container on a host with no store"; ok = let m = secretPublisherRemote; c = inAuthelia m; in !m.services.hyperhive.deploy.bao.enable - && (c.systemd.services ? authelia-swarm-client-secrets) - && !(m.systemd.services ? authelia-swarm-client-secrets) + && lib.hasInfix "mint_client_secret" c.systemd.services.authelia-swarm-secrets.script + && !(m.systemd.services ? authelia-swarm-secrets) # Control: the host's own unit set is the one read above. && (m.systemd.services ? "container@swarm-authelia"); } { # What makes the arm above able to fail. Without a certificate the # plaintext has no store to go to, so the mint keeps it in a file beside - # the digest as before, the migration does not exist, the bridge reads + # the digest as before, the bridge reads # that file, and nspawn gets no credential to load. - name = "an IdP host with no store identity keeps the client secrets in files and renders no migration"; + name = "an IdP host with no store identity keeps the client secrets in files"; ok = let m = secretPublisherNoIdentity; @@ -414,7 +411,6 @@ let mint = c.systemd.services.authelia-swarm-secrets.script; in m.services.hyperhive.deploy.authelia.enable - && !(c.systemd.services ? authelia-swarm-client-secrets) && lib.hasInfix "mint() {" mint && !(lib.hasInfix "mint_client_secret" mint) && @@ -423,7 +419,6 @@ let && !(c.systemd.services.swarm-authelia-bridge.serviceConfig ? RuntimeDirectory) && !(lib.any (lib.hasPrefix "--load-credential=") m.containers.swarm-authelia.extraFlags) # Controls: with an identity, the same lookups find the other shape. - && ((inAuthelia secretPublisherHere).systemd.services ? authelia-swarm-client-secrets) && lib.hasInfix "mint_client_secret" ( (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script ) @@ -456,20 +451,16 @@ let && lib.hasInfix "client_secret_read ${leaf}" bridge.preStart && lib.hasInfix "mint_client_secret swarm-authelia-bridge ${leaf}" here.systemd.services.authelia-swarm-secrets.script && lib.elem "publisher-bao-cert" bridge.serviceConfig.LoadCredential - && bridge.serviceConfig.Restart == "on-failure" - # The file it reads from is the one the earlier release left, until the - # migration moves it into the store. - && lib.hasInfix "legacy=/var/lib/authelia-swarm/oidc-clients/swarm-authelia-bridge.secret" bridge.preStart; + && bridge.serviceConfig.Restart == "on-failure"; } { - # Every non-native client is migrated and minted at the leaf its reader + # Every non-native client is minted at the leaf its reader # fetches, not only the ones with a reader in the publisher's list: the # bridge's own, which only the bridge reads, is the one that list lacks. - name = "the migration visits every client at the leaf the mint creates, and a failure that no retry changes is not retried"; + name = "the mint stores every client at the leaf its reader fetches"; ok = let c = inAuthelia secretPublisherHere; - migrate = c.systemd.services.authelia-swarm-client-secrets; mint = c.systemd.services.authelia-swarm-secrets; pairs = [ "hive-h1 secret/swarm/hives/h1/oidc/client" @@ -479,19 +470,11 @@ let "swarm-authelia-bridge secret/swarm/services/swarm-authelia-bridge/oidc/client" ]; in - lib.all ( - p: - lib.hasInfix "migrate_client_secret ${p}" migrate.script - && lib.hasInfix "mint_client_secret ${p}" mint.script - ) pairs - # A native client is public: it has no secret to mint or move. - && !(lib.hasInfix "swarmctl" migrate.script) + lib.all (p: lib.hasInfix "mint_client_secret ${p}" mint.script) pairs + # A native client is public: it has no secret to mint. && !(lib.hasInfix "mint_client_secret swarmctl" mint.script) - && migrate.serviceConfig.RestartPreventExitStatus == 3 # The mint answers authelia's password prompt through a pty. - && lib.any (p: lib.hasPrefix "util-linux" (p.name or "")) mint.path - && migrate.serviceConfig.Restart == "on-failure" - && migrate.serviceConfig.Type == "oneshot"; + && lib.any (p: lib.hasPrefix "util-linux" (p.name or "")) mint.path; } { # Every key authelia signs or encrypts with comes from the store, seeded @@ -532,8 +515,8 @@ let name = "the store identity reaches authelia's container as system credentials"; ok = let - dir = "/run/credentials/authelia-swarm-client-secrets.service"; - unit = m: (inAuthelia m).systemd.services.authelia-swarm-client-secrets; + dir = "/run/credentials/authelia-swarm-secrets.service"; + unit = m: (inAuthelia m).systemd.services.authelia-swarm-secrets; flags = m: m.containers.swarm-authelia.extraFlags; here = unit secretPublisherHere; in @@ -566,33 +549,6 @@ let ) && (unit secretPublisherRemote).environment.BAO_CLIENT_CERT == "${dir}/publisher-bao-cert"; } - { - # Authelia is the swarm's IdP and has to come up while the store is down. - # Nothing it starts after or needs may be the migration, and the - # migration may not order itself in front of authelia. - name = "authelia does not wait for the client secret migration"; - ok = - let - c = inAuthelia secretPublisherHere; - authelia = c.systemd.services.authelia-swarm; - publisher = c.systemd.services.authelia-swarm-client-secrets; - unit = "authelia-swarm-client-secrets.service"; - in - !(lib.any (l: lib.elem unit (authelia.${l} or [ ])) [ - "after" - "requires" - "wants" - "bindsTo" - "requisite" - ]) - && !(lib.any (l: lib.elem "authelia-swarm.service" (publisher.${l} or [ ])) [ - "before" - "requiredBy" - ]) - # Controls: both lookups find the real units. - && authelia.serviceConfig ? ExecStart - && lib.elem "authelia-swarm-secrets.service" publisher.after; - } ]; in runGroup "secret-publisher" cases diff --git a/nix/script-tests/oidc-client-secrets.nix b/nix/script-tests/oidc-client-secrets.nix index ac438220..08b823fc 100644 --- a/nix/script-tests/oidc-client-secrets.nix +++ b/nix/script-tests/oidc-client-secrets.nix @@ -1,6 +1,5 @@ # `checks.script-test-oidc-client-secrets` — runs -# ../host-modules/lib/oidc-client-secrets.nix's `mint_client_secret` and -# `migrate_client_secret` against a stub `bao` that keeps its leaves in files +# ../host-modules/lib/oidc-client-secrets.nix's `mint_client_secret` against a stub `bao` that keeps its leaves in files # and a stub `authelia` that prints a fixed password and digest. The cases are # in ./oidc-client-secrets.sh. # diff --git a/nix/script-tests/oidc-client-secrets.sh b/nix/script-tests/oidc-client-secrets.sh index 4a0a128f..e6a5f97f 100644 --- a/nix/script-tests/oidc-client-secrets.sh +++ b/nix/script-tests/oidc-client-secrets.sh @@ -2,7 +2,7 @@ # digest and the plaintext file an earlier release left, and the stub's # behaviour, runs the function under `set -e` as the units' scripts do, and # compares its status, what the store holds, what is left on disk and whether -# `authelia` ran. +# `authelia` minted. set -uo pipefail failures=0 @@ -172,27 +172,57 @@ if [ "$(stat -c %a "$digest")" != 600 ]; then fail "digest mode is $(stat -c %a if ! logged "oidc client $id: wrote the digest from the secret in the store"; then fail "no info line"; fi no_secret_in_argv -setup "mint: a plaintext file with its digest is left for the migration" +setup "mint: a plaintext file the store lacks seeds the store once, mints nothing and keeps the digest and the file" printf 'old-secret\n' >"$legacy" printf 'old-digest\n' >"$digest" run mint_client_secret expect 0 +held old-secret digest_is old-digest if ran_authelia; then fail "minted although a file held the secret"; fi -if store_called; then fail "called the store"; fi +if [ "$(cat "$legacy")" != old-secret ]; then fail "changed or removed the file"; fi +if ! grep -q '^kv put -cas=0 ' "$FAKE_BAO_LOG"; then fail "no create-only write was made"; fi +no_secret_in_argv -setup "mint: a plaintext file with no digest gets the digest of the file's secret, and the file and the store are untouched" +setup "mint: a plaintext file with no digest seeds the store and gets the digest of the file's secret" printf 'old-secret\n' >"$legacy" run mint_client_secret expect 0 +held old-secret digest_is digest-of-old-secret -held "" if [ "$(cat "$legacy")" != old-secret ]; then fail "changed the file"; fi if ran_authelia; then fail "minted over the file's secret"; fi -if store_called; then fail "called the store"; fi -if ! logged "oidc client $id: wrote the digest from the secret in $legacy"; then fail "no info line"; fi no_secret_in_argv +setup "mint: the store's secret wins over a different file, silently, and nothing is written" +printf 'old-secret\n' >"$legacy" +printf 'other-secret' >"$stored" +printf 'old-digest\n' >"$digest" +run mint_client_secret +expect 0 +held other-secret +digest_is old-digest +if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote to a leaf that held a value"; fi +if ran_authelia; then fail "minted"; fi + +setup "mint: a file with a store that refuses the login keeps the digest and mints nothing" +printf 'old-secret\n' >"$legacy" +printf 'old-digest\n' >"$digest" +export FAKE_BAO_LOGIN_RC=1 +run mint_client_secret +expect 0 +digest_is old-digest +if ran_authelia; then fail "minted with the store down"; fi + +setup "mint: a file whose seed write is refused is a failure, and nothing is minted" +printf 'old-secret\n' >"$legacy" +printf 'old-digest\n' >"$digest" +export FAKE_BAO_DENY_WRITE=1 +run mint_client_secret +expect 1 +held "" +if ran_authelia; then fail "minted although the file held the secret"; fi + setup "mint: a secret whose digest cannot be computed is a failure and writes no digest" printf 'old-secret' >"$stored" export FAKE_AUTHELIA_NODIGEST=1 @@ -232,104 +262,6 @@ if [ "$(id -u)" != 0 ]; then digest_is "" fi -# --- migrate_client_secret ------------------------------------------------ - -setup "migrate: a file the store lacks is stored, read back and removed" -printf 'old-secret\n' >"$legacy" -run migrate_client_secret -expect 0 -held old-secret -if [ -e "$legacy" ]; then fail "left the file"; fi -if ! logged "stored in the store, removed the file $legacy"; then fail "no stored line"; fi -if ran_authelia; then fail "minted"; fi -no_secret_in_argv - -setup "migrate: a file the store already holds is removed, and the store is not written" -printf 'old-secret\n' >"$legacy" -printf 'old-secret\n' >"$stored" -run migrate_client_secret -expect 0 -held "old-secret" -if [ -e "$legacy" ]; then fail "left the file"; fi -if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote to a leaf that held a value"; fi -if ! logged "already in the store, removed the file $legacy"; then fail "no removed line"; fi - -setup "migrate: a store holding a different secret keeps both and reports an error" -printf 'old-secret\n' >"$legacy" -printf 'other-secret' >"$stored" -run migrate_client_secret -expect 3 -held other-secret -if [ "$(cat "$legacy")" != old-secret ]; then fail "changed the file"; fi -if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote to a leaf that held a value"; fi -if ! logged "<3>oidc client $id"; then fail "no err-priority line"; fi - -setup "migrate: a store that refuses the login keeps the file and retries" -printf 'old-secret\n' >"$legacy" -export FAKE_BAO_LOGIN_RC=1 -run migrate_client_secret -expect 1 -if [ ! -e "$legacy" ]; then fail "removed the file"; fi -if store_called; then fail "called kv after a refused login"; fi - -setup "migrate: a store that refuses the read keeps the file and retries" -printf 'old-secret\n' >"$legacy" -export FAKE_BAO_DENY_READ=1 -run migrate_client_secret -expect 1 -if [ ! -e "$legacy" ]; then fail "removed the file"; fi -if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote although the read was refused"; fi - -setup "migrate: a store that refuses the write keeps the file and retries" -printf 'old-secret\n' >"$legacy" -export FAKE_BAO_DENY_WRITE=1 -run migrate_client_secret -expect 1 -if [ ! -e "$legacy" ]; then fail "removed the file"; fi -held "" - -setup "migrate: losing the race to a different secret keeps the file and reports an error" -printf 'old-secret\n' >"$legacy" -export FAKE_BAO_RACE=winner-secret -run migrate_client_secret -expect 3 -held winner-secret -if [ ! -e "$legacy" ]; then fail "removed the file"; fi - -setup "migrate: an empty file is kept and reported" -: >"$legacy" -run migrate_client_secret -expect 3 -if [ ! -e "$legacy" ]; then fail "removed the file"; fi -held "" - -setup "migrate: no file is nothing to do, store or no store" -export FAKE_BAO_LOGIN_RC=1 -run migrate_client_secret -expect 0 -if store_called; then fail "called the store"; fi - -if [ "$(id -u)" != 0 ]; then - setup "migrate: a file that cannot be removed is a failure, not a move" - printf 'old-secret\n' >"$legacy" - chmod 500 "$clients" - run migrate_client_secret - chmod 700 "$clients" - expect 1 - held old-secret - if ! [ -e "$legacy" ]; then fail "file vanished"; fi - if logged "removed the file"; then fail "logged a removal that did not happen"; fi - - setup "migrate: a file the store already holds that cannot be removed is a failure" - printf 'old-secret\n' >"$legacy" - printf 'old-secret' >"$stored" - chmod 500 "$clients" - run migrate_client_secret - chmod 700 "$clients" - expect 1 - if logged "removed the file"; then fail "logged a removal that did not happen"; fi -fi - if [ "$failures" != 0 ]; then echo "$failures of $count cases failed" >&2 exit 1