diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index 0e7468b3a..705b7b450 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -152,7 +152,7 @@ the label isn't taken from the payload at all. **Nothing to set where the hive holds its store identity.** The hive's collector reads its secret from the swarm's secret store under the hive's own leaf (`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches -`swarm/hives//oidc/client`, which the publisher in authelia's container +`swarm/hives//oidc/client`, which authelia's mint, in its container, writes, into `/run/hive-otel-oidc/` (tmpfs, so never on persistent disk). It runs at boot and retries every 30s until the store answers; until then the collector runs on an empty file and the swarm refuses its exports. When a diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 65f679367..1144d0f72 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -120,8 +120,8 @@ token endpoint is the swarm's single address: Both default to the swarm's own addresses on every hive, so there is nothing to set for them. The client secret isn't an option: hive-c0re reads it from the swarm's secret store, at `swarm/hives//oidc/client`, under the hive's -own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, in -authelia's container, puts it there. A hive without a store leaf doesn't +own store leaf (`deploy.bao.clientCertFile`). Authelia's mint, in its +container, puts it there. A hive without a store leaf doesn't publish; setting `natsUrl` to null turns publishing off too. Until the secret is in the store, hive-c0re shows a `swarm_queue_config` diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index c75ee094f..03cac2f9e 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -23,7 +23,7 @@ the rest of the cell says how. | `swarm/agents//matrix-recovery/` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes | | `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice sync` in `hive-matrix`: seeded from the container's own tokens when the store holds none, a new pair only when the container has none either | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` write the new pair into the container and restart the homeserver | ✅ `sync` re-reads it every 10 minutes | -| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | +| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it | | `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 | | `swarm/controller/swarm-controller/forge/webhook-secret` | `swarm-controller`, at start, from its old `webhook-secret` file; a new one only when there is no file | `swarm-controller` | ❌ minted once; a rotated secret fails every delivery until the forge hooks are re-registered | ❌ read at start | @@ -38,7 +38,7 @@ the rest of the cell says how. | `swarm/hives//forge/token` | `swarm-controller`, every 5 minutes | `swarm-controller` and the hive | ✅ when missing or no longer matching the forge | ✅ read on every forge call | | `swarm/agents//proxy-credential` | `swarm-controller`: 32 random bytes, at start and every 5 minutes for every agent that holds a store identity | the hive (`hive-c0re`) and the agent, which fetches it | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` re-reads it every 30 seconds | | `swarm/hives//squid-selfcheck` | `swarm-controller`, in the same pass as the agents' credentials | the hive: `hive-c0re` and `squid-selfcheck-credential` (root) | ❌ never replaced: written only while the path holds nothing | ✅ `hive-c0re` every 30 seconds, the root unit every 10 minutes | -| `swarm/hives//oidc/client` | authelia, first boot; `swarm-secret-publish` copies it in | the hive | ❌ minted once; `swarm-secret-publish` republishes a re-mint | ✅ re-read when authelia refuses it | +| `swarm/hives//oidc/client` | authelia, first boot, straight into the store | the hive | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it | | `swarm/services//oidc/client` | authelia | that service | must be stated | must be stated | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge, at start and every 5 minutes | `swarm-bao-ci-runner` on the runner's host | ❌ the forge keeps one token until an admin resets it; the controller then writes the new one within 5 minutes | ✅ every 10 minutes; restarts the runner on a change | | `swarm/services//metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, when none is stored | `swarm-metrics-auth` and `swarm-logs-grafana-auth` | ❌ delete it and restart `swarm-bao-grafana-metrics-bearer` | ✅ the auth services re-read within a minute; ❌ Grafana restarts when its copy changes | @@ -108,44 +108,44 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. ### Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | -| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | -| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice sync` in `hive-matrix`. The store's pair wins; a store holding none takes the container's own tokens, so the value doesn't change by moving. Only `appservice render` on a container with neither mints a new pair | `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` in `hive-matrix`, the last copy `sync` took, which tuwunel starts from; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down leaves the homeserver running on the files | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store ([how](credentials.md#hive-level-secrets)) | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) | -| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | -| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | -| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services//secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs Grafana on that file | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it | -| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 | -| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | -| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | -| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | -| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives//` and drop the setting | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | -| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | -| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents//backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file | -| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents//proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it | -| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | -| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | -| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | -| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | -| the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | -| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's plaintext half (`swarm.authelia.otel.clientId`). `swarm-secret-publish` copies it to `swarm/services//oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes. The two copies can differ while the publisher hasn't caught up, for example while the store is down after a re-mint. Authelia then refuses the forwarder's token request, and the collector retries until the next read lands the new value and restarts it | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. The mint's own copy stays in `/var/lib/authelia-swarm/oidc-clients/.secret` | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | +| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | +| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice sync` in `hive-matrix`. The store's pair wins; a store holding none takes the container's own tokens, so the value doesn't change by moving. Only `appservice render` on a container with neither mints a new pair | `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` in `hive-matrix`, the last copy `sync` took, which tuwunel starts from; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down leaves the homeserver running on the files | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives//oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services//oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) | +| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | +| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | +| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services//secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs Grafana on that file | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it | +| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 | +| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | +| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on the `oidc/client` leaves it mints, the controller's client and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `authelia-swarm-secrets`, `swarm-authelia-bridge` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | +| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | +| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives//` and drop the setting | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | +| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents//backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file | +| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents//proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it | +| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | +| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | +| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | +| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | +| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | +| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services//oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | ### Hive-level — one of each per hive diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index b2c209e98..12a57d919 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -84,6 +84,24 @@ there is neither. Nothing rotates a key: the storage-encryption key encrypts aut database, and a new issuer key invalidates every token issued. A container without the publisher's identity generates the files as before. +The same unit mints the client secrets, straight into the store. It reads +`swarm/hives//oidc/client` or `swarm/services//oidc/client` first, and +only when the store answers and holds none does it run `authelia crypto hash generate`, +create the leaf and write the digest. Authelia keeps only the digest on disk, in +`oidc-clients/.digest`, and starts from it while the store is down. A store that +can't answer a read never mints, because a second secret would rotate the first. A client with a +secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest, +with the password typed into its prompt through a pty, never in a command line. +`swarm-authelia-bridge` fetches its own secret from +`swarm/services/swarm-authelia-bridge/oidc/client` into +`/run/swarm-authelia-bridge/client-secret` before each start, and a store that doesn't +answer keeps the bridge down while it retries. A container without the publisher's +identity keeps the plaintext in `oidc-clients/.secret`, as before. + +The mint reads an `oidc-clients/.secret` file an earlier release left once: when the store holds +no value for that client, it stores the file's value with `-cas=0`, so the secret keeps its value. +A store that holds a value wins, whatever the file says, and the file stays on disk, unused. + Three telemetry collectors exist, and each one fails that test: each reads its own secret out of the store, under its own identity. @@ -263,7 +281,7 @@ The rows are in the [reference](credentials-reference.md#hive-level--one-of-each The forge and the homeserver read their copies out of the store, the way Grafana does below: the forge under the forge publisher's leaf, the homeserver under matrix-ctl's. Each **fails loudly** and retries until the -publisher in authelia's container has written the secret, rather than skipping. +mint in authelia's container has stored the secret, rather than skipping. A silent skip produces a service whose login button always fails, which is a symptom many layers from its cause. @@ -271,7 +289,7 @@ The store's **first reader** is Grafana's OIDC client secret, a _swarm service's_ own credential rather than a hive's. It has exactly one delivery route: `swarm-bao-grafana-oidc.service` reads `swarm/services//oidc/client` out of the store, on **every** host -that runs Grafana. Where authelia is doesn't enter into it — the publisher in +that runs Grafana. Where authelia is doesn't enter into it — the mint in authelia's container writes that path whether the reader is a network away or in the container next door. @@ -586,7 +604,7 @@ works too. | piece | written by | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | authelia client, with redirect `swarm.bao.ui.oidc.redirectUri` | `glue-bao-ui-oidc-client.nix`, wherever authelia runs | -| its secret at `swarm/services/swarm-bao-ui/oidc/client` | `swarm-secret-publish`, like every other service's | +| its secret at `swarm/services/swarm-bao-ui/oidc/client` | authelia's mint, like every other service's | | the `oidc` auth mount | `swarm-bao-granter-role`, with the bootstrap token | | `swarm-operator-viewer` policy, the mount's config and role `swarm-operator-viewer` | `swarm-bao-operator-viewer-policy`, as the granter, which reads the secret above into the config | diff --git a/docs/swarm/services.md b/docs/swarm/services.md index e8188c862..000fca4e8 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -142,13 +142,13 @@ The swarm collector scrapes authelia's own metrics at container. The gateway admits only the collector's OAuth2 client there; any other session gets `deny`. -The container also runs `swarm-secret-publish`, which copies the client -secrets authelia mints into the swarm secret store under the secret -publisher's own store leaf, handed in as a system credential; see -[`credentials.md`](credentials.md). Authelia doesn't wait for it, so the IdP comes up -while the store is down, and the publisher retries every 30 seconds. The -container's journal forwarder reads its client secret back from the store -under the same leaf, and is off where the container has none. +The container's `authelia-swarm-secrets` unit creates each client secret directly in +the swarm secret store, under the secret publisher's own store leaf handed in as +a system credential, and writes only the digest to disk; see +[`credentials.md`](credentials.md). Authelia starts from its digests, so the IdP +comes up while the store is down, and a unit that needs the store retries every +30 seconds. The container's journal forwarder reads its client secret back from +the store under the same leaf, and is off where the container has none. See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC relying-party flow, and [`credentials.md`](credentials.md) for where authelia diff --git a/docs/swarm/sso.md b/docs/swarm/sso.md index 957b0b89f..4caeedb3a 100644 --- a/docs/swarm/sso.md +++ b/docs/swarm/sso.md @@ -122,8 +122,8 @@ mechanism with flags. ### 1. All-local — one host runs both Nothing to configure at all. authelia mints each client's secret at its -first boot, and `swarm-secret-publish` in its container writes it to the swarm -secret store at `swarm/services//oidc/client`. A host-side unit +first boot, and the mint in its container creates it in the swarm +secret store at `swarm/services//oidc/client`, keeping only the digest on disk. A host-side unit then reads it from the store and writes it into the service's container: `swarm-bao-forge-oidc` for the forge, under the forge publisher's store leaf, into `/run/hive-forge-oidc/.secret` on the host, which @@ -143,7 +143,7 @@ rejected login with no error text worth reading. ⚠️ nixos-container refuses to start a container whose bind source is missing, and this secret doesn't exist until authelia's first boot has run -and the publisher has written it. The forge therefore binds the **directory** +and the mint has stored it. The forge therefore binds the **directory** `/run/hive-forge-oidc`, which a tmpfiles rule creates at boot, never the file. The directory is tmpfs, so the secret is gone after a reboot until the fetch runs again; forgejo keeps it in its login-source row, so logins diff --git a/hive-c0re/src/swarm_queue.rs b/hive-c0re/src/swarm_queue.rs index c7d77cb18..1e9b61c06 100644 --- a/hive-c0re/src/swarm_queue.rs +++ b/hive-c0re/src/swarm_queue.rs @@ -14,7 +14,7 @@ //! environment. The client secret comes from the swarm's secret store, at //! `swarm_secret_client::queue::hive_client_path`, read under this hive's own //! certificate, held only in memory and read again when authelia refuses it -//! (`swarm_secret_client::stored_secret`). `swarm-secret-publish`, in authelia's +//! (`swarm_secret_client::stored_secret`). authelia's mint (`authelia-swarm-secrets`), in its //! container, puts it there. //! //! Connects lazily on first use rather than at boot — nothing here @@ -290,8 +290,8 @@ async fn fetch_secret() -> Result { match stored { Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()), _ => bail!( - "nothing at {path}: swarm-secret-publish, in authelia's container, has not \ - published this hive's client secret" + "nothing at {path}: authelia's mint, in its container, has not \ + stored this hive's client secret" ), } } diff --git a/nix/checks.nix b/nix/checks.nix index fd89b2adb..a35ed7b2b 100644 --- a/nix/checks.nix +++ b/nix/checks.nix @@ -88,6 +88,11 @@ in # Runs ./host-modules/lib/store-seeded-secret.nix's seed, store-wins and mint # decisions against a stub `bao` that keeps its leaves in files. script-test-store-seeded-secret = import ./script-tests/store-seeded-secret.nix { inherit pkgs; }; + # Runs ./host-modules/lib/oidc-client-secrets.nix's mint decisions for + # authelia's client secrets against stub `bao` and `authelia`: + # what is minted, seeded from a leftover file or left alone when the store + # cannot answer. + script-test-oidc-client-secrets = import ./script-tests/oidc-client-secrets.nix { inherit pkgs; }; module-eval-nats-tls = import ./module-eval/nats-tls.nix { inherit pkgs self nixosSystem; inherit (pkgs) lib; diff --git a/nix/container-modules/swarm-secret-publisher.nix b/nix/container-modules/swarm-secret-publisher.nix deleted file mode 100644 index 27a786673..000000000 --- a/nix/container-modules/swarm-secret-publisher.nix +++ /dev/null @@ -1,180 +0,0 @@ -# The unit that puts swarm-level secrets into the swarm's secret store, so -# whoever needs one can read it there: a hive, a service or the controller its -# own. It mints nothing: every -# value is an OIDC client secret authelia minted, and this unit copies it. -# -# Imported inside authelia's container, beside the mint: it reads each secret -# where authelia writes it. The host module that declares the container sets -# the options below, as for ./swarm-journal-forwarder.nix. -# -# ⚠️ "Whoever", including a reader on authelia's own host. A swarm service's -# secret goes into the store even when the service runs beside authelia, -# because its reader fetches it from the store in every deployment — -# ../host-modules/swarm-grafana.nix states the ruling that made that the only -# route. Publishing "only when the reader is elsewhere" would be a second shape -# of this unit, gated on a fact about another host, to save a round trip on -# the one host that can afford it. -# -# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite -# every hive's policy and login role; a unit that writes a handful of secrets -# has no business holding it. ../host-modules/swarm-bao.nix grants this one -# `create`/`update` under the hive prefix and nothing else — no `read`. -# -# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every -# argument is world-readable in /proc for the life of the call — the value is -# passed as `@` and read by bao itself. -# -# ⚠️ Nothing in the container waits for this unit. Authelia is the swarm's IdP -# and starts whether or not the store answers; a store that is down or sealed -# is a failed run that restarts. -{ - config, - lib, - pkgs, - ... -}: -let - cfg = config.services.hyperhive.swarmSecretPublisher; -in -{ - options.services.hyperhive.swarmSecretPublisher = { - enable = lib.mkEnableOption "copying the OIDC client secrets authelia mints into the swarm secret store"; - - clientSecretDir = lib.mkOption { - type = lib.types.str; - description = '' - Directory authelia mints each client's plaintext into, as - `.secret`. A path inside the container. - ''; - }; - - secrets = lib.mkOption { - type = lib.types.listOf ( - lib.types.submodule { - options = { - clientId = lib.mkOption { - type = lib.types.str; - description = "The minted client, whose `.secret` is copied."; - }; - path = lib.mkOption { - type = lib.types.str; - example = "secret/swarm/services/swarm-grafana/oidc/client"; - description = "KV path its reader fetches the value from."; - }; - }; - } - ); - description = "Every secret this unit copies, and where to."; - }; - - store = { - package = lib.mkOption { - type = lib.types.package; - description = "The `bao` CLI the unit logs in and writes with."; - }; - addr = lib.mkOption { - type = lib.types.str; - description = "The swarm secret store's address, by name."; - }; - clientCertFile = lib.mkOption { - type = lib.types.str; - description = '' - The publisher's store identity, as a path inside the container. Its - subject must be - {option}`services.hyperhive.deploy.bao.secretPublisherCommonName`. - A path, never a value. - ''; - }; - clientKeyFile = lib.mkOption { - type = lib.types.str; - description = "Private key for `store.clientCertFile`. A path, never a value."; - }; - caFile = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - description = "CA the store's server certificate chains to; null means the system trust store."; - }; - }; - }; - - config = lib.mkIf cfg.enable { - # Re-publish when authelia rotates a secret. The mint writes the file, so - # the file is the event — there is no signal from authelia to subscribe to. - systemd.paths.swarm-secret-publish = { - description = "watch for minted OIDC client secrets to publish"; - wantedBy = [ "multi-user.target" ]; - pathConfig = { - PathChanged = cfg.clientSecretDir; - Unit = "swarm-secret-publish.service"; - }; - }; - - systemd.services.swarm-secret-publish = { - description = "publish the swarm's minted secrets to the swarm secret store"; - wantedBy = [ "multi-user.target" ]; - after = [ "network.target" ]; - path = [ - cfg.store.package - pkgs.coreutils - ]; - # ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns - # every later start from the path unit above into a no-op. - serviceConfig = { - Type = "oneshot"; - # Bounded here rather than left to systemd's default, so the number a - # boot waits on is in the file that waits. A sealed store answers on - # the port and never answers the write. - TimeoutStartSec = 60; - Restart = "on-failure"; - RestartSec = 30; - }; - environment = { - BAO_ADDR = cfg.store.addr; - BAO_CLIENT_CERT = cfg.store.clientCertFile; - BAO_CLIENT_KEY = cfg.store.clientKeyFile; - } - // lib.optionalAttrs (cfg.store.caFile != null) { - BAO_CACERT = cfg.store.caFile; - }; - script = '' - set -euo pipefail - - # Cert auth is a login, not a transport setting. The `BAO_CLIENT_*` - # variables above only decide which certificate the TLS handshake - # presents; without a token `bao` asks its token helper instead, and - # that is a `sh` this unit's `path` does not carry. `-token-only` - # answers on stdout and skips the helper on both sides. - # - # Unhandled on purpose: `Restart=on-failure` above is what a store that - # cannot authenticate this container should get. Degrading here would - # report "published 0" as an ordinary quiet day. - BAO_TOKEN="$(bao login -method=cert -token-only)" - export BAO_TOKEN - - published=0 - skipped=0 - - ${lib.concatMapStringsSep "\n" (s: '' - src=${lib.escapeShellArg "${cfg.clientSecretDir}/${s.clientId}.secret"} - if [ -s "$src" ]; then - # `value=@$src` hands bao the PATH: bao opens the file itself, so - # the plaintext is never an argument of this process. Writing it as - # `value="$(cat "$src")"` would publish it to /proc for anyone in - # the container to read. - bao kv put ${lib.escapeShellArg s.path} \ - value=@"$src" - published=$((published + 1)) - else - # Not an error: authelia mints on its FIRST BOOT, so an absent file - # is "not yet", and the path unit above re-runs this when it lands. - # A client registered nowhere in this swarm is never minted at all. - echo "no minted secret at $src yet; the path unit will re-run this" >&2 - skipped=$((skipped + 1)) - fi - '') cfg.secrets} - - echo "published $published client secret(s), skipped $skipped" - ''; - }; - }; -} diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index b60417462..f7be29ecb 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -49,7 +49,7 @@ let statusSecretRemoved = '' hive-c0re reads this hive's status-publishing client secret from the swarm's secret store, under the hive's own certificate - (deploy.bao.clientCertFile), and swarm-secret-publish in authelia's container + (deploy.bao.clientCertFile), and authelia's mint, in its container, puts it there. Remove this definition; the file it named is read by nothing now and can be deleted. ''; diff --git a/nix/host-modules/glue-bao-tls.nix b/nix/host-modules/glue-bao-tls.nix index e74869978..27d4e2023 100644 --- a/nix/host-modules/glue-bao-tls.nix +++ b/nix/host-modules/glue-bao-tls.nix @@ -204,7 +204,7 @@ in # The secret publisher's, minted here for the reason the controller's # line above gives — and it serves that case more often, not less: the - # publisher runs in AUTHELIA's container, on the one host + # publisher's unit runs in AUTHELIA's container, on the one host # guaranteed not to be this one once the store has a host of its own. [ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \ ${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth diff --git a/nix/host-modules/glue-builder-oidc-client.nix b/nix/host-modules/glue-builder-oidc-client.nix index 6d8b72e2e..38c1131f5 100644 --- a/nix/host-modules/glue-builder-oidc-client.nix +++ b/nix/host-modules/glue-builder-oidc-client.nix @@ -2,7 +2,7 @@ # # ONE PAIRING PER FILE — swarm-builder ← authelia, and nothing else. Deleting # this leaves a builder whose token requests authelia answers with a 401, and -# no minted secret for swarm-secret-publish to put in the store. +# no minted secret for authelia's mint to put in the store. # # ⚠️ Gated on authelia being HERE, and NOT on this host running the builder, # for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS diff --git a/nix/host-modules/glue-claude-login-oidc-client.nix b/nix/host-modules/glue-claude-login-oidc-client.nix index 6c857dde1..a13954ee5 100644 --- a/nix/host-modules/glue-claude-login-oidc-client.nix +++ b/nix/host-modules/glue-claude-login-oidc-client.nix @@ -3,7 +3,7 @@ # # ONE PAIRING PER FILE — swarm-claude-login ← authelia, and nothing else. # Deleting this leaves a minter whose token requests authelia answers with a -# 401, and no minted secret for swarm-secret-publish to put in the store. +# 401, and no minted secret for authelia's mint to put in the store. # # ⚠️ Gated on authelia being HERE, and NOT on this host running the minter, # for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS diff --git a/nix/host-modules/glue-controller-oidc-client.nix b/nix/host-modules/glue-controller-oidc-client.nix index 255add9b0..ac06fe537 100644 --- a/nix/host-modules/glue-controller-oidc-client.nix +++ b/nix/host-modules/glue-controller-oidc-client.nix @@ -2,7 +2,7 @@ # # ONE PAIRING PER FILE — swarm-controller ← authelia, and nothing else. Deleting # this leaves a controller whose token requests authelia answers with a 401, and -# no minted secret for swarm-secret-publish to put in the store. +# no minted secret for authelia's mint to put in the store. # # ⚠️ Gated on authelia being HERE, and NOT on this host running the controller. # A client is a row in THIS host's provider config, so it is declared where that diff --git a/nix/host-modules/glue-matrix-oidc-client.nix b/nix/host-modules/glue-matrix-oidc-client.nix index 0bf531d39..fef647305 100644 --- a/nix/host-modules/glue-matrix-oidc-client.nix +++ b/nix/host-modules/glue-matrix-oidc-client.nix @@ -3,7 +3,7 @@ # # ONE PAIRING PER FILE — matrix ← authelia, and nothing else. Deleting this # leaves a homeserver whose SSO login authelia has never heard of, and no -# minted secret for swarm-secret-publish to put in the store. +# minted secret for authelia's mint to put in the store. # # ⚠️ Gated on authelia being HERE, and NOT on this host running the # homeserver. A client is a row in THIS host's provider config, so it is diff --git a/nix/host-modules/glue-nats-oidc-client.nix b/nix/host-modules/glue-nats-oidc-client.nix index 49b237f42..a5760e5a4 100644 --- a/nix/host-modules/glue-nats-oidc-client.nix +++ b/nix/host-modules/glue-nats-oidc-client.nix @@ -2,7 +2,7 @@ # # ONE PAIRING PER FILE — swarm-nats ← authelia, and nothing else. Deleting this # leaves a queue responder whose introspection calls authelia refuses, so it -# denies every client, and no minted secret for swarm-secret-publish to put in +# denies every client, and no minted secret for authelia's mint to put in # the store. # # ⚠️ Gated on authelia being HERE, and NOT on this host running the queue. A diff --git a/nix/host-modules/glue-notify-router-oidc-client.nix b/nix/host-modules/glue-notify-router-oidc-client.nix index 1991508ef..d96eab065 100644 --- a/nix/host-modules/glue-notify-router-oidc-client.nix +++ b/nix/host-modules/glue-notify-router-oidc-client.nix @@ -3,7 +3,7 @@ # # ONE PAIRING PER FILE — swarm-notify-router ← authelia, and nothing else. # Deleting this leaves a router whose token requests authelia answers with a -# 401, and no minted secret for swarm-secret-publish to put in the store. +# 401, and no minted secret for authelia's mint to put in the store. # # ⚠️ Gated on authelia being HERE, and NOT on this host running the router, # for ./glue-controller-oidc-client.nix's reason: a client is a row in THIS diff --git a/nix/host-modules/glue-review-classifier-oidc-client.nix b/nix/host-modules/glue-review-classifier-oidc-client.nix index 58ec8c363..8ae948d59 100644 --- a/nix/host-modules/glue-review-classifier-oidc-client.nix +++ b/nix/host-modules/glue-review-classifier-oidc-client.nix @@ -3,7 +3,7 @@ # # ONE PAIRING PER FILE — swarm-review-classifier ← authelia, and nothing else. # Deleting this leaves a classifier whose token requests authelia answers with -# a 401, and no minted secret for swarm-secret-publish to put in the store. +# a 401, and no minted secret for authelia's mint to put in the store. # # ⚠️ Gated on authelia being HERE, and NOT on this host running the # classifier, for ./glue-controller-oidc-client.nix's reason: a client is a row diff --git a/nix/host-modules/glue-secret-publisher-bao-identity.nix b/nix/host-modules/glue-secret-publisher-bao-identity.nix index 342f06393..599ca2b1b 100644 --- a/nix/host-modules/glue-secret-publisher-bao-identity.nix +++ b/nix/host-modules/glue-secret-publisher-bao-identity.nix @@ -6,7 +6,7 @@ # # ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the # leaf, because the thing that owns a private key owns issuing from it. What -# belongs here is the pairing: which host paths go to the publisher in +# belongs here is the pairing: which host paths go to the publisher's identity in # authelia's container. # # ⚠️ Gated on a PKI directory existing, not on the store alone — the same rule diff --git a/nix/host-modules/hive-forge/default.nix b/nix/host-modules/hive-forge/default.nix index 26cb5fa08..59548c806 100644 --- a/nix/host-modules/hive-forge/default.nix +++ b/nix/host-modules/hive-forge/default.nix @@ -40,7 +40,7 @@ let # root. Removed by the tmpfiles rule below. oldOidcSecretPath = "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret"; - # Where swarm-secret-publish in authelia's container leaves this client's + # Where authelia's mint, in its container, stores this client's # secret. # The forge publisher's store role grants read on this path. oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client"; @@ -1201,8 +1201,8 @@ in # The forge's OIDC client secret, out of the swarm secret store under the # forge publisher's leaf — the one route, wherever authelia runs. Same # shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the - # read both fail loudly, and `storeRetry` retries until the publisher in - # authelia's container has written the secret. + # read both fail loudly, and `storeRetry` retries until authelia's mint, + # in its container, has stored the secret. # # On the host, which reads that leaf from its own files. A host with no # publisher leaf fails here, naming the options, rather than rendering no diff --git a/nix/host-modules/hive-forge/service.nix b/nix/host-modules/hive-forge/service.nix index 3f35946c7..4f017b9d8 100644 --- a/nix/host-modules/hive-forge/service.nix +++ b/nix/host-modules/hive-forge/service.nix @@ -164,8 +164,8 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-forge-otel-oidc-client.nix`), the swarm collector admits it on the container's receiver - (`swarm.otel.serviceProducers`), and the secret publisher carries - its secret to the store. + (`swarm.otel.serviceProducers`), and authelia's mint stores + its secret in the store. ''; }; }; diff --git a/nix/host-modules/hive-matrix-service.nix b/nix/host-modules/hive-matrix-service.nix index a96d964a3..fa0bb38b1 100644 --- a/nix/host-modules/hive-matrix-service.nix +++ b/nix/host-modules/hive-matrix-service.nix @@ -175,8 +175,8 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-matrix-otel-oidc-client.nix`), the swarm collector admits it on the container's receiver - (`swarm.otel.serviceProducers`), and the secret publisher carries - its secret to the store. + (`swarm.otel.serviceProducers`), and authelia's mint stores + its secret in the store. ''; }; }; diff --git a/nix/host-modules/hive-matrix.nix b/nix/host-modules/hive-matrix.nix index 447b25a4a..694ce7590 100644 --- a/nix/host-modules/hive-matrix.nix +++ b/nix/host-modules/hive-matrix.nix @@ -52,7 +52,7 @@ let # Where the secret sat on persistent disk before it moved to /run. legacyHostSecretPath = "/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret"; - # Where swarm-secret-publish in authelia's container leaves this client's + # Where authelia's mint, in its container, stores this client's # secret. # matrix-ctl's store role grants read on this path. storeSecretPath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client"; @@ -1064,8 +1064,7 @@ in # # `Restart=on-failure` covers a sealed store or one still starting; # the timer below repeats the run, which a unit without `RemainAfterExit` allows. - # The start timeout is bounded for the reason - # ./swarm-secret-publisher.nix states: a sealed store answers on the + # The start timeout is bounded because a sealed store answers on the # port and never answers the read. systemd.services.swarm-matrix-appservice-sync = lib.mkIf ctlActive { description = "keep the swarm's appservice registration in step with the swarm secret store"; diff --git a/nix/host-modules/lib/oidc-client-secrets.nix b/nix/host-modules/lib/oidc-client-secrets.nix new file mode 100644 index 000000000..57e8bdfe4 --- /dev/null +++ b/nix/host-modules/lib/oidc-client-secrets.nix @@ -0,0 +1,122 @@ +# Shared shell steps for authelia's OIDC client secrets, whose plaintext half +# lives in the swarm secret store and never on disk. authelia keeps only a +# digest of each, in `$clients/.digest`, and starts from it whether or not +# the store answers. +# +# `mint_client_secret` keeps every client with a digest. A client without one +# gets it from the store's secret, which an earlier release's plaintext file +# seeds once when the store holds none. Only a client with neither is minted, +# and then the store's leaf is created (`-cas=0`, so an existing value is never +# replaced) before the digest is written, so a digest on disk always has a +# secret behind it. A store that cannot be read never mints, because a second +# secret would rotate the first. +# +# Return statuses: 0 done, 1 retry later (the store could not answer, or a +# digest could not be computed). +# +# Pure function — NOT a NixOS module. Call it after `./store-seeded-secret.nix` +# and `./atomic-write-secret.nix`, with `store_init` and `store_login` run and +# `$clients` set to the digest directory. Needs `authelia`, `bao`, `script` +# (util-linux), `coreutils`, `grep` and `sed` on the caller's `path`. +{ }: +'' + # . Sets CLIENT_SECRET_VALUE. 0: the store holds a value. 1: it holds + # none. 2: it cannot tell (not logged in, down, or refusing the read). + client_secret_read() { + CLIENT_SECRET_VALUE="" + [ "$STORE_UP" = 1 ] || return 2 + if CLIENT_SECRET_VALUE="$(store_bao kv get -field=value "$1" 2>"$STORE_ERR")" \ + && [ -n "$CLIENT_SECRET_VALUE" ]; then + return 0 + fi + CLIENT_SECRET_VALUE="" + if grep -q '^No value found at ' "$STORE_ERR"; then + return 1 + fi + cat "$STORE_ERR" >&2 + return 2 + } + + # . Prints authelia's digest of it, the form `mint` would have + # written. The CLI takes a password only on a terminal prompt or in argv, + # and argv is world-readable in /proc, so the prompt is answered through a + # pty from stdin. The pty echoes the password into the pipe, and only the + # `Digest:` line leaves it. + client_secret_digest() { + printf '%s\n' "$1" \ + | timeout 60 script -qec "authelia crypto hash generate pbkdf2 --variant sha512 --no-confirm" /dev/null \ + | tr -d '\r' | sed -n 's/^Digest: *//p' + } + + # . Writes the digest of an + # existing secret; the secret itself is untouched. + client_secret_derive_digest() { + local id="$1" secret="$2" source="$3" digest + digest="$(client_secret_digest "$secret")" || digest="" + if [ -z "$digest" ]; then + echo "<3>oidc client $id: could not compute a digest from the secret in $source" >&2 + return 1 + fi + atomic_write_secret 0600 "" "$clients/$id.digest" "$digest" || return 1 + MINTED=1 + echo "oidc client $id: wrote the digest from the secret in $source; the secret is unchanged" + } + + # . Sets MINTED when it wrote a digest. + mint_client_secret() { + local id="$1" leaf="$2" dig="$clients/$1.digest" legacy="$clients/$1.secret" + local state=0 out secret digest + client_secret_read "$leaf" || state=$? + if [ "$state" = 1 ] && [ -s "$legacy" ]; then + # A file an earlier release left: its value goes into the store once, so + # the secret keeps its value. The file stays, unused. + printf '%s' "$(< "$legacy")" | store_bao kv put -cas=0 "$leaf" value=- >/dev/null 2>"$STORE_ERR" \ + || cat "$STORE_ERR" >&2 + state=0 + client_secret_read "$leaf" || state=$? + if [ "$state" = 1 ]; then + echo "oidc client $id: could not store the secret from $legacy in $leaf; not minting" >&2 + return 1 + fi + echo "oidc client $id: stored the secret from $legacy in the store" + fi + if [ "$state" = 0 ]; then + if [ -s "$dig" ]; then + chmod 0600 "$dig" + return 0 + fi + client_secret_derive_digest "$id" "$CLIENT_SECRET_VALUE" "the store at $leaf" + return + fi + if [ "$state" = 2 ]; then + if [ -s "$dig" ]; then + chmod 0600 "$dig" + echo "oidc client $id: the store cannot be read; keeping its digest" >&2 + return 0 + fi + echo "oidc client $id: the store cannot be read and there is no digest; not minting" >&2 + return 1 + fi + echo "oidc client $id: the store holds no secret at $leaf; minting one" + out="$(authelia crypto hash generate pbkdf2 --variant sha512 --random)" + secret="$(printf '%s' "$out" | sed -n 's/^Random Password: *//p')" + digest="$(printf '%s' "$out" | sed -n 's/^Digest: *//p')" + if [ -z "$secret" ] || [ -z "$digest" ]; then + echo "<3>authelia crypto hash generate produced no secret/digest for $id" >&2 + return 1 + fi + if ! printf '%s' "$secret" | store_bao kv put -cas=0 "$leaf" value=- >/dev/null 2>"$STORE_ERR"; then + cat "$STORE_ERR" >&2 + echo "could not create $leaf; reading it back in case another run created it" >&2 + fi + client_secret_read "$leaf" || return 1 + if [ "$CLIENT_SECRET_VALUE" != "$secret" ]; then + echo "oidc client $id: $leaf holds a value another run created; using it" + client_secret_derive_digest "$id" "$CLIENT_SECRET_VALUE" "the store at $leaf" + return + fi + atomic_write_secret 0600 "" "$dig" "$digest" || return 1 + MINTED=1 + echo "minted client secret for $id into the store" + } +'' diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index fa39ce436..edcc615d0 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -314,7 +314,7 @@ in # ./glue-hive-otel-authelia.nix restates this gate. waits = senderAuth && !fetches; secretWaitUnit = "hive-otel-client-secret"; - # Written by swarm-secret-publish in authelia's container. The nix half of + # Written by authelia's mint, in its container. The nix half of # `swarm_secret_client::queue::hive_client_path`. storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client"; atomicWriteSecret = import ./lib/atomic-write-secret.nix { }; @@ -786,8 +786,8 @@ in # This hive's OIDC client secret, out of the swarm secret store under the # hive's own leaf. Same shape as ./hive-matrix.nix's # `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and - # `storeRetry` retries until the publisher in authelia's container has - # written the secret, because the collector does not start without it. + # `storeRetry` retries until authelia's mint, in its container, has + # stored the secret, because the collector does not start without it. # # The collector re-reads `client_secret_file` on every token fetch, but # the file it reads is `LoadCredential`'s copy, made at start: a changed diff --git a/nix/host-modules/swarm-attic-options.nix b/nix/host-modules/swarm-attic-options.nix index 37848437f..ebe18010f 100644 --- a/nix/host-modules/swarm-attic-options.nix +++ b/nix/host-modules/swarm-attic-options.nix @@ -24,7 +24,7 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-attic-otel-oidc-client.nix`), the swarm collector admits it on the container's receiver (`swarm.otel.serviceProducers`), - and the secret publisher carries its secret to the store. + and authelia's mint stores its secret in the store. ''; }; diff --git a/nix/host-modules/swarm-authelia-service.nix b/nix/host-modules/swarm-authelia-service.nix index a6046de5f..4cde3bba4 100644 --- a/nix/host-modules/swarm-authelia-service.nix +++ b/nix/host-modules/swarm-authelia-service.nix @@ -373,8 +373,8 @@ in Read-only because three ends have to agree on it: authelia registers the client, the swarm collector admits it on the container's receiver - (`swarm.otel.serviceProducers`), and the secret publisher carries its - secret to the store, where the forwarder reads it back under the + (`swarm.otel.serviceProducers`), and authelia's mint stores its + secret in the store, where the forwarder reads it back under the publisher's identity. ''; }; diff --git a/nix/host-modules/swarm-authelia.nix b/nix/host-modules/swarm-authelia.nix index 39783827f..b52e878a3 100644 --- a/nix/host-modules/swarm-authelia.nix +++ b/nix/host-modules/swarm-authelia.nix @@ -212,7 +212,7 @@ let # no other collector shares it. The module requires a value. forwarderTelemetryPort = 8901; # A client this container registers and mints for itself. Its secret goes - # through the store like every other forwarder's: the publisher carries it + # through the store like every other forwarder's: the mint creates it # there and the forwarder reads it back under the same identity. forwarderClient = { id = forwarderClientId; @@ -227,8 +227,8 @@ let accessTokenSignedResponseAlg = "RS256"; }; - # The secret publisher runs in this container, beside the mint whose secrets - # it copies. It needs a store identity, and a reader is defined by holding a + # The mint runs in this container and creates its secrets in the store. It + # needs a store identity, and a reader is defined by holding a # certificate the store accepts, never by standing next to it — the rule # ./swarm-bao.nix's `deploy.bao` options block states in full. # `deploy.bao.enable` here would be the co-location assumption itself. @@ -237,7 +237,6 @@ let publisherCfg.enable && publisherCfg.baoClientCertFile != null && publisherCfg.baoClientKeyFile != null; - publisherUnit = "swarm-secret-publish"; forwarderFetchUnit = "swarm-journal-forwarder-oidc"; # The publisher's store identity, handed into the container as system # credentials (credential id → host source). The forwarder's fetch presents @@ -465,39 +464,33 @@ let '' ); - # The OIDC half of the first-boot generator, kept out of the script - # body so neither is read through the other's indentation. - oidcGenScript = lib.optionalString oidcEnabled '' - ${lib.optionalString (!publishes) '' - # The issuer key is the one secret here that is NOT interchangeable - # with a random blob: it *signs* id tokens, and every relying party - # verifies them against the public half served at `/jwks.json`. A - # symmetric secret cannot do that, so this one is an RSA pair. - # - # Rotating it invalidates every token already issued, which is why - # it is generated once and left alone — same reason as the session - # and storage keys above. - issuer=${lib.escapeShellArg stateDir}/oidc-issuer.key - if [ ! -s "$issuer" ]; then - openssl genrsa -out "$issuer" 4096 - echo "generated $issuer" - fi - chmod 0600 "$issuer" - ''} - # Each client's secret, minted once and kept as two files: the - # plaintext its relying party authenticates with, and the digest - # authelia compares against. The two live in different containers, - # so neither side can generate it alone — this is the only place - # that sees both. - # - # `--random` is why no plaintext ever reaches an argv: authelia - # generates the password itself and prints it beside its digest, - # so nothing has to be handed to a second process on a command - # line. - clients=${lib.escapeShellArg clientsDir} - mkdir -p "$clients" - chmod 0700 "$clients" + oidcClientSecrets = import ./lib/oidc-client-secrets.nix { }; + # Every client with a secret to mint: a native client is public and has none. + secretClients = lib.filter (c: c.kind != "native") cfg.oidc.clients; + + # Where a client's plaintext lives in the store: the path its reader fetches + # it from, which is `deploy.swarm-secret-publisher.secrets` for every client + # something reads. The rest, the bridge's own among them, sit with the + # services. + clientLeaf = + id: + let + listed = lib.findFirst (s: s.clientId == id) null publisherCfg.secrets; + in + if listed == null then "secret/swarm/services/${id}/oidc/client" else listed.path; + + # Each secret is kept as two halves: the plaintext its relying party + # authenticates with, and the digest authelia compares against. The two live + # in different containers, so neither side can generate it alone, and this is + # the only place that sees both. + # + # `--random` is why no plaintext reaches an argv: authelia generates the + # password itself and prints it beside its digest. + # + # Without a store identity the plaintext has nowhere else to go, so it stays + # in a file beside the digest. + fileMintScript = '' mint() { sec="$clients/$1.secret" dig="$clients/$1.digest" @@ -522,7 +515,45 @@ let ${lib.concatMapStrings (c: '' mint ${lib.escapeShellArg c.id} - '') (lib.filter (c: c.kind != "native") cfg.oidc.clients)} + '') secretClients} + ''; + + # The plaintext goes straight into the store and only the digest is written + # to disk. A digest on disk is enough for authelia to start while the store is + # down; a client with no digest cannot be rendered, so that run fails and + # retries (./lib/store-retry.nix). A client whose digest is missing gets it + # from the secret that exists, so nothing is minted again. + storeMintScript = '' + ${oidcClientSecrets} + ${lib.concatMapStrings (c: '' + mint_client_secret ${lib.escapeShellArg c.id} ${lib.escapeShellArg (clientLeaf c.id)} || exit $? + '') secretClients} + ''; + + # The OIDC half of the first-boot generator, kept out of the script + # body so neither is read through the other's indentation. + oidcGenScript = lib.optionalString oidcEnabled '' + ${lib.optionalString (!publishes) '' + # The issuer key is the one secret here that is NOT interchangeable + # with a random blob: it *signs* id tokens, and every relying party + # verifies them against the public half served at `/jwks.json`. A + # symmetric secret cannot do that, so this one is an RSA pair. + # + # Rotating it invalidates every token already issued, which is why + # it is generated once and left alone — same reason as the session + # and storage keys above. + issuer=${lib.escapeShellArg stateDir}/oidc-issuer.key + if [ ! -s "$issuer" ]; then + openssl genrsa -out "$issuer" 4096 + echo "generated $issuer" + fi + chmod 0600 "$issuer" + ''} + clients=${lib.escapeShellArg clientsDir} + mkdir -p "$clients" + chmod 0700 "$clients" + + ${if publishes then storeMintScript else fileMintScript} # Re-rendered every boot, deliberately: the secret is minted once, # but the metadata around it (a new redirect URI, a renamed client) @@ -537,10 +568,37 @@ let } > ${lib.escapeShellArg "${clientsFile}.tmp"} chmod 0600 ${lib.escapeShellArg "${clientsFile}.tmp"} mv ${lib.escapeShellArg "${clientsFile}.tmp"} ${lib.escapeShellArg clientsFile} + ${lib.optionalString publishes '' + # authelia reads its clients once, as it starts. + if [ -n "''${MINTED-}" ]; then + refresh_unit ${lib.escapeShellArg cfg.unit} ${lib.escapeShellArg clientsFile} + fi + ''} ''; storeRetry = import ./lib/store-retry.nix { }; + # The bridge's own client secret, in the memory-backed runtime directory of + # its unit, fetched from the store before each start. + bridgeSecretFile = "/run/swarm-authelia-bridge/client-secret"; + bridgeSecretFetch = '' + set -euo pipefail + + ${atomicWriteSecret} + ${storeSeededSecret} + ${oidcClientSecrets} + + store_init + store_login + if client_secret_read ${lib.escapeShellArg (clientLeaf bridgeClientId)}; then + value="$CLIENT_SECRET_VALUE" + else + echo "the store holds no readable secret for ${bridgeClientId}" >&2 + exit 1 + fi + atomic_write_secret 0600 "" ${lib.escapeShellArg bridgeSecretFile} "$value" + ''; + # authelia's keys, one leaf each under its container's name. The nix half of # `autheliaKeyLeaves` in ./swarm-bao.nix, which grants the publisher # `create` and `read` on exactly these. @@ -1019,19 +1077,19 @@ in imports = [ ../container-modules/swarm-container.nix ../container-modules/swarm-journal-forwarder.nix - ../container-modules/swarm-secret-publisher.nix (import ../container-modules/swarm-container-resolver.nix { inherit (networkCfg) bridgeIp; # The forwarder dials the swarm collector and authelia by name, - # and its fetch, the publisher and the key unit the store. + # and its fetch, the bridge's fetch, the key and secret units the + # store. dnsConsumers = [ cfg.unit ] ++ lib.optionals publishes [ "opentelemetry-collector.service" "${forwarderFetchUnit}.service" - "${publisherUnit}.service" "${unitName}-secrets.service" + "swarm-authelia-bridge.service" ]; }) caBundleModule @@ -1068,22 +1126,6 @@ in serviceConfig.LoadCredential = lib.attrNames storeFiles; }; - # Copies what the mint below leaves in `clientsDir` into the store, - # under the publisher's own identity, a system credential. Nothing - # orders authelia after it. - services.hyperhive.swarmSecretPublisher = lib.mkIf publishes { - enable = true; - clientSecretDir = clientsDir; - inherit (publisherCfg) secrets; - store = storeIdentityIn publisherUnit; - }; - systemd.services.${publisherUnit} = lib.mkIf publishes { - # A run before the mint would only skip every secret and wait - # for the path unit. - after = [ "${unitName}-secrets.service" ]; - serviceConfig.LoadCredential = lib.attrNames storeFiles; - }; - # The authelia binary itself, so an operator who gets a shell # in here can run `authelia crypto hash generate` to make a # password for the users file. Without it the container runs @@ -1121,7 +1163,9 @@ in ] ++ lib.optionals publishes [ deployCfg.bao.package + pkgs.gnugrep pkgs.systemd + pkgs.util-linux ]; # A store that cannot answer, with a key missing on disk, is a # failed run that retries (./lib/store-retry.nix). @@ -1189,6 +1233,10 @@ in # load-bearing at start, since nothing calls the bridge yet at # boot, but a clean dependency order beats a would-be-transient # failure on the first real request). + # + # With the store identity its client secret is read from the store + # into `bridgeSecretFile` before each start (`bridgeSecretFetch`). A + # store that does not answer fails the start, and the unit retries. systemd.services.swarm-authelia-bridge = { description = "swarm-authelia-bridge: the only writer of authelia's users database"; wantedBy = [ "multi-user.target" ]; @@ -1200,12 +1248,24 @@ in "${unitName}-secrets.service" "${unitName}.service" ]; + path = lib.optionals publishes [ + pkgs.coreutils + pkgs.gnugrep + deployCfg.bao.package + ]; + preStart = lib.mkIf publishes bridgeSecretFetch; + startLimitBurst = lib.mkIf publishes storeRetry.startLimitBurst; + startLimitIntervalSec = lib.mkIf publishes storeRetry.startLimitIntervalSec; serviceConfig = { ExecStart = "${deployCfg.authelia.bridgePackage}/bin/swarm-authelia-bridge"; User = unitName; Group = socketGroup; Restart = "on-failure"; - RestartSec = "5s"; + RestartSec = if publishes then storeRetry.serviceConfig.RestartSec else "5s"; + } + // lib.optionalAttrs publishes { + RuntimeDirectory = "swarm-authelia-bridge"; + LoadCredential = lib.attrNames storeFiles; }; environment = { SWARM_AUTHELIA_BRIDGE_SOCKET = bridgeSocket; @@ -1230,17 +1290,27 @@ in # uses, so there is one idiom rather than two. SWARM_AUTHELIA_BRIDGE_INTROSPECTION_URL = "${cfg.url}/api/oidc/introspection"; SWARM_AUTHELIA_BRIDGE_CLIENT_ID = bridgeClientId; - # Minted by `${unitName}-secrets`'s `mint` loop (it iterates - # every entry in `cfg.oidc.clients`, which now always - # includes `bridgeClient`) — same file this container's own - # `renderClient` reads the digest half of. - SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE = "${clientsDir}/${bridgeClientId}.secret"; + # Minted by `${unitName}-secrets` (it iterates every entry in + # `cfg.oidc.clients`, which always includes `bridgeClient`). + SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE = + if publishes then bridgeSecretFile else "${clientsDir}/${bridgeClientId}.secret"; # The one client the bridge answers. Every other active token # (each hive and its agents hold one) is refused, on every # request, since `/bridge/` is reachable from any host. SWARM_AUTHELIA_BRIDGE_CONTROLLER_CLIENT_ID = hyperhiveCfg.swarm.controller.queueClientId; SWARM_AUTHELIA_BRIDGE_OPERATOR_GROUP = operatorGroup; - }; + } + // lib.optionalAttrs publishes ( + let + store = storeIdentityIn "swarm-authelia-bridge"; + in + { + BAO_ADDR = store.addr; + BAO_CLIENT_CERT = store.clientCertFile; + BAO_CLIENT_KEY = store.clientKeyFile; + } + // lib.optionalAttrs (store.caFile != null) { BAO_CACERT = store.caFile; } + ); }; # Upstream's unit is `ProtectSystem=strict`, so the socket directory diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 8a836acb6..d514b3385 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -539,14 +539,14 @@ let } ''; - # The identity that copies authelia's minted OIDC client secrets into the - # store. Named outside `hive-*` for the same reason as the controller's: the + # The identity authelia's mint creates its OIDC client secrets in the store + # with. Named outside `hive-*` for the same reason as the controller's: the # controller may create policies under that prefix, and a policy it can # rewrite is not a constraint on anything. secretPublisherPolicyName = "swarm-secret-publisher"; secretPublisherCn = baoDeploy.secretPublisherCommonName; - # Four grants and six denials, and every narrowing in each is load-bearing. + # Five grants and six denials, and every narrowing in each is load-bearing. # # `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than # written by the caller — same trap as the controller's grant above. @@ -559,9 +559,9 @@ let # here too — a grant widens when a path gains a WRITER, not when a kind is # declared. # - # Write-only but for its forwarder's client secret and authelia's keys, read at - # exact leaves: a read on a prefix would let a file-copier recover every - # hive's credentials. The forwarder's stanza repeats `create`/`update`. + # Reads only the `oidc/client` leaves it mints, the controller's client and + # authelia's keys: a read on a prefix would recover every hive's credentials. + # Those stanzas repeat `create`/`update`: a narrower path replaces the prefix's. # # The queue callout's three seeds sit under `swarm/services/` and are no # secret this principal copies, so their exact paths, which outrank the @@ -618,10 +618,14 @@ let } path "${credentialMountPath}/data/${controllerClientLeaf}" { - capabilities = ["create", "update"] + capabilities = ["create", "update", "read"] } - path "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.authelia.otel.clientId}/oidc/client" { + path "${credentialMountPath}/data/swarm/hives/+/oidc/client" { + capabilities = ["create", "update", "read"] + } + + path "${credentialMountPath}/data/swarm/services/+/oidc/client" { capabilities = ["create", "update", "read"] } '' @@ -681,7 +685,7 @@ let swarmAppserviceRegistrationLeaf = "swarm/controller/swarm-controller/matrix/appservice-registration"; # The controller's own OIDC client secret, the nix half of - # `swarm_secret_client::queue::controller_client_path`. The publisher writes + # `swarm_secret_client::queue::controller_client_path`. Authelia's mint writes # it; the controller reads it and holds no other copy. controllerClientLeaf = "swarm/controller/swarm-controller/oidc/client"; @@ -767,7 +771,7 @@ let # and read like the API key above. classifierAcpKeyLeaf = "swarm/services/review-classifier/acp-provider"; - # The UI's OIDC client secret. The publisher writes it; the granter reads it + # The UI's OIDC client secret. Authelia's mint writes it; the granter reads it # into the `oidc` auth method's config, which is the only copy bao uses. baoUiClientLeaf = "swarm/services/${cfg.ui.oidc.clientId}/oidc/client"; @@ -1184,7 +1188,7 @@ let grafanaSecretKeyLeaf = "swarm/services/${hyperhiveCfg.swarm.grafana.oidc.clientId}/secret-key"; # authelia's jwt, session, storage-encryption, OIDC HMAC and OIDC issuer keys, - # which only the secret publisher in authelia's container creates and reads. + # which only `authelia-swarm-secrets` in authelia's container creates and reads. # The nix half of `keyLeaf` in ./swarm-authelia.nix. `create` without # `update`, so a stored key is never replaced: the storage-encryption key # encrypts authelia's database and the issuer key signs every id token. @@ -1668,7 +1672,7 @@ let # own and lands it in a collector's container — because that route is # proven and there is no second one worth inventing. # - # Where the publisher in authelia's container leaves it. A swarm service's + # Where authelia's mint, in its container, stores it. A swarm service's # client is registered once for the whole swarm, so its secret is one value. # The `swarm-forwarder-oidc` role's grant is this one path, so any other # answers 403 however correct it looks. @@ -1979,8 +1983,8 @@ in example = "swarm-secret-publisher.svc"; description = '' Subject the store's secret-publisher cert-auth role accepts — the - identity that copies authelia's minted OIDC client secrets into the - store so a hive that does not host authelia can read its own. + identity authelia's mint creates its OIDC client secrets in the + store with, so a hive that does not host authelia can read its own. A **second** identity rather than reuse of {option}`services.hyperhive.deploy.bao.controllerCommonName`: the @@ -4090,7 +4094,7 @@ in ;; esac - # Published by the secret publisher in authelia's container, which + # Stored by authelia's mint, in its container, which # this boot does not wait on, so absence is retried. if ! secret="$(bao kv get -field=value ${lib.escapeShellArg "${credentialMountPath}/${baoUiClientLeaf}"} 2>"$err")"; then echo ${lib.escapeShellArg "the store did not return ${credentialMountPath}/${baoUiClientLeaf}: the UI's OIDC client secret is not published yet."} >&2 diff --git a/nix/host-modules/swarm-builder.nix b/nix/host-modules/swarm-builder.nix index d98b87e54..2ab058e54 100644 --- a/nix/host-modules/swarm-builder.nix +++ b/nix/host-modules/swarm-builder.nix @@ -105,8 +105,8 @@ in description = '' The OAuth2 client id the swarm builder presents to the swarm queue. Read-only: the host running authelia registers it, the queue's - auth-callout responder grants it the build subject, and the secret - publisher carries its secret to the store, all from this one value. + auth-callout responder grants it the build subject, and + authelia's mint stores its secret in the store, all from this one value. ''; }; diff --git a/nix/host-modules/swarm-ci-metrics.nix b/nix/host-modules/swarm-ci-metrics.nix index 49a68793f..0cefb9d72 100644 --- a/nix/host-modules/swarm-ci-metrics.nix +++ b/nix/host-modules/swarm-ci-metrics.nix @@ -83,7 +83,7 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-ci-metrics-otel-oidc-client.nix`), the swarm collector admits it on the poller's receiver (`swarm.otel.serviceProducers`), and - the secret publisher carries its secret to the store. + authelia's mint stores its secret in the store. ''; }; diff --git a/nix/host-modules/swarm-ci-runner.nix b/nix/host-modules/swarm-ci-runner.nix index a67cd9c56..50ea4264e 100644 --- a/nix/host-modules/swarm-ci-runner.nix +++ b/nix/host-modules/swarm-ci-runner.nix @@ -139,7 +139,7 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-ci-runner-otel-oidc-client.nix`), the swarm collector admits it on the container's receiver (`swarm.otel.serviceProducers`), - and the secret publisher carries its secret to the store. + and authelia's mint stores its secret in the store. ''; }; diff --git a/nix/host-modules/swarm-claude-login.nix b/nix/host-modules/swarm-claude-login.nix index c7d3b7826..c38eff808 100644 --- a/nix/host-modules/swarm-claude-login.nix +++ b/nix/host-modules/swarm-claude-login.nix @@ -96,8 +96,8 @@ in description = '' The OAuth2 client id the claude login minter presents to the swarm queue. Read-only: the host running authelia registers it, the queue's - auth-callout responder grants it the login subject, and the secret - publisher carries its secret to the store, all from this one value. + auth-callout responder grants it the login subject, and + authelia's mint stores its secret in the store, all from this one value. ''; }; diff --git a/nix/host-modules/swarm-controller.nix b/nix/host-modules/swarm-controller.nix index 374363daf..f0ddfea29 100644 --- a/nix/host-modules/swarm-controller.nix +++ b/nix/host-modules/swarm-controller.nix @@ -182,8 +182,8 @@ let queueClientId = cfg.queueClientId; # No secret here. The daemon reads its client secret from the store at - # start, under `baoEnv`'s identity, from the path `swarm-secret-publish` - # writes it to in authelia's container (`queue_identity.rs`). + # start, under `baoEnv`'s identity, from the path authelia's mint + # writes in its container (`queue_identity.rs`). # # Every value here comes from an option rather than from what happens to # run on this host. The queue is not optional for a controller, but diff --git a/nix/host-modules/swarm-grafana-service.nix b/nix/host-modules/swarm-grafana-service.nix index a0aa6c779..710efaeeb 100644 --- a/nix/host-modules/swarm-grafana-service.nix +++ b/nix/host-modules/swarm-grafana-service.nix @@ -50,8 +50,8 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-grafana-otel-oidc-client.nix`), the swarm collector admits it on the container's receiver - (`swarm.otel.serviceProducers`), and the secret publisher carries - its secret to the store. + (`swarm.otel.serviceProducers`), and authelia's mint stores + its secret in the store. ''; }; diff --git a/nix/host-modules/swarm-grafana.nix b/nix/host-modules/swarm-grafana.nix index ee1b864fc..eb2bbbaf7 100644 --- a/nix/host-modules/swarm-grafana.nix +++ b/nix/host-modules/swarm-grafana.nix @@ -351,7 +351,7 @@ let "/var/lib/grafana-metrics/bearer" ]; - # Where the publisher in authelia's container leaves this client's secret. A + # Where authelia's mint, in its container, stores this client's secret. A # swarm service's client is registered once for the whole swarm, so its # secret is one value and not one per hive. Both ends compose this from the # same swarm-wide client id, so a rename cannot leave one of them behind. @@ -719,8 +719,8 @@ in ++ map (p: "r ${p}") legacyHostSecretPaths; # THE delivery unit — one route, in every deployment. The secret authelia - # minted arrives out of the swarm secret store, which the publisher in - # authelia's container wrote it into, and that is true whether authelia is + # minted arrives out of the swarm secret store, which authelia's mint, + # in its container, wrote it into, and that is true whether authelia is # a network away or in the container next door. # # 🩸 No second unit copies the plaintext out of authelia's tree @@ -737,7 +737,7 @@ in # there" into a value. # # ⚠️ An ABSENT secret is "not published yet", not a fault of this host's. - # The publisher runs in authelia's container and authelia mints on its + # The mint runs in authelia's container and authelia mints on its # first boot, so early in a swarm's life there is legitimately nothing to # read. # Nothing here writes a stand-in: there is no such thing as a locally valid diff --git a/nix/host-modules/swarm-notify-router.nix b/nix/host-modules/swarm-notify-router.nix index 06e8d4927..83e43692b 100644 --- a/nix/host-modules/swarm-notify-router.nix +++ b/nix/host-modules/swarm-notify-router.nix @@ -85,8 +85,8 @@ in description = '' The OAuth2 client id the notification router presents to the swarm queue. Read-only: the host running authelia registers it, the queue's - auth-callout responder grants it the router's subjects, and the secret - publisher carries its secret to the store, all from this one value. + auth-callout responder grants it the router's subjects, and + authelia's mint stores its secret in the store, all from this one value. ''; }; diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 58982a00e..31063e98d 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -261,7 +261,7 @@ let # regression rather than a choice anyone made. hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; - # Where the publisher in authelia's container leaves this client's secret — + # Where authelia's mint, in its container, stores this client's secret — # composed from the same swarm-wide `clientId` the registration in # ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of # them behind. The same prefix ./swarm-grafana.nix reads its own client @@ -554,8 +554,8 @@ in # ./swarm-grafana.nix has for its own client. # THE delivery unit — one route, in every deployment. The secret authelia - # minted arrives out of the swarm secret store, which the publisher in - # authelia's container wrote it into, whether authelia is a network away or + # minted arrives out of the swarm secret store, which authelia's mint, + # in its container, wrote it into, whether authelia is a network away or # in the container next door. # # 🩸 No unit here copies the plaintext directly out of authelia's diff --git a/nix/host-modules/swarm-review-classifier.nix b/nix/host-modules/swarm-review-classifier.nix index 02a110a1d..05ccd87fa 100644 --- a/nix/host-modules/swarm-review-classifier.nix +++ b/nix/host-modules/swarm-review-classifier.nix @@ -119,8 +119,8 @@ in description = '' The OAuth2 client id the review classifier presents to the swarm queue. Read-only: the host running authelia registers it, the queue's - auth-callout responder grants it the try-out subject, and the secret - publisher carries its secret to the store, all from this one value. + auth-callout responder grants it the try-out subject, and + authelia's mint stores its secret in the store, all from this one value. ''; }; diff --git a/nix/host-modules/swarm-secret-publisher.nix b/nix/host-modules/swarm-secret-publisher.nix index 0c78a4662..b16bbe2d7 100644 --- a/nix/host-modules/swarm-secret-publisher.nix +++ b/nix/host-modules/swarm-secret-publisher.nix @@ -1,8 +1,7 @@ -# What a host decides about the swarm's secret publisher: whether it runs, and -# the store identity it presents, plus the list of secrets it copies. The unit -# itself is ../container-modules/swarm-secret-publisher.nix, inside authelia's -# container beside the mint whose secrets it copies; ./swarm-authelia.nix -# imports it there and hands the identity in as system credentials. +# What a host decides about authelia's store identity: whether authelia's +# container holds one, and the list of secrets it creates in the store with it. +# The mint is `authelia-swarm-secrets` in ./swarm-authelia.nix, inside +# authelia's container, which hands the identity in as system credentials. { lib, config, @@ -106,11 +105,12 @@ in type = lib.types.bool; default = false; description = '' - Publish the OIDC client secrets this host mints into the swarm's - secret store, so a hive that does not run authelia, a swarm service - or the controller can read its own — from wherever it runs, this host - included. The unit, `swarm-secret-publish`, runs inside authelia's - container, so it exists only where authelia runs. + Keep the OIDC client secrets this host mints in the swarm's secret + store, so a hive that does not run authelia, a swarm service or the + controller can read its own — from wherever it runs, this host + included. Authelia's mint, `authelia-swarm-secrets` inside its + container, creates each secret in the store and writes only its + digest to disk, so the unit exists only where authelia runs. Defaults to whether this host mints the OIDC secrets (set by `glue-secret-publisher-authelia.nix`), which is the only half of the @@ -170,9 +170,10 @@ in internal = true; default = secrets; description = '' - Every secret the publisher copies: the minted client's id and the store - path its reader fetches. Derived from `swarm.*` alone, so it is the same list on every - host; ./swarm-authelia.nix hands it to the unit in its container. + Where the mint creates each client's secret: the minted client's id and + the store path its reader fetches. Derived from `swarm.*` alone, so it + is the same list on every host. A client missing here gets + `secret/swarm/services//oidc/client`, as the bridge's does. ''; }; }; diff --git a/nix/host-modules/swarm-victorialogs-service.nix b/nix/host-modules/swarm-victorialogs-service.nix index 31eddc5c5..59819b9bf 100644 --- a/nix/host-modules/swarm-victorialogs-service.nix +++ b/nix/host-modules/swarm-victorialogs-service.nix @@ -42,8 +42,8 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-victorialogs-otel-oidc-client.nix`), the swarm collector admits it on the container's receiver - (`swarm.otel.serviceProducers`), and the secret publisher carries - its secret to the store. + (`swarm.otel.serviceProducers`), and authelia's mint stores + its secret in the store. ''; }; diff --git a/nix/host-modules/swarm-victoriametrics-service.nix b/nix/host-modules/swarm-victoriametrics-service.nix index dea2bf8ae..08fc87c97 100644 --- a/nix/host-modules/swarm-victoriametrics-service.nix +++ b/nix/host-modules/swarm-victoriametrics-service.nix @@ -42,8 +42,8 @@ in Read-only because three hosts have to agree on it: authelia registers the client (`glue-victoriametrics-otel-oidc-client.nix`), the swarm collector admits it on the container's receiver - (`swarm.otel.serviceProducers`), and the secret publisher carries - its secret to the store. + (`swarm.otel.serviceProducers`), and authelia's mint stores + its secret in the store. ''; }; diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index e3dd6dcd1..08c387e2e 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -418,7 +418,7 @@ in A URL alone does not publish. This hive also needs its own store identity ({option}`services.hyperhive.deploy.bao.clientCertFile`), under which hive-c0re reads its `hive-` client secret - from the swarm's secret store; `swarm-secret-publish`, in authelia's + from the swarm's secret store; authelia's mint, in its container, puts it there. Without an identity, or with this set to null, this hive computes its own readiness as always and offers it to nobody, and the swarm controller reports it diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index c1b43d95e..9b56c63df 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -418,28 +418,31 @@ let # The whole point of a second principal. The two prefixes and one leaf it # publishes to and not `swarm/`, so it cannot touch an agent's # credentials or the appservice token beside the controller's client; - # and `read` on six exact leaves alone: the client secret of the journal forwarder - # that shares its identity in authelia's container, and authelia's five keys - # (the next case), so a unit whose job is copying a file cannot recover - # what is already there. Pinned as the full capability list per path, - # because an added capability is exactly what a presence check misses. - # The forwarder's `read` is found at its leaf, and the six together are - # counted, so a read on any other path fails the count. - name = "the publisher's grant writes the hive and service prefixes and the controller's client, and reads its forwarder's client and authelia's keys alone"; + # and `read` on the `oidc/client` leaves it mints alone (the controller's + # exactly, each hive's and each service's by one `+` segment) and + # authelia's five keys (the next case), so a unit whose job is minting + # cannot recover what is already there beyond its own secrets. Pinned as + # the full capability list per path, because an added capability is + # exactly what a presence check misses. The `oidc/client` stanzas repeat + # `create` and `update`, since the narrower path replaces the prefix's + # capabilities. The eight reads are counted, so a read on any other path + # fails the count. + name = "the publisher's grant writes the hive and service prefixes and the controller's client, and reads the oidc client leaves and authelia's keys alone"; ok = let s = baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy.script; in lib.hasInfix "path \"secret/data/swarm/hives/*\" {\n capabilities = [\"create\", \"update\"]" s && lib.hasInfix "path \"secret/data/swarm/services/*\" {\n capabilities = [\"create\", \"update\"]" s - && lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/oidc/client\" {\n capabilities = [\"create\", \"update\"]\n}" s + && lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/oidc/client\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}" s + && lib.hasInfix "path \"secret/data/swarm/hives/+/oidc/client\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}" s + && lib.hasInfix "path \"secret/data/swarm/services/+/oidc/client\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}" s && !(lib.hasInfix "secret/data/swarm/controller/*" s) && !(lib.hasInfix "appservice-token" s) && !(lib.hasInfix "secret/data/swarm/agents" s) && !(lib.hasInfix "secret/data/swarm/*" s) && !(lib.hasInfix "sys/policies/acl" s) - && lib.hasInfix "path \"secret/data/swarm/services/swarm-authelia-collector/oidc/client\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}" s - && lib.length (lib.splitString "\"read\"" s) == 7 + && lib.length (lib.splitString "\"read\"" s) == 9 && !(lib.hasInfix "\"list\"" s); } { @@ -2414,7 +2417,7 @@ let && !(lib.hasInfix "appservice-registration" baoGrantHere.systemd.services.swarm-bao-controller-policy.script); } { - # The controller's own OIDC client secret, which the publisher writes and + # The controller's own OIDC client secret, which authelia's mint writes and # the controller reads at start. Pinned as the whole stanza, so an added # capability fails. name = "the controller reads its own client secret and cannot write it"; diff --git a/nix/module-eval/bao-otel-collector.nix b/nix/module-eval/bao-otel-collector.nix index f0edb1eeb..ddae66801 100644 --- a/nix/module-eval/bao-otel-collector.nix +++ b/nix/module-eval/bao-otel-collector.nix @@ -421,7 +421,7 @@ let # for: the reader's grant is one path under `services`, so a secret # filed under the hive that happens to run the store would be refused # rather than missing, however correct the path reads. - name = "the forwarder's secret is read from the prefix the publisher writes"; + name = "the forwarder's secret is read from the prefix the mint writes"; ok = let s = baoWithCollector.systemd.services.swarm-bao-forwarder-oidc.script; diff --git a/nix/module-eval/builder.nix b/nix/module-eval/builder.nix index e7e402440..76f67a18e 100644 --- a/nix/module-eval/builder.nix +++ b/nix/module-eval/builder.nix @@ -311,7 +311,7 @@ let ok = lib.hasInfix "--builder-client swarm-builder" natsHere.containers.swarm-nats.config.systemd.services.swarm-nats-auth.serviceConfig.ExecStart; } { - name = "the builder's client is registered as a machine, and the publisher carries its secret"; + name = "the builder's client is registered as a machine, and the mint stores its secret"; ok = let c = lib.findFirst ( @@ -321,7 +321,7 @@ let c != null && c.kind == "machine" && lib.hasInfix "secret/swarm/services/swarm-builder/oidc/client" ( - autheliaHere.containers.swarm-authelia.config.systemd.services.swarm-secret-publish.script + autheliaHere.containers.swarm-authelia.config.systemd.services.authelia-swarm-secrets.script ); } { diff --git a/nix/module-eval/ci-metrics.nix b/nix/module-eval/ci-metrics.nix index 52a41ad2e..a38245793 100644 --- a/nix/module-eval/ci-metrics.nix +++ b/nix/module-eval/ci-metrics.nix @@ -95,7 +95,7 @@ let && !(bare.systemd.services ? swarm-ci-metrics-ca-bundle); } { - name = "the poller's client is registered as a machine, and the publisher carries its secret"; + name = "the poller's client is registered as a machine, and the mint stores its secret"; ok = let c = lib.findFirst ( @@ -106,7 +106,7 @@ let && c.kind == "machine" && c.audience == [ clientId ] && lib.hasInfix "secret/swarm/services/${clientId}/oidc/client" ( - allLocal.containers.swarm-authelia.config.systemd.services.swarm-secret-publish.script + allLocal.containers.swarm-authelia.config.systemd.services.authelia-swarm-secrets.script ); } ]; diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index ea25aa283..471b35986 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -537,7 +537,11 @@ let ] && !(lib.hasInfix "client_secret" client) && !(lib.hasInfix "mint swarmctl" s) - && lib.hasInfix "mint swarm-authelia-bridge" s; + && !(lib.hasInfix "mint_client_secret swarmctl" s) + && ( + lib.hasInfix "mint swarm-authelia-bridge" s + || lib.hasInfix "mint_client_secret swarm-authelia-bridge" s + ); } { # swarm-controller checks no caller itself: this location's authelia diff --git a/nix/module-eval/grafana.nix b/nix/module-eval/grafana.nix index 617ffad73..783df8bf8 100644 --- a/nix/module-eval/grafana.nix +++ b/nix/module-eval/grafana.nix @@ -315,7 +315,7 @@ let # refused rather than empty, however correct it reads. The negative arm is # the rename this is exposed to — a secret filed under the hive that runs # the service instead of under the service itself. - name = "grafana's OIDC secret is read from the prefix the publisher writes"; + name = "grafana's OIDC secret is read from the prefix the mint writes"; ok = let s = grafanaRemoteAuthelia.systemd.services.swarm-bao-grafana-oidc.script; diff --git a/nix/module-eval/hive-otel.nix b/nix/module-eval/hive-otel.nix index ec0398c70..658f5fed4 100644 --- a/nix/module-eval/hive-otel.nix +++ b/nix/module-eval/hive-otel.nix @@ -352,14 +352,15 @@ let { # The writer and the reader spell the path independently; this is the # one place that sees both rendered. - name = "the hive collector reads the path the secret publisher writes"; + name = "the hive collector reads the path the mint writes"; ok = let fetch = builtins.unsafeDiscardStringContext (hiveOtelFetch hiveOtelAllLocal).script; - publish = builtins.unsafeDiscardStringContext hiveOtelAllLocal.containers.swarm-authelia.config.systemd.services.swarm-secret-publish.script; + publish = builtins.unsafeDiscardStringContext hiveOtelAllLocal.containers.swarm-authelia.config.systemd.services.authelia-swarm-secrets.script; path = "secret/swarm/hives/h1/oidc/client"; in - lib.hasInfix "bao kv get -field=value ${path}" fetch && lib.hasInfix "bao kv put ${path} " publish; + lib.hasInfix "bao kv get -field=value ${path}" fetch + && lib.hasInfix "mint_client_secret hive-h1 ${path}" publish; } { # The store's units exist only on its host. The PKI unit is wanted, never diff --git a/nix/module-eval/secret-publisher.nix b/nix/module-eval/secret-publisher.nix index cdc6b169a..115095c51 100644 --- a/nix/module-eval/secret-publisher.nix +++ b/nix/module-eval/secret-publisher.nix @@ -24,7 +24,7 @@ let # The IdP and the store on one machine: the shape where minted plaintext and # a store identity are both present without an operator placing anything. - # Two hives in the roster, because the publisher walks it — an arm written + # Two hives in the roster, because the mint walks it — an arm written # against a single-hive fixture passes on a hardcoded name. secretPublisherHere = hive { deploy.bao.enable = true; @@ -73,47 +73,45 @@ let # pair is what makes the reader's absence arm mean anything. inherit (shared) baoWithMatrix; - # The publisher runs inside authelia's container. + # The mint runs inside authelia's container. inAuthelia = m: m.containers.${m.services.hyperhive.swarm.authelia.machine}.config; cases = [ { # Each hive's hive-c0re reads this exact path # (`swarm_secret_client::queue::hive_client_path`, pinned by that crate's # own test), for every hive in the roster, not only the one this host is. - name = "the publisher writes every hive's own client under that hive's prefix"; + name = "the mint stores every hive's own client under that hive's prefix"; ok = let - s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script; + s = (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script; in - lib.all - (h: lib.hasInfix "secret/swarm/hives/${h}/oidc/client" s && lib.hasInfix "/hive-${h}.secret" s) - [ - "h1" - "h2" - ]; + lib.all (h: lib.hasInfix "mint_client_secret hive-${h} secret/swarm/hives/${h}/oidc/client" s) [ + "h1" + "h2" + ]; } { # The controller reads this exact path # (`swarm_secret_client::queue::controller_client_path`, pinned by that # crate's own test). Under `controller/`, which no hive reads: this # client may read every hive's status. - name = "the publisher writes the controller's client under controller/, never services/"; + name = "the mint stores the controller's client under controller/, never services/"; ok = let - s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script; + s = (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script; in lib.hasInfix "secret/swarm/controller/swarm-controller/oidc/client" s - && lib.hasInfix "/swarm-controller.secret" s + && lib.hasInfix "mint_client_secret swarm-controller secret/swarm/controller/swarm-controller/oidc/client" s && !(lib.hasInfix "secret/swarm/services/swarm-controller" s); } { # Both ends of a wire nothing at eval time carries end to end: the - # publisher in authelia's container writes the path the reader on + # mint in authelia's container writes the path the reader on # Grafana's host reads, and the two files agree only because both compose # it from the same swarm-wide client id. - name = "the publisher writes the swarm service path grafana reads"; + name = "the mint stores the swarm service path grafana reads"; ok = lib.hasInfix "secret/swarm/services/swarm-grafana/oidc/client" ( - (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script + (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script ); } { @@ -180,30 +178,27 @@ let } { # The other half of the queue's, the homeserver's and the forge's one - # route: the publisher on an IdP host that runs none of them carries each + # route: the mint on an IdP host that runs none of them stores each # secret to the path its reader fetches. - name = "the publisher on an authelia host carries the queue's, homeserver's and forge's secrets"; + name = "the mint on an authelia host stores the queue's, homeserver's and forge's secrets"; ok = let - s = (inAuthelia secretPublisherRemote).systemd.services.swarm-secret-publish.script; + s = (inAuthelia secretPublisherRemote).systemd.services.authelia-swarm-secrets.script; in - lib.hasInfix "secret/swarm/services/swarm-nats/oidc/client" s - && lib.hasInfix "/swarm-nats.secret" s - && lib.hasInfix "secret/swarm/services/tuwunel/oidc/client" s - && lib.hasInfix "/tuwunel.secret" s + lib.hasInfix "mint_client_secret swarm-nats secret/swarm/services/swarm-nats/oidc/client" s + && lib.hasInfix "mint_client_secret tuwunel secret/swarm/services/tuwunel/oidc/client" s && !secretPublisherRemote.services.hyperhive.deploy.forgejo.enable - && lib.hasInfix "secret/swarm/services/forgejo/oidc/client" s - && lib.hasInfix "/forgejo.secret" s; + && lib.hasInfix "mint_client_secret forgejo secret/swarm/services/forgejo/oidc/client" s; } { # Both ends of a wire nothing at eval time carries end to end: the - # publisher in authelia's container writes the path the reader on the + # mint in authelia's container writes the path the reader on the # collector's host reads, and the two files agree only because both # compose it from the same swarm-wide client id. `secretPublisherHere` # already grew this client when `serviceClientIds` did. - name = "the publisher writes the swarm service path the collector reads"; + name = "the mint stores the swarm service path the collector reads"; ok = lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" ( - (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script + (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script ); } { @@ -212,22 +207,22 @@ let # registers and mints for, so every layer above looks complete while # the one hop that CARRIES the value skips it and the reader on the # store's host waits forever on a path nothing writes. - name = "the publisher carries the store forwarder's secret too"; + name = "the mint stores the store forwarder's secret too"; ok = lib.hasInfix "secret/swarm/services/swarm-bao-collector/oidc/client" ( - (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script + (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script ); } { # The forwarders in the CI runner's, Grafana's, the log store's, the # forge's, the homeserver's and authelia's containers read their secrets # from these paths, and the metrics store's host reads its container's. - name = "the publisher carries the CI runner, Grafana, log store, cache, forge, homeserver, metrics store and authelia forwarders' secrets"; + name = "the mint stores the CI runner, Grafana, log store, cache, forge, homeserver, metrics store and authelia forwarders' secrets"; ok = lib.all ( id: lib.hasInfix "secret/swarm/services/${id}/oidc/client" ( - (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script + (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script ) ) [ @@ -244,7 +239,7 @@ let { # The review classifier reads its queue client secret from the store # inside its container; authelia registers it on every IdP host. - name = "the review classifier's client is registered as a machine, and the publisher carries its secret"; + name = "the review classifier's client is registered as a machine, and the mint stores its secret"; ok = let c = lib.findFirst ( @@ -254,15 +249,15 @@ let c != null && c.kind == "machine" && lib.hasInfix "secret/swarm/services/swarm-review-classifier/oidc/client" ( - (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script + (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script ); } { # The store's UI login: authelia registers the client with the callback - # the store's `oidc` role allows, and the publisher carries its secret + # the store's `oidc` role allows, and the mint stores its secret # to the path `swarm-bao-operator-viewer-policy` reads. Not registered # where authelia is not. - name = "the store UI's client is registered interactive at its callback, and the publisher carries its secret"; + name = "the store UI's client is registered interactive at its callback, and the mint stores its secret"; ok = let ui = lib.findFirst ( @@ -273,7 +268,7 @@ let && ui.kind == "interactive" && ui.redirectUris == [ "https://bao-ui.t.local/ui/vault/auth/oidc/oidc/callback" ] && lib.hasInfix "secret/swarm/services/swarm-bao-ui/oidc/client" ( - (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script + (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script ) && !(lib.any ( c: c.id == "swarm-bao-ui" @@ -307,42 +302,43 @@ let && !bare.services.hyperhive.deploy.swarm-secret-publisher.enable; } { - # The one security property of this unit, and why its push cannot be - # rewritten into the obvious shape: `bao` is an external binary, so an - # argument is world-readable in /proc for the life of the call. - # `value=@` hands it the path and bao opens the file itself. + # The one security property of the mint's write: `bao` is an external + # binary, so an argument is world-readable in /proc for the life of the + # call. `value=-` makes bao read the value from stdin. # - # The second arm is what makes the first mean anything — `value=@` can - # sit one line above a command substitution that put the plaintext in - # argv anyway. + # The arms below the real script are what make it mean anything: a + # `value=-` elsewhere in the text can sit beside a put that passes the + # plaintext as an argument. # # ⚠️ Comments are stripped first, and that is not tidiness. A `script` - # renders its own comments into the text, and this unit's comments name - # the hazard verbatim so the next editor does not reintroduce it. Without - # the strip this case reads that warning and fails — a check the artifact - # defeats by DESCRIBING the thing it is checked for. - name = "the publisher hands bao the secret's path, never the secret"; + # renders its own comments into the text, and a comment can name the + # hazard verbatim. Without the strip this case would read that warning + # and fail — a check the artifact defeats by DESCRIBING the thing it is + # checked for. + name = "the mint hands bao the secret on stdin, never as an argument"; ok = let - s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script; - code = lib.concatStringsSep "\n" ( - lib.filter (l: builtins.match "[[:space:]]*#.*" l == null) (lib.splitString "\n" s) - ); + code = s: lib.filter (l: builtins.match "[[:space:]]*#.*" l == null) (lib.splitString "\n" s); + puts = s: lib.filter (l: lib.hasInfix "kv put" l) (code s); + onStdin = s: puts s != [ ] && lib.all (l: lib.hasInfix "value=-" l) (puts s); in - lib.hasInfix "value=@" code && !(lib.hasInfix "$(cat" code); + onStdin (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script + && !(onStdin "printf '%s' \"$secret\" | bao kv put -cas=0 x value=\"$secret\"") + && !(onStdin "# kv put x value=-\nbao kv put x value=@/tmp/f") + && onStdin "printf '%s' \"$secret\" | bao kv put -cas=0 x value=-"; } { # No hive holds a shared queue credential for its agents, so authelia - # mints no agent client and the publisher writes no such path. The - # control: the same script writes each hive's own client. - name = "the publisher writes no hive-shared agent queue credential"; + # mints no agent client and the mint stores no such path. The + # control: the same script stores each hive's own client. + name = "the mint stores no hive-shared agent queue credential"; ok = let - s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script; + s = (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script; in !(lib.hasInfix "/queue/agent" s) - && !(lib.hasInfix "-agent.secret" s) - && lib.hasInfix "secret/swarm/hives/h1/oidc/client" s; + && !(lib.hasInfix "hive-h1-agent" s) + && lib.hasInfix "mint_client_secret hive-h1 secret/swarm/hives/h1/oidc/client" s; } { # A property of the SET, not of one unit: both of these authenticate by @@ -371,7 +367,8 @@ let && lib.hasInfix "-token-only" c && !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c))); in - holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script + holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script + && holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-authelia-bridge.preStart && holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-matrix-oidc.script # Controls, so a clean verdict above means something. In order: a bare # read is refused, a read placed before the login is refused, and a @@ -389,40 +386,96 @@ let # would have left the unit rendering only on the one deployment that has # no use for it. It runs beside the mint, in authelia's container, and # never on the host. - name = "a publisher holding an identity runs in authelia's container on a host with no store"; + name = "the mint holding an identity runs in authelia's container on a host with no store"; ok = let m = secretPublisherRemote; c = inAuthelia m; in !m.services.hyperhive.deploy.bao.enable - && (c.systemd.services ? swarm-secret-publish) - && (c.systemd.paths ? swarm-secret-publish) - && !(m.systemd.services ? swarm-secret-publish) - && !(m.systemd.paths ? swarm-secret-publish) + && lib.hasInfix "mint_client_secret" c.systemd.services.authelia-swarm-secrets.script + && !(m.systemd.services ? authelia-swarm-secrets) # Control: the host's own unit set is the one read above. && (m.systemd.services ? "container@swarm-authelia"); } { - # What makes the arm above able to fail. Minting the secrets is not being - # able to publish them: with no certificate the unit would fail a TLS - # handshake on every rotation, so it must not exist at all, and nspawn - # gets no credential to load. - name = "an IdP host with no store identity renders no publisher"; + # What makes the arm above able to fail. Without a certificate the + # plaintext has no store to go to, so the mint keeps it in a file beside + # the digest as before, the bridge reads + # that file, and nspawn gets no credential to load. + name = "an IdP host with no store identity keeps the client secrets in files"; ok = let m = secretPublisherNoIdentity; c = inAuthelia m; + mint = c.systemd.services.authelia-swarm-secrets.script; in m.services.hyperhive.deploy.authelia.enable - && !(c.systemd.services ? swarm-secret-publish) - && !(c.systemd.paths ? swarm-secret-publish) - && !(m.systemd.services ? swarm-secret-publish) + && lib.hasInfix "mint() {" mint + && !(lib.hasInfix "mint_client_secret" mint) + && + c.systemd.services.swarm-authelia-bridge.environment.SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE + == "/var/lib/authelia-swarm/oidc-clients/swarm-authelia-bridge.secret" + && !(c.systemd.services.swarm-authelia-bridge.serviceConfig ? RuntimeDirectory) && !(lib.any (lib.hasPrefix "--load-credential=") m.containers.swarm-authelia.extraFlags) - # Controls: with an identity, the same lookups find both. - && ((inAuthelia secretPublisherHere).systemd.services ? swarm-secret-publish) + # Controls: with an identity, the same lookups find the other shape. + && lib.hasInfix "mint_client_secret" ( + (inAuthelia secretPublisherHere).systemd.services.authelia-swarm-secrets.script + ) && lib.any (lib.hasPrefix "--load-credential=") secretPublisherHere.containers.swarm-authelia.extraFlags; } + { + # With a store identity no unit writes a plaintext client secret under + # /var/lib: the mint keeps no `$sec` file, no redirect in the container + # writes one, and the bridge reads its secret from a file in /run that it + # fetches from the store before each start. The no-identity host is the + # control: the same lookups find the write there. + name = "no unit writes a plaintext oidc client secret to disk, and the bridge's source is the store"; + ok = + let + writesPlaintext = + m: + lib.any (u: lib.hasInfix "> \"$sec\"" (u.script or "")) ( + lib.attrValues (inAuthelia m).systemd.services + ); + here = inAuthelia secretPublisherHere; + bridge = here.systemd.services.swarm-authelia-bridge; + leaf = "secret/swarm/services/swarm-authelia-bridge/oidc/client"; + in + !(writesPlaintext secretPublisherHere) + && writesPlaintext secretPublisherNoIdentity + && + bridge.environment.SWARM_AUTHELIA_BRIDGE_CLIENT_SECRET_FILE + == "/run/swarm-authelia-bridge/client-secret" + && bridge.serviceConfig.RuntimeDirectory == "swarm-authelia-bridge" + && lib.hasInfix "client_secret_read ${leaf}" bridge.preStart + && lib.hasInfix "mint_client_secret swarm-authelia-bridge ${leaf}" here.systemd.services.authelia-swarm-secrets.script + && lib.elem "publisher-bao-cert" bridge.serviceConfig.LoadCredential + && bridge.serviceConfig.Restart == "on-failure"; + } + { + # Every non-native client is minted at the leaf its reader + # fetches, not only the ones with a reader in the publisher's list: the + # bridge's own, which only the bridge reads, is the one that list lacks. + name = "the mint stores every client at the leaf its reader fetches"; + ok = + let + c = inAuthelia secretPublisherHere; + mint = c.systemd.services.authelia-swarm-secrets; + pairs = [ + "hive-h1 secret/swarm/hives/h1/oidc/client" + "hive-h2 secret/swarm/hives/h2/oidc/client" + "swarm-grafana secret/swarm/services/swarm-grafana/oidc/client" + "swarm-controller secret/swarm/controller/swarm-controller/oidc/client" + "swarm-authelia-bridge secret/swarm/services/swarm-authelia-bridge/oidc/client" + ]; + in + lib.all (p: lib.hasInfix "mint_client_secret ${p}" mint.script) pairs + # A native client is public: it has no secret to mint. + && !(lib.hasInfix "mint_client_secret swarmctl" mint.script) + # The mint answers authelia's password prompt through a pty. + && lib.any (p: lib.hasPrefix "util-linux" (p.name or "")) mint.path; + } { # Every key authelia signs or encrypts with comes from the store, seeded # from the file it was generated into, and lands at the path the config @@ -453,37 +506,17 @@ let && !(onDisk.environment ? BAO_ADDR) && !(onDisk.serviceConfig ? LoadCredential); } - { - # The path unit's start is a no-op while its target is `active (exited)`, - # so a secret minted after the first run would never be published. - # `Type` is read too, so a missing `serviceConfig` cannot pass the arm. - # The watched directory is the mint's own, inside the container. - name = "the publisher's path unit watches the mint's directory and starts a oneshot that does not remain active"; - ok = - let - c = inAuthelia secretPublisherHere; - sc = c.systemd.services.swarm-secret-publish.serviceConfig; - in - c.systemd.paths.swarm-secret-publish.pathConfig.Unit == "swarm-secret-publish.service" - && - c.systemd.paths.swarm-secret-publish.pathConfig.PathChanged - == "/var/lib/authelia-swarm/oidc-clients" - && lib.hasInfix "/var/lib/authelia-swarm/oidc-clients/swarm-grafana.secret" c.systemd.services.swarm-secret-publish.script - && !(lib.hasInfix "/var/lib/nixos-containers" c.systemd.services.swarm-secret-publish.script) - && sc.Type == "oneshot" - && !(sc.RemainAfterExit or false); - } { # The leaf crosses into the container as nspawn system credentials, and # the unit loads them and names them in its environment. On the store's # host the container waits for the unit that mints the leaf, since nspawn # refuses a missing credential source; with the leaf placed by hand, # those paths cross. - name = "the publisher's store identity reaches authelia's container as system credentials"; + name = "the store identity reaches authelia's container as system credentials"; ok = let - dir = "/run/credentials/swarm-secret-publish.service"; - unit = m: (inAuthelia m).systemd.services.swarm-secret-publish; + dir = "/run/credentials/authelia-swarm-secrets.service"; + unit = m: (inAuthelia m).systemd.services.authelia-swarm-secrets; flags = m: m.containers.swarm-authelia.extraFlags; here = unit secretPublisherHere; in @@ -516,33 +549,6 @@ let ) && (unit secretPublisherRemote).environment.BAO_CLIENT_CERT == "${dir}/publisher-bao-cert"; } - { - # Authelia is the swarm's IdP and has to come up while the store is down. - # Nothing it starts after or needs may be the publisher, and the - # publisher may not order itself in front of authelia. - name = "authelia does not wait for the secret publisher"; - ok = - let - c = inAuthelia secretPublisherHere; - authelia = c.systemd.services.authelia-swarm; - publisher = c.systemd.services.swarm-secret-publish; - unit = "swarm-secret-publish.service"; - in - !(lib.any (l: lib.elem unit (authelia.${l} or [ ])) [ - "after" - "requires" - "wants" - "bindsTo" - "requisite" - ]) - && !(lib.any (l: lib.elem "authelia-swarm.service" (publisher.${l} or [ ])) [ - "before" - "requiredBy" - ]) - # Controls: both lookups find the real units. - && authelia.serviceConfig ? ExecStart - && lib.elem "authelia-swarm-secrets.service" publisher.after; - } ]; in runGroup "secret-publisher" cases diff --git a/nix/module-eval/swarm-otel-core.nix b/nix/module-eval/swarm-otel-core.nix index 5662d5a39..b122664ee 100644 --- a/nix/module-eval/swarm-otel-core.nix +++ b/nix/module-eval/swarm-otel-core.nix @@ -125,7 +125,7 @@ let # Same 403-not-a-miss reason as grafana's arm above: the reader's grant # covers the `services` prefix, so a path outside it is refused rather # than empty, however correct it reads. - name = "the collector's OIDC secret is read from the prefix the publisher writes"; + name = "the collector's OIDC secret is read from the prefix the mint writes"; ok = let s = otelBaoRemoteAuthelia.systemd.services.swarm-bao-otel-oidc.script; diff --git a/nix/script-tests/oidc-client-secrets.nix b/nix/script-tests/oidc-client-secrets.nix new file mode 100644 index 000000000..08b823fca --- /dev/null +++ b/nix/script-tests/oidc-client-secrets.nix @@ -0,0 +1,110 @@ +# `checks.script-test-oidc-client-secrets` — runs +# ../host-modules/lib/oidc-client-secrets.nix's `mint_client_secret` against a stub `bao` that keeps its leaves in files +# and a stub `authelia` that prints a fixed password and digest. The cases are +# in ./oidc-client-secrets.sh. +# +# What runs is the rendered helper, sourced as the units' scripts include it, +# after the two helpers it builds on. +{ + pkgs, + ... +}: +let + # `login` answers from `FAKE_BAO_LOGIN_RC`. `kv get` and `kv put -cas=0` act + # on a file per leaf under `FAKE_BAO_KV`, and answer as bao does: a missing + # leaf is "No value found at " on stderr, status 2. `FAKE_BAO_DENY_READ` + # and `FAKE_BAO_DENY_WRITE` answer 403 as a policy that grants neither does. + # `FAKE_BAO_RACE` has another run create the leaf between this one's read and + # its write. Every call is logged without the value `put` reads from stdin. + fakeBao = pkgs.writeShellScriptBin "bao" '' + echo "$*" >> "$FAKE_BAO_LOG" + case "$1 $2" in + "login -method=cert") + [ "$FAKE_BAO_LOGIN_RC" = 0 ] || { echo "fake bao: certificate refused" >&2; exit 2; } + printf 's.fake' + ;; + "kv get") + [ "''${BAO_TOKEN-}" = s.fake ] || { echo "fake bao: kv without the login's token" >&2; exit 97; } + [ "$3" = -field=value ] || { echo "fake bao: get without -field=value" >&2; exit 95; } + leaf="$FAKE_BAO_KV/$(printf '%s' "$4" | tr / _)" + [ -z "''${FAKE_BAO_DENY_READ-}" ] || { echo "fake bao: permission denied" >&2; exit 2; } + if [ -n "''${FAKE_BAO_RACE-}" ] && [ ! -e "$FAKE_BAO_KV/.raced" ]; then + touch "$FAKE_BAO_KV/.raced" + printf '%s' "$FAKE_BAO_RACE" > "$leaf" + echo "No value found at secret/data/x" >&2 + exit 2 + fi + [ -e "$leaf" ] || { echo "No value found at secret/data/x" >&2; exit 2; } + cat "$leaf" + echo + ;; + "kv put") + [ "''${BAO_TOKEN-}" = s.fake ] || { echo "fake bao: kv without the login's token" >&2; exit 97; } + [ "$3" = -cas=0 ] || { echo "fake bao: put without -cas=0" >&2; exit 96; } + [ "$5" = value=- ] || { echo "fake bao: put with the value as an argument" >&2; exit 94; } + leaf="$FAKE_BAO_KV/$(printf '%s' "$4" | tr / _)" + [ -z "''${FAKE_BAO_DENY_WRITE-}" ] || { echo "fake bao: permission denied" >&2; exit 2; } + [ ! -e "$leaf" ] || { echo "fake bao: check-and-set parameter did not match" >&2; exit 2; } + cat > "$leaf" + ;; + *) + echo "fake bao: unexpected call: $*" >&2 + exit 98 + ;; + esac + ''; + + # With `--random`, prints what `authelia crypto hash generate pbkdf2 --random` + # prints. Without it, answers the password prompt from stdin the way the CLI + # does on a terminal: echo, then the digest of what was typed. Prints nothing + # when `FAKE_AUTHELIA_EMPTY` is set, and fails without a digest when + # `FAKE_AUTHELIA_NODIGEST` is set. Every call is logged. + fakeAuthelia = pkgs.writeShellScriptBin "authelia" '' + echo "$*" >> "$FAKE_AUTHELIA_LOG" + [ -z "''${FAKE_AUTHELIA_EMPTY-}" ] || exit 0 + case " $* " in + *" --random "*) + printf 'Random Password: minted-secret\n\nDigest: $pbkdf2-sha512$fake\n' + ;; + *) + [ -z "''${FAKE_AUTHELIA_NODIGEST-}" ] || exit 1 + IFS= read -r typed + printf 'Enter Password: %s\r\nDigest: digest-of-%s\r\n' "$typed" "$typed" + ;; + esac + ''; + + # Runs the command it is given, as util-linux's `script -qec /dev/null` + # does, without needing a pty in the build sandbox. The real pty is checked + # against the real authelia by hand. + fakeScript = pkgs.writeShellScriptBin "script" '' + echo "script $1 $3" >> "$FAKE_AUTHELIA_LOG" + [ "$1" = -qec ] || exit 99 + exec ${pkgs.bash}/bin/bash -c "$2" + ''; +in +pkgs.runCommand "hyperhive-script-test-oidc-client-secrets" + { + HELPER = pkgs.writeText "oidc-client-secrets.sh" ( + import ../host-modules/lib/store-seeded-secret.nix { } + + import ../host-modules/lib/atomic-write-secret.nix { } + + import ../host-modules/lib/oidc-client-secrets.nix { } + ); + FAKE_BIN = pkgs.symlinkJoin { + name = "fake-bao-and-authelia"; + paths = [ + fakeBao + fakeAuthelia + fakeScript + ]; + }; + nativeBuildInputs = [ + pkgs.coreutils + pkgs.gnugrep + pkgs.gnused + ]; + } + '' + ${pkgs.bash}/bin/bash ${./oidc-client-secrets.sh} + touch "$out" + '' diff --git a/nix/script-tests/oidc-client-secrets.sh b/nix/script-tests/oidc-client-secrets.sh new file mode 100644 index 000000000..e6a5f97f6 --- /dev/null +++ b/nix/script-tests/oidc-client-secrets.sh @@ -0,0 +1,269 @@ +# Cases for ./oidc-client-secrets.nix. Each case sets up the store's leaf, the +# digest and the plaintext file an earlier release left, and the stub's +# behaviour, runs the function under `set -e` as the units' scripts do, and +# compares its status, what the store holds, what is left on disk and whether +# `authelia` minted. +set -uo pipefail + +failures=0 +count=0 + +fail() { + echo "FAILED: $case: $*" >&2 + failures=$((failures + 1)) +} + +id=swarm-grafana +leaf=secret/swarm/services/swarm-grafana/oidc/client + +# setup +setup() { + case=$1 + count=$((count + 1)) + dir=$(mktemp -d) + mkdir "$dir/kv" "$dir/clients" + export FAKE_BAO_KV=$dir/kv FAKE_BAO_LOG=$dir/bao.log FAKE_BAO_LOGIN_RC=0 + export FAKE_AUTHELIA_LOG=$dir/authelia.log + unset FAKE_BAO_DENY_READ FAKE_BAO_DENY_WRITE FAKE_BAO_RACE FAKE_AUTHELIA_EMPTY FAKE_AUTHELIA_NODIGEST + : >"$FAKE_BAO_LOG" + : >"$FAKE_AUTHELIA_LOG" + clients=$dir/clients + digest=$clients/$id.digest + legacy=$clients/$id.secret + stored=$FAKE_BAO_KV/$(printf '%s' "$leaf" | tr / _) +} + +# run — the function for $id and $leaf, with the stubs first on +# PATH, after the store login the units make first. Sets $rc and $minted. +run() { + minted=$( + set -e + PATH="$FAKE_BIN/bin:$PATH" + unset BAO_TOKEN + # shellcheck source=/dev/null + . "$HELPER" + clients=$dir/clients + store_init + store_login 2>>"$dir/err" + rc=0 + "$1" "$id" "$leaf" 2>>"$dir/err" >>"$dir/out" || rc=$? + echo "${MINTED-}" + exit "$rc" + ) + rc=$? +} + +# expect +expect() { + if [ "$rc" != "$1" ]; then fail "exited $rc, wanted $1: $(cat "$dir/err" "$dir/out" 2>/dev/null)"; fi +} + +# held , or held "" for a leaf the store does not have. +held() { + if [ -z "$1" ]; then + if [ -e "$stored" ]; then fail "the store holds '$(cat "$stored")', wanted nothing"; fi + elif [ "$(cat "$stored" 2>/dev/null)" != "$1" ]; then + fail "the store holds '$(cat "$stored" 2>/dev/null)', wanted '$1'" + fi +} + +# digest_is , or digest_is "" for no digest file. +digest_is() { + if [ -z "$1" ]; then + if [ -e "$digest" ]; then fail "a digest was written: '$(cat "$digest")'"; fi + elif [ "$(cat "$digest" 2>/dev/null)" != "$1" ]; then + fail "the digest is '$(cat "$digest" 2>/dev/null)', wanted '$1'" + fi +} + +# Whether a new secret was minted, as opposed to a digest computed. +ran_authelia() { + grep -q -e "--random" "$FAKE_AUTHELIA_LOG" +} + +store_called() { + grep -q '^kv' "$FAKE_BAO_LOG" +} + +logged() { + grep -qF -- "$1" "$dir/err" "$dir/out" 2>/dev/null +} + +no_secret_in_argv() { + if grep -q -e minted-secret -e old-secret "$FAKE_BAO_LOG" "$FAKE_AUTHELIA_LOG"; then + fail "a secret reached a command line" + fi +} + +# --- mint_client_secret --------------------------------------------------- + +setup "mint: a fresh client is minted into the store and only its digest is written" +run mint_client_secret +expect 0 +held minted-secret +digest_is '$pbkdf2-sha512$fake' +if [ "$minted" != 1 ]; then fail "did not report the mint"; fi +if [ -e "$legacy" ]; then fail "wrote a plaintext file"; fi +if [ "$(stat -c %a "$digest")" != 600 ]; then fail "digest mode is $(stat -c %a "$digest")"; fi +if ! logged "minted client secret for $id into the store"; then fail "no mint line"; fi +no_secret_in_argv + +setup "mint: a client the store and the disk both hold is left alone" +printf 'old-secret' >"$stored" +printf 'old-digest\n' >"$digest" +run mint_client_secret +expect 0 +held old-secret +digest_is old-digest +if ran_authelia; then fail "minted although the store held a value"; fi +if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote to a leaf that held a value"; fi +if [ "$minted" = 1 ]; then fail "reported a mint"; fi + +setup "mint: a digest whose secret is gone from the store is minted again" +printf 'old-digest\n' >"$digest" +run mint_client_secret +expect 0 +held minted-secret +digest_is '$pbkdf2-sha512$fake' +if [ "$minted" != 1 ]; then fail "did not report the mint"; fi + +setup "mint: a store that refuses the login leaves the digest, and mints nothing" +printf 'old-digest\n' >"$digest" +export FAKE_BAO_LOGIN_RC=1 +run mint_client_secret +expect 0 +digest_is old-digest +if ran_authelia; then fail "minted with the store down"; fi +if store_called; then fail "called kv after a refused login"; fi + +setup "mint: a store that refuses the read leaves the digest, and mints nothing" +printf 'old-digest\n' >"$digest" +export FAKE_BAO_DENY_READ=1 +run mint_client_secret +expect 0 +digest_is old-digest +if ran_authelia; then fail "minted although the read was refused"; fi +if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote although the read was refused"; fi + +setup "mint: no digest and a store that is down fails to retry, and mints nothing" +export FAKE_BAO_LOGIN_RC=1 +run mint_client_secret +expect 1 +digest_is "" +if ran_authelia; then fail "minted with the store down"; fi + +setup "mint: no digest and a store that refuses the read fails to retry, and mints nothing" +export FAKE_BAO_DENY_READ=1 +run mint_client_secret +expect 1 +digest_is "" +if ran_authelia; then fail "minted although the read was refused"; fi + +setup "mint: no digest but a secret in the store gets the digest of that secret, and the secret is unchanged" +printf 'old-secret' >"$stored" +run mint_client_secret +expect 0 +held old-secret +digest_is digest-of-old-secret +if ran_authelia; then fail "minted over the store's secret"; fi +if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote to a leaf that held a value"; fi +if [ "$minted" != 1 ]; then fail "did not report the new digest"; fi +if [ "$(stat -c %a "$digest")" != 600 ]; then fail "digest mode is $(stat -c %a "$digest")"; fi +if ! logged "oidc client $id: wrote the digest from the secret in the store"; then fail "no info line"; fi +no_secret_in_argv + +setup "mint: a plaintext file the store lacks seeds the store once, mints nothing and keeps the digest and the file" +printf 'old-secret\n' >"$legacy" +printf 'old-digest\n' >"$digest" +run mint_client_secret +expect 0 +held old-secret +digest_is old-digest +if ran_authelia; then fail "minted although a file held the secret"; fi +if [ "$(cat "$legacy")" != old-secret ]; then fail "changed or removed the file"; fi +if ! grep -q '^kv put -cas=0 ' "$FAKE_BAO_LOG"; then fail "no create-only write was made"; fi +no_secret_in_argv + +setup "mint: a plaintext file with no digest seeds the store and gets the digest of the file's secret" +printf 'old-secret\n' >"$legacy" +run mint_client_secret +expect 0 +held old-secret +digest_is digest-of-old-secret +if [ "$(cat "$legacy")" != old-secret ]; then fail "changed the file"; fi +if ran_authelia; then fail "minted over the file's secret"; fi +no_secret_in_argv + +setup "mint: the store's secret wins over a different file, silently, and nothing is written" +printf 'old-secret\n' >"$legacy" +printf 'other-secret' >"$stored" +printf 'old-digest\n' >"$digest" +run mint_client_secret +expect 0 +held other-secret +digest_is old-digest +if grep -q '^kv put' "$FAKE_BAO_LOG"; then fail "wrote to a leaf that held a value"; fi +if ran_authelia; then fail "minted"; fi + +setup "mint: a file with a store that refuses the login keeps the digest and mints nothing" +printf 'old-secret\n' >"$legacy" +printf 'old-digest\n' >"$digest" +export FAKE_BAO_LOGIN_RC=1 +run mint_client_secret +expect 0 +digest_is old-digest +if ran_authelia; then fail "minted with the store down"; fi + +setup "mint: a file whose seed write is refused is a failure, and nothing is minted" +printf 'old-secret\n' >"$legacy" +printf 'old-digest\n' >"$digest" +export FAKE_BAO_DENY_WRITE=1 +run mint_client_secret +expect 1 +held "" +if ran_authelia; then fail "minted although the file held the secret"; fi + +setup "mint: a secret whose digest cannot be computed is a failure and writes no digest" +printf 'old-secret' >"$stored" +export FAKE_AUTHELIA_NODIGEST=1 +run mint_client_secret +expect 1 +digest_is "" +held old-secret + +setup "mint: an authelia that prints nothing writes nothing" +export FAKE_AUTHELIA_EMPTY=1 +run mint_client_secret +expect 1 +held "" +digest_is "" + +setup "mint: a store that refuses the write leaves no digest" +export FAKE_BAO_DENY_WRITE=1 +run mint_client_secret +expect 1 +held "" +digest_is "" + +setup "mint: losing the race to another run takes the winner's secret and writes its digest" +export FAKE_BAO_RACE=winner-secret +run mint_client_secret +expect 0 +held winner-secret +digest_is digest-of-winner-secret + +if [ "$(id -u)" != 0 ]; then + setup "mint: a digest that cannot be written is a failure, not a mint" + chmod 500 "$clients" + run mint_client_secret + chmod 700 "$clients" + expect 1 + if [ "$minted" = 1 ]; then fail "reported a mint without a digest"; fi + digest_is "" +fi + +if [ "$failures" != 0 ]; then + echo "$failures of $count cases failed" >&2 + exit 1 +fi +echo "$count cases passed" diff --git a/swarm-builder/src/main.rs b/swarm-builder/src/main.rs index c8f1470e3..aecc2ff51 100644 --- a/swarm-builder/src/main.rs +++ b/swarm-builder/src/main.rs @@ -538,8 +538,8 @@ async fn client_secret(client_id: &str) -> Result { match stored { Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()), _ => bail!( - "nothing at {path}: swarm-secret-publish, in authelia's container, has not \ - published this service's client secret" + "nothing at {path}: authelia's mint, in its container, has not \ + stored this service's client secret" ), } } diff --git a/swarm-ci-metrics/src/main.rs b/swarm-ci-metrics/src/main.rs index 1c6e36785..68d8b58f4 100644 --- a/swarm-ci-metrics/src/main.rs +++ b/swarm-ci-metrics/src/main.rs @@ -296,7 +296,7 @@ async fn forge_token() -> Result { } } -/// This service's OIDC client secret, which the secret publisher writes. +/// This service's OIDC client secret, which authelia's mint writes. async fn client_secret(client_id: &str) -> Result { let path = client_secret_path(client_id)?; let stored: Option = login() @@ -307,8 +307,8 @@ async fn client_secret(client_id: &str) -> Result { match stored { Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()), _ => bail!( - "nothing at {path}: swarm-secret-publish, in authelia's container, has not \ - published this service's client secret" + "nothing at {path}: authelia's mint, in its container, has not \ + stored this service's client secret" ), } } diff --git a/swarm-claude-login/src/main.rs b/swarm-claude-login/src/main.rs index f9dce0400..9cc7cd649 100644 --- a/swarm-claude-login/src/main.rs +++ b/swarm-claude-login/src/main.rs @@ -179,8 +179,8 @@ async fn client_secret(client_id: &str) -> Result { match stored { Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()), _ => bail!( - "nothing at {path}: swarm-secret-publish, in authelia's container, has not \ - published this service's client secret" + "nothing at {path}: authelia's mint, in its container, has not \ + stored this service's client secret" ), } } diff --git a/swarm-controller/src/queue_identity.rs b/swarm-controller/src/queue_identity.rs index 7bc110622..f15ab0d8e 100644 --- a/swarm-controller/src/queue_identity.rs +++ b/swarm-controller/src/queue_identity.rs @@ -6,7 +6,7 @@ //! `swarm_secret_client::queue::controller_client_path`, read at start under //! this daemon's own certificate, held only in memory and read again when //! authelia refuses it (`swarm_secret_client::stored_secret`). It is never on -//! this host's disk. `swarm-secret-publish`, in authelia's +//! this host's disk. authelia's mint (`authelia-swarm-secrets`), in its //! container, puts it there. //! //! [`init`] runs once in `main`, before anything calls [`get`]. @@ -114,8 +114,8 @@ async fn fetch_secret() -> Result { match stored { Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()), _ => bail!( - "nothing at {path}: swarm-secret-publish, in authelia's container, has not \ - published this daemon's client secret" + "nothing at {path}: authelia's mint, in its container, has not \ + stored this daemon's client secret" ), } } diff --git a/swarm-nats-auth/src/oidc_secret.rs b/swarm-nats-auth/src/oidc_secret.rs index 4ee306870..3f2ca1c37 100644 --- a/swarm-nats-auth/src/oidc_secret.rs +++ b/swarm-nats-auth/src/oidc_secret.rs @@ -2,8 +2,8 @@ //! with. Read from the secret store as the responder starts and held in //! memory, never in a file, `argv` or the environment. //! -//! The secret can change under a running responder: the publisher in authelia's -//! container re-publishes it when it is re-minted. So when the `IdP` refuses an +//! The secret can change under a running responder: authelia's mint, in its +//! container, writes a new one when it re-mints. So when the `IdP` refuses an //! introspection as a credential (a 400 or 401), the secret is read from the //! store again ([`StoredSecret::after_refusal`]) and, when it changed, the //! introspection is tried once more. Both fit in the one request's @@ -25,8 +25,9 @@ use crate::introspect::{Answer, Caller}; use crate::keystore::KeyStore; /// The path under the store's KV mount holding `client_id`'s secret. -/// `swarm-secret-publisher.nix` writes it, and `natsAuthReaders` in -/// `swarm-bao.nix` grants this responder's role the read. +/// Authelia's mint (`authelia-swarm-secrets`) writes it, at the path +/// `swarm-secret-publisher.nix` lists, and `natsAuthReaders` in `swarm-bao.nix` +/// grants this responder's role the read. /// /// # Errors /// When `client_id` is not a single path segment. @@ -35,17 +36,15 @@ pub fn path(client_id: &str) -> Result { Ok(format!("{prefix}/oidc/client")) } -/// The secret at `path`, with surrounding whitespace trimmed: the publisher -/// stores the minted file as is, and a trailing newline is not part of the -/// secret. Errors name the path, never the value. +/// The secret at `path`, with surrounding whitespace trimmed: an earlier +/// release stored the minted file as is, and a trailing newline is not part of +/// the secret. Errors name the path, never the value. /// /// # Errors /// When the store cannot be read, or holds nothing or only whitespace there. pub async fn read(store: &impl KeyStore, path: &str) -> Result { let stored = store.read(path).await?.with_context(|| { - format!( - "nothing is stored at {path}; swarm-secret-publish in authelia's container writes it" - ) + format!("nothing is stored at {path}; authelia's mint, in its container, writes it") })?; let trimmed = stored.trim(); if trimmed.is_empty() { diff --git a/swarm-notify-router/src/main.rs b/swarm-notify-router/src/main.rs index a7cb97845..3758ce06e 100644 --- a/swarm-notify-router/src/main.rs +++ b/swarm-notify-router/src/main.rs @@ -381,8 +381,8 @@ async fn client_secret(client_id: &str) -> Result { match stored { Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()), _ => bail!( - "nothing at {path}: swarm-secret-publish, in authelia's container, has not \ - published this service's client secret" + "nothing at {path}: authelia's mint, in its container, has not \ + stored this service's client secret" ), } } diff --git a/swarm-review-classifier/src/main.rs b/swarm-review-classifier/src/main.rs index a2cbd3a35..b5cc39559 100644 --- a/swarm-review-classifier/src/main.rs +++ b/swarm-review-classifier/src/main.rs @@ -171,8 +171,8 @@ async fn client_secret(client_id: &str) -> Result { match stored { Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()), _ => bail!( - "nothing at {path}: swarm-secret-publish, in authelia's container, has not \ - published this service's client secret" + "nothing at {path}: authelia's mint, in its container, has not \ + stored this service's client secret" ), } } diff --git a/swarm-secret-client/src/builder.rs b/swarm-secret-client/src/builder.rs index 2e3444c2d..d573400b6 100644 --- a/swarm-secret-client/src/builder.rs +++ b/swarm-secret-client/src/builder.rs @@ -52,7 +52,7 @@ pub struct AtticPushToken { } /// The path holding the queue client secret of the builder's OIDC client -/// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's. +/// `client_id`, where authelia's mint puts every swarm service's. /// /// # Errors /// [`Error::PathSegment`] when `client_id` is not a single path segment. @@ -66,7 +66,7 @@ pub fn client_secret_path(client_id: &str) -> Result { /// No `Debug`: `value` authenticates the builder at the queue. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ClientSecret { - /// The secret, under the field name the publisher writes. + /// The secret, under the field name authelia's mint writes. pub value: String, } diff --git a/swarm-secret-client/src/ci_metrics.rs b/swarm-secret-client/src/ci_metrics.rs index 37ecfbaea..29097b918 100644 --- a/swarm-secret-client/src/ci_metrics.rs +++ b/swarm-secret-client/src/ci_metrics.rs @@ -27,7 +27,7 @@ pub fn forge_token_path() -> Result { } /// The path holding the secret of the poller's OIDC client `client_id`, where -/// `swarm-secret-publisher.nix` puts every swarm service's. +/// authelia's mint puts every swarm service's. /// /// # Errors /// [`Error::PathSegment`] when `client_id` is not a single path segment. @@ -41,7 +41,7 @@ pub fn client_secret_path(client_id: &str) -> Result { /// No `Debug`: `value` authenticates the poller at the collector. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ClientSecret { - /// The secret, under the field name the publisher writes. + /// The secret, under the field name authelia's mint writes. pub value: String, } diff --git a/swarm-secret-client/src/claude_login.rs b/swarm-secret-client/src/claude_login.rs index 3cfcda4b6..f54774542 100644 --- a/swarm-secret-client/src/claude_login.rs +++ b/swarm-secret-client/src/claude_login.rs @@ -46,7 +46,7 @@ impl std::fmt::Debug for Credentials { } /// The path holding the queue client secret of the minter's OIDC client -/// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's. +/// `client_id`, where authelia's mint puts every swarm service's. /// /// # Errors /// [`Error::PathSegment`] when `client_id` is not a single path segment. diff --git a/swarm-secret-client/src/queue.rs b/swarm-secret-client/src/queue.rs index 0c32251d2..53a63ba9a 100644 --- a/swarm-secret-client/src/queue.rs +++ b/swarm-secret-client/src/queue.rs @@ -44,8 +44,9 @@ pub fn agent_queue_path(agent: &str) -> Result { /// Under [`Kind::Controller`] because no hive's policy reads that kind, and /// this client may read every hive's status. /// -/// The nix half is `swarm-secret-publisher.nix`, which writes it, and -/// `swarm-bao.nix`'s `controllerPolicyText`, which grants the read. +/// The nix half is `swarm-secret-publisher.nix`, which lists the path +/// authelia's mint writes it at, and `swarm-bao.nix`'s `controllerPolicyText`, +/// which grants the read. /// /// # Errors /// Never in practice: the name segment is a constant. The `Result` is @@ -70,7 +71,8 @@ pub struct ControllerCredential { /// Under the hive's own prefix, which the hive's policy reads /// ([`crate::policy::render_shared_hives`]). /// -/// The nix half is `swarm-secret-publisher.nix`, which writes it. +/// The nix half is `swarm-secret-publisher.nix`, which lists the path authelia's +/// mint writes it at. /// /// # Errors /// [`Error::PathSegment`] when `hive` contains anything but `[A-Za-z0-9_-]`. @@ -117,8 +119,8 @@ pub struct AgentCredential { mod tests { use super::*; - /// Spelled out in full because `swarm-secret-publisher.nix` and - /// `swarm-bao.nix` write the same string by hand. + /// Spelled out in full because the path layout in `swarm-secret-publisher.nix` + /// and `swarm-bao.nix` is written out by hand there too. #[test] fn the_controller_client_lands_under_the_controller() { assert_eq!( @@ -145,8 +147,9 @@ mod tests { assert!(reads(&own), "{own} should be readable under {granted:?}"); } - /// Spelled out in full because `swarm-secret-publisher.nix` writes the - /// same string by hand. + /// Spelled out in full because the path layout in + /// `swarm-secret-publisher.nix`, where authelia's mint takes it from, is + /// written out by hand there too. #[test] fn a_hives_own_client_lands_under_its_own_principal_prefix() { assert_eq!( diff --git a/swarm-secret-client/src/review_classifier.rs b/swarm-secret-client/src/review_classifier.rs index 7d573e2a4..bcb5ac6d5 100644 --- a/swarm-secret-client/src/review_classifier.rs +++ b/swarm-secret-client/src/review_classifier.rs @@ -60,7 +60,7 @@ pub fn forge_token_path() -> Result { } /// The path holding the queue client secret of the classifier's OIDC client -/// `client_id`, where `swarm-secret-publisher.nix` puts every swarm service's. +/// `client_id`, where authelia's mint puts every swarm service's. /// /// # Errors /// [`Error::PathSegment`] when `client_id` is not a single path segment. @@ -74,7 +74,7 @@ pub fn client_secret_path(client_id: &str) -> Result { /// No `Debug`: `value` authenticates the classifier at the queue. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ClientSecret { - /// The secret, under the field name the publisher writes. + /// The secret, under the field name authelia's mint writes. pub value: String, } diff --git a/swarm-secret-client/src/stored_secret.rs b/swarm-secret-client/src/stored_secret.rs index 51a4ee3f3..d4f51b47c 100644 --- a/swarm-secret-client/src/stored_secret.rs +++ b/swarm-secret-client/src/stored_secret.rs @@ -1,8 +1,8 @@ //! A client secret held in memory and read again from where it is kept when //! the `IdP` refuses it. //! -//! The secret can change under a running process: the publisher in authelia's -//! container re-publishes it when it is re-minted. A refusal asks for a re-read +//! The secret can change under a running process: authelia's mint, in its +//! container, writes a new one when it re-mints. A refusal asks for a re-read //! ([`StoredSecret::after_refusal`]); when the read lands a different secret, //! the caller tries once more with it. One re-read runs at a time, and a //! refusal while one is running waits for it rather than starting another.