diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 6199f6b8..f73686b7 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -117,7 +117,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. | the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives//` and drop the setting | | the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | diff --git a/nix/host-modules/otel.nix b/nix/host-modules/otel.nix index 6ac3ecdf..7d89639a 100644 --- a/nix/host-modules/otel.nix +++ b/nix/host-modules/otel.nix @@ -51,6 +51,8 @@ let inherit (config.services.hyperhive.deploy.hive-controller) hiveName; }; fetchedSecretPath = hiveOtelSecret.path; + + pkiDir = import ./lib/bao-pki-dir.nix; in { imports = [ @@ -883,13 +885,22 @@ in # The fetch unit writes `fetchedSecretPath` now, so a file named at the # old default is delivered by nobody and never refreshed. - warnings = lib.optional (hiveName != null && otel.clientSecretFile == hiveOtelSecret.legacyPath) '' - services.hyperhive.otel.clientSecretFile is ${hiveOtelSecret.legacyPath}, the path - swarm-bao-hive-otel-oidc.service used to write. That unit now writes - ${fetchedSecretPath} and, with the option set to anything else, does - not exist, so nothing refreshes the file. Remove the setting to fetch - from the swarm secret store, or deliver the file yourself. - ''; + warnings = + lib.optional (hiveName != null && otel.clientSecretFile == hiveOtelSecret.legacyPath) '' + services.hyperhive.otel.clientSecretFile is ${hiveOtelSecret.legacyPath}, the path + swarm-bao-hive-otel-oidc.service used to write. That unit now writes + ${fetchedSecretPath} and, with the option set to anything else, does + not exist, so nothing refreshes the file. Remove the setting to fetch + from the swarm secret store, or deliver the file yourself. + '' + ++ lib.optional (hiveName != null && !hiveLeaf && otel.clientSecretFile != null) '' + services.hyperhive.otel.clientSecretFile is ${otel.clientSecretFile} and this + host holds no store identity (services.hyperhive.deploy.bao.clientCertFile + and clientKeyFile). Place the hive's leaf from ${pkiDir}/hives/${hiveName}/ + on the host that runs the store, set those two options, then drop + services.hyperhive.otel.clientSecretFile: the collector then fetches its + secret from the swarm secret store under that leaf. + ''; assertions = [ { diff --git a/nix/module-eval/hive-otel.nix b/nix/module-eval/hive-otel.nix index 3db70378..2e409033 100644 --- a/nix/module-eval/hive-otel.nix +++ b/nix/module-eval/hive-otel.nix @@ -49,6 +49,10 @@ let deploy.authelia.enable = true; }; + # Neither a leaf nor a secret file: the one shape the collector has no + # credential for. + hiveOtelBare = hiveWithController { }; + hiveOtelFetch = machine: machine.systemd.services.swarm-bao-hive-otel-oidc; hiveOtelPipelines = hiveOtel.services.opentelemetry-collector.settings.service.pipelines; @@ -262,6 +266,27 @@ let && !(builtins.elem "f ${run} 0400 root root -" (rules hiveOtel)) && warned hiveOtel; } + { + # Without a store identity, a hand-named secret file is warned to place + # the leaf; with the leaf nothing is warned and the fetch unit exists; + # with neither, the assertion still refuses the build. `hiveOtelBare` + # and `hiveOtelLeaf` are the controls: a lookup that never matches + # would pass the negated arms. + name = "a hive host with no leaf and an operator secret file is warned to place the leaf"; + ok = + let + warned = m: lib.any (lib.hasInfix "holds no store identity") m.warnings; + refused = m: lib.any (a: !a.assertion && lib.hasInfix "identity to present" a.message) m.assertions; + in + warned hiveOtel + && lib.any (lib.hasInfix "/var/lib/swarm-bao-pki/hives/h1/") hiveOtel.warnings + && !(warned hiveOtelLeaf) + && hiveOtelLeaf.systemd.services ? swarm-bao-hive-otel-oidc + && !(warned hiveOtelBare) + && refused hiveOtelBare + && !(refused hiveOtel) + && !(refused hiveOtelLeaf); + } { # `LoadCredential` copies the file at the collector's start, so a value # landed later reaches it only through a restart. Retries every 30s