From a572fb6a7fcdb23f39818f9982b87bf3325b3ca8 Mon Sep 17 00:00:00 2001 From: atlas Date: Tue, 27 Oct 2026 15:22:55 +0100 Subject: [PATCH 1/2] forward the attic container's journal to the swarm collector --- docs/swarm/credentials-reference.md | 2 +- docs/swarm/services.md | 9 +- nix/host-modules/default.nix | 1 + .../glue-attic-otel-oidc-client.nix | 37 ++++++ nix/host-modules/swarm-attic-options.nix | 15 +++ nix/host-modules/swarm-attic.nix | 121 +++++++++++++++++- nix/host-modules/swarm-bao.nix | 7 +- nix/host-modules/swarm-otel-service.nix | 8 +- nix/host-modules/swarm-otel.nix | 2 +- nix/host-modules/swarm-secret-publisher.nix | 5 +- nix/module-eval/bao-grants.nix | 11 +- nix/module-eval/builder.nix | 30 +++++ nix/module-eval/secret-publisher.nix | 3 +- 13 files changed, 230 insertions(+), 21 deletions(-) create mode 100644 nix/host-modules/glue-attic-otel-oidc-client.nix diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 3e34a0194..9976fada4 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -121,7 +121,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. | an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | | the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the CI runner, Grafana, log store, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | +| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | | the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | | the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | | the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | diff --git a/docs/swarm/services.md b/docs/swarm/services.md index 905fee4ab..913e2e161 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -359,16 +359,16 @@ gateway either way. The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`, `swarm-notify-router`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`, -`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `hive-forge` +`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `swarm-attic`, `hive-forge` and `hive-matrix` containers each run a collector of their own that forwards the container's journal here, on a route and receiver per service (`https:////`), authenticated as an OIDC client of that service's. The queue, builder, login minter, notification router and classifier reuse the service's own client; the store, CI runner, Grafana, log store, metrics store, -authelia, forge and homeserver have a forwarder client of their own +authelia, cache, forge and homeserver have a forwarder client of their own (`swarm..otel.clientId`). Their records carry `service.name` (`bao`, `nats`, `builder`, `claude-login`, `notify-router`, `review-classifier`, `ci-runner`, -`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `forgejo`, +`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `attic`, `forgejo`, `tuwunel`) and no `hive` label. The forwarder of all but the store's is `nix/container-modules/swarm-journal-forwarder.nix`. @@ -451,6 +451,9 @@ store, each only while absent. On a host without the store, that leaf is the one file to copy. The builder pushes green builds to it. How it works, and the external alternative, are in [`builder.md`](builder.md#binary-cache). +The container forwards atticd's journal to the swarm collector like the other +service containers (see [Telemetry collector](#telemetry-collector-otel)). The two host units' lines +stay in the host's journal. ### CI metrics The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index fdb721f59..c76477c3a 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -52,6 +52,7 @@ ./glue-ci-runner-otel-oidc-client.nix ./glue-claude-login-oidc-client.nix ./glue-notify-router-oidc-client.nix + ./glue-attic-otel-oidc-client.nix ./glue-forge-otel-oidc-client.nix ./glue-grafana-otel-oidc-client.nix ./glue-matrix-otel-oidc-client.nix diff --git a/nix/host-modules/glue-attic-otel-oidc-client.nix b/nix/host-modules/glue-attic-otel-oidc-client.nix new file mode 100644 index 000000000..1d4de83d7 --- /dev/null +++ b/nix/host-modules/glue-attic-otel-oidc-client.nix @@ -0,0 +1,37 @@ +# Glue: register the cache container's journal forwarder as an OIDC client +# wherever authelia runs. +# +# ONE PAIRING PER FILE: that forwarder ← authelia, and nothing else. +# Without it authelia mints no secret, the publisher has nothing to copy into +# the store, and the forwarder's export is a 401. +# +# Gated on authelia being HERE, not on this host running the cache: a client is +# a row in THIS host's provider config, the split +# ./glue-victorialogs-otel-oidc-client.nix makes for the same reason. +{ + lib, + config, + ... +}: +let + hyperhiveCfg = config.services.hyperhive; + clientId = hyperhiveCfg.swarm.attic.otel.clientId; +in +{ + config = lib.mkIf hyperhiveCfg.deploy.authelia.enable { + services.hyperhive.swarm.authelia.oidc.clients = [ + { + id = clientId; + description = "HyperHive cache journal forwarder"; + kind = "machine"; + redirectUris = [ ]; + # Its own id as its own permitted audience, the one the swarm + # collector's receiver for this container checks. + audience = [ clientId ]; + # The receiver verifies the token offline against `/jwks.json`; + # authelia's default opaque handle fails there. + accessTokenSignedResponseAlg = "RS256"; + } + ]; + }; +} diff --git a/nix/host-modules/swarm-attic-options.nix b/nix/host-modules/swarm-attic-options.nix index 222d6ae85..5adbf9e06 100644 --- a/nix/host-modules/swarm-attic-options.nix +++ b/nix/host-modules/swarm-attic-options.nix @@ -13,6 +13,21 @@ let in { options.services.hyperhive.swarm.attic = { + otel.clientId = lib.mkOption { + type = lib.types.str; + readOnly = true; + default = "swarm-attic-collector"; + description = '' + OAuth2 client id the journal forwarder inside the cache's container + authenticates as, and the audience it asks its token for. + + Read-only because three hosts have to agree on it: authelia registers + the client (`glue-attic-otel-oidc-client.nix`), the swarm collector + admits it on the container's receiver (`swarm.otel.serviceProducers`), + and the secret publisher carries its secret to the store. + ''; + }; + ownDomain = lib.mkOption { type = lib.types.str; readOnly = true; diff --git a/nix/host-modules/swarm-attic.nix b/nix/host-modules/swarm-attic.nix index d95270523..2ef51c22e 100644 --- a/nix/host-modules/swarm-attic.nix +++ b/nix/host-modules/swarm-attic.nix @@ -29,6 +29,9 @@ let atticCfg = hyperhiveCfg.swarm.attic; baoCfg = hyperhiveCfg.swarm.bao; baoDeploy = hyperhiveCfg.deploy.bao; + autheliaCfg = hyperhiveCfg.swarm.authelia; + networkCfg = hyperhiveCfg.network; + gatewayCfg = hyperhiveCfg.gateway; machine = "swarm-attic"; unit = "swarm-attic"; @@ -87,6 +90,33 @@ let haveClientIdentity = baoDeploy.atticClientCertFile != null && baoDeploy.atticClientKeyFile != null; + # The container's journal forwarder pushes to its own receiver on the swarm + # collector, the route `serviceProducers.swarm-attic` in + # ./swarm-otel-service.nix declares. + forwarderClientId = atticCfg.otel.clientId; + forwarderEndpoint = "https://${hyperhiveCfg.swarm.otel.domain}/${machine}"; + forwarderUnit = "swarm-journal-forwarder-oidc"; + # In the host netns: distinct from every other collector's there. + forwarderTelemetryPort = 8903; + forwarderCredential = id: "/run/credentials/${forwarderUnit}.service/${id}"; + # The cache's store identity, handed into the container as system credentials + # for the forwarder's fetch unit. The host units keep reading the same files. + forwarderStoreFiles = lib.filterAttrs (_: source: source != null) { + attic-bao-cert = baoDeploy.atticClientCertFile; + attic-bao-key = baoDeploy.atticClientKeyFile; + attic-bao-server-ca = baoDeploy.serverCaFile; + }; + # nixos-container expands `extraFlags` unquoted, so a source path must be + # one word. + oneWord = path: builtins.match ".*[[:space:]].*" path == null; + + # The forwarder reaches the swarm collector and authelia by name over TLS; in + # self-signed mode their leaves chain to the hive CA. + caTrust = import ./lib/hive-ca-trust.nix { + inherit lib gatewayCfg; + tlsCfg = hyperhiveCfg.deploy.hive-controller.tls; + }; + # Both host units log in as the cache's principal. storeEnvironment = { BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}"; @@ -171,6 +201,35 @@ in is set to null. ''; } + { + assertion = lib.all oneWord (lib.attrValues forwarderStoreFiles); + message = '' + The cache's store identity paths must not contain whitespace: + nspawn's --load-credential receives them through nixos-container's + unquoted extra flags. + ''; + } + { + assertion = + (hyperhiveCfg.swarm.otel.serviceProducers.${machine}.clientId or null) == forwarderClientId; + message = '' + services.hyperhive.swarm.otel.serviceProducers has no entry + "${machine}" with clientId "${forwarderClientId}", so the swarm + collector has no receiver for the cache's journal forwarder. + Declare it in swarm-otel-service.nix. + ''; + } + ]; + + # nspawn fails to start the container when a credential source is + # missing, so where this host mints the leaf the container waits for it. The + # CA ordering keeps the trust bundle's bind source there first. + systemd.services."container@${machine}" = lib.mkMerge [ + caTrust.containerOrdering + { + after = [ "swarm-bao-pki.service" ]; + wants = [ "swarm-bao-pki.service" ]; + } ]; services.hyperhive.gateway.localNames = [ atticCfg.domain ]; @@ -279,19 +338,71 @@ in containers.${machine} = { autoStart = true; ephemeral = false; + # No `--link-journal=host`: the journal stays in the guest, where the + # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix). + extraFlags = lib.mapAttrsToList ( + id: source: "--load-credential=${id}:${source}" + ) forwarderStoreFiles; privateNetwork = false; - bindMounts.${secretDir} = { - hostPath = secretDir; - isReadOnly = true; - }; + bindMounts = { + ${secretDir} = { + hostPath = secretDir; + isReadOnly = true; + }; + } + // caTrust.bindMount; config = { ... }: { - imports = [ ../container-modules/swarm-container.nix ]; + imports = [ + ../container-modules/swarm-container.nix + ../container-modules/swarm-journal-forwarder.nix + (import ../container-modules/swarm-container-resolver.nix { + inherit (networkCfg) bridgeIp; + # The forwarder and its secret fetch dial the swarm collector, + # authelia and the store by name. + dnsConsumers = [ + "opentelemetry-collector.service" + "${forwarderUnit}.service" + ]; + }) + # Go: `SSL_CERT_FILE` replaces its trust store, for the swarm + # collector's and authelia's names. + (caTrust.trustBundle { + inherit pkgs; + name = machine; + consumers = [ "opentelemetry-collector" ]; + }) + ]; services.hyperhive.swarmContainer.privateNetwork = false; + # Authenticated as the forwarder's own OIDC client, with the cache's + # store identity: one identity for the whole container. It is a + # system credential, so the fetch unit loads it. + services.hyperhive.swarmJournalForwarder = { + enable = true; + serviceName = "attic"; + endpoint = forwarderEndpoint; + clientId = forwarderClientId; + tokenUrl = "${toString autheliaCfg.url}/api/oidc/token"; + telemetryPort = forwarderTelemetryPort; + store = { + package = baoDeploy.package; + addr = "https://${baoCfg.domain}:${toString baoCfg.port}"; + secretPath = "secret/swarm/services/${forwarderClientId}/oidc/client"; + clientCertFile = forwarderCredential "attic-bao-cert"; + clientKeyFile = forwarderCredential "attic-bao-key"; + caFile = + if forwarderStoreFiles ? attic-bao-server-ca then + forwarderCredential "attic-bao-server-ca" + else + null; + }; + }; + systemd.services.${forwarderUnit}.serviceConfig.LoadCredential = lib.attrNames forwarderStoreFiles; + systemd.services.${unit} = { description = "attic binary cache server"; wantedBy = [ "multi-user.target" ]; diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index cac6f8ee3..2784f8c10 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -1187,7 +1187,9 @@ let # The in-swarm cache's three secrets, create and read and nothing more: its # host units create each one only while it is absent, so none needs `update`, - # and a rewrite of a signing key or a token an operator holds is refused. + # and a rewrite of a signing key or a token an operator holds is refused. The + # cache container's journal forwarder reads its client secret under the same + # leaf. atticWriters = [ { name = "swarm-attic"; @@ -1203,7 +1205,8 @@ let atticJwtLeaf atticPushTokenLeaf atticPullTokenLeaf - ]; + ] + + readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.attic.otel.clientId}/oidc/client"; } ]; diff --git a/nix/host-modules/swarm-otel-service.nix b/nix/host-modules/swarm-otel-service.nix index d9317a59a..d911165ce 100644 --- a/nix/host-modules/swarm-otel-service.nix +++ b/nix/host-modules/swarm-otel-service.nix @@ -304,8 +304,12 @@ in swarm-notify-router = { clientId = hyperhiveCfg.swarm.notify-router.clientId; }; - # The forwarders in ./swarm-ci-runner.nix, ./swarm-grafana.nix and - # ./swarm-victorialogs.nix, each authenticated as a client of its own. + # The forwarders in ./swarm-attic.nix, ./swarm-ci-runner.nix, + # ./swarm-grafana.nix and ./swarm-victorialogs.nix, each authenticated as a + # client of its own. + swarm-attic = { + clientId = hyperhiveCfg.swarm.attic.otel.clientId; + }; swarm-ci-runner = { clientId = hyperhiveCfg.swarm.ci-runner.otel.clientId; }; diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index c4beb06dc..9c73bb81c 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -202,7 +202,7 @@ let # `swarm-builder`, `swarm-claude-login`, `swarm-notify-router`, # `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`, # `swarm-victorialogs`, `hive-forge`, `hive-matrix`, `swarm-victoriametrics`, - # `swarm-authelia`) forwards its own + # `swarm-authelia`, `swarm-attic`) forwards its own # journal from its own collector, so this receiver never sees their units. hostJournalDir = "/var/log/journal"; hostJournalMount = "/run/host-journal"; diff --git a/nix/host-modules/swarm-secret-publisher.nix b/nix/host-modules/swarm-secret-publisher.nix index 6c21cc74c..0c78a4662 100644 --- a/nix/host-modules/swarm-secret-publisher.nix +++ b/nix/host-modules/swarm-secret-publisher.nix @@ -55,8 +55,9 @@ let # The notification router, which reads it itself inside its container. hyperhiveCfg.swarm.notify-router.clientId # The journal forwarders in the CI runner's, Grafana's, the log store's, - # the forge's and the homeserver's containers, each reading it inside its - # container. + # the cache's, the forge's and the homeserver's containers, each reading it + # inside its container. + hyperhiveCfg.swarm.attic.otel.clientId hyperhiveCfg.swarm.ci-runner.otel.clientId hyperhiveCfg.swarm.grafana.otel.clientId hyperhiveCfg.swarm.victorialogs.otel.clientId diff --git a/nix/module-eval/bao-grants.nix b/nix/module-eval/bao-grants.nix index d11af2e30..76cc9641a 100644 --- a/nix/module-eval/bao-grants.nix +++ b/nix/module-eval/bao-grants.nix @@ -1980,9 +1980,10 @@ let } { # The cache's host units create each of its three secrets and read them - # back, and may not replace one: no `update`. The notification router's - # grant is the control that `grantsIn` reads a one-path grant too. - name = "the cache's grant is create and read on its signing key and its two tokens, and nothing else"; + # back, and may not replace one: no `update`. Its container's journal + # forwarder reads its own client secret, read only. The notification + # router's grant is the control that `grantsIn` reads a one-path grant too. + name = "the cache's grant is create and read on its signing key and its two tokens, read on its forwarder client, and nothing else"; ok = let s = baoGrantHere.systemd.services.swarm-bao-attic-policy.script; @@ -1995,7 +1996,9 @@ let (grantFor grants "secret/data/swarm/services/attic/jwt-secret").caps == caps && (grantFor grants "secret/data/swarm/services/attic/push-token").caps == caps && (grantFor grants "secret/data/swarm/services/attic/pull-token").caps == caps - && lib.length grants == 3 + && + (grantFor grants "secret/data/swarm/services/swarm-attic-collector/oidc/client").caps == [ "read" ] + && lib.length grants == 4 && lib.length (grantsIn baoGrantHere.systemd.services.swarm-bao-notify-router-policy.script) == 1 && lib.hasInfix "auth/cert/certs/swarm-attic" s && lib.hasInfix "allowed_common_names=swarm-attic" s diff --git a/nix/module-eval/builder.nix b/nix/module-eval/builder.nix index e1ee2ca91..e4cdc06be 100644 --- a/nix/module-eval/builder.nix +++ b/nix/module-eval/builder.nix @@ -385,6 +385,36 @@ let && lib.hasInfix "--mode monolithic" atticUnit.serviceConfig.ExecStart && atticDeploy.retentionPeriod == "14 days"; } + { + # atticd's journal reaches the swarm collector through the container's own + # forwarder, under the cache's store leaf. The host with the switch forced + # off is the control that nothing of it renders there, and `atticHere` + # that the same lookups find it where the cache runs. + name = "the cache's container forwards its journal to the swarm collector as its own client, and only where the cache runs"; + ok = + let + inner = atticHere.containers.swarm-attic; + f = inner.config.services.hyperhive.swarmJournalForwarder; + fetch = inner.config.systemd.services.swarm-journal-forwarder-oidc; + collector = inner.config.systemd.services.opentelemetry-collector; + in + f.enable + && f.serviceName == "attic" + && f.clientId == "swarm-attic-collector" + && f.endpoint == "https://${atticHere.services.hyperhive.swarm.otel.domain}/swarm-attic" + && f.store.secretPath == "secret/swarm/services/swarm-attic-collector/oidc/client" + && f.store.clientCertFile == "/run/credentials/swarm-journal-forwarder-oidc.service/attic-bao-cert" + && lib.elem "attic-bao-key" fetch.serviceConfig.LoadCredential + && lib.elem "--load-credential=attic-bao-cert:/var/lib/swarm-bao-pki/attic.pem" inner.extraFlags + && !(lib.elem "--link-journal=host" inner.extraFlags) + && collector.enable + && + atticHere.services.hyperhive.swarm.otel.serviceProducers.swarm-attic.clientId + == "swarm-attic-collector" + && lib.hasInfix "services/swarm-attic-collector/oidc/client" atticHere.systemd.services.swarm-bao-attic-policy.script + && !(atticExternal.containers ? swarm-attic) + && atticExternal.services.hyperhive.swarm.otel.serviceProducers ? swarm-attic; + } { # attic checks its own tokens and a nix client cannot follow a login # redirect, so the vhost carries no `auth_request`; the tokens ride in a diff --git a/nix/module-eval/secret-publisher.nix b/nix/module-eval/secret-publisher.nix index ad9082256..cdc6b169a 100644 --- a/nix/module-eval/secret-publisher.nix +++ b/nix/module-eval/secret-publisher.nix @@ -221,7 +221,7 @@ let # The forwarders in the CI runner's, Grafana's, the log store's, the # forge's, the homeserver's and authelia's containers read their secrets # from these paths, and the metrics store's host reads its container's. - name = "the publisher carries the CI runner, Grafana, log store, forge, homeserver, metrics store and authelia forwarders' secrets"; + name = "the publisher carries the CI runner, Grafana, log store, cache, forge, homeserver, metrics store and authelia forwarders' secrets"; ok = lib.all ( @@ -238,6 +238,7 @@ let "swarm-matrix-collector" "swarm-victoriametrics-collector" "swarm-authelia-collector" + "swarm-attic-collector" ]; } { From fb6cd36b5f54f199e0734bcee1a51e7ecbd25adf Mon Sep 17 00:00:00 2001 From: atlas Date: Tue, 27 Oct 2026 16:03:44 +0100 Subject: [PATCH 2/2] format the docs tables and the services page --- docs/swarm/credentials-reference.md | 70 ++++++++++++++--------------- docs/swarm/services.md | 1 + 2 files changed, 36 insertions(+), 35 deletions(-) diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 9976fada4..2a5a31af4 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -98,41 +98,41 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. ### Swarm-level — one of each per swarm -| secret | generated by | lives at | hive elsewhere | -| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | -| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | -| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | -| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store ([how](credentials.md#hive-level-secrets)) | -| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | -| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) | -| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) | -| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | -| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | -| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services//secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs Grafana on that file | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it | -| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 | -| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | -| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | -| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | -| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | -| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | -| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | -| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | -| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | -| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | -| the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | -| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's plaintext half (`swarm.authelia.otel.clientId`). `swarm-secret-publish` copies it to `swarm/services//oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes. The two copies can differ while the publisher hasn't caught up, for example while the store is down after a re-mint. Authelia then refuses the forwarder's token request, and the collector retries until the next read lands the new value and restarts it | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. The mint's own copy stays in `/var/lib/authelia-swarm/oidc-clients/.secret` | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf | -| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | -| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | -| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | -| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | -| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | +| secret | generated by | lives at | hive elsewhere | +| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | +| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | +| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | +| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | `/var/lib/authelia-swarm/oidc-clients/.secret` | operator provides the file and names it in whichever option reads it — `sso.clientSecretFile` for a service; the forge and, by default, the hive collector need none — `swarm-bao-forge-oidc` and `swarm-bao-hive-otel-oidc` fetch it from the store ([how](credentials.md#hive-level-secrets)) | +| OIDC client secret, digest half | the same mint | `oidc-clients/.digest` | authelia's own half; merged at runtime via `settingsFiles` | +| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) | +| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/run/swarm-grafana-secrets/.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see [hive-level secrets](credentials.md#hive-level-secrets) | +| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services//admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services//admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](credentials.md#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | +| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services//metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services//metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer `: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | +| Grafana's datasource encryption key | `swarm-bao-grafana-secret-key.service` stores it in the swarm secret store at `swarm/services//secret-key` when the store holds none, then copies it out, **on every host that runs Grafana**. It stores the key an earlier release generated at `/var/lib/grafana-secret/secret_key` in the container, so the key doesn't change, and mints a new one only on a host with no such file. Same grant as the admin password, create without update. The earlier file stays on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs Grafana on that file | `/run/swarm-grafana-secrets/secret-key` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container: `security.secret_key` | same unit, same path. Grafana doesn't start until the file exists. Changing the key leaves Grafana unable to read the datasource secrets it stored, so never delete the store path to rotate it | +| the swarm controller's webhook secret | `swarm-controller` stores it in the swarm secret store at `swarm/controller/swarm-controller/forge/webhook-secret` when the store holds none: the value of `webhook-secret` in its state directory, or a new one when there is no such file. The state directory file stays on disk and is the fallback while the store is unreachable or its policy doesn't grant the path yet | the swarm secret store; the controller holds it in memory and registers the same value with the forge's hooks | nothing to place: the controller's store leaf already exists. A store that stays unreachable on a start with no state directory file leaves the webhook endpoint answering 503 | +| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | +| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret and authelia's five keys alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` | +| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | +| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | +| an agent's claude login (`.credentials.json`) | `swarm-claude-login.service`, in its container, when an operator completes `claude auth login` from the agent's card in the swarm UI; it stores the file as a new version at `swarm/agents//claude-credentials` under its own leaf, which may create and update that path and read nothing under an agent | the swarm secret store at that path; the agent's `hive-agent` reads it under the agent's own leaf every 30 seconds while it waits for a login and writes it to `~/.claude/.credentials.json` | the same read, co-located or not; nothing for an operator to place beyond the agent's store leaf. When the store's copy and the agent's differ the KV version decides: the agent installs only a version newer than the last it installed (`claude-credentials-version` under the harness dir), because claude rotates the refresh token in its own copy and the stored one goes stale. The agent ignores a stale or equal version, and a relogin writes a newer one — [claude-login.md](claude-login.md) | +| the queue responder's copy of its OIDC secret | the responder reads it out of the swarm secret store itself as it starts, under its own leaf, and again when authelia refuses it | the daemon's memory only; never on disk | same read, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue container's journal forwarder's copy of the queue's OIDC secret | `swarm-journal-forwarder-oidc.service` inside the `swarm-nats` container reads it out of the swarm secret store under the responder's store leaf, and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. After a reboot the collector restarts until the file exists, then resumes from its saved journal cursor. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the CI runner, Grafana, log store, cache, forge and homeserver containers' journal forwarders' OIDC secrets | `swarm-journal-forwarder-oidc.service` inside each container reads its forwarder client's secret (`swarm..otel.clientId`) out of the swarm secret store under the store leaf the host already holds for that service, handed in as a system credential (bound in read-only, for the homeserver's), and again every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`, never on persistent disk; `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond those leaves (`deploy.bao.ciRunnerClientCertFile`, `deploy.bao.grafanaOidcClientCertFile`, `deploy.victorialogs.authBaoClientCertFile`, `deploy.bao.forgePublisherClientCertFile`, `deploy.bao.atticClientCertFile`, `deploy.matrix.ctlBaoClientCertFile`) | +| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | +| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | +| the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | +| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's plaintext half (`swarm.authelia.otel.clientId`). `swarm-secret-publish` copies it to `swarm/services//oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes. The two copies can differ while the publisher hasn't caught up, for example while the store is down after a re-mint. Authelia then refuses the forwarder's token request, and the collector retries until the next read lands the new value and restarts it | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. The mint's own copy stays in `/var/lib/authelia-swarm/oidc-clients/.secret` | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf | +| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | +| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | +| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | +| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | +| the secret store's own contents | openbao, on first `bao operator init` — **an operator action, not a unit** | inside the `swarm-bao` container, at its own `/var/lib/openbao`, kept across rebuilds by `ephemeral = false`. ⚠️ **Not a host path**: `nixos-container destroy swarm-bao` takes the raft data with it, so back up the container's tree, not `/var/lib/`. Only the store's TLS material (`/var/lib/swarm-bao-tls`) and its PKCS11 token (`/var/lib/swarm-bao-token`) are host-level | n/a — there is one store; a hive elsewhere is a _client_ of it and holds none of this | +| the secret store's unseal material | the HSM/TPM under `deploy.bao.seal = "pkcs11"`; openbao itself under `"shamir"` | in the token; or held by whoever ran `bao operator init`, which is what `"shamir"` means and why it's stated rather than inferred | n/a — only the host running the store seals anything | ### Hive-level — one of each per hive diff --git a/docs/swarm/services.md b/docs/swarm/services.md index 913e2e161..2247dbbee 100644 --- a/docs/swarm/services.md +++ b/docs/swarm/services.md @@ -454,6 +454,7 @@ external alternative, are in [`builder.md`](builder.md#binary-cache). The container forwards atticd's journal to the swarm collector like the other service containers (see [Telemetry collector](#telemetry-collector-otel)). The two host units' lines stay in the host's journal. + ### CI metrics The CI run poller, a `swarm-ci-metrics` oneshot on a 5-minute timer on the host