From b68a8ef29060db40032d3933e048cdeb83be5bd0 Mon Sep 17 00:00:00 2001 From: atlas Date: Fri, 16 Oct 2026 15:05:29 +0200 Subject: [PATCH] swarm-authelia: run the secret publisher and a journal forwarder inside the container swarm-secret-publish and its path unit now run inside authelia's container, beside the mint, and read each client secret at the path authelia writes it. The host unit and host path unit, which read the secrets through the container's root on the host, are gone, and so is deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps its store identity, the secret-publisher leaf. The host hands the leaf and the store's CA into the container as nspawn system credentials, and the unit loads them; where this host mints the leaf, the container waits for swarm-bao-pki. Authelia does not wait for the publisher, which retries every 30 seconds while the store is down. The container also runs the journal forwarder the other service containers run, pushing to a receiver of its own on the swarm collector (serviceProducers swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId) that authelia registers for itself and the publisher carries to the store. The fetch unit reads it back under the publisher's leaf, also handed to it as a system credential, so the swarm-secret-publisher policy gains read on that one leaf; its stanza keeps create and update, which the exact path would otherwise take from the services/* prefix. Without the leaf the forwarder is off. The container drops --link-journal=host, and the host collector stops reading it. The services' "nothing at " errors name the publisher in authelia's container. --- docs/scheduler/observability.md | 2 +- docs/swarm/README.md | 4 +- docs/swarm/credentials.md | 56 ++--- docs/swarm/secrets.md | 12 +- docs/swarm/services.md | 18 +- docs/swarm/sso.md | 2 +- hive-c0re/src/swarm_queue.rs | 6 +- .../swarm-secret-publisher.nix | 188 ++++++++++++++ nix/host-modules/deploy.nix | 6 +- nix/host-modules/glue-bao-tls.nix | 4 +- .../glue-queue-agent-credential.nix | 2 +- .../glue-secret-publisher-authelia.nix | 2 +- .../glue-secret-publisher-bao-identity.nix | 3 +- nix/host-modules/hive-forge/default.nix | 7 +- nix/host-modules/hive-matrix.nix | 3 +- nix/host-modules/otel.nix | 6 +- nix/host-modules/swarm-authelia-service.nix | 16 ++ nix/host-modules/swarm-authelia.nix | 205 ++++++++++++--- nix/host-modules/swarm-bao.nix | 32 ++- nix/host-modules/swarm-controller.nix | 2 +- nix/host-modules/swarm-grafana.nix | 15 +- nix/host-modules/swarm-otel-service.nix | 5 + nix/host-modules/swarm-otel.nix | 16 +- nix/host-modules/swarm-secret-publisher.nix | 236 +++++------------- nix/host-modules/swarm.nix | 4 +- nix/module-eval/bao-grants.nix | 16 +- nix/module-eval/builder.nix | 2 +- nix/module-eval/hive-otel.nix | 2 +- nix/module-eval/nats-authelia.nix | 69 +++-- nix/module-eval/secret-publisher.nix | 151 ++++++++--- swarm-builder/src/main.rs | 2 +- swarm-claude-login/src/main.rs | 2 +- swarm-controller/src/queue_identity.rs | 6 +- swarm-nats-auth/src/oidc_secret.rs | 8 +- swarm-queue-client/src/stored_secret.rs | 4 +- swarm-review-classifier/src/main.rs | 2 +- 36 files changed, 753 insertions(+), 363 deletions(-) create mode 100644 nix/container-modules/swarm-secret-publisher.nix diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index c57328cb..299a16db 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -137,7 +137,7 @@ the label isn't taken from the payload at all. **Nothing to set where the hive holds its store identity.** The hive's collector reads its secret from the swarm's secret store under the hive's own leaf (`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches -`swarm/hives//oidc/client`, which the publisher on authelia's host +`swarm/hives//oidc/client`, which the publisher in authelia's container writes. It runs at boot and retries every 30s until the store answers; when a run lands a changed value it restarts the collector. A timer runs it again every ten minutes, so a secret rotated later reaches the collector within diff --git a/docs/swarm/README.md b/docs/swarm/README.md index a276f305..2ca6acb2 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -108,8 +108,8 @@ token endpoint is the swarm's single address: Both default to the swarm's own addresses on every hive, so there is nothing to set for them. The client secret isn't an option: hive-c0re reads it from the swarm's secret store, at `swarm/hives//oidc/client`, under the hive's -own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, on the -host that runs authelia, puts it there. A hive without a store leaf doesn't +own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, in +authelia's container, puts it there. A hive without a store leaf doesn't publish; setting `natsUrl` to null turns publishing off too. Until the secret is in the store, hive-c0re shows a `swarm_queue_config` diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 89bdfcf0..f902cab6 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -55,34 +55,34 @@ of the cell says how. -| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | -| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `swarm/agents//matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | -| `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it | -| `swarm/agents//matrix-recovery/` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | -| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | -| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | -| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | -| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | -| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | -| `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | -| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | -| `swarm/agents//forge/