From 7294707d7d9350a57fb11b41d5239de91dd3644d Mon Sep 17 00:00:00 2001 From: atlas Date: Fri, 16 Oct 2026 12:05:10 +0200 Subject: [PATCH] swarm-victoriametrics: forward the container's journal to the swarm collector The metrics store's container now runs the journal forwarder the other service containers run, pushing to a receiver of its own on the swarm collector (serviceProducers swarm-victoriametrics) as a new machine OIDC client (swarm.victoriametrics.otel.clientId), registered by a glue file wherever authelia runs and published to the store by the secret publisher. The container keeps its loopback-only netns. The collector cannot dial a unix socket, so swarm-victoriametrics-egress-relay, a socket-activated systemd-socket-proxyd inside the container, takes its connections on 127.0.0.1:4380 and passes them to /run/swarm-victoriametrics-egress/egress.sock. The host's nginx listens there and sends /swarm-victoriametrics/ to the swarm collector and /api/oidc/token to authelia, both by name and verified, and refuses every other path. nginx gets ReadWritePaths on that directory and removes a stale socket before it starts. The container cannot reach the store, so the client secret is fetched on the host by swarm-victoriametrics-forwarder-oidc under swarm-metrics-auth's leaf, the shape swarm-bao-forwarder-oidc has, and bound into the container read-only. The forwarder module gains hostSecretFile for that: set, it renders no fetch unit. swarm-metrics-auth's policy gains one read stanza on the forwarder's client path. The container drops --link-journal=host, and the host collector stops reading it. Also lists the forge publisher's and matrix-ctl's leaves in the forwarder secrets row of docs/swarm/secrets.md, which the hive-forge/hive-matrix change missed. --- docs/swarm/credentials.md | 54 ++-- docs/swarm/secrets.md | 5 +- docs/swarm/services.md | 27 +- .../swarm-journal-forwarder.nix | 31 ++- nix/host-modules/default.nix | 1 + .../glue-victoriametrics-otel-oidc-client.nix | 37 +++ nix/host-modules/swarm-bao.nix | 13 +- nix/host-modules/swarm-otel-service.nix | 6 + nix/host-modules/swarm-otel.nix | 6 +- nix/host-modules/swarm-secret-publisher.nix | 2 + .../swarm-victoriametrics-service.nix | 16 ++ nix/host-modules/swarm-victoriametrics.nix | 231 +++++++++++++++++- nix/module-eval/bao-grants.nix | 12 +- nix/module-eval/nats-authelia.nix | 70 ++++++ nix/module-eval/secret-publisher.nix | 5 +- 15 files changed, 454 insertions(+), 62 deletions(-) create mode 100644 nix/host-modules/glue-victoriametrics-otel-oidc-client.nix diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index 572d16fb..5643374a 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -55,33 +55,33 @@ of the cell says how. -| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | -| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `swarm/agents//matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | -| `swarm/agents//matrix/` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it | -| `swarm/agents//matrix-recovery/` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | -| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | -| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | -| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | -| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | -| `swarm/agents//bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | -| `swarm/agents//queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | -| `swarm/agents//forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | -| `swarm/agents//forge/