mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-authelia: run the secret publisher and a journal forwarder inside the container

Browse source 
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.

The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.

The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
This commit is contained in:
atlas 2026-10-16 15:05:29 +02:00 • committed by mara
commit b68a8ef290
36 changed files with 753 additions and 363 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/scheduler/observability.md
View file

@ -137,7 +137,7 @@ the label isn't taken from the payload at all.
**Nothing to set where the hive holds its store identity.** The hive's collector **Nothing to set where the hive holds its store identity.** The hive's collector
reads its secret from the swarm's secret store under the hive's own leaf reads its secret from the swarm's secret store under the hive's own leaf
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches (`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client`, which the publisher on authelia's host `swarm/hives/<hive>/oidc/client`, which the publisher in authelia's container
writes. It runs at boot and retries every 30s until the store answers; when a writes. It runs at boot and retries every 30s until the store answers; when a
run lands a changed value it restarts the collector. A timer runs it again run lands a changed value it restarts the collector. A timer runs it again
every ten minutes, so a secret rotated later reaches the collector within every ten minutes, so a secret rotated later reaches the collector within

4
docs/swarm/README.md
View file

@ -108,8 +108,8 @@ token endpoint is the swarm's single address:
Both default to the swarm's own addresses on every hive, so there is nothing to Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, on the own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, in
host that runs authelia, puts it there. A hive without a store leaf doesn't authelia's container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too. publish; setting `natsUrl` to null turns publishing off too.
Until the secret is in the store, hive-c0re shows a `swarm_queue_config` Until the secret is in the store, hive-c0re shows a `swarm_queue_config`

56
docs/swarm/credentials.md
View file

@ -55,34 +55,34 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value | | `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds | | `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it | | `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s, authelia's journal forwarder's under `swarm-secret-publisher`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes | | `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run | | `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts | | `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build | | `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account (read on the Actions unit of every organisation's repos through a `ci-metrics` team in each), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-ci-metrics` on its host, under its own certificate, once per pass; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the poller reads the store on every pass | | `swarm/services/ci-metrics/forge/token` | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account (read on the Actions unit of every organisation's repos through a `ci-metrics` team in each), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-ci-metrics` on its host, under its own certificate, once per pass; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the poller reads the store on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

12
docs/swarm/secrets.md
View file

@ -54,6 +54,7 @@ neither is a renaming of the other.
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` | | Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes | | Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) | | the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
@ -66,6 +67,7 @@ neither is a renaming of the other.
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) | | the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store | | the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
| the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | | the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's plaintext half (`swarm.authelia.otel.clientId`). `swarm-secret-publish` copies it to `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes. The two copies can differ while the publisher hasn't caught up, for example while the store is down after a re-mint. Authelia then refuses the forwarder's token request, and the collector retries until the next read lands the new value and restarts it | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. The mint's own copy stays in `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
@ -262,8 +264,8 @@ such as the root token.
The forge and the homeserver read their copies out of the store, the way The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the homeserver under matrix-ctl's. Each **fails loudly** and retries until the
publisher on authelia's host has written the secret, rather than skipping. A publisher in authelia's container has written the secret, rather than skipping.
silent skip produces a service whose login button always fails, which is a A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause. symptom many layers from its cause.
The store's **first reader** is the agent containers' queue credential: The store's **first reader** is the agent containers' queue credential:
@ -272,7 +274,7 @@ the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is the secret by path, the id by value. No local fallback exists, and none is
possible, so absent files mean an agent built before `swarm-logs` read its own possible, so absent files mean an agent built before `swarm-logs` read its own
queue secret has no working `swarm-logs`, which is what a swarm looks like queue secret has no working `swarm-logs`, which is what a swarm looks like
before the publisher on the authelia host has run. before the publisher in authelia's container has run.
The reader runs before `hive-c0re.service`, which wants it (doesn't require The reader runs before `hive-c0re.service`, which wants it (doesn't require
it), so an agent container never renders ahead of the credential; an it), so an agent container never renders ahead of the credential; an
unreachable store delays the daemon's start rather than failing it. unreachable store delays the daemon's start rather than failing it.
@ -296,8 +298,8 @@ The **second reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly that's a _swarm service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host `swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the publisher on that runs Grafana. Where authelia is doesn't enter into it — the publisher in
authelia's host writes that path whether the reader is a network away or in the authelia's container writes that path whether the reader is a network away or in the
container next door. container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext That's a deliberate refusal of the shortcut. Copying the plaintext

18
docs/swarm/services.md
View file

@ -127,6 +127,14 @@ The swarm collector scrapes authelia's own metrics at
container. The gateway admits only the collector's OAuth2 client there; container. The gateway admits only the collector's OAuth2 client there;
any other session gets `deny`. any other session gets `deny`.
The container also runs `swarm-secret-publish`, which copies the client
secrets authelia mints into the swarm secret store under the secret
publisher's own store leaf, handed in as a system credential; see
[`secrets.md`](secrets.md). Authelia doesn't wait for it, so the IdP comes up
while the store is down, and the publisher retries every 30 seconds. The
container's journal forwarder reads its client secret back from the store
under the same leaf, and is off where the container has none.
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`secrets.md`](secrets.md) for where authelia relying-party flow, and [`secrets.md`](secrets.md) for where authelia
generates and reads each of its keys. generates and reads each of its keys.
@ -301,17 +309,17 @@ gateway either way.
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`, The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`, `swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `hive-forge` and `hive-matrix` `swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `hive-forge`
containers each run a collector of their own that forwards the container's and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client (`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter and classifier reuse the of that service's. The queue, builder, login minter and classifier reuse the
service's own client; the store, CI runner, Grafana, log store, metrics store, service's own client; the store, CI runner, Grafana, log store, metrics store,
forge and homeserver have a forwarder client of their own authelia, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`, (`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `review-classifier`, `ci-runner`, `nats`, `builder`, `claude-login`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `forgejo`, `tuwunel`) and no `grafana`, `victorialogs`, `victoriametrics`, `authelia`, `forgejo`,
`hive` label. The forwarder of all but the store's is `tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`. `nix/container-modules/swarm-journal-forwarder.nix`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable` **Both store exporters are unconditional**, and `deploy.victoriametrics.enable`

2
docs/swarm/sso.md
View file

@ -122,7 +122,7 @@ mechanism with flags.
### 1. All-local — one host runs both ### 1. All-local — one host runs both
Nothing to configure at all. authelia mints each client's secret at its Nothing to configure at all. authelia mints each client's secret at its
first boot, and `swarm-secret-publish` on its host writes it to the swarm first boot, and `swarm-secret-publish` in its container writes it to the swarm
secret store at `swarm/services/<client id>/oidc/client`. A host-side unit secret store at `swarm/services/<client id>/oidc/client`. A host-side unit
then reads it from the store and writes it into the service's container: then reads it from the store and writes it into the service's container:
`swarm-bao-forge-oidc` for the forge, under the forge publisher's store `swarm-bao-forge-oidc` for the forge, under the forge publisher's store

6
hive-c0re/src/swarm_queue.rs
View file

@ -14,8 +14,8 @@
//! environment. The client secret comes from the swarm's secret store, at //! environment. The client secret comes from the swarm's secret store, at
//! `swarm_secret_client::queue::hive_client_path`, read under this hive's own //! `swarm_secret_client::queue::hive_client_path`, read under this hive's own
//! certificate, held only in memory and read again when authelia refuses it //! certificate, held only in memory and read again when authelia refuses it
//! (`swarm_queue_client::stored_secret`). `swarm-secret-publish`, on the host //! (`swarm_queue_client::stored_secret`). `swarm-secret-publish`, in authelia's
//! that runs authelia, puts it there. //! container, puts it there.
//! //!
//! Connects lazily on first use rather than at boot — nothing here //! Connects lazily on first use rather than at boot — nothing here
//! blocks `hive-c0re` starting up on hosts with no queue configured, //! blocks `hive-c0re` starting up on hosts with no queue configured,
@ -286,7 +286,7 @@ async fn fetch_secret() -> Result<String> {
match stored { match stored {
Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()), Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()),
_ => bail!( _ => bail!(
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \ "nothing at {path}: swarm-secret-publish, in authelia's container, has not \
published this hive's client secret" published this hive's client secret"
), ),
} }

188
nix/container-modules/swarm-secret-publisher.nix Normal file
View file

@ -0,0 +1,188 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its own and its agents'
# credentials, a service or the controller its own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it.
#
# Imported inside authelia's container, beside the mint: it reads each secret
# where authelia writes it. The host module that declares the container sets
# the options below, as for ./swarm-journal-forwarder.nix.
#
# ⚠️ "Whoever", including a reader on authelia's own host. A swarm service's
# secret goes into the store even when the service runs beside authelia,
# because its reader fetches it from the store in every deployment —
# ../host-modules/swarm-grafana.nix states the ruling that made that the only
# route. Publishing "only when the reader is elsewhere" would be a second shape
# of this unit, gated on a fact about another host, to save a round trip on
# the one host that can afford it.
#
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit that writes a handful of secrets
# has no business holding it. ../host-modules/swarm-bao.nix grants this one
# `create`/`update` under the hive prefix and nothing else — no `read`.
#
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself.
#
# ⚠️ Nothing in the container waits for this unit. Authelia is the swarm's IdP
# and starts whether or not the store answers; a store that is down or sealed
# is a failed run that restarts.
{
config,
lib,
pkgs,
...
}:
let
cfg = config.services.hyperhive.swarmSecretPublisher;
in
{
options.services.hyperhive.swarmSecretPublisher = {
enable = lib.mkEnableOption "copying the OIDC client secrets authelia mints into the swarm secret store";
clientSecretDir = lib.mkOption {
type = lib.types.str;
description = ''
Directory authelia mints each client's plaintext into, as
`<client id>.secret`. A path inside the container.
'';
};
secrets = lib.mkOption {
type = lib.types.listOf (
lib.types.submodule {
options = {
clientId = lib.mkOption {
type = lib.types.str;
description = "The minted client, whose `<client id>.secret` is copied.";
};
path = lib.mkOption {
type = lib.types.str;
example = "secret/swarm/services/swarm-grafana/oidc/client";
description = "KV path its reader fetches the value from.";
};
withClientId = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Also store the client id beside the value, as `client_id`, for
a reader that has to reconstruct it from the store alone.
'';
};
};
}
);
description = "Every secret this unit copies, and where to.";
};
store = {
package = lib.mkOption {
type = lib.types.package;
description = "The `bao` CLI the unit logs in and writes with.";
};
addr = lib.mkOption {
type = lib.types.str;
description = "The swarm secret store's address, by name.";
};
clientCertFile = lib.mkOption {
type = lib.types.str;
description = ''
The publisher's store identity, as a path inside the container. Its
subject must be
{option}`services.hyperhive.deploy.bao.secretPublisherCommonName`.
A path, never a value.
'';
};
clientKeyFile = lib.mkOption {
type = lib.types.str;
description = "Private key for `store.clientCertFile`. A path, never a value.";
};
caFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = "CA the store's server certificate chains to; null means the system trust store.";
};
};
};
config = lib.mkIf cfg.enable {
# Re-publish when authelia rotates a secret. The mint writes the file, so
# the file is the event — there is no signal from authelia to subscribe to.
systemd.paths.swarm-secret-publish = {
description = "watch for minted OIDC client secrets to publish";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = cfg.clientSecretDir;
Unit = "swarm-secret-publish.service";
};
};
systemd.services.swarm-secret-publish = {
description = "publish the swarm's minted secrets to the swarm secret store";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
path = [
cfg.store.package
pkgs.coreutils
];
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
# every later start from the path unit above into a no-op.
serviceConfig = {
Type = "oneshot";
# Bounded here rather than left to systemd's default, so the number a
# boot waits on is in the file that waits. A sealed store answers on
# the port and never answers the write.
TimeoutStartSec = 60;
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = cfg.store.addr;
BAO_CLIENT_CERT = cfg.store.clientCertFile;
BAO_CLIENT_KEY = cfg.store.clientKeyFile;
}
// lib.optionalAttrs (cfg.store.caFile != null) {
BAO_CACERT = cfg.store.caFile;
};
script = ''
set -euo pipefail
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Unhandled on purpose: `Restart=on-failure` above is what a store that
# cannot authenticate this container should get. Degrading here would
# report "published 0" as an ordinary quiet day.
BAO_TOKEN="$(bao login -method=cert -token-only)"
export BAO_TOKEN
published=0
skipped=0
${lib.concatMapStringsSep "\n" (s: ''
src=${lib.escapeShellArg "${cfg.clientSecretDir}/${s.clientId}.secret"}
if [ -s "$src" ]; then
# `value=@$src` hands bao the PATH: bao opens the file itself, so
# the plaintext is never an argument of this process. Writing it as
# `value="$(cat "$src")"` would publish it to /proc for anyone in
# the container to read.
bao kv put ${lib.escapeShellArg s.path} \
value=@"$src"${lib.optionalString s.withClientId " client_id=${lib.escapeShellArg s.clientId}"}
published=$((published + 1))
else
# Not an error: authelia mints on its FIRST BOOT, so an absent file
# is "not yet", and the path unit above re-runs this when it lands.
# A client registered nowhere in this swarm is never minted at all.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') cfg.secrets}
echo "published $published client secret(s), skipped $skipped"
'';
};
};
}

6
nix/host-modules/deploy.nix
View file

@ -49,9 +49,9 @@ let
statusSecretRemoved = '' statusSecretRemoved = ''
hive-c0re reads this hive's status-publishing client secret from the hive-c0re reads this hive's status-publishing client secret from the
swarm's secret store, under the hive's own certificate swarm's secret store, under the hive's own certificate
(deploy.bao.clientCertFile), and swarm-secret-publish on the host that runs (deploy.bao.clientCertFile), and swarm-secret-publish in authelia's container
authelia puts it there. Remove this definition; the file it named is read puts it there. Remove this definition; the file it named is read by
by nothing now and can be deleted. nothing now and can be deleted.
''; '';
in in
{ {

4
nix/host-modules/glue-bao-tls.nix
View file

@ -197,8 +197,8 @@ in
# The secret publisher's, minted here for the reason the controller's # The secret publisher's, minted here for the reason the controller's
# line above gives — and it serves that case more often, not less: the # line above gives — and it serves that case more often, not less: the
# publisher runs beside AUTHELIA, which is the one host guaranteed not # publisher runs in AUTHELIA's container, on the one host
# to be this one whenever the store has a host of its own. # guaranteed not to be this one once the store has a host of its own.
[ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \ [ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \
${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth

2
nix/host-modules/glue-queue-agent-credential.nix
View file

@ -102,7 +102,7 @@ in
to make and the parent stays c0re's. to make and the parent stays c0re's.
Absent files mean this hive has no queue credential yet, which is what Absent files mean this hive has no queue credential yet, which is what
a swarm looks like before the publisher on the authelia host has run. a swarm looks like before the publisher in authelia's container has run.
''; '';
}; };
}; };

2
nix/host-modules/glue-secret-publisher-authelia.nix
View file

@ -2,7 +2,7 @@
# #
# ONE PAIRING PER FILE — publisher ← authelia, and nothing else. Deleting this # ONE PAIRING PER FILE — publisher ← authelia, and nothing else. Deleting this
# leaves the publisher off unless an operator enables it, so every OIDC client # leaves the publisher off unless an operator enables it, so every OIDC client
# secret authelia mints stays on this host and no reader gets one, this # secret authelia mints stays in its container and no reader gets one, this
# host's included. # host's included.
# #
# ⚠️ Gated on authelia alone, NOT on `deploy.bao.enable`: a publisher beside a # ⚠️ Gated on authelia alone, NOT on `deploy.bao.enable`: a publisher beside a

3
nix/host-modules/glue-secret-publisher-bao-identity.nix
View file

@ -6,7 +6,8 @@
# #
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the # ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, because the thing that owns a private key owns issuing from it. What # leaf, because the thing that owns a private key owns issuing from it. What
# belongs here is the pairing: which paths this host's publisher reads. # belongs here is the pairing: which host paths go to the publisher in
# authelia's container.
# #
# ⚠️ Gated on the leaf existing, not on the store being enabled — the same rule # ⚠️ Gated on the leaf existing, not on the store being enabled — the same rule
# ./glue-controller-bao-identity.nix states. A publisher on the store's own # ./glue-controller-bao-identity.nix states. A publisher on the store's own

7
nix/host-modules/hive-forge/default.nix
View file

@ -40,7 +40,8 @@ let
# root. Removed by the tmpfiles rule below. # root. Removed by the tmpfiles rule below.
oldOidcSecretPath = "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret"; oldOidcSecretPath = "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
# Where swarm-secret-publish on authelia's host leaves this client's secret. # Where swarm-secret-publish in authelia's container leaves this client's
# secret.
# The forge publisher's store role grants read on this path. # The forge publisher's store role grants read on this path.
oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client"; oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
@ -1196,8 +1197,8 @@ in
# The forge's OIDC client secret, out of the swarm secret store under the # The forge's OIDC client secret, out of the swarm secret store under the
# forge publisher's leaf — the one route, wherever authelia runs. Same # forge publisher's leaf — the one route, wherever authelia runs. Same
# shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the # shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
# read both fail loudly, and `storeRetry` retries until the publisher on # read both fail loudly, and `storeRetry` retries until the publisher in
# authelia's host has written the secret. # authelia's container has written the secret.
# #
# On the host, which reads that leaf from its own files. A host with no # On the host, which reads that leaf from its own files. A host with no
# publisher leaf fails here, naming the options, rather than rendering no # publisher leaf fails here, naming the options, rather than rendering no

3
nix/host-modules/hive-matrix.nix
View file

@ -48,7 +48,8 @@ let
# Where the secret sat on persistent disk before it moved to /run. # Where the secret sat on persistent disk before it moved to /run.
legacyHostSecretPath = "/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret"; legacyHostSecretPath = "/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret";
# Where swarm-secret-publish on authelia's host leaves this client's secret. # Where swarm-secret-publish in authelia's container leaves this client's
# secret.
# matrix-ctl's store role grants read on this path. # matrix-ctl's store role grants read on this path.
storeSecretPath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client"; storeSecretPath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";

6
nix/host-modules/otel.nix
View file

@ -369,7 +369,7 @@ in
# ./glue-hive-otel-authelia.nix restates this gate. # ./glue-hive-otel-authelia.nix restates this gate.
waits = senderAuth && !fetches; waits = senderAuth && !fetches;
secretWaitUnit = "hive-otel-client-secret"; secretWaitUnit = "hive-otel-client-secret";
# Written by swarm-secret-publish on authelia's host. The nix half of # Written by swarm-secret-publish in authelia's container. The nix half of
# `swarm_secret_client::queue::hive_client_path`. # `swarm_secret_client::queue::hive_client_path`.
storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client"; storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { }; atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
@ -829,8 +829,8 @@ in
# This hive's OIDC client secret, out of the swarm secret store under the # This hive's OIDC client secret, out of the swarm secret store under the
# hive's own leaf. Same shape as ./hive-matrix.nix's # hive's own leaf. Same shape as ./hive-matrix.nix's
# `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and # `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and
# `storeRetry` retries until the publisher on authelia's host has written # `storeRetry` retries until the publisher in authelia's container has
# the secret, because the collector does not start without it. # written the secret, because the collector does not start without it.
# #
# The collector re-reads `client_secret_file` on every token fetch, but # The collector re-reads `client_secret_file` on every token fetch, but
# the file it reads is `LoadCredential`'s copy, made at start: a changed # the file it reads is `LoadCredential`'s copy, made at start: a changed

16
nix/host-modules/swarm-authelia-service.nix
View file

@ -440,5 +440,21 @@ in
`swarm-controller`'s OIDC client. `swarm-controller`'s OIDC client.
''; '';
}; };
otel.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-authelia-collector";
description = ''
OAuth2 client id the journal forwarder inside authelia's container
authenticates as, and the audience it asks its token for.
Read-only because three ends have to agree on it: authelia registers
the client, the swarm collector admits it on the container's receiver
(`swarm.otel.serviceProducers`), and the secret publisher carries its
secret to the store, where the forwarder reads it back under the
publisher's identity.
'';
};
}; };
} }

205
nix/host-modules/swarm-authelia.nix
View file

@ -181,9 +181,82 @@ let
caBundleModule = caTrust.trustBundle { caBundleModule = caTrust.trustBundle {
inherit pkgs; inherit pkgs;
name = cfg.machine; name = cfg.machine;
consumers = [ "swarm-authelia-bridge" ]; consumers = [
"swarm-authelia-bridge"
]
# Go, for the swarm collector's name and authelia's own. Only where the
# forwarder runs: naming it would otherwise define an empty unit.
++ lib.optional publishes "opentelemetry-collector";
}; };
# The container's journal forwarder pushes to its own receiver on the swarm
# collector, the route `serviceProducers.swarm-authelia` in
# ./swarm-otel-service.nix declares, as a client of its own. The port suffix
# is elided on 443, as in ./swarm-bao.nix.
forwarderClientId = cfg.otel.clientId;
gatewayPortSuffix = lib.optionalString (
gatewayCfg.httpsPort != 443
) ":${toString gatewayCfg.httpsPort}";
forwarderEndpoint = "https://${hyperhiveCfg.swarm.otel.domain}${gatewayPortSuffix}/${cfg.machine}";
# In the host netns: distinct from every other collector's there, which the
# assertion below checks.
forwarderTelemetryPort = 8901;
# A client this container registers and mints for itself. Its secret goes
# through the store like every other forwarder's: the publisher carries it
# there and the forwarder reads it back under the same identity.
forwarderClient = {
id = forwarderClientId;
description = "HyperHive authelia journal forwarder";
kind = "machine";
redirectUris = [ ];
# Its own id as its own permitted audience, the one the swarm collector's
# receiver for this container checks.
audience = [ forwarderClientId ];
# The receiver verifies the token offline against `/jwks.json`;
# authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
};
# The secret publisher runs in this container, beside the mint whose secrets
# it copies. It needs a store identity, and a reader is defined by holding a
# certificate the store accepts, never by standing next to it — the rule
# ./glue-queue-agent-credential.nix states in full. `deploy.bao.enable` here
# would be the co-location assumption itself.
publisherCfg = deployCfg.swarm-secret-publisher;
publishes =
publisherCfg.enable
&& publisherCfg.baoClientCertFile != null
&& publisherCfg.baoClientKeyFile != null;
publisherUnit = "swarm-secret-publish";
forwarderFetchUnit = "swarm-journal-forwarder-oidc";
# The publisher's store identity, handed into the container as system
# credentials (credential id → host source). The forwarder's fetch presents
# it too: one identity for the whole container.
storeFiles = lib.optionalAttrs publishes (
lib.filterAttrs (_: source: source != null) {
publisher-bao-cert = publisherCfg.baoClientCertFile;
publisher-bao-key = publisherCfg.baoClientKeyFile;
publisher-bao-server-ca = deployCfg.bao.serverCaFile;
}
);
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
# The store options both units take, with the credentials as each loads them.
storeIdentityIn =
unit:
let
credential = id: "/run/credentials/${unit}.service/${id}";
in
{
package = deployCfg.bao.package;
addr = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
clientCertFile = credential "publisher-bao-cert";
clientKeyFile = credential "publisher-bao-key";
caFile =
if storeFiles ? publisher-bao-server-ca then credential "publisher-bao-server-ca" else null;
};
# The SWARM's domain, because that is where the protected apps now live # The SWARM's domain, because that is where the protected apps now live
# (`forge.<swarm>`, `chat.<swarm>`, `auth.<swarm>`). It moves in the # (`forge.<swarm>`, `chat.<swarm>`, `auth.<swarm>`). It moves in the
# same commit as `domain` (./swarm-authelia-service.nix) and cannot lag # same commit as `domain` (./swarm-authelia-service.nix) and cannot lag
@ -529,7 +602,7 @@ in
# register every service checks itself against and its port, is # register every service checks itself against and its port, is
# `swarm.authelia` in ./swarm-authelia-service.nix. What the host running # `swarm.authelia` in ./swarm-authelia-service.nix. What the host running
# the container decides is here — which two builds it # the container decides is here — which two builds it
# runs, and three filesystem paths that only exist on the machine running # runs, and two filesystem paths that only exist on the machine running
# `swarm-authelia`. A hive that does not run it has nothing at any of those # `swarm-authelia`. A hive that does not run it has nothing at any of those
# paths. `enable` already lives in ./deploy.nix, which also carries the # paths. `enable` already lives in ./deploy.nix, which also carries the
# renames. # renames.
@ -578,28 +651,6 @@ in
''; '';
}; };
hostClientSecretDir = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "/var/lib/nixos-containers/${cfg.machine}${clientsDir}";
description = ''
Where the minted client secrets sit **as seen from the host** —
`<id>.secret` holds a plaintext, `<id>.digest` the hash authelia
itself reads.
Published for the same reason as `hostUsersFile`: the plaintext's
other reader lives in a **different container**, and containers
that share this host's network namespace still have separate
filesystem roots. The host is the only place both trees are
addressable, so the host is where a delivery step has to run.
⚠️ Nothing here exists until authelia's **first boot** has run.
A consumer must wait for it — it cannot be a `bindMounts` source,
because nixos-container refuses to start when a bind source is
missing, and that turns a fresh hive into a boot-order deadlock.
'';
};
hostUsersFile = lib.mkOption { hostUsersFile = lib.mkOption {
type = lib.types.str; type = lib.types.str;
readOnly = true; readOnly = true;
@ -628,7 +679,10 @@ in
# — see `oidcEnabled`'s comment above. # — see `oidcEnabled`'s comment above.
services.hyperhive.swarm.authelia.oidc.clients = services.hyperhive.swarm.authelia.oidc.clients =
lib.optionals cfg.oidc.hiveIdentities (hiveClients ++ agentClients) lib.optionals cfg.oidc.hiveIdentities (hiveClients ++ agentClients)
++ [ bridgeClient ] ++ [
bridgeClient
forwarderClient
]
++ lib.optional (swarmDomain != null) swarmctlClient; ++ lib.optional (swarmDomain != null) swarmctlClient;
# A redirect URI on a machine client is not harmless-but-unused: it # A redirect URI on a machine client is not harmless-but-unused: it
@ -772,6 +826,39 @@ in
+ "evaluating and deploying cleanly. A browser client has a user " + "evaluating and deploying cleanly. A browser client has a user "
+ "to authorise and does not need it."; + "to authorise and does not need it.";
} }
{
assertion = lib.all oneWord (lib.attrValues storeFiles);
message = ''
The secret publisher's store identity paths must not contain
whitespace: nspawn's --load-credential receives them through
nixos-container's unquoted extra flags.
'';
}
{
assertion =
(hyperhiveCfg.swarm.otel.serviceProducers.${cfg.machine}.clientId or null) == forwarderClientId;
message = ''
services.hyperhive.swarm.otel.serviceProducers has no entry
"${cfg.machine}" with clientId "${forwarderClientId}", so the swarm
collector has no receiver for authelia's journal forwarder. Declare
it in swarm-otel-service.nix.
'';
}
{
assertion =
!(lib.elem forwarderTelemetryPort [
hyperhiveCfg.otel.telemetryPort
hyperhiveCfg.swarm.otel.telemetryPort
hyperhiveCfg.swarm.bao.otel.telemetryPort
hyperhiveCfg.swarm.nats.otel.telemetryPort
]);
message = ''
authelia's journal forwarder's telemetry port (${toString forwarderTelemetryPort})
collides with another collector's in the host netns: the hive
collector's, the swarm collector's, the store forwarder's or the
queue forwarder's.
'';
}
]; ];
# Authelia's own gateway surface: the vhost that fronts it and the # Authelia's own gateway surface: the vhost that fronts it and the
@ -899,15 +986,25 @@ in
}; };
# Order the container after the host CA generator, so the bind source # Order the container after the host CA generator, so the bind source
# exists before nspawn sets the mount up. # exists before nspawn sets the mount up. nspawn also fails to start it
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering; # when a credential source is missing, so where this host mints the
# publisher's leaf the container pulls the minting unit in and waits for
# it, as ./swarm-grafana.nix's does. The minting is local: the store need
# not be up.
systemd.services."container@${cfg.machine}" = lib.mkMerge [
caTrust.containerOrdering
(lib.mkIf publishes {
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];
})
];
containers.${cfg.machine} = { containers.${cfg.machine} = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes # No `--link-journal=host`: the journal stays in the guest, where the
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. # container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = [ "--link-journal=host" ]; extraFlags = lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
inherit privateNetwork; inherit privateNetwork;
# Public trust bundle only, read-only. Empty when the gateway is not # Public trust bundle only, read-only. Empty when the gateway is not
# self-signed, so the whole trust path drops out cleanly. # self-signed, so the whole trust path drops out cleanly.
@ -918,15 +1015,61 @@ in
{ {
imports = [ imports = [
../container-modules/swarm-container.nix ../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
../container-modules/swarm-secret-publisher.nix
(import ../container-modules/swarm-container-resolver.nix { (import ../container-modules/swarm-container-resolver.nix {
inherit (networkCfg) bridgeIp; inherit (networkCfg) bridgeIp;
dnsConsumers = [ cfg.unit ]; # The forwarder dials the swarm collector and authelia by name,
# and its fetch and the publisher the store.
dnsConsumers = [
cfg.unit
]
++ lib.optionals publishes [
"opentelemetry-collector.service"
"${forwarderFetchUnit}.service"
"${publisherUnit}.service"
];
}) })
caBundleModule caBundleModule
]; ];
services.hyperhive.swarmContainer = { inherit privateNetwork; }; services.hyperhive.swarmContainer = { inherit privateNetwork; };
# Authenticated as the forwarder's own OIDC client, its secret read
# from the store with the publisher's identity: one identity for the
# whole container. With no identity there is nothing to read it with,
# and the forwarder is off.
services.hyperhive.swarmJournalForwarder = lib.mkIf publishes {
enable = true;
serviceName = "authelia";
endpoint = forwarderEndpoint;
clientId = forwarderClientId;
tokenUrl = "${toString cfg.url}/api/oidc/token";
telemetryPort = forwarderTelemetryPort;
store = storeIdentityIn forwarderFetchUnit // {
secretPath = "secret/swarm/services/${forwarderClientId}/oidc/client";
};
};
systemd.services.${forwarderFetchUnit} = lib.mkIf publishes {
serviceConfig.LoadCredential = lib.attrNames storeFiles;
};
# Copies what the mint below leaves in `clientsDir` into the store,
# under the publisher's own identity, a system credential. Nothing
# orders authelia after it.
services.hyperhive.swarmSecretPublisher = lib.mkIf publishes {
enable = true;
clientSecretDir = clientsDir;
inherit (publisherCfg) secrets;
store = storeIdentityIn publisherUnit;
};
systemd.services.${publisherUnit} = lib.mkIf publishes {
# A run before the mint would only skip every secret and wait
# for the path unit.
after = [ "${unitName}-secrets.service" ];
serviceConfig.LoadCredential = lib.attrNames storeFiles;
};
# The authelia binary itself, so an operator who gets a shell # The authelia binary itself, so an operator who gets a shell
# in here can run `authelia crypto hash generate` to make a # in here can run `authelia crypto hash generate` to make a
# password for the users file. Without it the container runs # password for the users file. Without it the container runs

32
nix/host-modules/swarm-bao.nix
View file

@ -481,7 +481,7 @@ let
secretPublisherPolicyName = "swarm-secret-publisher"; secretPublisherPolicyName = "swarm-secret-publisher";
secretPublisherCn = baoDeploy.secretPublisherCommonName; secretPublisherCn = baoDeploy.secretPublisherCommonName;
# Three grants and one denial, and every narrowing in each is load-bearing. # Four grants and six denials, and every narrowing in each is load-bearing.
# #
# `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than # `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than
# written by the caller — same trap as the controller's grant above. # written by the caller — same trap as the controller's grant above.
@ -494,8 +494,9 @@ let
# here too — a grant widens when a path gains a WRITER, not when a kind is # here too — a grant widens when a path gains a WRITER, not when a kind is
# declared. # declared.
# #
# Write-only. It copies secrets in and never reads one back; a read # Write-only but for one leaf, the client secret of its container's forwarder:
# capability would let a file-copier recover every hive's credentials. # a read on a prefix would let a file-copier recover every hive's credentials.
# That stanza repeats `create`/`update`, which its exact path would drop.
# #
# The queue callout's three seeds sit under `swarm/services/` and are no # The queue callout's three seeds sit under `swarm/services/` and are no
# secret this principal copies, so their exact paths, which outrank the # secret this principal copies, so their exact paths, which outrank the
@ -550,6 +551,10 @@ let
path "${credentialMountPath}/data/${controllerClientLeaf}" { path "${credentialMountPath}/data/${controllerClientLeaf}" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
path "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.authelia.otel.clientId}/oidc/client" {
capabilities = ["create", "update", "read"]
}
''; '';
# The identity the matrix container's `swarm-matrix-ctl` presents. Named outside `hive-*` # The identity the matrix container's `swarm-matrix-ctl` presents. Named outside `hive-*`
@ -1448,10 +1453,10 @@ let
# own and lands it in a collector's container — because that route is # own and lands it in a collector's container — because that route is
# proven and there is no second one worth inventing. # proven and there is no second one worth inventing.
# #
# Where the publisher on authelia's host leaves it. A swarm service's client # Where the publisher in authelia's container leaves it. A swarm service's
# is registered once for the whole swarm, so its secret is one value. The # client is registered once for the whole swarm, so its secret is one value.
# `swarm-forwarder-oidc` role's grant is this one path, so any other answers # The `swarm-forwarder-oidc` role's grant is this one path, so any other
# 403 however correct it looks. # answers 403 however correct it looks.
forwarderStoreSecretPath = "secret/swarm/services/${cfg.otel.clientId}/oidc/client"; forwarderStoreSecretPath = "secret/swarm/services/${cfg.otel.clientId}/oidc/client";
# On the host's /run (tmpfs), so the secret never reaches persistent disk: # On the host's /run (tmpfs), so the secret never reaches persistent disk:
# after a reboot `swarm-bao-forwarder-oidc` fetches it again. The container # after a reboot `swarm-bao-forwarder-oidc` fetches it again. The container
@ -1770,7 +1775,9 @@ in
controller's grant includes rewriting every hive's policy and login controller's grant includes rewriting every hive's policy and login
role, which is not something a unit whose whole job is copying one file role, which is not something a unit whose whole job is copying one file
should be able to do. This one's grant is `create`/`update` under the should be able to do. This one's grant is `create`/`update` under the
hive prefix and nothing else. hive and service prefixes and on the controller's client, and `read` on
one leaf: the client secret of the journal forwarder in authelia's
container, which presents this identity too.
⚠️ Same collision as its sibling above, and the same answer: ./swarm.nix ⚠️ Same collision as its sibling above, and the same answer: ./swarm.nix
feeds this value into the guard on feeds this value into the guard on
@ -2967,9 +2974,8 @@ in
# matters most and is the least obvious: # matters most and is the least obvious:
# 1. `nixos-container` refuses to start when a bind source is missing, # 1. `nixos-container` refuses to start when a bind source is missing,
# and a certificate minted on this same boot does not exist yet when # and a certificate minted on this same boot does not exist yet when
# the container is ordered. Same trap `hostClientSecretDir` # the container is ordered. Ordering against whatever mints it
# documents in ./swarm-authelia.nix. Ordering against whatever # belongs with whatever named the path, not here.
# mints it belongs with whatever named the path, not here.
# 2. `tlsDir` is already mounted at the same path inside, so a copy # 2. `tlsDir` is already mounted at the same path inside, so a copy
# needs no second mount. # needs no second mount.
# 3. A directory holding a leaf usually holds the CA's private key # 3. A directory holding a leaf usually holds the CA's private key
@ -3797,8 +3803,8 @@ in
;; ;;
esac esac
# Published by ./swarm-secret-publisher.nix on authelia's host, # Published by the secret publisher in authelia's container, which
# which this boot does not wait on, so absence is retried. # this boot does not wait on, so absence is retried.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg "${credentialMountPath}/${baoUiClientLeaf}"} 2>"$err")"; then if ! secret="$(bao kv get -field=value ${lib.escapeShellArg "${credentialMountPath}/${baoUiClientLeaf}"} 2>"$err")"; then
echo ${lib.escapeShellArg "the store did not return ${credentialMountPath}/${baoUiClientLeaf}: the UI's OIDC client secret is not published yet."} >&2 echo ${lib.escapeShellArg "the store did not return ${credentialMountPath}/${baoUiClientLeaf}: the UI's OIDC client secret is not published yet."} >&2
cat "$err" >&2 cat "$err" >&2

2
nix/host-modules/swarm-controller.nix
View file

@ -177,7 +177,7 @@ let
# No secret here. The daemon reads its client secret from the store at # No secret here. The daemon reads its client secret from the store at
# start, under `baoEnv`'s identity, from the path `swarm-secret-publish` # start, under `baoEnv`'s identity, from the path `swarm-secret-publish`
# writes it to on authelia's host (`queue_identity.rs`). # writes it to in authelia's container (`queue_identity.rs`).
# #
# Every value here comes from an option rather than from what happens to # Every value here comes from an option rather than from what happens to
# run on this host. The queue is not optional for a controller, but # run on this host. The queue is not optional for a controller, but

Some files were not shown because too many files have changed in this diff Show more