vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-authelia: run the secret publisher and a journal forwarder inside the container

Browse source 
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.

The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.

The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
This commit is contained in:
atlas 2026-10-16 15:05:29 +02:00 • committed by mara
commit b68a8ef290
36 changed files with 753 additions and 363 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/scheduler/observability.md
View file

@ -137,7 +137,7 @@ the label isn't taken from the payload at all.
**Nothing to set where the hive holds its store identity.** The hive's collector
reads its secret from the swarm's secret store under the hive's own leaf
(`deploy.bao.clientCertFile`): `swarm-bao-hive-otel-oidc.service` fetches
`swarm/hives/<hive>/oidc/client`, which the publisher on authelia's host
`swarm/hives/<hive>/oidc/client`, which the publisher in authelia's container
writes. It runs at boot and retries every 30s until the store answers; when a
run lands a changed value it restarts the collector. A timer runs it again
every ten minutes, so a secret rotated later reaches the collector within

4
docs/swarm/README.md
View file

@ -108,8 +108,8 @@ token endpoint is the swarm's single address:
Both default to the swarm's own addresses on every hive, so there is nothing to
set for them. The client secret isn't an option: hive-c0re reads it from the
swarm's secret store, at `swarm/hives/<hiveName>/oidc/client`, under the hive's
own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, on the
host that runs authelia, puts it there. A hive without a store leaf doesn't
own store leaf (`deploy.bao.clientCertFile`). `swarm-secret-publish`, in
authelia's container, puts it there. A hive without a store leaf doesn't
publish; setting `natsUrl` to null turns publishing off too.
Until the secret is in the store, hive-c0re shows a `swarm_queue_config`

56
docs/swarm/credentials.md
View file

@ -55,34 +55,34 @@ of the cell says how.
<!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account (read on the Actions unit of every organisation's repos through a `ci-metrics` team in each), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-ci-metrics` on its host, under its own certificate, once per pass; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the poller reads the store on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | ✅ a secondary that didn't come up at start, or whose token the homeserver rejected, is read from the store again two minutes later, then at doubling waits up to sixteen minutes, without restarting the daemon or its other accounts; one that's up and then rejected ends its sync loop, the daemon exits and its timer restarts it |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase for every account listed under the agent's `matrix/`, in a five-minute pass | the agent container itself, under the certificate its hive passed in, when `hive-matrix-daemon` brings the account up | ❌ never replaced: the account's server-side key backup opens only with the passphrase it was enabled with, so the pass writes only to an empty path | ❌ read when the account comes up; an account whose passphrase lands later turns backup on at the next daemon start |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, at start and on refusal | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read at start and held in memory; read again when authelia refuses it, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/forge/token` | the forge's `forgejo-swarm-controller-account`, in the `hive-forge` container, once, under the forge publisher's certificate with a write-only grant; it then revokes its earlier tokens | `swarm-controller`, under its own certificate, at start and on refusal | ❌ the forge mints it once and keeps only the token's name in the container, at `/var/lib/forgejo/swarm-controller-token-name`; deleting that file mints a new token on the unit's next start | ✅ the controller reads it at start, and in the background with backoff until it first finds it. When the forge answers 401 it reads it again and sends the request once more with the new token, at most once per backoff of 5 s doubling to 5 min while the store holds the refused value |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host, under Grafana's certificate: a token for the `swarm-controller` Viewer service account, minted with Grafana's admin password | `swarm-controller`, under its own certificate, on the first `/api/alerts` request and again when Grafana refuses it | ✅ a timer starts the unit again 10 minutes after each run; it keeps a stored token Grafana accepts, and mints a new one when Grafana answers 401 or the store holds none | ✅ the controller holds the token in memory; when Grafana answers 401 it reads the store once more and retries the request |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config; then `hive-agent-bao-identity` in the agent container, under the leaf it holds | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ✅ `hive-agent-bao-identity` re-reads it every 10 minutes and swaps a new leaf into `/run/hive-agent-bao-identity/`, which every store login in the container presents. It reads under the leaf it holds, so it renews only while that leaf is valid |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive; `swarm-logs` in the agent container, under the same certificate, on every query, as the bearer it presents at the log store; `swarm-nats-auth` and `swarm-logs-auth`, each under its own certificate, to check a presented secret | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff. `swarm-logs` reads it on every query and once more after a 401. `swarm-logs-auth` keeps granting a secret it verified for `deploy.victorialogs.authVerdictTtl` (60 s by default) without reading the path, so a replaced or deleted secret still works at the log store for up to that long |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer, and `hive-github-notify` re-reads the file on every poll |
| `swarm/agents/<agent>/claude-credentials` | `swarm-claude-login`, under its own certificate, when an operator logs the agent in from its card in the swarm UI; a new version per login | `hive-agent` in the agent container, under the agent's own certificate, into `~/.claude/.credentials.json`, only while the agent waits for a login and only a version newer than the one it installed last | ❌ an operator's login; a relogin writes a new version | ✅ the waiting agent reads the store every 30 seconds |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ✅ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held in memory and read again when authelia refuses it |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s, the review classifier's under `swarm-review-classifier`'s, the CI runner's, Grafana's, the log store's, the metrics store's, the forge's and the homeserver's journal forwarders' under `swarm-ci-runner`'s, `swarm-grafana-oidc`'s, `swarm-logs-auth`'s, `swarm-metrics-auth`'s, `swarm-forge-publisher`'s and `swarm-matrix-ctl`'s, authelia's journal forwarder's under `swarm-secret-publisher`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file under the host's `/run` that only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ✅ `swarm-bao-ci-runner` reads it at boot, on deploy and every ten minutes, and restarts the runner when the value changed |
| `swarm/services/<grafana client id>/metrics/bearer` | `swarm-bao-grafana-metrics-bearer` on Grafana's host, under Grafana's certificate, when the store holds none | `swarm-metrics-auth` on the metrics store's host, under its own certificate, at most once a minute, to check the bearer Grafana's datasource presents at the store's read API, and `swarm-logs-grafana-auth` on the log store's host, under `swarm-logs-auth`'s certificate, to check the same bearer at the log store's query API | ❌ nothing re-mints it; an operator deletes it and restarts `swarm-bao-grafana-metrics-bearer` | ✅ `swarm-metrics-auth` reads it again once its copy is a minute old. ❌ Grafana reads its file copy only as it starts; the bearer unit restarts Grafana when the copy changes |
| `swarm/services/review-classifier/anthropic-api-key` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every API key run; held for that run and handed to `claude`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/acp-provider` | the operator, by hand in the store's UI, field `api_key` | `swarm-review-classifier` in its container, under its own certificate, at the start of every ACP run; held for that run and handed to `opencode`'s environment only; no other principal's policy reads it | ❌ an operator's key; replaced by writing a new one | ✅ read on every run, so a written or removed key takes effect on the next run |
| `swarm/services/review-classifier/forge/token` | `swarm-controller`, with the forge's admin API, for its `review-classifier` account, at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-controller` under its own certificate, before it decides whether to mint, and on every review it posts as the account; the classifier service's policy doesn't read it | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the controller reads the store on every review it posts |
| `swarm/services/builder/forge/token` | `swarm-controller`, with the forge's admin API, for its `builder` account (read on every config repo through the `hives` team), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-builder` in its container, under its own certificate, on every build; held for that build and handed to `git` through the environment only; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the builder reads the store on every build |
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account (read on the Actions unit of every organisation's repos through a `ci-metrics` team in each), at start and in a five-minute pass; the secret publisher's policy denies it | `swarm-ci-metrics` on its host, under its own certificate, once per pass; `swarm-controller` under its own certificate, before it decides whether to mint | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the poller reads the store on every pass |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES -->

12
docs/swarm/secrets.md
View file

@ -54,6 +54,7 @@ neither is a renaming of the other.
| Grafana's admin password | `swarm-bao-grafana-admin.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/admin/password` when the store holds none, then copies it out, **on every host that runs Grafana**. Grafana's leaf may create that leaf but not update it, so the store keeps the first value, and the secret publisher can't write it. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/admin/password` with a token that may, such as the root token (Grafana's, the publisher's and the controller's can't), then `systemctl restart swarm-bao-grafana-admin`; a plain `bao kv delete` isn't enough, see [the callout xkey's section](#the-queues-callout-xkey) | `/run/swarm-grafana-secrets/admin-password` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, never on persistent disk: `security.admin_password` for a new database, and `grafana cli admin reset-admin-password` in `grafana.service`'s `preStart` for an existing one, on every start | same unit, same path. After a reboot the unit fetches the file again, and Grafana doesn't start until it exists, so it stays down rather than falling back to the upstream default `admin` |
| Grafana's metrics bearer | `swarm-bao-grafana-metrics-bearer.service` creates it in the swarm secret store at `swarm/services/<grafana client id>/metrics/bearer` when the store holds none, then copies it out, **on every host that runs Grafana**. Same grant as the admin password, create without update. To rotate, run `bao kv metadata delete secret/swarm/services/<grafana client id>/metrics/bearer` with a token that may, then `systemctl restart swarm-bao-grafana-metrics-bearer` | `/run/swarm-grafana-secrets/metrics-bearer` on the host's tmpfs, bind-mounted read-only into the `swarm-grafana` container, holding `Bearer <value>`: the metrics datasource's `Authorization` header. The logs datasource sends the same header. `swarm-metrics-auth` on the metrics store's host, and a second instance on the log store's host (`swarm-logs-grafana-auth`), read the stored value themselves, each under its own leaf | same unit, same path. The unit restarts Grafana when the file changes |
| the metrics store responder's own store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.metricsAuthCommonName`. `swarm-bao-metrics-auth-policy.service` writes the `swarm-metrics-auth` role and its read grants on the bearer leaf and on the metrics store forwarder's client secret, nothing else | `/var/lib/swarm-bao-pki/metrics-auth.pem` and `metrics-auth-key.pem` on the store's host, handed to `swarm-metrics-auth` through `LoadCredential` (paths in argv, never values) and read by path by `swarm-victoriametrics-forwarder-oidc` | **the operator** places a leaf with that subject on the metrics store's host and names it in `deploy.victoriametrics.authBaoClientCertFile` / `authBaoClientKeyFile`; evaluation fails naming both until then. If the store is on another host, deploy that host first: the role must exist before the responder's login succeeds, and until it does the gateway refuses every Grafana query (fail closed) |
| the secret publisher's store leaf (cert + key) | `signLeaf` in `glue-bao-tls.nix` (`swarm-bao-pki.service`), on the host that runs the secret store, once, subject `deploy.bao.secretPublisherCommonName`. `swarm-bao-secret-publisher-policy.service` writes the `swarm-secret-publisher` role: create and update under `swarm/hives/*`, `swarm/services/*` and the controller's client, and read on authelia's forwarder's client secret alone | `/var/lib/swarm-bao-pki/secret-publisher.pem` and `secret-publisher-key.pem` on the host, handed into authelia's container as nspawn system credentials, which `swarm-secret-publish` and the forwarder's `swarm-journal-forwarder-oidc` load with `LoadCredential`. nspawn won't start authelia's container while a named file is missing | **the operator** copies the leaf to authelia's host, beside the hive's own store leaf, where `glue-secret-publisher-bao-identity.nix` looks, or names it in `deploy.swarm-secret-publisher.baoClientCertFile` / `baoClientKeyFile` |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf** | `/var/lib/hive-otel-oidc/hive-<name>.secret` on the hive's host, 0400 root, loaded by `LoadCredential` | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
@ -66,6 +67,7 @@ neither is a renaming of the other.
| the metrics store container's journal forwarder's OIDC secret | `swarm-victoriametrics-forwarder-oidc.service` on the store's host reads its forwarder client's secret (`swarm.victoriametrics.otel.clientId`) out of the swarm secret store under the metrics store responder's leaf, and again every 10 minutes: the container has no network to reach the store | `/run/swarm-victoriametrics-forwarder-oidc/<id>.secret` on the host (tmpfs), root `0400`, never on persistent disk; bound into the container read-only, where `LoadCredential` hands it to the collector | same unit, same path. Nothing for an operator to place beyond that leaf (`deploy.victoriametrics.authBaoClientCertFile`) |
| the CI run poller's forge token | `swarm-controller`, with the forge's admin API, for its `ci-metrics` account, at start and in a five-minute pass | the swarm secret store at `swarm/services/ci-metrics/forge/token`; `swarm-ci-metrics` reads it once per pass under its own leaf and holds it in memory for that pass. When the store's token no longer matches the forge's, the controller's next pass mints a new one; until then the poller's forge calls answer 401, the unit fails and no cursor moves | nothing to place beyond the poller's leaf (`deploy.bao.ciMetricsClientCertFile`), on a host without the store |
| the CI run poller's copy of its OIDC secret | authelia; `swarm-secret-publish` copies it into the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's plaintext half (`swarm.authelia.otel.clientId`). `swarm-secret-publish` copies it to `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes. The two copies can differ while the publisher hasn't caught up, for example while the store is down after a re-mint. Authelia then refuses the forwarder's token request, and the collector retries until the next read lands the new value and restarts it | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. The mint's own copy stays in `/var/lib/authelia-swarm/oidc-clients/<id>.secret` | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
@ -262,8 +264,8 @@ such as the root token.
The forge and the homeserver read their copies out of the store, the way
Grafana does below: the forge under the forge publisher's leaf, the
homeserver under matrix-ctl's. Each **fails loudly** and retries until the
publisher on authelia's host has written the secret, rather than skipping. A
silent skip produces a service whose login button always fails, which is a
publisher in authelia's container has written the secret, rather than skipping.
A silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is the agent containers' queue credential:
@ -272,7 +274,7 @@ the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is
possible, so absent files mean an agent built before `swarm-logs` read its own
queue secret has no working `swarm-logs`, which is what a swarm looks like
before the publisher on the authelia host has run.
before the publisher in authelia's container has run.
The reader runs before `hive-c0re.service`, which wants it (doesn't require
it), so an agent container never renders ahead of the credential; an
unreachable store delays the daemon's start rather than failing it.
@ -296,8 +298,8 @@ The **second reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
that runs Grafana. Where authelia is doesn't enter into it — the publisher on
authelia's host writes that path whether the reader is a network away or in the
that runs Grafana. Where authelia is doesn't enter into it — the publisher in
authelia's container writes that path whether the reader is a network away or in the
container next door.
That's a deliberate refusal of the shortcut. Copying the plaintext

18
docs/swarm/services.md
View file

@ -127,6 +127,14 @@ The swarm collector scrapes authelia's own metrics at
container. The gateway admits only the collector's OAuth2 client there;
any other session gets `deny`.
The container also runs `swarm-secret-publish`, which copies the client
secrets authelia mints into the swarm secret store under the secret
publisher's own store leaf, handed in as a system credential; see
[`secrets.md`](secrets.md). Authelia doesn't wait for it, so the IdP comes up
while the store is down, and the publisher retries every 30 seconds. The
container's journal forwarder reads its client secret back from the store
under the same leaf, and is off where the container has none.
See [`sso.md`](sso.md) for bootstrapping the first user and the OIDC
relying-party flow, and [`secrets.md`](secrets.md) for where authelia
generates and reads each of its keys.
@ -301,17 +309,17 @@ gateway either way.
The `swarm-bao`, `swarm-nats`, `swarm-builder`, `swarm-claude-login`,
`swarm-review-classifier`, `swarm-ci-runner`, `swarm-grafana`,
`swarm-victorialogs`, `swarm-victoriametrics`, `hive-forge` and `hive-matrix`
containers each run a collector of their own that forwards the container's
`swarm-victorialogs`, `swarm-victoriametrics`, `swarm-authelia`, `hive-forge`
and `hive-matrix` containers each run a collector of their own that forwards the container's
journal here, on a route and receiver per service
(`https://<swarm.otel.domain>/<container>/`), authenticated as an OIDC client
of that service's. The queue, builder, login minter and classifier reuse the
service's own client; the store, CI runner, Grafana, log store, metrics store,
forge and homeserver have a forwarder client of their own
authelia, forge and homeserver have a forwarder client of their own
(`swarm.<service>.otel.clientId`). Their records carry `service.name` (`bao`,
`nats`, `builder`, `claude-login`, `review-classifier`, `ci-runner`,
`grafana`, `victorialogs`, `victoriametrics`, `forgejo`, `tuwunel`) and no
`hive` label. The forwarder of all but the store's is
`grafana`, `victorialogs`, `victoriametrics`, `authelia`, `forgejo`,
`tuwunel`) and no `hive` label. The forwarder of all but the store's is
`nix/container-modules/swarm-journal-forwarder.nix`.
**Both store exporters are unconditional**, and `deploy.victoriametrics.enable`

2
docs/swarm/sso.md
View file

@ -122,7 +122,7 @@ mechanism with flags.
### 1. All-local — one host runs both
Nothing to configure at all. authelia mints each client's secret at its
first boot, and `swarm-secret-publish` on its host writes it to the swarm
first boot, and `swarm-secret-publish` in its container writes it to the swarm
secret store at `swarm/services/<client id>/oidc/client`. A host-side unit
then reads it from the store and writes it into the service's container:
`swarm-bao-forge-oidc` for the forge, under the forge publisher's store

6
hive-c0re/src/swarm_queue.rs
View file

@ -14,8 +14,8 @@
//! environment. The client secret comes from the swarm's secret store, at
//! `swarm_secret_client::queue::hive_client_path`, read under this hive's own
//! certificate, held only in memory and read again when authelia refuses it
//! (`swarm_queue_client::stored_secret`). `swarm-secret-publish`, on the host
//! that runs authelia, puts it there.
//! (`swarm_queue_client::stored_secret`). `swarm-secret-publish`, in authelia's
//! container, puts it there.
//!
//! Connects lazily on first use rather than at boot — nothing here
//! blocks `hive-c0re` starting up on hosts with no queue configured,
@ -286,7 +286,7 @@ async fn fetch_secret() -> Result<String> {
match stored {
Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()),
_ => bail!(
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \
"nothing at {path}: swarm-secret-publish, in authelia's container, has not \
published this hive's client secret"
),
}

188
nix/container-modules/swarm-secret-publisher.nix Normal file
View file

@ -0,0 +1,188 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its own and its agents'
# credentials, a service or the controller its own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it.
#
# Imported inside authelia's container, beside the mint: it reads each secret
# where authelia writes it. The host module that declares the container sets
# the options below, as for ./swarm-journal-forwarder.nix.
#
# ⚠️ "Whoever", including a reader on authelia's own host. A swarm service's
# secret goes into the store even when the service runs beside authelia,
# because its reader fetches it from the store in every deployment —
# ../host-modules/swarm-grafana.nix states the ruling that made that the only
# route. Publishing "only when the reader is elsewhere" would be a second shape
# of this unit, gated on a fact about another host, to save a round trip on
# the one host that can afford it.
#
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit that writes a handful of secrets
# has no business holding it. ../host-modules/swarm-bao.nix grants this one
# `create`/`update` under the hive prefix and nothing else — no `read`.
#
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself.
#
# ⚠️ Nothing in the container waits for this unit. Authelia is the swarm's IdP
# and starts whether or not the store answers; a store that is down or sealed
# is a failed run that restarts.
{
config,
lib,
pkgs,
...
}:
let
cfg = config.services.hyperhive.swarmSecretPublisher;
in
{
options.services.hyperhive.swarmSecretPublisher = {
enable = lib.mkEnableOption "copying the OIDC client secrets authelia mints into the swarm secret store";
clientSecretDir = lib.mkOption {
type = lib.types.str;
description = ''
Directory authelia mints each client's plaintext into, as
`<client id>.secret`. A path inside the container.
'';
};
secrets = lib.mkOption {
type = lib.types.listOf (
lib.types.submodule {
options = {
clientId = lib.mkOption {
type = lib.types.str;
description = "The minted client, whose `<client id>.secret` is copied.";
};
path = lib.mkOption {
type = lib.types.str;
example = "secret/swarm/services/swarm-grafana/oidc/client";
description = "KV path its reader fetches the value from.";
};
withClientId = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Also store the client id beside the value, as `client_id`, for
a reader that has to reconstruct it from the store alone.
'';
};
};
}
);
description = "Every secret this unit copies, and where to.";
};
store = {
package = lib.mkOption {
type = lib.types.package;
description = "The `bao` CLI the unit logs in and writes with.";
};
addr = lib.mkOption {
type = lib.types.str;
description = "The swarm secret store's address, by name.";
};
clientCertFile = lib.mkOption {
type = lib.types.str;
description = ''
The publisher's store identity, as a path inside the container. Its
subject must be
{option}`services.hyperhive.deploy.bao.secretPublisherCommonName`.
A path, never a value.
'';
};
clientKeyFile = lib.mkOption {
type = lib.types.str;
description = "Private key for `store.clientCertFile`. A path, never a value.";
};
caFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
description = "CA the store's server certificate chains to; null means the system trust store.";
};
};
};
config = lib.mkIf cfg.enable {
# Re-publish when authelia rotates a secret. The mint writes the file, so
# the file is the event — there is no signal from authelia to subscribe to.
systemd.paths.swarm-secret-publish = {
description = "watch for minted OIDC client secrets to publish";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = cfg.clientSecretDir;
Unit = "swarm-secret-publish.service";
};
};
systemd.services.swarm-secret-publish = {
description = "publish the swarm's minted secrets to the swarm secret store";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
path = [
cfg.store.package
pkgs.coreutils
];
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
# every later start from the path unit above into a no-op.
serviceConfig = {
Type = "oneshot";
# Bounded here rather than left to systemd's default, so the number a
# boot waits on is in the file that waits. A sealed store answers on
# the port and never answers the write.
TimeoutStartSec = 60;
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = cfg.store.addr;
BAO_CLIENT_CERT = cfg.store.clientCertFile;
BAO_CLIENT_KEY = cfg.store.clientKeyFile;
}
// lib.optionalAttrs (cfg.store.caFile != null) {
BAO_CACERT = cfg.store.caFile;
};
script = ''
set -euo pipefail
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Unhandled on purpose: `Restart=on-failure` above is what a store that
# cannot authenticate this container should get. Degrading here would
# report "published 0" as an ordinary quiet day.
BAO_TOKEN="$(bao login -method=cert -token-only)"
export BAO_TOKEN
published=0
skipped=0
${lib.concatMapStringsSep "\n" (s: ''
src=${lib.escapeShellArg "${cfg.clientSecretDir}/${s.clientId}.secret"}
if [ -s "$src" ]; then
# `value=@$src` hands bao the PATH: bao opens the file itself, so
# the plaintext is never an argument of this process. Writing it as
# `value="$(cat "$src")"` would publish it to /proc for anyone in
# the container to read.
bao kv put ${lib.escapeShellArg s.path} \
value=@"$src"${lib.optionalString s.withClientId " client_id=${lib.escapeShellArg s.clientId}"}
published=$((published + 1))
else
# Not an error: authelia mints on its FIRST BOOT, so an absent file
# is "not yet", and the path unit above re-runs this when it lands.
# A client registered nowhere in this swarm is never minted at all.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') cfg.secrets}
echo "published $published client secret(s), skipped $skipped"
'';
};
};
}

6
nix/host-modules/deploy.nix
View file

@ -49,9 +49,9 @@ let
statusSecretRemoved = ''
hive-c0re reads this hive's status-publishing client secret from the
swarm's secret store, under the hive's own certificate
(deploy.bao.clientCertFile), and swarm-secret-publish on the host that runs
authelia puts it there. Remove this definition; the file it named is read
by nothing now and can be deleted.
(deploy.bao.clientCertFile), and swarm-secret-publish in authelia's container
puts it there. Remove this definition; the file it named is read by
nothing now and can be deleted.
'';
in
{

4
nix/host-modules/glue-bao-tls.nix
View file

@ -197,8 +197,8 @@ in
# The secret publisher's, minted here for the reason the controller's
# line above gives — and it serves that case more often, not less: the
# publisher runs beside AUTHELIA, which is the one host guaranteed not
# to be this one whenever the store has a host of its own.
# publisher runs in AUTHELIA's container, on the one host
# guaranteed not to be this one once the store has a host of its own.
[ -s ${pkiDir}/secret-publisher.pem ] || ${signLeaf} ${pkiDir} secret-publisher \
${lib.escapeShellArg deployCfg.bao.secretPublisherCommonName} "" clientAuth

2
nix/host-modules/glue-queue-agent-credential.nix
View file

@ -102,7 +102,7 @@ in
to make and the parent stays c0re's.
Absent files mean this hive has no queue credential yet, which is what
a swarm looks like before the publisher on the authelia host has run.
a swarm looks like before the publisher in authelia's container has run.
'';
};
};

2
nix/host-modules/glue-secret-publisher-authelia.nix
View file

@ -2,7 +2,7 @@
#
# ONE PAIRING PER FILE — publisher ← authelia, and nothing else. Deleting this
# leaves the publisher off unless an operator enables it, so every OIDC client
# secret authelia mints stays on this host and no reader gets one, this
# secret authelia mints stays in its container and no reader gets one, this
# host's included.
#
# ⚠️ Gated on authelia alone, NOT on `deploy.bao.enable`: a publisher beside a

3
nix/host-modules/glue-secret-publisher-bao-identity.nix
View file

@ -6,7 +6,8 @@
#
# ⚠️ The minting is NOT here. ./glue-bao-tls.nix holds the CA and signs the
# leaf, because the thing that owns a private key owns issuing from it. What
# belongs here is the pairing: which paths this host's publisher reads.
# belongs here is the pairing: which host paths go to the publisher in
# authelia's container.
#
# ⚠️ Gated on the leaf existing, not on the store being enabled — the same rule
# ./glue-controller-bao-identity.nix states. A publisher on the store's own

7
nix/host-modules/hive-forge/default.nix
View file

@ -40,7 +40,8 @@ let
# root. Removed by the tmpfiles rule below.
oldOidcSecretPath = "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
# Where swarm-secret-publish on authelia's host leaves this client's secret.
# Where swarm-secret-publish in authelia's container leaves this client's
# secret.
# The forge publisher's store role grants read on this path.
oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
@ -1196,8 +1197,8 @@ in
# The forge's OIDC client secret, out of the swarm secret store under the
# forge publisher's leaf — the one route, wherever authelia runs. Same
# shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
# read both fail loudly, and `storeRetry` retries until the publisher on
# authelia's host has written the secret.
# read both fail loudly, and `storeRetry` retries until the publisher in
# authelia's container has written the secret.
#
# On the host, which reads that leaf from its own files. A host with no
# publisher leaf fails here, naming the options, rather than rendering no

3
nix/host-modules/hive-matrix.nix
View file

@ -48,7 +48,8 @@ let
# Where the secret sat on persistent disk before it moved to /run.
legacyHostSecretPath = "/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret";
# Where swarm-secret-publish on authelia's host leaves this client's secret.
# Where swarm-secret-publish in authelia's container leaves this client's
# secret.
# matrix-ctl's store role grants read on this path.
storeSecretPath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";

6
nix/host-modules/otel.nix
View file

@ -369,7 +369,7 @@ in
# ./glue-hive-otel-authelia.nix restates this gate.
waits = senderAuth && !fetches;
secretWaitUnit = "hive-otel-client-secret";
# Written by swarm-secret-publish on authelia's host. The nix half of
# Written by swarm-secret-publish in authelia's container. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
storeSecretPath = "secret/swarm/hives/${toString hiveName}/oidc/client";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
@ -829,8 +829,8 @@ in
# This hive's OIDC client secret, out of the swarm secret store under the
# hive's own leaf. Same shape as ./hive-matrix.nix's
# `swarm-bao-matrix-oidc`: the login and the read both fail loudly, and
# `storeRetry` retries until the publisher on authelia's host has written
# the secret, because the collector does not start without it.
# `storeRetry` retries until the publisher in authelia's container has
# written the secret, because the collector does not start without it.
#
# The collector re-reads `client_secret_file` on every token fetch, but
# the file it reads is `LoadCredential`'s copy, made at start: a changed

16
nix/host-modules/swarm-authelia-service.nix
View file

@ -440,5 +440,21 @@ in
`swarm-controller`'s OIDC client.
'';
};
otel.clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "swarm-authelia-collector";
description = ''
OAuth2 client id the journal forwarder inside authelia's container
authenticates as, and the audience it asks its token for.
Read-only because three ends have to agree on it: authelia registers
the client, the swarm collector admits it on the container's receiver
(`swarm.otel.serviceProducers`), and the secret publisher carries its
secret to the store, where the forwarder reads it back under the
publisher's identity.
'';
};
};
}

205
nix/host-modules/swarm-authelia.nix
View file

@ -181,9 +181,82 @@ let
caBundleModule = caTrust.trustBundle {
inherit pkgs;
name = cfg.machine;
consumers = [ "swarm-authelia-bridge" ];
consumers = [
"swarm-authelia-bridge"
]
# Go, for the swarm collector's name and authelia's own. Only where the
# forwarder runs: naming it would otherwise define an empty unit.
++ lib.optional publishes "opentelemetry-collector";
};
# The container's journal forwarder pushes to its own receiver on the swarm
# collector, the route `serviceProducers.swarm-authelia` in
# ./swarm-otel-service.nix declares, as a client of its own. The port suffix
# is elided on 443, as in ./swarm-bao.nix.
forwarderClientId = cfg.otel.clientId;
gatewayPortSuffix = lib.optionalString (
gatewayCfg.httpsPort != 443
) ":${toString gatewayCfg.httpsPort}";
forwarderEndpoint = "https://${hyperhiveCfg.swarm.otel.domain}${gatewayPortSuffix}/${cfg.machine}";
# In the host netns: distinct from every other collector's there, which the
# assertion below checks.
forwarderTelemetryPort = 8901;
# A client this container registers and mints for itself. Its secret goes
# through the store like every other forwarder's: the publisher carries it
# there and the forwarder reads it back under the same identity.
forwarderClient = {
id = forwarderClientId;
description = "HyperHive authelia journal forwarder";
kind = "machine";
redirectUris = [ ];
# Its own id as its own permitted audience, the one the swarm collector's
# receiver for this container checks.
audience = [ forwarderClientId ];
# The receiver verifies the token offline against `/jwks.json`;
# authelia's default opaque handle fails there.
accessTokenSignedResponseAlg = "RS256";
};
# The secret publisher runs in this container, beside the mint whose secrets
# it copies. It needs a store identity, and a reader is defined by holding a
# certificate the store accepts, never by standing next to it — the rule
# ./glue-queue-agent-credential.nix states in full. `deploy.bao.enable` here
# would be the co-location assumption itself.
publisherCfg = deployCfg.swarm-secret-publisher;
publishes =
publisherCfg.enable
&& publisherCfg.baoClientCertFile != null
&& publisherCfg.baoClientKeyFile != null;
publisherUnit = "swarm-secret-publish";
forwarderFetchUnit = "swarm-journal-forwarder-oidc";
# The publisher's store identity, handed into the container as system
# credentials (credential id → host source). The forwarder's fetch presents
# it too: one identity for the whole container.
storeFiles = lib.optionalAttrs publishes (
lib.filterAttrs (_: source: source != null) {
publisher-bao-cert = publisherCfg.baoClientCertFile;
publisher-bao-key = publisherCfg.baoClientKeyFile;
publisher-bao-server-ca = deployCfg.bao.serverCaFile;
}
);
# nixos-container expands `extraFlags` unquoted, so a source path must be
# one word.
oneWord = path: builtins.match ".*[[:space:]].*" path == null;
# The store options both units take, with the credentials as each loads them.
storeIdentityIn =
unit:
let
credential = id: "/run/credentials/${unit}.service/${id}";
in
{
package = deployCfg.bao.package;
addr = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
clientCertFile = credential "publisher-bao-cert";
clientKeyFile = credential "publisher-bao-key";
caFile =
if storeFiles ? publisher-bao-server-ca then credential "publisher-bao-server-ca" else null;
};
# The SWARM's domain, because that is where the protected apps now live
# (`forge.<swarm>`, `chat.<swarm>`, `auth.<swarm>`). It moves in the
# same commit as `domain` (./swarm-authelia-service.nix) and cannot lag
@ -529,7 +602,7 @@ in
# register every service checks itself against and its port, is
# `swarm.authelia` in ./swarm-authelia-service.nix. What the host running
# the container decides is here — which two builds it
# runs, and three filesystem paths that only exist on the machine running
# runs, and two filesystem paths that only exist on the machine running
# `swarm-authelia`. A hive that does not run it has nothing at any of those
# paths. `enable` already lives in ./deploy.nix, which also carries the
# renames.
@ -578,28 +651,6 @@ in
'';
};
hostClientSecretDir = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = "/var/lib/nixos-containers/${cfg.machine}${clientsDir}";
description = ''
Where the minted client secrets sit **as seen from the host** —
`<id>.secret` holds a plaintext, `<id>.digest` the hash authelia
itself reads.
Published for the same reason as `hostUsersFile`: the plaintext's
other reader lives in a **different container**, and containers
that share this host's network namespace still have separate
filesystem roots. The host is the only place both trees are
addressable, so the host is where a delivery step has to run.
⚠️ Nothing here exists until authelia's **first boot** has run.
A consumer must wait for it — it cannot be a `bindMounts` source,
because nixos-container refuses to start when a bind source is
missing, and that turns a fresh hive into a boot-order deadlock.
'';
};
hostUsersFile = lib.mkOption {
type = lib.types.str;
readOnly = true;
@ -628,7 +679,10 @@ in
# — see `oidcEnabled`'s comment above.
services.hyperhive.swarm.authelia.oidc.clients =
lib.optionals cfg.oidc.hiveIdentities (hiveClients ++ agentClients)
++ [ bridgeClient ]
++ [
bridgeClient
forwarderClient
]
++ lib.optional (swarmDomain != null) swarmctlClient;
# A redirect URI on a machine client is not harmless-but-unused: it
@ -772,6 +826,39 @@ in
+ "evaluating and deploying cleanly. A browser client has a user "
+ "to authorise and does not need it.";
}
{
assertion = lib.all oneWord (lib.attrValues storeFiles);
message = ''
The secret publisher's store identity paths must not contain
whitespace: nspawn's --load-credential receives them through
nixos-container's unquoted extra flags.
'';
}
{
assertion =
(hyperhiveCfg.swarm.otel.serviceProducers.${cfg.machine}.clientId or null) == forwarderClientId;
message = ''
services.hyperhive.swarm.otel.serviceProducers has no entry
"${cfg.machine}" with clientId "${forwarderClientId}", so the swarm
collector has no receiver for authelia's journal forwarder. Declare
it in swarm-otel-service.nix.
'';
}
{
assertion =
!(lib.elem forwarderTelemetryPort [
hyperhiveCfg.otel.telemetryPort
hyperhiveCfg.swarm.otel.telemetryPort
hyperhiveCfg.swarm.bao.otel.telemetryPort
hyperhiveCfg.swarm.nats.otel.telemetryPort
]);
message = ''
authelia's journal forwarder's telemetry port (${toString forwarderTelemetryPort})
collides with another collector's in the host netns: the hive
collector's, the swarm collector's, the store forwarder's or the
queue forwarder's.
'';
}
];
# Authelia's own gateway surface: the vhost that fronts it and the
@ -899,15 +986,25 @@ in
};
# Order the container after the host CA generator, so the bind source
# exists before nspawn sets the mount up.
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
# exists before nspawn sets the mount up. nspawn also fails to start it
# when a credential source is missing, so where this host mints the
# publisher's leaf the container pulls the minting unit in and waits for
# it, as ./swarm-grafana.nix's does. The minting is local: the store need
# not be up.
systemd.services."container@${cfg.machine}" = lib.mkMerge [
caTrust.containerOrdering
(lib.mkIf publishes {
after = [ "swarm-bao-pki.service" ];
wants = [ "swarm-bao-pki.service" ];
})
];
containers.${cfg.machine} = {
autoStart = true;
ephemeral = false;
# Journal files on the host, not inside the container: nixpkgs hardcodes
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [ "--link-journal=host" ];
# No `--link-journal=host`: the journal stays in the guest, where the
# container's own forwarder reads it (../container-modules/swarm-journal-forwarder.nix).
extraFlags = lib.mapAttrsToList (id: source: "--load-credential=${id}:${source}") storeFiles;
inherit privateNetwork;
# Public trust bundle only, read-only. Empty when the gateway is not
# self-signed, so the whole trust path drops out cleanly.
@ -918,15 +1015,61 @@ in
{
imports = [
../container-modules/swarm-container.nix
../container-modules/swarm-journal-forwarder.nix
../container-modules/swarm-secret-publisher.nix
(import ../container-modules/swarm-container-resolver.nix {
inherit (networkCfg) bridgeIp;
dnsConsumers = [ cfg.unit ];
# The forwarder dials the swarm collector and authelia by name,
# and its fetch and the publisher the store.
dnsConsumers = [
cfg.unit
]
++ lib.optionals publishes [
"opentelemetry-collector.service"
"${forwarderFetchUnit}.service"
"${publisherUnit}.service"
];
})
caBundleModule
];
services.hyperhive.swarmContainer = { inherit privateNetwork; };
# Authenticated as the forwarder's own OIDC client, its secret read
# from the store with the publisher's identity: one identity for the
# whole container. With no identity there is nothing to read it with,
# and the forwarder is off.
services.hyperhive.swarmJournalForwarder = lib.mkIf publishes {
enable = true;
serviceName = "authelia";
endpoint = forwarderEndpoint;
clientId = forwarderClientId;
tokenUrl = "${toString cfg.url}/api/oidc/token";
telemetryPort = forwarderTelemetryPort;
store = storeIdentityIn forwarderFetchUnit // {
secretPath = "secret/swarm/services/${forwarderClientId}/oidc/client";
};
};
systemd.services.${forwarderFetchUnit} = lib.mkIf publishes {
serviceConfig.LoadCredential = lib.attrNames storeFiles;
};
# Copies what the mint below leaves in `clientsDir` into the store,
# under the publisher's own identity, a system credential. Nothing
# orders authelia after it.
services.hyperhive.swarmSecretPublisher = lib.mkIf publishes {
enable = true;
clientSecretDir = clientsDir;
inherit (publisherCfg) secrets;
store = storeIdentityIn publisherUnit;
};
systemd.services.${publisherUnit} = lib.mkIf publishes {
# A run before the mint would only skip every secret and wait
# for the path unit.
after = [ "${unitName}-secrets.service" ];
serviceConfig.LoadCredential = lib.attrNames storeFiles;
};
# The authelia binary itself, so an operator who gets a shell
# in here can run `authelia crypto hash generate` to make a
# password for the users file. Without it the container runs

32
nix/host-modules/swarm-bao.nix
View file

@ -481,7 +481,7 @@ let
secretPublisherPolicyName = "swarm-secret-publisher";
secretPublisherCn = baoDeploy.secretPublisherCommonName;
# Three grants and one denial, and every narrowing in each is load-bearing.
# Four grants and six denials, and every narrowing in each is load-bearing.
#
# `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than
# written by the caller — same trap as the controller's grant above.
@ -494,8 +494,9 @@ let
# here too — a grant widens when a path gains a WRITER, not when a kind is
# declared.
#
# Write-only. It copies secrets in and never reads one back; a read
# capability would let a file-copier recover every hive's credentials.
# Write-only but for one leaf, the client secret of its container's forwarder:
# a read on a prefix would let a file-copier recover every hive's credentials.
# That stanza repeats `create`/`update`, which its exact path would drop.
#
# The queue callout's three seeds sit under `swarm/services/` and are no
# secret this principal copies, so their exact paths, which outrank the
@ -550,6 +551,10 @@ let
path "${credentialMountPath}/data/${controllerClientLeaf}" {
capabilities = ["create", "update"]
}
path "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.authelia.otel.clientId}/oidc/client" {
capabilities = ["create", "update", "read"]
}
'';
# The identity the matrix container's `swarm-matrix-ctl` presents. Named outside `hive-*`
@ -1448,10 +1453,10 @@ let
# own and lands it in a collector's container — because that route is
# proven and there is no second one worth inventing.
#
# Where the publisher on authelia's host leaves it. A swarm service's client
# is registered once for the whole swarm, so its secret is one value. The
# `swarm-forwarder-oidc` role's grant is this one path, so any other answers
# 403 however correct it looks.
# Where the publisher in authelia's container leaves it. A swarm service's
# client is registered once for the whole swarm, so its secret is one value.
# The `swarm-forwarder-oidc` role's grant is this one path, so any other
# answers 403 however correct it looks.
forwarderStoreSecretPath = "secret/swarm/services/${cfg.otel.clientId}/oidc/client";
# On the host's /run (tmpfs), so the secret never reaches persistent disk:
# after a reboot `swarm-bao-forwarder-oidc` fetches it again. The container
@ -1770,7 +1775,9 @@ in
controller's grant includes rewriting every hive's policy and login
role, which is not something a unit whose whole job is copying one file
should be able to do. This one's grant is `create`/`update` under the
hive prefix and nothing else.
hive and service prefixes and on the controller's client, and `read` on
one leaf: the client secret of the journal forwarder in authelia's
container, which presents this identity too.
⚠️ Same collision as its sibling above, and the same answer: ./swarm.nix
feeds this value into the guard on
@ -2967,9 +2974,8 @@ in
# matters most and is the least obvious:
# 1. `nixos-container` refuses to start when a bind source is missing,
# and a certificate minted on this same boot does not exist yet when
# the container is ordered. Same trap `hostClientSecretDir`
# documents in ./swarm-authelia.nix. Ordering against whatever
# mints it belongs with whatever named the path, not here.
# the container is ordered. Ordering against whatever mints it
# belongs with whatever named the path, not here.
# 2. `tlsDir` is already mounted at the same path inside, so a copy
# needs no second mount.
# 3. A directory holding a leaf usually holds the CA's private key
@ -3797,8 +3803,8 @@ in
;;
esac
# Published by ./swarm-secret-publisher.nix on authelia's host,
# which this boot does not wait on, so absence is retried.
# Published by the secret publisher in authelia's container, which
# this boot does not wait on, so absence is retried.
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg "${credentialMountPath}/${baoUiClientLeaf}"} 2>"$err")"; then
echo ${lib.escapeShellArg "the store did not return ${credentialMountPath}/${baoUiClientLeaf}: the UI's OIDC client secret is not published yet."} >&2
cat "$err" >&2

2
nix/host-modules/swarm-controller.nix
View file

@ -177,7 +177,7 @@ let
# No secret here. The daemon reads its client secret from the store at
# start, under `baoEnv`'s identity, from the path `swarm-secret-publish`
# writes it to on authelia's host (`queue_identity.rs`).
# writes it to in authelia's container (`queue_identity.rs`).
#
# Every value here comes from an option rather than from what happens to
# run on this host. The queue is not optional for a controller, but

15
nix/host-modules/swarm-grafana.nix
View file

@ -278,7 +278,7 @@ let
"/var/lib/grafana-metrics/bearer"
];
# Where the publisher on authelia's host leaves this client's secret. A
# Where the publisher in authelia's container leaves this client's secret. A
# swarm service's client is registered once for the whole swarm, so its
# secret is one value and not one per hive. Both ends compose this from the
# same swarm-wide client id, so a rename cannot leave one of them behind.
@ -649,11 +649,11 @@ in
++ map (p: "r ${p}") legacyHostSecretPaths;
# THE delivery unit — one route, in every deployment. The secret authelia
# minted arrives out of the swarm secret store, which the publisher on
# authelia's host wrote it into, and that is true whether authelia is a
# network away or in the container next door.
# minted arrives out of the swarm secret store, which the publisher in
# authelia's container wrote it into, and that is true whether authelia is
# a network away or in the container next door.
#
# 🩸 No second unit copies the plaintext out of authelia's host tree
# 🩸 No second unit copies the plaintext out of authelia's tree
# where the two are co-located. The store exists so a host holds ONE
# out-of-band secret — its client certificate — and reads everything else
# with it, so skipping the store when the producer is local would buy a
@ -667,8 +667,9 @@ in
# that degrades QUIETLY, since no retry turns "no value there" into a value.
#
# ⚠️ An ABSENT secret is "not published yet", not a fault of this host's.
# The publisher runs on the authelia host and authelia mints on its first
# boot, so early in a swarm's life there is legitimately nothing to read.
# The publisher runs in authelia's container and authelia mints on its
# first boot, so early in a swarm's life there is legitimately nothing to
# read.
# Nothing here writes a stand-in: there is no such thing as a locally valid
# OIDC client secret, and a placeholder would turn a Grafana that cannot
# log anyone in into one whose token exchange is refused.

5
nix/host-modules/swarm-otel-service.nix
View file

@ -442,6 +442,11 @@ in
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
port = lib.mkDefault 4403;
};
# The forwarder in ./swarm-authelia.nix.
swarm-authelia = {
clientId = hyperhiveCfg.swarm.authelia.otel.clientId;
port = lib.mkDefault 4402;
};
};
# Not under any `enable`: see `publishedScrapeTargets` for why every host

16
nix/host-modules/swarm-otel.nix
View file

@ -216,9 +216,9 @@ let
# The `serviceProducers` containers (`swarm-bao`, `swarm-nats`,
# `swarm-builder`, `swarm-claude-login`, `swarm-review-classifier`,
# `swarm-ci-runner`, `swarm-grafana`, `swarm-victorialogs`, `hive-forge`,
# `hive-matrix`, `swarm-victoriametrics`) are the exceptions on purpose: their
# journals stay inside the container for its own collector instead, so this
# receiver never sees their units.
# `hive-matrix`, `swarm-victoriametrics`, `swarm-authelia`) are the exceptions
# on purpose: their journals stay inside the container for its own collector
# instead, so this receiver never sees their units.
hostJournalDir = "/var/log/journal";
# Each store's OTLP route, by domain. One binding because the same string is
@ -274,7 +274,7 @@ let
# regression rather than a choice anyone made.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the publisher on authelia's host leaves this client's secret —
# Where the publisher in authelia's container leaves this client's secret —
# composed from the same swarm-wide `clientId` the registration in
# ./glue-swarm-otel-oidc-client.nix uses, so a rename cannot leave one of
# them behind. The same prefix ./swarm-grafana.nix reads its own client
@ -409,11 +409,11 @@ in
# ./swarm-grafana.nix has for its own client.
# THE delivery unit — one route, in every deployment. The secret authelia
# minted arrives out of the swarm secret store, which the publisher on
# authelia's host wrote it into, whether authelia is a network away or in
# the container next door.
# minted arrives out of the swarm secret store, which the publisher in
# authelia's container wrote it into, whether authelia is a network away or
# in the container next door.
#
# 🩸 No unit here copies the plaintext directly out of authelia's host
# 🩸 No unit here copies the plaintext directly out of authelia's
# tree: that is reachable only where authelia shares this host's network
# namespace, and breaks the moment authelia is on another host. The store
# exists so a host holds ONE out-of-band secret — its client certificate —

236
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,55 +1,15 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its own and its agents'
# credentials, a service or the controller its own. It mints nothing: every
# value is an OIDC client secret authelia minted, and this unit copies it.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
# reader fetches it from the store in every deployment — ./swarm-grafana.nix
# states the ruling that made that the only route. Publishing "only when the
# reader is elsewhere" would be a second shape of this unit, gated on a fact
# about another host, to save a round trip on the one host that can afford it.
#
# ⚠️ IT RUNS WHERE AUTHELIA DOES, and that is the whole reason it exists as a
# separate thing. `deploy.authelia.hostClientSecretDir`'s own description says
# why: the plaintext's other reader lives in a **different container**, and
# containers that share this host's network namespace still have separate
# filesystem roots, so "the host is the only place both trees are addressable".
# A delivery step therefore runs on the minting host, not the store's or the
# reading hive's.
#
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit that writes a handful of secrets
# has no business holding it. ./swarm-bao.nix grants this one `create`/`update`
# under the hive prefix and nothing else — no `read`.
#
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself.
# What a host decides about the swarm's secret publisher: whether it runs, and
# the store identity it presents, plus the list of secrets it copies. The unit
# itself is ../container-modules/swarm-secret-publisher.nix, inside authelia's
# container beside the mint whose secrets it copies; ./swarm-authelia.nix
# imports it there and hands the identity in as system credentials.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoDeploy = deployCfg.bao;
cfg = deployCfg.swarm-secret-publisher;
# A reader is defined by holding a certificate the store accepts, never by
# standing next to it — the rule ./glue-queue-agent-credential.nix states in
# full.
# `deploy.bao.enable` here would be the co-location assumption itself.
haveClientIdentity = cfg.baoClientCertFile != null && cfg.baoClientKeyFile != null;
# Both halves have to be here: the mint (authelia, for the plaintext) and an
# identity (for the store). Neither implies the other. Authelia here is a
# requirement, not a co-location toggle: every file the unit copies is under
# its `hostClientSecretDir`.
active = cfg.enable && deployCfg.authelia.enable && haveClientIdentity;
hiveNames = lib.attrNames hyperhiveCfg.swarm.hives;
# The client id agent containers present, per hive — composed exactly as
@ -110,12 +70,46 @@ let
hyperhiveCfg.swarm.victoriametrics.otel.clientId
# The CI run poller, which reads it itself on its host.
hyperhiveCfg.swarm.ci-metrics.clientId
# Authelia's container's journal forwarder, which reads it back under the
# publisher's own identity.
hyperhiveCfg.swarm.authelia.otel.clientId
];
# The swarm controller's OIDC client, which it reads back from the store
# under its own certificate.
controllerClientId = hyperhiveCfg.swarm.controller.queueClientId;
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
secrets =
# `client_id` beside the value: an agent's id is derived per hive and has
# to be reconstructable from the store alone.
map (hive: {
clientId = agentClientId hive;
path = "secret/swarm/hives/${hive}/queue/agent";
withClientId = true;
}) hiveNames
# Each hive's own client, which that hive's hive-c0re reads under its own
# certificate. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
++ map (hive: {
clientId = hiveClientId hive;
path = "secret/swarm/hives/${hive}/oidc/client";
}) hiveNames
# No `client_id`: a service's client id is the swarm-wide option both
# ends already read.
++ map (id: {
clientId = id;
path = "secret/swarm/services/${id}/oidc/client";
}) serviceClientIds
# Under `controller/`, which no hive reads: this client may read every
# hive's status. The nix half of
# `swarm_secret_client::queue::controller_client_path`.
++ [
{
clientId = controllerClientId;
path = controllerSecretPath;
}
];
in
{
options.services.hyperhive.deploy.swarm-secret-publisher = {
@ -127,7 +121,8 @@ in
secret store, so a hive that does not run authelia can read its
own and its agents' credentials and a swarm service or the
controller can read its own — from wherever it runs, this host
included.
included. The unit, `swarm-secret-publish`, runs inside authelia's
container, so it exists only where authelia runs.
Defaults to whether this host mints the OIDC secrets (set by
`glue-secret-publisher-authelia.nix`), which is the only half of the
@ -160,6 +155,11 @@ in
must be {option}`services.hyperhive.deploy.bao.secretPublisherCommonName`
— cert auth matches on the CN, and the role accepts nothing else.
A path on the host. ./swarm-authelia.nix hands it into authelia's
container as an nspawn system credential, and nspawn refuses to
start the container while a credential's source is missing: set,
the file has to exist before authelia's container starts.
No default: a module that guessed would be holding the CA opinion
./swarm-bao.nix deliberately does not hold.
./glue-secret-publisher-bao-identity.nix points it at the leaf
@ -172,141 +172,21 @@ in
default = null;
description = ''
Private key for {option}`services.hyperhive.deploy.swarm-secret-publisher.baoClientCertFile`.
Both or neither — the unit does not exist unless each is set.
Both or neither — the unit does not exist unless each is set. A host
path, handed in like the certificate.
'';
};
};
config = lib.mkIf active {
# Re-publish when authelia rotates a secret. The mint writes the file, so
# the file is the event — there is no signal from authelia to subscribe to.
systemd.paths.swarm-secret-publish = {
description = "watch for minted OIDC client secrets to publish";
wantedBy = [ "multi-user.target" ];
pathConfig = {
PathChanged = deployCfg.authelia.hostClientSecretDir;
Unit = "swarm-secret-publish.service";
};
};
systemd.services.swarm-secret-publish = {
description = "publish the swarm's minted secrets to the swarm secret store";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
path = [
baoDeploy.package
pkgs.coreutils
];
# ⚠️ No `RemainAfterExit`. A oneshot that stays `active (exited)` turns
# every later start from the path unit above into a no-op.
serviceConfig = {
Type = "oneshot";
# Bounded here rather than left to systemd's default, so the number a
# boot waits on is in the file that waits. A sealed store answers on
# the port and never answers the write.
TimeoutStartSec = 60;
Restart = "on-failure";
RestartSec = 30;
};
environment = {
BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
BAO_CLIENT_CERT = cfg.baoClientCertFile;
BAO_CLIENT_KEY = cfg.baoClientKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Unhandled on purpose: `Restart=on-failure` above is what a store that
# cannot authenticate this host should get. Degrading here would report
# "published 0" as an ordinary quiet day.
BAO_TOKEN="$(bao login -method=cert -token-only)"
export BAO_TOKEN
published=0
skipped=0
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"}
if [ -s "$src" ]; then
# `value=@$src` hands bao the PATH: bao opens the file itself, so
# the plaintext is never an argument of this process. Writing it as
# `value="$(cat "$src")"` would publish it to /proc for anyone on
# the host to read.
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/queue/agent"} \
value=@"$src" \
client_id=${lib.escapeShellArg (agentClientId hive)}
published=$((published + 1))
else
# Not an error: authelia mints on its FIRST BOOT, so an absent file
# is "not yet", and the path unit above re-runs this when it lands.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') hiveNames}
# Each hive's own client, which that hive's hive-c0re reads under its
# own certificate. The nix half of
# `swarm_secret_client::queue::hive_client_path`.
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${hiveClientId hive}.secret"}
if [ -s "$src" ]; then
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/oidc/client"} \
value=@"$src"
published=$((published + 1))
else
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') hiveNames}
${lib.concatMapStringsSep "\n" (id: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${id}.secret"}
if [ -s "$src" ]; then
# `value=@$src` for the same reason as the hive loop above: bao
# opens the file itself, so the plaintext is never an argument of
# this process.
#
# No `client_id` field beside it, unlike a hive's credential: that
# one is derived per hive and has to be reconstructable from the
# store alone, whereas a service's client id is the swarm-wide
# option both ends already read.
bao kv put ${lib.escapeShellArg "secret/swarm/services/${id}/oidc/client"} \
value=@"$src"
published=$((published + 1))
else
# Not an error, and the ordinary state of a swarm that runs this
# service nowhere: nothing registered the client, so authelia minted
# nothing to publish.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
'') serviceClientIds}
# The controller's own client, under `controller/`, which no hive
# reads: this client may read every hive's status. The nix half of
# `swarm_secret_client::queue::controller_client_path`.
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${controllerClientId}.secret"}
if [ -s "$src" ]; then
bao kv put ${lib.escapeShellArg controllerSecretPath} \
value=@"$src"
published=$((published + 1))
else
# Not an error: authelia mints on its first boot, and the path unit
# above re-runs this when it lands.
echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1))
fi
echo "published $published client secret(s), skipped $skipped"
secrets = lib.mkOption {
type = lib.types.listOf lib.types.attrs;
readOnly = true;
internal = true;
default = secrets;
description = ''
Every secret the publisher copies: the minted client's id, the store
path its reader fetches, and whether `client_id` is stored beside the
value. Derived from `swarm.*` alone, so it is the same list on every
host; ./swarm-authelia.nix hands it to the unit in its container.
'';
};
};

4
nix/host-modules/swarm.nix
View file

@ -488,8 +488,8 @@ in
A URL alone does not publish. This hive also needs its own store
identity ({option}`services.hyperhive.deploy.bao.clientCertFile`),
under which hive-c0re reads its `hive-<hiveName>` client secret
from the swarm's secret store; `swarm-secret-publish`, on the host
that runs authelia, puts it there. Without an identity, or with
from the swarm's secret store; `swarm-secret-publish`, in authelia's
container, puts it there. Without an identity, or with
this set to null, this hive computes its own readiness as always
and offers it to nobody, and the swarm controller reports it
`never_reported`, which is the honest reading.

16
nix/module-eval/bao-grants.nix
View file

@ -414,10 +414,13 @@ let
# The whole point of a second principal. The two prefixes and one leaf it
# publishes to and not `swarm/`, so it cannot touch an agent's
# credentials or the appservice token beside the controller's client;
# and no `read`, so a unit whose job is copying a file cannot recover
# what is already there. Pinned as the full capability list per path,
# because an added capability is exactly what a presence check misses.
name = "the publisher's grant is write-only and reaches the hive and service prefixes and the controller's client alone";
# and `read` on one leaf alone, the client secret of the journal forwarder
# that shares its identity in authelia's container, so a unit whose job
# is copying a file cannot recover what is already there. Pinned as the
# full capability list per path, because an added capability is exactly
# what a presence check misses. The one `read` is found at that leaf and
# counted once, so a read on any other path fails the count.
name = "the publisher's grant writes the hive and service prefixes and the controller's client, and reads its forwarder's client alone";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-secret-publisher-policy.script;
@ -429,7 +432,10 @@ let
&& !(lib.hasInfix "appservice-token" s)
&& !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/*" s)
&& !(lib.hasInfix "sys/policies/acl" s);
&& !(lib.hasInfix "sys/policies/acl" s)
&& lib.hasInfix "path \"secret/data/swarm/services/swarm-authelia-collector/oidc/client\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}" s
&& lib.length (lib.splitString "\"read\"" s) == 2
&& !(lib.hasInfix "\"list\"" s);
}
{
# The publisher's `services/*` write covers the callout xkey's seed, and

2
nix/module-eval/builder.nix
View file

@ -138,7 +138,7 @@ let
c != null
&& c.kind == "machine"
&& lib.hasInfix "secret/swarm/services/swarm-builder/oidc/client" (
autheliaHere.systemd.services.swarm-secret-publish.script
autheliaHere.containers.swarm-authelia.config.systemd.services.swarm-secret-publish.script
);
}
];

2
nix/module-eval/hive-otel.nix
View file

@ -303,7 +303,7 @@ let
ok =
let
fetch = builtins.unsafeDiscardStringContext (hiveOtelFetch hiveOtelAllLocal).script;
publish = builtins.unsafeDiscardStringContext hiveOtelAllLocal.systemd.services.swarm-secret-publish.script;
publish = builtins.unsafeDiscardStringContext hiveOtelAllLocal.containers.swarm-authelia.config.systemd.services.swarm-secret-publish.script;
path = "secret/swarm/hives/h1/oidc/client";
in
lib.hasInfix "bao kv get -field=value ${path}" fetch && lib.hasInfix "bao kv put ${path} " publish;

69
nix/module-eval/nats-authelia.nix
View file

@ -25,18 +25,13 @@ let
# No store identity, which the first case below relies on.
natsQueueOnly = shared.natsQueueOnly;
# The IdP and the queue on one host. A reader of authelia's host paths has to
# follow the `deploy` namespace; seven of those reads go through an alias a
# path-shaped grep cannot see.
# The IdP and the queue on one host.
autheliaAndNats = hive {
deploy.authelia.enable = true;
deploy.nats.enable = true;
deploy.authelia.usersFile = "/var/lib/test-authelia/users.yml";
deploy.authelia.package = pkgs.emptyDirectory;
deploy.authelia.bridgePackage = pkgs.emptyDirectory;
# Renders the publisher, which reads the host client-secret dir.
deploy.swarm-secret-publisher.baoClientCertFile = "/etc/pki/publisher.pem";
deploy.swarm-secret-publisher.baoClientKeyFile = "/etc/pki/publisher-key.pem";
};
# The queue with a store identity for its responder, placed by hand: the
@ -671,16 +666,6 @@ let
&& lib.hasInfix "[ -s /var/lib/swarm-bao-pki/nats-auth.pem ] ||" mint
&& lib.hasInfix "/var/lib/swarm-bao-pki nats-auth " mint;
}
{
# Not a rename test. `hostClientSecretDir` is `readOnly`, so the fixture
# cannot define it; what can break is a reader left pointing at
# `swarm.authelia` instead of `deploy.authelia`. Several modules read this
# through an `autheliaCfg` alias, where a path-shaped grep does not see it.
name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace";
ok =
autheliaAndNats.systemd.paths.swarm-secret-publish.pathConfig.PathChanged
== "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients";
}
{
# authelia exits at startup on `users: {}`, so an empty database, and a
# file holding exactly that document, must be replaced by the disabled
@ -1017,6 +1002,58 @@ let
&& lib.elem "-/run/swarm-victoriametrics-egress" h.systemd.services.nginx.serviceConfig.ReadWritePaths
&& lib.hasInfix "rm -f ${sock}" h.systemd.services.nginx.preStart;
}
{
# Authelia's forwarder reads its secret from the store under the
# publisher's identity, handed in for both units. The IdP host with no
# identity is the control: the forwarder is off there, and on the
# services host it is on, so the lookup is the right one.
name = "the authelia container forwards its journal to its own route under the publisher's identity";
ok =
let
h = servicesHost;
container = h.containers.swarm-authelia;
c = container.config;
s = c.services.opentelemetry-collector.settings;
otel = h.services.hyperhive.swarm.otel;
clientId = h.services.hyperhive.swarm.authelia.otel.clientId;
fetch = c.systemd.services.swarm-journal-forwarder-oidc;
dir = forwarderFetchDir;
registered = lib.findFirst (
x: x.id == clientId
) null h.services.hyperhive.swarm.authelia.oidc.clients;
reader = lib.head s.service.telemetry.metrics.readers;
taken = [
8888
8889
8890
h.services.hyperhive.swarm.nats.otel.telemetryPort
]
++ map (o: o.port) hostNetnsForwarders;
in
c.services.opentelemetry-collector.enable
&& s.exporters.otlphttp.endpoint == "https://${otel.domain}/swarm-authelia"
&& s.extensions."oauth2client/swarm".client_id == clientId
&& s.extensions."oauth2client/swarm".endpoint_params.audience == clientId
&& otel.serviceProducers.swarm-authelia.clientId == clientId
&&
(lib.findFirst (a: a.key == "service.name") null s.processors.resource.attributes).value
== "authelia"
&& fetch.environment.BAO_CLIENT_CERT == "${dir}/publisher-bao-cert"
&& fetch.environment.BAO_CLIENT_KEY == "${dir}/publisher-bao-key"
&& lib.elem "publisher-bao-cert" fetch.serviceConfig.LoadCredential
&& lib.elem "publisher-bao-key" fetch.serviceConfig.LoadCredential
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/${clientId}/oidc/client" fetch.script
&& registered.kind == "machine"
&& registered.audience == [ clientId ]
&& !(lib.elem "--link-journal=host" container.extraFlags)
&& lib.any (lib.hasPrefix "--load-credential=publisher-bao-cert:") container.extraFlags
&& reader.pull.exporter.prometheus.host == "127.0.0.1"
&& !(lib.elem reader.pull.exporter.prometheus.port taken)
&& !autheliaColocated.containers.swarm-authelia.config.services.opentelemetry-collector.enable
&& !(
autheliaColocated.containers.swarm-authelia.config.systemd.services ? swarm-journal-forwarder-oidc
);
}
]
++ forwarderCases
++ hostNetnsForwarderCases;

151
nix/module-eval/secret-publisher.nix
View file

@ -72,6 +72,9 @@ let
# The store and a service that reads from it, versus the store alone. The
# pair is what makes the reader's absence arm mean anything.
inherit (shared) baoWithMatrix;
# The publisher runs inside authelia's container.
inAuthelia = m: m.containers.${m.services.hyperhive.swarm.authelia.machine}.config;
cases = [
{
# Each hive's hive-c0re reads this exact path
@ -81,7 +84,7 @@ let
name = "the publisher writes every hive's own client under that hive's prefix";
ok =
let
s = secretPublisherHere.systemd.services.swarm-secret-publish.script;
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
in
lib.all
(h: lib.hasInfix "secret/swarm/hives/${h}/oidc/client" s && lib.hasInfix "/hive-${h}.secret" s)
@ -99,7 +102,7 @@ let
name = "the publisher writes the controller's client under controller/, never services/";
ok =
let
s = secretPublisherHere.systemd.services.swarm-secret-publish.script;
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
in
lib.hasInfix "secret/swarm/controller/swarm-controller/oidc/client" s
&& lib.hasInfix "/swarm-controller.secret" s
@ -107,12 +110,12 @@ let
}
{
# Both ends of a wire nothing at eval time carries end to end: the
# publisher on authelia's host writes the path the reader on Grafana's host
# reads, and the two files agree only because both compose it from the same
# swarm-wide client id.
# publisher in authelia's container writes the path the reader on
# Grafana's host reads, and the two files agree only because both compose
# it from the same swarm-wide client id.
name = "the publisher writes the swarm service path grafana reads";
ok = lib.hasInfix "secret/swarm/services/swarm-grafana/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
(inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
);
}
{
@ -184,7 +187,7 @@ let
name = "the publisher on an authelia host carries the queue's, homeserver's and forge's secrets";
ok =
let
s = secretPublisherRemote.systemd.services.swarm-secret-publish.script;
s = (inAuthelia secretPublisherRemote).systemd.services.swarm-secret-publish.script;
in
lib.hasInfix "secret/swarm/services/swarm-nats/oidc/client" s
&& lib.hasInfix "/swarm-nats.secret" s
@ -196,13 +199,13 @@ let
}
{
# Both ends of a wire nothing at eval time carries end to end: the
# publisher on authelia's host writes the path the reader on the
# publisher in authelia's container writes the path the reader on the
# collector's host reads, and the two files agree only because both
# compose it from the same swarm-wide client id. `secretPublisherHere`
# already grew this client when `serviceClientIds` did.
name = "the publisher writes the swarm service path the collector reads";
ok = lib.hasInfix "secret/swarm/services/swarm-collector/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
(inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
);
}
{
@ -213,20 +216,20 @@ let
# store's host waits forever on a path nothing writes.
name = "the publisher carries the store forwarder's secret too";
ok = lib.hasInfix "secret/swarm/services/swarm-bao-collector/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
(inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
);
}
{
# The forwarders in the CI runner's, Grafana's, the log store's, the
# forge's and the homeserver's containers read their secrets from these
# paths, and the metrics store's host reads its container's.
name = "the publisher carries the CI runner, Grafana, log store, forge, homeserver and metrics store forwarders' secrets";
# forge's, the homeserver's and authelia's containers read their secrets
# from these paths, and the metrics store's host reads its container's.
name = "the publisher carries the CI runner, Grafana, log store, forge, homeserver, metrics store and authelia forwarders' secrets";
ok =
lib.all
(
id:
lib.hasInfix "secret/swarm/services/${id}/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
(inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
)
)
[
@ -236,6 +239,7 @@ let
"swarm-forge-collector"
"swarm-matrix-collector"
"swarm-victoriametrics-collector"
"swarm-authelia-collector"
];
}
{
@ -251,7 +255,7 @@ let
c != null
&& c.kind == "machine"
&& lib.hasInfix "secret/swarm/services/swarm-review-classifier/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
(inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
);
}
{
@ -270,7 +274,7 @@ let
&& ui.kind == "interactive"
&& ui.redirectUris == [ "https://bao-ui.t.local/ui/vault/auth/oidc/oidc/callback" ]
&& lib.hasInfix "secret/swarm/services/swarm-bao-ui/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
(inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
)
&& !(lib.any (
c: c.id == "swarm-bao-ui"
@ -321,7 +325,7 @@ let
name = "the publisher hands bao the secret's path, never the secret";
ok =
let
s = secretPublisherHere.systemd.services.swarm-secret-publish.script;
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
code = lib.concatStringsSep "\n" (
lib.filter (l: builtins.match "[[:space:]]*#.*" l == null) (lib.splitString "\n" s)
);
@ -336,7 +340,7 @@ let
name = "the publisher writes every hive in the roster to that hive's own queue path";
ok =
let
s = secretPublisherHere.systemd.services.swarm-secret-publish.script;
s = (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script;
in
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s
&& lib.hasInfix "secret/swarm/hives/h2/queue/agent" s;
@ -368,7 +372,7 @@ let
&& lib.hasInfix "-token-only" c
&& !(lib.hasInfix "bao kv" (lib.head (lib.splitString "bao login" c)));
in
holdsTokenFirst secretPublisherHere.systemd.services.swarm-secret-publish.script
holdsTokenFirst (inAuthelia secretPublisherHere).systemd.services.swarm-secret-publish.script
&& holdsTokenFirst baoWithMatrix.systemd.services.swarm-bao-queue-agent.script
# Controls, so a clean verdict above means something. In order: a bare
# read is refused, a read placed before the login is refused, and a
@ -384,43 +388,132 @@ let
# keep: a client is defined by holding a certificate the store accepts,
# never by standing next to the store. Gating this on `deploy.bao.enable`
# would have left the unit rendering only on the one deployment that has
# no use for it.
name = "a publisher holding an identity runs on a host with no store";
# no use for it. It runs beside the mint, in authelia's container, and
# never on the host.
name = "a publisher holding an identity runs in authelia's container on a host with no store";
ok =
let
m = secretPublisherRemote;
c = inAuthelia m;
in
!m.services.hyperhive.deploy.bao.enable
&& (m.systemd.services ? swarm-secret-publish)
&& (m.systemd.paths ? swarm-secret-publish);
&& (c.systemd.services ? swarm-secret-publish)
&& (c.systemd.paths ? swarm-secret-publish)
&& !(m.systemd.services ? swarm-secret-publish)
&& !(m.systemd.paths ? swarm-secret-publish)
# Control: the host's own unit set is the one read above.
&& (m.systemd.services ? "container@swarm-authelia");
}
{
# What makes the arm above able to fail. Minting the secrets is not being
# able to publish them: with no certificate the unit would fail a TLS
# handshake on every rotation, so it must not exist at all.
# handshake on every rotation, so it must not exist at all, and nspawn
# gets no credential to load.
name = "an IdP host with no store identity renders no publisher";
ok =
let
m = secretPublisherNoIdentity;
c = inAuthelia m;
in
m.services.hyperhive.deploy.authelia.enable
&& !(c.systemd.services ? swarm-secret-publish)
&& !(c.systemd.paths ? swarm-secret-publish)
&& !(m.systemd.services ? swarm-secret-publish)
&& !(m.systemd.paths ? swarm-secret-publish);
&& !(lib.any (lib.hasPrefix "--load-credential=") m.containers.swarm-authelia.extraFlags)
# Controls: with an identity, the same lookups find both.
&& ((inAuthelia secretPublisherHere).systemd.services ? swarm-secret-publish)
&& lib.any (lib.hasPrefix "--load-credential=") secretPublisherHere.containers.swarm-authelia.extraFlags;
}
{
# The path unit's start is a no-op while its target is `active (exited)`,
# so a secret minted after the first run would never be published.
# `Type` is read too, so a missing `serviceConfig` cannot pass the arm.
name = "the publisher's path unit starts a oneshot that does not remain active";
# The watched directory is the mint's own, inside the container.
name = "the publisher's path unit watches the mint's directory and starts a oneshot that does not remain active";
ok =
let
m = secretPublisherHere;
sc = m.systemd.services.swarm-secret-publish.serviceConfig;
c = inAuthelia secretPublisherHere;
sc = c.systemd.services.swarm-secret-publish.serviceConfig;
in
m.systemd.paths.swarm-secret-publish.pathConfig.Unit == "swarm-secret-publish.service"
c.systemd.paths.swarm-secret-publish.pathConfig.Unit == "swarm-secret-publish.service"
&&
c.systemd.paths.swarm-secret-publish.pathConfig.PathChanged
== "/var/lib/authelia-swarm/oidc-clients"
&& lib.hasInfix "/var/lib/authelia-swarm/oidc-clients/swarm-grafana.secret" c.systemd.services.swarm-secret-publish.script
&& !(lib.hasInfix "/var/lib/nixos-containers" c.systemd.services.swarm-secret-publish.script)
&& sc.Type == "oneshot"
&& !(sc.RemainAfterExit or false);
}
{
# The leaf crosses into the container as nspawn system credentials, and
# the unit loads them and names them in its environment. On the store's
# host the container waits for the unit that mints the leaf, since nspawn
# refuses a missing credential source; with the leaf placed by hand,
# those paths cross.
name = "the publisher's store identity reaches authelia's container as system credentials";
ok =
let
dir = "/run/credentials/swarm-secret-publish.service";
unit = m: (inAuthelia m).systemd.services.swarm-secret-publish;
flags = m: m.containers.swarm-authelia.extraFlags;
here = unit secretPublisherHere;
in
lib.elem "--load-credential=publisher-bao-cert:/var/lib/swarm-bao-pki/secret-publisher.pem" (
flags secretPublisherHere
)
&& lib.elem "--load-credential=publisher-bao-key:/var/lib/swarm-bao-pki/secret-publisher-key.pem" (
flags secretPublisherHere
)
&& lib.elem "--load-credential=publisher-bao-server-ca:/var/lib/swarm-bao-pki/ca.pem" (
flags secretPublisherHere
)
&&
lib.sort lib.lessThan here.serviceConfig.LoadCredential == [
"publisher-bao-cert"
"publisher-bao-key"
"publisher-bao-server-ca"
]
&& here.environment.BAO_CLIENT_CERT == "${dir}/publisher-bao-cert"
&& here.environment.BAO_CLIENT_KEY == "${dir}/publisher-bao-key"
&& here.environment.BAO_CACERT == "${dir}/publisher-bao-server-ca"
&&
lib.elem "swarm-bao-pki.service"
secretPublisherHere.systemd.services."container@swarm-authelia".after
&& lib.elem "--load-credential=publisher-bao-cert:/etc/pki/publisher.pem" (
flags secretPublisherRemote
)
&& lib.elem "--load-credential=publisher-bao-key:/etc/pki/publisher-key.pem" (
flags secretPublisherRemote
)
&& (unit secretPublisherRemote).environment.BAO_CLIENT_CERT == "${dir}/publisher-bao-cert";
}
{
# Authelia is the swarm's IdP and has to come up while the store is down.
# Nothing it starts after or needs may be the publisher, and the
# publisher may not order itself in front of authelia.
name = "authelia does not wait for the secret publisher";
ok =
let
c = inAuthelia secretPublisherHere;
authelia = c.systemd.services.authelia-swarm;
publisher = c.systemd.services.swarm-secret-publish;
unit = "swarm-secret-publish.service";
in
!(lib.any (l: lib.elem unit (authelia.${l} or [ ])) [
"after"
"requires"
"wants"
"bindsTo"
"requisite"
])
&& !(lib.any (l: lib.elem "authelia-swarm.service" (publisher.${l} or [ ])) [
"before"
"requiredBy"
])
# Controls: both lookups find the real units.
&& authelia.serviceConfig ? ExecStart
&& lib.elem "authelia-swarm-secrets.service" publisher.after;
}
];
in
runGroup "secret-publisher" cases

2
swarm-builder/src/main.rs
View file

@ -191,7 +191,7 @@ async fn client_secret(client_id: &str) -> Result<String> {
match stored {
Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()),
_ => bail!(
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \
"nothing at {path}: swarm-secret-publish, in authelia's container, has not \
published this service's client secret"
),
}

2
swarm-claude-login/src/main.rs
View file

@ -195,7 +195,7 @@ async fn client_secret(client_id: &str) -> Result<String> {
match stored {
Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()),
_ => bail!(
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \
"nothing at {path}: swarm-secret-publish, in authelia's container, has not \
published this service's client secret"
),
}

6
swarm-controller/src/queue_identity.rs
View file

@ -6,8 +6,8 @@
//! `swarm_secret_client::queue::controller_client_path`, read at start under
//! this daemon's own certificate, held only in memory and read again when
//! authelia refuses it (`swarm_queue_client::stored_secret`). It is never on
//! this host's disk. `swarm-secret-publish`, on the host that runs
//! authelia, puts it there.
//! this host's disk. `swarm-secret-publish`, in authelia's
//! container, puts it there.
//!
//! [`init`] runs once in `main`, before anything calls [`get`].
@ -116,7 +116,7 @@ async fn fetch_secret() -> Result<String> {
match stored {
Some(c) if !c.value.trim().is_empty() => Ok(c.value.trim().to_owned()),
_ => bail!(
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \
"nothing at {path}: swarm-secret-publish, in authelia's container, has not \
published this daemon's client secret"
),
}

8
swarm-nats-auth/src/oidc_secret.rs
View file

@ -2,8 +2,8 @@
//! with. Read from the secret store as the responder starts and held in
//! memory, never in a file, `argv` or the environment.
//!
//! The secret can change under a running responder: authelia's host
//! re-publishes it when it is re-minted. So when the `IdP` refuses an
//! The secret can change under a running responder: the publisher in authelia's
//! container re-publishes it when it is re-minted. So when the `IdP` refuses an
//! introspection as a credential (a 400 or 401), the secret is read from the
//! store again ([`StoredSecret::after_refusal`]) and, when it changed, the
//! introspection is tried once more. Both fit in the one request's
@ -43,7 +43,9 @@ pub fn path(client_id: &str) -> Result<String> {
/// When the store cannot be read, or holds nothing or only whitespace there.
pub async fn read(store: &impl KeyStore, path: &str) -> Result<String> {
let stored = store.read(path).await?.with_context(|| {
format!("nothing is stored at {path}; swarm-secret-publish on authelia's host writes it")
format!(
"nothing is stored at {path}; swarm-secret-publish in authelia's container writes it"
)
})?;
let trimmed = stored.trim();
if trimmed.is_empty() {

4
swarm-queue-client/src/stored_secret.rs
View file

@ -1,8 +1,8 @@
//! A client secret held in memory and read again from where it is kept when
//! the `IdP` refuses it.
//!
//! The secret can change under a running process: authelia's host
//! re-publishes it when it is re-minted. A refusal asks for a re-read
//! The secret can change under a running process: the publisher in authelia's
//! container re-publishes it when it is re-minted. A refusal asks for a re-read
//! ([`StoredSecret::after_refusal`]); when the read lands a different secret,
//! the caller tries once more with it. One re-read runs at a time, and a
//! refusal while one is running waits for it rather than starting another.

2
swarm-review-classifier/src/main.rs
View file

@ -185,7 +185,7 @@ async fn client_secret(client_id: &str) -> Result<String> {
match stored {
Some(s) if !s.value.trim().is_empty() => Ok(s.value.trim().to_owned()),
_ => bail!(
"nothing at {path}: swarm-secret-publish, on the host that runs authelia, has not \
"nothing at {path}: swarm-secret-publish, in authelia's container, has not \
published this service's client secret"
),
}