vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

forge: fetch the OIDC client secret from bao instead of a host-side copy

Browse source 
The forge's login source read its OIDC client secret from a file that
`hive-forge-oidc-secret` copied out of authelia's container tree. That
unit only existed where authelia ran on the forge's host (`ssoLocal`),
so a forge anywhere else needed the file placed by hand.

The publisher on authelia's host now carries the forge's client secret
to `secret/swarm/services/<forge client id>/oidc/client`, like every
other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy
of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf
(the forge host's one store identity, whose policy gains read on that
one path) and writes the file `forgejo-sso-source` reads, re-running the
login source when the value changes. It retries on the shared
store-retry shape while the store is down or the secret is not yet
published.

`hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the
in-container path wherever the forge runs, and the `ssoLocal` gate on
the login source's resolver ordering goes with it: the login source
fetches the issuer's discovery document on every forge host.

Refs #5015
This commit is contained in:
atlas 2026-10-09 23:30:17 +02:00 • committed by mara
commit f6ca5f1433
10 changed files with 332 additions and 159 deletions
Download patch file Download diff file Expand all files Collapse all files

38
docs/swarm/credentials.md
View file

@ -55,25 +55,25 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts |
| `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart | | `swarm/controller/swarm-controller/forge/token` | the forge, inside the `hive-forge` container, once; `swarm-bao-forge-publisher` on the forge's host copies it in under its own certificate, with a write-only grant | `swarm-controller`, under its own certificate, at start | ❌ the forge mints it once and keeps its copy in the container; the publisher writes that copy each time the forge container starts and each time the mint writes the file | ❌ the controller reads it at start, and in the background with backoff until it first finds it; a replaced token reaches it with the next restart |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read | | `swarm/hives/<hive>/forge/token` | `swarm-controller`, with the forge's admin API, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `forge::reader_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ hive-c0re reads the store on every forge read |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held until hive-c0re restarts | | `swarm/hives/<hive>/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | hive-c0re, under the hive's own certificate, when it first connects to the queue | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read until the first connect succeeds, retried with a backoff of up to five minutes while it's missing; then held until hive-c0re restarts |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s, the forge's under `swarm-forge-publisher`'s; the bao UI's by the store's granter | must be stated | must be stated |
| `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed | | `swarm/services/ci-runner/registration-token` | `swarm-controller`, from the forge's admin API, at start and in a five-minute pass; its grant on the path is write-only | `swarm-bao-ci-runner` on the runner's host, under the runner's own certificate, into a file only `swarm-ci-runner-register` in the runner's container loads; no other principal's policy reads it | ❌ the forge hands out one token until an admin resets it; the controller writes a reset token within five minutes | ❌ `swarm-bao-ci-runner` reads it when it starts, at boot or deploy, and restarts the runner when the value changed |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

63
docs/swarm/secrets.md
View file

@ -243,15 +243,16 @@ such as the root token.
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` | | hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container | | the forge's copy of its OIDC secret | `swarm-bao-forge-oidc.service` reads it out of the swarm secret store under the forge publisher's leaf, wherever authelia runs | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) | | the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
The forge's delivery unit waits for authelia's first boot to mint the secret — a The forge and the homeserver read their copies out of the store, the way
bounded wait, 120s — and then **fails loudly** rather than skipping. A silent skip Grafana does below: the forge under the forge publisher's leaf, the
produces a service whose login button always fails, which is a symptom many homeserver under matrix-ctl's. Each **fails loudly** and retries until the
layers from its cause. The homeserver's reads the store under matrix-ctl's leaf, publisher on authelia's host has written the secret, rather than skipping. A
the way Grafana's does below. silent skip produces a service whose login button always fails, which is a
symptom many layers from its cause.
The store's **first reader** is the agent containers' queue credential: The store's **first reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and `glue-queue-agent-credential.nix` lands it as two files, the client secret and
@ -304,13 +305,16 @@ either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config, (`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. The swarm controller, the queue and the homeserver register the running Grafana. The swarm controller, the queue, the homeserver and the forge
same way (`glue-controller-oidc-client.nix`, `glue-nats-oidc-client.nix`, register the same way (`glue-controller-oidc-client.nix`,
`glue-matrix-oidc-client.nix`), and the queue's responder and the homeserver read `glue-nats-oidc-client.nix`, `glue-matrix-oidc-client.nix`,
their secrets from the store too: the responder itself under its own leaf, `glue-forge-oidc-client.nix`), and the queue's responder, the homeserver and the
`swarm-bao-matrix-oidc.service` under matrix-ctl's. Each fails, naming the forge read their secrets from the store too: the responder itself under its own
option pair, when that leaf is unset — neither consumer starts without its leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's,
secret. `swarm-bao-forge-oidc.service` under the forge publisher's. Each fails, naming
the option pair, when that leaf is unset — neither the responder nor the
homeserver starts without its secret, and the forge registers no login source
without its.
The **third reader** is the swarm collector's own OIDC client secret, the The **third reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads same shape one level down: `swarm-bao-otel-oidc.service` reads
@ -386,17 +390,18 @@ next controller restart.
### Per-principal identities ### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. These six units reach one path identity and sharing one merges the identities. Each of these six leaves has a
each and each holds a leaf, a role and a policy of its own: role and a policy of its own, and reaches one path, except the forge host's,
which reaches two:
| unit | option pair under `deploy.bao.` | reads, or writes | | unit | option pair under `deploy.bao.` | reads, or writes |
| --------------------------- | -------------------------------------------------------------- | -------------------------------------------------------- | | --------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` | | `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` | | `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` | | `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
| `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` | | `swarm-bao-ci-runner` | `ciRunnerClientCertFile` / `ciRunnerClientKeyFile` | `swarm/services/ci-runner/registration-token` |
| `swarm-bao-forge-publisher` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token` | | `swarm-bao-forge-publisher`, `swarm-bao-forge-oidc` | `forgePublisherClientCertFile` / `forgePublisherClientKeyFile` | writes `swarm/controller/swarm-controller/forge/token`; reads `swarm/services/<forge client id>/oidc/client` |
The first exists **per hive**, because the path it reads carries a hive name The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is and every hive runs its own reader. Its subject is
@ -406,17 +411,19 @@ role. The other five reach a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`, `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName`,
`deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName` and `deploy.bao.forwarderOidcCommonName`, `deploy.bao.ciRunnerCommonName` and
`deploy.bao.forgePublisherCommonName`. The last is the only writer: the other `deploy.bao.forgePublisherCommonName`. The last is the only writer, and the one
five read. leaf two units share: the forge's host holds one store identity for the token it
publishes and the OIDC client secret it reads.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all six and defaults On a host that mints its own PKI, `glue-bao-tls.nix` signs all six and defaults
all twelve options, and there is nothing to do. Elsewhere you issue each leaf from all twelve options, and there is nothing to do. Elsewhere you issue each leaf from
that CA out of band and name it here — one file per principal rather than one file that CA out of band and name it here — one file per principal rather than one file
shared by six, which is the whole of what this buys. `swarm-ci-runner.nix` shared by six, which is the whole of what this buys. `swarm-ci-runner.nix`
refuses the build wherever the runner runs without its pair, as refuses the build wherever the runner runs without its pair, as
`swarm-grafana.nix` does for Grafana. `swarm-bao-forge-publisher` refuses `swarm-grafana.nix` does for Grafana. The forge's pair refuses nothing: without
nothing: without its pair it renders, fails at start naming both options, and it `swarm-bao-forge-publisher` and `swarm-bao-forge-oidc` render, fail at start
the controller runs without forge access until an operator places the leaf. naming both options, and the controller runs without forge access and the forge
without a login source until an operator places the leaf.
On a host without the store, where the forwarder never runs, forgetting one of On a host without the store, where the forwarder never runs, forgetting one of
the other six isn't a quiet degrade. A hand-configured remote-store hive that the other six isn't a quiet degrade. A hand-configured remote-store hive that

5
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -71,6 +71,11 @@ let
here = deployCfg.matrix.enable; here = deployCfg.matrix.enable;
policies = [ "swarm-bao-matrix-ctl-policy" ]; policies = [ "swarm-bao-matrix-ctl-policy" ];
}; };
# ./hive-forge/default.nix, under the forge publisher's leaf.
swarm-bao-forge-oidc = {
here = deployCfg.forgejo.enable;
policies = [ "swarm-bao-forge-publisher-policy" ];
};
}; };
orderAfterPolicies = orderAfterPolicies =

218
nix/host-modules/hive-forge/default.nix
View file

@ -28,18 +28,21 @@ let
autheliaUrl = autheliaCfg.url; autheliaUrl = autheliaCfg.url;
autheliaDiscoveryUrl = "${toString autheliaUrl}/.well-known/openid-configuration"; autheliaDiscoveryUrl = "${toString autheliaUrl}/.well-known/openid-configuration";
# The all-local case: this host runs BOTH the forge and the swarm's
# authelia, so the secret can be moved without an operator. The other
# two cases (swarm side via swarmctl, remote hive) leave
# `clientSecretFile` to be set explicitly — see docs/swarm/.
ssoLocal = deployCfg.authelia.enable;
# Where the plaintext lands inside the forge container. Under # Where the plaintext lands inside the forge container. Under
# /var/lib rather than /run: the forge may start before the delivery # /var/lib rather than /run: the forge may start before the delivery
# unit on a later boot, and a secret that evaporates on reboot turns a # unit on a later boot, and a secret that evaporates on reboot turns a
# working login into an intermittent one. # working login into an intermittent one.
forgeSecretPath = "/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret"; forgeSecretPath = "/var/lib/forgejo-oidc/${cfg.sso.clientId}.secret";
# The same file seen from the host, where `swarm-bao-forge-oidc` writes it.
# Follows `clientSecretFile`, so an operator-chosen path is where the
# fetched secret lands.
hostSecretPath = "/var/lib/nixos-containers/hive-forge${toString deployCfg.forgejo.sso.clientSecretFile}";
# Where swarm-secret-publish on authelia's host leaves this client's secret.
# The forge publisher's store role grants read on this one path.
oidcStorePath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
# The swarm-controller's forge account. Same name as # The swarm-controller's forge account. Same name as
# `swarm-controller.nix`'s existing `queueClientId` — the account is # `swarm-controller.nix`'s existing `queueClientId` — the account is
# provisioned to *match* that identity, not invented independently — # provisioned to *match* that identity, not invented independently —
@ -66,6 +69,7 @@ let
baoCfg = config.services.hyperhive.swarm.bao; baoCfg = config.services.hyperhive.swarm.bao;
storeRetry = import ../lib/store-retry.nix { }; storeRetry = import ../lib/store-retry.nix { };
atomicWriteSecret = import ../lib/atomic-write-secret.nix { }; atomicWriteSecret = import ../lib/atomic-write-secret.nix { };
refreshConsumer = import ../lib/refresh-consumer.nix { };
havePublisherIdentity = havePublisherIdentity =
deployCfg.bao.forgePublisherClientCertFile != null deployCfg.bao.forgePublisherClientCertFile != null
&& deployCfg.bao.forgePublisherClientKeyFile != null; && deployCfg.bao.forgePublisherClientKeyFile != null;
@ -251,10 +255,13 @@ in
plaintext), and a literal written here would be rendered into plaintext), and a literal written here would be rendered into
the world-readable nix store. the world-readable nix store.
Required when `enable` is set — deliberately no fallback. A Set to `/var/lib/forgejo-oidc/<client id>.secret` wherever the
forge that boots with SSO half-configured presents as a login forge runs. `swarm-bao-forge-oidc` writes the secret here from the
button that always fails, which is harder to diagnose than an swarm secret store, whichever path this names.
eval error.
Null is refused at eval: a forge that boots with SSO
half-configured presents as a login button that always fails,
which is harder to diagnose than an eval error.
''; '';
}; };
}; };
@ -363,9 +370,9 @@ in
(inside the forge container) holding the OIDC client secret's (inside the forge container) holding the OIDC client secret's
plaintext. plaintext.
On a hive that also runs the swarm's authelia this is wired up It is set by default wherever the forge runs, and
for you. Set it explicitly when authelia lives on another swarm-bao-forge-oidc writes the secret there from the swarm
host: see docs/swarm/ for which secret goes where. secret store. Remove the definition that set it to null.
''; '';
} }
{ {
@ -483,8 +490,8 @@ in
inherit (networkCfg) bridgeIp; inherit (networkCfg) bridgeIp;
dnsConsumers = [ dnsConsumers = [
"forgejo.service" "forgejo.service"
] "forgejo-sso-source.service"
++ lib.optional ssoLocal "forgejo-sso-source.service"; ];
}) })
# Forgejo is Go, and `SSL_CERT_FILE` *replaces* the default store # Forgejo is Go, and `SSL_CERT_FILE` *replaces* the default store
@ -495,10 +502,7 @@ in
# BOTH units that make an outbound HTTPS call are consumers, not # BOTH units that make an outbound HTTPS call are consumers, not
# just the obvious one: `forgejo-sso-source` fetches the issuer's # just the obvious one: `forgejo-sso-source` fetches the issuer's
# `.well-known/openid-configuration` under the store's root, and it # `.well-known/openid-configuration` under the store's root, and it
# once shipped without the trust its sibling had. `optional` # once shipped without the trust its sibling had.
# rather than a flat list because that unit only exists when SSO
# is on — naming an absent unit would order nothing and quietly
# define a serviceless one.
(caTrust.trustBundle { (caTrust.trustBundle {
inherit pkgs; inherit pkgs;
name = "hive-forge"; name = "hive-forge";
@ -946,12 +950,10 @@ in
token_path=${lib.escapeShellArg swarmControllerTokenPath} token_path=${lib.escapeShellArg swarmControllerTokenPath}
# Token minting is idempotent by the token FILE's existence, # Token minting is idempotent by the token FILE's existence,
# not a separate stamp — same reasoning `hive-forge-oidc-secret` # not a separate stamp: the file is both the record of "already
# (below) uses for its own delivery: the file is both the # done" and the thing that has to survive, so a state wipe that
# record of "already done" and the thing that has to survive, # deletes it correctly triggers a re-mint instead of being
# so a state wipe that deletes it correctly triggers a # silently masked by a stamp that outlived what it claims exists.
# re-mint instead of being silently masked by a stamp that
# outlived what it claims exists.
if [ ! -s "$token_path" ]; then if [ ! -s "$token_path" ]; then
# `write:admin` added for `swarm-controller::forge::Client:: # `write:admin` added for `swarm-controller::forge::Client::
# ensure_agent_user` (the `CreateForgeUser` jobq node) — # ensure_agent_user` (the `CreateForgeUser` jobq node) —
@ -994,76 +996,128 @@ in
}; };
# The client this login source authenticates as is registered by # The client this login source authenticates as is registered by
# ../glue-forge-oidc-client.nix, wherever authelia runs. When authelia # ../glue-forge-oidc-client.nix, wherever authelia runs.
# is here too, this host minted the secret, so it can say where the services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkDefault forgeSecretPath;
# forge will find it.
services.hyperhive.deploy.forgejo.sso.clientSecretFile = lib.mkIf ssoLocal (
lib.mkDefault forgeSecretPath
);
# The delivery. It runs on the HOST because that is the only place # The forge's OIDC client secret, out of the swarm secret store under the
# both container trees are addressable: they share this host's # forge publisher's leaf — the one route, wherever authelia runs. Same
# network namespace, which makes them feel co-located, but their # shape as ../hive-matrix.nix's `swarm-bao-matrix-oidc`: the login and the
# filesystem roots are separate — the forge cannot open a path inside # read both fail loudly, and `storeRetry` retries until the publisher on
# authelia's tree however local the port looks. # authelia's host has written the secret.
# #
# ⚠️ Deliberately a copy and not a `bindMounts` entry. # On the host, where that leaf is: the container holds no store identity.
# nixos-container refuses to start when a bind source is missing, and # A host with no publisher leaf fails here, naming the options, rather than
# this secret does not exist until authelia's first boot has minted # rendering no unit.
# it — so binding it would make the forge wait on a file that waits systemd.services.swarm-bao-forge-oidc = {
# on a container that starts after it. On a fresh hive that is a description = "fetch the forge's OIDC client secret from the swarm secret store";
# permanent stall presenting as "the forge is broken", several layers # The store's units exist only where the store runs; `Requires=` on an
# from its cause. # absent unit fails the job, so off-host the timeout below bounds the
systemd.services.hive-forge-oidc-secret = lib.mkIf ssoLocal { # attempt instead.
description = "deliver the forge's OIDC client secret from authelia"; after = lib.optionals deployCfg.bao.enable [
after = [ "container@${autheliaCfg.machine}.service" ]; "swarm-bao-pki.service"
requires = [ "container@${autheliaCfg.machine}.service" ]; "container@${baoCfg.machine}.service"
];
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
before = [ "container@hive-forge.service" ]; before = [ "container@hive-forge.service" ];
wantedBy = [ "container@hive-forge.service" ]; wantedBy = [
serviceConfig = { "multi-user.target"
"container@hive-forge.service"
];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ../lib/store-retry.nix. The container waits for one attempt; a secret a
# later attempt lands re-runs `forgejo-sso-source` (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
SyslogIdentifier = "hive-forge-oidc-secret"; SyslogIdentifier = "swarm-bao-forge-oidc";
# Longer than the 120s bounded wait below. `DefaultTimeoutStartSec` TimeoutStartSec = 30;
# is 90s, so without this systemd kills the unit at 90 — before it
# can emit the message naming the file it was waiting for.
TimeoutStartSec = "180s";
}; };
path = [ pkgs.coreutils ]; environment = {
script = '' BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
set -euo pipefail }
// lib.optionalAttrs havePublisherIdentity {
BAO_CLIENT_CERT = deployCfg.bao.forgePublisherClientCertFile;
BAO_CLIENT_KEY = deployCfg.bao.forgePublisherClientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
script =
if !havePublisherIdentity then
''
echo "the forge's host has no forge-publisher store identity, so it cannot read the forge's OIDC client secret: set services.hyperhive.deploy.bao.forgePublisherClientCertFile and forgePublisherClientKeyFile (they default to the leaf the store mints only where services.hyperhive.deploy.bao.enable is set on this host)" >&2
exit 1
''
else
''
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.sso.clientId}.secret"} ${atomicWriteSecret}
dst=${lib.escapeShellArg "/var/lib/nixos-containers/hive-forge${forgeSecretPath}"} ${refreshConsumer}
# authelia's container is up, but its first-boot generator may err="$(mktemp)"
# still be minting. Bounded wait, then fail: a silent skip here trap 'rm -f "$err"' EXIT
# produces a forge whose SSO button 401s, which is the failure
# this whole design is trying not to ship.
for _ in $(seq 1 60); do
[ -s "$src" ] && break
sleep 2
done
if [ ! -s "$src" ]; then
echo "authelia has not minted $src after 120s" >&2
exit 1
fi
# The owning uid is DISCOVERED, not assumed: read it off the # `-token-only` answers on stdout and skips the token helper, a
# forge container's own state dir. Whatever uid maps to forgejo # `sh` this unit's `path` does not carry.
# inside that container is by definition the one that owns the if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
# directory it was created with, and hardcoding a number here echo "could not log in to swarm-bao with the forge publisher's certificate; leaving the forge's OIDC client secret as it is." >&2
# would be a second place for it to be wrong. if [ -s "$err" ]; then
uid=$(stat -c %u /var/lib/nixos-containers/hive-forge/var/lib/forgejo) cat "$err" >&2
install -D -m 0400 -o "$uid" -g "$uid" "$src" "$dst" else
''; echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg oidcStorePath} 2>"$err")"; then
echo "swarm-bao did not return ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${oidcStorePath}; the forge has no OIDC client secret yet." >&2
exit 1
fi
# `forgejo-sso-source` reads the file as forgejo. The owning uid is
# read off the container's forgejo state dir rather than assumed,
# which exists only once the container has started.
state=/var/lib/nixos-containers/hive-forge/var/lib/forgejo
if [ ! -d "$state" ]; then
echo "the forge container has not created $state yet, so the forge's OIDC client secret has no owner to be written for." >&2
exit 1
fi
uid="$(stat -c %u "$state")"
mkdir -p ${lib.escapeShellArg (builtins.dirOf hostSecretPath)}
if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
atomic_write_secret 0400 "$uid:$uid" ${lib.escapeShellArg hostSecretPath} "$secret"
fi
# The login source is a row `forgejo-sso-source` writes from the
# file each time it runs.
refresh_consumer hive-forge forgejo-sso-source.service ${lib.escapeShellArg hostSecretPath}
'';
}; };
# The reverse direction of the unit above: publish the swarm-controller's # The reverse direction of the unit above: publish the swarm-controller's
# freshly-minted forge token OUT of the container into the swarm secret # freshly-minted forge token OUT of the container into the swarm secret
# store, where the controller reads it wherever it runs. On the host for the # store, where the controller reads it wherever it runs. On the host and
# reason `hive-forge-oidc-secret` gives, and under the forge's own store # under the forge's own store identity for the reason the unit above gives,
# identity, so nothing here asks where the store or the controller is. # so nothing here asks where the store or the controller is.
# #
# It runs at boot and on a switch that adds or changes it # It runs at boot and on a switch that adds or changes it
# (`multi-user.target`), each time the container starts, and each time the # (`multi-user.target`), each time the container starts, and each time the

3
nix/host-modules/otel.nix
View file

@ -692,9 +692,6 @@ in
# exist. This oneshot converts the fast failure into a slow one, which # exist. This oneshot converts the fast failure into a slow one, which
# is what that restart policy is actually good at. # is what that restart policy is actually good at.
# #
# Copied from `hive-forge-oidc-secret.service`, which solves exactly
# this for the forge — see `docs/swarm/secrets.md`.
#
# Not rendered beside the store fetch below, which restarts the # Not rendered beside the store fetch below, which restarts the
# collector itself when the file lands. # collector itself when the file lands.
systemd.services.${secretWaitUnit} = lib.mkIf waits { systemd.services.${secretWaitUnit} = lib.mkIf waits {

24
nix/host-modules/swarm-bao.nix
View file

@ -931,9 +931,11 @@ let
} }
]; ];
# The forge's host, publishing the controller's forge token. The one writer # The forge host's one store identity. `swarm-bao-forge-publisher` writes
# of the path, and write-only like the secret publisher: it copies a file in # the controller's forge token under it, the one writer of that path and
# and never reads one back. # write-only: it copies a file in and never reads one back.
# `swarm-bao-forge-oidc` reads the forge's own OIDC client secret, that one
# leaf and no other service's.
forgePublisherWriters = [ forgePublisherWriters = [
{ {
name = "swarm-forge-publisher"; name = "swarm-forge-publisher";
@ -942,7 +944,8 @@ let
path "${credentialMountPath}/data/${controllerForgeTokenLeaf}" { path "${credentialMountPath}/data/${controllerForgeTokenLeaf}" {
capabilities = ["create", "update"] capabilities = ["create", "update"]
} }
''; ''
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.forge.sso.clientId}/oidc/client";
} }
]; ];
@ -1754,9 +1757,11 @@ in
default = "swarm-forge-publisher"; default = "swarm-forge-publisher";
description = '' description = ''
Subject the store's `swarm-forge-publisher` cert-auth role accepts: the Subject the store's `swarm-forge-publisher` cert-auth role accepts: the
identity `swarm-bao-forge-publisher`, the unit on the forge's host that forge host's one store identity, presented by `swarm-bao-forge-publisher`,
publishes the swarm-controller's forge token, presents. Its grant is which publishes the swarm-controller's forge token, and by
one path, `swarm/controller/swarm-controller/forge/token`, and write `swarm-bao-forge-oidc`, which fetches the forge's OIDC client secret.
Its grant is two paths: `swarm/controller/swarm-controller/forge/token`,
write only, and `swarm/services/<forge client id>/oidc/client`, read
only. only.
⚠️ Reserved as a hive name by ./swarm.nix, like its siblings. ⚠️ Reserved as a hive name by ./swarm.nix, like its siblings.
@ -2001,8 +2006,9 @@ in
default = null; default = null;
example = "/var/lib/swarm-bao-pki/forge-publisher.pem"; example = "/var/lib/swarm-bao-pki/forge-publisher.pem";
description = '' description = ''
Certificate the unit that publishes the swarm-controller's forge token Certificate the forge's host presents to the store, both to publish the
presents to the store. Its subject must be swarm-controller's forge token and to fetch the forge's OIDC client
secret. Its subject must be
{option}`services.hyperhive.deploy.bao.forgePublisherCommonName`. {option}`services.hyperhive.deploy.bao.forgePublisherCommonName`.
./glue-bao-tls.nix points it at the leaf it mints on the store's host. ./glue-bao-tls.nix points it at the leaf it mints on the store's host.

7
nix/host-modules/swarm-secret-publisher.nix
View file

@ -88,6 +88,8 @@ let
hyperhiveCfg.swarm.nats.clientId hyperhiveCfg.swarm.nats.clientId
# The homeserver, read by `swarm-bao-matrix-oidc` on its host. # The homeserver, read by `swarm-bao-matrix-oidc` on its host.
hyperhiveCfg.swarm.matrix.sso.clientId hyperhiveCfg.swarm.matrix.sso.clientId
# The forge's login source, read by `swarm-bao-forge-oidc` on its host.
hyperhiveCfg.swarm.forge.sso.clientId
]; ];
# The swarm controller's OIDC client, which it reads back from the store # The swarm controller's OIDC client, which it reads back from the store
@ -121,8 +123,9 @@ in
Turning it off leaves every hive but this one without its agents' Turning it off leaves every hive but this one without its agents'
credential, every hive unable to publish its status, the swarm credential, every hive unable to publish its status, the swarm
controller unable to start, the swarm's Grafana without any login at controller unable to start, the swarm's Grafana without any login at
all, its collector pushing unauthenticated, the all, its collector pushing unauthenticated, the queue's responder
queue's responder unable to start, and the homeserver unable to start. unable to start, the homeserver unable to start, and the forge's login
source without its secret.
Each secret has exactly one route and this is the producer's Each secret has exactly one route and this is the producer's
end of it, so the honest reason to set it false is a deployment end of it, so the honest reason to set it false is a deployment
delivering those secrets by some other mechanism it owns. delivering those secrets by some other mechanism it owns.

23
nix/module-eval/bao-grants.nix
View file

@ -1373,16 +1373,19 @@ let
lib.hasInfix "/ci-runner.pem ]" s && lib.hasInfix "swarm-ci-runner \"\" clientAuth" s; lib.hasInfix "/ci-runner.pem ]" s && lib.hasInfix "swarm-ci-runner \"\" clientAuth" s;
} }
{ {
# The forge's host is the one writer of the controller's forge token. # The forge's host is the one writer of the controller's forge token,
# Pinned as the whole stanza, so an added `read` fails: it copies a file # and reads the forge's own OIDC client secret under the same leaf.
# in and never reads one back. # Pinned as two whole stanzas, so an added `read` on the token or a
name = "the forge publisher's grant is writing the controller's forge token and nothing else"; # write or wildcard on the secret fails.
name = "the forge publisher's grant is writing the controller's forge token and reading the forge's OIDC secret, and nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-forge-publisher-policy.script; s = baoGrantHere.systemd.services.swarm-bao-forge-publisher-policy.script;
in in
lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/forge/token\" {\n capabilities = [\"create\", \"update\"]\n}" s lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/forge/token\" {\n capabilities = [\"create\", \"update\"]\n}" s
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1 && lib.hasInfix "path \"secret/data/swarm/services/forgejo/oidc/client\" {\n capabilities = [\"read\"]\n}" s
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
&& lib.hasInfix "auth/cert/certs/swarm-forge-publisher" s && lib.hasInfix "auth/cert/certs/swarm-forge-publisher" s
&& lib.hasInfix "allowed_common_names=swarm-forge-publisher" s && lib.hasInfix "allowed_common_names=swarm-forge-publisher" s
&& lib.hasInfix "token_policies=swarm-forge-publisher" s; && lib.hasInfix "token_policies=swarm-forge-publisher" s;
@ -1413,6 +1416,16 @@ let
"swarm-bao-forge-publisher-policy" "swarm-bao-forge-publisher-policy"
]; ];
} }
{
# The forge's OIDC client secret by name: the forge publisher reads it,
# and the secret publisher writes it under its `swarm/services/*` grant,
# which names no service.
name = "no granting unit but the forge publisher's names the forge's OIDC secret";
ok =
lib.filter (
u: lib.hasInfix "swarm/services/forgejo/oidc/client" baoGrantHere.systemd.services.${u}.script
) grantingUnitNames == [ "swarm-bao-forge-publisher-policy" ];
}
{ {
name = "the PKI unit signs the forge publisher's leaf under its own subject"; name = "the PKI unit signs the forge publisher's leaf under its own subject";
ok = ok =

87
nix/module-eval/forge-placement.nix
View file

@ -40,10 +40,21 @@ let
deploy.forgejo.enable = false; deploy.forgejo.enable = false;
}; };
# The forge without authelia: the other half of the split. # The forge without authelia: the other half of the split, with its secret's
# path and its store identity chosen by hand.
forgeNoAuthelia = hive { forgeNoAuthelia = hive {
deploy.forgejo.enable = true; deploy.forgejo.enable = true;
deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret"; deploy.forgejo.sso.clientSecretFile = "/var/lib/forgejo-oidc/by-hand.secret";
deploy.bao.forgePublisherClientCertFile = "/etc/pki/forge-publisher.pem";
deploy.bao.forgePublisherClientKeyFile = "/etc/pki/forge-publisher-key.pem";
};
# The forge with neither authelia nor the store, its store identity placed by
# hand: the deployment the store fetch exists for.
forgeAlone = hive {
deploy.forgejo.enable = true;
deploy.bao.forgePublisherClientCertFile = "/etc/pki/forge-publisher.pem";
deploy.bao.forgePublisherClientKeyFile = "/etc/pki/forge-publisher-key.pem";
}; };
# The runner alone: no hive, no forge, no store, its store identity placed # The runner alone: no hive, no forge, no store, its store identity placed
@ -105,6 +116,7 @@ let
refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions; refusedOver = m: needle: lib.any (a: !a.assertion && lib.hasInfix needle a.message) m.assertions;
publisher = m: m.systemd.services.swarm-bao-forge-publisher; publisher = m: m.systemd.services.swarm-bao-forge-publisher;
oidcFetch = m: m.systemd.services.swarm-bao-forge-oidc;
forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings; forgeSettings = forgeHere.containers.hive-forge.config.services.forgejo.settings;
@ -329,6 +341,79 @@ let
&& lib.hasInfix "exit 1" s && lib.hasInfix "exit 1" s
&& !(lib.hasInfix "bao kv put" s); && !(lib.hasInfix "bao kv put" s);
} }
{
# The forge's OIDC secret has one route, the store, under the forge
# publisher's leaf, landing where `clientSecretFile` points inside the
# container. No unit copies it out of authelia's tree, even on a host
# that runs authelia, and a forge host without authelia needs no path
# set by hand.
name = "the forge reads its OIDC secret from the store under the forge publisher's leaf";
ok =
let
u = oidcFetch forgeAlone;
d = forgeAlone.services.hyperhive.deploy;
in
!d.authelia.enable
&& !d.bao.enable
&& d.forgejo.sso.clientSecretFile == "/var/lib/forgejo-oidc/forgejo.secret"
&& !(refusedOver forgeAlone "deploy.forgejo.sso.clientSecretFile")
&& u.environment.BAO_CLIENT_CERT == "/etc/pki/forge-publisher.pem"
&& u.environment.BAO_CLIENT_KEY == "/etc/pki/forge-publisher-key.pem"
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/forgejo/oidc/client" u.script
&& lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/forgejo.secret" u.script
&& lib.hasInfix "refresh_consumer hive-forge forgejo-sso-source.service" u.script
# An unpublished or empty secret fails the unit, so `storeRetry` retries it.
&& !(lib.hasInfix "exit 0" u.script)
&& u.serviceConfig.Restart == "on-failure"
&& lib.elem "container@hive-forge.service" u.before
&& lib.elem "multi-user.target" u.wantedBy
&& (oidcFetch allLocal).environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/forge-publisher.pem"
&& !(forgeAlone.systemd.services ? hive-forge-oidc-secret)
&& !(allLocal.systemd.services ? hive-forge-oidc-secret)
&& !(bare.systemd.services ? swarm-bao-forge-oidc);
}
{
# An operator-chosen path is where the fetched secret lands.
name = "the forge's OIDC secret lands where clientSecretFile points";
ok =
let
s = (oidcFetch forgeNoAuthelia).script;
in
lib.hasInfix "/var/lib/nixos-containers/hive-forge/var/lib/forgejo-oidc/by-hand.secret" s;
}
{
# Beside the store, the fetch waits for the unit writing the forge
# publisher's role, the leaf it logs in with.
name = "the forge's OIDC secret fetch is ordered after the forge publisher's policy unit";
ok =
let
u = oidcFetch allLocal;
in
lib.elem "swarm-bao-forge-publisher-policy.service" u.after
&& lib.elem "swarm-bao-forge-publisher-policy.service" u.wants;
}
{
# The absence arm: no publisher leaf, no secret, and the unit says which
# options to set.
name = "a forge host with no publisher leaf renders an OIDC secret fetch that fails naming the options";
ok =
let
u = oidcFetch forgeHere;
in
!(u.environment ? BAO_CLIENT_CERT)
&& lib.hasInfix "forgePublisherClientCertFile" u.script
&& lib.hasInfix "exit 1" u.script
&& !(lib.hasInfix "bao login" u.script);
}
{
# The login source fetches the issuer's discovery document by name
# wherever the forge runs, so it waits for the resolver on a forge host
# without authelia too.
name = "the login source waits for the container's resolver on a forge host without authelia";
ok = lib.elem "forgejo-sso-source.service" (
forgeAlone.containers.hive-forge.config.systemd.services.swarm-bridge-dns.before
);
}
{ {
# The host copy is removed where the forge runs: nothing reads it. # The host copy is removed where the forge runs: nothing reads it.
name = "the forge's host removes its old copy of the controller's forge token"; name = "the forge's host removes its old copy of the controller's forge token";

13
nix/module-eval/secret-publisher.nix
View file

@ -189,10 +189,10 @@ let
]); ]);
} }
{ {
# The other half of the queue's and the homeserver's one route: the # The other half of the queue's, the homeserver's and the forge's one
# publisher on an IdP host that runs neither carries both secrets to the # route: the publisher on an IdP host that runs none of them carries each
# paths their readers fetch. # secret to the path its reader fetches.
name = "the publisher on an authelia host carries the queue's and homeserver's secrets"; name = "the publisher on an authelia host carries the queue's, homeserver's and forge's secrets";
ok = ok =
let let
s = secretPublisherRemote.systemd.services.swarm-secret-publish.script; s = secretPublisherRemote.systemd.services.swarm-secret-publish.script;
@ -200,7 +200,10 @@ let
lib.hasInfix "secret/swarm/services/swarm-nats/oidc/client" s lib.hasInfix "secret/swarm/services/swarm-nats/oidc/client" s
&& lib.hasInfix "/swarm-nats.secret" s && lib.hasInfix "/swarm-nats.secret" s
&& lib.hasInfix "secret/swarm/services/tuwunel/oidc/client" s && lib.hasInfix "secret/swarm/services/tuwunel/oidc/client" s
&& lib.hasInfix "/tuwunel.secret" s; && lib.hasInfix "/tuwunel.secret" s
&& !secretPublisherRemote.services.hyperhive.deploy.forgejo.enable
&& lib.hasInfix "secret/swarm/services/forgejo/oidc/client" s
&& lib.hasInfix "/forgejo.secret" s;
} }
{ {
# Both ends of a wire nothing at eval time carries end to end: the # Both ends of a wire nothing at eval time carries end to end: the