vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm: OIDC clients registered wherever authelia runs; nats + matrix secrets from bao

Browse source 
The three client registrations move into glue-{controller,nats,matrix}-oidc-client.nix,
gated only on deploy.authelia.enable and reading their ids from swarm.* options,
so the host running authelia registers every one of them whichever host runs
the service. The matrix callback is a new read-only swarm.matrix.sso.redirectUri,
read by both the homeserver's config and its client entry.

swarm-secret-publish now also publishes the queue's and the homeserver's minted
secrets to swarm/services/<id>/oidc/client. The queue host reads its responder's
secret there with swarm-bao-nats-auth-oidc under the responder's own leaf
(swarm-nats-auth role gains that one read path), and the homeserver's host with
swarm-bao-matrix-oidc under matrix-ctl's leaf (swarm-matrix-ctl role gains its
one read path). The host-file copies are gone: swarm-nats-auth-secrets now
delivers only the callout seeds and the store identity, and
hive-matrix-oidc-secret is removed. A fetch unit whose leaf is unset still
renders and fails naming the option pair.

Closes #4943
This commit is contained in:
atlas 2026-10-05 02:56:18 +02:00 • committed by mara
commit 962da3b784
20 changed files with 702 additions and 293 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/integrations/matrix.md
View file

@ -359,11 +359,13 @@ surprising behaviour:
SSO is unconditional, so the three below are requirements of running a SSO is unconditional, so the three below are requirements of running a
homeserver at all rather than of a setting: homeserver at all rather than of a setting:
- **Set `deploy.matrix.sso.clientSecretFile`** — fails at eval, not at boot: - **Keep `deploy.matrix.sso.clientSecretFile` set** — it defaults to
tuwunel reads its identity providers from the config file, so a `/var/lib/tuwunel-oidc/<client id>.secret` wherever the homeserver runs,
half-configured one can stop the homeserver from starting outright and `swarm-bao-matrix-oidc` writes the secret there from the swarm secret
rather than merely hiding a login button. On a host that also runs store under matrix-ctl's leaf. A null fails at eval, not at boot: tuwunel
the swarm's authelia it's wired up for you. reads its identity providers from the config file, so a half-configured
one can stop the homeserver from starting outright rather than merely
hiding a login button.
- **Set `swarm.authelia.url`** — without a provider URL there - **Set `swarm.authelia.url`** — without a provider URL there
is nothing to discover against. is nothing to discover against.
- **Set `swarm.matrix.gatewayHost != null`** — the SSO callback URL is - **Set `swarm.matrix.gatewayHost != null`** — the SSO callback URL is

6
docs/swarm/credentials.md
View file

@ -56,11 +56,11 @@ of the cell says how.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
| store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull | | store path | minter | reader — pulls at runtime, holds in memory | automatic re-mint | automatic re-pull |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again | | `swarm/agents/<agent>/matrix/main` | `swarm-controller`, with the swarm's appservice token, at agent creation and in a five-minute pass | the agent container itself, under the certificate its hive passed in | ✅ the pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ `hive-matrix-daemon` exits when the homeserver rejects its token, and a five-minute timer restarts it, which reads the store again |
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller` | the agent container itself, under the certificate its hive passed in | must be stated | must be stated |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl`, inside the `hive-matrix` container, once | `swarm-controller`, under its own certificate | ❌ `swarm-matrix-ctl` mints it once; the container keeps its copy and republishes it when the store's differs | ✅ the controller reads it on every five-minute matrix pass |
| `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot, where the controller registers its client; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts | | `swarm/controller/swarm-controller/oidc/client` | authelia, at its first boot; `swarm-secret-publish` copies it in | `swarm-controller`, under its own certificate, once at start | ❌ authelia mints it once. A re-mint is republished by `swarm-secret-publish`'s path unit | ❌ read once at start; the controller holds the old value until it restarts |
| `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires | | `swarm/agents/<agent>/bao-mtls` | the store's agent PKI mount (`deploy.bao.agentPkiMountPath`), which generates the key, at `swarm-controller`'s request at agent creation | `hive-c0re`, under the hive's own certificate, when it writes the agent's container config | ✅ `swarm-controller`'s five-minute pass re-issues a live agent's leaf once it's past half its validity (45 of 90 days, read from the certificate itself) | ❌ `hive-c0re` reads it when it writes the container config, so the agent presents a new leaf from its next start; the old leaf stays valid until it expires |
| `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff | | `swarm/agents/<agent>/queue` | `swarm-controller`, at agent creation | `hive-agent` in the agent container, under the agent's own certificate, held in memory — the identity it presents to the swarm queue, naming that one agent rather than its hive | ✅ `swarm-controller`'s five-minute pass re-mints a live agent's secret once it's 45 days old by `minted_at` on the stored object; a secret with no `minted_at` gets one stamped, value unchanged. The pass skips agents declared `Destroyed` — declaring an agent destroyed deletes every version of the path instead, the undo of the mint rather than another one | ✅ `hive-agent` reads the path before its first connect and again on every reconnect attempt, so a reconnect after a re-mint presents the new secret. An open connection keeps the secret it connected with; after a revocation the agent keeps retrying under the queue client's backoff |
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
@ -68,7 +68,7 @@ of the cell says how.
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate — the queue responder's under `swarm-nats-auth`'s, the homeserver's under `swarm-matrix-ctl`'s; the bao UI's by the store's granter | must be stated | must be stated |
| _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated | | _(not in the store)_ a hive's mTLS leaf | the store's own PKI, or an operator placing it by hand | its own client, off disk — the exception above, because it's what makes every other row's pull possible | must be stated | must be stated |
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->

18
docs/swarm/secrets.md
View file

@ -54,6 +54,7 @@ neither is a renaming of the other.
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/var/lib/swarm-otel-oidc/<id>.secret` inside the `swarm-otel` container | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see below |
| Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below | | Grafana's copy of its OIDC secret | `swarm-bao-grafana-oidc.service` reads it out of the swarm secret store, **on every host that runs Grafana** | `/var/lib/grafana-oidc/<id>.secret` inside the `swarm-grafana` container | same unit, same path — one route, co-located or not. Nothing for an operator to place beyond this host's store leaf, see below |
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| the queue responder's copy of its OIDC secret | `swarm-bao-nats-auth-oidc.service` reads it out of the swarm secret store, on the host that runs the queue, under the responder's own leaf | `/var/lib/swarm-nats-auth/oidc-client.secret` inside the `swarm-nats` container, handed to the responder through `LoadCredential` | same unit, same path, wherever authelia runs. Nothing for an operator to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when the operator sets `deploy.nats.autoGenerateCallout` | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-callout-keys` first-boot unit, when the operator sets `deploy.nats.autoGenerateCallout` | `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed`, `0600` | operator mints both with `nk` and names them in `deploy.nats.calloutUserSeedFile` / `deploy.nats.calloutIssuerSeedFile` |
@ -159,13 +160,14 @@ when the system builds. The server names the offending file and refuses to run.
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` | | hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container | | the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `swarm-bao-matrix-oidc.service` reads it out of the swarm secret store under matrix-ctl's leaf, wherever authelia runs | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) | | the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
Both delivery units wait for authelia's first boot to mint the secret — a The forge's delivery unit waits for authelia's first boot to mint the secret — a
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip bounded wait, 120s — and then **fails loudly** rather than skipping. A silent skip
produces a service whose login button always fails, which is a symptom many produces a service whose login button always fails, which is a symptom many
layers from its cause. layers from its cause. The homeserver's reads the store under matrix-ctl's leaf,
the way Grafana's does below.
The store's **first reader** is the agent containers' queue credential: The store's **first reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and `glue-queue-agent-credential.nix` lands it as two files, the client secret and
@ -218,7 +220,13 @@ either one left a Grafana with no SSO and no password box — nothing failed, so
nothing said anything. Registration is separate and stays where authelia is nothing said anything. Registration is separate and stays where authelia is
(`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config, (`glue-grafana-oidc-client.nix`): a client is a row in authelia's own config,
so it's declared on the host running authelia, which isn't necessarily the host so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. running Grafana. The swarm controller, the queue and the homeserver register the
same way (`glue-controller-oidc-client.nix`, `glue-nats-oidc-client.nix`,
`glue-matrix-oidc-client.nix`), and the queue's responder and the homeserver read
their secrets from the store too: `swarm-bao-nats-auth-oidc.service` under the
responder's own leaf, `swarm-bao-matrix-oidc.service` under matrix-ctl's. Each
renders wherever its consumer runs and fails, naming the option pair, when that
leaf is unset — neither consumer starts without its secret.
The **third reader** is the swarm collector's own OIDC client secret, the The **third reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads same shape one level down: `swarm-bao-otel-oidc.service` reads

3
nix/host-modules/default.nix
View file

@ -30,12 +30,15 @@
./glue-bao-tls.nix ./glue-bao-tls.nix
./glue-bao-ui-oidc-client.nix ./glue-bao-ui-oidc-client.nix
./glue-controller-bao-identity.nix ./glue-controller-bao-identity.nix
./glue-controller-oidc-client.nix
./glue-forge-oidc-client.nix ./glue-forge-oidc-client.nix
./glue-grafana-oidc-client.nix ./glue-grafana-oidc-client.nix
./glue-hive-tls-bao-identity.nix ./glue-hive-tls-bao-identity.nix
./glue-matrix-ctl-bao-identity.nix ./glue-matrix-ctl-bao-identity.nix
./glue-matrix-oidc-client.nix
./glue-nats-auth-bao-identity.nix ./glue-nats-auth-bao-identity.nix
./glue-nats-bao-identity.nix ./glue-nats-bao-identity.nix
./glue-nats-oidc-client.nix
./glue-queue-agent-credential.nix ./glue-queue-agent-credential.nix
./glue-secret-publisher-bao-identity.nix ./glue-secret-publisher-bao-identity.nix
./glue-services-issuer-bao-identity.nix ./glue-services-issuer-bao-identity.nix

26
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -47,15 +47,33 @@ let
swarm-bao-nats-tls = deployCfg.nats.enable; swarm-bao-nats-tls = deployCfg.nats.enable;
}; };
# Readers that log in under another unit's principal: reader unit → the
# gate its own module defines it under, and that principal's policy unit.
readersOfOtherPolicy = {
# ./swarm-nats.nix, under the queue responder's leaf.
swarm-bao-nats-auth-oidc = {
here = deployCfg.nats.enable;
policy = "swarm-bao-nats-auth-policy";
};
# ./hive-matrix.nix, under matrix-ctl's leaf.
swarm-bao-matrix-oidc = {
here = deployCfg.matrix.enable;
policy = "swarm-bao-matrix-ctl-policy";
};
};
orderAfterPolicy = orderAfterPolicy =
reader: here: reader: here: policy:
lib.mkIf (baoDeploy.enable && here) { lib.mkIf (baoDeploy.enable && here) {
systemd.services.${reader} = { systemd.services.${reader} = {
after = [ "${reader}-policy.service" ]; after = [ "${policy}.service" ];
wants = [ "${reader}-policy.service" ]; wants = [ "${policy}.service" ];
}; };
}; };
in in
{ {
config = lib.mkMerge (lib.mapAttrsToList orderAfterPolicy readersHere); config = lib.mkMerge (
lib.mapAttrsToList (reader: here: orderAfterPolicy reader here "${reader}-policy") readersHere
++ lib.mapAttrsToList (reader: r: orderAfterPolicy reader r.here r.policy) readersOfOtherPolicy
);
} }

48
nix/host-modules/glue-controller-oidc-client.nix Normal file
View file

@ -0,0 +1,48 @@
# Glue: register the swarm controller as an OIDC client wherever authelia runs.
#
# ONE PAIRING PER FILE — swarm-controller ← authelia, and nothing else. Deleting
# this leaves a controller whose token requests authelia answers with a 401, and
# no minted secret for swarm-secret-publish to put in the store.
#
# ⚠️ Gated on authelia being HERE, and NOT on this host running the controller.
# A client is a row in THIS host's provider config, so it is declared where that
# config is rendered; ./swarm-controller.nix's `config` hangs off
# `deploy.swarm-controller.enable`, which answers a question about a different
# host. Same shape as ./glue-grafana-oidc-client.nix.
#
# Registered whether or not the swarm runs a controller: nothing in `swarm.*`
# records that, so the cost of a swarm without one is one unused client and one
# unused minted secret.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
queueClientId = hyperhiveCfg.swarm.controller.queueClientId;
in
{
config = lib.mkIf deployCfg.authelia.enable {
# One declaration, two readers: the id is the read-only option the
# controller authenticates under, so a mismatch cannot be introduced here.
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = queueClientId;
description = "HyperHive swarm controller";
# `client_credentials`: a daemon authenticating as itself, with nobody
# to redirect. Declared rather than inferred from an empty redirect
# list, because authelia permits only the grants a client names and an
# omitted `grant_types` means authorization-code alone.
kind = "machine";
# Its own id as its own permitted audience. The controller mints a
# token for `SWARM_CONTROLLER_OTEL_AUDIENCE` (./swarm-controller.nix),
# and ./swarm-otel.nix's `oidc/swarm` authenticator checks for this
# same id; either half without the other is a 401 several layers from
# this line.
audience = [ queueClientId ];
}
];
};
}

46
nix/host-modules/glue-matrix-oidc-client.nix Normal file
View file

@ -0,0 +1,46 @@
# Glue: register the swarm's homeserver as an OIDC client wherever authelia
# runs.
#
# ONE PAIRING PER FILE — matrix ← authelia, and nothing else. Deleting this
# leaves a homeserver whose SSO login authelia has never heard of, and no
# minted secret for swarm-secret-publish to put in the store.
#
# ⚠️ Gated on authelia being HERE, and NOT on this host running the
# homeserver. A client is a row in THIS host's provider config, so it is
# declared where that config is rendered; ./hive-matrix.nix's `config` hangs
# off `deploy.matrix.enable`, which answers a question about a different host.
# Same shape as ./glue-grafana-oidc-client.nix.
#
# Registered whether or not the swarm runs a homeserver: nothing in `swarm.*`
# records that. The one swarm-level fact that rules a login out is a null
# `gatewayHost`, which leaves no callback a browser can reach, so that alone
# skips the client.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
matrixCfg = hyperhiveCfg.swarm.matrix;
in
{
config = lib.mkIf (deployCfg.authelia.enable && matrixCfg.sso.redirectUri != null) {
# One declaration, two readers: the callback is the read-only option the
# homeserver's own config renders, so it is spelled once.
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = matrixCfg.sso.clientId;
description = "HyperHive matrix";
redirectUris = [ matrixCfg.sso.redirectUri ];
# tuwunel authenticates at the token endpoint by putting the secret in
# the POST body. Authelia enforces the *registered* method, and its
# default is `client_secret_basic` — so without this the browser flow
# completes, consent is granted, and the last hop fails with a 401
# that names neither the secret nor the redirect.
tokenEndpointAuthMethod = "client_secret_post";
}
];
};
}

45
nix/host-modules/glue-nats-oidc-client.nix Normal file
View file

@ -0,0 +1,45 @@
# Glue: register the swarm queue as an OIDC client wherever authelia runs.
#
# ONE PAIRING PER FILE — swarm-nats ← authelia, and nothing else. Deleting this
# leaves a queue responder whose introspection calls authelia refuses, so it
# denies every client, and no minted secret for swarm-secret-publish to put in
# the store.
#
# ⚠️ Gated on authelia being HERE, and NOT on this host running the queue. A
# client is a row in THIS host's provider config, so it is declared where that
# config is rendered; ./swarm-nats.nix's `config` hangs off
# `deploy.nats.enable`, which answers a question about a different host. Same
# shape as ./glue-grafana-oidc-client.nix.
#
# Registered whether or not the swarm runs a queue: nothing in `swarm.*` records
# that, so the cost of a swarm without one is one unused client and one unused
# minted secret.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
natsCfg = hyperhiveCfg.swarm.nats;
in
{
config = lib.mkIf deployCfg.authelia.enable {
services.hyperhive.swarm.authelia.oidc.clients = [
{
id = natsCfg.clientId;
description = "HyperHive swarm queue";
# `client_credentials`: a queue's clients are daemons with nobody to
# redirect, and no redirect URI exists.
#
# ⚠️ `kind` says that, and an empty `redirectUris` does NOT: authelia
# permits only the grants a client names, and an omitted `grant_types`
# means authorization-code alone. Introspection, which is all the
# responder calls, works either way, so the token endpoint's
# `unauthorized_client` is the only place a wrong kind shows.
kind = "machine";
}
];
};
}

28
nix/host-modules/hive-matrix-service.nix
View file

@ -178,14 +178,34 @@ in
type = lib.types.str; type = lib.types.str;
default = "tuwunel"; default = "tuwunel";
description = '' description = ''
OAuth2 client id this homeserver identifies itself with. Must OAuth2 client id this homeserver identifies itself with. The host
match the `id` of the corresponding entry in running authelia registers it, whichever host runs the homeserver.
`services.hyperhive.swarm.authelia.oidc.clients`.
The secret it pairs with is a host path, so it lives at The homeserver's host reads the secret it pairs with out of the
swarm secret store, into
`services.hyperhive.deploy.matrix.sso.clientSecretFile`. `services.hyperhive.deploy.matrix.sso.clientSecretFile`.
''; '';
}; };
redirectUri = lib.mkOption {
type = lib.types.nullOr lib.types.str;
readOnly = true;
default =
if cfg.gatewayHost == null then
null
else
"https://${cfg.gatewayHost}/_matrix/client/unstable/login/sso/callback/${cfg.sso.clientId}";
defaultText = lib.literalExpression ''"https://''${services.hyperhive.swarm.matrix.gatewayHost}/_matrix/client/unstable/login/sso/callback/''${services.hyperhive.swarm.matrix.sso.clientId}"'';
description = ''
SSO callback authelia sends the browser back to, and the URI it
matches **exactly**. Format-locked by tuwunel: the callback host
must point directly at the homeserver, and the path is fixed.
Read-only: the homeserver's config and its authelia client entry
both read it from here. `null` when `gatewayHost` is, because the
homeserver then has no name a browser can reach.
'';
};
}; };
}; };
} }

219
nix/host-modules/hive-matrix.nix
View file

@ -33,18 +33,25 @@ let
baoCfg = config.services.hyperhive.swarm.bao; baoCfg = config.services.hyperhive.swarm.bao;
autheliaUrl = autheliaCfg.url; autheliaUrl = autheliaCfg.url;
# The all-local case: this host runs BOTH the homeserver and the swarm's
# authelia, so the secret can be moved without an operator. The other
# two cases (swarm side, remote hive) leave `clientSecretFile` to be set
# explicitly — same split the forge module documents.
ssoLocal = deployCfg.authelia.enable;
# Where the plaintext lands inside the matrix container. Under /var/lib # Where the plaintext lands inside the matrix container. Under /var/lib
# rather than /run: the homeserver may start before the delivery unit on # rather than /run: the homeserver may start before the delivery unit on
# a later boot, and a secret that evaporates on reboot turns a working # a later boot, and a secret that evaporates on reboot turns a working
# login into an intermittent one. # login into an intermittent one.
matrixSecretPath = "/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret"; matrixSecretPath = "/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret";
# The same file seen from the host, where `swarm-bao-matrix-oidc` writes it.
# Follows `clientSecretFile`, so an operator-chosen path is where the
# fetched secret lands.
hostSecretPath = "/var/lib/nixos-containers/hive-matrix${toString deployCfg.matrix.sso.clientSecretFile}";
# Where swarm-secret-publish on authelia's host leaves this client's secret.
# matrix-ctl's store role grants read on this one path.
storeSecretPath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
# ⚠️ tuwunel does NOT read the host path directly, and this indirection # ⚠️ tuwunel does NOT read the host path directly, and this indirection
# is not ceremony. Upstream's own words: "under systemd the path must be # is not ceremony. Upstream's own words: "under systemd the path must be
# visible to the service after sandboxing (ReadWritePaths / ProtectHome), # visible to the service after sandboxing (ReadWritePaths / ProtectHome),
@ -141,13 +148,6 @@ let
# create these names, not a monopoly on them. # create these names, not a monopoly on them.
appserviceUserRegex = "^@[a-z0-9._=/-]+:${lib.escapeRegex effectiveServerName}$"; appserviceUserRegex = "^@[a-z0-9._=/-]+:${lib.escapeRegex effectiveServerName}$";
# Format-locked by tuwunel, not chosen here: the callback host must point
# directly at the matrix server and the path is fixed at
# `/_matrix/client/unstable/login/sso/callback/<client_id>`. Built once
# and read twice — by the homeserver's own config and by the client entry
# handed to authelia — because a redirect-URI mismatch is a rejected
# login with no error text worth reading.
ssoCallbackUrl = "https://${toString cfg.gatewayHost}/_matrix/client/unstable/login/sso/callback/${cfg.sso.clientId}";
# Falls back to the SWARM domain: a swarm runs one homeserver, so its # Falls back to the SWARM domain: a swarm runs one homeserver, so its
# identifier belongs to the swarm rather than to whichever hive happens # identifier belongs to the swarm rather than to whichever hive happens
# to host it — otherwise moving the container between hives would look # to host it — otherwise moving the container between hives would look
@ -385,13 +385,11 @@ in
needs the plaintext), and a literal written here would be needs the plaintext), and a literal written here would be
rendered into the world-readable nix store. rendered into the world-readable nix store.
Deliberately no fallback — a homeserver that boots with SSO Set to `/var/lib/tuwunel-oidc/<client id>.secret` wherever the
half-configured is worse than one that fails to evaluate: homeserver runs. `swarm-bao-matrix-oidc` writes the secret here
tuwunel reads OIDC from its config file rather than a database from the swarm secret store, whichever path this names. The id
row, so a malformed block can stop the server outright instead this pairs with is `swarm.matrix.sso.clientId`, which is
of merely hiding a button. The id this pairs with is swarm-wide because authelia's host registers it.
`swarm.matrix.sso.clientId`, which is swarm-wide because it must
match authelia's register.
''; '';
}; };
@ -559,9 +557,9 @@ in
(inside the matrix container) holding the OIDC client secret's (inside the matrix container) holding the OIDC client secret's
plaintext. plaintext.
On a hive that also runs the swarm's authelia this is wired up It is set by default wherever the homeserver runs, and
for you. Set it explicitly when authelia lives on another swarm-bao-matrix-oidc writes the secret there from the swarm
host: see docs/swarm/ for which secret goes where. secret store. Remove the definition that set it to null.
''; '';
} }
{ {
@ -582,94 +580,113 @@ in
} }
]; ];
# One declaration, two readers. The homeserver knows its own callback # Where the homeserver finds its OIDC client secret, inside the container.
# URL; making the operator restate it in authelia's client list would services.hyperhive.deploy.matrix.sso.clientSecretFile = lib.mkDefault matrixSecretPath;
# be a second source of truth for a string whose mismatch is a silent
# rejected login. # The homeserver's OIDC client secret, out of the swarm secret store under
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf ssoLocal [ # matrix-ctl's leaf — the one route, wherever authelia runs. Same shape as
{ # ./swarm-grafana.nix's `swarm-bao-grafana-oidc`: a login that fails
id = cfg.sso.clientId; # loudly, since a retry fixes every state it fails on, then a read that
description = "HyperHive matrix"; # degrades quietly, since an absent value is "not published yet".
redirectUris = [ ssoCallbackUrl ]; #
# tuwunel authenticates at the token endpoint by putting the # ⚠️ tuwunel requires the file to exist and be non-empty, and its
# secret in the POST body. Authelia enforces the *registered* # `LoadCredential` source must exist for the unit to start, so a host with
# method rather than accepting whichever one arrives, and its # no matrix-ctl leaf fails here, naming the options, rather than rendering
# default is `client_secret_basic` — so without this the browser # no unit.
# flow completes, consent is granted, and the very last hop fails systemd.services.swarm-bao-matrix-oidc = {
# with a 401 that names neither the secret nor the redirect. description = "fetch the homeserver's OIDC client secret from the swarm secret store";
tokenEndpointAuthMethod = "client_secret_post"; # The store's units exist only where the store runs; `Requires=` on an
} # absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = lib.optionals deployCfg.bao.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
]; ];
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
# Same case, same reasoning: this host minted the secret, so it can say requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
# where the homeserver will find it.
services.hyperhive.deploy.matrix.sso.clientSecretFile = lib.mkIf ssoLocal (
lib.mkDefault matrixSecretPath
);
# The delivery. It runs on the HOST because that is the only place both
# container trees are addressable: they share this host's network
# namespace, which makes them feel co-located, but their filesystem
# roots are separate — the homeserver cannot open a path inside
# authelia's tree however local the port looks.
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry.
# nixos-container refuses to start when a bind source is missing, and
# this secret does not exist until authelia's first boot has minted it
# — so binding it would make the homeserver wait on a file that waits
# on a container that starts after it. On a fresh hive that is a
# permanent stall presenting as "matrix is broken", several layers from
# its cause.
#
# ⚠️ Rendering a placeholder first is no way around it: tuwunel
# requires the secret file to exist *and be non-empty*, so a zero-byte
# placeholder would satisfy the bind mount and then stop the homeserver
# from starting.
systemd.services.hive-matrix-oidc-secret = lib.mkIf ssoLocal {
description = "deliver the homeserver's OIDC client secret from authelia";
after = [ "container@${autheliaCfg.machine}.service" ];
requires = [ "container@${autheliaCfg.machine}.service" ];
before = [ "container@hive-matrix.service" ]; before = [ "container@hive-matrix.service" ];
wantedBy = [ "container@hive-matrix.service" ]; wantedBy = [
serviceConfig = { "multi-user.target"
"container@hive-matrix.service"
];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. The container waits for one attempt; a secret a
# later attempt lands restarts tuwunel (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
SyslogIdentifier = "hive-matrix-oidc-secret"; SyslogIdentifier = "swarm-bao-matrix-oidc";
# Longer than the 120s bounded wait below, and that is the whole TimeoutStartSec = 30;
# point: `DefaultTimeoutStartSec` is 90s, so without this systemd
# kills the unit at 90 — before it can emit the message naming the
# file it was waiting for. The failure then reads as a timeout with
# no cause rather than "authelia has not minted <path>", which is
# the one line that makes a fresh-hive SSO stall diagnosable.
TimeoutStartSec = "180s";
}; };
path = [ pkgs.coreutils ]; environment = {
script = '' BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs ctlActive {
BAO_CLIENT_CERT = deployCfg.matrix.ctlBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.matrix.ctlBaoClientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
script =
if !ctlActive then
''
echo "the homeserver's host has no matrix-ctl store identity, so it cannot read the homeserver's OIDC client secret: set services.hyperhive.deploy.matrix.ctlBaoClientCertFile and ctlBaoClientKeyFile" >&2
exit 1
''
else
''
set -euo pipefail set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.sso.clientId}.secret"} ${atomicWriteSecret}
dst=${lib.escapeShellArg "/var/lib/nixos-containers/hive-matrix${toString deployCfg.matrix.sso.clientSecretFile}"} ${refreshConsumer}
# authelia's container is up, but its first-boot generator may err="$(mktemp)"
# still be minting. Bounded wait, then fail: a silent skip here trap 'rm -f "$err"' EXIT
# produces a homeserver whose SSO login dead-ends, which is the
# failure this whole design is trying not to ship. # `-token-only` answers on stdout and skips the token helper, a
for _ in $(seq 1 60); do # `sh` this unit's `path` does not carry.
[ -s "$src" ] && break if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
sleep 2 echo "could not log in to swarm-bao with matrix-ctl's certificate; leaving the homeserver's OIDC client secret as it is." >&2
done if [ -s "$err" ]; then
if [ ! -s "$src" ]; then cat "$err" >&2
echo "authelia has not minted $src after 120s" >&2 else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1 exit 1
fi fi
export BAO_TOKEN
# root-owned 0400, and deliberately NOT the forge's `stat -c %u` if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
# uid discovery: that reads the service's state dir to learn which echo "swarm-bao did not return ${storeSecretPath}; the homeserver has no OIDC client secret yet." >&2
# uid to hand the file to, and tuwunel runs under `DynamicUser`, so if [ -s "$err" ]; then
# there is no stable uid to discover. It never reads this path cat "$err" >&2
# directly anyway — `LoadCredential` does, as root, before the else
# sandbox and the dynamic user exist. echo "bao failed without writing a diagnostic." >&2
install -D -m 0400 -o root -g root "$src" "$dst" fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# Root-owned 0400: tuwunel runs under `DynamicUser`, so there is no
# stable uid to hand the file to, and `LoadCredential` reads it as
# root before the sandbox exists.
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf hostSecretPath)}
if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
atomic_write_secret 0400 "" ${lib.escapeShellArg hostSecretPath} "$secret"
fi
# `LoadCredential` copies the file once, at tuwunel's start.
refresh_consumer hive-matrix tuwunel.service ${lib.escapeShellArg hostSecretPath}
''; '';
}; };
@ -887,7 +904,7 @@ in
client_id = cfg.sso.clientId; client_id = cfg.sso.clientId;
client_secret_file = matrixSecretCredential; client_secret_file = matrixSecretCredential;
issuer_url = toString autheliaUrl; issuer_url = toString autheliaUrl;
callback_url = ssoCallbackUrl; callback_url = cfg.sso.redirectUri;
# Explicit, though a lone provider is auto-defaulted: # Explicit, though a lone provider is auto-defaulted:
# relying on that logs a warning every startup, and a # relying on that logs a warning every startup, and a
# recurring warning that is expected is one nobody reads. # recurring warning that is expected is one nobody reads.

24
nix/host-modules/swarm-bao.nix
View file

@ -485,23 +485,28 @@ let
matrixCtlPolicyName = "swarm-matrix-ctl"; matrixCtlPolicyName = "swarm-matrix-ctl";
matrixCtlCn = baoDeploy.matrixCtlCommonName; matrixCtlCn = baoDeploy.matrixCtlCommonName;
# ONE path, and every narrowing in it is load-bearing. # Two paths, each spelled to the leaf, and every narrowing in them is
# load-bearing.
# #
# `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than # `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than
# written by the caller — the same trap as the two grants above. # written by the caller — the same trap as the two grants above.
# #
# Not `swarm/services/*` like the publisher's: a homeserver is not entitled # Not `swarm/services/*` like the publisher's: a homeserver is not entitled
# to overwrite Grafana's OIDC client. The path is spelled to the leaf for # to overwrite Grafana's OIDC client. The paths are spelled to the leaf for
# that reason, not for tidiness. # that reason, not for tidiness.
# #
# It is the swarm appservice's token, which matrix-ctl mints inside the # The first is the swarm appservice's token, which matrix-ctl mints inside
# container and publishes here for the controller. `read` as well as write, # the container and publishes here for the controller. `read` as well as
# unlike either sibling, because publish compares before it writes. # write, unlike either sibling, because publish compares before it writes.
#
# The second is the homeserver's own OIDC client secret, read-only:
# `swarm-bao-matrix-oidc` on the homeserver's host reads it under this leaf.
matrixCtlPolicyText = '' matrixCtlPolicyText = ''
path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" { path "${credentialMountPath}/data/${swarmAppserviceTokenLeaf}" {
capabilities = ["create", "update", "read"] capabilities = ["create", "update", "read"]
} }
''; ''
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.matrix.sso.clientId}/oidc/client";
# The swarm appservice token's leaf, the nix half of # The swarm appservice token's leaf, the nix half of
# `swarm_secret_client::matrix::swarm_appservice_token_path`. Under # `swarm_secret_client::matrix::swarm_appservice_token_path`. Under
@ -836,12 +841,15 @@ let
# The queue's auth-callout responder, which checks the secret an agent # The queue's auth-callout responder, which checks the secret an agent
# presents against the one stored for it. One leaf under every agent: `+` is # presents against the one stored for it. One leaf under every agent: `+` is
# one path segment, where `*` globs only at the end and would reach every # one path segment, where `*` globs only at the end and would reach every
# other credential an agent holds. # other credential an agent holds. Plus the responder's own OIDC client
# secret, which `swarm-bao-nats-auth-oidc` reads under this same leaf.
natsAuthReaders = [ natsAuthReaders = [
{ {
name = "swarm-nats-auth"; name = "swarm-nats-auth";
cn = baoDeploy.natsAuthCommonName; cn = baoDeploy.natsAuthCommonName;
policyText = readStanza "${credentialMountPath}/data/swarm/agents/+/queue"; policyText =
readStanza "${credentialMountPath}/data/swarm/agents/+/queue"
+ readStanza "${credentialMountPath}/data/swarm/services/${hyperhiveCfg.swarm.nats.clientId}/oidc/client";
} }
]; ];

38
nix/host-modules/swarm-controller.nix
View file

@ -301,7 +301,8 @@ let
OTEL_EXPORTER_OTLP_ENDPOINT = "https://${otelSwarmCfg.domain}/${otelSwarmCfg.producerName}"; OTEL_EXPORTER_OTLP_ENDPOINT = "https://${otelSwarmCfg.domain}/${otelSwarmCfg.producerName}";
# Checked by `swarm-otel.nix`'s `oidc/${producerName}` authenticator # Checked by `swarm-otel.nix`'s `oidc/${producerName}` authenticator
# against exactly this controller's own registered audience (see the # against exactly this controller's own registered audience (see the
# `audience = [ queueClientId ]` client entry below) — the two have # `audience = [ queueClientId ]` client entry in
# ./glue-controller-oidc-client.nix) — the two have
# to name the same value or the exporter mints a token the receiver # to name the same value or the exporter mints a token the receiver
# was never told to accept. # was never told to accept.
SWARM_CONTROLLER_OTEL_AUDIENCE = queueClientId; SWARM_CONTROLLER_OTEL_AUDIENCE = queueClientId;
@ -360,9 +361,10 @@ in
default = "swarm-controller"; default = "swarm-controller";
description = '' description = ''
The OAuth2 client id the controller presents to the swarm queue. The OAuth2 client id the controller presents to the swarm queue.
Read-only: it is what this module registers, published so the Read-only: it is the client the host running authelia registers,
auth-callout responder can be told which client may read every whichever host runs the controller, published so the auth-callout
hive's key without repeating the string. responder can be told which client may read every hive's key without
repeating the string.
The responder decides that from a client id, and a client id it The responder decides that from a client id, and a client id it
does not recognise is **denied**. A denial reaches a NATS client does not recognise is **denied**. A denial reaches a NATS client
@ -704,34 +706,6 @@ in
# exactly what it must not inherit. # exactly what it must not inherit.
environment.systemPackages = [ swarmctlConfigured ]; environment.systemPackages = [ swarmctlConfigured ];
# One declaration, two readers — the controller knows which client id it
# authenticates under, so making the operator restate it in authelia's
# client list would be a second source of truth for a string whose
# mismatch is an opaque 401 from the token endpoint. Same shape as the
# queue's own client declaration.
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [
{
id = queueClientId;
description = "HyperHive swarm controller";
# `client_credentials`: a daemon authenticating as itself, with
# nobody to redirect. Declared rather than inferred from an empty
# redirect list, because authelia permits only the grants a client
# names and an omitted `grant_types` means authorization-code alone.
kind = "machine";
# Its own id as its own permitted audience — same "client's own id
# is the value it may ask for" shape `swarm-authelia.nix`'s
# `hiveClients` uses, just self-referential instead of prefixed:
# this controller is not a hive (see `queueClientId`'s own doc
# comment), so it has no `hiveClientPrefix` name to reuse. This is
# what lets `vcs_metrics::authenticated_http_client()` mint a token
# for `SWARM_CONTROLLER_OTEL_AUDIENCE` and have `swarm-otel.nix`'s
# `oidc/swarm` authenticator (which checks for this same id) accept
# it — registering here without requesting there, or the reverse,
# both fail as a healthy-looking 401 several layers from this line.
audience = [ queueClientId ];
}
];
# A controller without a queue is not a lighter controller, it is a # A controller without a queue is not a lighter controller, it is a
# broken one: `/api/hives/status` reads the KV, so its whole aggregate # broken one: `/api/hives/status` reads the KV, so its whole aggregate
# half is off. The operator's rule is that the controller needs a queue # half is off. The operator's rule is that the controller needs a queue

6
nix/host-modules/swarm-nats-service.nix
View file

@ -107,9 +107,9 @@ in
default = "swarm-nats"; default = "swarm-nats";
description = '' description = ''
OAuth2 client id the queue's authentication path identifies OAuth2 client id the queue's authentication path identifies
itself with. Must match the `id` of the corresponding entry in itself with. The host running authelia registers it, whichever
`services.hyperhive.swarm.authelia.oidc.clients` — which this host runs the queue, and the queue's host reads its secret out of
module contributes for you when both run on this host. the swarm secret store.
''; '';
}; };

219
nix/host-modules/swarm-nats.nix
View file

@ -57,7 +57,10 @@ let
responderConfigured = responderConfigured =
deployCfg.nats.autoGenerateCallout deployCfg.nats.autoGenerateCallout
|| (deployCfg.nats.calloutUserSeedFile != "" && deployCfg.nats.calloutIssuerSeedFile != ""); || (deployCfg.nats.calloutUserSeedFile != "" && deployCfg.nats.calloutIssuerSeedFile != "");
clientSecretSource = "${deployCfg.authelia.hostClientSecretDir}/${cfg.clientId}.secret"; # Where swarm-secret-publish on authelia's host leaves this client's secret,
# composed from the same swarm-wide client id it publishes under. The
# responder's store role grants read on this one path.
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
introspectionUrl = "${toString autheliaUrl}/api/oidc/introspection"; introspectionUrl = "${toString autheliaUrl}/api/oidc/introspection";
# Where the responder's seeds actually come from. One name for two # Where the responder's seeds actually come from. One name for two
@ -234,6 +237,10 @@ let
); );
authCredential = id: "/run/credentials/swarm-nats-auth.service/${id}"; authCredential = id: "/run/credentials/swarm-nats-auth.service/${id}";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
# Re-issue once the leaf is past half of the 720h `pki/roles/swarm-nats` # Re-issue once the leaf is past half of the 720h `pki/roles/swarm-nats`
# grants it (./swarm-bao.nix), so the daily timer below has two weeks of # grants it (./swarm-bao.nix), so the daily timer below has two weeks of
# retries before it lapses. # retries before it lapses.
@ -430,13 +437,17 @@ in
example = "/var/lib/swarm-bao-pki/nats-auth.pem"; example = "/var/lib/swarm-bao-pki/nats-auth.pem";
description = '' description = ''
Client certificate the auth-callout responder presents to the swarm's Client certificate the auth-callout responder presents to the swarm's
secret store to read agent queue credentials. Its subject must be secret store to read agent queue credentials, and that
`swarm-bao-nats-auth-oidc` presents to read the responder's own OIDC
client secret. Its subject must be
{option}`services.hyperhive.deploy.bao.natsAuthCommonName`. {option}`services.hyperhive.deploy.bao.natsAuthCommonName`.
No default. ./glue-nats-auth-bao-identity.nix points it at the leaf No default. ./glue-nats-auth-bao-identity.nix points it at the leaf
./glue-bao-tls.nix mints, where this host mints one. Unset, or set to ./glue-bao-tls.nix mints, where this host mints one; on any other
a file that does not exist, the responder denies every agent token and queue host it is a file the operator places. Unset, or set to a file
admits OIDC clients as before. that does not exist, `swarm-bao-nats-auth-oidc` fails; a responder
with no OIDC client secret on disk cannot start, and the queue then
refuses every client.
A path, never a value. A path, never a value.
''; '';
@ -532,33 +543,6 @@ in
} }
]; ];
# One declaration, two readers. The queue knows which client id it
# authenticates under; making the operator restate it in authelia's
# client list would be a second source of truth for a string whose
# mismatch is an opaque 401 from the token endpoint.
#
# `client_credentials` rather than an authorization-code flow: a
# queue's clients are daemons with nobody to redirect, so a
# non-interactive grant is what makes a hive able to authenticate at
# all. No redirect URI exists or is wanted.
#
# ⚠️ `kind` says that, and an empty `redirectUris` does NOT. This
# declaration carried the comment above while rendering as an
# interactive client, because authelia permits only the grants a
# client names and an omitted `grant_types` means authorization-code
# alone. Measured against 4.39.20: the token endpoint answered
# `unauthorized_client: The OAuth 2.0 Client is not allowed to use
# authorization grant 'client_credentials'`. Introspection — which
# is all the responder needs today — worked throughout, which is why
# nothing was visibly broken while the comment was untrue.
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf deployCfg.authelia.enable [
{
id = cfg.clientId;
description = "HyperHive swarm queue";
kind = "machine";
}
];
# Declared here rather than in the collector's module: an entry then # Declared here rather than in the collector's module: an entry then
# exists only where the service that named it runs. Gated on a # exists only where the service that named it runs. Gated on a
# collector, because a target nobody reads asserts a collection that is # collector, because a target nobody reads asserts a collection that is
@ -882,17 +866,16 @@ in
}; };
}; };
# Deliver the responder's three credentials into the container before
# it starts. Same shape as `hive-matrix-oidc-secret`, and for the same
# reason it is a copy rather than a `bindMounts` entry: nixos-container
# refuses to start when a bind source is missing, so one absent seed
# would take down the **whole container including the queue**, not
# merely the responder. A far larger blast radius than the fault.
# Order the container after the host CA generator, so the bind source # Order the container after the host CA generator, so the bind source
# exists before nspawn sets the mount up. Without it a late CA fails the # exists before nspawn sets the mount up. Without it a late CA fails the
# container start outright rather than degrading. # container start outright rather than degrading.
systemd.services."container@${machine}" = caTrust.containerOrdering; systemd.services."container@${machine}" = caTrust.containerOrdering;
# Deliver the responder's seeds and store identity into the container
# before it starts. A copy rather than a `bindMounts` entry:
# nixos-container refuses to start when a bind source is missing, so one
# absent seed would take down the **whole container including the queue**,
# not merely the responder.
systemd.services.swarm-nats-auth-secrets = lib.mkIf responderConfigured { systemd.services.swarm-nats-auth-secrets = lib.mkIf responderConfigured {
description = "deliver the swarm queue responder's credentials"; description = "deliver the swarm queue responder's credentials";
before = [ "container@swarm-nats.service" ]; before = [ "container@swarm-nats.service" ];
@ -903,13 +886,6 @@ in
# or absent seed is worse than not running. # or absent seed is worse than not running.
after = after =
lib.optional deployCfg.nats.autoGenerateCallout "swarm-nats-callout-keys.service" lib.optional deployCfg.nats.autoGenerateCallout "swarm-nats-callout-keys.service"
# The third credential does not come from the generator above — it is
# minted by authelia's FIRST BOOT, inside its own container. Ordering
# after that container is necessary and NOT sufficient: the container
# being up says nothing about whether its in-container secrets unit
# has finished. The wait in the script is what actually closes it;
# this only stops us spinning for the full timeout on every boot.
++ lib.optional deployCfg.authelia.enable "container@${autheliaCfg.machine}.service"
# Where the store's PKI is minted on this host, the responder's leaf is # Where the store's PKI is minted on this host, the responder's leaf is
# one of its files. Ordering only: elsewhere the unit does not exist. # one of its files. Ordering only: elsewhere the unit does not exist.
++ lib.optional authStoreActive "swarm-bao-pki.service"; ++ lib.optional authStoreActive "swarm-bao-pki.service";
@ -917,14 +893,6 @@ in
serviceConfig = { serviceConfig = {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
# Must exceed the script's own wait below, and is set rather than
# left to the default for exactly that reason: systemd's
# `DefaultTimeoutStartSec` is 90s, so a 120s wait would be killed at
# 90 and the operator would get a generic unit timeout instead of
# the message that names the missing file and says what to do.
# The bound that matters belongs in one place, and this makes the
# two visibly related.
TimeoutStartSec = "180s";
SyslogIdentifier = "swarm-nats-auth-secrets"; SyslogIdentifier = "swarm-nats-auth-secrets";
}; };
path = [ pkgs.coreutils ]; path = [ pkgs.coreutils ];
@ -935,38 +903,10 @@ in
${lib.escapeShellArg (hostPath "callout-user.seed")} ${lib.escapeShellArg (hostPath "callout-user.seed")}
install -m 0400 ${lib.escapeShellArg issuerSeedFile} \ install -m 0400 ${lib.escapeShellArg issuerSeedFile} \
${lib.escapeShellArg (hostPath "issuer.seed")} ${lib.escapeShellArg (hostPath "issuer.seed")}
# Wait for authelia's minted secret rather than failing the instant
# it is absent. On a fresh boot this unit and authelia's first-boot
# generator race, and losing that race used to cost the WHOLE QUEUE:
# this exits 1, the responder never starts, and `auth_callout` with
# no responder refuses every client — fail-closed by design, so the
# symptom appears on every queue client and nowhere near the cause.
#
# Bounded, not indefinite. Where authelia runs on another host the
# file is never going to appear, and blocking the queue container
# forever would replace a clear failure with a hang. After the
# timeout this fails exactly as it did before, having first given
# the co-located case the seconds it actually needs.
secret=${lib.escapeShellArg clientSecretSource}
deadline=$(( SECONDS + 120 ))
while [ ! -s "$secret" ]; do
if [ "$SECONDS" -ge "$deadline" ]; then
echo "the swarm queue responder's OIDC secret never appeared at $secret" >&2
echo "(authelia mints it on first boot; if authelia runs on another host," >&2
echo " copy the secret there and this unit will pick it up)" >&2
exit 1
fi
sleep 2
done
install -m 0400 "$secret" \
${lib.escapeShellArg (hostPath "oidc-client.secret")}
'' ''
# The store identity is optional to the responder, so an absent source # The store identity is optional to the responder's agent-token path, so
# is delivered as an empty file rather than failing this unit: a missing # an absent source is delivered as an empty file rather than failing this
# `LoadCredential` source stops the responder starting, and a responder # unit: a missing `LoadCredential` source stops the responder starting.
# that does not start denies every client. An empty identity fails only
# the store login, which denies agent tokens.
+ lib.concatStrings ( + lib.concatStrings (
lib.mapAttrsToList (id: source: '' lib.mapAttrsToList (id: source: ''
if [ -s ${lib.escapeShellArg source} ]; then if [ -s ${lib.escapeShellArg source} ]; then
@ -979,6 +919,117 @@ in
); );
}; };
# The responder's OIDC client secret, out of the swarm secret store under
# the responder's own leaf — the one route, wherever authelia runs. Same
# shape as ./swarm-grafana.nix's `swarm-bao-grafana-oidc`: a login that
# fails loudly, since a retry fixes every state it fails on, then a read
# that degrades quietly, since an absent value is "not published yet".
#
# ⚠️ Without the secret the responder does not start (its `LoadCredential`
# source is missing), and `auth_callout` with no responder refuses every
# client. A host with no responder leaf therefore fails here, naming the
# options, rather than rendering no unit.
systemd.services.swarm-bao-nats-auth-oidc = {
description = "fetch the swarm queue responder's OIDC client secret from the swarm secret store";
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = [
"swarm-nats-auth-secrets.service"
]
++ lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${machine}.service" ];
wantedBy = [
"multi-user.target"
"container@${machine}.service"
];
path = [
baoDeploy.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. The container waits for one attempt; a secret a
# later attempt lands restarts the responder (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "swarm-bao-nats-auth-oidc";
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs authStoreActive {
BAO_CLIENT_CERT = deployCfg.nats.authBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.nats.authBaoClientKeyFile;
}
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script =
if !authStoreActive then
''
echo "the swarm queue responder has no store identity, so it cannot read its OIDC client secret: set services.hyperhive.deploy.nats.authBaoClientCertFile and authBaoClientKeyFile" >&2
exit 1
''
else
''
set -euo pipefail
${atomicWriteSecret}
${refreshConsumer}
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with the queue responder's certificate; leaving its OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; the queue responder has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# Root-owned 0400: `LoadCredential` reads it as root before the
# responder's `DynamicUser` exists. `printf` inside
# `atomic_write_secret` is a builtin, so the plaintext never
# becomes an argument in /proc.
install -d -m 0700 ${lib.escapeShellArg secretDir}
if secret_differs ${lib.escapeShellArg (hostPath "oidc-client.secret")} "$secret"; then
atomic_write_secret 0400 "" ${lib.escapeShellArg (hostPath "oidc-client.secret")} "$secret"
fi
# `LoadCredential` copies the file once, at the responder's start.
refresh_consumer ${lib.escapeShellArg machine} swarm-nats-auth.service ${lib.escapeShellArg (hostPath "oidc-client.secret")}
'';
};
# ⚠️ Minted on the HOST, not in the container, because the responder is # ⚠️ Minted on the HOST, not in the container, because the responder is
# a separate unit that needs the user seed: generating it inside would # a separate unit that needs the user seed: generating it inside would
# trap it there and require a secret-export path back out — the exact # trap it there and require a secret-export path back out — the exact

2
nix/host-modules/swarm-otel.nix
View file

@ -132,7 +132,7 @@ let
# Upstream's collector unit runs with `DynamicUser = true`, and the # Upstream's collector unit runs with `DynamicUser = true`, and the
# prometheus receiver opens `client_secret_file` ITSELF, at runtime, as that # prometheus receiver opens `client_secret_file` ITSELF, at runtime, as that
# user — so there is no stable uid to hand a file to, and the root-owned # user — so there is no stable uid to hand a file to, and the root-owned
# 0400 shape `hive-matrix-oidc-secret` delivers to would be unreadable. # 0400 shape `swarm-bao-matrix-oidc` delivers to would be unreadable.
# (Matrix gets away with it because `LoadCredential` reads the file as root # (Matrix gets away with it because `LoadCredential` reads the file as root
# before the sandbox exists, and tuwunel never opens that path itself.) # before the sandbox exists, and tuwunel never opens that path itself.)
# #

12
nix/host-modules/swarm-secret-publisher.nix
View file

@ -79,6 +79,11 @@ let
# The store's browser UI, whose `oidc` auth method the store's host # The store's browser UI, whose `oidc` auth method the store's host
# configures with this secret. # configures with this secret.
hyperhiveCfg.swarm.bao.ui.oidc.clientId hyperhiveCfg.swarm.bao.ui.oidc.clientId
# The queue's auth-callout responder, read by `swarm-bao-nats-auth-oidc`
# on the queue's host.
hyperhiveCfg.swarm.nats.clientId
# The homeserver, read by `swarm-bao-matrix-oidc` on its host.
hyperhiveCfg.swarm.matrix.sso.clientId
]; ];
# The swarm controller's OIDC client, which it reads back from the store # The swarm controller's OIDC client, which it reads back from the store
@ -110,7 +115,8 @@ in
Turning it off leaves every hive but this one without its agents' Turning it off leaves every hive but this one without its agents'
credential, the swarm controller unable to start, the swarm's Grafana credential, the swarm controller unable to start, the swarm's Grafana
without any login at all, and its collector pushing unauthenticated. without any login at all, its collector pushing unauthenticated, the
queue's responder unable to start, and the homeserver unable to start.
Each secret has exactly one route and this is the producer's Each secret has exactly one route and this is the producer's
end of it, so the honest reason to set it false is a deployment end of it, so the honest reason to set it false is a deployment
delivering those secrets by some other mechanism it owns. delivering those secrets by some other mechanism it owns.
@ -249,8 +255,8 @@ in
value=@"$src" value=@"$src"
published=$((published + 1)) published=$((published + 1))
else else
# Authelia mints it only where the controller registers its client, # Not an error: authelia mints on its first boot, and the path unit
# which is where the controller runs beside it. # above re-runs this when it lands.
echo "no minted secret at $src yet; the path unit will re-run this" >&2 echo "no minted secret at $src yet; the path unit will re-run this" >&2
skipped=$((skipped + 1)) skipped=$((skipped + 1))
fi fi

62
nix/module-eval/bao-grants.nix
View file

@ -98,6 +98,14 @@ let
deploy.swarm-otel.enable = true; deploy.swarm-otel.enable = true;
}; };
# The store and the homeserver on one host: the fetch of the homeserver's
# OIDC secret logs in under matrix-ctl's leaf, whose role this host writes.
baoGrantWithHomeserver = hive {
deploy.bao.enable = true;
deploy.bao.bootstrapTokenFile = "/run/secrets/bao-bootstrap.token";
deploy.matrix.enable = true;
};
# The store with none of the readers beside it. Grafana and the collector are # The store with none of the readers beside it. Grafana and the collector are
# simply off; the queue reader renders on any host holding # simply off; the queue reader renders on any host holding
# its leaf, which ./glue-bao-tls.nix mints here, so that leaf is taken away. # its leaf, which ./glue-bao-tls.nix mints here, so that leaf is taken away.
@ -426,23 +434,26 @@ let
); );
} }
{ {
# The third principal's grant, and the narrowest of the three: ONE path, # The third principal's grant: two paths, each spelled to the leaf. The
# spelled to the leaf. The negative arms are the property — a homeserver # negative arms are the property — a homeserver is not entitled to
# is not entitled to overwrite Grafana's OIDC client, so widening this to # overwrite Grafana's OIDC client, so widening this to the `services/`
# the `services/` prefix the publisher holds would be a real loss even # prefix the publisher holds would be a real loss even though it would
# though it would read as tidier. # read as tidier.
# #
# The leaf is the swarm appservice token, which matrix-ctl mints and # The first leaf is the swarm appservice token, which matrix-ctl mints
# publishes for the controller. Counted, so a second stanza fails rather # and publishes for the controller; the second is the homeserver's own
# than riding along beside a correct one. # OIDC client secret, read-only. Counted, so a third stanza fails rather
name = "matrix-ctl's grant is the swarm appservice token, nothing else"; # than riding along beside correct ones.
name = "matrix-ctl's grant is the swarm appservice token and the homeserver's OIDC secret, nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-matrix-ctl-policy.script; s = baoGrantHere.systemd.services.swarm-bao-matrix-ctl-policy.script;
in in
lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/matrix/appservice-token\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}" s lib.hasInfix "path \"secret/data/swarm/controller/swarm-controller/matrix/appservice-token\" {\n capabilities = [\"create\", \"update\", \"read\"]\n}" s
&& lib.length (lib.splitString "path \"" s) == 2 && lib.hasInfix "path \"secret/data/swarm/services/tuwunel/oidc/client\" {\n capabilities = [\"read\"]\n}" s
&& !(lib.hasInfix "secret/data/swarm/services" s) && lib.length (lib.splitString "path \"" s) == 3
&& !(lib.hasInfix "secret/data/swarm/services/*" s)
&& !(lib.hasInfix "swarm-grafana" s)
&& !(lib.hasInfix "secret/data/swarm/agents" s) && !(lib.hasInfix "secret/data/swarm/agents" s)
&& !(lib.hasInfix "secret/data/swarm/hives" s) && !(lib.hasInfix "secret/data/swarm/hives" s)
&& !(lib.hasInfix "sys/policies/acl" s); && !(lib.hasInfix "sys/policies/acl" s);
@ -559,18 +570,20 @@ let
{ {
# `+` is one path segment, so this reaches `swarm/agents/<agent>/queue` # `+` is one path segment, so this reaches `swarm/agents/<agent>/queue`
# and no other credential an agent holds; `swarm/agents/*` would reach # and no other credential an agent holds; `swarm/agents/*` would reach
# all of them. # all of them. The responder's own OIDC client secret is the one
name = "the queue responder's grant is every agent's queue credential and nothing else"; # `services/` leaf beside it.
name = "the queue responder's grant is every agent's queue credential and its own OIDC secret, nothing else";
ok = ok =
let let
s = baoGrantHere.systemd.services.swarm-bao-nats-auth-policy.script; s = baoGrantHere.systemd.services.swarm-bao-nats-auth-policy.script;
in in
lib.hasInfix "path \"secret/data/swarm/agents/+/queue\" {" s lib.hasInfix "path \"secret/data/swarm/agents/+/queue\" {" s
&& lib.hasInfix "capabilities = [\"read\"]" s && lib.hasInfix "path \"secret/data/swarm/services/swarm-nats/oidc/client\" {" s
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 1 && lib.length (lib.filter lib.isList (builtins.split "capabilities = \\[\"read\"]" s)) == 2
&& lib.length (lib.filter lib.isList (builtins.split "path \"" s)) == 2
&& !(lib.hasInfix "secret/data/swarm/agents/*" s) && !(lib.hasInfix "secret/data/swarm/agents/*" s)
&& !(lib.hasInfix "secret/data/swarm/hives" s) && !(lib.hasInfix "secret/data/swarm/hives" s)
&& !(lib.hasInfix "secret/data/swarm/services" s) && !(lib.hasInfix "secret/data/swarm/services/*" s)
&& lib.hasInfix "auth/cert/certs/swarm-nats-auth" s && lib.hasInfix "auth/cert/certs/swarm-nats-auth" s
&& lib.hasInfix "allowed_common_names=swarm-nats-auth" s && lib.hasInfix "allowed_common_names=swarm-nats-auth" s
&& lib.hasInfix "token_policies=swarm-nats-auth" s; && lib.hasInfix "token_policies=swarm-nats-auth" s;
@ -737,6 +750,23 @@ let
in in
lib.all waitsFor (policyReaders ++ [ "swarm-bao-forwarder-oidc" ]); lib.all waitsFor (policyReaders ++ [ "swarm-bao-forwarder-oidc" ]);
} }
{
# The homeserver's OIDC secret is read under matrix-ctl's leaf, so its
# reader waits for the unit writing matrix-ctl's role, not one named
# after itself. The second arm is the absence control: on the store's
# host without a homeserver or queue, the ordering defines no reader.
name = "the homeserver's OIDC secret reader is ordered after matrix-ctl's policy unit";
ok =
let
u = baoGrantWithHomeserver.systemd.services.swarm-bao-matrix-oidc;
noReaders = baoGrantNoReaders.systemd.services;
in
lib.elem "swarm-bao-matrix-ctl-policy.service" u.after
&& lib.elem "swarm-bao-matrix-ctl-policy.service" u.wants
&& !(lib.elem "swarm-bao-matrix-ctl-policy.service" u.requires)
&& !(noReaders ? swarm-bao-matrix-oidc)
&& !(noReaders ? swarm-bao-nats-auth-oidc);
}
{ {
# The ordering is set apart from each reader's own definition, so it can # The ordering is set apart from each reader's own definition, so it can
# define a reader by itself: `after` on a unit nothing else declares is a # define a reader by itself: `after` on a unit nothing else declares is a

36
nix/module-eval/matrix-core.nix
View file

@ -71,6 +71,42 @@ let
&& !(s.hive-c0re.environment ? BAO_ADDR) && !(s.hive-c0re.environment ? BAO_ADDR)
&& !(lib.any (c: lib.hasPrefix "bao-" c) s.hive-c0re.serviceConfig.LoadCredential); && !(lib.any (c: lib.hasPrefix "bao-" c) s.hive-c0re.serviceConfig.LoadCredential);
} }
{
# The homeserver's OIDC secret has one route, the store, under
# matrix-ctl's leaf, landing where `clientSecretFile` points inside the
# container. No unit copies it out of authelia's tree, even on a host
# that runs authelia.
name = "the homeserver reads its OIDC secret from the store under matrix-ctl's leaf";
ok =
let
s = matrixBodyCap.systemd.services;
u = s.swarm-bao-matrix-oidc;
m = matrixBodyCap.services.hyperhive.deploy.matrix;
in
matrixBodyCap.services.hyperhive.deploy.authelia.enable
&& u.environment.BAO_CLIENT_CERT == m.ctlBaoClientCertFile
&& u.environment.BAO_CLIENT_KEY == m.ctlBaoClientKeyFile
&& m.sso.clientSecretFile == "/var/lib/tuwunel-oidc/tuwunel.secret"
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/tuwunel/oidc/client" u.script
&& lib.hasInfix "/var/lib/nixos-containers/hive-matrix/var/lib/tuwunel-oidc/tuwunel.secret" u.script
&& lib.hasInfix "refresh_consumer hive-matrix tuwunel.service" u.script
&& lib.elem "container@hive-matrix.service" u.before
&& !(s ? hive-matrix-oidc-secret);
}
{
# No matrix-ctl leaf is a homeserver with no OIDC secret, which tuwunel
# refuses to start on. The unit still renders, and fails naming the
# options.
name = "a homeserver host with no matrix-ctl leaf renders a secret fetch that fails naming the options";
ok =
let
u = matrixNoBaoIdentity.systemd.services.swarm-bao-matrix-oidc;
in
!(u.environment ? BAO_CLIENT_CERT)
&& lib.hasInfix "deploy.matrix.ctlBaoClientCertFile" u.script
&& lib.hasInfix "exit 1" u.script
&& !(lib.hasInfix "bao login" u.script);
}
{ {
# hive-c0re logs in with the hive's matrix token over this URL, and on a # hive-c0re logs in with the hive's matrix token over this URL, and on a
# host without the homeserver that login crosses the network, so the # host without the homeserver that login crosses the network, so the

59
nix/module-eval/nats-authelia.nix
View file

@ -178,9 +178,8 @@ let
&& lib.hasPrefix "https://" u.environment.BAO_ADDR; && lib.hasPrefix "https://" u.environment.BAO_ADDR;
} }
{ {
# An absent leaf must not stop the responder, which would deny every # An absent leaf must not stop the responder's agent-token path: it is
# client: it is delivered empty, and the three credentials the responder # delivered empty, beside the two seeds.
# cannot run without are delivered too.
name = "the copy unit delivers the store identity, and an absent one as an empty file"; name = "the copy unit delivers the store identity, and an absent one as an empty file";
ok = ok =
let let
@ -189,7 +188,43 @@ let
lib.hasInfix "install -m 0400 /etc/pki/nats-auth.pem " s lib.hasInfix "install -m 0400 /etc/pki/nats-auth.pem " s
&& lib.hasInfix "install -m 0400 /etc/pki/nats-auth-key.pem " s && lib.hasInfix "install -m 0400 /etc/pki/nats-auth-key.pem " s
&& lib.hasInfix "install -m 0400 /dev/null " s && lib.hasInfix "install -m 0400 /dev/null " s
&& lib.hasInfix "oidc-client.secret" s; && lib.hasInfix "callout-user.seed" s
&& lib.hasInfix "issuer.seed" s;
}
{
# The responder's OIDC secret has one route, the store, under the
# responder's own leaf. The copy unit's arms are the control: on a queue
# host without authelia nothing reads authelia's tree, and nothing
# copies the secret from any host file.
name = "a queue host without authelia reads the responder's OIDC secret from the store, and copies none from a host file";
ok =
let
services = natsWithStoreIdentity.systemd.services;
fetch = services.swarm-bao-nats-auth-oidc;
copy = services.swarm-nats-auth-secrets.script;
in
!natsWithStoreIdentity.services.hyperhive.deploy.authelia.enable
&& fetch.environment.BAO_CLIENT_CERT == "/etc/pki/nats-auth.pem"
&& fetch.environment.BAO_CLIENT_KEY == "/etc/pki/nats-auth-key.pem"
&& lib.hasInfix "bao kv get -field=value secret/swarm/services/swarm-nats/oidc/client" fetch.script
&& lib.hasInfix "/var/lib/nixos-containers/swarm-nats/var/lib/swarm-nats-auth/oidc-client.secret" fetch.script
&& lib.hasInfix "refresh_consumer swarm-nats swarm-nats-auth.service" fetch.script
&& lib.elem "container@swarm-nats.service" fetch.before
&& !(lib.hasInfix "oidc-client" copy)
&& !(lib.hasInfix "authelia" copy);
}
{
# No responder leaf is a responder with no OIDC secret, which refuses
# every client. The unit still renders, and fails naming the options.
name = "a queue host with no responder leaf renders a secret fetch that fails naming the options";
ok =
let
fetch = natsOldPath.systemd.services.swarm-bao-nats-auth-oidc;
in
!(fetch.environment ? BAO_CLIENT_CERT)
&& lib.hasInfix "deploy.nats.authBaoClientCertFile" fetch.script
&& lib.hasInfix "exit 1" fetch.script
&& !(lib.hasInfix "bao login" fetch.script);
} }
{ {
# The control: no identity, no store wiring, and the responder's # The control: no identity, no store wiring, and the responder's
@ -203,6 +238,18 @@ let
&& !(u.environment ? BAO_CLIENT_CERT) && !(u.environment ? BAO_CLIENT_CERT)
&& lib.length u.serviceConfig.LoadCredential == 3; && lib.length u.serviceConfig.LoadCredential == 3;
} }
{
# On the store's host the fetch logs in under the responder's leaf, so it
# waits for the unit writing that role.
name = "on the store's host the responder's OIDC secret fetch uses its leaf and waits for its role";
ok =
let
u = natsOnStoreHost.systemd.services.swarm-bao-nats-auth-oidc;
in
u.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/nats-auth.pem"
&& lib.elem "swarm-bao-nats-auth-policy.service" u.after
&& lib.elem "swarm-bao-nats-auth-policy.service" u.wants;
}
{ {
# On the store's host the glue pairs the responder with a leaf of its # On the store's host the glue pairs the responder with a leaf of its
# own, never the queue's TLS-issuing one or the hive's. # own, never the queue's TLS-issuing one or the hive's.
@ -220,10 +267,10 @@ let
{ {
# Not a rename test. `hostClientSecretDir` is `readOnly`, so the fixture # Not a rename test. `hostClientSecretDir` is `readOnly`, so the fixture
# cannot define it; what can break is a reader left pointing at the # cannot define it; what can break is a reader left pointing at the
# namespace it moved out of. Five modules read this through an # namespace it moved out of. Several modules read this through an
# `autheliaCfg` alias, where a path-shaped grep does not see it. # `autheliaCfg` alias, where a path-shaped grep does not see it.
name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace"; name = "a consumer of authelia's host client-secret dir renders it from the deploy namespace";
ok = lib.hasInfix "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients/" autheliaOldPath.systemd.services.swarm-nats-auth-secrets.script; ok = lib.hasPrefix "/var/lib/nixos-containers/swarm-authelia/var/lib/authelia-swarm/oidc-clients/" autheliaOldPath.services.hyperhive.otel.clientSecretFile;
} }
{ {
# authelia exits at startup on `users: {}`, so an empty database, and a # authelia exits at startup on `users: {}`, so an empty database, and a

50
nix/module-eval/secret-publisher.nix
View file

@ -135,6 +135,56 @@ let
lib.elem "swarm-collector" (clients secretPublisherHere) lib.elem "swarm-collector" (clients secretPublisherHere)
&& !(lib.elem "swarm-collector" (clients otelBaoRemoteAuthelia)); && !(lib.elem "swarm-collector" (clients otelBaoRemoteAuthelia));
} }
{
# Registration follows authelia, not the service: the IdP host below runs
# no controller, queue or homeserver (the first arms), and still lists all
# three clients. The last arm is the control — a host with no IdP lists
# none of them.
name = "an authelia host running no controller, queue or homeserver registers all three clients";
ok =
let
d = secretPublisherRemote.services.hyperhive.deploy;
clients = m: map (c: c.id) m.services.hyperhive.swarm.authelia.oidc.clients;
byId =
id:
lib.findFirst (
c: c.id == id
) null secretPublisherRemote.services.hyperhive.swarm.authelia.oidc.clients;
controller = byId "swarm-controller";
nats = byId "swarm-nats";
matrix = byId "tuwunel";
in
!d.swarm-controller.enable
&& !d.nats.enable
&& !d.matrix.enable
&& controller.kind == "machine"
&& controller.audience == [ "swarm-controller" ]
&& nats.kind == "machine"
&&
matrix.redirectUris == [
"https://chat.t.local/_matrix/client/unstable/login/sso/callback/tuwunel"
]
&& matrix.tokenEndpointAuthMethod == "client_secret_post"
&& !(lib.any (id: lib.elem id (clients grafanaRemoteAuthelia)) [
"swarm-controller"
"swarm-nats"
"tuwunel"
]);
}
{
# The other half of the queue's and the homeserver's one route: the
# publisher on an IdP host that runs neither carries both secrets to the
# paths their readers fetch.
name = "the publisher on an authelia host carries the queue's and homeserver's secrets";
ok =
let
s = secretPublisherRemote.systemd.services.swarm-secret-publish.script;
in
lib.hasInfix "secret/swarm/services/swarm-nats/oidc/client" s
&& lib.hasInfix "/swarm-nats.secret" s
&& lib.hasInfix "secret/swarm/services/tuwunel/oidc/client" s
&& lib.hasInfix "/tuwunel.secret" s;
}
{ {
# Both ends of a wire nothing at eval time carries end to end: the # Both ends of a wire nothing at eval time carries end to end: the
# publisher on authelia's host writes the path the reader on the # publisher on authelia's host writes the path the reader on the