Watch
0
0
Fork
You've already forked hyperhive
0

matrix: drop the per-hive appservice and its unused token

Every hive's homeserver loaded a per-hive `hyperhive` appservice
registration, and the swarm minted and delivered an as_token for it
through bao. No client presented that token: swarm-controller creates
every `@hive-<hive>` sender and every agent account with the swarm
appservice token (hive_sender.rs, agent_token.rs), and no Rust code read
the per-hive one.

Removed, with everything that existed only to serve it:
- hive-matrix.nix: the `hyperhive` registration, its activation-time
  render, the host bind mount of /var/lib/hyperhive/matrix-appservice,
  its LoadCredential entry, and the internal options
  `deploy.matrix.appserviceTokenFile` / `appserviceRegistrationScript`.
  tuwunel now loads only the swarm registration.
- glue-matrix-bao-token.nix and its `swarm-bao-matrix-token` unit.
- glue-bao-tls.nix: the matrix-token leaf and its defaults.
- swarm-bao.nix: the `swarm-matrix-token-<hive>` roles and policies
  (`swarm-bao-matrix-token-policy`), and the options
  `deploy.bao.matrixTokenClient{Cert,Key}File` and
  `matrixTokenCommonNamePrefix`, now mkRemovedOptionModule.
- swarm-secret-publisher.nix: the per-hive token mint and its state dir.
- swarm.nix: the reserved `<matrixTokenCommonNamePrefix>-<hive>` CN.
- swarm-secret-client: `matrix::appservice_token_path`.

The swarm registration (swarm-matrix-ctl, swarm-controller) is untouched.
Comments and docs that pointed at the removed glue file now point at
glue-queue-agent-credential.nix, which has the same reader shape.

Refs #4428
This commit is contained in:
atlas 2026-10-04 22:52:22 +02:00 • committed by mara
commit 48067d07e8
31 changed files with 362 additions and 1233 deletions

View file

@ -105,8 +105,9 @@ delegation (the latter lives in `gateway.md::Discovery flow`).
## Provisioning flow (appservice) ## Provisioning flow (appservice)
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
Registration is closed. **Appservices** create accounts: agents never see Registration is closed. **The swarm's appservice** creates accounts: agents
an appservice token, and an agent only ever receives its own `access_token`. never see an appservice token, and an agent only ever receives its own
`access_token`.
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
The appservice has no URL (`url: null` in its registration), so the The appservice has no URL (`url: null` in its registration), so the
@ -116,48 +117,42 @@ no secret has to be equal on both sides of the wire, and why account
creation doesn't depend on registration being open to anyone who learns creation doesn't depend on registration being open to anyone who learns
a token. a token.
1. **System activation** mints a 32-byte random hex appservice token (64 1. **`swarm-matrix-ctl appservice render`**, inside the `hive-matrix`
chars) at `/var/lib/hyperhive/matrix-appservice-token` and its container and before tuwunel starts, mints the registration's
spec-required `hs_token` sibling, mode `0600 root:root`, then renders `as_token` and its spec-required `hs_token` when they're missing, and
the registration to renders the registration to
`/var/lib/hyperhive/matrix-appservice/hyperhive.yaml` (also `0600`). `/var/lib/swarm-matrix-appservice/swarm.yaml` (mode `0600 root:root`).
The script mints the tokens only when missing; the registration is 2. **systemd `LoadCredential=`** copies the registration into
re-rendered every time, because the token file can be overwritten in `/run/credentials/tuwunel.service/swarm-appservice.yaml`, owned by
place by the swarm secret store and a registration naming a stale tuwunel's dynamic user with mode `0400`, at service start. The source
token authenticates nobody. Runs at activation time, before any
container start, because the directory is bind-mounted and
nixos-container refuses to start when a bind source is missing.
2. **Read-only bind-mount** maps that directory into the tuwunel
container at the same path.
3. **systemd `LoadCredential=`** inside the container copies the
registration into
`/run/credentials/tuwunel.service/hyperhive-appservice.yaml`, owned by
tuwunel's dynamic user with mode `0400`, at service start. The host
file stays `root:root 0600` — no `chown :tuwunel` / `chmod 0640` / file stays `root:root 0600` — no `chown :tuwunel` / `chmod 0640` /
GID-pin gymnastics required. Keeps `DynamicUser = true` + GID-pin gymnastics required. Keeps `DynamicUser = true` +
`PrivateUsers = true` intact. `PrivateUsers = true` intact.
4. tuwunel's `appservice_dir` points at the credentials directory, not at 3. tuwunel's `appservice_dir` points at the credentials directory. It
the bind-mount path. It reads only `.yaml`/`.yml` entries from there, reads only `.yaml`/`.yml` entries from there, so the sibling
so the sibling credentials are invisible to it. The `.yaml` suffix on credentials are invisible to it. The `.yaml` suffix on the credential
the credential id is what makes this work. id is what makes this work.
5. **hive-c0re** doesn't read the appservice token and creates no 4. **hive-c0re** holds no appservice token and creates no account. Its
account. Its `@hive-<hive>:` token comes from the swarm store (below). `@hive-<hive>:` token comes from the swarm store (below).
6. **Agents' accounts aren't this hive's.** `swarm-controller` creates each 5. **`swarm-controller` creates every account**: each agent's, with its
one with the swarm's own appservice token (next section), stores its token stored at `swarm/agents/<agent>/matrix/main` for the agent's
token at `swarm/agents/<agent>/matrix/main`, and the agent's `hive-matrix-daemon` to read as the agent itself, and each hive's
`hive-matrix-daemon` reads it from there as the agent itself. sender account (below).
A homeserver whose `swarm-matrix-ctl` has no store identity
(`services.hyperhive.deploy.matrix.ctlBaoClientCertFile`) loads no
registration, and nothing creates accounts on it.
### The swarm's appservice, and its admin sender ### The swarm's appservice, and its admin sender
A second registration sits beside the hive's: id `swarm`, sender `@swarm`, The registration's id is `swarm`, its sender `@swarm`, and its user
the same non-exclusive namespace. It's the swarm's identity on the namespace non-exclusive. It's the swarm's identity on the homeserver:
homeserver, and `swarm-controller` creates every agent's account with it. `swarm-controller` creates every agent's account and every hive's sender
tuwunel loads every `.yaml` in `appservice_dir` and refuses only a account with it.
duplicate `id` or `as_token`, so the two overlapping namespaces coexist.
`swarm-matrix-ctl appservice render` mints its tokens inside the `swarm-matrix-ctl appservice render` mints its tokens inside the
`hive-matrix` container, before tuwunel starts, and renders the `hive-matrix` container, before tuwunel starts, and renders the
registration tuwunel loads as a second credential. `appservice publish` registration tuwunel loads as a credential. `appservice publish`
then writes its `as_token` to then writes its `as_token` to
`swarm/controller/swarm-controller/matrix/appservice-token`. No hive's `swarm/controller/swarm-controller/matrix/appservice-token`. No hive's
policy reaches that path: only matrix-ctl (which writes it) and policy reaches that path: only matrix-ctl (which writes it) and
@ -199,13 +194,12 @@ lands on.
`services.hyperhive.deploy.swarm-controller.authBridgeUrl` and `services.hyperhive.deploy.swarm-controller.authBridgeUrl` and
`services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl` set. `services.hyperhive.deploy.swarm-controller.matrixHomeserverUrl` set.
### The appservice's sender account, and why it isn't an admin ### A hive's sender account, and why it isn't an admin
`@hive-<hive>:<server_name>` is the appservice's own `sender_localpart`, `@hive-<hive>:<server_name>` is the account a hive acts as.
which the homeserver creates itself when it loads the registration — on a `swarm-controller` creates it with the swarm's appservice token, for every
zero-user database, inside startup, before the HTTP listener accepts hive in its directory. It's an **ordinary account**: nothing promotes it;
anything. It's an **ordinary account**: nothing promotes it; the the homeserver's `admin_execute` promotes only `@swarm`.
homeserver's `admin_execute` promotes only `@swarm`.
**One account per hive.** The localpart carries the hive's name, so a swarm **One account per hive.** The localpart carries the hive's name, so a swarm
whose hives share a homeserver gives each of them its own identity: the whose hives share a homeserver gives each of them its own identity: the
@ -220,8 +214,8 @@ appservice token and publishes it to `swarm/hives/<hive>/matrix/sender-token`, a
reads it from there under its own certificate. That path sits inside the reads it from there under its own certificate. That path sits inside the
hive's own read grant (`swarm/hives/<hive>/*`), so a hive fetches its own hive's own read grant (`swarm/hives/<hive>/*`), so a hive fetches its own
token and gets a refusal on any other hive's. The name says what it token and gets a refusal on any other hive's. The name says what it
authenticates as — an account the appservice registration brings into being authenticates as — an account the swarm's appservice creates — rather
— rather than any privilege level, because it carries none. than any privilege level, because it carries none.
It needs no promotion for what the hive does with it. Creating the hive It needs no promotion for what the hive does with it. Creating the hive
Space and the chat room, writing their hierarchy and join rules, and Space and the chat room, writing their hierarchy and join rules, and
@ -251,8 +245,8 @@ required.
<details><summary>Appservice identity</summary> <details><summary>Appservice identity</summary>
The activation script mints the appservice token and renders the `swarm-matrix-ctl appservice render` mints the appservice tokens and
registration before the homeserver restarts, so every boot has both renders the registration before tuwunel starts, so every boot has both
halves. halves.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
@ -261,11 +255,10 @@ halves.
too. too.
- **The homeserver's `admin_execute` promotes only `@swarm` at boot** - **The homeserver's `admin_execute` promotes only `@swarm` at boot**
(above); a hive's sender account is never promoted. (above); a hive's sender account is never promoted.
- **The value that matters lives at `matrix/appservice-token`.** - **The value that matters lives at
`swarm-secret-publish` on the authelia host mints and `put`s it `swarm/controller/swarm-controller/matrix/appservice-token`.**
there; the hive uses its locally minted token only until the first `appservice publish` writes the container's copy there whenever the
successful read. See [`../swarm/secrets.md`](../swarm/secrets.md) store's differs, and `swarm-controller` reads it from there.
for how that mint stays idempotent across runs.
<!-- vale write-good.Passive = YES --> <!-- vale write-good.Passive = YES -->
</details> </details>
@ -277,8 +270,8 @@ Initial rollout settings:
restart. `trusted_servers = []` keeps it effectively closed restart. `trusted_servers = []` keeps it effectively closed
until you list peers. until you list peers.
- `allow_registration = false`. tuwunel checks this flag only for - `allow_registration = false`. tuwunel checks this flag only for
requests that arrive **without** an appservice token, so the appservices requests that arrive **without** an appservice token, so the swarm's
still create accounts and tuwunel refuses everyone else. With no appservice still creates accounts and tuwunel refuses everyone else. With no
registration token configured, `allow_registration = true` makes registration token configured, `allow_registration = true` makes
tuwunel refuse to start unless tuwunel refuse to start unless
`yes_i_am_very_very_sure_…_open_registration_…` is also set. `yes_i_am_very_very_sure_…_open_registration_…` is also set.

View file

@ -66,7 +66,6 @@ of the cell says how.
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer | | `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer | | `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/hives/<hive>/matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated | must be stated |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs | | `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated | | `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate; the bao UI's by the store's granter | must be stated | must be stated | | `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate; the bao UI's by the store's granter | must be stated | must be stated |

View file

@ -90,8 +90,7 @@ with nothing to authenticate with still receives every hive's telemetry —
only its own pushes to the stores go out unauthenticated and get refused only its own pushes to the stores go out unauthenticated and get refused
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice `deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for `glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix`
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
uses for its non-optional one. The operator can still deliver the secret by uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches. collector this unit never reaches.
@ -159,7 +158,6 @@ when the system builds. The server names the offending file and refuses to run.
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` | | hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` | | the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container | | the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` | | the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) | | the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
@ -169,50 +167,7 @@ bounded wait, 120s — and then **fail loudly** rather than skipping. A silent s
produces a service whose login button always fails, which is a symptom many produces a service whose login button always fails, which is a symptom many
layers from its cause. layers from its cause.
The store's **first reader** is the matrix appservice token, and it's worth The store's **first reader** is the agent containers' queue credential:
saying why that one: it's an opaque 32-byte value with no second file and no
format. Authelia's OIDC secret needs a `.secret` _and_ a matching `.digest`, so
starting there would have meant debugging "can a reader authenticate and get
bytes back" and "is authelia's file format right" at once, with an
SSO outage as the failure mode.
`glue-matrix-bao-token.nix` fetches it, writes the file `hive-matrix.nix`
already reads, and then runs that module's own renderer to re-stamp the
appservice registration naming the token — so the homeserver never learns the
store exists. The re-render isn't housekeeping: the token is half an
agreement, and a registration carrying the previous value authenticates
nobody. Every failure path — no such key, sealed store, unreachable store,
empty value — leaves the locally minted token in place, so a hive with no store
behaves exactly as it did before.
The store path is `swarm/hives/<hive>/matrix/appservice-token`, and the
**producer is `swarm-secret-publish`** — the same unit that copies authelia's
OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s
it there, so the store is the source of truth and every hive converges on the
value it holds. The hive's own activation mint is still there, but it's a
first-boot fallback now: it fires only when the token file is absent, and the
reader overwrites whatever it produced.
The mint is **idempotent by keeping its own record**, not by asking the store.
The publisher's grant is `create`/`update` under `swarm/hives/*` with no
`read` — write-only on purpose, so a file-copier can't recover every hive's
credentials — which means it can't check whether a hive already has a token.
Instead it keeps the value it minted under its `StateDirectory`,
`/var/lib/swarm-secret-publish/matrix-appservice-token/<hive>` (`0700` dir,
`0600` file), and mints only when that file is missing or empty. The `put`
itself runs every time: re-putting the same bytes changes nothing for any
reader, while a mint whose publish failed must not leave behind a token
this host holds and no hive can reach.
Lose the state directory and the next run mints once more and republishes.
That rotates the token, which readers pick up on their next start — nothing
that already registered breaks, because the token authenticates the
appservice rather than living inside any account it created.
The path was `…/matrix/registration-token` while the homeserver still took a
shared registration secret; nothing reads a value left at the old path.
The **second reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and `glue-queue-agent-credential.nix` lands it as two files, the client secret and
the client id it authenticates, because that's how a queue client reads them — the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is the secret by path, the id by value. No local fallback exists, and none is
@ -237,7 +192,7 @@ never reads the bytes at any point: it runs as `hive-core`, which is the reason
a copy wasn't an option either. When the files aren't there the daemon forwards a copy wasn't an option either. When the files aren't there the daemon forwards
nothing and says so in its journal, and the container starts without them. nothing and says so in its journal, and the container starts without them.
The **third reader** is Grafana's OIDC client secret, and it's the first one The **second reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly that's a _swarm service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host `swarm/services/<client-id>/oidc/client` out of the store, on **every** host
@ -265,7 +220,7 @@ nothing said anything. Registration is separate and stays where authelia is
so it's declared on the host running authelia, which isn't necessarily the host so it's declared on the host running authelia, which isn't necessarily the host
running Grafana. running Grafana.
The **fourth reader** is the swarm collector's own OIDC client secret, the The **third reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that `swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in runs the collector, and registration lives separately in
@ -275,13 +230,12 @@ telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana `deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it does — it
doesn't render the reading unit without them, the shape doesn't render the reading unit without them, the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for `glue-queue-agent-credential.nix` uses for its own read.
their own optional reads.
The **fifth reader** is the secret store forwarder's OIDC client secret: The **fourth reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by `swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the `glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
fourth in both directions. It renders unconditionally, because the export it third in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it container it delivers into. Nothing circular sits behind that: the identity it
@ -340,59 +294,55 @@ next controller restart.
### Per-principal identities ### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. These five units read one path identity and sharing one merges the identities. These four units read one path
each and each holds a leaf, a role and a policy of its own: each and each holds a leaf, a role and a policy of its own:
| unit | option pair under `deploy.bao.` | reads | | unit | option pair under `deploy.bao.` | reads |
| -------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- | | -------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- |
| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives/<hive>/matrix/appservice-token` |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` | | `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` | | `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` | | `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` | | `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
The first two exist **per hive**, because the path they read carries a hive The first exists **per hive**, because the path it reads carries a hive name
name and every hive runs its own reader. Their subjects are and every hive runs its own reader. Its subject is
`<deploy.bao.matrixTokenCommonNamePrefix>-<hive>` and `<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves both composed spelling as a hive name, so nobody can name a hive into another hive's
composed spellings as hive names, so nobody can name a hive into another hive's
role. The other three read a path that names a swarm service rather than a hive, role. The other three read a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName` and `deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName` and
`deploy.bao.forwarderOidcCommonName`. `deploy.bao.forwarderOidcCommonName`.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all five and defaults On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults
all ten options, and there is nothing to do. Elsewhere you issue each leaf from all eight options, and there is nothing to do. Elsewhere you issue each leaf from
that CA out of band and name it here — one file per principal rather than one file that CA out of band and name it here — one file per principal rather than one file
shared by five, which is the whole of what this buys. shared by four, which is the whole of what this buys.
On a host without the store, where the fifth never runs, forgetting one of the On a host without the store, where the forwarder never runs, forgetting one of
other eight isn't a quiet degrade. Three of the four the other six isn't a quiet degrade. A hand-configured remote-store hive that
readers used to render only where their leaf existed, so a hand-configured names the hive's own `clientCertFile` and misses a principal's pair has the
remote-store hive that named the hive's own `clientCertFile` and missed a build refused, by the module of the reader it missed, naming that pair — see
principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing
the build. All four now refuse it, each naming its own option pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below. [what a missing leaf costs](#what-a-missing-leaf-costs) below.
### What a missing leaf costs ### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host Each of the three modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**. `clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without The refusal names the two options, so an operator acts on the message without
opening the nix. opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive That condition fires on exactly one shape, the hand-configured remote-store hive
that named seven of the eight. It stays silent everywhere else, and each half of that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own: that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so - **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should the one an operator passes through while bringing a hive up — neither should
fail to evaluate. fail to evaluate.
- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as - **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse. defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no homeserver, no collector, no Grafana — has no - **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false. Only the queue-credential reader unit to skip, so the second clause is false. Only the queue-credential reader
has no toggle of its own to check: every hive runs one for its own agents, so has no toggle of its own to check: every hive runs one for its own agents, so
reading the store at all is what asks for it. reading the store at all is what asks for it.

View file

@ -58,13 +58,9 @@ const HTTP_TIMEOUT_SECS: u64 = 10;
/// homeserver could not attribute an action to a hive and revoking one hive's /// homeserver could not attribute an action to a hive and revoking one hive's
/// standing revoked all of them. /// standing revoked all of them.
/// ///
/// Also the `sender_localpart` of the hive's appservice registration, /// `swarm-controller` creates this account with the swarm's appservice token
/// which is what creates this account on a homeserver that has never had /// (`matrix_account::hive_sender`) and stores its token where
/// one: the homeserver creates a registration's sender user itself, at /// [`stored_sender_token`] reads it; hive-c0re never creates it.
/// startup, before it accepts a request. See
/// `nix/host-modules/hive-matrix.nix`, where the same string is rendered as
/// `hiveLocalpart`. **The two must match**; nothing wires an override
/// across.
/// ///
/// An ordinary account, with no homeserver-admin standing: what it /// An ordinary account, with no homeserver-admin standing: what it
/// provisions — the Space, the chat room, the invites — it provisions as /// provisions — the Space, the chat room, the invites — it provisions as

View file

@ -33,7 +33,6 @@
./glue-forge-oidc-client.nix ./glue-forge-oidc-client.nix
./glue-grafana-oidc-client.nix ./glue-grafana-oidc-client.nix
./glue-hive-tls-bao-identity.nix ./glue-hive-tls-bao-identity.nix
./glue-matrix-bao-token.nix
./glue-matrix-ctl-bao-identity.nix ./glue-matrix-ctl-bao-identity.nix
./glue-nats-auth-bao-identity.nix ./glue-nats-auth-bao-identity.nix
./glue-nats-bao-identity.nix ./glue-nats-bao-identity.nix

View file

@ -257,23 +257,18 @@ in
# Both spellings of the registration token, removed rather than renamed: # Both spellings of the registration token, removed rather than renamed:
# the homeserver no longer accepts a shared registration secret at all, # the homeserver no longer accepts a shared registration secret at all,
# so there is no path to point a definition at. Accounts are created by # so there is no path to point a definition at. Accounts are created by
# the hive's appservice, whose token lives at a fixed path this module # the swarm's appservice.
# deliberately does not let anyone move.
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "matrix" "registrationTokenFile" ] '' (lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "matrix" "registrationTokenFile" ] ''
The matrix registration token is gone: hive-c0re creates accounts as the The matrix registration token is gone: swarm-controller creates accounts
hive's appservice instead, so there is no shared secret for tuwunel to as the swarm's appservice instead, so there is no shared secret for
check. Remove this definition — nothing needs replacing, the appservice tuwunel to check. Remove this definition — nothing needs replacing.
token is minted and delivered automatically
(/var/lib/hyperhive/matrix-appservice-token, not operator-settable).
The old token file at /var/lib/hyperhive/matrix-register-token is read by The old token file at /var/lib/hyperhive/matrix-register-token is read by
nothing now and can be deleted. See docs/integrations/matrix.md. nothing now and can be deleted. See docs/integrations/matrix.md.
'') '')
(lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "matrix" "registrationTokenFile" ] '' (lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "matrix" "registrationTokenFile" ] ''
The matrix registration token is gone: hive-c0re creates accounts as the The matrix registration token is gone: swarm-controller creates accounts
hive's appservice instead, so there is no shared secret for tuwunel to as the swarm's appservice instead, so there is no shared secret for
check. Remove this definition — nothing needs replacing, the appservice tuwunel to check. Remove this definition — nothing needs replacing.
token is minted and delivered automatically
(/var/lib/hyperhive/matrix-appservice-token, not operator-settable).
The old token file at /var/lib/hyperhive/matrix-register-token is read by The old token file at /var/lib/hyperhive/matrix-register-token is read by
nothing now and can be deleted. See docs/integrations/matrix.md. nothing now and can be deleted. See docs/integrations/matrix.md.
'') '')

View file

@ -30,10 +30,6 @@ let
# Reader unit → the gate its own module defines it under. # Reader unit → the gate its own module defines it under.
readersHere = { readersHere = {
# ./glue-matrix-bao-token.nix
swarm-bao-matrix-token =
havePair baoDeploy.matrixTokenClientCertFile baoDeploy.matrixTokenClientKeyFile
&& deployCfg.matrix.enable;
# ./glue-queue-agent-credential.nix # ./glue-queue-agent-credential.nix
swarm-bao-queue-agent = swarm-bao-queue-agent =
deployCfg.hive-controller.enable deployCfg.hive-controller.enable

View file

@ -86,26 +86,23 @@ in
clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem"; clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem";
serverCaFile = lib.mkDefault "${pkiDir}/ca.pem"; serverCaFile = lib.mkDefault "${pkiDir}/ca.pem";
# 🩸 Four readers that used to present `client.pem` above, each now # 🩸 Readers that used to present `client.pem` above, each now pointed at
# pointed at a leaf of its own. The leaf is what bao sees, so this pairing # a leaf of its own. The leaf is what bao sees, so this pairing is the
# is the whole of what turns "one principal with the union of four grants" # whole of what turns "one principal with the union of their grants" into
# into four principals with one grant each — see ./swarm-bao.nix's # one principal per grant — see ./swarm-bao.nix's `perHiveReaders` block
# `perHiveReaders` block for the grants themselves. # for the grants themselves.
# #
# Defaulted here rather than in four `glue-<consumer>-bao-identity.nix` # Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files:
# files: those exist where the consumer is a CONTAINER with a # those exist where the consumer is a CONTAINER with a
# `deploy.<service>.*` namespace of its own to point at. These four are # `deploy.<service>.*` namespace of its own to point at. These are host
# host units reading the store, which is the pairing this file already is. # units reading the store, which is the pairing this file already is.
matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem";
matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem";
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem"; queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem"; queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem"; grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem"; grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem"; otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem"; otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem";
# The fifth, the store forwarder's: the reader the four-way split left # The store forwarder's.
# on `client.pem`.
forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem"; forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem";
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem"; forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
@ -196,26 +193,23 @@ in
[ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \ [ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \
${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth ${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth
# 🩸 The four readers that used to present `client.pem` above. Each # 🩸 The readers that used to present `client.pem` above. Each carries
# carries its own subject, which is the entire mechanism: bao matches a # its own subject, which is the entire mechanism: bao matches a
# cert-auth role on the CN, so four units sharing one leaf were one # cert-auth role on the CN, so units sharing one leaf were one principal
# principal holding the union of four grants, and the union included # holding the union of their grants, and the union included every agent
# every agent credential in the swarm. # credential in the swarm.
# #
# ⚠️ The first two carry THIS hive's name in the subject, unlike the # ⚠️ The first carries THIS hive's name in the subject, unlike the three
# three service leaves above. Their grants name one hive's path, because # service leaves above. Its grant names one hive's path, because a queue
# a matrix token and a queue credential live under `swarm/hives/<name>/` # credential lives under `swarm/hives/<name>/` and every hive runs a
# and every hive runs a reader for its own; ./swarm-bao.nix writes one # reader for its own; ./swarm-bao.nix writes one role per hive in the
# role per hive in the swarm directory to match. The two OIDC readers # swarm directory to match. The OIDC readers need no such segment — a
# need no such segment — a client is registered once per swarm. # client is registered once per swarm.
# #
# Minted whether or not the consumer runs here, for the reason the three # Minted whether or not the consumer runs here, for the reason the three
# leaves above give: on a hive that does not run the store this is the # leaves above give: on a hive that does not run the store this is the
# file an operator copies, and a leaf that only appears where its # file an operator copies, and a leaf that only appears where its
# consumer does is one nobody can copy from anywhere. # consumer does is one nobody can copy from anywhere.
[ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \
${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \ [ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth ${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
@ -265,8 +259,8 @@ in
${lib.optionalString deployCfg.hive-controller.enable '' ${lib.optionalString deployCfg.hive-controller.enable ''
# This hive's gateway-leaf identity, which opens # This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its # `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
# subject carries this hive's name, like the matrix-token and # subject carries this hive's name, like the queue-agent leaf
# queue-agent leaves above, because ./swarm-bao.nix writes one role # above, because ./swarm-bao.nix writes one role
# per hive. Only where a hive runs; hives on other hosts are issued # per hive. Only where a hive runs; hives on other hosts are issued
# theirs out of band. # theirs out of band.
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \ [ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \

View file

@ -1,256 +0,0 @@
# Glue: the matrix appservice token comes from the secret store.
#
# The store's first reader, and deliberately a small one. It fetches an opaque
# 32-byte value and writes it where ./hive-matrix.nix already looks, then asks
# that module's own renderer to re-stamp the appservice registration naming it
# — the homeserver never learns the store exists, and its config is unchanged.
#
# ⚠️ Why this credential first. It has no second file and no format: authelia's
# OIDC secret needs a `.secret` *and* a matching `.digest`, so shipping that
# one first would debug "can a reader authenticate and get bytes back" and
# "did we write authelia's file format right" at the same time, with an SSO
# outage as the failure mode. Here the failure is narrow — new agent accounts
# cannot be provisioned, existing ones are untouched, nothing crash-loops.
#
# ⚠️ The fallback is today's behaviour, not a new one. `hive-matrix.nix`'s
# activation script still mints a token when the file is absent; this unit
# overwrites it with the swarm's copy when the store has one. A store that is
# empty or unreachable leaves a working hive with a local token.
#
# 📌 This runs wherever a client identity is configured, NOT only where the
# store is. `deploy.bao.enable` would have been the co-location assumption
# itself; the reader needs a certificate, not a neighbour. On the store's own
# host ./glue-bao-tls.nix supplies one as a `mkDefault` and nothing changes;
# elsewhere an operator places the leaf and names it, and the same unit works.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
#
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four
# units used to share. Bao matches a cert-auth role on the CN, so one leaf for
# four readers was ONE principal holding the union of four grants: read on
# `swarm/agents/*` AND `swarm/hives/<hive>/*` AND `swarm/services/*`, when
# this unit reads one appservice token and nothing else. Its own leaf carries
# `<matrixTokenCommonNamePrefix>-<hive>` and its role grants the single path
# below.
haveClientIdentity =
baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has always had to place by hand. Only
# used to decide whether a missing per-principal leaf is a mistake or a
# deployment that has no store: a host holding neither is the supported
# no-store shape, and one holding this pair but not the pair above named
# seven of the eight options and stopped.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the token lives in the store. A path, not a convention to guess at:
# whoever writes it and whoever reads it must agree, and the agreement
# belongs in one visible place.
#
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
# own grant covers. The store's read policy grants `swarm/agents/*` and
# `swarm/hives/<this hive>/*` and nothing else, so a path outside those is a
# 403 rather than a miss, however correct it looks. `swarm-secret-client`'s
# `matrix::appservice_token_path` builds the same string from the same
# pieces; this literal is the nix half of that one agreement.
#
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
# its own use of it: it is asserted set on every host that runs the
# homeserver.
tokenPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/matrix/appservice-token";
# A literal, not an option — ./hive-matrix.nix names its container
# `containers.hive-matrix` directly and declares no `machine` to derive it
# from, which the trust-bundle call in that file already says out loud.
# ⚠️ A `swarm.matrix.machine` read parses fine and fails at module-system
# resolution, so this is the kind of mistake only reading the target module
# catches.
matrixMachine = "hive-matrix";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
config = lib.mkMerge [
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
# assertion written inside it could never be reached in the state it
# exists to report.
#
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
# same refusal, named to this principal's own pair. What differs is the
# gate. Grafana asserts wherever Grafana runs, because a Grafana with no
# store identity has no way in at all; a homeserver with no store identity
# is a hive that has no store, which is supported. So this one additionally
# requires `hiveReaderIdentity`: the host demonstrably reads the store, and
# named every option but this pair.
(lib.mkIf (deployCfg.matrix.enable && hiveReaderIdentity) {
assertions = [
{
assertion = haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set) and runs a homeserver, so it needs the matrix appservice
token reader's own client identity: set both
services.hyperhive.deploy.bao.matrixTokenClientCertFile
services.hyperhive.deploy.bao.matrixTokenClientKeyFile
swarm-bao-matrix-token.service fetches this hive's appservice token
out of the store, and without these it is not rendered at all —
leaving the homeserver authenticating hive-c0re against whatever is
already on disk, which is a 401 on every request naming nothing.
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is
the hive's, and its grant reads every secret in the store; this role
reads the one appservice-token path. Pointing this option at the
hive's leaf would evaluate, deploy and log in — and undo the split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/secrets.md.
'';
}
];
})
(lib.mkIf (haveClientIdentity && deployCfg.matrix.enable) {
systemd.services.swarm-bao-matrix-token = {
description = "fetch the matrix appservice token from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering is
# conditional even though the read is not: off-host there is nothing local
# to wait for, and the timeout below is what bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${matrixMachine}.service" ];
wantedBy = [ "container@${matrixMachine}.service" ];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. This unit is `Before=` the homeserver's
# container, which waits for one attempt and then starts on the token it
# already has; a token a later attempt lands restarts the homeserver
# (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds each attempt below. Stated here rather than
# left to systemd's default, so the number a boot waits on is in
# the file that waits.
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile;
BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
${refreshConsumer}
# A sealed or uninitialised store answers on the port and never
# answers the read, so "the store is up" is not the same as "the
# store can answer". `TimeoutStartSec` above bounds each attempt and
# a timed-out attempt is retried like any other failure; the
# homeserver only `Wants=` this unit, so a failed attempt lets it
# start on the local token rather than failing it.
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one
# rather than asserting all three in a sentence of ours — a reader
# that cannot say why it read nothing is indistinguishable from a
# broken one.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the read below: the three states a login failure
# covers — store not up, sealed, role not written yet — are all things
# a retry fixes, and `Restart=on-failure` above is what retries. Exiting
# 0 here spends the whole boot on a condition that was seconds old.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then
echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$token" ]; then
echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2
exit 0
fi
if secret_differs ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} "$token"; then
atomic_write_secret 0600 "" ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} "$token"
fi
# Re-stamp the registration file from the token on disk. The
# token is half an agreement — the registration the homeserver loads
# has to carry the same value — so writing the file and stopping
# would leave the homeserver authenticating hive-c0re against
# whatever activation put there: a 401 on every request, naming
# nothing. Unconditional rather than on-change, because this unit
# has no way to know what the registration currently says.
#
# hive-matrix's own renderer rather than a `printf` here, so the
# registration's shape has one home.
${deployCfg.matrix.appserviceRegistrationScript}
# tuwunel loads the registration through `LoadCredential`, a copy
# taken at start, while hive-c0re reads the token file on every call:
# a changed token splits the two until the homeserver restarts.
refresh_consumer ${lib.escapeShellArg matrixMachine} tuwunel.service ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
'';
};
})
];
}

View file

@ -1,9 +1,8 @@
# Glue: this hive's agent queue credential comes out of the secret store. # Glue: this hive's agent queue credential comes out of the secret store.
# #
# The store's second reader, and deliberately the same shape as its first # A cert login that fails LOUDLY because every state it fails on is one a
# (./glue-matrix-bao-token.nix): a cert login that fails LOUDLY because every # retry fixes, then a read that degrades QUIETLY because no retry turns "no
# state it fails on is one a retry fixes, then a read that degrades QUIETLY # value there" into a value.
# because no retry turns "no value there" into a value.
# #
# ⚠️ Two files, not one, and that is the consumer's shape rather than a # ⚠️ Two files, not one, and that is the consumer's shape rather than a
# preference. `swarm_queue_client::QueueConfig::from_env` takes the secret as # preference. `swarm_queue_client::QueueConfig::from_env` takes the secret as
@ -15,15 +14,17 @@
# that is correct rather than degraded. The publisher runs on the authelia # that is correct rather than degraded. The publisher runs on the authelia
# host and authelia mints on its first boot, so "nothing at that path yet" is # host and authelia mints on its first boot, so "nothing at that path yet" is
# the ordinary early state of a swarm. Nothing here writes a local stand-in: # the ordinary early state of a swarm. Nothing here writes a local stand-in:
# unlike a matrix appservice token there is no such thing as a locally valid # there is no such thing as a locally valid OIDC client secret, so a placeholder would turn a hive that cannot connect
# OIDC client secret, so a placeholder would turn a hive that cannot connect
# into one that is refused, which reaches the agent as a timeout. # into one that is refused, which reaches the agent as a timeout.
# #
# 📌 This runs wherever a client identity is configured, NOT only where the # 📌 This runs wherever a client identity is configured, NOT only where the
# store is — the rule ./glue-matrix-bao-token.nix states in full — and # store is. `deploy.bao.enable` would have been the co-location assumption
# wherever this host runs a hive (`deploy.hive-controller.enable`): the # itself; the reader needs a certificate, not a neighbour. On the store's own
# credential is for the agents hive-c0re creates here at runtime, and the unit # host ./glue-bao-tls.nix supplies one as a `mkDefault`; elsewhere an operator
# writes under hive-c0re's own state directory. # places the leaf and names it, and the same unit works. It also needs this
# host to run a hive (`deploy.hive-controller.enable`): the credential is for
# the agents hive-c0re creates here at runtime, and the unit writes under
# hive-c0re's own state directory.
{ {
pkgs, pkgs,
lib, lib,
@ -55,8 +56,8 @@ let
# one thing a remote-store deployment has always had to place by hand. Only # one thing a remote-store deployment has always had to place by hand. Only
# used to decide whether a missing per-principal leaf is a mistake or a # used to decide whether a missing per-principal leaf is a mistake or a
# deployment that has no store: a host holding neither is the supported # deployment that has no store: a host holding neither is the supported
# no-store shape, and one holding this pair but not the pair above named # no-store shape, and one holding this pair but not the pair above stopped
# seven of the eight options and stopped. # short of naming this reader's own.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null; hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir; credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;

View file

@ -333,8 +333,8 @@ in
# reachable from an unprivileged daemon without widening either. # reachable from an unprivileged daemon without widening either.
# The gate is the identity, never `deploy.bao.enable`: a hive that # The gate is the identity, never `deploy.bao.enable`: a hive that
# reads a store on another machine holds one of these and runs no # reads a store on another machine holds one of these and runs no
# store. ./glue-matrix-bao-token.nix gates its own reader the same # store. ./glue-queue-agent-credential.nix gates its own reader the
# way. # same way.
++ lib.optionals baoClientIdentity [ ++ lib.optionals baoClientIdentity [
"bao-client.pem:${baoDeploy.clientCertFile}" "bao-client.pem:${baoDeploy.clientCertFile}"
"bao-client-key.pem:${baoDeploy.clientKeyFile}" "bao-client-key.pem:${baoDeploy.clientKeyFile}"

View file

@ -11,7 +11,7 @@ let
cfg = config.services.hyperhive.c0re; cfg = config.services.hyperhive.c0re;
baoDeploy = config.services.hyperhive.deploy.bao; baoDeploy = config.services.hyperhive.deploy.bao;
baoCfg = config.services.hyperhive.swarm.bao; baoCfg = config.services.hyperhive.swarm.bao;
# The same test ../glue-matrix-bao-token.nix applies, and for the same # The same test ../glue-queue-agent-credential.nix applies, and for the same
# reason: a reader is defined by holding a certificate the store accepts, # reason: a reader is defined by holding a certificate the store accepts,
# not by sharing a host with the store. Deriving this from # not by sharing a host with the store. Deriving this from
# `deploy.bao.enable` would be the co-location assumption itself. # `deploy.bao.enable` would be the co-location assumption itself.

View file

@ -50,79 +50,20 @@ let
# visible to the service after sandboxing (ReadWritePaths / ProtectHome), # visible to the service after sandboxing (ReadWritePaths / ProtectHome),
# typically by placing the file under /etc/tuwunel/" — which this # typically by placing the file under /etc/tuwunel/" — which this
# container has no writable etc for. `LoadCredential` is the answer # container has no writable etc for. `LoadCredential` is the answer
# already in use for the appservice registration below, and for the # already in use for the swarm appservice registration below, and for the
# same reason: it keeps `DynamicUser=true` + `PrivateUsers=true` intact # same reason: it keeps `DynamicUser=true` + `PrivateUsers=true` intact
# with no host-side chown or GID pinning. # with no host-side chown or GID pinning.
matrixSecretCredential = "/run/credentials/tuwunel.service/oidc_client_secret"; matrixSecretCredential = "/run/credentials/tuwunel.service/oidc_client_secret";
# The hive's appservice registration, whose `url` is null. Null is a legal # Where tuwunel reads appservice registrations from: its own credentials
# `url` (ruma's `Registration` types it `Option<String>`), and it is the # directory. A registration is a 0600 root-owned file and the homeserver
# whole point — with no URL the homeserver # runs under `DynamicUser=true`, so `LoadCredential` copies it to a 0400
# never calls out, so there is no HTTP service to run and no daemon to # dynamic-user-owned path as root, before the sandbox and that user exist.
# operate. tuwunel loads the registration and creates its sender account
# (below). No client presents its `as_token`: `swarm-controller` mints the
# sender's token with the swarm registration's.
appserviceId = "hyperhive";
# The appservice's own user, and the account the hive acts as. An
# ordinary user, not a homeserver admin: everything the hive does with
# it — the Space, the chat room, the invites — it does as the creator of
# those rooms. Loading a registration CREATES its `sender_localpart` user
# when absent (tuwunel `src/service/appservice/mod.rs`), on a zero-user
# database, inside `Services::start()` — before the HTTP listener accepts
# anything, so the account exists on the very first boot of a fresh
# homeserver without hive-c0re racing to register it.
# #
# 🩸 **Derived from the hive name, so it is one account per hive.** It was # Safe rather than clever: tuwunel's loader takes only `.yaml`/`.yml`
# the bare `hive` and a swarm runs one homeserver: every hive on it acted
# as the same `@hive:`, which is one identity with no attribution and no
# way to revoke a single hive. The hives that do not run this container
# register their own `hive-<name>` through the appservice namespace below,
# which already admits it.
#
# ⚠️ Must equal `swarm_secret_client::matrix::hive_localpart`, which
# hive-c0re and swarm-controller both derive from independently with
# nothing wiring an override across — same agreement, and same reason for
# saying so, as the token path below.
#
# `hiveName` has no fallback here for the reason ./glue-matrix-bao-token.nix
# gives at its own use of it: it is asserted set on every host that runs the
# homeserver.
hiveLocalpart = "hive-${toString config.services.hyperhive.hiveName}";
# The `as_token`, and the `hs_token` the spec requires alongside it. Both
# minted by the render script below, mode 0600; the `as_token` is the one
# the swarm secret store overwrites (see
# `glue-matrix-bao-token.nix`). The `hs_token` authenticates the homeserver
# TO the appservice, which with `url = null` is nobody — it exists because
# the registration format requires it.
#
# `appserviceTokenPath` is the single source for the option's `default`,
# its `config`-level `mkDefault` self-definition, and the assertion that
# rejects a moved path. See `appserviceTokenFile`'s own comment below.
appserviceTokenPath = "/var/lib/hyperhive/matrix-appservice-token";
appserviceHsTokenPath = "/var/lib/hyperhive/matrix-appservice-hs-token";
# The registration file, and the directory that holds it. A directory
# rather than a `[global.appservice.<id>]` stanza because a stanza's
# `as_token` would be a nix literal, and a nix literal is a world-readable
# store path — the same rule that sends the OIDC client secret through a
# runtime file. `docs/swarm/secrets.md` has the general form of it.
appserviceDir = "/var/lib/hyperhive/matrix-appservice";
appserviceRegistrationPath = "${appserviceDir}/${appserviceId}.yaml";
# ⚠️ tuwunel does not read the host path directly, for exactly the reason
# given for the two secrets above: the file is 0600 root-owned and the
# homeserver runs under `DynamicUser=true`. `LoadCredential` copies it to a
# 0400 dynamic-user-owned path as root, before the sandbox and that user
# exist.
#
# So `appservice_dir` points at the credentials directory ITSELF. That is
# safe rather than clever: tuwunel's loader takes only `.yaml`/`.yml`
# entries from the directory and skips every other file, so the sibling # entries from the directory and skips every other file, so the sibling
# credentials (the OIDC secret) are invisible to it. A credential id is a # credentials (the OIDC secret) are invisible to it. A credential id is a
# free-form filename, which is what lets ours end in `.yaml`. # free-form filename, which is what lets a registration's end in `.yaml`.
appserviceCredentialId = "${appserviceId}-appservice.yaml";
appserviceCredentialDir = "/run/credentials/tuwunel.service"; appserviceCredentialDir = "/run/credentials/tuwunel.service";
# ── swarm-matrix-ctl ──────────────────────────────────────────────── # ── swarm-matrix-ctl ────────────────────────────────────────────────
@ -147,9 +88,8 @@ let
# derived rather than assumed, because an operator naming paths by hand is # derived rather than assumed, because an operator naming paths by hand is
# exactly the deployment the gate above is written for. # exactly the deployment the gate above is written for.
# #
# Directories rather than the files, for the reason `appserviceDir`'s own # Directories rather than the files: a re-issued leaf is a new inode, and
# comment gives: a re-issued leaf is a new inode, and binding the file would # binding the file would pin the one the container saw when it started.
# pin the one the container saw when it started.
ctlPkiDirs = lib.optionals ctlActive ( ctlPkiDirs = lib.optionals ctlActive (
lib.unique ( lib.unique (
map builtins.dirOf ( map builtins.dirOf (
@ -169,9 +109,9 @@ let
# ── the swarm's own appservice ────────────────────────────────────── # ── the swarm's own appservice ──────────────────────────────────────
# #
# A second registration beside the hive's `hyperhive` one, and the swarm's # The homeserver's one appservice registration, and the swarm's identity on
# identity on this homeserver: `swarm-controller` creates every agent's # it: `swarm-controller` creates every agent's account and every hive's
# account with its token. Minted INSIDE this container by # `@hive-<hive>` sender with its token. Minted INSIDE this container by
# `swarm-matrix-ctl appservice render` and published to the store by # `swarm-matrix-ctl appservice render` and published to the store by
# `appservice publish`, which is why it is gated on matrix-ctl's store # `appservice publish`, which is why it is gated on matrix-ctl's store
# identity: with nobody to publish it, a registration here would be an # identity: with nobody to publish it, a registration here would be an
@ -187,8 +127,8 @@ let
swarmAppserviceDir = "/var/lib/swarm-matrix-appservice"; swarmAppserviceDir = "/var/lib/swarm-matrix-appservice";
swarmAppserviceCredentialId = "swarm-appservice.yaml"; swarmAppserviceCredentialId = "swarm-appservice.yaml";
# Every local user this hive may provision — agents and `@hive-<hive>:` # Every local user the swarm appservice may create — agents and every
# itself — which is the whole matrix localpart charset. # hive's `@hive-<hive>:` sender — which is the whole matrix localpart charset.
# #
# ⚠️ Anchored deliberately: tuwunel compiles a namespace into a `RegexSet` # ⚠️ Anchored deliberately: tuwunel compiles a namespace into a `RegexSet`
# and asks it for a MATCH, not a full match, so an unanchored # and asks it for a MATCH, not a full match, so an unanchored
@ -201,73 +141,6 @@ let
# create these names, not a monopoly on them. # create these names, not a monopoly on them.
appserviceUserRegex = "^@[a-z0-9._=/-]+:${lib.escapeRegex effectiveServerName}$"; appserviceUserRegex = "^@[a-z0-9._=/-]+:${lib.escapeRegex effectiveServerName}$";
# Mint the tokens if they are absent, then render the registration from
# whatever they now hold. **One script with two callers** — the activation
# script below and `glue-matrix-bao-token.nix`, which overwrites the
# `as_token` with the swarm store's copy and has to re-render afterwards.
#
# ⚠️ The `as_token` mint here is the FIRST-BOOT FALLBACK, not the normal
# route. `swarm-secret-publisher.nix` mints one per hive swarm-side and
# publishes it to the store; the reader above overwrites this file before
# the homeserver's container starts, so what a hive actually runs on is the
# swarm's value. This keeps a hive that has never reached the store working
# — and it fires only when the file is absent, so it never clobbers one the
# store delivered. (`hs_token` has no swarm half: it is how the homeserver
# authenticates to *us*, so it stays hive-local and is minted here for real.)
# A second copy of the registration's shape would be a second source of
# truth for a format whose mismatch is silent: the homeserver would load a
# registration naming a token nobody holds, and every request hive-c0re
# makes would come back 401 naming nothing.
#
# Re-rendering unconditionally is the point rather than thoroughness: "the
# token file exists" does not mean "the registration carries what is in
# it".
appserviceRegistrationScript = pkgs.writeShellApplication {
name = "hive-matrix-appservice-registration";
runtimeInputs = [ pkgs.coreutils ];
text = ''
# Both the tokens and the rendered registration are secrets; 077
# covers every file this script creates rather than each one
# separately.
umask 077
mkdir -p ${lib.escapeShellArg appserviceDir}
for f in ${lib.escapeShellArg appserviceTokenPath} ${lib.escapeShellArg appserviceHsTokenPath}; do
if [ ! -s "$f" ]; then
head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$f"
echo >> "$f"
echo "hive-matrix: generated appservice token at $f"
fi
chmod 0600 "$f"
done
# Read into shell variables and emitted with `printf`, a shell
# builtin: a token passed as an argument to a real command would land
# in that process's argv, which is world-readable for its lifetime.
asToken="$(cat ${lib.escapeShellArg appserviceTokenPath})"
hsToken="$(cat ${lib.escapeShellArg appserviceHsTokenPath})"
# The quoted heredoc keeps the regex's own `$` and `\` out of the
# shell's hands; the YAML single quotes keep them out of YAML's.
{
cat <<'REGISTRATION'
id: ${appserviceId}
url: null
sender_localpart: ${hiveLocalpart}
rate_limited: false
namespaces:
users:
- exclusive: false
regex: '${appserviceUserRegex}'
aliases: []
rooms: []
REGISTRATION
printf 'as_token: %s\nhs_token: %s\n' "$asToken" "$hsToken"
} > ${lib.escapeShellArg appserviceRegistrationPath}
chmod 0600 ${lib.escapeShellArg appserviceRegistrationPath}
chmod 0700 ${lib.escapeShellArg appserviceDir}
'';
};
# Format-locked by tuwunel, not chosen here: the callback host must point # Format-locked by tuwunel, not chosen here: the callback host must point
# directly at the matrix server and the path is fixed at # directly at the matrix server and the path is fixed at
# `/_matrix/client/unstable/login/sso/callback/<client_id>`. Built once # `/_matrix/client/unstable/login/sso/callback/<client_id>`. Built once
@ -482,61 +355,6 @@ in
''; '';
}; };
appserviceTokenFile = lib.mkOption {
type = lib.types.path;
internal = true;
default = appserviceTokenPath;
description = ''
Host path to a file containing this hive's matrix appservice
token (`as_token`). Minted automatically on first activation
(32-byte random hex, mode 0600) and rendered into the
appservice registration the homeserver loads at boot. Agents
never see it; an agent only ever receives its own
`access_token`.
Not operator-settable — the module's registration renderer reads
this same path as a literal, not through the option,
with nothing wiring an override across, so a moved path desyncs
the two silently. An externally-managed token is delivered by
writing into *this* fixed path instead of moving it — see
`glue-matrix-bao-token.nix`, which fetches from the swarm secret
store, overwrites this file in place, and re-renders the
registration that names it.
Enforced by an `assertions` entry below rather than `readOnly`:
the `config` block gives this option its own `mkDefault`
definition (lowest priority) so a real override still resolves
cleanly instead of crashing eval with nixpkgs' generic
"read-only, set multiple times" message; the assertion then
names the actual problem. (`readOnly` was the first attempt on
this option's predecessor — atlas caught that it only rejects a
*second* definition, so with nothing else defining the option a
lone override sailed through silently, same desync as before
with a lock that wasn't locked. Fixed by defining the value here
instead of leaving it on `default` alone, but a proper
`assertions` message beat re-adding `readOnly` on top once the
option had a real definition either way.)
'';
};
appserviceRegistrationScript = lib.mkOption {
type = lib.types.path;
internal = true;
default = "${appserviceRegistrationScript}/bin/hive-matrix-appservice-registration";
defaultText = lib.literalMD "the module's own registration renderer";
description = ''
The script that mints the appservice tokens when absent and
(re-)renders the registration file from them. An option only so
that `glue-matrix-bao-token.nix` can run the same one after
overwriting the token with the swarm store's copy, rather than
carrying a second copy of the registration's shape.
Not operator-settable, and not a hook: replacing it means
deciding what the homeserver's appservice registration says,
which is this module's job.
'';
};
gui.package = lib.mkOption { gui.package = lib.mkOption {
type = lib.types.package; type = lib.types.package;
default = fluffychat-web-fixed; default = fluffychat-web-fixed;
@ -629,13 +447,6 @@ in
}; };
config = lib.mkIf deployCfg.matrix.enable { config = lib.mkIf deployCfg.matrix.enable {
# The option's own value, defined explicitly rather than left on its
# bare `default` — `mkDefault` so a real override still resolves
# cleanly rather than crashing eval; the `assertions` entry below is
# what actually rejects it, with a message naming the reason. See that
# option's own comment above.
services.hyperhive.deploy.matrix.appserviceTokenFile = lib.mkDefault appserviceTokenPath;
# Matrix's own gateway surface: the sub-domain vhost, the name the # Matrix's own gateway surface: the sub-domain vhost, the name the
# hive resolver answers for, and the Accept-header map that vhost's # hive resolver answers for, and the Accept-header map that vhost's
# SPA fallback reads. All three are matrix knowledge and none of # SPA fallback reads. All three are matrix knowledge and none of
@ -769,25 +580,6 @@ in
httpPort and has no such name. httpPort and has no such name.
''; '';
} }
{
# mkDefault above lets an override resolve instead of crashing
# eval — this is the actual rejection, with a message that names
# the real fix instead of nixpkgs' generic conflicting-definition
# text.
assertion = deployCfg.matrix.appserviceTokenFile == appserviceTokenPath;
message = ''
services.hyperhive.deploy.matrix.appserviceTokenFile is fixed at
${appserviceTokenPath} and cannot be moved — the registration
renderer reads this same path as a literal and has no way to learn
an override, so moving it desyncs the two silently instead of
loudly.
Integrating an externally-managed appservice token? Deliver it
by writing into ${appserviceTokenPath} instead of pointing this
option elsewhere — see glue-matrix-bao-token.nix, which does
exactly that from the swarm secret store.
'';
}
]; ];
# One declaration, two readers. The homeserver knows its own callback # One declaration, two readers. The homeserver knows its own callback
@ -829,11 +621,10 @@ in
# permanent stall presenting as "matrix is broken", several layers from # permanent stall presenting as "matrix is broken", several layers from
# its cause. # its cause.
# #
# The appservice registration dodges that with an activation script # ⚠️ Rendering a placeholder first is no way around it: tuwunel
# that renders the file first. ⚠️ That dodge is NOT available here: # requires the secret file to exist *and be non-empty*, so a zero-byte
# tuwunel requires the secret file to exist *and be non-empty*, so a # placeholder would satisfy the bind mount and then stop the homeserver
# zero-byte placeholder would satisfy the bind mount and then stop the # from starting.
# homeserver from starting.
systemd.services.hive-matrix-oidc-secret = lib.mkIf ssoLocal { systemd.services.hive-matrix-oidc-secret = lib.mkIf ssoLocal {
description = "deliver the homeserver's OIDC client secret from authelia"; description = "deliver the homeserver's OIDC client secret from authelia";
after = [ "container@${autheliaCfg.machine}.service" ]; after = [ "container@${autheliaCfg.machine}.service" ];
@ -938,23 +729,6 @@ in
'' ''
); );
# Mint the appservice tokens and render the registration, before any
# container start.
#
# An activation script rather than a unit: the directory below is
# bind-mounted into the container and nixos-container refuses to start
# when a bind source is missing, so this has to have run first.
# Activation is what runs before the container both on a switch and on
# every boot. (The registration token this replaced used an activation
# script for the same reason, and additionally to dodge nspawn creating
# an empty file at a missing bind target — which tuwunel then read as
# "no token", refusing every registration until the next restart. A
# missing registration file is not silent in that way: the homeserver
# fails its appservice load loudly.)
system.activationScripts.hive-matrix-appservice = lib.stringAfter [ "var" ] ''
${deployCfg.matrix.appserviceRegistrationScript}
'';
containers.hive-matrix = { containers.hive-matrix = {
autoStart = true; autoStart = true;
ephemeral = false; ephemeral = false;
@ -962,23 +736,12 @@ in
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it. # --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [ "--link-journal=host" ]; extraFlags = [ "--link-journal=host" ];
inherit privateNetwork; inherit privateNetwork;
# Read-only bind of the host-managed appservice registration; tuwunel bindMounts =
# reads it via systemd LoadCredential below (not directly). caTrust.bindMount
# # matrix-ctl's bao client identity, and nothing else of the hive's. See
# The directory, not the file inside it: the registration is # `ctlPkiDirs` above for why it is a derived set of directories
# re-rendered rather than edited, and binding the file would pin the # rather than one literal.
# inode the container saw when it started. // ctlBindMounts;
bindMounts = {
${appserviceDir} = {
hostPath = appserviceDir;
isReadOnly = true;
};
}
// caTrust.bindMount
# matrix-ctl's bao client identity, and nothing else of the hive's. See
# `ctlPkiDirs` above for why it is a derived set of directories
# rather than one literal.
// ctlBindMounts;
config = config =
{ ... }: { ... }:
{ {
@ -1061,10 +824,10 @@ in
allow_federation = true; allow_federation = true;
trusted_servers = deployCfg.matrix.trustedServers; trusted_servers = deployCfg.matrix.trustedServers;
# Nobody registers themselves here. Accounts are created by # Nobody registers themselves here. Accounts are created by
# the hive's appservice, which this flag does not gate — # the swarm's appservice, which this flag does not gate —
# tuwunel checks it only for requests that arrive WITHOUT an # tuwunel checks it only for requests that arrive WITHOUT an
# appservice token, so hive-c0re provisions exactly as before # appservice token, so swarm-controller provisions and
# and everyone else is refused outright. # everyone else is refused outright.
# #
# ⚠️ Not a hardening afterthought: `allow_registration = true` # ⚠️ Not a hardening afterthought: `allow_registration = true`
# with no registration token configured makes tuwunel REFUSE # with no registration token configured makes tuwunel REFUSE
@ -1074,7 +837,7 @@ in
# homeserver, it is a homeserver that does not boot. # homeserver, it is a homeserver that does not boot.
allow_registration = false; allow_registration = false;
# Where the hive's appservice registration is read from — the # Where the swarm's appservice registration is read from — the
# credentials directory, for the reasons at # credentials directory, for the reasons at
# `appserviceCredentialDir`'s own definition. Loaded inside # `appserviceCredentialDir`'s own definition. Loaded inside
# `Services::start()`, before the listener accepts anything. # `Services::start()`, before the listener accepts anything.
@ -1083,8 +846,8 @@ in
} }
# Exactly one account is promoted to homeserver admin at boot: # Exactly one account is promoted to homeserver admin at boot:
# the swarm appservice's sender. Its token is read by # the swarm appservice's sender. Its token is read by
# matrix-ctl and swarm-controller only. `@${hiveLocalpart}` # matrix-ctl and swarm-controller only. Each `@hive-<hive>`
# stays an ordinary user, because every hive reads its # stays an ordinary user, because its hive reads its
# credential; what it does (the Space, the chat room, the # credential; what it does (the Space, the chat room, the
# invites) rides on being the rooms' creator, not on admin. # invites) rides on being the rooms' creator, not on admin.
# #
@ -1161,20 +924,15 @@ in
# host-side chown :tuwunel / GID-pin gymnastics needed. # host-side chown :tuwunel / GID-pin gymnastics needed.
# See `man systemd.exec` → LoadCredential. # See `man systemd.exec` → LoadCredential.
systemd.services.tuwunel.serviceConfig.LoadCredential = [ systemd.services.tuwunel.serviceConfig.LoadCredential = [
# The credential id carries a `.yaml` suffix on purpose: # tuwunel re-reads this file on every OAuth exchange, not just at startup, so it has to
# `appservice_dir` above names this very directory, and tuwunel
# takes only `.yaml`/`.yml` entries from it.
"${appserviceCredentialId}:${appserviceRegistrationPath}"
# Same mechanism, second secret. tuwunel re-reads this file on
# every OAuth exchange, not just at startup, so it has to
# outlive the unit's start — a credentials path does. # outlive the unit's start — a credentials path does.
"oidc_client_secret:${toString deployCfg.matrix.sso.clientSecretFile}" "oidc_client_secret:${toString deployCfg.matrix.sso.clientSecretFile}"
] ]
# The swarm registration, a second `.yaml` in the same directory: # The swarm registration. The credential id carries a `.yaml`
# tuwunel loads every one it finds (`appservice/mod.rs`), and # suffix on purpose: `appservice_dir` above names this very
# refuses only a duplicate `id` or `as_token`, never an # directory, and tuwunel takes only `.yaml`/`.yml` entries from
# overlapping namespace. A missing source fails this unit, which # it. A missing source fails this unit, which is why the render
# is why the render below is `requiredBy` it and local only. # below is `requiredBy` it and local only.
++ lib.optional ctlActive "${swarmAppserviceCredentialId}:${swarmAppserviceDir}/swarm.yaml"; ++ lib.optional ctlActive "${swarmAppserviceCredentialId}:${swarmAppserviceDir}/swarm.yaml";
# The swarm registration, minted and rendered before the homeserver # The swarm registration, minted and rendered before the homeserver
@ -1196,9 +954,8 @@ in
environment = { environment = {
MATRIX_APPSERVICE_DIR = swarmAppserviceDir; MATRIX_APPSERVICE_DIR = swarmAppserviceDir;
MATRIX_APPSERVICE_SENDER = swarmSenderLocalpart; MATRIX_APPSERVICE_SENDER = swarmSenderLocalpart;
# The hive registration's namespace. Agents' localparts are # Agents' localparts are bare `[a-z0-9-]`, so nothing narrower
# bare `[a-z0-9-]`, so nothing narrower covers them without # covers them without also covering people.
# also covering people.
MATRIX_APPSERVICE_USER_REGEX = appserviceUserRegex; MATRIX_APPSERVICE_USER_REGEX = appserviceUserRegex;
}; };
}; };

View file

@ -156,7 +156,7 @@ in
# Each of these deployments turns `hiveName` into an identifier with no # Each of these deployments turns `hiveName` into an identifier with no
# fallback: hive-c0re's OIDC client and matrix localpart, the CN of the # fallback: hive-c0re's OIDC client and matrix localpart, the CN of the
# store's `client.pem` (./glue-bao-tls.nix), and the `hives/<name>` path # store's `client.pem` (./glue-bao-tls.nix), and the `hives/<name>` path
# the matrix token and queue credential readers fetch. A null there renders # the queue credential reader fetches. A null there renders
# as an empty string that evaluates and deploys. # as an empty string that evaluates and deploys.
config.assertions = [ config.assertions = [
{ {

View file

@ -755,32 +755,31 @@ let
} }
''; '';
# ── the four principals that used to share the hive's own leaf ───────────── # ── the principals that used to share the hive's own leaf ──────────────────
# #
# 🩸 Each of the four reads exactly ONE path in the store, and until this # 🩸 Each of these reads exactly ONE path in the store, and until this split
# split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf, # each did it holding `deploy.bao.clientCertFile` — the hive's own leaf,
# whose policy (`swarm-secret-client`'s `policy::render`) granted read on # whose policy (`swarm-secret-client`'s `policy::render`) granted read on
# `swarm/agents/*`, `swarm/hives/<hive>/*` AND `swarm/services/*`. Four # `swarm/agents/*`, `swarm/hives/<hive>/*` AND `swarm/services/*`. Principals
# principals presenting one certificate are ONE principal to bao, so the union # presenting one certificate are ONE principal to bao, so the union of what
# of what the four need was the only grant expressible: the unit that fetches # they need was the only grant expressible: the unit that fetches Grafana's
# Grafana's OIDC client secret could fetch every agent credential in the # OIDC client secret could fetch every agent credential in the swarm, and the
# swarm, and the unit that fetches this hive's matrix token could fetch # unit that fetches this hive's queue credential could fetch Grafana's. Least
# Grafana's. Least privilege was not misconfigured, it was unrepresentable. # privilege was not misconfigured, it was unrepresentable.
# #
# Each now gets its own leaf, its own cert-auth role and a policy holding the # Each now gets its own leaf, its own cert-auth role and a policy holding the
# one `secret/data/…` path that principal's script actually names — spelled to # one `secret/data/…` path that principal's script actually names — spelled to
# the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above # the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above
# gives for doing the same. # gives for doing the same.
# #
# ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the # ⚠️ The queue-credential reader is PER HIVE and the OIDC readers are not,
# shape of the paths rather than a preference. A matrix appservice token and a # and that asymmetry is the shape of the paths rather than a preference. A
# queue credential live under `swarm/hives/<name>/`, and every hive runs a # queue credential lives under `swarm/hives/<name>/`, and every hive runs a
# reader for its own; an OIDC client secret lives under # reader for its own; an OIDC client secret lives under
# `swarm/services/<client-id>/` and a swarm registers each of those exactly # `swarm/services/<client-id>/` and a swarm registers each of those exactly
# once. A single role for a per-hive reader would have to name `hives/*` to # once. A single role for a per-hive reader would have to name `hives/*` to
# serve every hive — which would let one hive read another's matrix token, a # serve every hive — which would let one hive read another's queue
# reach no hive has today and one this change exists to remove rather than # credential.
# create.
readStanza = path: '' readStanza = path: ''
path "${path}" { path "${path}" {
capabilities = ["read"] capabilities = ["read"]
@ -799,12 +798,6 @@ let
policyText = readStanza (spec.path hiveName); policyText = readStanza (spec.path hiveName);
}) hyperhiveCfg.swarm.hives; }) hyperhiveCfg.swarm.hives;
matrixTokenReaders = perHiveReaders {
rolePrefix = "swarm-matrix-token";
cnPrefix = baoDeploy.matrixTokenCommonNamePrefix;
path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token";
};
queueAgentReaders = perHiveReaders { queueAgentReaders = perHiveReaders {
rolePrefix = "swarm-queue-agent"; rolePrefix = "swarm-queue-agent";
cnPrefix = baoDeploy.queueAgentCommonNamePrefix; cnPrefix = baoDeploy.queueAgentCommonNamePrefix;
@ -812,7 +805,7 @@ let
}; };
# Singletons, so a list of one rather than a second shape: the granting unit # Singletons, so a list of one rather than a second shape: the granting unit
# below takes a list either way and the four units stay identical apart from # below takes a list either way and the units stay identical apart from
# the objects they write. # the objects they write.
grafanaOidcReaders = [ grafanaOidcReaders = [
{ {
@ -830,8 +823,7 @@ let
} }
]; ];
# The fifth, and the one the four-way split left on the hive's leaf: the # The store's own forwarder. Its path is `forwarderStoreSecretPath` below with
# store's own forwarder. Its path is `forwarderStoreSecretPath` below with
# KV v2's `data/` inserted. # KV v2's `data/` inserted.
forwarderOidcReaders = [ forwarderOidcReaders = [
{ {
@ -876,10 +868,10 @@ let
display_name=${lib.escapeShellArg obj.cn} display_name=${lib.escapeShellArg obj.cn}
''; '';
# One unit per PRINCIPAL, not one unit for the four: these names are # One unit per PRINCIPAL, not one unit for all of them: these names are
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy` # operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
# above gives for not being more script inside the controller's unit. A hive # above gives for not being more script inside the controller's unit. A hive
# whose matrix token never arrives is diagnosed by the state of the unit named # whose queue credential never arrives is diagnosed by the state of the unit named
# after it. # after it.
# #
# `after` and not `requires`, for the reason the publisher's unit states: the # `after` and not `requires`, for the reason the publisher's unit states: the
@ -1224,7 +1216,22 @@ in
and this option named a write grant matrix-ctl no longer holds. Delete and this option named a write grant matrix-ctl no longer holds. Delete
this setting. this setting.
'') '')
]; ]
++
map
(
option:
lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "bao" option ] ''
The per-hive matrix appservice token and the unit that fetched it are
gone: swarm-controller creates every hive's and every agent's matrix
account with the swarm's appservice token. Delete this setting.
''
)
[
"matrixTokenClientCertFile"
"matrixTokenClientKeyFile"
"matrixTokenCommonNamePrefix"
];
# One service, two namespaces, and the split decides who may set what. # One service, two namespaces, and the split decides who may set what.
# #
@ -1236,7 +1243,8 @@ in
# `serverCaFile` live here too, and they are set by whoever authenticates TO # `serverCaFile` live here too, and they are set by whoever authenticates TO
# the store — which includes a host that runs none of it. A reader is defined # the store — which includes a host that runs none of it. A reader is defined
# by holding a certificate the store accepts, never by sharing a host with # by holding a certificate the store accepts, never by sharing a host with
# one; ./glue-matrix-bao-token.nix gates on exactly that and not on `enable`. # one; ./glue-queue-agent-credential.nix gates on exactly that and not on
# `enable`.
# Reading this block as store-runner-only is what makes an off-host reader # Reading this block as store-runner-only is what makes an off-host reader
# look inexpressible when it is already supported. # look inexpressible when it is already supported.
# #
@ -1629,27 +1637,6 @@ in
''; '';
}; };
matrixTokenCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-matrix-token";
example = "swarm-bao-matrix-token.svc";
description = ''
Prefix of the subject the store's per-hive matrix-token roles accept.
The role for hive `h` accepts `<prefix>-h` and nothing else, and grants
read on exactly `swarm/hives/h/matrix/appservice-token`.
A **prefix** rather than one subject because every hive runs its own
reader and the path it reads carries that hive's name. One subject for
all of them would need a `hives/*` grant, which would let one hive read
another hive's appservice token — a reach no hive has today.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
fixed-subject siblings are: cert auth trusts the CA and
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
carried one of these spellings would receive that grant.
'';
};
queueAgentCommonNamePrefix = lib.mkOption { queueAgentCommonNamePrefix = lib.mkOption {
type = lib.types.str; type = lib.types.str;
default = "swarm-bao-queue-agent"; default = "swarm-bao-queue-agent";
@ -1660,10 +1647,15 @@ in
exactly `swarm/hives/h/queue/agent` — the client credential that hive's exactly `swarm/hives/h/queue/agent` — the client credential that hive's
agent containers authenticate to the swarm queue with. agent containers authenticate to the swarm queue with.
Per hive for the same reason A **prefix** rather than one subject because every hive runs its own
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is, reader and the path it reads carries that hive's name. One subject for
and more sharply: every hive in the swarm runs this reader, so a single all of them would need a `hives/*` grant, which would let one hive read
subject would have to be granted `hives/*`. another hive's queue credential.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
fixed-subject siblings are: cert auth trusts the CA and
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
carried one of these spellings would receive that grant.
''; '';
}; };
@ -1678,7 +1670,7 @@ in
for `h`'s domain and `*.` that domain and nothing else. for `h`'s domain and `*.` that domain and nothing else.
Per hive for the reason Per hive for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is: {option}`services.hyperhive.deploy.bao.queueAgentCommonNamePrefix` is:
one subject for every hive could ask for any hive's certificate. one subject for every hive could ask for any hive's certificate.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings. ⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
@ -1725,7 +1717,7 @@ in
presents. Its grant is one path, presents. Its grant is one path,
`swarm/services/<collector client id>/oidc/client`, and read only. `swarm/services/<collector client id>/oidc/client`, and read only.
A **fourth** identity rather than reuse of Its **own** identity rather than reuse of
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two
read different services' client secrets, and a collector is not read different services' client secrets, and a collector is not
entitled to Grafana's. entitled to Grafana's.
@ -1742,46 +1734,13 @@ in
own forwarder's OIDC client secret, presents. Its grant is one path, own forwarder's OIDC client secret, presents. Its grant is one path,
`swarm/services/<forwarder client id>/oidc/client`, and read only. `swarm/services/<forwarder client id>/oidc/client`, and read only.
A **fifth** identity rather than reuse of Its **own** identity rather than reuse of
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`: the two {option}`services.hyperhive.deploy.bao.otelOidcCommonName`: the two
read different clients' secrets, and the store's forwarder is not read different clients' secrets, and the store's forwarder is not
entitled to the swarm collector's. entitled to the swarm collector's.
''; '';
}; };
matrixTokenClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/matrix-token.pem";
description = ''
Certificate the unit that fetches this hive's matrix appservice token
presents to the store. Its subject must be
`<matrixTokenCommonNamePrefix>-<this hive's name>`; cert auth matches on
the CN and the role accepts nothing else.
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one token a credential that reads every
secret in the store. That is what this option exists to end.
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
wherever this host mints its own PKI. A deployment that reads a store on
another host names it by hand, alongside the leaf itself — one file per
principal placed out of band instead of one shared by four.
'';
};
matrixTokenClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/matrix-token-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both
or neither — a certificate with no key authenticates nothing.
'';
};
queueAgentClientCertFile = lib.mkOption { queueAgentClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str; type = lib.types.nullOr lib.types.str;
default = null; default = null;
@ -1789,10 +1748,17 @@ in
description = '' description = ''
Certificate the unit that fetches this hive's queue client credential Certificate the unit that fetches this hive's queue client credential
presents to the store. Its subject must be presents to the store. Its subject must be
`<queueAgentCommonNamePrefix>-<this hive's name>`. `<queueAgentCommonNamePrefix>-<this hive's name>`; cert auth matches on
the CN and the role accepts nothing else.
⚠️ Not the hive's own leaf, for the reason ⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one credential a certificate that reads every
secret in the store.
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
wherever this host mints its own PKI. A deployment that reads a store on
another host names it by hand, alongside the leaf itself.
''; '';
}; };
@ -1816,7 +1782,7 @@ in
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`. {option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason ⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. {option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
''; '';
}; };
@ -1840,7 +1806,7 @@ in
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`. {option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason ⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. {option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
''; '';
}; };
@ -1864,7 +1830,7 @@ in
{option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`. {option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason ⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives. {option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
''; '';
}; };
@ -2924,13 +2890,11 @@ in
''; '';
}; };
# 🩸 The four principals that used to log in as the hive. See the block # 🩸 The principals that used to log in as the hive. See the block around
# around `perHiveReaders` for what each grant is and why the first two are # `perHiveReaders` for what each grant is and why the first is per hive;
# per hive; the four units are identical apart from the objects they # the units are identical apart from the objects they
# write, so the shape lives in `readerPolicyUnit` and the difference is # write, so the shape lives in `readerPolicyUnit` and the difference is
# visible here in one line each. # visible here in one line each.
systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders;
systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders; systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders;
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders; systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
@ -3139,7 +3103,6 @@ in
"swarm-bao-controller-policy.service" "swarm-bao-controller-policy.service"
"swarm-bao-secret-publisher-policy.service" "swarm-bao-secret-publisher-policy.service"
"swarm-bao-matrix-ctl-policy.service" "swarm-bao-matrix-ctl-policy.service"
"swarm-bao-matrix-token-policy.service"
"swarm-bao-queue-agent-policy.service" "swarm-bao-queue-agent-policy.service"
"swarm-bao-grafana-oidc-policy.service" "swarm-bao-grafana-oidc-policy.service"
"swarm-bao-otel-oidc-policy.service" "swarm-bao-otel-oidc-policy.service"

View file

@ -206,7 +206,7 @@ let
# A reader of the store is defined by holding a certificate the store # A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule # accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. # ./glue-queue-agent-credential.nix states in full.
# #
# Also asserted rather than gating anything, and that is the ruling this # Also asserted rather than gating anything, and that is the ruling this
# module's second delivery route was deleted under: the store exists so that # module's second delivery route was deleted under: the store exists so that

View file

@ -241,11 +241,10 @@ let
# A reader of the store is defined by holding a certificate the store # A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule # accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical- # ./glue-queue-agent-credential.nix states in full. Unlike Grafana's identical-
# looking flag, this one does not gate an assertion on its own: the # looking flag, this one does not gate an assertion on its own: the
# store-reading unit below simply does not render without it, the same choice # store-reading unit below simply does not render without it, the same choice
# ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for # ./glue-queue-agent-credential.nix makes for its own reader, because a collector with no client identity is
# their own readers, because a collector with no client identity is
# `haveCollectorSecret = false` above, and that is already a supported, # `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all. What # merely degraded shape rather than a service with no way in at all. What
# IS asserted is narrower and lives in `assertions` below — see # IS asserted is narrower and lives in `assertions` below — see

View file

@ -1,7 +1,7 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so # The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its agents' credential, a # whoever needs one can read it there: a hive its agents' credential, a
# service or the controller its own, a hive its appservice token. OIDC secrets # service or the controller its own. It mints nothing: every value is an OIDC
# it only copies; the appservice token it MINTS, having no swarm-side producer. # client secret authelia minted, and this unit copies it.
# #
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes # ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its # into the store even when the service runs beside authelia, because its
@ -21,13 +21,11 @@
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite # ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit that writes a handful of secrets # every hive's policy and login role; a unit that writes a handful of secrets
# has no business holding it. ./swarm-bao.nix grants this one `create`/`update` # has no business holding it. ./swarm-bao.nix grants this one `create`/`update`
# under the hive prefix and nothing else — no `read`, which is why the mint # under the hive prefix and nothing else — no `read`.
# below keeps its own record of what it published.
# #
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every # ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is # argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself. Same rule governs the mint: it # passed as `@<path>` and read by bao itself.
# redirects into a file and never holds the token in a shell variable.
{ {
pkgs, pkgs,
lib, lib,
@ -41,7 +39,8 @@ let
cfg = deployCfg.swarm-secret-publisher; cfg = deployCfg.swarm-secret-publisher;
# A reader is defined by holding a certificate the store accepts, never by # A reader is defined by holding a certificate the store accepts, never by
# standing next to it — the rule ./glue-matrix-bao-token.nix states in full. # standing next to it — the rule ./glue-queue-agent-credential.nix states in
# full.
# `deploy.bao.enable` here would be the co-location assumption itself. # `deploy.bao.enable` here would be the co-location assumption itself.
haveClientIdentity = cfg.baoClientCertFile != null && cfg.baoClientKeyFile != null; haveClientIdentity = cfg.baoClientCertFile != null && cfg.baoClientKeyFile != null;
@ -86,21 +85,6 @@ let
# under its own certificate. # under its own certificate.
controllerClientId = hyperhiveCfg.swarm.controller.queueClientId; controllerClientId = hyperhiveCfg.swarm.controller.queueClientId;
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client"; controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
# Where this unit keeps the appservice tokens it minted, and the whole reason
# a re-publish is idempotent. The store cannot be that record: ./swarm-bao.nix
# grants this principal `create`/`update` and deliberately no `read`, so "does
# this hive already have a token?" is a question it can only ask locally.
# Without the file every run would mint a fresh value and rotate the swarm's
# token for no reason — and a rotation is not free, because the homeserver
# only reloads the registration naming it when its container restarts.
#
# `StateDirectory=` below is what creates it, so the name there is relative
# and this unit owns the tree. Losing it mints once more and republishes;
# readers pick that up on their next start, and nothing that already
# registered breaks — the token authenticates the appservice, it does not
# live inside any account it created.
matrixTokenDir = "/var/lib/swarm-secret-publish/matrix-appservice-token";
in in
{ {
options.services.hyperhive.deploy.swarm-secret-publisher = { options.services.hyperhive.deploy.swarm-secret-publisher = {
@ -112,9 +96,7 @@ in
Publish the OIDC client secrets this host mints into the swarm's Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its secret store, so a hive that does not run authelia can read its
agents' credential and a swarm service or the controller can read its agents' credential and a swarm service or the controller can read its
own — from wherever it runs, this host included. Also mints each hive's matrix own — from wherever it runs, this host included.
appservice token, which has no other swarm-side producer, and
publishes it the same way.
Defaults to whether this host mints the OIDC secrets, which is the Defaults to whether this host mints the OIDC secrets, which is the
only half of the question that is a property of *this* host. only half of the question that is a property of *this* host.
@ -128,9 +110,8 @@ in
Turning it off leaves every hive but this one without its agents' Turning it off leaves every hive but this one without its agents'
credential, the swarm controller unable to start, the swarm's Grafana credential, the swarm controller unable to start, the swarm's Grafana
without any login at all, its collector pushing unauthenticated, and without any login at all, and its collector pushing unauthenticated.
every hive falling back to the appservice token its own first boot Each secret has exactly one route and this is the producer's
minted — so two hives never agree on one. Each secret has exactly one route and this is the producer's
end of it, so the honest reason to set it false is a deployment end of it, so the honest reason to set it false is a deployment
delivering those secrets by some other mechanism it owns. delivering those secrets by some other mechanism it owns.
''; '';
@ -190,12 +171,6 @@ in
TimeoutStartSec = 60; TimeoutStartSec = 60;
Restart = "on-failure"; Restart = "on-failure";
RestartSec = 30; RestartSec = 30;
# The minted appservice tokens, see `matrixTokenDir` above. Named
# relative to /var/lib because that is the form systemd creates; the
# mode is narrowed from systemd's 0755 default because the directory
# listing is the roster of hives and every file in it is a secret.
StateDirectory = "swarm-secret-publish/matrix-appservice-token";
StateDirectoryMode = "0700";
}; };
environment = { environment = {
BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}"; BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
@ -222,7 +197,6 @@ in
published=0 published=0
skipped=0 skipped=0
tokens=0
${lib.concatMapStringsSep "\n" (hive: '' ${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"} src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"}
@ -281,43 +255,7 @@ in
skipped=$((skipped + 1)) skipped=$((skipped + 1))
fi fi
# The matrix appservice token, per hive. Unlike the two loops above echo "published $published client secret(s), skipped $skipped"
# there is nothing to copy: authelia never minted this one, and each
# homeserver's own host minted its own — which is exactly why two hives
# never converged on a value the swarm has to agree on. The mint moves
# here because this unit already holds a store identity and already
# writes under the hive prefix; ../host-modules/hive-matrix.nix keeps
# its activation mint as the first-boot fallback for a hive that has
# never reached the store.
#
# 32 bytes of urandom as lowercase hex, the shape that file has
# generated all along. Written with no trailing newline, unlike that
# file: what goes into the store is the value, and
# ./glue-matrix-bao-token.nix adds the newline the hive's token file
# carries.
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${matrixTokenDir}/${hive}"}
if [ ! -s "$src" ]; then
# The subshell keeps the umask local — the loops above rely on
# bao's own file handling and should not inherit one.
( umask 077; head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$src" )
echo "minted a matrix appservice token for ${hive}" >&2
fi
# `value=@$src` for the same reason as both loops above: bao opens
# the file itself, so the token is never an argument of this process.
#
# Put on every run, not only on the run that minted — a put of the
# value already there changes nothing for any reader, while a mint
# whose publish failed must not be left as a token this host holds
# and no hive can reach. The `-s` test above is what keeps the VALUE
# stable across those runs: rotating on every boot would hand the
# homeserver a registration its running container does not carry.
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/matrix/appservice-token"} \
value=@"$src"
tokens=$((tokens + 1))
'') hiveNames}
echo "published $published client secret(s), skipped $skipped, and $tokens matrix appservice token(s)"
''; '';
}; };
}; };

View file

@ -63,7 +63,6 @@ let
# Every spelling is reserved for every declared hive, including the one the # Every spelling is reserved for every declared hive, including the one the
# CN belongs to — a hive cannot be named after its own reader either. # CN belongs to — a hive cannot be named after its own reader either.
++ lib.concatMap (hive: [ ++ lib.concatMap (hive: [
"${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}"
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}" "${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
"${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}" "${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}"
]) hiveNames; ]) hiveNames;

View file

@ -162,13 +162,6 @@ let
name = "an operator's own certificate path beats the glue's default"; name = "an operator's own certificate path beats the glue's default";
ok = baoExplicitCerts.services.hyperhive.deploy.bao.serverCertFile == "/etc/pki/bao.pem"; ok = baoExplicitCerts.services.hyperhive.deploy.bao.serverCertFile == "/etc/pki/bao.pem";
} }
{
# Absence arm. A store with nothing to serve renders no reader, so the
# unit is a function of the PAIRING rather than of the store — which is
# the property that makes it glue instead of a feature of either side.
name = "a store with no homeserver beside it renders no token reader";
ok = !(baoPkcs11.systemd.services ? swarm-bao-matrix-token);
}
{ {
# The name a reader dials has to resolve where the store runs; a # The name a reader dials has to resolve where the store runs; a
# multi-host swarm resolves it upstream instead. # multi-host swarm resolves it upstream instead.

Some files were not shown because too many files have changed in this diff Show more