vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

matrix: drop the per-hive appservice and its unused token

Browse source 
Every hive's homeserver loaded a per-hive `hyperhive` appservice
registration, and the swarm minted and delivered an as_token for it
through bao. No client presented that token: swarm-controller creates
every `@hive-<hive>` sender and every agent account with the swarm
appservice token (hive_sender.rs, agent_token.rs), and no Rust code read
the per-hive one.

Removed, with everything that existed only to serve it:
- hive-matrix.nix: the `hyperhive` registration, its activation-time
  render, the host bind mount of /var/lib/hyperhive/matrix-appservice,
  its LoadCredential entry, and the internal options
  `deploy.matrix.appserviceTokenFile` / `appserviceRegistrationScript`.
  tuwunel now loads only the swarm registration.
- glue-matrix-bao-token.nix and its `swarm-bao-matrix-token` unit.
- glue-bao-tls.nix: the matrix-token leaf and its defaults.
- swarm-bao.nix: the `swarm-matrix-token-<hive>` roles and policies
  (`swarm-bao-matrix-token-policy`), and the options
  `deploy.bao.matrixTokenClient{Cert,Key}File` and
  `matrixTokenCommonNamePrefix`, now mkRemovedOptionModule.
- swarm-secret-publisher.nix: the per-hive token mint and its state dir.
- swarm.nix: the reserved `<matrixTokenCommonNamePrefix>-<hive>` CN.
- swarm-secret-client: `matrix::appservice_token_path`.

The swarm registration (swarm-matrix-ctl, swarm-controller) is untouched.
Comments and docs that pointed at the removed glue file now point at
glue-queue-agent-credential.nix, which has the same reader shape.

Refs #4428
This commit is contained in:
atlas 2026-10-04 22:52:22 +02:00 • committed by mara
commit 48067d07e8
31 changed files with 362 additions and 1233 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/swarm/credentials.md
View file

@ -66,7 +66,6 @@ of the cell says how.
| `swarm/agents/<agent>/forge-token` | `swarm-controller`, at agent creation and in a pass every 5 minutes over every agent with a store identity | the agent container itself, under its own certificate, fetched to `/run/hive-agent-forge-token/token` | ✅ the controller re-mints when the stored token is missing or no longer matches the forge (last eight characters and scopes) | ✅ the agent re-fetches on a 10-minute timer |
| `swarm/agents/<agent>/forge/<label>` | `swarm-controller`, when an operator links an external forge account in the swarm UI | `hive-agent-forge-accounts` in the agent container, under the agent's own certificate, into `<state>/forge-<label>-token` and `forge-<label>.json` | ❌ an operator's token; replaced only by deleting it and linking the label again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/agents/<agent>/github-token` | `swarm-controller`, when an operator links a GitHub account in the swarm UI | `hive-agent-github-token` in the agent container, under the agent's own certificate, into `<state>/github-token` | ❌ an operator's token; replaced only by deleting it and linking it again | ✅ the agent re-fetches on a 2-minute timer |
| `swarm/hives/<hive>/matrix/appservice-token` | one minter, on the authelia host | the hive process that presents the token to its homeserver, under the hive's own certificate | must be stated | must be stated |
| `swarm/hives/<hive>/matrix/sender-token` | `swarm-controller`, with the swarm's appservice token, for every hive in its directory in a five-minute pass | `swarm-controller` under its own certificate, before it decides whether to mint, and hive-c0re's `stored_sender_token()`, under the hive's own certificate | ✅ the controller's pass re-mints when the stored token is missing, unknown to the homeserver, or someone else's | ✅ hive-c0re's matrix sweep reads the store every run and overwrites its token file when the store's token differs |
| `swarm/hives/<hive>/queue/agent` | authelia | `swarm-bao-queue-agent` on the hive's host, under its own per-hive certificate; no agent's policy reaches it | must be stated | must be stated |
| `swarm/services/<clientId>/oidc/client` | authelia | that service's own reader, under its own certificate; the bao UI's by the store's granter | must be stated | must be stated |

96
docs/swarm/secrets.md
View file

@ -90,8 +90,7 @@ with nothing to authenticate with still receives every hive's telemetry —
only its own pushes to the stores go out unauthenticated and get refused
there. `swarm-bao-otel-oidc.service` therefore doesn't render at all without
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile`, the same choice
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` make for
their own optional readers, rather than the hard assertion `swarm-grafana.nix`
`glue-queue-agent-credential.nix` makes for its own reader, rather than the hard assertion `swarm-grafana.nix`
uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
@ -159,7 +158,6 @@ when the system builds. The server names the offending file and refuses to run.
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive leaf certs | `hive-tls.nix`, issued by the secret store's `pki` mount | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| the hive's own store identity | `glue-bao-tls.nix` on the store's host; placed by hand on every other host ([`ca.md`](ca.md#the-gateway-leaf-comes-from-the-secret-store)) | `deploy.hive-controller.tls.baoHiveIdentityCertFile` / `baoHiveIdentityKeyFile` |
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
@ -169,50 +167,7 @@ bounded wait, 120s — and then **fail loudly** rather than skipping. A silent s
produces a service whose login button always fails, which is a symptom many
layers from its cause.
The store's **first reader** is the matrix appservice token, and it's worth
saying why that one: it's an opaque 32-byte value with no second file and no
format. Authelia's OIDC secret needs a `.secret` _and_ a matching `.digest`, so
starting there would have meant debugging "can a reader authenticate and get
bytes back" and "is authelia's file format right" at once, with an
SSO outage as the failure mode.
`glue-matrix-bao-token.nix` fetches it, writes the file `hive-matrix.nix`
already reads, and then runs that module's own renderer to re-stamp the
appservice registration naming the token — so the homeserver never learns the
store exists. The re-render isn't housekeeping: the token is half an
agreement, and a registration carrying the previous value authenticates
nobody. Every failure path — no such key, sealed store, unreachable store,
empty value — leaves the locally minted token in place, so a hive with no store
behaves exactly as it did before.
The store path is `swarm/hives/<hive>/matrix/appservice-token`, and the
**producer is `swarm-secret-publish`** — the same unit that copies authelia's
OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s
it there, so the store is the source of truth and every hive converges on the
value it holds. The hive's own activation mint is still there, but it's a
first-boot fallback now: it fires only when the token file is absent, and the
reader overwrites whatever it produced.
The mint is **idempotent by keeping its own record**, not by asking the store.
The publisher's grant is `create`/`update` under `swarm/hives/*` with no
`read` — write-only on purpose, so a file-copier can't recover every hive's
credentials — which means it can't check whether a hive already has a token.
Instead it keeps the value it minted under its `StateDirectory`,
`/var/lib/swarm-secret-publish/matrix-appservice-token/<hive>` (`0700` dir,
`0600` file), and mints only when that file is missing or empty. The `put`
itself runs every time: re-putting the same bytes changes nothing for any
reader, while a mint whose publish failed must not leave behind a token
this host holds and no hive can reach.
Lose the state directory and the next run mints once more and republishes.
That rotates the token, which readers pick up on their next start — nothing
that already registered breaks, because the token authenticates the
appservice rather than living inside any account it created.
The path was `…/matrix/registration-token` while the homeserver still took a
shared registration secret; nothing reads a value left at the old path.
The **second reader** is the agent containers' queue credential:
The store's **first reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and
the client id it authenticates, because that's how a queue client reads them —
the secret by path, the id by value. No local fallback exists, and none is
@ -237,7 +192,7 @@ never reads the bytes at any point: it runs as `hive-core`, which is the reason
a copy wasn't an option either. When the files aren't there the daemon forwards
nothing and says so in its journal, and the container starts without them.
The **third reader** is Grafana's OIDC client secret, and it's the first one
The **second reader** is Grafana's OIDC client secret, and it's the first one
that's a _swarm service's_ own credential rather than a hive's. It has exactly
one delivery route: `swarm-bao-grafana-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, on **every** host
@ -265,7 +220,7 @@ nothing said anything. Registration is separate and stays where authelia is
so it's declared on the host running authelia, which isn't necessarily the host
running Grafana.
The **fourth reader** is the swarm collector's own OIDC client secret, the
The **third reader** is the swarm collector's own OIDC client secret, the
same shape one level down: `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store on every host that
runs the collector, and registration lives separately in
@ -275,13 +230,12 @@ telemetry with none, so `swarm-otel.nix` doesn't assert
`deploy.bao.otelOidcClientCertFile` / `otelOidcClientKeyFile` the way Grafana
does — it
doesn't render the reading unit without them, the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
their own optional reads.
`glue-queue-agent-credential.nix` uses for its own read.
The **fifth reader** is the secret store forwarder's OIDC client secret:
The **fourth reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
fourth in both directions. It renders unconditionally, because the export it
third in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
@ -340,59 +294,55 @@ next controller restart.
### Per-principal identities
Bao matches a cert-auth role on the certificate's subject, so a certificate is an
identity and sharing one merges the identities. These five units read one path
identity and sharing one merges the identities. These four units read one path
each and each holds a leaf, a role and a policy of its own:
| unit | option pair under `deploy.bao.` | reads |
| -------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- |
| `swarm-bao-matrix-token` | `matrixTokenClientCertFile` / `matrixTokenClientKeyFile` | `swarm/hives/<hive>/matrix/appservice-token` |
| `swarm-bao-queue-agent` | `queueAgentClientCertFile` / `queueAgentClientKeyFile` | `swarm/hives/<hive>/queue/agent` |
| `swarm-bao-grafana-oidc` | `grafanaOidcClientCertFile` / `grafanaOidcClientKeyFile` | `swarm/services/<grafana client id>/oidc/client` |
| `swarm-bao-otel-oidc` | `otelOidcClientCertFile` / `otelOidcClientKeyFile` | `swarm/services/<collector client id>/oidc/client` |
| `swarm-bao-forwarder-oidc` | `forwarderOidcClientCertFile` / `forwarderOidcClientKeyFile` | `swarm/services/<store forwarder client id>/oidc/client` |
The first two exist **per hive**, because the path they read carries a hive
name and every hive runs its own reader. Their subjects are
`<deploy.bao.matrixTokenCommonNamePrefix>-<hive>` and
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves both
composed spellings as hive names, so nobody can name a hive into another hive's
The first exists **per hive**, because the path it reads carries a hive name
and every hive runs its own reader. Its subject is
`<deploy.bao.queueAgentCommonNamePrefix>-<hive>`; `swarm.nix` reserves that
composed spelling as a hive name, so nobody can name a hive into another hive's
role. The other three read a path that names a swarm service rather than a hive,
so one role each is enough and their subjects are the flat
`deploy.bao.grafanaOidcCommonName`, `deploy.bao.otelOidcCommonName` and
`deploy.bao.forwarderOidcCommonName`.
On a host that mints its own PKI, `glue-bao-tls.nix` signs all five and defaults
all ten options, and there is nothing to do. Elsewhere you issue each leaf from
On a host that mints its own PKI, `glue-bao-tls.nix` signs all four and defaults
all eight options, and there is nothing to do. Elsewhere you issue each leaf from
that CA out of band and name it here — one file per principal rather than one file
shared by five, which is the whole of what this buys.
shared by four, which is the whole of what this buys.
On a host without the store, where the fifth never runs, forgetting one of the
other eight isn't a quiet degrade. Three of the four
readers used to render only where their leaf existed, so a hand-configured
remote-store hive that named the hive's own `clientCertFile` and missed a
principal's pair just lost that unit; `swarm-grafana.nix` was alone in refusing
the build. All four now refuse it, each naming its own option pair — see
On a host without the store, where the forwarder never runs, forgetting one of
the other six isn't a quiet degrade. A hand-configured remote-store hive that
names the hive's own `clientCertFile` and misses a principal's pair has the
build refused, by the module of the reader it missed, naming that pair — see
[what a missing leaf costs](#what-a-missing-leaf-costs) below.
### What a missing leaf costs
Each of the four modules asserts its own pair, on the same condition: **this host
Each of the three modules asserts its own pair, on the same condition: **this host
already reads the store** — it holds `deploy.bao.clientCertFile` and
`clientKeyFile` — **and the consumer is on, and this principal's pair is null**.
The refusal names the two options, so an operator acts on the message without
opening the nix.
That condition fires on exactly one shape, the hand-configured remote-store hive
that named seven of the eight. It stays silent everywhere else, and each half of
that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of
that matters on its own:
- **A host with no store identity at all** holds no `clientCertFile` either, so
the first clause is false. That's the supported no-store deployment, and also
the one an operator passes through while bringing a hive up — neither should
fail to evaluate.
- **A host that runs the store** gets all eight from `glue-bao-tls.nix` as
- **A host that runs the store** gets every pair from `glue-bao-tls.nix` as
defaults, so the third clause is false. Nothing to set, nothing to refuse.
- **A consumer that's off** — no homeserver, no collector, no Grafana — has no
- **A consumer that's off** — no collector, no Grafana — has no
unit to skip, so the second clause is false. Only the queue-credential reader
has no toggle of its own to check: every hive runs one for its own agents, so
reading the store at all is what asks for it.