vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

matrix: publish the appservice token from the swarm, not just read it

Browse source 
`glue-matrix-bao-token.nix` has read
`secret/swarm/hives/<hive>/matrix/appservice-token` since it landed, but
nothing ever wrote that path. The store was empty in every deployment, so
every read degraded to "keep what activation minted" and each hive stayed
the origin of a value the swarm has to agree on — two hives never
converged.

`swarm-secret-publish` is now the producer. It already holds a store
identity, already writes under the hive prefix, and already runs per
hive in the roster, so the mint is a third loop beside the two OIDC
copies rather than a second shape of this unit.

Idempotence comes from a record of its own, not from the store: this
principal is granted `create`/`update` with no `read`, so it cannot ask
whether a hive already has a token. It keeps what it minted under
`StateDirectory=` (0700 dir, 0600 file) and mints only when that file is
missing or empty; the `put` runs every time, because re-putting the same
bytes changes nothing for a reader while a mint whose publish failed must
not be left as a token this host holds and no hive can reach.

The token never becomes a nix literal and never reaches argv: the mint
redirects into a file, and the publish hands bao `value=@<path>` so bao
opens it itself — the same handling the OIDC loops use.

`hive-matrix.nix`'s activation mint stays as the genuine first-boot
fallback. It already fires only when the token file is absent, so it
cannot clobber a value the store delivered; `hs_token` has no swarm half
and is still minted there for real.

Refs #4402
This commit is contained in:
atlas 2026-09-15 20:57:49 +02:00
commit 199afa41c8
5 changed files with 167 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/integrations/matrix.md
View file

@ -192,8 +192,12 @@ restarts, so the first boot after the switch already has both halves.
sets it fails to evaluate with a message naming the appservice — a hive
that never set it (the default) is unaffected.
- **A swarm store holding the old `matrix/registration-token` path** is
no longer read at all; the hive uses its locally minted appservice
token until someone `put`s a value at `matrix/appservice-token`.
no longer read at all. The value that matters now lives at
`matrix/appservice-token`, and `swarm-secret-publish` on the authelia
host mints and `put`s it there — the hive uses its locally minted
token only until the first successful read. See
[`../swarm/secrets.md`](../swarm/secrets.md) for how that mint stays
idempotent across runs.
</details>

46
docs/swarm/secrets.md
View file

@ -132,14 +132,14 @@ when the system builds. The server names the offending file and refuses to run.
## Hive-level — one of each per hive
| secret | generated by | lives at |
| ---------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix appservice token | a host activation script, on first boot | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
| secret | generated by | lives at |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| hive CA cert + key | `hive-tls.nix` first-boot unit | `<deploy.hive-controller.tls.stateDir>/ca.pem`, `ca-key.pem` (`0600`) |
| hive leaf certs | `hive-tls.nix`, signed by the hive CA | `<deploy.hive-controller.tls.stateDir>/<name>.pem` |
| matrix appservice token | `swarm-secret-publish`, published to the store; a host activation script only as a first-boot fallback | `/var/lib/hyperhive/matrix-appservice-token` (`0600`) |
| the forge's copy of its OIDC secret | `hive-forge-oidc-secret.service` copies it from authelia's tree | `/var/lib/forgejo-oidc/<id>.secret` inside the forge container |
| the homeserver's copy of its OIDC secret | `hive-matrix-oidc-secret.service`, same shape | `/var/lib/tuwunel-oidc/<id>.secret`, handed to tuwunel through `LoadCredential` |
| the agent containers' queue credential | authelia, published to the store by `swarm-secret-publish` | `<deploy.hive-controller.queue.agentCredentialDir>/secret` (`0600`) and `/client_id` (`0644`) |
Both delivery units wait for authelia's first boot to mint the secret — a
bounded wait, 120s — and then **fail loudly** rather than skipping. A silent skip
@ -162,10 +162,32 @@ nobody. Every failure path — no such key, sealed store, unreachable store,
empty value — leaves the locally minted token in place, so a hive with no store
behaves exactly as it did before.
The store path is `swarm/hives/<hive>/matrix/appservice-token`. It was
`…/matrix/registration-token` while the homeserver still took a shared
registration secret; a value left at the old path is read by nothing, and the
hive falls back to its local token until someone `put`s the new one.
The store path is `swarm/hives/<hive>/matrix/appservice-token`, and the
**producer is `swarm-secret-publish`** — the same unit that copies authelia's
OIDC secrets in. It mints one token per hive in the swarm's roster and `put`s
it there, so the store is the source of truth and every hive converges on the
value it holds. The hive's own activation mint is still there, but it's a
first-boot fallback now: it fires only when the token file is absent, and the
reader overwrites whatever it produced.
The mint is **idempotent by keeping its own record**, not by asking the store.
The publisher's grant is `create`/`update` under `swarm/hives/*` with no
`read` — write-only on purpose, so a file-copier can't recover every hive's
credentials — which means it can't check whether a hive already has a token.
Instead it keeps the value it minted under its `StateDirectory`,
`/var/lib/swarm-secret-publish/matrix-appservice-token/<hive>` (`0700` dir,
`0600` file), and mints only when that file is missing or empty. The `put`
itself runs every time: re-putting the same bytes changes nothing for any
reader, while a mint whose publish failed must not be left as a token this
host holds and no hive can reach.
Lose the state directory and the next run mints once more and republishes.
That rotates the token, which readers pick up on their next start — nothing
that already registered breaks, because the token authenticates the
appservice rather than living inside any account it created.
The path was `…/matrix/registration-token` while the homeserver still took a
shared registration secret; a value left at the old path is read by nothing.
The **second reader** is the agent containers' queue credential:
`glue-queue-agent-credential.nix` lands it as two files, the client secret and

9
nix/host-modules/hive-matrix.nix
View file

@ -126,6 +126,15 @@ let
# whatever they now hold. **One script with two callers** — the activation
# script below and `glue-matrix-bao-token.nix`, which overwrites the
# `as_token` with the swarm store's copy and has to re-render afterwards.
#
# ⚠️ The `as_token` mint here is the FIRST-BOOT FALLBACK, not the normal
# route. `swarm-secret-publisher.nix` mints one per hive swarm-side and
# publishes it to the store; the reader above overwrites this file before
# the homeserver's container starts, so what a hive actually runs on is the
# swarm's value. This keeps a hive that has never reached the store working
# — and it fires only when the file is absent, so it never clobbers one the
# store delivered. (`hs_token` has no swarm half: it is how the homeserver
# authenticates to *us*, so it stays hive-local and is minted here for real.)
# A second copy of the registration's shape would be a second source of
# truth for a format whose mismatch is silent: the homeserver would load a
# registration naming a token nobody holds, and every request hive-c0re

104
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,6 +1,7 @@
# The unit that copies authelia's minted OIDC client secrets into the swarm's
# secret store, so whoever needs one can read it there: a hive its agents'
# credential, a swarm service its own.
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its agents' credential, a swarm
# service its own, a hive its matrix appservice token. The OIDC secrets it only
# copies; the appservice token it MINTS, having had no swarm-side producer.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
@ -14,19 +15,19 @@
# why: the plaintext's other reader lives in a **different container**, and
# containers that share this host's network namespace still have separate
# filesystem roots, so "the host is the only place both trees are addressable".
# A delivery step therefore runs on the host that mints — not on the store's
# host, and not on the reading hive's.
# A delivery step therefore runs on the minting host, not the store's or the
# reading hive's.
#
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit whose entire job is copying one
# file has no business holding it. ./swarm-bao.nix grants this one
# `create`/`update` under the hive prefix and nothing else.
# every hive's policy and login role; a unit that writes a handful of secrets
# has no business holding it. ./swarm-bao.nix grants this one `create`/`update`
# under the hive prefix and nothing else — no `read`, which is why the mint
# below keeps its own record of what it published.
#
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself. (`/knowledge/secret-hygiene.md`:
# "a path keeps the secret out of the store, and reading it into a shell
# variable puts it straight into argv".)
# passed as `@<path>` and read by bao itself. Same rule governs the mint: it
# redirects into a file and never holds the token in a shell variable.
{
pkgs,
lib,
@ -71,6 +72,21 @@ let
hyperhiveCfg.swarm.grafana.oidc.clientId
hyperhiveCfg.swarm.otel.clientId
];
# Where this unit keeps the appservice tokens it minted, and the whole reason
# a re-publish is idempotent. The store cannot be that record: ./swarm-bao.nix
# grants this principal `create`/`update` and deliberately no `read`, so "does
# this hive already have a token?" is a question it can only ask locally.
# Without the file every run would mint a fresh value and rotate the swarm's
# token for no reason — and a rotation is not free, because the homeserver
# only reloads the registration naming it when its container restarts.
#
# `StateDirectory=` below is what creates it, so the name there is relative
# and this unit owns the tree. Losing it mints once more and republishes;
# readers pick that up on their next start, and nothing that already
# registered breaks — the token authenticates the appservice, it does not
# live inside any account it created.
matrixTokenDir = "/var/lib/swarm-secret-publish/matrix-appservice-token";
in
{
options.services.hyperhive.deploy.swarm-secret-publisher = {
@ -82,10 +98,12 @@ in
Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its
agents' credential and a swarm service can read its own — from
wherever it runs, this host included.
wherever it runs, this host included. Also mints each hive's matrix
appservice token, which has no other swarm-side producer, and
publishes it the same way.
Defaults to whether this host mints them, which is the only half of
the question that is a property of *this* host.
Defaults to whether this host mints the OIDC secrets, which is the
only half of the question that is a property of *this* host.
⚠️ Deliberately **not** `deploy.bao.enable`. That asks whether the
store stands here, and a publisher beside a remote store, holding a
@ -95,11 +113,12 @@ in
identity's job (see `baoClientCertFile`), not this option's.
Turning it off leaves every hive but this one without its agents'
credential, the swarm's Grafana without any login at all, and its
collector pushing unauthenticated — each secret has exactly one route
and this is the producer's end of it. So the honest reason to set it
false is a deployment delivering those secrets by some other
mechanism it owns.
credential, the swarm's Grafana without any login at all, its
collector pushing unauthenticated, and every hive falling back to the
appservice token its own first boot minted — so two hives never agree
on one. Each secret has exactly one route and this is the producer's
end of it, so the honest reason to set it false is a deployment
delivering those secrets by some other mechanism it owns.
'';
};
@ -143,7 +162,7 @@ in
};
systemd.services.swarm-secret-publish = {
description = "publish minted OIDC client secrets to the swarm secret store";
description = "publish the swarm's minted secrets to the swarm secret store";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
path = [
@ -159,6 +178,12 @@ in
TimeoutStartSec = 60;
Restart = "on-failure";
RestartSec = 30;
# The minted appservice tokens, see `matrixTokenDir` above. Named
# relative to /var/lib because that is the form systemd creates; the
# mode is narrowed from systemd's 0755 default because the directory
# listing is the roster of hives and every file in it is a secret.
StateDirectory = "swarm-secret-publish/matrix-appservice-token";
StateDirectoryMode = "0700";
};
environment = {
BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
@ -185,6 +210,7 @@ in
published=0
skipped=0
tokens=0
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"}
@ -228,7 +254,43 @@ in
fi
'') serviceClientIds}
echo "published $published client secret(s), skipped $skipped"
# The matrix appservice token, per hive. Unlike the two loops above
# there is nothing to copy: authelia never minted this one, and each
# homeserver's own host minted its own — which is exactly why two hives
# never converged on a value the swarm has to agree on. The mint moves
# here because this unit already holds a store identity and already
# writes under the hive prefix; ../host-modules/hive-matrix.nix keeps
# its activation mint as the first-boot fallback for a hive that has
# never reached the store.
#
# 32 bytes of urandom as lowercase hex, the shape that file has
# generated all along. Written with no trailing newline, unlike that
# file: what goes into the store is the value, and
# ./glue-matrix-bao-token.nix adds the newline the hive's token file
# carries.
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${matrixTokenDir}/${hive}"}
if [ ! -s "$src" ]; then
# The subshell keeps the umask local — the loops above rely on
# bao's own file handling and should not inherit one.
( umask 077; head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$src" )
echo "minted a matrix appservice token for ${hive}" >&2
fi
# `value=@$src` for the same reason as both loops above: bao opens
# the file itself, so the token is never an argument of this process.
#
# Put on every run, not only on the run that minted — a put of the
# value already there changes nothing for any reader, while a mint
# whose publish failed must not be left as a token this host holds
# and no hive can reach. The `-s` test above is what keeps the VALUE
# stable across those runs: rotating on every boot would hand the
# homeserver a registration its running container does not carry.
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/matrix/appservice-token"} \
value=@"$src"
tokens=$((tokens + 1))
'') hiveNames}
echo "published $published client secret(s), skipped $skipped, and $tokens matrix appservice token(s)"
'';
};
};

35
nix/module-eval.nix
View file

@ -1477,6 +1477,41 @@ let
lib.hasInfix "secret/swarm/hives/h1/queue/agent" s
&& lib.hasInfix "secret/swarm/hives/h2/queue/agent" s;
}
{
# The producer's end of the read `glue-matrix-bao-token.nix` already did.
# Both hives are asserted for the reason the queue case above gives: a
# publisher that knew one name rather than the roster would pass on a
# single-hive fixture and strand the second hive ever added — which is
# the two-hives-never-converge shape this slice exists to close.
name = "the publisher mints an appservice token for every hive and writes it to that hive's matrix path";
ok =
let
s = secretPublisherHere.systemd.services.swarm-secret-publish.script;
in
lib.hasInfix "secret/swarm/hives/h1/matrix/appservice-token" s
&& lib.hasInfix "secret/swarm/hives/h2/matrix/appservice-token" s
&& lib.hasInfix "/dev/urandom" s;
}
{
# What makes a re-publish idempotent. This principal is granted
# `create`/`update` and no `read`, so it cannot ask the store whether a
# hive already has a token — with nowhere to keep one, every run would
# mint a fresh value and rotate the swarm's token. A state directory is
# that somewhere, and nothing else in this unit needs one, so its absence
# means exactly this.
#
# The second arm is the mint's own guard: the state file is only written
# when it is missing or empty. Dropping that test leaves a unit that
# still has a state directory and still rotates on every boot.
name = "the publisher keeps the tokens it minted, and mints only when it holds none";
ok =
let
u = secretPublisherHere.systemd.services.swarm-secret-publish;
in
lib.hasInfix "matrix-appservice-token" (u.serviceConfig.StateDirectory or "")
&& u.serviceConfig.StateDirectoryMode or null == "0700"
&& lib.hasInfix "if [ ! -s \"$src\" ]" u.script;
}
{
# A property of the SET, not of one unit: both of these authenticate by
# certificate, and `BAO_CLIENT_CERT` is transport rather than identity, so