vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm: OIDC clients registered wherever authelia runs; nats + matrix secrets from bao

Browse source 
The three client registrations move into glue-{controller,nats,matrix}-oidc-client.nix,
gated only on deploy.authelia.enable and reading their ids from swarm.* options,
so the host running authelia registers every one of them whichever host runs
the service. The matrix callback is a new read-only swarm.matrix.sso.redirectUri,
read by both the homeserver's config and its client entry.

swarm-secret-publish now also publishes the queue's and the homeserver's minted
secrets to swarm/services/<id>/oidc/client. The queue host reads its responder's
secret there with swarm-bao-nats-auth-oidc under the responder's own leaf
(swarm-nats-auth role gains that one read path), and the homeserver's host with
swarm-bao-matrix-oidc under matrix-ctl's leaf (swarm-matrix-ctl role gains its
one read path). The host-file copies are gone: swarm-nats-auth-secrets now
delivers only the callout seeds and the store identity, and
hive-matrix-oidc-secret is removed. A fetch unit whose leaf is unset still
renders and fails naming the option pair.

Closes #4943
This commit is contained in:
atlas 2026-10-05 02:56:18 +02:00 • committed by mara
commit 962da3b784
20 changed files with 702 additions and 293 deletions
Download patch file Download diff file Expand all files Collapse all files

225
nix/host-modules/hive-matrix.nix
View file

@ -33,18 +33,25 @@ let
baoCfg = config.services.hyperhive.swarm.bao;
autheliaUrl = autheliaCfg.url;
# The all-local case: this host runs BOTH the homeserver and the swarm's
# authelia, so the secret can be moved without an operator. The other
# two cases (swarm side, remote hive) leave `clientSecretFile` to be set
# explicitly — same split the forge module documents.
ssoLocal = deployCfg.authelia.enable;
# Where the plaintext lands inside the matrix container. Under /var/lib
# rather than /run: the homeserver may start before the delivery unit on
# a later boot, and a secret that evaporates on reboot turns a working
# login into an intermittent one.
matrixSecretPath = "/var/lib/tuwunel-oidc/${cfg.sso.clientId}.secret";
# The same file seen from the host, where `swarm-bao-matrix-oidc` writes it.
# Follows `clientSecretFile`, so an operator-chosen path is where the
# fetched secret lands.
hostSecretPath = "/var/lib/nixos-containers/hive-matrix${toString deployCfg.matrix.sso.clientSecretFile}";
# Where swarm-secret-publish on authelia's host leaves this client's secret.
# matrix-ctl's store role grants read on this one path.
storeSecretPath = "secret/swarm/services/${cfg.sso.clientId}/oidc/client";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
# ⚠️ tuwunel does NOT read the host path directly, and this indirection
# is not ceremony. Upstream's own words: "under systemd the path must be
# visible to the service after sandboxing (ReadWritePaths / ProtectHome),
@ -141,13 +148,6 @@ let
# create these names, not a monopoly on them.
appserviceUserRegex = "^@[a-z0-9._=/-]+:${lib.escapeRegex effectiveServerName}$";
# Format-locked by tuwunel, not chosen here: the callback host must point
# directly at the matrix server and the path is fixed at
# `/_matrix/client/unstable/login/sso/callback/<client_id>`. Built once
# and read twice — by the homeserver's own config and by the client entry
# handed to authelia — because a redirect-URI mismatch is a rejected
# login with no error text worth reading.
ssoCallbackUrl = "https://${toString cfg.gatewayHost}/_matrix/client/unstable/login/sso/callback/${cfg.sso.clientId}";
# Falls back to the SWARM domain: a swarm runs one homeserver, so its
# identifier belongs to the swarm rather than to whichever hive happens
# to host it — otherwise moving the container between hives would look
@ -385,13 +385,11 @@ in
needs the plaintext), and a literal written here would be
rendered into the world-readable nix store.
Deliberately no fallback — a homeserver that boots with SSO
half-configured is worse than one that fails to evaluate:
tuwunel reads OIDC from its config file rather than a database
row, so a malformed block can stop the server outright instead
of merely hiding a button. The id this pairs with is
`swarm.matrix.sso.clientId`, which is swarm-wide because it must
match authelia's register.
Set to `/var/lib/tuwunel-oidc/<client id>.secret` wherever the
homeserver runs. `swarm-bao-matrix-oidc` writes the secret here
from the swarm secret store, whichever path this names. The id
this pairs with is `swarm.matrix.sso.clientId`, which is
swarm-wide because authelia's host registers it.
'';
};
@ -559,9 +557,9 @@ in
(inside the matrix container) holding the OIDC client secret's
plaintext.
On a hive that also runs the swarm's authelia this is wired up
for you. Set it explicitly when authelia lives on another
host: see docs/swarm/ for which secret goes where.
It is set by default wherever the homeserver runs, and
swarm-bao-matrix-oidc writes the secret there from the swarm
secret store. Remove the definition that set it to null.
'';
}
{
@ -582,95 +580,114 @@ in
}
];
# One declaration, two readers. The homeserver knows its own callback
# URL; making the operator restate it in authelia's client list would
# be a second source of truth for a string whose mismatch is a silent
# rejected login.
services.hyperhive.swarm.authelia.oidc.clients = lib.mkIf ssoLocal [
{
id = cfg.sso.clientId;
description = "HyperHive matrix";
redirectUris = [ ssoCallbackUrl ];
# tuwunel authenticates at the token endpoint by putting the
# secret in the POST body. Authelia enforces the *registered*
# method rather than accepting whichever one arrives, and its
# default is `client_secret_basic` — so without this the browser
# flow completes, consent is granted, and the very last hop fails
# with a 401 that names neither the secret nor the redirect.
tokenEndpointAuthMethod = "client_secret_post";
}
];
# Where the homeserver finds its OIDC client secret, inside the container.
services.hyperhive.deploy.matrix.sso.clientSecretFile = lib.mkDefault matrixSecretPath;
# Same case, same reasoning: this host minted the secret, so it can say
# where the homeserver will find it.
services.hyperhive.deploy.matrix.sso.clientSecretFile = lib.mkIf ssoLocal (
lib.mkDefault matrixSecretPath
);
# The delivery. It runs on the HOST because that is the only place both
# container trees are addressable: they share this host's network
# namespace, which makes them feel co-located, but their filesystem
# roots are separate — the homeserver cannot open a path inside
# authelia's tree however local the port looks.
# The homeserver's OIDC client secret, out of the swarm secret store under
# matrix-ctl's leaf — the one route, wherever authelia runs. Same shape as
# ./swarm-grafana.nix's `swarm-bao-grafana-oidc`: a login that fails
# loudly, since a retry fixes every state it fails on, then a read that
# degrades quietly, since an absent value is "not published yet".
#
# ⚠️ Deliberately a copy and not a `bindMounts` entry.
# nixos-container refuses to start when a bind source is missing, and
# this secret does not exist until authelia's first boot has minted it
# — so binding it would make the homeserver wait on a file that waits
# on a container that starts after it. On a fresh hive that is a
# permanent stall presenting as "matrix is broken", several layers from
# its cause.
#
# ⚠️ Rendering a placeholder first is no way around it: tuwunel
# requires the secret file to exist *and be non-empty*, so a zero-byte
# placeholder would satisfy the bind mount and then stop the homeserver
# from starting.
systemd.services.hive-matrix-oidc-secret = lib.mkIf ssoLocal {
description = "deliver the homeserver's OIDC client secret from authelia";
after = [ "container@${autheliaCfg.machine}.service" ];
requires = [ "container@${autheliaCfg.machine}.service" ];
# ⚠️ tuwunel requires the file to exist and be non-empty, and its
# `LoadCredential` source must exist for the unit to start, so a host with
# no matrix-ctl leaf fails here, naming the options, rather than rendering
# no unit.
systemd.services.swarm-bao-matrix-oidc = {
description = "fetch the homeserver's OIDC client secret from the swarm secret store";
# The store's units exist only where the store runs; `Requires=` on an
# absent unit fails the job, so off-host the timeout below bounds the
# attempt instead.
after = lib.optionals deployCfg.bao.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals deployCfg.bao.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals deployCfg.bao.enable [ "swarm-bao-pki.service" ];
before = [ "container@hive-matrix.service" ];
wantedBy = [ "container@hive-matrix.service" ];
serviceConfig = {
wantedBy = [
"multi-user.target"
"container@hive-matrix.service"
];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. The container waits for one attempt; a secret a
# later attempt lands restarts tuwunel (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "hive-matrix-oidc-secret";
# Longer than the 120s bounded wait below, and that is the whole
# point: `DefaultTimeoutStartSec` is 90s, so without this systemd
# kills the unit at 90 — before it can emit the message naming the
# file it was waiting for. The failure then reads as a timeout with
# no cause rather than "authelia has not minted <path>", which is
# the one line that makes a fresh-hive SSO stall diagnosable.
TimeoutStartSec = "180s";
SyslogIdentifier = "swarm-bao-matrix-oidc";
TimeoutStartSec = 30;
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
}
// lib.optionalAttrs ctlActive {
BAO_CLIENT_CERT = deployCfg.matrix.ctlBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.matrix.ctlBaoClientKeyFile;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
script =
if !ctlActive then
''
echo "the homeserver's host has no matrix-ctl store identity, so it cannot read the homeserver's OIDC client secret: set services.hyperhive.deploy.matrix.ctlBaoClientCertFile and ctlBaoClientKeyFile" >&2
exit 1
''
else
''
set -euo pipefail
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${cfg.sso.clientId}.secret"}
dst=${lib.escapeShellArg "/var/lib/nixos-containers/hive-matrix${toString deployCfg.matrix.sso.clientSecretFile}"}
${atomicWriteSecret}
${refreshConsumer}
# authelia's container is up, but its first-boot generator may
# still be minting. Bounded wait, then fail: a silent skip here
# produces a homeserver whose SSO login dead-ends, which is the
# failure this whole design is trying not to ship.
for _ in $(seq 1 60); do
[ -s "$src" ] && break
sleep 2
done
if [ ! -s "$src" ]; then
echo "authelia has not minted $src after 120s" >&2
exit 1
fi
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# root-owned 0400, and deliberately NOT the forge's `stat -c %u`
# uid discovery: that reads the service's state dir to learn which
# uid to hand the file to, and tuwunel runs under `DynamicUser`, so
# there is no stable uid to discover. It never reads this path
# directly anyway — `LoadCredential` does, as root, before the
# sandbox and the dynamic user exist.
install -D -m 0400 -o root -g root "$src" "$dst"
'';
# `-token-only` answers on stdout and skips the token helper, a
# `sh` this unit's `path` does not carry.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with matrix-ctl's certificate; leaving the homeserver's OIDC client secret as it is." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! secret="$(bao kv get -field=value ${lib.escapeShellArg storeSecretPath} 2>"$err")"; then
echo "swarm-bao did not return ${storeSecretPath}; the homeserver has no OIDC client secret yet." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$secret" ]; then
echo "swarm-bao returned an empty ${storeSecretPath}; leaving the file as it is." >&2
exit 0
fi
# Root-owned 0400: tuwunel runs under `DynamicUser`, so there is no
# stable uid to hand the file to, and `LoadCredential` reads it as
# root before the sandbox exists.
install -d -m 0700 ${lib.escapeShellArg (builtins.dirOf hostSecretPath)}
if secret_differs ${lib.escapeShellArg hostSecretPath} "$secret"; then
atomic_write_secret 0400 "" ${lib.escapeShellArg hostSecretPath} "$secret"
fi
# `LoadCredential` copies the file once, at tuwunel's start.
refresh_consumer hive-matrix tuwunel.service ${lib.escapeShellArg hostSecretPath}
'';
};
# ⚠️ Deliberately NO `networking.hosts` entry for authelia's name. This
@ -887,7 +904,7 @@ in
client_id = cfg.sso.clientId;
client_secret_file = matrixSecretCredential;
issuer_url = toString autheliaUrl;
callback_url = ssoCallbackUrl;
callback_url = cfg.sso.redirectUri;
# Explicit, though a lone provider is auto-defaulted:
# relying on that logs a warning every startup, and a
# recurring warning that is expected is one nobody reads.