vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

matrix: drop the per-hive appservice and its unused token

Browse source 
Every hive's homeserver loaded a per-hive `hyperhive` appservice
registration, and the swarm minted and delivered an as_token for it
through bao. No client presented that token: swarm-controller creates
every `@hive-<hive>` sender and every agent account with the swarm
appservice token (hive_sender.rs, agent_token.rs), and no Rust code read
the per-hive one.

Removed, with everything that existed only to serve it:
- hive-matrix.nix: the `hyperhive` registration, its activation-time
  render, the host bind mount of /var/lib/hyperhive/matrix-appservice,
  its LoadCredential entry, and the internal options
  `deploy.matrix.appserviceTokenFile` / `appserviceRegistrationScript`.
  tuwunel now loads only the swarm registration.
- glue-matrix-bao-token.nix and its `swarm-bao-matrix-token` unit.
- glue-bao-tls.nix: the matrix-token leaf and its defaults.
- swarm-bao.nix: the `swarm-matrix-token-<hive>` roles and policies
  (`swarm-bao-matrix-token-policy`), and the options
  `deploy.bao.matrixTokenClient{Cert,Key}File` and
  `matrixTokenCommonNamePrefix`, now mkRemovedOptionModule.
- swarm-secret-publisher.nix: the per-hive token mint and its state dir.
- swarm.nix: the reserved `<matrixTokenCommonNamePrefix>-<hive>` CN.
- swarm-secret-client: `matrix::appservice_token_path`.

The swarm registration (swarm-matrix-ctl, swarm-controller) is untouched.
Comments and docs that pointed at the removed glue file now point at
glue-queue-agent-credential.nix, which has the same reader shape.

Refs #4428
This commit is contained in:
atlas 2026-10-04 22:52:22 +02:00 • committed by mara
commit 48067d07e8
31 changed files with 362 additions and 1233 deletions
Download patch file Download diff file Expand all files Collapse all files

1
nix/host-modules/default.nix
View file

@ -33,7 +33,6 @@
./glue-forge-oidc-client.nix
./glue-grafana-oidc-client.nix
./glue-hive-tls-bao-identity.nix
./glue-matrix-bao-token.nix
./glue-matrix-ctl-bao-identity.nix
./glue-nats-auth-bao-identity.nix
./glue-nats-bao-identity.nix

19
nix/host-modules/deploy.nix
View file

@ -257,23 +257,18 @@ in
# Both spellings of the registration token, removed rather than renamed:
# the homeserver no longer accepts a shared registration secret at all,
# so there is no path to point a definition at. Accounts are created by
# the hive's appservice, whose token lives at a fixed path this module
# deliberately does not let anyone move.
# the swarm's appservice.
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "matrix" "registrationTokenFile" ] ''
The matrix registration token is gone: hive-c0re creates accounts as the
hive's appservice instead, so there is no shared secret for tuwunel to
check. Remove this definition — nothing needs replacing, the appservice
token is minted and delivered automatically
(/var/lib/hyperhive/matrix-appservice-token, not operator-settable).
The matrix registration token is gone: swarm-controller creates accounts
as the swarm's appservice instead, so there is no shared secret for
tuwunel to check. Remove this definition — nothing needs replacing.
The old token file at /var/lib/hyperhive/matrix-register-token is read by
nothing now and can be deleted. See docs/integrations/matrix.md.
'')
(lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "matrix" "registrationTokenFile" ] ''
The matrix registration token is gone: hive-c0re creates accounts as the
hive's appservice instead, so there is no shared secret for tuwunel to
check. Remove this definition — nothing needs replacing, the appservice
token is minted and delivered automatically
(/var/lib/hyperhive/matrix-appservice-token, not operator-settable).
The matrix registration token is gone: swarm-controller creates accounts
as the swarm's appservice instead, so there is no shared secret for
tuwunel to check. Remove this definition — nothing needs replacing.
The old token file at /var/lib/hyperhive/matrix-register-token is read by
nothing now and can be deleted. See docs/integrations/matrix.md.
'')

4
nix/host-modules/glue-bao-readers-policy-order.nix
View file

@ -30,10 +30,6 @@ let
# Reader unit → the gate its own module defines it under.
readersHere = {
# ./glue-matrix-bao-token.nix
swarm-bao-matrix-token =
havePair baoDeploy.matrixTokenClientCertFile baoDeploy.matrixTokenClientKeyFile
&& deployCfg.matrix.enable;
# ./glue-queue-agent-credential.nix
swarm-bao-queue-agent =
deployCfg.hive-controller.enable

52
nix/host-modules/glue-bao-tls.nix
View file

@ -86,26 +86,23 @@ in
clientKeyFile = lib.mkDefault "${pkiDir}/client-key.pem";
serverCaFile = lib.mkDefault "${pkiDir}/ca.pem";
# 🩸 Four readers that used to present `client.pem` above, each now
# pointed at a leaf of its own. The leaf is what bao sees, so this pairing
# is the whole of what turns "one principal with the union of four grants"
# into four principals with one grant each — see ./swarm-bao.nix's
# `perHiveReaders` block for the grants themselves.
# 🩸 Readers that used to present `client.pem` above, each now pointed at
# a leaf of its own. The leaf is what bao sees, so this pairing is the
# whole of what turns "one principal with the union of their grants" into
# one principal per grant — see ./swarm-bao.nix's `perHiveReaders` block
# for the grants themselves.
#
# Defaulted here rather than in four `glue-<consumer>-bao-identity.nix`
# files: those exist where the consumer is a CONTAINER with a
# `deploy.<service>.*` namespace of its own to point at. These four are
# host units reading the store, which is the pairing this file already is.
matrixTokenClientCertFile = lib.mkDefault "${pkiDir}/matrix-token.pem";
matrixTokenClientKeyFile = lib.mkDefault "${pkiDir}/matrix-token-key.pem";
# Defaulted here rather than in `glue-<consumer>-bao-identity.nix` files:
# those exist where the consumer is a CONTAINER with a
# `deploy.<service>.*` namespace of its own to point at. These are host
# units reading the store, which is the pairing this file already is.
queueAgentClientCertFile = lib.mkDefault "${pkiDir}/queue-agent.pem";
queueAgentClientKeyFile = lib.mkDefault "${pkiDir}/queue-agent-key.pem";
grafanaOidcClientCertFile = lib.mkDefault "${pkiDir}/grafana-oidc.pem";
grafanaOidcClientKeyFile = lib.mkDefault "${pkiDir}/grafana-oidc-key.pem";
otelOidcClientCertFile = lib.mkDefault "${pkiDir}/otel-oidc.pem";
otelOidcClientKeyFile = lib.mkDefault "${pkiDir}/otel-oidc-key.pem";
# The fifth, the store forwarder's: the reader the four-way split left
# on `client.pem`.
# The store forwarder's.
forwarderOidcClientCertFile = lib.mkDefault "${pkiDir}/forwarder-oidc.pem";
forwarderOidcClientKeyFile = lib.mkDefault "${pkiDir}/forwarder-oidc-key.pem";
@ -196,26 +193,23 @@ in
[ -s ${pkiDir}/matrix-ctl.pem ] || ${signLeaf} ${pkiDir} matrix-ctl \
${lib.escapeShellArg deployCfg.bao.matrixCtlCommonName} "" clientAuth
# 🩸 The four readers that used to present `client.pem` above. Each
# carries its own subject, which is the entire mechanism: bao matches a
# cert-auth role on the CN, so four units sharing one leaf were one
# principal holding the union of four grants, and the union included
# every agent credential in the swarm.
# 🩸 The readers that used to present `client.pem` above. Each carries
# its own subject, which is the entire mechanism: bao matches a
# cert-auth role on the CN, so units sharing one leaf were one principal
# holding the union of their grants, and the union included every agent
# credential in the swarm.
#
# ⚠️ The first two carry THIS hive's name in the subject, unlike the
# three service leaves above. Their grants name one hive's path, because
# a matrix token and a queue credential live under `swarm/hives/<name>/`
# and every hive runs a reader for its own; ./swarm-bao.nix writes one
# role per hive in the swarm directory to match. The two OIDC readers
# need no such segment — a client is registered once per swarm.
# ⚠️ The first carries THIS hive's name in the subject, unlike the three
# service leaves above. Its grant names one hive's path, because a queue
# credential lives under `swarm/hives/<name>/` and every hive runs a
# reader for its own; ./swarm-bao.nix writes one role per hive in the
# swarm directory to match. The OIDC readers need no such segment — a
# client is registered once per swarm.
#
# Minted whether or not the consumer runs here, for the reason the three
# leaves above give: on a hive that does not run the store this is the
# file an operator copies, and a leaf that only appears where its
# consumer does is one nobody can copy from anywhere.
[ -s ${pkiDir}/matrix-token.pem ] || ${signLeaf} ${pkiDir} matrix-token \
${lib.escapeShellArg "${deployCfg.bao.matrixTokenCommonNamePrefix}-${clientCn}"} "" clientAuth
[ -s ${pkiDir}/queue-agent.pem ] || ${signLeaf} ${pkiDir} queue-agent \
${lib.escapeShellArg "${deployCfg.bao.queueAgentCommonNamePrefix}-${clientCn}"} "" clientAuth
@ -265,8 +259,8 @@ in
${lib.optionalString deployCfg.hive-controller.enable ''
# This hive's gateway-leaf identity, which opens
# `pki/issue/<hiveLeafPkiRolePrefix>-<hive>` and nothing else. Its
# subject carries this hive's name, like the matrix-token and
# queue-agent leaves above, because ./swarm-bao.nix writes one role
# subject carries this hive's name, like the queue-agent leaf
# above, because ./swarm-bao.nix writes one role
# per hive. Only where a hive runs; hives on other hosts are issued
# theirs out of band.
[ -s ${pkiDir}/hive-leaf.pem ] || ${signLeaf} ${pkiDir} hive-leaf \

256
nix/host-modules/glue-matrix-bao-token.nix
View file

@ -1,256 +0,0 @@
# Glue: the matrix appservice token comes from the secret store.
#
# The store's first reader, and deliberately a small one. It fetches an opaque
# 32-byte value and writes it where ./hive-matrix.nix already looks, then asks
# that module's own renderer to re-stamp the appservice registration naming it
# — the homeserver never learns the store exists, and its config is unchanged.
#
# ⚠️ Why this credential first. It has no second file and no format: authelia's
# OIDC secret needs a `.secret` *and* a matching `.digest`, so shipping that
# one first would debug "can a reader authenticate and get bytes back" and
# "did we write authelia's file format right" at the same time, with an SSO
# outage as the failure mode. Here the failure is narrow — new agent accounts
# cannot be provisioned, existing ones are untouched, nothing crash-loops.
#
# ⚠️ The fallback is today's behaviour, not a new one. `hive-matrix.nix`'s
# activation script still mints a token when the file is absent; this unit
# overwrites it with the swarm's copy when the store has one. A store that is
# empty or unreachable leaves a working hive with a local token.
#
# 📌 This runs wherever a client identity is configured, NOT only where the
# store is. `deploy.bao.enable` would have been the co-location assumption
# itself; the reader needs a certificate, not a neighbour. On the store's own
# host ./glue-bao-tls.nix supplies one as a `mkDefault` and nothing changes;
# elsewhere an operator places the leaf and names it, and the same unit works.
{
pkgs,
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
baoCfg = hyperhiveCfg.swarm.bao;
baoDeploy = deployCfg.bao;
# What decides whether this unit exists at all. A reader is defined by holding
# a certificate the store accepts, and that is true on the store's own host
# and on a hive three networks away for exactly the same reason.
#
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four
# units used to share. Bao matches a cert-auth role on the CN, so one leaf for
# four readers was ONE principal holding the union of four grants: read on
# `swarm/agents/*` AND `swarm/hives/<hive>/*` AND `swarm/services/*`, when
# this unit reads one appservice token and nothing else. Its own leaf carries
# `<matrixTokenCommonNamePrefix>-<hive>` and its role grants the single path
# below.
haveClientIdentity =
baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null;
# Does this host read the store at all — the hive's own leaf, which is the
# one thing a remote-store deployment has always had to place by hand. Only
# used to decide whether a missing per-principal leaf is a mistake or a
# deployment that has no store: a host holding neither is the supported
# no-store shape, and one holding this pair but not the pair above named
# seven of the eight options and stopped.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
# Where the token lives in the store. A path, not a convention to guess at:
# whoever writes it and whoever reads it must agree, and the agreement
# belongs in one visible place.
#
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
# own grant covers. The store's read policy grants `swarm/agents/*` and
# `swarm/hives/<this hive>/*` and nothing else, so a path outside those is a
# 403 rather than a miss, however correct it looks. `swarm-secret-client`'s
# `matrix::appservice_token_path` builds the same string from the same
# pieces; this literal is the nix half of that one agreement.
#
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
# its own use of it: it is asserted set on every host that runs the
# homeserver.
tokenPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/matrix/appservice-token";
# A literal, not an option — ./hive-matrix.nix names its container
# `containers.hive-matrix` directly and declares no `machine` to derive it
# from, which the trust-bundle call in that file already says out loud.
# ⚠️ A `swarm.matrix.machine` read parses fine and fails at module-system
# resolution, so this is the kind of mistake only reading the target module
# catches.
matrixMachine = "hive-matrix";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
refreshConsumer = import ./lib/refresh-consumer.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
config = lib.mkMerge [
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
# assertion written inside it could never be reached in the state it
# exists to report.
#
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
# same refusal, named to this principal's own pair. What differs is the
# gate. Grafana asserts wherever Grafana runs, because a Grafana with no
# store identity has no way in at all; a homeserver with no store identity
# is a hive that has no store, which is supported. So this one additionally
# requires `hiveReaderIdentity`: the host demonstrably reads the store, and
# named every option but this pair.
(lib.mkIf (deployCfg.matrix.enable && hiveReaderIdentity) {
assertions = [
{
assertion = haveClientIdentity;
message = ''
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
is set) and runs a homeserver, so it needs the matrix appservice
token reader's own client identity: set both
services.hyperhive.deploy.bao.matrixTokenClientCertFile
services.hyperhive.deploy.bao.matrixTokenClientKeyFile
swarm-bao-matrix-token.service fetches this hive's appservice token
out of the store, and without these it is not rendered at all —
leaving the homeserver authenticating hive-c0re against whatever is
already on disk, which is a 401 on every request naming nothing.
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is
the hive's, and its grant reads every secret in the store; this role
reads the one appservice-token path. Pointing this option at the
hive's leaf would evaluate, deploy and log in — and undo the split.
On a hive that runs the store, glue-bao-tls.nix supplies both as
defaults and there is nothing to do. Elsewhere the leaf is issued
from that CA out of band and named here — see docs/swarm/secrets.md.
'';
}
];
})
(lib.mkIf (haveClientIdentity && deployCfg.matrix.enable) {
systemd.services.swarm-bao-matrix-token = {
description = "fetch the matrix appservice token from the swarm secret store";
# Every one of these names a unit that exists only where the store runs.
# `Requires=` on an absent unit fails the job outright, so the ordering is
# conditional even though the read is not: off-host there is nothing local
# to wait for, and the timeout below is what bounds the attempt instead.
after = lib.optionals baoDeploy.enable [
"swarm-bao-pki.service"
"container@${baoCfg.machine}.service"
];
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
before = [ "container@${matrixMachine}.service" ];
wantedBy = [ "container@${matrixMachine}.service" ];
path = [
deployCfg.bao.package
pkgs.coreutils
pkgs.systemd
];
# ./lib/store-retry.nix. This unit is `Before=` the homeserver's
# container, which waits for one attempt and then starts on the token it
# already has; a token a later attempt lands restarts the homeserver
# (below).
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds each attempt below. Stated here rather than
# left to systemd's default, so the number a boot waits on is in
# the file that waits.
TimeoutStartSec = 30;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile;
BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile;
}
# Absent means the system trust store, which is what a deployment with a
# real CA wants and what a self-signed one must not be left with.
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
BAO_CACERT = baoDeploy.serverCaFile;
};
script = ''
set -euo pipefail
${atomicWriteSecret}
${refreshConsumer}
# A sealed or uninitialised store answers on the port and never
# answers the read, so "the store is up" is not the same as "the
# store can answer". `TimeoutStartSec` above bounds each attempt and
# a timed-out attempt is retried like any other failure; the
# homeserver only `Wants=` this unit, so a failed attempt lets it
# start on the local token rather than failing it.
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one
# rather than asserting all three in a sentence of ours — a reader
# that cannot say why it read nothing is indistinguishable from a
# broken one.
err="$(mktemp)"
trap 'rm -f "$err"' EXIT
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
# variables above only decide which certificate the TLS handshake
# presents; without a token `bao` asks its token helper instead, and
# that is a `sh` this unit's `path` does not carry. `-token-only`
# answers on stdout and skips the helper on both sides.
#
# Fails LOUDLY, unlike the read below: the three states a login failure
# covers — store not up, sealed, role not written yet — are all things
# a retry fixes, and `Restart=on-failure` above is what retries. Exiting
# 0 here spends the whole boot on a condition that was seconds old.
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 1
fi
export BAO_TOKEN
if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then
echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2
if [ -s "$err" ]; then
cat "$err" >&2
else
echo "bao failed without writing a diagnostic." >&2
fi
exit 0
fi
if [ -z "$token" ]; then
echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2
exit 0
fi
if secret_differs ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} "$token"; then
atomic_write_secret 0600 "" ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} "$token"
fi
# Re-stamp the registration file from the token on disk. The
# token is half an agreement — the registration the homeserver loads
# has to carry the same value — so writing the file and stopping
# would leave the homeserver authenticating hive-c0re against
# whatever activation put there: a 401 on every request, naming
# nothing. Unconditional rather than on-change, because this unit
# has no way to know what the registration currently says.
#
# hive-matrix's own renderer rather than a `printf` here, so the
# registration's shape has one home.
${deployCfg.matrix.appserviceRegistrationScript}
# tuwunel loads the registration through `LoadCredential`, a copy
# taken at start, while hive-c0re reads the token file on every call:
# a changed token splits the two until the homeserver restarts.
refresh_consumer ${lib.escapeShellArg matrixMachine} tuwunel.service ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)}
'';
};
})
];
}

25
nix/host-modules/glue-queue-agent-credential.nix
View file

@ -1,9 +1,8 @@
# Glue: this hive's agent queue credential comes out of the secret store.
#
# The store's second reader, and deliberately the same shape as its first
# (./glue-matrix-bao-token.nix): a cert login that fails LOUDLY because every
# state it fails on is one a retry fixes, then a read that degrades QUIETLY
# because no retry turns "no value there" into a value.
# A cert login that fails LOUDLY because every state it fails on is one a
# retry fixes, then a read that degrades QUIETLY because no retry turns "no
# value there" into a value.
#
# ⚠️ Two files, not one, and that is the consumer's shape rather than a
# preference. `swarm_queue_client::QueueConfig::from_env` takes the secret as
@ -15,15 +14,17 @@
# that is correct rather than degraded. The publisher runs on the authelia
# host and authelia mints on its first boot, so "nothing at that path yet" is
# the ordinary early state of a swarm. Nothing here writes a local stand-in:
# unlike a matrix appservice token there is no such thing as a locally valid
# OIDC client secret, so a placeholder would turn a hive that cannot connect
# there is no such thing as a locally valid OIDC client secret, so a placeholder would turn a hive that cannot connect
# into one that is refused, which reaches the agent as a timeout.
#
# 📌 This runs wherever a client identity is configured, NOT only where the
# store is — the rule ./glue-matrix-bao-token.nix states in full — and
# wherever this host runs a hive (`deploy.hive-controller.enable`): the
# credential is for the agents hive-c0re creates here at runtime, and the unit
# writes under hive-c0re's own state directory.
# store is. `deploy.bao.enable` would have been the co-location assumption
# itself; the reader needs a certificate, not a neighbour. On the store's own
# host ./glue-bao-tls.nix supplies one as a `mkDefault`; elsewhere an operator
# places the leaf and names it, and the same unit works. It also needs this
# host to run a hive (`deploy.hive-controller.enable`): the credential is for
# the agents hive-c0re creates here at runtime, and the unit writes under
# hive-c0re's own state directory.
{
pkgs,
lib,
@ -55,8 +56,8 @@ let
# one thing a remote-store deployment has always had to place by hand. Only
# used to decide whether a missing per-principal leaf is a mistake or a
# deployment that has no store: a host holding neither is the supported
# no-store shape, and one holding this pair but not the pair above named
# seven of the eight options and stopped.
# no-store shape, and one holding this pair but not the pair above stopped
# short of naming this reader's own.
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
credentialDir = toString deployCfg.hive-controller.queue.agentCredentialDir;

4
nix/host-modules/hive-c0re/default.nix
View file

@ -333,8 +333,8 @@ in
# reachable from an unprivileged daemon without widening either.
# The gate is the identity, never `deploy.bao.enable`: a hive that
# reads a store on another machine holds one of these and runs no
# store. ./glue-matrix-bao-token.nix gates its own reader the same
# way.
# store. ./glue-queue-agent-credential.nix gates its own reader the
# same way.
++ lib.optionals baoClientIdentity [
"bao-client.pem:${baoDeploy.clientCertFile}"
"bao-client-key.pem:${baoDeploy.clientKeyFile}"

2
nix/host-modules/hive-c0re/environment.nix
View file

@ -11,7 +11,7 @@ let
cfg = config.services.hyperhive.c0re;
baoDeploy = config.services.hyperhive.deploy.bao;
baoCfg = config.services.hyperhive.swarm.bao;
# The same test ../glue-matrix-bao-token.nix applies, and for the same
# The same test ../glue-queue-agent-credential.nix applies, and for the same
# reason: a reader is defined by holding a certificate the store accepts,
# not by sharing a host with the store. Deriving this from
# `deploy.bao.enable` would be the co-location assumption itself.

319
nix/host-modules/hive-matrix.nix
View file

@ -50,79 +50,20 @@ let
# visible to the service after sandboxing (ReadWritePaths / ProtectHome),
# typically by placing the file under /etc/tuwunel/" — which this
# container has no writable etc for. `LoadCredential` is the answer
# already in use for the appservice registration below, and for the
# already in use for the swarm appservice registration below, and for the
# same reason: it keeps `DynamicUser=true` + `PrivateUsers=true` intact
# with no host-side chown or GID pinning.
matrixSecretCredential = "/run/credentials/tuwunel.service/oidc_client_secret";
# The hive's appservice registration, whose `url` is null. Null is a legal
# `url` (ruma's `Registration` types it `Option<String>`), and it is the
# whole point — with no URL the homeserver
# never calls out, so there is no HTTP service to run and no daemon to
# operate. tuwunel loads the registration and creates its sender account
# (below). No client presents its `as_token`: `swarm-controller` mints the
# sender's token with the swarm registration's.
appserviceId = "hyperhive";
# The appservice's own user, and the account the hive acts as. An
# ordinary user, not a homeserver admin: everything the hive does with
# it — the Space, the chat room, the invites — it does as the creator of
# those rooms. Loading a registration CREATES its `sender_localpart` user
# when absent (tuwunel `src/service/appservice/mod.rs`), on a zero-user
# database, inside `Services::start()` — before the HTTP listener accepts
# anything, so the account exists on the very first boot of a fresh
# homeserver without hive-c0re racing to register it.
# Where tuwunel reads appservice registrations from: its own credentials
# directory. A registration is a 0600 root-owned file and the homeserver
# runs under `DynamicUser=true`, so `LoadCredential` copies it to a 0400
# dynamic-user-owned path as root, before the sandbox and that user exist.
#
# 🩸 **Derived from the hive name, so it is one account per hive.** It was
# the bare `hive` and a swarm runs one homeserver: every hive on it acted
# as the same `@hive:`, which is one identity with no attribution and no
# way to revoke a single hive. The hives that do not run this container
# register their own `hive-<name>` through the appservice namespace below,
# which already admits it.
#
# ⚠️ Must equal `swarm_secret_client::matrix::hive_localpart`, which
# hive-c0re and swarm-controller both derive from independently with
# nothing wiring an override across — same agreement, and same reason for
# saying so, as the token path below.
#
# `hiveName` has no fallback here for the reason ./glue-matrix-bao-token.nix
# gives at its own use of it: it is asserted set on every host that runs the
# homeserver.
hiveLocalpart = "hive-${toString config.services.hyperhive.hiveName}";
# The `as_token`, and the `hs_token` the spec requires alongside it. Both
# minted by the render script below, mode 0600; the `as_token` is the one
# the swarm secret store overwrites (see
# `glue-matrix-bao-token.nix`). The `hs_token` authenticates the homeserver
# TO the appservice, which with `url = null` is nobody — it exists because
# the registration format requires it.
#
# `appserviceTokenPath` is the single source for the option's `default`,
# its `config`-level `mkDefault` self-definition, and the assertion that
# rejects a moved path. See `appserviceTokenFile`'s own comment below.
appserviceTokenPath = "/var/lib/hyperhive/matrix-appservice-token";
appserviceHsTokenPath = "/var/lib/hyperhive/matrix-appservice-hs-token";
# The registration file, and the directory that holds it. A directory
# rather than a `[global.appservice.<id>]` stanza because a stanza's
# `as_token` would be a nix literal, and a nix literal is a world-readable
# store path — the same rule that sends the OIDC client secret through a
# runtime file. `docs/swarm/secrets.md` has the general form of it.
appserviceDir = "/var/lib/hyperhive/matrix-appservice";
appserviceRegistrationPath = "${appserviceDir}/${appserviceId}.yaml";
# ⚠️ tuwunel does not read the host path directly, for exactly the reason
# given for the two secrets above: the file is 0600 root-owned and the
# homeserver runs under `DynamicUser=true`. `LoadCredential` copies it to a
# 0400 dynamic-user-owned path as root, before the sandbox and that user
# exist.
#
# So `appservice_dir` points at the credentials directory ITSELF. That is
# safe rather than clever: tuwunel's loader takes only `.yaml`/`.yml`
# Safe rather than clever: tuwunel's loader takes only `.yaml`/`.yml`
# entries from the directory and skips every other file, so the sibling
# credentials (the OIDC secret) are invisible to it. A credential id is a
# free-form filename, which is what lets ours end in `.yaml`.
appserviceCredentialId = "${appserviceId}-appservice.yaml";
# free-form filename, which is what lets a registration's end in `.yaml`.
appserviceCredentialDir = "/run/credentials/tuwunel.service";
# ── swarm-matrix-ctl ────────────────────────────────────────────────
@ -147,9 +88,8 @@ let
# derived rather than assumed, because an operator naming paths by hand is
# exactly the deployment the gate above is written for.
#
# Directories rather than the files, for the reason `appserviceDir`'s own
# comment gives: a re-issued leaf is a new inode, and binding the file would
# pin the one the container saw when it started.
# Directories rather than the files: a re-issued leaf is a new inode, and
# binding the file would pin the one the container saw when it started.
ctlPkiDirs = lib.optionals ctlActive (
lib.unique (
map builtins.dirOf (
@ -169,9 +109,9 @@ let
# ── the swarm's own appservice ──────────────────────────────────────
#
# A second registration beside the hive's `hyperhive` one, and the swarm's
# identity on this homeserver: `swarm-controller` creates every agent's
# account with its token. Minted INSIDE this container by
# The homeserver's one appservice registration, and the swarm's identity on
# it: `swarm-controller` creates every agent's account and every hive's
# `@hive-<hive>` sender with its token. Minted INSIDE this container by
# `swarm-matrix-ctl appservice render` and published to the store by
# `appservice publish`, which is why it is gated on matrix-ctl's store
# identity: with nobody to publish it, a registration here would be an
@ -187,8 +127,8 @@ let
swarmAppserviceDir = "/var/lib/swarm-matrix-appservice";
swarmAppserviceCredentialId = "swarm-appservice.yaml";
# Every local user this hive may provision — agents and `@hive-<hive>:`
# itself — which is the whole matrix localpart charset.
# Every local user the swarm appservice may create — agents and every
# hive's `@hive-<hive>:` sender — which is the whole matrix localpart charset.
#
# ⚠️ Anchored deliberately: tuwunel compiles a namespace into a `RegexSet`
# and asks it for a MATCH, not a full match, so an unanchored
@ -201,73 +141,6 @@ let
# create these names, not a monopoly on them.
appserviceUserRegex = "^@[a-z0-9._=/-]+:${lib.escapeRegex effectiveServerName}$";
# Mint the tokens if they are absent, then render the registration from
# whatever they now hold. **One script with two callers** — the activation
# script below and `glue-matrix-bao-token.nix`, which overwrites the
# `as_token` with the swarm store's copy and has to re-render afterwards.
#
# ⚠️ The `as_token` mint here is the FIRST-BOOT FALLBACK, not the normal
# route. `swarm-secret-publisher.nix` mints one per hive swarm-side and
# publishes it to the store; the reader above overwrites this file before
# the homeserver's container starts, so what a hive actually runs on is the
# swarm's value. This keeps a hive that has never reached the store working
# — and it fires only when the file is absent, so it never clobbers one the
# store delivered. (`hs_token` has no swarm half: it is how the homeserver
# authenticates to *us*, so it stays hive-local and is minted here for real.)
# A second copy of the registration's shape would be a second source of
# truth for a format whose mismatch is silent: the homeserver would load a
# registration naming a token nobody holds, and every request hive-c0re
# makes would come back 401 naming nothing.
#
# Re-rendering unconditionally is the point rather than thoroughness: "the
# token file exists" does not mean "the registration carries what is in
# it".
appserviceRegistrationScript = pkgs.writeShellApplication {
name = "hive-matrix-appservice-registration";
runtimeInputs = [ pkgs.coreutils ];
text = ''
# Both the tokens and the rendered registration are secrets; 077
# covers every file this script creates rather than each one
# separately.
umask 077
mkdir -p ${lib.escapeShellArg appserviceDir}
for f in ${lib.escapeShellArg appserviceTokenPath} ${lib.escapeShellArg appserviceHsTokenPath}; do
if [ ! -s "$f" ]; then
head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$f"
echo >> "$f"
echo "hive-matrix: generated appservice token at $f"
fi
chmod 0600 "$f"
done
# Read into shell variables and emitted with `printf`, a shell
# builtin: a token passed as an argument to a real command would land
# in that process's argv, which is world-readable for its lifetime.
asToken="$(cat ${lib.escapeShellArg appserviceTokenPath})"
hsToken="$(cat ${lib.escapeShellArg appserviceHsTokenPath})"
# The quoted heredoc keeps the regex's own `$` and `\` out of the
# shell's hands; the YAML single quotes keep them out of YAML's.
{
cat <<'REGISTRATION'
id: ${appserviceId}
url: null
sender_localpart: ${hiveLocalpart}
rate_limited: false
namespaces:
users:
- exclusive: false
regex: '${appserviceUserRegex}'
aliases: []
rooms: []
REGISTRATION
printf 'as_token: %s\nhs_token: %s\n' "$asToken" "$hsToken"
} > ${lib.escapeShellArg appserviceRegistrationPath}
chmod 0600 ${lib.escapeShellArg appserviceRegistrationPath}
chmod 0700 ${lib.escapeShellArg appserviceDir}
'';
};
# Format-locked by tuwunel, not chosen here: the callback host must point
# directly at the matrix server and the path is fixed at
# `/_matrix/client/unstable/login/sso/callback/<client_id>`. Built once
@ -482,61 +355,6 @@ in
'';
};
appserviceTokenFile = lib.mkOption {
type = lib.types.path;
internal = true;
default = appserviceTokenPath;
description = ''
Host path to a file containing this hive's matrix appservice
token (`as_token`). Minted automatically on first activation
(32-byte random hex, mode 0600) and rendered into the
appservice registration the homeserver loads at boot. Agents
never see it; an agent only ever receives its own
`access_token`.
Not operator-settable — the module's registration renderer reads
this same path as a literal, not through the option,
with nothing wiring an override across, so a moved path desyncs
the two silently. An externally-managed token is delivered by
writing into *this* fixed path instead of moving it — see
`glue-matrix-bao-token.nix`, which fetches from the swarm secret
store, overwrites this file in place, and re-renders the
registration that names it.
Enforced by an `assertions` entry below rather than `readOnly`:
the `config` block gives this option its own `mkDefault`
definition (lowest priority) so a real override still resolves
cleanly instead of crashing eval with nixpkgs' generic
"read-only, set multiple times" message; the assertion then
names the actual problem. (`readOnly` was the first attempt on
this option's predecessor — atlas caught that it only rejects a
*second* definition, so with nothing else defining the option a
lone override sailed through silently, same desync as before
with a lock that wasn't locked. Fixed by defining the value here
instead of leaving it on `default` alone, but a proper
`assertions` message beat re-adding `readOnly` on top once the
option had a real definition either way.)
'';
};
appserviceRegistrationScript = lib.mkOption {
type = lib.types.path;
internal = true;
default = "${appserviceRegistrationScript}/bin/hive-matrix-appservice-registration";
defaultText = lib.literalMD "the module's own registration renderer";
description = ''
The script that mints the appservice tokens when absent and
(re-)renders the registration file from them. An option only so
that `glue-matrix-bao-token.nix` can run the same one after
overwriting the token with the swarm store's copy, rather than
carrying a second copy of the registration's shape.
Not operator-settable, and not a hook: replacing it means
deciding what the homeserver's appservice registration says,
which is this module's job.
'';
};
gui.package = lib.mkOption {
type = lib.types.package;
default = fluffychat-web-fixed;
@ -629,13 +447,6 @@ in
};
config = lib.mkIf deployCfg.matrix.enable {
# The option's own value, defined explicitly rather than left on its
# bare `default` — `mkDefault` so a real override still resolves
# cleanly rather than crashing eval; the `assertions` entry below is
# what actually rejects it, with a message naming the reason. See that
# option's own comment above.
services.hyperhive.deploy.matrix.appserviceTokenFile = lib.mkDefault appserviceTokenPath;
# Matrix's own gateway surface: the sub-domain vhost, the name the
# hive resolver answers for, and the Accept-header map that vhost's
# SPA fallback reads. All three are matrix knowledge and none of
@ -769,25 +580,6 @@ in
httpPort and has no such name.
'';
}
{
# mkDefault above lets an override resolve instead of crashing
# eval — this is the actual rejection, with a message that names
# the real fix instead of nixpkgs' generic conflicting-definition
# text.
assertion = deployCfg.matrix.appserviceTokenFile == appserviceTokenPath;
message = ''
services.hyperhive.deploy.matrix.appserviceTokenFile is fixed at
${appserviceTokenPath} and cannot be moved — the registration
renderer reads this same path as a literal and has no way to learn
an override, so moving it desyncs the two silently instead of
loudly.
Integrating an externally-managed appservice token? Deliver it
by writing into ${appserviceTokenPath} instead of pointing this
option elsewhere — see glue-matrix-bao-token.nix, which does
exactly that from the swarm secret store.
'';
}
];
# One declaration, two readers. The homeserver knows its own callback
@ -829,11 +621,10 @@ in
# permanent stall presenting as "matrix is broken", several layers from
# its cause.
#
# The appservice registration dodges that with an activation script
# that renders the file first. ⚠️ That dodge is NOT available here:
# tuwunel requires the secret file to exist *and be non-empty*, so a
# zero-byte placeholder would satisfy the bind mount and then stop the
# homeserver from starting.
# ⚠️ Rendering a placeholder first is no way around it: tuwunel
# requires the secret file to exist *and be non-empty*, so a zero-byte
# placeholder would satisfy the bind mount and then stop the homeserver
# from starting.
systemd.services.hive-matrix-oidc-secret = lib.mkIf ssoLocal {
description = "deliver the homeserver's OIDC client secret from authelia";
after = [ "container@${autheliaCfg.machine}.service" ];
@ -938,23 +729,6 @@ in
''
);
# Mint the appservice tokens and render the registration, before any
# container start.
#
# An activation script rather than a unit: the directory below is
# bind-mounted into the container and nixos-container refuses to start
# when a bind source is missing, so this has to have run first.
# Activation is what runs before the container both on a switch and on
# every boot. (The registration token this replaced used an activation
# script for the same reason, and additionally to dodge nspawn creating
# an empty file at a missing bind target — which tuwunel then read as
# "no token", refusing every registration until the next restart. A
# missing registration file is not silent in that way: the homeserver
# fails its appservice load loudly.)
system.activationScripts.hive-matrix-appservice = lib.stringAfter [ "var" ] ''
${deployCfg.matrix.appserviceRegistrationScript}
'';
containers.hive-matrix = {
autoStart = true;
ephemeral = false;
@ -962,23 +736,12 @@ in
# --link-journal=try-guest, and EXTRA_NSPAWN_FLAGS expands after it.
extraFlags = [ "--link-journal=host" ];
inherit privateNetwork;
# Read-only bind of the host-managed appservice registration; tuwunel
# reads it via systemd LoadCredential below (not directly).
#
# The directory, not the file inside it: the registration is
# re-rendered rather than edited, and binding the file would pin the
# inode the container saw when it started.
bindMounts = {
${appserviceDir} = {
hostPath = appserviceDir;
isReadOnly = true;
};
}
// caTrust.bindMount
# matrix-ctl's bao client identity, and nothing else of the hive's. See
# `ctlPkiDirs` above for why it is a derived set of directories
# rather than one literal.
// ctlBindMounts;
bindMounts =
caTrust.bindMount
# matrix-ctl's bao client identity, and nothing else of the hive's. See
# `ctlPkiDirs` above for why it is a derived set of directories
# rather than one literal.
// ctlBindMounts;
config =
{ ... }:
{
@ -1061,10 +824,10 @@ in
allow_federation = true;
trusted_servers = deployCfg.matrix.trustedServers;
# Nobody registers themselves here. Accounts are created by
# the hive's appservice, which this flag does not gate —
# the swarm's appservice, which this flag does not gate —
# tuwunel checks it only for requests that arrive WITHOUT an
# appservice token, so hive-c0re provisions exactly as before
# and everyone else is refused outright.
# appservice token, so swarm-controller provisions and
# everyone else is refused outright.
#
# ⚠️ Not a hardening afterthought: `allow_registration = true`
# with no registration token configured makes tuwunel REFUSE
@ -1074,7 +837,7 @@ in
# homeserver, it is a homeserver that does not boot.
allow_registration = false;
# Where the hive's appservice registration is read from — the
# Where the swarm's appservice registration is read from — the
# credentials directory, for the reasons at
# `appserviceCredentialDir`'s own definition. Loaded inside
# `Services::start()`, before the listener accepts anything.
@ -1083,8 +846,8 @@ in
}
# Exactly one account is promoted to homeserver admin at boot:
# the swarm appservice's sender. Its token is read by
# matrix-ctl and swarm-controller only. `@${hiveLocalpart}`
# stays an ordinary user, because every hive reads its
# matrix-ctl and swarm-controller only. Each `@hive-<hive>`
# stays an ordinary user, because its hive reads its
# credential; what it does (the Space, the chat room, the
# invites) rides on being the rooms' creator, not on admin.
#
@ -1161,20 +924,15 @@ in
# host-side chown :tuwunel / GID-pin gymnastics needed.
# See `man systemd.exec` → LoadCredential.
systemd.services.tuwunel.serviceConfig.LoadCredential = [
# The credential id carries a `.yaml` suffix on purpose:
# `appservice_dir` above names this very directory, and tuwunel
# takes only `.yaml`/`.yml` entries from it.
"${appserviceCredentialId}:${appserviceRegistrationPath}"
# Same mechanism, second secret. tuwunel re-reads this file on
# every OAuth exchange, not just at startup, so it has to
# tuwunel re-reads this file on every OAuth exchange, not just at startup, so it has to
# outlive the unit's start — a credentials path does.
"oidc_client_secret:${toString deployCfg.matrix.sso.clientSecretFile}"
]
# The swarm registration, a second `.yaml` in the same directory:
# tuwunel loads every one it finds (`appservice/mod.rs`), and
# refuses only a duplicate `id` or `as_token`, never an
# overlapping namespace. A missing source fails this unit, which
# is why the render below is `requiredBy` it and local only.
# The swarm registration. The credential id carries a `.yaml`
# suffix on purpose: `appservice_dir` above names this very
# directory, and tuwunel takes only `.yaml`/`.yml` entries from
# it. A missing source fails this unit, which is why the render
# below is `requiredBy` it and local only.
++ lib.optional ctlActive "${swarmAppserviceCredentialId}:${swarmAppserviceDir}/swarm.yaml";
# The swarm registration, minted and rendered before the homeserver
@ -1196,9 +954,8 @@ in
environment = {
MATRIX_APPSERVICE_DIR = swarmAppserviceDir;
MATRIX_APPSERVICE_SENDER = swarmSenderLocalpart;
# The hive registration's namespace. Agents' localparts are
# bare `[a-z0-9-]`, so nothing narrower covers them without
# also covering people.
# Agents' localparts are bare `[a-z0-9-]`, so nothing narrower
# covers them without also covering people.
MATRIX_APPSERVICE_USER_REGEX = appserviceUserRegex;
};
};

2
nix/host-modules/hyperhive.nix
View file

@ -156,7 +156,7 @@ in
# Each of these deployments turns `hiveName` into an identifier with no
# fallback: hive-c0re's OIDC client and matrix localpart, the CN of the
# store's `client.pem` (./glue-bao-tls.nix), and the `hives/<name>` path
# the matrix token and queue credential readers fetch. A null there renders
# the queue credential reader fetches. A null there renders
# as an empty string that evaluates and deploys.
config.assertions = [
{

165
nix/host-modules/swarm-bao.nix
View file

@ -755,32 +755,31 @@ let
}
'';
# ── the four principals that used to share the hive's own leaf ─────────────
# ── the principals that used to share the hive's own leaf ──────────────────
#
# 🩸 Each of the four reads exactly ONE path in the store, and until this
# split each did it holding `deploy.bao.clientCertFile` — the hive's own leaf,
# 🩸 Each of these reads exactly ONE path in the store, and until this split
# each did it holding `deploy.bao.clientCertFile` — the hive's own leaf,
# whose policy (`swarm-secret-client`'s `policy::render`) granted read on
# `swarm/agents/*`, `swarm/hives/<hive>/*` AND `swarm/services/*`. Four
# principals presenting one certificate are ONE principal to bao, so the union
# of what the four need was the only grant expressible: the unit that fetches
# Grafana's OIDC client secret could fetch every agent credential in the
# swarm, and the unit that fetches this hive's matrix token could fetch
# Grafana's. Least privilege was not misconfigured, it was unrepresentable.
# `swarm/agents/*`, `swarm/hives/<hive>/*` AND `swarm/services/*`. Principals
# presenting one certificate are ONE principal to bao, so the union of what
# they need was the only grant expressible: the unit that fetches Grafana's
# OIDC client secret could fetch every agent credential in the swarm, and the
# unit that fetches this hive's queue credential could fetch Grafana's. Least
# privilege was not misconfigured, it was unrepresentable.
#
# Each now gets its own leaf, its own cert-auth role and a policy holding the
# one `secret/data/…` path that principal's script actually names — spelled to
# the LEAF rather than to a prefix, for the reason `matrixCtlPolicyText` above
# gives for doing the same.
#
# ⚠️ Two of the four are PER HIVE and two are not, and that asymmetry is the
# shape of the paths rather than a preference. A matrix appservice token and a
# queue credential live under `swarm/hives/<name>/`, and every hive runs a
# ⚠️ The queue-credential reader is PER HIVE and the OIDC readers are not,
# and that asymmetry is the shape of the paths rather than a preference. A
# queue credential lives under `swarm/hives/<name>/`, and every hive runs a
# reader for its own; an OIDC client secret lives under
# `swarm/services/<client-id>/` and a swarm registers each of those exactly
# once. A single role for a per-hive reader would have to name `hives/*` to
# serve every hive — which would let one hive read another's matrix token, a
# reach no hive has today and one this change exists to remove rather than
# create.
# serve every hive — which would let one hive read another's queue
# credential.
readStanza = path: ''
path "${path}" {
capabilities = ["read"]
@ -799,12 +798,6 @@ let
policyText = readStanza (spec.path hiveName);
}) hyperhiveCfg.swarm.hives;
matrixTokenReaders = perHiveReaders {
rolePrefix = "swarm-matrix-token";
cnPrefix = baoDeploy.matrixTokenCommonNamePrefix;
path = h: "${credentialMountPath}/data/swarm/hives/${h}/matrix/appservice-token";
};
queueAgentReaders = perHiveReaders {
rolePrefix = "swarm-queue-agent";
cnPrefix = baoDeploy.queueAgentCommonNamePrefix;
@ -812,7 +805,7 @@ let
};
# Singletons, so a list of one rather than a second shape: the granting unit
# below takes a list either way and the four units stay identical apart from
# below takes a list either way and the units stay identical apart from
# the objects they write.
grafanaOidcReaders = [
{
@ -830,8 +823,7 @@ let
}
];
# The fifth, and the one the four-way split left on the hive's leaf: the
# store's own forwarder. Its path is `forwarderStoreSecretPath` below with
# The store's own forwarder. Its path is `forwarderStoreSecretPath` below with
# KV v2's `data/` inserted.
forwarderOidcReaders = [
{
@ -876,10 +868,10 @@ let
display_name=${lib.escapeShellArg obj.cn}
'';
# One unit per PRINCIPAL, not one unit for the four: these names are
# One unit per PRINCIPAL, not one unit for all of them: these names are
# operator-facing strings, the same reason `swarm-bao-secret-publisher-policy`
# above gives for not being more script inside the controller's unit. A hive
# whose matrix token never arrives is diagnosed by the state of the unit named
# whose queue credential never arrives is diagnosed by the state of the unit named
# after it.
#
# `after` and not `requires`, for the reason the publisher's unit states: the
@ -1224,7 +1216,22 @@ in
and this option named a write grant matrix-ctl no longer holds. Delete
this setting.
'')
];
]
++
map
(
option:
lib.mkRemovedOptionModule [ "services" "hyperhive" "deploy" "bao" option ] ''
The per-hive matrix appservice token and the unit that fetched it are
gone: swarm-controller creates every hive's and every agent's matrix
account with the swarm's appservice token. Delete this setting.
''
)
[
"matrixTokenClientCertFile"
"matrixTokenClientKeyFile"
"matrixTokenCommonNamePrefix"
];
# One service, two namespaces, and the split decides who may set what.
#
@ -1236,7 +1243,8 @@ in
# `serverCaFile` live here too, and they are set by whoever authenticates TO
# the store — which includes a host that runs none of it. A reader is defined
# by holding a certificate the store accepts, never by sharing a host with
# one; ./glue-matrix-bao-token.nix gates on exactly that and not on `enable`.
# one; ./glue-queue-agent-credential.nix gates on exactly that and not on
# `enable`.
# Reading this block as store-runner-only is what makes an off-host reader
# look inexpressible when it is already supported.
#
@ -1629,27 +1637,6 @@ in
'';
};
matrixTokenCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-matrix-token";
example = "swarm-bao-matrix-token.svc";
description = ''
Prefix of the subject the store's per-hive matrix-token roles accept.
The role for hive `h` accepts `<prefix>-h` and nothing else, and grants
read on exactly `swarm/hives/h/matrix/appservice-token`.
A **prefix** rather than one subject because every hive runs its own
reader and the path it reads carries that hive's name. One subject for
all of them would need a `hives/*` grant, which would let one hive read
another hive's appservice token — a reach no hive has today.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
fixed-subject siblings are: cert auth trusts the CA and
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
carried one of these spellings would receive that grant.
'';
};
queueAgentCommonNamePrefix = lib.mkOption {
type = lib.types.str;
default = "swarm-bao-queue-agent";
@ -1660,10 +1647,15 @@ in
exactly `swarm/hives/h/queue/agent` — the client credential that hive's
agent containers authenticate to the swarm queue with.
Per hive for the same reason
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is,
and more sharply: every hive in the swarm runs this reader, so a single
subject would have to be granted `hives/*`.
A **prefix** rather than one subject because every hive runs its own
reader and the path it reads carries that hive's name. One subject for
all of them would need a `hives/*` grant, which would let one hive read
another hive's queue credential.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, for the reason its
fixed-subject siblings are: cert auth trusts the CA and
`allowed_common_names` is the whole narrowing, so a hive whose own leaf
carried one of these spellings would receive that grant.
'';
};
@ -1678,7 +1670,7 @@ in
for `h`'s domain and `*.` that domain and nothing else.
Per hive for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenCommonNamePrefix` is:
{option}`services.hyperhive.deploy.bao.queueAgentCommonNamePrefix` is:
one subject for every hive could ask for any hive's certificate.
⚠️ Reserved as a hive name by ./swarm.nix, per hive, like its siblings.
@ -1725,7 +1717,7 @@ in
presents. Its grant is one path,
`swarm/services/<collector client id>/oidc/client`, and read only.
A **fourth** identity rather than reuse of
Its **own** identity rather than reuse of
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`: the two
read different services' client secrets, and a collector is not
entitled to Grafana's.
@ -1742,46 +1734,13 @@ in
own forwarder's OIDC client secret, presents. Its grant is one path,
`swarm/services/<forwarder client id>/oidc/client`, and read only.
A **fifth** identity rather than reuse of
Its **own** identity rather than reuse of
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`: the two
read different clients' secrets, and the store's forwarder is not
entitled to the swarm collector's.
'';
};
matrixTokenClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/matrix-token.pem";
description = ''
Certificate the unit that fetches this hive's matrix appservice token
presents to the store. Its subject must be
`<matrixTokenCommonNamePrefix>-<this hive's name>`; cert auth matches on
the CN and the role accepts nothing else.
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one token a credential that reads every
secret in the store. That is what this option exists to end.
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
wherever this host mints its own PKI. A deployment that reads a store on
another host names it by hand, alongside the leaf itself — one file per
principal placed out of band instead of one shared by four.
'';
};
matrixTokenClientKeyFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
example = "/var/lib/swarm-bao-pki/matrix-token-key.pem";
description = ''
Private key for
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile`. Both
or neither — a certificate with no key authenticates nothing.
'';
};
queueAgentClientCertFile = lib.mkOption {
type = lib.types.nullOr lib.types.str;
default = null;
@ -1789,10 +1748,17 @@ in
description = ''
Certificate the unit that fetches this hive's queue client credential
presents to the store. Its subject must be
`<queueAgentCommonNamePrefix>-<this hive's name>`.
`<queueAgentCommonNamePrefix>-<this hive's name>`; cert auth matches on
the CN and the role accepts nothing else.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
⚠️ **Not** {option}`services.hyperhive.deploy.bao.clientCertFile`.
Pointing this at the hive's own leaf would evaluate, deploy and log in —
and hand a unit that reads one credential a certificate that reads every
secret in the store.
No default here: ./glue-bao-tls.nix points it at the leaf it mints,
wherever this host mints its own PKI. A deployment that reads a store on
another host names it by hand, alongside the leaf itself.
'';
};
@ -1816,7 +1782,7 @@ in
{option}`services.hyperhive.deploy.bao.grafanaOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
'';
};
@ -1840,7 +1806,7 @@ in
{option}`services.hyperhive.deploy.bao.otelOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
'';
};
@ -1864,7 +1830,7 @@ in
{option}`services.hyperhive.deploy.bao.forwarderOidcCommonName`.
⚠️ Not the hive's own leaf, for the reason
{option}`services.hyperhive.deploy.bao.matrixTokenClientCertFile` gives.
{option}`services.hyperhive.deploy.bao.queueAgentClientCertFile` gives.
'';
};
@ -2924,13 +2890,11 @@ in
'';
};
# 🩸 The four principals that used to log in as the hive. See the block
# around `perHiveReaders` for what each grant is and why the first two are
# per hive; the four units are identical apart from the objects they
# 🩸 The principals that used to log in as the hive. See the block around
# `perHiveReaders` for what each grant is and why the first is per hive;
# the units are identical apart from the objects they
# write, so the shape lives in `readerPolicyUnit` and the difference is
# visible here in one line each.
systemd.services.swarm-bao-matrix-token-policy = readerPolicyUnit "write the per-hive matrix-token bao policies and cert-auth roles" matrixTokenReaders;
systemd.services.swarm-bao-queue-agent-policy = readerPolicyUnit "write the per-hive queue-credential bao policies and cert-auth roles" queueAgentReaders;
systemd.services.swarm-bao-grafana-oidc-policy = readerPolicyUnit "write Grafana's OIDC-secret-reader bao policy and cert-auth role" grafanaOidcReaders;
@ -3139,7 +3103,6 @@ in
"swarm-bao-controller-policy.service"
"swarm-bao-secret-publisher-policy.service"
"swarm-bao-matrix-ctl-policy.service"
"swarm-bao-matrix-token-policy.service"
"swarm-bao-queue-agent-policy.service"
"swarm-bao-grafana-oidc-policy.service"
"swarm-bao-otel-oidc-policy.service"

2
nix/host-modules/swarm-grafana.nix
View file

@ -206,7 +206,7 @@ let
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full.
# ./glue-queue-agent-credential.nix states in full.
#
# Also asserted rather than gating anything, and that is the ruling this
# module's second delivery route was deleted under: the store exists so that

5
nix/host-modules/swarm-otel.nix
View file

@ -241,11 +241,10 @@ let
# A reader of the store is defined by holding a certificate the store
# accepts, never by standing next to it — the rule
# ./glue-matrix-bao-token.nix states in full. Unlike Grafana's identical-
# ./glue-queue-agent-credential.nix states in full. Unlike Grafana's identical-
# looking flag, this one does not gate an assertion on its own: the
# store-reading unit below simply does not render without it, the same choice
# ./glue-matrix-bao-token.nix and ./glue-queue-agent-credential.nix make for
# their own readers, because a collector with no client identity is
# ./glue-queue-agent-credential.nix makes for its own reader, because a collector with no client identity is
# `haveCollectorSecret = false` above, and that is already a supported,
# merely degraded shape rather than a service with no way in at all. What
# IS asserted is narrower and lives in `assertions` below — see

82
nix/host-modules/swarm-secret-publisher.nix
View file

@ -1,7 +1,7 @@
# The unit that puts swarm-level secrets into the swarm's secret store, so
# whoever needs one can read it there: a hive its agents' credential, a
# service or the controller its own, a hive its appservice token. OIDC secrets
# it only copies; the appservice token it MINTS, having no swarm-side producer.
# service or the controller its own. It mints nothing: every value is an OIDC
# client secret authelia minted, and this unit copies it.
#
# ⚠️ "Whoever", including a reader on THIS host. A swarm service's secret goes
# into the store even when the service runs beside authelia, because its
@ -21,13 +21,11 @@
# ⚠️ ITS OWN STORE IDENTITY, not swarm-controller's. That principal may rewrite
# every hive's policy and login role; a unit that writes a handful of secrets
# has no business holding it. ./swarm-bao.nix grants this one `create`/`update`
# under the hive prefix and nothing else — no `read`, which is why the mint
# below keeps its own record of what it published.
# under the hive prefix and nothing else — no `read`.
#
# ⚠️ THE SECRET NEVER REACHES `argv`. `bao` is an external binary, so every
# argument is world-readable in /proc for the life of the call — the value is
# passed as `@<path>` and read by bao itself. Same rule governs the mint: it
# redirects into a file and never holds the token in a shell variable.
# passed as `@<path>` and read by bao itself.
{
pkgs,
lib,
@ -41,7 +39,8 @@ let
cfg = deployCfg.swarm-secret-publisher;
# A reader is defined by holding a certificate the store accepts, never by
# standing next to it — the rule ./glue-matrix-bao-token.nix states in full.
# standing next to it — the rule ./glue-queue-agent-credential.nix states in
# full.
# `deploy.bao.enable` here would be the co-location assumption itself.
haveClientIdentity = cfg.baoClientCertFile != null && cfg.baoClientKeyFile != null;
@ -86,21 +85,6 @@ let
# under its own certificate.
controllerClientId = hyperhiveCfg.swarm.controller.queueClientId;
controllerSecretPath = "secret/swarm/controller/swarm-controller/oidc/client";
# Where this unit keeps the appservice tokens it minted, and the whole reason
# a re-publish is idempotent. The store cannot be that record: ./swarm-bao.nix
# grants this principal `create`/`update` and deliberately no `read`, so "does
# this hive already have a token?" is a question it can only ask locally.
# Without the file every run would mint a fresh value and rotate the swarm's
# token for no reason — and a rotation is not free, because the homeserver
# only reloads the registration naming it when its container restarts.
#
# `StateDirectory=` below is what creates it, so the name there is relative
# and this unit owns the tree. Losing it mints once more and republishes;
# readers pick that up on their next start, and nothing that already
# registered breaks — the token authenticates the appservice, it does not
# live inside any account it created.
matrixTokenDir = "/var/lib/swarm-secret-publish/matrix-appservice-token";
in
{
options.services.hyperhive.deploy.swarm-secret-publisher = {
@ -112,9 +96,7 @@ in
Publish the OIDC client secrets this host mints into the swarm's
secret store, so a hive that does not run authelia can read its
agents' credential and a swarm service or the controller can read its
own — from wherever it runs, this host included. Also mints each hive's matrix
appservice token, which has no other swarm-side producer, and
publishes it the same way.
own — from wherever it runs, this host included.
Defaults to whether this host mints the OIDC secrets, which is the
only half of the question that is a property of *this* host.
@ -128,9 +110,8 @@ in
Turning it off leaves every hive but this one without its agents'
credential, the swarm controller unable to start, the swarm's Grafana
without any login at all, its collector pushing unauthenticated, and
every hive falling back to the appservice token its own first boot
minted — so two hives never agree on one. Each secret has exactly one route and this is the producer's
without any login at all, and its collector pushing unauthenticated.
Each secret has exactly one route and this is the producer's
end of it, so the honest reason to set it false is a deployment
delivering those secrets by some other mechanism it owns.
'';
@ -190,12 +171,6 @@ in
TimeoutStartSec = 60;
Restart = "on-failure";
RestartSec = 30;
# The minted appservice tokens, see `matrixTokenDir` above. Named
# relative to /var/lib because that is the form systemd creates; the
# mode is narrowed from systemd's 0755 default because the directory
# listing is the roster of hives and every file in it is a secret.
StateDirectory = "swarm-secret-publish/matrix-appservice-token";
StateDirectoryMode = "0700";
};
environment = {
BAO_ADDR = "https://${hyperhiveCfg.swarm.bao.domain}:${toString hyperhiveCfg.swarm.bao.port}";
@ -222,7 +197,6 @@ in
published=0
skipped=0
tokens=0
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${deployCfg.authelia.hostClientSecretDir}/${agentClientId hive}.secret"}
@ -281,43 +255,7 @@ in
skipped=$((skipped + 1))
fi
# The matrix appservice token, per hive. Unlike the two loops above
# there is nothing to copy: authelia never minted this one, and each
# homeserver's own host minted its own — which is exactly why two hives
# never converged on a value the swarm has to agree on. The mint moves
# here because this unit already holds a store identity and already
# writes under the hive prefix; ../host-modules/hive-matrix.nix keeps
# its activation mint as the first-boot fallback for a hive that has
# never reached the store.
#
# 32 bytes of urandom as lowercase hex, the shape that file has
# generated all along. Written with no trailing newline, unlike that
# file: what goes into the store is the value, and
# ./glue-matrix-bao-token.nix adds the newline the hive's token file
# carries.
${lib.concatMapStringsSep "\n" (hive: ''
src=${lib.escapeShellArg "${matrixTokenDir}/${hive}"}
if [ ! -s "$src" ]; then
# The subshell keeps the umask local — the loops above rely on
# bao's own file handling and should not inherit one.
( umask 077; head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > "$src" )
echo "minted a matrix appservice token for ${hive}" >&2
fi
# `value=@$src` for the same reason as both loops above: bao opens
# the file itself, so the token is never an argument of this process.
#
# Put on every run, not only on the run that minted — a put of the
# value already there changes nothing for any reader, while a mint
# whose publish failed must not be left as a token this host holds
# and no hive can reach. The `-s` test above is what keeps the VALUE
# stable across those runs: rotating on every boot would hand the
# homeserver a registration its running container does not carry.
bao kv put ${lib.escapeShellArg "secret/swarm/hives/${hive}/matrix/appservice-token"} \
value=@"$src"
tokens=$((tokens + 1))
'') hiveNames}
echo "published $published client secret(s), skipped $skipped, and $tokens matrix appservice token(s)"
echo "published $published client secret(s), skipped $skipped"
'';
};
};

1
nix/host-modules/swarm.nix
View file

@ -63,7 +63,6 @@ let
# Every spelling is reserved for every declared hive, including the one the
# CN belongs to — a hive cannot be named after its own reader either.
++ lib.concatMap (hive: [
"${deployCfg.bao.matrixTokenCommonNamePrefix}-${hive}"
"${deployCfg.bao.queueAgentCommonNamePrefix}-${hive}"
"${deployCfg.bao.hiveLeafCommonNamePrefix}-${hive}"
]) hiveNames;