vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

matrix: swarm-controller is the only minter

Browse source 
Every hive is in a swarm and every swarm runs matrix, so every swarm has a
swarm-controller, and since #4810 its hive_sender pass mints each hive's
@hive-<hive>: sender token into the store every five minutes. The two
other minters of that token go:

- swarm-matrix-ctl mint: the systemd.services.swarm-matrix-ctl unit in the
  hive-matrix container, Command::Mint and src/mint.rs. The binary, its
  appservice render/publish verbs, ctlPackage, ctlActive and the ctl cert
  role stay. bao-matrix-reader's checks on the deleted unit are removed;
  the leaf-identity and no-token-in-env checks now look at
  swarm-matrix-appservice-publish, which runs under the same identity.
- the hive-side mint ladder in hive-c0re's ensure_hive_user
  (register/appservice-login/password-login with the local as_token), with
  read_appservice_token, paths::matrix_appservice_token and the helpers
  only it used. ensure_hive_user now takes the store's token, keeps the
  file when the store has none or can't be reached, and fails otherwise.
- hivectl matrix sync-admin: the verb, HostRequest::MatrixSyncAdmin and
  handle_matrix_sync_admin. The periodic MatrixSweep (ensure_all) is
  unchanged apart from no longer reading the local as_token.

This removes the double-mint race #4810's review flagged: two minters
logging in on one pinned device could leave a dead token in the store
until the next pass.

Closes #4813
Closes #4814
This commit is contained in:
atlas 2026-09-29 23:49:57 +02:00 • committed by mara
commit ddb7d7196d
22 changed files with 187 additions and 1162 deletions
Download patch file Download diff file Expand all files Collapse all files

115
nix/host-modules/hive-matrix.nix
View file

@ -81,7 +81,7 @@ let
# which already admits it.
#
# ⚠️ Must equal `swarm_secret_client::matrix::hive_localpart`, which
# hive-c0re and swarm-matrix-ctl both derive from independently with
# hive-c0re and swarm-controller both derive from independently with
# nothing wiring an override across — same agreement, and same reason for
# saying so, as the token path below.
#
@ -92,7 +92,7 @@ let
# The `as_token`, and the `hs_token` the spec requires alongside it. Both
# minted by the render script below, mode 0600; the `as_token` is the one
# hive-c0re reads and the one the swarm secret store overwrites (see
# the swarm secret store overwrites (see
# `glue-matrix-bao-token.nix`). The `hs_token` authenticates the homeserver
# TO the appservice, which with `url = null` is nobody — it exists because
# the registration format requires it.
@ -127,18 +127,12 @@ let
# ── swarm-matrix-ctl ────────────────────────────────────────────────
#
# The oneshot that publishes the appservice sender account's access token to
# the swarm's secret store. It runs INSIDE the container, beside tuwunel,
# because the appservice token that authorises the mint is already in here —
# `appserviceDir` below is bound read-only precisely so the homeserver can
# load it — and minting anywhere else would create a second holder of that
# secret, which is the thing this whole arrangement exists to stop.
#
# Gated on the identity, not on `deploy.bao.enable`: a swarm's ONE homeserver
# is the host least likely to also be the host running the store, so
# "co-located with bao" would leave the intended deployment silently minting
# nothing. Same rule ./swarm-secret-publisher.nix's `haveClientIdentity`
# states, for a sharper reason.
# The container's own store identity, which the swarm appservice units below
# run under. Gated on the identity, not on `deploy.bao.enable`: a swarm's ONE
# homeserver is the host least likely to also be the host running the store,
# so "co-located with bao" would leave the intended deployment silently
# publishing nothing. Same rule ./swarm-secret-publisher.nix's
# `haveClientIdentity` states, for a sharper reason.
ctlActive =
deployCfg.matrix.ctlBaoClientCertFile != null && deployCfg.matrix.ctlBaoClientKeyFile != null;
@ -179,9 +173,9 @@ let
# identity on this homeserver: `swarm-controller` creates every agent's
# account with its token. Minted INSIDE this container by
# `swarm-matrix-ctl appservice render` and published to the store by
# `appservice publish`, which is why it is gated on the same identity as
# the sender mint: with nobody to publish it, a registration here would be
# an admin credential nobody reads.
# `appservice publish`, which is why it is gated on matrix-ctl's store
# identity: with nobody to publish it, a registration here would be an
# admin credential nobody reads.
#
# Its sender is promoted to homeserver admin at boot (`admin_execute`
# below), so its token goes only to a store path no hive's policy reaches.
@ -193,13 +187,6 @@ let
swarmAppserviceDir = "/var/lib/swarm-matrix-appservice";
swarmAppserviceCredentialId = "swarm-appservice.yaml";
# Where a reader of the published credential is told the token is good for.
# Empty when this hive serves no vhost: `matrix::Credential.homeserver` is an
# `Option`, and matrix-ctl reads an empty variable as absent rather than as
# the string "null" — which is what a hive with no gateway host actually
# knows about itself.
ctlHomeserverUrl = if cfg.gatewayHost == null then "" else "https://${toString cfg.gatewayHost}";
# Every local user this hive may provision — agents and `@hive-<hive>:`
# itself — which is the whole matrix localpart charset.
#
@ -677,15 +664,14 @@ in
default = appserviceTokenPath;
description = ''
Host path to a file containing this hive's matrix appservice
token (`as_token`) — the identity `hive-c0re` creates and logs
into accounts with. Minted automatically on first activation
token (`as_token`). Minted automatically on first activation
(32-byte random hex, mode 0600) and rendered into the
appservice registration the homeserver loads at boot. Agents
never see it; an agent only ever receives its own
`access_token`.
Not operator-settable — `hive-c0re`'s Rust side derives this
same path independently (`paths::matrix_appservice_token()`)
Not operator-settable — the module's registration renderer reads
this same path as a literal, not through the option,
with nothing wiring an override across, so a moved path desyncs
the two silently. An externally-managed token is delivered by
writing into *this* fixed path instead of moving it — see
@ -1015,8 +1001,8 @@ in
assertion = deployCfg.matrix.appserviceTokenFile == appserviceTokenPath;
message = ''
services.hyperhive.deploy.matrix.appserviceTokenFile is fixed at
${appserviceTokenPath} and cannot be moved — hive-c0re's Rust
side derives this same path independently and has no way to learn
${appserviceTokenPath} and cannot be moved — the registration
renderer reads this same path as a literal and has no way to learn
an override, so moving it desyncs the two silently instead of
loudly.
@ -1415,66 +1401,6 @@ in
# is why the render below is `requiredBy` it and local only.
++ lib.optional ctlActive "${swarmAppserviceCredentialId}:${swarmAppserviceDir}/swarm.yaml";
# Publish the appservice sender account's access token to the swarm
# store, once, under an identity that belongs to this container and
# not to the hive. See `ctlActive` above for why it runs here.
#
# A `oneshot` with no timer and no retry loop of its own: the whole
# of "and only once" is the binary's first act, a read of the path it
# would write. `Restart=on-failure` covers a store that is sealed or
# a homeserver still starting; `RemainAfterExit` is deliberately NOT
# set, because the unit having succeeded is not the idempotency
# record — the store is, and it outlives this machine.
systemd.services.swarm-matrix-ctl = lib.mkIf ctlActive {
description = "publish the matrix sender token to the swarm secret store";
# Ordered after the homeserver because both of the ladder's arms
# are client-server API calls. `wants`, not `requires`: a run that
# finds the credential already published never touches tuwunel at
# all, so a homeserver that is slow to come up should delay this,
# not cancel it.
after = [ "tuwunel.service" ];
wants = [ "tuwunel.service" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
# The verb is part of the contract: `swarm-matrix-ctl` is a
# subcommand binary and refuses a bare invocation, so dropping
# `mint` here fails the unit rather than doing something else.
ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl mint";
Restart = "on-failure";
RestartSec = 30;
# Bounded here rather than left to systemd's default, for the
# reason ./swarm-secret-publisher.nix states: a sealed store
# answers on the port and never answers the read.
TimeoutStartSec = 60;
SyslogIdentifier = "swarm-matrix-ctl";
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.matrix.ctlBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.matrix.ctlBaoClientKeyFile;
MATRIX_MINT_CERT_ROLE = ctlCertRole;
# Loopback: this container shares the host netns, so the
# homeserver it must talk to is the one in this very unit's
# netns and needs no name, no vhost and no TLS.
MATRIX_MINT_API_URL = "http://127.0.0.1:${toString cfg.httpPort}";
# The bind-mounted registration, which IS the as_token. A path,
# never a value.
MATRIX_MINT_REGISTRATION = appserviceRegistrationPath;
MATRIX_MINT_LOCALPART = hiveLocalpart;
# The hive segment of the store path the token is published
# under, and so the thing that keeps this hive's token out of
# every other hive's reach: the grant that reaches it is the
# hive's own `swarm/hives/<name>/*` stanza. ./swarm-bao.nix
# spells the same name into matrix-ctl's write grant.
MATRIX_MINT_HIVE = toString config.services.hyperhive.hiveName;
MATRIX_MINT_HOMESERVER = ctlHomeserverUrl;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
};
# The swarm registration, minted and rendered before the homeserver
# loads it. No network and no store: this is on tuwunel's start
# path, and a store outage must not keep the homeserver down.
@ -1502,9 +1428,12 @@ in
};
# Hand the swarm registration's token to swarm-controller, through
# the store. Same identity and retry shape as `swarm-matrix-ctl`
# above; the write lands at a path only matrix-ctl and the
# controller are granted (./swarm-bao.nix).
# the store, under matrix-ctl's identity; the write lands at a path
# only matrix-ctl and the controller are granted (./swarm-bao.nix).
# `Restart=on-failure` covers a sealed store or one still starting;
# the start timeout is bounded for the reason
# ./swarm-secret-publisher.nix states: a sealed store answers on the
# port and never answers the read.
systemd.services.swarm-matrix-appservice-publish = lib.mkIf ctlActive {
description = "publish the swarm's appservice token to the swarm secret store";
after = [ "swarm-matrix-appservice-render.service" ];

49
nix/host-modules/swarm-bao.nix
View file

@ -471,10 +471,9 @@ let
# `secret/data/` is KV v2's ACL prefix, inserted by the engine rather than
# written by the caller — the same trap as the two grants above.
#
# Not `swarm/services/*` like the publisher's: this principal produces
# exactly one secret, its own hive's matrix sender account access token, and
# a homeserver is not entitled to overwrite Grafana's OIDC client. The path
# is spelled to the leaf for that reason, not for tidiness.
# Not `swarm/services/*` like the publisher's: a homeserver is not entitled
# to overwrite Grafana's OIDC client. Each path is spelled to the leaf for
# that reason, not for tidiness.
#
# 🩸 And the leaf now carries a HIVE segment, which is the narrowing that
# matters: the credential used to live at `swarm/services/matrix/sender-token`
@ -484,16 +483,12 @@ let
# another's. `matrixCtlHive` below is the name this principal may write, and
# it is one hive rather than a `hives/*` wildcard for the same reason.
#
# `read` as well as write, unlike either sibling, and it is what makes "and
# only once" mechanical: matrix-ctl's first act is to read this path back and
# stop if something is there, so without the capability every container
# restart would mint a second access token and invalidate the hive's. A read
# here recovers one secret this principal itself wrote, which is a much
# narrower grant than the publisher's would have been.
# ⚠️ Nothing presenting this identity writes the first stanza's path:
# `swarm-controller` is the sender token's only minter. The stanza is unused.
#
# The second stanza is the swarm appservice's token, which matrix-ctl mints
# inside the container and publishes here for the controller. `read` for the
# same reason: publish compares before it writes.
# inside the container and publishes here for the controller. `read` as well
# as write, unlike either sibling, because publish compares before it writes.
matrixCtlPolicyText = ''
path "${credentialMountPath}/data/swarm/hives/${matrixCtlHive}/matrix/sender-token" {
capabilities = ["create", "update", "read"]
@ -504,15 +499,7 @@ let
}
'';
# Which hive matrix-ctl mints for. This host's own by default, which is right
# whenever the store and the homeserver are co-located and is the shape the
# `mkDefault` deployments produce; an operator running them apart names the
# homeserver's hive here, because the policy is written where the store is
# and the container runs where the homeserver is.
#
# A wrong value is loud rather than silent: matrix-ctl's write comes back 403
# with the store's own message and the hive falls back to minting its account
# locally, which is the same degrade a store that was never deployed gives.
# The hive the unused first stanza above names.
matrixCtlHive = baoDeploy.matrixCtlHiveName;
# The swarm appservice token's leaf, the nix half of
@ -1460,8 +1447,8 @@ in
example = "swarm-matrix-ctl.svc";
description = ''
Subject the store's matrix-ctl cert-auth role accepts — the
identity the oneshot inside the matrix container presents when it
publishes the appservice sender account's access token.
identity the matrix container's `swarm-matrix-appservice-publish`
unit presents when it publishes the swarm appservice's token.
A **third** identity rather than reuse of either sibling above, and
the narrowest of the three: its grant is one path, that
@ -1632,20 +1619,8 @@ in
description = ''
Hive whose matrix sender token the store's matrix-ctl role may write.
The sender account's access token is **per hive**: it lives at
`swarm/hives/<name>/matrix/sender-token`, and the only read grant that
reaches it is that hive's own. So matrix-ctl's write grant names one
hive too — the hive whose homeserver container it runs in.
Defaults to this host's own {option}`services.hyperhive.hiveName`,
which is correct whenever the store and the homeserver are co-located.
Set it when they are not: the policy is written where the store runs,
and the oneshot runs where the homeserver does.
A wrong value degrades rather than breaks — matrix-ctl's write is
refused with the store's own message and the hive mints its account
locally instead, the same fallback a swarm that never deployed the
store already uses.
Unused: nothing presenting that identity writes the sender token;
`swarm-controller` is its only minter.
'';
};

12
nix/module-eval/bao-grants.nix
View file

@ -451,14 +451,10 @@ let
&& !(lib.hasInfix "sys/policies/acl" s);
}
{
# 🩸 `read` is load-bearing here, and the publisher — the one sibling
# that still has no `read` — shows what its absence costs. matrix-ctl's
# first act is to read this path back and stop if something is there —
# that read IS "and only once", so without the capability every container
# restart would mint a second access token and invalidate the hive's.
# (The controller holds `read` for the same idempotency reason, on the
# agent prefix.)
name = "matrix-ctl may read back the one path it writes";
# 🩸 `read` is load-bearing here, unlike on the publisher: matrix-ctl's
# `appservice publish` reads the swarm appservice token back and writes
# only when the store's copy differs.
name = "matrix-ctl may read back what it writes";
ok =
let
s = baoGrantHere.systemd.services.swarm-bao-matrix-ctl-policy.script;

85
nix/module-eval/bao-matrix-reader.nix
View file

@ -292,7 +292,8 @@ let
name = "matrix-ctl presents its own leaf, never the hive's store-wide one";
ok =
let
env = baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-ctl.environment;
env =
baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-appservice-publish.environment;
hiveLeaf = baoWithMatrix.services.hyperhive.deploy.bao.clientCertFile;
in
env.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/matrix-ctl.pem" && env.BAO_CLIENT_CERT != hiveLeaf;
@ -316,81 +317,37 @@ let
&& mounts ? "/var/lib/hyperhive/matrix-appservice";
}
{
# What the unit is for, read as the two agreements it cannot get wrong:
# the cert role ./host-modules/swarm-bao.nix writes, and a homeserver
# address that is loopback because the container shares the host netns. A
# vhost here would be a request out through the gateway and back.
name = "matrix-ctl is handed the store role and the loopback homeserver";
# The cert role ./host-modules/swarm-bao.nix writes, which is the one
# agreement the unit cannot get wrong, and the verb: a bare invocation
# exits non-zero with clap's usage, a deploy-time failure with no local
# signal.
name = "the publish unit is handed the store role and invokes its verb";
ok =
let
m = baoWithMatrix;
u = m.containers.hive-matrix.config.systemd.services.swarm-matrix-ctl;
port = m.services.hyperhive.swarm.matrix.httpPort;
u = baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-appservice-publish;
in
u.environment.MATRIX_MINT_CERT_ROLE == "swarm-matrix-ctl"
&& u.environment.MATRIX_MINT_API_URL == "http://127.0.0.1:${toString port}"
&& u.environment.MATRIX_MINT_REGISTRATION == "/var/lib/hyperhive/matrix-appservice/hyperhive.yaml"
&& u.serviceConfig.Type == "oneshot";
u.environment.MATRIX_APPSERVICE_CERT_ROLE == "swarm-matrix-ctl"
&& lib.hasSuffix "/bin/swarm-matrix-ctl appservice publish" u.serviceConfig.ExecStart;
}
{
# 🩸 The per-hive minting identity, read off the rendered unit rather than
# off the option: the binary builds its store path out of
# `MATRIX_MINT_HIVE` and logs in as `MATRIX_MINT_LOCALPART`, so a unit
# that passed the old bare `hive` would publish one identity for the
# whole swarm again and nothing in the Rust tests could see it. Both
# spellings are pinned, and the localpart is pinned as *derived from* the
# hive name rather than as a literal, which is the agreement
# `swarm_secret_client::matrix::hive_localpart` owns.
name = "matrix-ctl is told which hive it mints for, and acts as that hive's account";
# 🩸 A secret is a path, never a value. Every variable the unit is given
# names a file or an address; the token itself is read out of the state
# dir at runtime, so nothing here can be a token and an environment block
# is world-readable through `systemctl show`.
name = "the publish unit's environment carries paths and addresses, never a token";
ok =
let
m = baoWithMatrix;
u = m.containers.hive-matrix.config.systemd.services.swarm-matrix-ctl;
hive = m.services.hyperhive.hiveName;
in
u.environment.MATRIX_MINT_HIVE == hive
&& u.environment.MATRIX_MINT_LOCALPART == "hive-${hive}"
&& u.environment.MATRIX_MINT_LOCALPART != "hive";
}
{
# 🩸 The crate is a `*ctl` with subcommands, so the unit has to name a
# VERB. This is the one end of that contract nix owns: the binary's own
# test pins how `mint` is spelled, but only a rendered `ExecStart` can
# say the unit actually passes it. A bare invocation exits non-zero with
# clap's usage — which is a deploy-time failure with no local signal, and
# exactly what the next verb added here is most likely to disturb.
name = "the unit invokes a verb rather than the bare binary";
ok =
let
exec =
baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-ctl.serviceConfig.ExecStart;
in
lib.hasSuffix "/bin/swarm-matrix-ctl mint" exec;
}
{
# 🩸 A secret is a path, never a value — checked on the one unit in this
# tree whose whole job is an `as_token`. Every variable it is given names
# a file or an address; the token itself is read out of the bind-mounted
# registration at runtime, so nothing here can be a token and an
# environment block is world-readable through `systemctl show`.
name = "matrix-ctl's environment carries paths and addresses, never a token";
ok =
let
env = baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-ctl.environment;
env =
baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-appservice-publish.environment;
in
!(lib.any (v: lib.hasInfix "as_token" v || lib.hasInfix "syt_" v) (lib.attrValues env));
}
{
# The absence arm, and the deployment it protects: a homeserver on a hive
# with no store identity at all. Without it the unit would exist naming
# `null` as its certificate, which nixos renders as the literal string.
name = "a matrix container with no store identity runs no matrix-ctl and binds no PKI";
ok =
let
units = matrixNoBaoIdentity.containers.hive-matrix.config.systemd.services;
in
!(units ? swarm-matrix-ctl)
&& !(matrixNoBaoIdentity.containers.hive-matrix.bindMounts ? "/var/lib/swarm-bao-pki");
# with no store identity at all. Without it the mount would name `null`
# as its source, which nixos renders as the literal string.
name = "a matrix container with no store identity binds no PKI";
ok = !(matrixNoBaoIdentity.containers.hive-matrix.bindMounts ? "/var/lib/swarm-bao-pki");
}
{
# 🩸 The privilege arm: exactly one account is a homeserver admin, the