vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

render the matrix appservice registration from the store onto tmpfs

Browse source 
swarm-matrix-ctl appservice render now reads the registration's tokens
from the store and renders swarm.yaml under /run/swarm-matrix-appservice,
which tuwunel loads through LoadCredential. It mints a pair only when the
store answers and holds none; a host that has the old files stores their
pair once instead. A store that does not answer fails the unit and keeps
tuwunel down; sync starts it once the store answers.

Refs #4998 (item 3)
This commit is contained in:
atlas 2026-10-30 05:47:39 +01:00 • committed by mara
commit 3a4b0ffabf
8 changed files with 594 additions and 378 deletions
Download patch file Download diff file Expand all files Collapse all files

31
docs/integrations/matrix.md
View file

@ -125,11 +125,12 @@ creation doesn't depend on registration being open to anyone who learns
a token. a token.
1. **`swarm-matrix-ctl appservice render`**, inside the `hive-matrix` 1. **`swarm-matrix-ctl appservice render`**, inside the `hive-matrix`
container and before tuwunel starts, renders the registration to container and before tuwunel starts, fetches the `as_token` and
`/var/lib/swarm-matrix-appservice/swarm.yaml` (mode `0600 root:root`) spec-required `hs_token` from the store and renders the registration to
from the `as_token` and spec-required `hs_token` beside it. It mints `/run/swarm-matrix-appservice/swarm.yaml` (mode `0600 root:root`) on the
them only on a container that has none, and never reaches the store, so container's tmpfs, so no token sits on disk. It mints a pair only when
a store outage can't keep the homeserver down. the store answers and holds none. While the store doesn't answer, `render`
fails and retries, and tuwunel stays down.
2. **systemd `LoadCredential=`** copies the registration into 2. **systemd `LoadCredential=`** copies the registration into
`/run/credentials/tuwunel.service/swarm-appservice.yaml`, owned by `/run/credentials/tuwunel.service/swarm-appservice.yaml`, owned by
tuwunel's dynamic user with mode `0400`, at service start. The source tuwunel's dynamic user with mode `0400`, at service start. The source
@ -167,15 +168,17 @@ account with it.
The swarm secret store holds the registration's tokens, at The swarm secret store holds the registration's tokens, at
`swarm/controller/swarm-controller/matrix/appservice-registration`. `swarm/controller/swarm-controller/matrix/appservice-registration`.
`swarm-matrix-ctl appservice render` renders the registration tuwunel `swarm-matrix-ctl appservice render` fetches them and renders the
loads as a credential from the copy of them in the `hive-matrix` registration tuwunel loads as a credential onto the `hive-matrix`
container, before tuwunel starts and without the store. container's tmpfs, before tuwunel starts.
`appservice sync` runs beside tuwunel on a timer. The store's tokens win: `appservice sync` runs beside tuwunel on a timer. The store's tokens win:
when they differ from the container's, it writes them there. It when they differ from the rendered copy, it rewrites it. It
restarts tuwunel whenever the registration is newer than tuwunel's last restarts tuwunel whenever the registration is newer than tuwunel's last
start, so the next run retries one that failed after writing. A store that holds none takes the container's start, so the next run retries one that failed after writing, and starts
tokens with a create-only write, so a running swarm keeps its value, and one that failed because `render` couldn't reach the store. A store that
a container that has none mints a pair. `sync` then writes the `as_token` holds none takes the tokens an earlier version kept in
`/var/lib/swarm-matrix-appservice` with a create-only write, so a running
swarm keeps its value; the files stay, unused. A host with neither files nor stored tokens mints a pair. `sync` then writes the `as_token`
to `swarm/controller/swarm-controller/matrix/appservice-token`. No hive's to `swarm/controller/swarm-controller/matrix/appservice-token`. No hive's
policy reaches either path: only matrix-ctl (which holds `create` and policy reaches either path: only matrix-ctl (which holds `create` and
`read` on the registration, and writes the token) and `swarm-controller` `read` on the registration, and writes the token) and `swarm-controller`
@ -263,8 +266,8 @@ both are swarm-level operations.
<details><summary>Appservice identity</summary> <details><summary>Appservice identity</summary>
`swarm-matrix-ctl appservice render` renders the registration from the `swarm-matrix-ctl appservice render` renders the registration from the
container's copy of the appservice tokens before tuwunel starts, so every store's copy of the appservice tokens before tuwunel starts, so every
boot has both halves, with the store up or down. boot has both halves. With the store down or sealed, tuwunel waits for it.
<!-- vale write-good.Passive = NO --> <!-- vale write-good.Passive = NO -->
- **Access tokens live on the device that minted them.** - **Access tokens live on the device that minted them.**

4
docs/swarm/credentials-reference.md
View file

@ -22,7 +22,7 @@ the rest of the cell says how.
| `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection | | `swarm/agents/<agent>/matrix/<account>` | `swarm-controller`, from a credential you hand it | the agent | must be stated | ✅ retried within 2 to 16 minutes after a rejection |
| `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up | | `swarm/agents/<agent>/matrix-recovery/<account>` | `swarm-controller`, a random passphrase per account | the agent | ❌ never replaced: the account's key backup opens only with this passphrase | ❌ read when the account comes up |
| `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes | | `swarm/controller/swarm-controller/matrix/appservice-token` | `swarm-matrix-ctl appservice sync` in `hive-matrix`, from the registration's `as_token` | `swarm-controller` | ❌ follows the registration below | ✅ read every 5 minutes |
| `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice sync` in `hive-matrix`: seeded from the container's own tokens when the store holds none, a new pair only when the container has none either | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` write the new pair into the container and restart the homeserver | ✅ `sync` re-reads it every 10 minutes | | `swarm/controller/swarm-controller/matrix/appservice-registration` | `swarm-matrix-ctl appservice render` in `hive-matrix`, when the store answers and holds none; before that, the tokens an earlier version kept in the container, stored unchanged | `swarm-matrix-ctl` in `hive-matrix`; the controller has no grant to it | ❌ minted once; an operator who replaces it makes `sync` render the new pair and restart the homeserver | ✅ `sync` re-reads it every 10 minutes |
| `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it | | `swarm/controller/swarm-controller/oidc/client` | authelia, first boot | `swarm-controller` | ❌ minted once, straight into the store; minted again only when the store holds none | ✅ re-read when authelia refuses it |
| `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 | | `swarm/controller/swarm-controller/forge/token` | the forge, once | `swarm-controller` | ❌ minted once; delete `/var/lib/forgejo/swarm-controller-token-name` in the forge container to mint a new one | ✅ re-read on a 401 |
| `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 | | `swarm/controller/swarm-controller/grafana/token` | `swarm-bao-grafana-controller-token` on Grafana's host | `swarm-controller` | ✅ every 10 minutes, when Grafana refuses it or none is stored | ✅ re-read on a 401 |
@ -113,7 +113,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
| swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity | | swarm-services root CA (cert + key) | `swarm-bao.nix`'s granting unit, inside the store's `pki` mount | the **key** never leaves the mount; the **cert** lands at `/var/lib/swarm-bao-services-pki/services-root.pem` (0644) and at `/var/lib/hive-tls/swarm-services-root.pem` on every hive | nothing to copy — every hive receives the cert with the leaf it asks the store for, or from `pki/ca/pem` where it fronts no swarm name and needs a store identity |
| authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/<name>` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting | | authelia session, JWT, storage-encryption and OIDC HMAC keys | `authelia-swarm-secrets` in authelia's container. It stores each in the swarm secret store at `swarm/services/swarm-authelia/keys/<name>` when the store holds none: the value of the key an earlier release generated, or a new one when there is no such file. The files stay on disk. While the store's policy doesn't grant the leaf yet, or the store is down, the unit runs on the files. A container with no secret publisher identity generates the files, as before | `/var/lib/authelia-swarm/{session,jwt,storage-encryption,oidc-hmac}.key` | nothing to place: the publisher's store leaf is the one in the row below. A store that stays unreachable on a start with no key file keeps authelia from starting |
| authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` | | authelia OIDC issuer key (RSA) | same unit, same store path (`oidc-issuer`) | `/var/lib/authelia-swarm/oidc-issuer.key` | same — relying parties verify against the **public** half at `/jwks.json` |
| swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice sync` in `hive-matrix`. The store's pair wins; a store holding none takes the container's own tokens, so the value doesn't change by moving. Only `appservice render` on a container with neither mints a new pair | `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` in `hive-matrix`, the last copy `sync` took, which tuwunel starts from; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down leaves the homeserver running on the files | | swarm appservice registration (`as_token`, `hs_token`) | `swarm-matrix-ctl appservice render` in `hive-matrix`. The store's pair wins; a store holding none takes the tokens an earlier version kept in the container, so the value doesn't change by moving. Only a store that answers, holds none, and a container with no such files gets a new pair | `/run/swarm-matrix-appservice/swarm.yaml` in `hive-matrix` (tmpfs), which tuwunel loads through `LoadCredential`. The earlier `/var/lib/swarm-matrix-appservice/{as-token,hs-token,swarm.yaml}` stay, unused, and bao wins if they differ; the store path is `swarm/controller/swarm-controller/matrix/appservice-registration` | nothing to place: matrix-ctl's store leaf is the one the container binds. A store that's down keeps the homeserver down and mints nothing, and `render` retries every 30 seconds for 24 hours |
| OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) | | OIDC client secret, plaintext half | `authelia crypto hash generate --random` | the swarm secret store, at the path its reader fetches: `swarm/hives/<hive>/oidc/client`, `swarm/controller/swarm-controller/oidc/client` or `swarm/services/<client id>/oidc/client`. The bridge reads its own into `/run/swarm-authelia-bridge/client-secret` before each start. A container with no secret publisher identity keeps it in `/var/lib/authelia-swarm/oidc-clients/<id>.secret`; a file an earlier release left seeds the store once when it holds no value, and stays unused | nothing to place: readers fetch it from the store ([how](credentials.md#hive-level-secrets)) |
| OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` | | OIDC client secret, digest half | the same mint | `oidc-clients/<id>.digest` | authelia's own half; merged at runtime via `settingsFiles` |
| the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) | | the swarm collector's copy of its OIDC secret | `swarm-bao-otel-oidc.service` reads it out of the swarm secret store, **on every host that runs the collector and holds a store identity** | `/run/swarm-otel-oidc/<id>.secret` on the host's tmpfs, bind-mounted read-only into the `swarm-otel` container — never on persistent disk, so a reboot fetches it again | same unit, same path — one route, co-located or not. A collector with no store identity has `clientSecretFile == null`, its already-supported unauthenticated-push degrade — see [swarm-level secrets](credentials.md#swarm-level-secrets) |

122
nix/host-modules/hive-matrix.nix
View file

@ -100,6 +100,25 @@ let
# environment, so the binary itself spells no role at all. # environment, so the binary itself spells no role at all.
ctlCertRole = "swarm-matrix-ctl"; ctlCertRole = "swarm-matrix-ctl";
# What `swarm-matrix-ctl appservice render` and `sync` are both given: the
# store address and matrix-ctl's identity, and the registration's shape. A
# secret is a path, never a value, so nothing here is a token.
appserviceEnv = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.matrix.ctlBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.matrix.ctlBaoClientKeyFile;
MATRIX_APPSERVICE_CERT_ROLE = ctlCertRole;
MATRIX_APPSERVICE_RUNTIME_DIR = swarmAppserviceRuntimeDir;
MATRIX_APPSERVICE_LEGACY_DIR = swarmAppserviceLegacyDir;
MATRIX_APPSERVICE_SENDER = swarmSenderLocalpart;
# Agents' localparts are bare `[a-z0-9-]`, so nothing narrower covers
# them without also covering people.
MATRIX_APPSERVICE_USER_REGEX = appserviceUserRegex;
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
};
# Every host directory matrix-ctl's bao identity is spread across. Normally # Every host directory matrix-ctl's bao identity is spread across. Normally
# one — ./glue-bao-tls.nix puts all three files in ./glue's PKI dir — but # one — ./glue-bao-tls.nix puts all three files in ./glue's PKI dir — but
# derived rather than assumed, because an operator naming paths by hand is # derived rather than assumed, because an operator naming paths by hand is
@ -140,20 +159,25 @@ let
# The homeserver's one appservice registration, and the swarm's identity on # The homeserver's one appservice registration, and the swarm's identity on
# it: `swarm-controller` creates every agent's account and every hive's # it: `swarm-controller` creates every agent's account and every hive's
# `@hive-<hive>` sender with its token. The store holds its tokens; # `@hive-<hive>` sender with its token. The store holds its tokens;
# `swarm-matrix-ctl appservice sync` keeps this container's copy in step with # `swarm-matrix-ctl appservice render` fetches them into this container's
# them, which is why it is gated on matrix-ctl's store identity: with nobody # tmpfs before each homeserver start, and `appservice sync` keeps that copy
# to sync it, a registration here would be an admin credential nobody reads. # in step with them. Both are gated on matrix-ctl's store identity: with
# `appservice render` minted a copy here only for a host that has none, and # nobody to fetch it, a registration here would be an admin credential
# renders the one the homeserver starts from. # nobody reads. `render` mints a pair only when the store answers and holds
# none, and the homeserver stays down while the store does not answer.
# #
# Its sender is promoted to homeserver admin at boot (`admin_execute` # Its sender is promoted to homeserver admin at boot (`admin_execute`
# below), so its token goes only to a store path no hive's policy reaches. # below), so its token goes only to a store path no hive's policy reaches.
# `swarm` is in ../reserved-names.nix, so no agent can be created as this # `swarm` is in ../reserved-names.nix, so no agent can be created as this
# account. # account.
swarmSenderLocalpart = "swarm"; swarmSenderLocalpart = "swarm";
# Inside the container: the render unit's `StateDirectory`. `ephemeral = # Inside the container, on its tmpfs: the registration tuwunel loads, owned
# false` keeps it across restarts, so "mint when absent" is mint once. # by the two units below through `RuntimeDirectory`.
swarmAppserviceDir = "/var/lib/swarm-matrix-appservice"; swarmAppserviceRuntimeName = "swarm-matrix-appservice";
swarmAppserviceRuntimeDir = "/run/${swarmAppserviceRuntimeName}";
# Where earlier versions kept the tokens. Read to seed an empty store, never
# written.
swarmAppserviceLegacyDir = "/var/lib/swarm-matrix-appservice";
swarmAppserviceCredentialId = "swarm-appservice.yaml"; swarmAppserviceCredentialId = "swarm-appservice.yaml";
# Every local user the swarm appservice may create — agents and every # Every local user the swarm appservice may create — agents and every
@ -1021,77 +1045,71 @@ in
# suffix on purpose: `appservice_dir` above names this very # suffix on purpose: `appservice_dir` above names this very
# directory, and tuwunel takes only `.yaml`/`.yml` entries from # directory, and tuwunel takes only `.yaml`/`.yml` entries from
# it. A missing source fails this unit, which is why the render # it. A missing source fails this unit, which is why the render
# below is `requiredBy` it and local only. # below is `requiredBy` it.
++ lib.optional ctlActive "${swarmAppserviceCredentialId}:${swarmAppserviceDir}/swarm.yaml"; ++ lib.optional ctlActive "${swarmAppserviceCredentialId}:${swarmAppserviceRuntimeDir}/swarm.yaml";
# The swarm registration, rendered before the homeserver loads it from # The swarm registration, fetched from the store and rendered onto
# the tokens in the state dir: the last copy of the store's that # the tmpfs before the homeserver loads it. A store that does not
# `swarm-matrix-appservice-sync` took, or a minted pair on a host with # answer fails this unit, and with it the homeserver's start: the
# none. No network and no store: this is on tuwunel's start path, and # tokens are on no disk to start from. It retries like the other
# a store outage must not keep the homeserver down. # store fetches, and `swarm-matrix-appservice-sync` below starts the
# homeserver once the registration is there.
#
# `RuntimeDirectoryPreserve` because the sync unit shares the
# directory: neither unit's stop may remove it from under the other.
systemd.services.swarm-matrix-appservice-render = lib.mkIf ctlActive { systemd.services.swarm-matrix-appservice-render = lib.mkIf ctlActive {
description = "render the swarm's appservice registration"; description = "render the swarm's appservice registration";
before = [ "tuwunel.service" ]; before = [ "tuwunel.service" ];
requiredBy = [ "tuwunel.service" ]; requiredBy = [ "tuwunel.service" ];
serviceConfig = { inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
RemainAfterExit = true; RemainAfterExit = true;
ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice render"; ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice render";
StateDirectory = baseNameOf swarmAppserviceDir; # A sealed store answers on the port and never answers the read.
StateDirectoryMode = "0700"; TimeoutStartSec = 60;
RuntimeDirectory = swarmAppserviceRuntimeName;
RuntimeDirectoryMode = "0700";
RuntimeDirectoryPreserve = "yes";
UMask = "0077"; UMask = "0077";
SyslogIdentifier = "swarm-matrix-appservice"; SyslogIdentifier = "swarm-matrix-appservice";
}; };
environment = { environment = appserviceEnv;
MATRIX_APPSERVICE_DIR = swarmAppserviceDir;
MATRIX_APPSERVICE_SENDER = swarmSenderLocalpart;
# Agents' localparts are bare `[a-z0-9-]`, so nothing narrower
# covers them without also covering people.
MATRIX_APPSERVICE_USER_REGEX = appserviceUserRegex;
};
}; };
# The store holds the swarm registration; this keeps the state dir # Keeps the rendered registration in step with the store, and
# the render above reads in step with it, and publishes the # publishes the `as_token` to the path `swarm-controller` reads,
# `as_token` to the path `swarm-controller` reads, under matrix-ctl's # under matrix-ctl's identity (the grants are in ./swarm-bao.nix).
# identity (the grants are in ./swarm-bao.nix). A store without the # When the registration is newer than tuwunel's last start, this
# registration is seeded from the state dir, so nothing rotates. # restarts tuwunel onto it; a tuwunel that failed because the render
# did is started the same way.
# #
# ⚠️ Not `before`/`requiredBy` tuwunel, and nothing orders tuwunel # `wants`, not `requires`, the render: this unit must still run when
# after it: the homeserver starts from the state dir's last copy # the render has failed, since it is what brings the homeserver up
# whether or not the store answers. When the registration is newer than # once the store answers. `Restart=on-failure` with the shared budget
# tuwunel's last start, this restarts tuwunel onto it. # covers a sealed store or one still starting; the timer below
# # repeats the run, which a unit without `RemainAfterExit` allows.
# `Restart=on-failure` covers a sealed store or one still starting;
# the timer below repeats the run, which a unit without `RemainAfterExit` allows.
# The start timeout is bounded because a sealed store answers on the # The start timeout is bounded because a sealed store answers on the
# port and never answers the read. # port and never answers the read.
systemd.services.swarm-matrix-appservice-sync = lib.mkIf ctlActive { systemd.services.swarm-matrix-appservice-sync = lib.mkIf ctlActive {
description = "keep the swarm's appservice registration in step with the swarm secret store"; description = "keep the swarm's appservice registration in step with the swarm secret store";
after = [ "swarm-matrix-appservice-render.service" ]; after = [ "swarm-matrix-appservice-render.service" ];
requires = [ "swarm-matrix-appservice-render.service" ]; wants = [ "swarm-matrix-appservice-render.service" ];
wantedBy = [ "multi-user.target" ]; wantedBy = [ "multi-user.target" ];
path = [ pkgs.systemd ]; path = [ pkgs.systemd ];
serviceConfig = { inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot"; Type = "oneshot";
ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice sync"; ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice sync";
Restart = "on-failure";
RestartSec = 30;
TimeoutStartSec = 60; TimeoutStartSec = 60;
RuntimeDirectory = swarmAppserviceRuntimeName;
RuntimeDirectoryMode = "0700";
RuntimeDirectoryPreserve = "yes";
UMask = "0077";
SyslogIdentifier = "swarm-matrix-appservice"; SyslogIdentifier = "swarm-matrix-appservice";
}; };
environment = { environment = appserviceEnv // {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
BAO_CLIENT_CERT = deployCfg.matrix.ctlBaoClientCertFile;
BAO_CLIENT_KEY = deployCfg.matrix.ctlBaoClientKeyFile;
MATRIX_APPSERVICE_CERT_ROLE = ctlCertRole;
MATRIX_APPSERVICE_DIR = swarmAppserviceDir;
MATRIX_APPSERVICE_SENDER = swarmSenderLocalpart;
MATRIX_APPSERVICE_USER_REGEX = appserviceUserRegex;
MATRIX_APPSERVICE_RESTART_UNIT = "tuwunel.service"; MATRIX_APPSERVICE_RESTART_UNIT = "tuwunel.service";
}
// lib.optionalAttrs (deployCfg.bao.serverCaFile != null) {
BAO_CACERT = deployCfg.bao.serverCaFile;
}; };
}; };

9
nix/host-modules/swarm-bao.nix
View file

@ -653,7 +653,8 @@ let
# write, unlike either sibling, because sync compares before it writes. # write, unlike either sibling, because sync compares before it writes.
# #
# The second is the registration's tokens: the store's copy, which matrix-ctl # The second is the registration's tokens: the store's copy, which matrix-ctl
# seeds from the container's state dir and then follows. `create` and `read` # creates (from the tokens an earlier version kept in the container, or a
# minted pair) and then follows. `create` and `read`
# only, never `update`: a registration that exists is replaced by an # only, never `update`: a registration that exists is replaced by an
# operator, not by a unit. # operator, not by a unit.
# #
@ -2008,9 +2009,9 @@ in
example = "swarm-matrix-ctl.svc"; example = "swarm-matrix-ctl.svc";
description = '' description = ''
Subject the store's matrix-ctl cert-auth role accepts — the Subject the store's matrix-ctl cert-auth role accepts — the
identity the matrix container's `swarm-matrix-appservice-sync` identity the matrix container's `swarm-matrix-appservice-render` and
unit presents when it syncs the swarm appservice's registration and `swarm-matrix-appservice-sync` units present when they fetch the swarm
publishes its token. appservice's registration and publish its token.
A **third** identity rather than reuse of either sibling above. Its A **third** identity rather than reuse of either sibling above. Its
grant is that credential and the registration it derives from, plus grant is that credential and the registration it derives from, plus

70
nix/module-eval/bao-matrix-reader.nix
View file

@ -258,13 +258,15 @@ let
creds = creds =
baoWithMatrix.containers.hive-matrix.config.systemd.services.tuwunel.serviceConfig.LoadCredential; baoWithMatrix.containers.hive-matrix.config.systemd.services.tuwunel.serviceConfig.LoadCredential;
in in
lib.elem "swarm-appservice.yaml:/var/lib/swarm-matrix-appservice/swarm.yaml" creds lib.elem "swarm-appservice.yaml:/run/swarm-matrix-appservice/swarm.yaml" creds
&& lib.length (lib.filter (c: lib.hasInfix ".yaml:" c) creds) == 1; && lib.length (lib.filter (c: lib.hasInfix ".yaml:" c) creds) == 1;
} }
{ {
# A missing `LoadCredential` source fails tuwunel, so the render has to # A missing `LoadCredential` source fails tuwunel, so the render has to
# have run first — and it is local only, so it cannot fail on a store. # have run first. It fetches from the store, so it carries the store's
name = "the swarm registration is rendered before tuwunel and required by it, without the store"; # address and matrix-ctl's identity, and retries on failure: a store
# that does not answer keeps the homeserver down until it does.
name = "the swarm registration is rendered before tuwunel and required by it, from the store, with retries";
ok = ok =
let let
r = baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-appservice-render; r = baoWithMatrix.containers.hive-matrix.config.systemd.services.swarm-matrix-appservice-render;
@ -272,17 +274,19 @@ let
lib.elem "tuwunel.service" r.before lib.elem "tuwunel.service" r.before
&& lib.elem "tuwunel.service" r.requiredBy && lib.elem "tuwunel.service" r.requiredBy
&& lib.hasSuffix "swarm-matrix-ctl appservice render" r.serviceConfig.ExecStart && lib.hasSuffix "swarm-matrix-ctl appservice render" r.serviceConfig.ExecStart
&& !(r.environment ? BAO_ADDR); && r.environment.BAO_ADDR != ""
&& r.environment.BAO_CLIENT_CERT == "/var/lib/swarm-bao-pki/matrix-ctl.pem"
&& r.environment.MATRIX_APPSERVICE_CERT_ROLE == "swarm-matrix-ctl"
&& r.serviceConfig.Restart == "on-failure";
} }
{ {
# The start path stays free of the store. The render is what tuwunel # The sync unit is what starts a homeserver the render's failure left
# waits for and it carries no store address or identity; the sync unit # down, so it must run when the render has failed: it is after the
# that does is neither before tuwunel nor required by it, and tuwunel # render, and neither requires nor binds to it. It stays off tuwunel's
# names it nowhere, so a store outage leaves the homeserver starting on # own start path, and tuwunel names it nowhere. The presence arms are the
# the state dir's last copy. The presence arms are the render's own # render's own ordering (the same lookup finds `before`/`requiredBy`
# ordering (the same lookup finds `before`/`requiredBy` where they are) # where they are) and the sync's order after the render.
# and the sync's order after the render. name = "the sync unit runs after the render without requiring it, and is off tuwunel's start path";
name = "the sync unit is off tuwunel's start path and the render holds no store identity";
ok = ok =
let let
c = baoWithMatrix.containers.hive-matrix.config.systemd; c = baoWithMatrix.containers.hive-matrix.config.systemd;
@ -298,15 +302,51 @@ let
(u.bindsTo or [ ]) (u.bindsTo or [ ])
]; ];
sync = "swarm-matrix-appservice-sync.service"; sync = "swarm-matrix-appservice-sync.service";
render = "swarm-matrix-appservice-render.service";
in in
lib.elem "tuwunel.service" r.before lib.elem "tuwunel.service" r.before
&& lib.elem "swarm-matrix-appservice-render.service" s.after && lib.elem render s.after
&& !(lib.elem render (s.requires or [ ]))
&& !(lib.elem render (s.bindsTo or [ ]))
&& !(lib.elem "tuwunel.service" (s.before or [ ])) && !(lib.elem "tuwunel.service" (s.before or [ ]))
&& !(lib.elem "tuwunel.service" (s.requiredBy or [ ])) && !(lib.elem "tuwunel.service" (s.requiredBy or [ ]))
&& !(lib.elem "tuwunel.service" (s.wantedBy or [ ])) && !(lib.elem "tuwunel.service" (s.wantedBy or [ ]))
&& !(lib.elem sync (names t)) && !(lib.elem sync (names t))
&& !(lib.elem sync (r.before or [ ])) && !(lib.elem sync (r.before or [ ]));
&& !(lib.any (lib.hasPrefix "BAO_") (lib.attrNames r.environment)); }
{
# 🩸 The registration holds the tokens that create users, so it is
# rendered onto the container's tmpfs and no unit keeps the tokens in
# /var/lib. tuwunel's credential source, both units' runtime directory
# and the directory they are told to render into are one path under
# /run (the presence arms), and neither unit has a state directory or
# a writable path under /var/lib. The legacy directory is named in the
# environment, to be read, and nowhere else.
name = "the swarm registration is rendered under /run and no unit writes it under /var/lib";
ok =
let
c = baoWithMatrix.containers.hive-matrix.config.systemd;
units = [
c.services.swarm-matrix-appservice-render
c.services.swarm-matrix-appservice-sync
];
creds = c.services.tuwunel.serviceConfig.LoadCredential;
runDir = "/run/swarm-matrix-appservice";
writesVarLib =
u:
lib.any (lib.hasPrefix "/var/lib") (
lib.toList (u.serviceConfig.ReadWritePaths or [ ]) ++ lib.toList (u.serviceConfig.BindPaths or [ ])
)
|| (u.serviceConfig ? StateDirectory);
in
lib.elem "swarm-appservice.yaml:${runDir}/swarm.yaml" creds
&& lib.all (u: u.environment.MATRIX_APPSERVICE_RUNTIME_DIR == runDir) units
&& lib.all (u: u.serviceConfig.RuntimeDirectory == "swarm-matrix-appservice") units
&& !(lib.any writesVarLib units)
&& !(lib.any (cr: lib.hasInfix "/var/lib/swarm-matrix-appservice" cr) creds)
&& lib.all (
u: u.environment.MATRIX_APPSERVICE_LEGACY_DIR == "/var/lib/swarm-matrix-appservice"
) units;
} }
{ {
# A timer repeats the sync, which needs a unit that does not stay # A timer repeats the sync, which needs a unit that does not stay

17
swarm-matrix-ctl/README.md
View file

@ -13,18 +13,19 @@ crate.
### `appservice render` ### `appservice render`
Renders the swarm appservice registration tuwunel loads from the tokens in the Renders the swarm appservice registration tuwunel loads from the store's tokens
state dir, minting them only on a host that has none. Runs before the onto the container's tmpfs, minting a pair only when the store answers and holds
homeserver and needs no network: the tokens on disk are the last copy `sync` none. Runs before the homeserver, which stays down while the store does not
took from the store. answer. Tokens an earlier version kept in the container's `/var/lib` are stored
first when the store holds none; the files stay, unused.
### `appservice sync` ### `appservice sync`
The store holds the registration's tokens. This writes them into the state dir The store holds the registration's tokens. This rewrites the rendered copy
when they differ, restarts the homeserver whenever the registration is when they differ, restarts the homeserver whenever the registration is
newer than its last start, and publishes the `as_token` for `swarm-controller`. A store holding none is seeded from the newer than its last start, and publishes the `as_token` for `swarm-controller`.
state dir, create-only, so nothing rotates. It runs beside the homeserver, so a It runs the same reconciliation as `render`, and starts a homeserver that
store outage leaves the homeserver running on the last copy. failed because `render` could not reach the store.
Both are configured entirely by the `MATRIX_APPSERVICE_*` environment the units Both are configured entirely by the `MATRIX_APPSERVICE_*` environment the units
set — no flags. A systemd `Environment=` block is what a nix module can render; a set — no flags. A systemd `Environment=` block is what a nix module can render; a

750
swarm-matrix-ctl/src/appservice.rs
View file

@ -1,21 +1,22 @@
//! `swarm-matrix-ctl appservice` — the **swarm's** own appservice //! `swarm-matrix-ctl appservice` — the **swarm's** own appservice
//! registration. The store holds it; this container keeps a copy the //! registration. The store holds it; this container renders a copy the
//! homeserver starts from. //! homeserver starts from onto its own tmpfs, and keeps none on disk.
//! //!
//! Two verbs, because they have opposite failure rules: //! Two verbs, both reading the store:
//! //!
//! - [`render`] runs before tuwunel and touches nothing but this container's //! - [`render`] runs before tuwunel. tuwunel loads the file it writes through
//! state dir. tuwunel loads the file it writes through `LoadCredential`, and //! `LoadCredential`, and a missing credential source fails the homeserver's
//! a missing credential source fails the homeserver's start, so this half //! start, so a store that does not answer fails this unit and keeps the
//! must not be able to fail on a network. It renders from the tokens on //! homeserver down until it does. It mints a pair only when the store
//! disk, which are the last copy [`sync`] took from the store, and mints //! answers and holds none.
//! them only on a host that has none. //! - [`sync`] runs beside the homeserver and follows the store: it rewrites
//! - [`sync`] runs beside the homeserver, never before it, and needs the store. //! the file when the store's tokens differ, restarts the homeserver
//! The store's tokens win: it writes the state dir only when they differ, and //! whenever the registration is newer than the homeserver's last start, and
//! restarts the homeserver whenever the registration is newer than the //! publishes the `as_token` for the controller.
//! homeserver's last start. A store holding none is seeded from //!
//! the state dir, create-only, so nothing rotates. A sealed store delays it //! Where the store holds none, both first store the tokens an earlier version
//! and nothing else. //! kept in the legacy state dir (see [`resolve`]), so a host that has run for
//! months keeps its value. The files stay, unused.
//! //!
//! This registration's sender is promoted to homeserver admin at boot //! This registration's sender is promoted to homeserver admin at boot
//! (`nix/host-modules/hive-matrix.nix`), which is why its `as_token` goes to //! (`nix/host-modules/hive-matrix.nix`), which is why its `as_token` goes to
@ -33,14 +34,17 @@ use swarm_secret_client::{
matrix::{self, Registration}, matrix::{self, Registration},
}; };
/// This container's state dir for the registration and its two tokens. /// Where the homeserver's copy is rendered: a runtime directory on this
const ENV_DIR: &str = "MATRIX_APPSERVICE_DIR"; /// container's tmpfs.
const ENV_RUNTIME_DIR: &str = "MATRIX_APPSERVICE_RUNTIME_DIR";
/// The directory earlier versions kept the tokens in. Read once, never written.
const ENV_LEGACY_DIR: &str = "MATRIX_APPSERVICE_LEGACY_DIR";
/// The registration's `sender_localpart`: the account the homeserver creates /// The registration's `sender_localpart`: the account the homeserver creates
/// for it and the one `admin_execute` promotes. /// for it and the one `admin_execute` promotes.
const ENV_SENDER: &str = "MATRIX_APPSERVICE_SENDER"; const ENV_SENDER: &str = "MATRIX_APPSERVICE_SENDER";
/// The user namespace regex, rendered by nix. /// The user namespace regex, rendered by nix.
const ENV_USER_REGEX: &str = "MATRIX_APPSERVICE_USER_REGEX"; const ENV_USER_REGEX: &str = "MATRIX_APPSERVICE_USER_REGEX";
/// Role on the store's `cert` auth mount that [`sync`] logs in with. /// Role on the store's `cert` auth mount that both verbs log in with.
const ENV_CERT_ROLE: &str = "MATRIX_APPSERVICE_CERT_ROLE"; const ENV_CERT_ROLE: &str = "MATRIX_APPSERVICE_CERT_ROLE";
/// The unit that loads the registration, which [`sync`] restarts when the /// The unit that loads the registration, which [`sync`] restarts when the
/// registration is newer than its last start. /// registration is newer than its last start.
@ -48,7 +52,8 @@ const ENV_RESTART_UNIT: &str = "MATRIX_APPSERVICE_RESTART_UNIT";
/// The registration's `id`. /// The registration's `id`.
const ID: &str = "swarm"; const ID: &str = "swarm";
/// File names inside [`ENV_DIR`]. `REGISTRATION` is what tuwunel loads. /// `REGISTRATION` is what tuwunel loads, in [`ENV_RUNTIME_DIR`]. All three are
/// the names earlier versions used in [`ENV_LEGACY_DIR`].
const AS_TOKEN: &str = "as-token"; const AS_TOKEN: &str = "as-token";
const HS_TOKEN: &str = "hs-token"; const HS_TOKEN: &str = "hs-token";
const REGISTRATION: &str = "swarm.yaml"; const REGISTRATION: &str = "swarm.yaml";
@ -62,41 +67,51 @@ fn required(get: &impl Fn(&str) -> Option<String>, var: &'static str) -> Result<
.with_context(|| format!("{var} is unset or empty")) .with_context(|| format!("{var} is unset or empty"))
} }
/// Mint the tokens if absent and render the registration from them. /// The store, logged in as matrix-ctl.
async fn connect(cert_role: &str) -> Result<Bao> {
let settings = Settings::from_env().context("reading the store's BAO_* environment")?;
let store = SecretStore::connect(&settings, cert_role, DEFAULT_CERT_MOUNT)
.await
.with_context(|| {
format!("logging in to the swarm secret store as cert role {cert_role}")
})?;
Ok(Bao(store))
}
/// Render the registration from the store's tokens, which are minted first
/// when the store holds none.
/// ///
/// # Errors /// # Errors
/// If a variable is missing, or the state dir cannot be read or written. /// If a variable is missing, or the store refuses the login, a read or a
pub fn render() -> Result<()> { /// write, or the runtime dir cannot be written.
pub async fn render() -> Result<()> {
let get = |k: &str| std::env::var(k).ok(); let get = |k: &str| std::env::var(k).ok();
let dir = PathBuf::from(required(&get, ENV_DIR)?); let runtime = PathBuf::from(required(&get, ENV_RUNTIME_DIR)?);
let legacy = PathBuf::from(required(&get, ENV_LEGACY_DIR)?);
let sender = required(&get, ENV_SENDER)?; let sender = required(&get, ENV_SENDER)?;
let regex = required(&get, ENV_USER_REGEX)?; let regex = required(&get, ENV_USER_REGEX)?;
render_into(&dir, &sender, &regex)?; let store = connect(&required(&get, ENV_CERT_ROLE)?).await?;
tracing::info!(path = %dir.join(REGISTRATION).display(), "rendered the swarm appservice registration"); render_with(&store, &runtime, &legacy, &sender, &regex, mint).await?;
tracing::info!(path = %runtime.join(REGISTRATION).display(), "rendered the swarm appservice registration");
Ok(()) Ok(())
} }
/// [`render`] against an explicit directory, so a test can run it twice. /// [`render`] against an explicit store and directories.
fn render_into(dir: &Path, sender: &str, regex: &str) -> Result<()> { async fn render_with(
let as_token = existing_or_minted(&dir.join(AS_TOKEN))?; store: &impl Store,
let hs_token = existing_or_minted(&dir.join(HS_TOKEN))?; runtime: &Path,
legacy: &Path,
sender: &str,
regex: &str,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<()> {
let current = resolve(store, legacy, mint).await?;
write_secret( write_secret(
&dir.join(REGISTRATION), &runtime.join(REGISTRATION),
&registration_yaml(sender, regex, &as_token, &hs_token), &registration_yaml(sender, regex, &current.as_token, &current.hs_token),
) )
} }
/// The token at `path`, minting and writing one first when there is none.
fn existing_or_minted(path: &Path) -> Result<String> {
if let Some(t) = read_token(path)? {
return Ok(t);
}
let token = swarm_matrix_client::random_hex(TOKEN_BYTES)?;
write_secret(path, &token)?;
tracing::info!(path = %path.display(), "minted a swarm appservice token");
Ok(token)
}
/// The trimmed token at `path`; `None` when the file is missing or blank. /// The trimmed token at `path`; `None` when the file is missing or blank.
fn read_token(path: &Path) -> Result<Option<String>> { fn read_token(path: &Path) -> Result<Option<String>> {
match std::fs::read_to_string(path) { match std::fs::read_to_string(path) {
@ -107,21 +122,43 @@ fn read_token(path: &Path) -> Result<Option<String>> {
} }
} }
/// A temp path beside `path`, unique per call and per process, so two writers
/// of the same file never share one. Same directory, so the rename stays
/// atomic.
fn temp_path(path: &Path) -> PathBuf {
static NEXT: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0);
let mut name = path.file_name().unwrap_or_default().to_os_string();
name.push(format!(
".{}-{}.tmp",
std::process::id(),
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
));
path.with_file_name(name)
}
/// Write `contents` to `path` as a `0600` file, through a rename so a reader /// Write `contents` to `path` as a `0600` file, through a rename so a reader
/// never sees half of it. /// never sees half of it. A failure removes the temp file.
fn write_secret(path: &Path, contents: &str) -> Result<()> { fn write_secret(path: &Path, contents: &str) -> Result<()> {
let tmp = path.with_extension("tmp"); let tmp = temp_path(path);
let _ = std::fs::remove_file(&tmp); let written = write_temp(&tmp, contents).and_then(|()| {
std::fs::rename(&tmp, path).with_context(|| format!("renaming onto {}", path.display()))
});
if written.is_err() {
let _ = std::fs::remove_file(&tmp);
}
written
}
fn write_temp(tmp: &Path, contents: &str) -> Result<()> {
let mut f = std::fs::OpenOptions::new() let mut f = std::fs::OpenOptions::new()
.write(true) .write(true)
.create_new(true) .create_new(true)
.mode(0o600) .mode(0o600)
.open(&tmp) .open(tmp)
.with_context(|| format!("creating {}", tmp.display()))?; .with_context(|| format!("creating {}", tmp.display()))?;
f.write_all(contents.as_bytes()) f.write_all(contents.as_bytes())
.and_then(|()| f.sync_all()) .and_then(|()| f.sync_all())
.with_context(|| format!("writing {}", tmp.display()))?; .with_context(|| format!("writing {}", tmp.display()))
std::fs::rename(&tmp, path).with_context(|| format!("renaming onto {}", path.display()))
} }
/// The registration tuwunel loads. /// The registration tuwunel loads.
@ -146,7 +183,7 @@ fn registration_yaml(sender: &str, regex: &str, as_token: &str, hs_token: &str)
) )
} }
/// What [`sync`] needs of the store, so the decisions can be tested without /// What [`render`] and [`sync`] need of the store, so the decisions can be tested without
/// one. /// one.
trait Store { trait Store {
/// The registration's tokens, or `None` when nothing is stored. /// The registration's tokens, or `None` when nothing is stored.
@ -201,55 +238,55 @@ impl Store for Bao {
} }
} }
/// Make the state dir hold the store's registration, and the controller's token /// Make the runtime dir hold the store's registration, and the controller's
/// match it. /// token match it.
/// ///
/// # Errors /// # Errors
/// If a variable is missing, the store refuses the login, a read or a write, /// If a variable is missing, the store refuses the login, a read or a write,
/// the state dir cannot be written, or the homeserver cannot be restarted. /// the runtime dir cannot be written, or the homeserver cannot be restarted.
pub async fn sync() -> Result<()> { pub async fn sync() -> Result<()> {
let get = |k: &str| std::env::var(k).ok(); let get = |k: &str| std::env::var(k).ok();
let dir = PathBuf::from(required(&get, ENV_DIR)?); let runtime = PathBuf::from(required(&get, ENV_RUNTIME_DIR)?);
let legacy = PathBuf::from(required(&get, ENV_LEGACY_DIR)?);
let sender = required(&get, ENV_SENDER)?; let sender = required(&get, ENV_SENDER)?;
let regex = required(&get, ENV_USER_REGEX)?; let regex = required(&get, ENV_USER_REGEX)?;
let cert_role = required(&get, ENV_CERT_ROLE)?;
let unit = required(&get, ENV_RESTART_UNIT)?; let unit = required(&get, ENV_RESTART_UNIT)?;
let store = connect(&required(&get, ENV_CERT_ROLE)?).await?;
let settings = Settings::from_env().context("reading the store's BAO_* environment")?; sync_with(
let store = SecretStore::connect(&settings, &cert_role, DEFAULT_CERT_MOUNT) &store,
.await &runtime,
.with_context(|| { &legacy,
format!("logging in to the swarm secret store as cert role {cert_role}") &sender,
})?; &regex,
sync_with(&Bao(store), &dir, &sender, &regex, mint, &Systemd(unit)).await mint,
&Systemd(unit),
)
.await
} }
/// [`sync`] against an explicit store, directory and homeserver unit. /// [`sync`] against an explicit store, directories and homeserver unit.
/// ///
/// The store's registration wins. Where it holds none, the state dir's tokens /// The homeserver is restarted whenever the registration was written after it
/// seed it, and only a host with neither mints a pair. The homeserver is /// last started, so a run that failed between the write and the restart is
/// restarted whenever the registration was written after it last started, so /// finished by the next.
/// a run that failed between the write and the restart is finished by the next.
async fn sync_with( async fn sync_with(
store: &impl Store, store: &impl Store,
dir: &Path, runtime: &Path,
legacy: &Path,
sender: &str, sender: &str,
regex: &str, regex: &str,
mint: impl FnOnce() -> Result<Registration>, mint: impl FnOnce() -> Result<Registration>,
unit: &impl Homeserver, unit: &impl Homeserver,
) -> Result<()> { ) -> Result<()> {
let local = read_local(dir)?; let current = resolve(store, legacy, mint).await?;
let current = stored_or_seeded(store, local.as_ref(), mint).await?;
let path = runtime.join(REGISTRATION);
let want = registration_yaml(sender, regex, &current.as_token, &current.hs_token); let want = registration_yaml(sender, regex, &current.as_token, &current.hs_token);
let have = std::fs::read_to_string(dir.join(REGISTRATION)).ok(); if std::fs::read_to_string(&path).ok().as_deref() != Some(&want) {
if local.as_ref() != Some(&current) || have.as_deref() != Some(&want) { write_secret(&path, &want)?;
write_secret(&dir.join(AS_TOKEN), &current.as_token)?;
write_secret(&dir.join(HS_TOKEN), &current.hs_token)?;
write_secret(&dir.join(REGISTRATION), &want)?;
tracing::info!("took the swarm appservice registration from the store"); tracing::info!("took the swarm appservice registration from the store");
} }
restart_if_stale(unit, &dir.join(REGISTRATION))?; restart_if_stale(unit, &path)?;
if needs_publish(store.controller_token().await?.as_ref(), &current.as_token) { if needs_publish(store.controller_token().await?.as_ref(), &current.as_token) {
store.write_controller_token(&current.as_token).await?; store.write_controller_token(&current.as_token).await?;
@ -258,17 +295,31 @@ async fn sync_with(
Ok(()) Ok(())
} }
/// The store's registration; where it holds none, `local` or a minted pair is /// The registration the store holds. Where it holds none, the tokens an
/// created there first. A refused create is a lost race or a denial, and the /// earlier version kept in `legacy` are stored once (create-only, so nothing
/// read-back tells which: it finds the winner's value, or nothing. /// rotates), and only a host with neither mints a pair. A store that cannot
async fn stored_or_seeded( /// be read is an error and mints nothing. The files stay where they are,
/// unused.
async fn resolve(
store: &impl Store,
legacy: &Path,
mint: impl FnOnce() -> Result<Registration>,
) -> Result<Registration> {
match store.registration().await? {
Some(stored) => Ok(stored),
None => seed_store(store, read_local(legacy)?.as_ref(), mint).await,
}
}
/// Create the registration in the empty store from `local`, or from a minted
/// pair when there are no files, and return what the store then holds. A
/// refused create is a lost race or a denial, and the read-back tells which:
/// it finds the winner's pair, or nothing.
async fn seed_store(
store: &impl Store, store: &impl Store,
local: Option<&Registration>, local: Option<&Registration>,
mint: impl FnOnce() -> Result<Registration>, mint: impl FnOnce() -> Result<Registration>,
) -> Result<Registration> { ) -> Result<Registration> {
if let Some(stored) = store.registration().await? {
return Ok(stored);
}
let seed = match local { let seed = match local {
Some(l) => l.clone(), Some(l) => l.clone(),
None => mint()?, None => mint()?,
@ -282,7 +333,7 @@ async fn stored_or_seeded(
.context("the store holds no swarm appservice registration after creating one") .context("the store holds no swarm appservice registration after creating one")
} }
/// The tokens on disk, when both are there. /// The tokens in `dir`, when both are there.
fn read_local(dir: &Path) -> Result<Option<Registration>> { fn read_local(dir: &Path) -> Result<Option<Registration>> {
let as_token = read_token(&dir.join(AS_TOKEN))?; let as_token = read_token(&dir.join(AS_TOKEN))?;
let hs_token = read_token(&dir.join(HS_TOKEN))?; let hs_token = read_token(&dir.join(HS_TOKEN))?;
@ -447,18 +498,24 @@ mod tests {
const REGEX: &str = "^@[a-z0-9._=/-]+:t\\.local$"; const REGEX: &str = "^@[a-z0-9._=/-]+:t\\.local$";
/// A directory of its own per call. A counter, not the clock, keeps two /// A runtime dir and a legacy dir of their own per call; the legacy one
/// tests that start in the same clock tick out of one directory. /// is not created, as on a host that never had the files. A counter, not
fn scratch() -> PathBuf { /// the clock, keeps two tests that start in the same clock tick out of one
/// directory.
fn scratch() -> (PathBuf, PathBuf) {
static NEXT: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0); static NEXT: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0);
let dir = std::env::temp_dir().join(format!( let root = std::env::temp_dir().join(format!(
"swarm-appservice-{}-{}", "swarm-appservice-{}-{}",
std::process::id(), std::process::id(),
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed) NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
)); ));
std::fs::remove_dir_all(&dir).ok(); std::fs::remove_dir_all(&root).ok();
std::fs::create_dir_all(&dir).expect("temp dir"); std::fs::create_dir_all(root.join("run")).expect("temp dir");
dir (root.join("run"), root.join("legacy"))
}
fn cleanup(runtime: &Path) {
std::fs::remove_dir_all(runtime.parent().expect("a parent")).ok();
} }
fn reg(tag: &str) -> Registration { fn reg(tag: &str) -> Registration {
@ -469,18 +526,34 @@ mod tests {
} }
/// A store in memory. `down` fails every call, as a sealed or unreachable /// A store in memory. `down` fails every call, as a sealed or unreachable
/// store does; `racer` is what another writer stores between this run's /// store does; `deny_create` refuses a create and stores nothing; `racer`
/// read and its create. /// is what another writer stores between this run's read and its create.
#[derive(Default)] #[derive(Default)]
struct Fake { struct Fake {
registration: RefCell<Option<Registration>>, registration: RefCell<Option<Registration>>,
token: RefCell<Option<String>>, token: RefCell<Option<String>>,
down: bool, down: bool,
deny_create: bool,
racer: RefCell<Option<Registration>>, racer: RefCell<Option<Registration>>,
creates: Cell<u32>, creates: Cell<u32>,
token_writes: Cell<u32>, token_writes: Cell<u32>,
} }
impl Fake {
fn holding(r: Registration) -> Self {
let f = Self::default();
*f.registration.borrow_mut() = Some(r);
f
}
fn down() -> Self {
Self {
down: true,
..Self::default()
}
}
}
impl Store for Fake { impl Store for Fake {
async fn registration(&self) -> Result<Option<Registration>> { async fn registration(&self) -> Result<Option<Registration>> {
if self.down { if self.down {
@ -491,6 +564,9 @@ mod tests {
async fn create_registration(&self, r: &Registration) -> Result<()> { async fn create_registration(&self, r: &Registration) -> Result<()> {
self.creates.set(self.creates.get() + 1); self.creates.set(self.creates.get() + 1);
if self.deny_create {
bail!("permission denied");
}
if let Some(winner) = self.racer.borrow_mut().take() { if let Some(winner) = self.racer.borrow_mut().take() {
*self.registration.borrow_mut() = Some(winner); *self.registration.borrow_mut() = Some(winner);
bail!("already created"); bail!("already created");
@ -513,8 +589,9 @@ mod tests {
} }
} }
/// Write the state dir as `render` leaves it for `r`. /// The three files an earlier version left in the legacy dir for `r`.
fn host_with(dir: &Path, r: &Registration) { fn legacy_with(dir: &Path, r: &Registration) {
std::fs::create_dir_all(dir).expect("legacy dir");
write_secret(&dir.join(AS_TOKEN), &r.as_token).expect("as"); write_secret(&dir.join(AS_TOKEN), &r.as_token).expect("as");
write_secret(&dir.join(HS_TOKEN), &r.hs_token).expect("hs"); write_secret(&dir.join(HS_TOKEN), &r.hs_token).expect("hs");
write_secret( write_secret(
@ -524,11 +601,25 @@ mod tests {
.expect("yaml"); .expect("yaml");
} }
fn files(dir: &Path) -> Vec<String> { /// What is left of the legacy files: the dir's file names, sorted.
[AS_TOKEN, HS_TOKEN, REGISTRATION] fn legacy_left(dir: &Path) -> Vec<String> {
.iter() let mut names: Vec<String> = std::fs::read_dir(dir)
.map(|f| std::fs::read_to_string(dir.join(f)).unwrap_or_default()) .map(|d| {
.collect() d.map(|e| e.expect("entry").file_name().to_string_lossy().into_owned())
.collect()
})
.unwrap_or_default();
names.sort();
names
}
/// The registration the runtime dir holds, or empty.
fn rendered(runtime: &Path) -> String {
std::fs::read_to_string(runtime.join(REGISTRATION)).unwrap_or_default()
}
fn yaml_for(r: &Registration) -> String {
registration_yaml("swarm", REGEX, &r.as_token, &r.hs_token)
} }
/// The homeserver unit. `started` is when it last became active; a restart /// The homeserver unit. `started` is when it last became active; a restart
@ -574,12 +665,24 @@ mod tests {
} }
} }
/// Run one render, returning how often it minted.
async fn render_run(store: &Fake, runtime: &Path, legacy: &Path) -> (Result<()>, u32) {
let mints = Cell::new(0);
let out = render_with(store, runtime, legacy, "swarm", REGEX, || {
mints.set(mints.get() + 1);
Ok(reg("minted"))
})
.await;
(out, mints.get())
}
/// Run one sync against `unit`, returning how often it minted. /// Run one sync against `unit`, returning how often it minted.
async fn run(store: &Fake, dir: &Path, unit: &Unit) -> (Result<()>, u32) { async fn run(store: &Fake, runtime: &Path, legacy: &Path, unit: &Unit) -> (Result<()>, u32) {
let mints = Cell::new(0); let mints = Cell::new(0);
let out = sync_with( let out = sync_with(
store, store,
dir, runtime,
legacy,
"swarm", "swarm",
REGEX, REGEX,
|| { || {
@ -592,28 +695,154 @@ mod tests {
(out, mints.get()) (out, mints.get())
} }
#[test] #[tokio::test]
fn the_rendered_registration_carries_the_token_it_minted() { async fn an_empty_store_and_no_files_mint_a_pair_into_the_store() {
let dir = scratch(); let (runtime, legacy) = scratch();
render_into(&dir, "swarm", REGEX).expect("renders"); let store = Fake::default();
let yaml = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered");
let token = std::fs::read_to_string(dir.join(AS_TOKEN)).expect("minted"); let (out, mints) = render_run(&store, &runtime, &legacy).await;
assert_eq!(token.len(), TOKEN_BYTES * 2);
assert!(yaml.contains(&format!("\nas_token: {token}\n")), "{yaml}"); out.expect("renders");
std::fs::remove_dir_all(&dir).ok(); assert_eq!(mints, 1);
assert_eq!(*store.registration.borrow(), Some(reg("minted")));
assert_eq!(rendered(&runtime), yaml_for(&reg("minted")));
assert!(!legacy.exists(), "nothing is written to the legacy dir");
cleanup(&runtime);
}
#[tokio::test]
async fn an_empty_store_takes_the_files_value_once() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("old"));
let store = Fake::default();
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(mints, 0, "a host that has the value mints nothing");
assert_eq!(*store.registration.borrow(), Some(reg("old")));
assert_eq!(rendered(&runtime), yaml_for(&reg("old")));
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_mints_nothing_and_renders_nothing() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let (out, mints) = render_run(&Fake::down(), &runtime, &legacy).await;
assert!(out.is_err(), "the unit fails and retries");
assert_eq!(mints, 0);
assert_eq!(rendered(&runtime), "", "nothing for tuwunel to load");
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_on_a_fresh_host_mints_nothing() {
let (runtime, legacy) = scratch();
let (out, mints) = render_run(&Fake::down(), &runtime, &legacy).await;
assert!(out.is_err());
assert_eq!(
mints, 0,
"a pair minted while the store is away is a rotation"
);
assert_eq!(rendered(&runtime), "");
cleanup(&runtime);
}
#[tokio::test]
async fn a_lost_race_adopts_the_winners_pair() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let store = Fake::default();
*store.racer.borrow_mut() = Some(reg("winner"));
let (out, _) = render_run(&store, &runtime, &legacy).await;
out.expect("renders");
assert_eq!(rendered(&runtime), yaml_for(&reg("winner")));
cleanup(&runtime);
}
#[tokio::test]
async fn a_refused_create_fails_the_run() {
let (runtime, legacy) = scratch();
legacy_with(&legacy, &reg("disk"));
let store = Fake {
deny_create: true,
..Fake::default()
};
let (out, _) = render_run(&store, &runtime, &legacy).await;
assert!(out.is_err(), "no read-back finds the pair");
assert_eq!(rendered(&runtime), "");
cleanup(&runtime);
}
#[tokio::test]
async fn a_second_render_keeps_the_pair() {
// "Only once": a re-mint on every boot would hand the controller a
// token the homeserver no longer loads until sync catches up.
let (runtime, legacy) = scratch();
let store = Fake::default();
render_run(&store, &runtime, &legacy)
.await
.0
.expect("first");
let first = rendered(&runtime);
let (out, mints) = render_run(&store, &runtime, &legacy).await;
out.expect("second");
assert_eq!(mints, 0);
assert_eq!(rendered(&runtime), first);
cleanup(&runtime);
} }
#[test] #[test]
fn a_second_render_keeps_the_token() { fn two_writers_of_one_file_use_different_temp_files() {
// "Only once": a re-mint on every boot would hand the controller a let path = Path::new("/run/x/swarm.yaml");
// token the homeserver no longer loads until sync catches up. let (a, b) = (temp_path(path), temp_path(path));
let dir = scratch(); assert_ne!(a, b);
render_into(&dir, "swarm", REGEX).expect("first"); assert_eq!(
let first = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered"); a.parent(),
render_into(&dir, "swarm", REGEX).expect("second"); path.parent(),
let second = std::fs::read_to_string(dir.join(REGISTRATION)).expect("rendered"); "the rename stays in one directory"
assert_eq!(first, second); );
std::fs::remove_dir_all(&dir).ok(); }
#[test]
fn overlapping_writers_all_succeed_and_leave_no_temp_file() {
let (runtime, _) = scratch();
let path = runtime.join(REGISTRATION);
let writers: Vec<_> = (0..8)
.map(|_| {
let path = path.clone();
std::thread::spawn(move || {
(0..50)
.try_for_each(|_| write_secret(&path, "same\n"))
.is_ok()
})
})
.collect();
assert!(writers.into_iter().all(|w| w.join().expect("joins")));
assert_eq!(std::fs::read_to_string(&path).expect("written"), "same\n");
assert_eq!(legacy_left(&runtime), [REGISTRATION]);
cleanup(&runtime);
}
#[test]
fn a_failed_write_removes_its_temp_file() {
let (runtime, _) = scratch();
// A directory where the file goes: the rename fails.
std::fs::create_dir(runtime.join(REGISTRATION)).expect("dir");
write_secret(&runtime.join(REGISTRATION), "x").expect_err("cannot replace a directory");
assert_eq!(legacy_left(&runtime), [REGISTRATION]);
cleanup(&runtime);
} }
#[test] #[test]
@ -624,172 +853,86 @@ mod tests {
assert!(y.contains("\n - exclusive: false\n"), "{y}"); assert!(y.contains("\n - exclusive: false\n"), "{y}");
assert!(y.contains(&format!("regex: '{REGEX}'")), "{y}"); assert!(y.contains(&format!("regex: '{REGEX}'")), "{y}");
assert!(y.contains("\nurl: null\n"), "{y}"); assert!(y.contains("\nurl: null\n"), "{y}");
assert!(y.contains("\nas_token: aa\n"), "{y}");
assert!(y.contains("\nhs_token: bb\n"), "{y}");
} }
#[test] #[tokio::test]
fn the_registration_and_tokens_are_owner_only() { async fn the_rendered_registration_is_owner_only() {
use std::os::unix::fs::PermissionsExt as _; use std::os::unix::fs::PermissionsExt as _;
let dir = scratch(); let (runtime, legacy) = scratch();
render_into(&dir, "swarm", REGEX).expect("renders"); render_run(&Fake::default(), &runtime, &legacy)
for f in [AS_TOKEN, HS_TOKEN, REGISTRATION] {
let mode = std::fs::metadata(dir.join(f))
.expect("exists")
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600, "{f}");
}
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn seeding_from_the_host_leaves_every_file_unchanged() {
// A host that has run for months with the store empty: the store takes
// its value, and nothing on the host moves, so nothing rotates.
let dir = scratch();
let on_disk = reg("old");
host_with(&dir, &on_disk);
let before = files(&dir);
let store = Fake::default();
let unit = Unit::just_started();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(*store.registration.borrow(), Some(on_disk));
assert_eq!(files(&dir), before);
assert_eq!(mints, 0, "a host that has the value mints nothing");
assert_eq!(restarts, 0, "an unchanged host does not restart");
assert_eq!(store.token.borrow().as_deref(), Some("as-old"));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn the_store_wins_when_both_hold_a_value() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
*store.token.borrow_mut() = Some("as-disk".to_owned());
let unit = Unit::long_running();
let (out, _) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
let want = registration_yaml("swarm", REGEX, "as-store", "hs-store");
assert_eq!(
files(&dir),
vec!["as-store".to_owned(), "hs-store".to_owned(), want]
);
assert_eq!(*store.registration.borrow(), Some(reg("store")));
assert_eq!(store.creates.get(), 0, "a held value is never rewritten");
assert_eq!(restarts, 1);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_host_without_the_files_takes_them_from_the_store() {
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
let unit = Unit::long_running();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(read_local(&dir).expect("reads"), Some(reg("store")));
assert!(dir.join(REGISTRATION).exists());
assert_eq!((restarts, mints), (1, 0));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_fresh_host_mints_into_the_store() {
let dir = scratch();
let store = Fake::default();
let unit = Unit::long_running();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(mints, 1);
assert_eq!(*store.registration.borrow(), Some(reg("minted")));
assert_eq!(read_local(&dir).expect("reads"), Some(reg("minted")));
assert_eq!(restarts, 1);
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_lost_race_adopts_the_winners_registration() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let store = Fake::default();
*store.racer.borrow_mut() = Some(reg("winner"));
let unit = Unit::long_running();
let (out, _) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
out.expect("syncs");
assert_eq!(read_local(&dir).expect("reads"), Some(reg("winner")));
assert_eq!(restarts, 1);
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn a_store_that_is_down_leaves_the_host_as_it_is() {
let dir = scratch();
host_with(&dir, &reg("disk"));
let before = files(&dir);
let store = Fake {
down: true,
..Fake::default()
};
let unit = Unit::just_started();
let (out, mints) = run(&store, &dir, &unit).await;
let restarts = unit.restarts.get();
assert!(out.is_err(), "the unit reports the outage");
assert_eq!(files(&dir), before, "the homeserver keeps the last copy");
assert_eq!((restarts, mints), (0, 0));
std::fs::remove_dir_all(&dir).ok();
}
#[tokio::test]
async fn the_start_path_renders_the_copy_sync_took_from_the_store() {
// `render` has no store: after an outage it renders what the last sync
// wrote, without minting over it.
let dir = scratch();
let store = Fake::default();
*store.registration.borrow_mut() = Some(reg("store"));
run(&store, &dir, &Unit::long_running())
.await .await
.0 .0
.expect("syncs"); .expect("renders");
let synced = files(&dir); let mode = std::fs::metadata(runtime.join(REGISTRATION))
.expect("exists")
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600);
cleanup(&runtime);
}
render_into(&dir, "swarm", REGEX).expect("renders"); #[tokio::test]
async fn the_store_wins_when_the_rendered_copy_differs() {
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("old"))).expect("rendered");
let store = Fake::holding(reg("store"));
*store.token.borrow_mut() = Some("as-old".to_owned());
assert_eq!(files(&dir), synced); let unit = Unit::long_running();
std::fs::remove_dir_all(&dir).ok(); let (out, _) = run(&store, &runtime, &legacy, &unit).await;
out.expect("syncs");
assert_eq!(rendered(&runtime), yaml_for(&reg("store")));
assert_eq!(*store.registration.borrow(), Some(reg("store")));
assert_eq!(store.creates.get(), 0, "a held value is never rewritten");
assert_eq!(unit.restarts.get(), 1);
assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
cleanup(&runtime);
}
#[tokio::test]
async fn a_sync_with_nothing_rendered_takes_the_stores_pair() {
let (runtime, legacy) = scratch();
let store = Fake::holding(reg("store"));
let unit = Unit::long_running();
let (out, mints) = run(&store, &runtime, &legacy, &unit).await;
out.expect("syncs");
assert_eq!(rendered(&runtime), yaml_for(&reg("store")));
assert_eq!((unit.restarts.get(), mints), (1, 0));
cleanup(&runtime);
}
#[tokio::test]
async fn a_store_that_is_down_leaves_the_rendered_copy_as_it_is() {
let (runtime, legacy) = scratch();
std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("disk"))).expect("rendered");
let unit = Unit::just_started();
let (out, mints) = run(&Fake::down(), &runtime, &legacy, &unit).await;
assert!(out.is_err(), "the unit reports the outage");
assert_eq!(rendered(&runtime), yaml_for(&reg("disk")));
assert_eq!((unit.restarts.get(), mints), (0, 0));
cleanup(&runtime);
} }
#[tokio::test] #[tokio::test]
async fn a_second_sync_restarts_nothing() { async fn a_second_sync_restarts_nothing() {
let dir = scratch(); let (runtime, legacy) = scratch();
let store = Fake::default(); let store = Fake::holding(reg("store"));
*store.registration.borrow_mut() = Some(reg("store"));
let unit = Unit::long_running(); let unit = Unit::long_running();
run(&store, &dir, &unit).await.0.expect("first"); run(&store, &runtime, &legacy, &unit)
.await
.0
.expect("first");
let first = unit.restarts.get(); let first = unit.restarts.get();
let (out, _) = run(&store, &dir, &unit).await; let (out, _) = run(&store, &runtime, &legacy, &unit).await;
let second = unit.restarts.get() - first; let second = unit.restarts.get() - first;
out.expect("syncs"); out.expect("syncs");
@ -799,48 +942,48 @@ mod tests {
1, 1,
"the controller's copy is written once" "the controller's copy is written once"
); );
std::fs::remove_dir_all(&dir).ok(); cleanup(&runtime);
} }
#[tokio::test] #[tokio::test]
async fn a_restart_that_failed_is_retried_by_the_next_run() { async fn a_restart_that_failed_is_retried_by_the_next_run() {
// The files are written and the restart is refused. The next run finds // The file is written and the restart is refused. The next run finds
// nothing to write, and still restarts: the decision comes from the // nothing to write, and still restarts: the decision comes from the
// files' age against the unit's start, not from this run's writes. // file's age against the unit's start, not from this run's writes.
let dir = scratch(); let (runtime, legacy) = scratch();
let store = Fake::default(); let store = Fake::holding(reg("store"));
*store.registration.borrow_mut() = Some(reg("store"));
let unit = Unit::long_running(); let unit = Unit::long_running();
unit.fail_restart.set(true); unit.fail_restart.set(true);
let (first, _) = run(&store, &dir, &unit).await; let (first, _) = run(&store, &runtime, &legacy, &unit).await;
assert!(first.is_err(), "the refused restart fails the run"); assert!(first.is_err(), "the refused restart fails the run");
let written = files(&dir); let written = rendered(&runtime);
unit.fail_restart.set(false); unit.fail_restart.set(false);
let (second, _) = run(&store, &dir, &unit).await; let (second, _) = run(&store, &runtime, &legacy, &unit).await;
second.expect("the retry succeeds"); second.expect("the retry succeeds");
assert_eq!(files(&dir), written, "nothing was left to write"); assert_eq!(rendered(&runtime), written, "nothing was left to write");
assert_eq!(unit.restarts.get(), 2); assert_eq!(unit.restarts.get(), 2);
assert_eq!(read_local(&dir).expect("reads"), Some(reg("store"))); cleanup(&runtime);
std::fs::remove_dir_all(&dir).ok();
} }
#[tokio::test] #[tokio::test]
async fn a_unit_that_started_after_the_files_is_left_alone() { async fn a_unit_that_started_after_the_file_is_left_alone() {
// The control for the retry above: the same files, a unit that started // The control for the retry above: the same file, a unit that started
// after them, and no restart. // after it, and no restart.
let dir = scratch(); let (runtime, legacy) = scratch();
host_with(&dir, &reg("store")); std::fs::write(runtime.join(REGISTRATION), yaml_for(&reg("store"))).expect("rendered");
let store = Fake::default(); let store = Fake::holding(reg("store"));
*store.registration.borrow_mut() = Some(reg("store"));
let unit = Unit::just_started(); let unit = Unit::just_started();
run(&store, &dir, &unit).await.0.expect("syncs"); run(&store, &runtime, &legacy, &unit)
.await
.0
.expect("syncs");
assert_eq!(unit.restarts.get(), 0); assert_eq!(unit.restarts.get(), 0);
std::fs::remove_dir_all(&dir).ok(); cleanup(&runtime);
} }
#[tokio::test] #[tokio::test]
@ -855,17 +998,25 @@ mod tests {
Ok(()) Ok(())
} }
} }
let dir = scratch(); let (runtime, legacy) = scratch();
let store = Fake::default(); let store = Fake::holding(reg("store"));
*store.registration.borrow_mut() = Some(reg("store"));
let unit = Unreadable(Cell::new(0)); let unit = Unreadable(Cell::new(0));
let out = sync_with(&store, &dir, "swarm", REGEX, || Ok(reg("minted")), &unit).await; let out = sync_with(
&store,
&runtime,
&legacy,
"swarm",
REGEX,
|| Ok(reg("minted")),
&unit,
)
.await;
out.expect("the run goes on"); out.expect("the run goes on");
assert_eq!(unit.0.get(), 0); assert_eq!(unit.0.get(), 0);
assert_eq!(store.token.borrow().as_deref(), Some("as-store")); assert_eq!(store.token.borrow().as_deref(), Some("as-store"));
std::fs::remove_dir_all(&dir).ok(); cleanup(&runtime);
} }
/// `restart_via` against a `systemctl` that exits with the given codes for the /// `restart_via` against a `systemctl` that exits with the given codes for the
@ -939,7 +1090,8 @@ mod tests {
#[test] #[test]
fn every_variable_is_scoped_to_the_verb() { fn every_variable_is_scoped_to_the_verb() {
for var in [ for var in [
ENV_DIR, ENV_RUNTIME_DIR,
ENV_LEGACY_DIR,
ENV_SENDER, ENV_SENDER,
ENV_USER_REGEX, ENV_USER_REGEX,
ENV_CERT_ROLE, ENV_CERT_ROLE,

17
swarm-matrix-ctl/src/main.rs
View file

@ -7,8 +7,8 @@
//! identity plumbing to add one action, so the next thing that has to run in //! identity plumbing to add one action, so the next thing that has to run in
//! here is a verb below, not a new crate. //! here is a verb below, not a new crate.
//! //!
//! [`appservice`] mints the **swarm's** own appservice registration and //! [`appservice`] renders the **swarm's** own appservice registration from
//! keeps it in step with the store and publishes its token for `swarm-controller`. //! the store and publishes its token for `swarm-controller`.
//! //!
//! 🩸 **A secret is a path, never a value.** The only identifier any verb here //! 🩸 **A secret is a path, never a value.** The only identifier any verb here
//! logs is the store path; see `swarm_matrix_client`'s module doc for the same rule //! logs is the store path; see `swarm_matrix_client`'s module doc for the same rule
@ -39,12 +39,13 @@ enum Command {
#[derive(Debug, Subcommand)] #[derive(Debug, Subcommand)]
enum Appservice { enum Appservice {
/// Mint the tokens when absent and render the registration tuwunel loads. /// Render the registration tuwunel loads from the store, minting the tokens
/// Local only: it runs before the homeserver and must not need a network. /// first when the store holds none. Runs before the homeserver, which stays
/// down until the store answers.
Render, Render,
/// Make the state dir hold the store's registration, seeding the store /// Make the rendered registration match the store's, restart the homeserver
/// from it when the store holds none, and restart the homeserver when the /// when the registration is newer than its last start, and publish the
/// registration is newer than its last start. Runs beside the homeserver, never before it. /// `as_token` for `swarm-controller`. Runs beside the homeserver.
Sync, Sync,
} }
@ -58,7 +59,7 @@ async fn main() -> Result<()> {
.init(); .init();
match Cli::parse().command { match Cli::parse().command {
Command::Appservice(Appservice::Render) => appservice::render(), Command::Appservice(Appservice::Render) => appservice::render().await,
Command::Appservice(Appservice::Sync) => appservice::sync().await, Command::Appservice(Appservice::Sync) => appservice::sync().await,
} }
} }