770f68c272
The pipe contract had a gap: if a producer piped into atomic_write_secret exited non-zero after writing partial output, cat still saw a clean EOF and wrote that partial content through to the live target via mv — pipefail only reported the failure afterward, once the bad write was already committed. The helper now takes the value as its 4th argument and writes it itself with printf (a shell builtin, so the value never touches an external process's own argv/environ, same as a function argument never does), so there is no pipe left to fail silently. Callers that compute the value with a command now capture it into a variable first (`value=$(cmd)`), which fails under `set -e` before atomic_write_secret is ever called — swarm-bao.nix's pin.env site is the one that needed this (`pin_env_value="BAO_HSM_PIN=$(cat ...)"`). All seven call sites converted; output is byte-identical (same printf '%s\n' framing, now applied inside the helper instead of by each caller). Refs #4723
261 lines
13 KiB
Nix
261 lines
13 KiB
Nix
# Glue: the matrix appservice token comes from the secret store.
|
||
#
|
||
# The store's first reader, and deliberately a small one. It fetches an opaque
|
||
# 32-byte value and writes it where ./hive-matrix.nix already looks, then asks
|
||
# that module's own renderer to re-stamp the appservice registration naming it
|
||
# — the homeserver never learns the store exists, and its config is unchanged.
|
||
#
|
||
# ⚠️ Why this credential first. It has no second file and no format: authelia's
|
||
# OIDC secret needs a `.secret` *and* a matching `.digest`, so shipping that
|
||
# one first would debug "can a reader authenticate and get bytes back" and
|
||
# "did we write authelia's file format right" at the same time, with an SSO
|
||
# outage as the failure mode. Here the failure is narrow — new agent accounts
|
||
# cannot be provisioned, existing ones are untouched, nothing crash-loops.
|
||
#
|
||
# ⚠️ The fallback is today's behaviour, not a new one. `hive-matrix.nix`'s
|
||
# activation script still mints a token when the file is absent; this unit
|
||
# overwrites it with the swarm's copy when the store has one. A store that is
|
||
# empty or unreachable leaves a working hive with a local token.
|
||
#
|
||
# 📌 This runs wherever a client identity is configured, NOT only where the
|
||
# store is. `deploy.bao.enable` would have been the co-location assumption
|
||
# itself; the reader needs a certificate, not a neighbour. On the store's own
|
||
# host ./glue-bao-tls.nix supplies one as a `mkDefault` and nothing changes;
|
||
# elsewhere an operator places the leaf and names it, and the same unit works.
|
||
{
|
||
pkgs,
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
deployCfg = hyperhiveCfg.deploy;
|
||
baoCfg = hyperhiveCfg.swarm.bao;
|
||
|
||
baoDeploy = deployCfg.bao;
|
||
|
||
# What decides whether this unit exists at all. A reader is defined by holding
|
||
# a certificate the store accepts, and that is true on the store's own host
|
||
# and on a hive three networks away for exactly the same reason.
|
||
#
|
||
# 🩸 This principal's OWN leaf, not `clientCertFile` — the hive's, which four
|
||
# units used to share. Bao matches a cert-auth role on the CN, so one leaf for
|
||
# four readers was ONE principal holding the union of four grants: read on
|
||
# `swarm/agents/*` AND `swarm/hives/<hive>/*` AND `swarm/services/*`, when
|
||
# this unit reads one appservice token and nothing else. Its own leaf carries
|
||
# `<matrixTokenCommonNamePrefix>-<hive>` and its role grants the single path
|
||
# below.
|
||
haveClientIdentity =
|
||
baoDeploy.matrixTokenClientCertFile != null && baoDeploy.matrixTokenClientKeyFile != null;
|
||
|
||
# Does this host read the store at all — the hive's own leaf, which is the
|
||
# one thing a remote-store deployment has always had to place by hand. Only
|
||
# used to decide whether a missing per-principal leaf is a mistake or a
|
||
# deployment that has no store: a host holding neither is the supported
|
||
# no-store shape, and one holding this pair but not the pair above named
|
||
# seven of the eight options and stopped.
|
||
hiveReaderIdentity = baoDeploy.clientCertFile != null && baoDeploy.clientKeyFile != null;
|
||
|
||
# Where the token lives in the store. A path, not a convention to guess at:
|
||
# whoever writes it and whoever reads it must agree, and the agreement
|
||
# belongs in one visible place.
|
||
#
|
||
# ⚠️ The `hives/<name>` segment is not decoration — it is what the reader's
|
||
# own grant covers. The store's read policy grants `swarm/agents/*` and
|
||
# `swarm/hives/<this hive>/*` and nothing else, so a path outside those is a
|
||
# 403 rather than a miss, however correct it looks. `swarm-secret-client`'s
|
||
# `matrix::appservice_token_path` builds the same string from the same
|
||
# pieces; this literal is the nix half of that one agreement.
|
||
#
|
||
# `hiveName` has no fallback here for the reason ./glue-bao-tls.nix gives at
|
||
# its own use of it: it is asserted set on every host that runs the
|
||
# homeserver.
|
||
tokenPath = "secret/swarm/hives/${hyperhiveCfg.hiveName}/matrix/appservice-token";
|
||
|
||
# A literal, not an option — ./hive-matrix.nix names its container
|
||
# `containers.hive-matrix` directly and declares no `machine` to derive it
|
||
# from, which the trust-bundle call in that file already says out loud.
|
||
# ⚠️ A `swarm.matrix.machine` read parses fine and fails at module-system
|
||
# resolution, so this is the kind of mistake only reading the target module
|
||
# catches.
|
||
matrixMachine = "hive-matrix";
|
||
|
||
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
|
||
in
|
||
{
|
||
config = lib.mkMerge [
|
||
# ⚠️ A SEPARATE arm from the unit below, and that separation is the whole
|
||
# mechanism: the unit's arm is gated on `haveClientIdentity`, so an
|
||
# assertion written inside it could never be reached in the state it
|
||
# exists to report.
|
||
#
|
||
# Shaped after ./swarm-grafana.nix's `haveClientIdentity` assertion — the
|
||
# same refusal, named to this principal's own pair. What differs is the
|
||
# gate. Grafana asserts wherever Grafana runs, because a Grafana with no
|
||
# store identity has no way in at all; a homeserver with no store identity
|
||
# is a hive that has no store, which is supported. So this one additionally
|
||
# requires `hiveReaderIdentity`: the host demonstrably reads the store, and
|
||
# named every option but this pair.
|
||
(lib.mkIf (deployCfg.matrix.enable && hiveReaderIdentity) {
|
||
assertions = [
|
||
{
|
||
assertion = haveClientIdentity;
|
||
message = ''
|
||
This host reads the swarm secret store (services.hyperhive.deploy.bao.clientCertFile
|
||
is set) and runs a homeserver, so it needs the matrix appservice
|
||
token reader's own client identity: set both
|
||
|
||
services.hyperhive.deploy.bao.matrixTokenClientCertFile
|
||
services.hyperhive.deploy.bao.matrixTokenClientKeyFile
|
||
|
||
swarm-bao-matrix-token.service fetches this hive's appservice token
|
||
out of the store, and without these it is not rendered at all —
|
||
leaving the homeserver authenticating hive-c0re against whatever is
|
||
already on disk, which is a 401 on every request naming nothing.
|
||
|
||
⚠️ This reader's OWN leaf, not deploy.bao.clientCertFile. That one is
|
||
the hive's, and its grant reads every secret in the store; this role
|
||
reads the one appservice-token path. Pointing this option at the
|
||
hive's leaf would evaluate, deploy and log in — and undo the split.
|
||
|
||
On a hive that runs the store, glue-bao-tls.nix supplies both as
|
||
defaults and there is nothing to do. Elsewhere the leaf is issued
|
||
from that CA out of band and named here — see docs/swarm/secrets.md.
|
||
'';
|
||
}
|
||
];
|
||
})
|
||
|
||
(lib.mkIf (haveClientIdentity && deployCfg.matrix.enable) {
|
||
# Same rule as the unit's own gate: this reader exists on a host that has a
|
||
# client identity and a homeserver, which is not every host that runs the
|
||
# store, so the store's module cannot name it.
|
||
services.hyperhive.swarm.otel.journaldUnits = [ "swarm-bao-matrix-token" ];
|
||
|
||
systemd.services.swarm-bao-matrix-token = {
|
||
description = "fetch the matrix appservice token from the swarm secret store";
|
||
# Every one of these names a unit that exists only where the store runs.
|
||
# `Requires=` on an absent unit fails the job outright, so the ordering is
|
||
# conditional even though the read is not: off-host there is nothing local
|
||
# to wait for, and the timeout below is what bounds the attempt instead.
|
||
after = lib.optionals baoDeploy.enable [
|
||
"swarm-bao-pki.service"
|
||
"container@${baoCfg.machine}.service"
|
||
];
|
||
wants = lib.optionals baoDeploy.enable [ "container@${baoCfg.machine}.service" ];
|
||
requires = lib.optionals baoDeploy.enable [ "swarm-bao-pki.service" ];
|
||
before = [ "container@${matrixMachine}.service" ];
|
||
wantedBy = [ "container@${matrixMachine}.service" ];
|
||
path = [
|
||
deployCfg.bao.package
|
||
pkgs.coreutils
|
||
];
|
||
# Sized for the race this loses, not for an unseal. `swarm-bao` comes up
|
||
# seconds before this unit asks, and the cert-auth role it logs in
|
||
# against is written seconds after — so a few short attempts cover it.
|
||
# ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy.
|
||
# That unit blocks nothing; this one is `Before=` the homeserver's
|
||
# container, and whether that ordering waits across an auto-restart is
|
||
# unverified — so an hours-long window would be a bet on an unknown,
|
||
# where a minute is not. A store still sealed after it keeps the degrade
|
||
# below, as today.
|
||
#
|
||
# `StartLimit*` are `[Unit]` settings, so they go here and not in
|
||
# `serviceConfig` — systemd ignores them under `[Service]`. The window
|
||
# has to exceed `RestartSec × burst`.
|
||
startLimitBurst = 4;
|
||
startLimitIntervalSec = 300;
|
||
serviceConfig = {
|
||
Type = "oneshot";
|
||
RemainAfterExit = true;
|
||
# What actually bounds the read below. Stated here rather than
|
||
# left to systemd's default, so the number a boot waits on is in
|
||
# the file that waits.
|
||
TimeoutStartSec = 30;
|
||
Restart = "on-failure";
|
||
RestartSec = 15;
|
||
};
|
||
environment = {
|
||
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
|
||
BAO_CLIENT_CERT = baoDeploy.matrixTokenClientCertFile;
|
||
BAO_CLIENT_KEY = baoDeploy.matrixTokenClientKeyFile;
|
||
}
|
||
# Absent means the system trust store, which is what a deployment with a
|
||
# real CA wants and what a self-signed one must not be left with.
|
||
// lib.optionalAttrs (baoDeploy.serverCaFile != null) {
|
||
BAO_CACERT = baoDeploy.serverCaFile;
|
||
};
|
||
script = ''
|
||
set -euo pipefail
|
||
|
||
${atomicWriteSecret}
|
||
|
||
# A sealed or uninitialised store answers on the port and never
|
||
# answers the read, so "the store is up" is not the same as "the
|
||
# store can answer". `TimeoutStartSec` above is the bound; the
|
||
# homeserver only `Wants=` this unit, so hitting it degrades to
|
||
# keeping the local token rather than holding up the container.
|
||
# `bao`'s own message is the only thing separating a missing value
|
||
# from a refused identity from an unreachable host. This unit's
|
||
# degraded mode is correct for all three, so it reports which one
|
||
# rather than asserting all three in a sentence of ours — a reader
|
||
# that cannot say why it read nothing is indistinguishable from a
|
||
# broken one.
|
||
err="$(mktemp)"
|
||
trap 'rm -f "$err"' EXIT
|
||
|
||
# Cert auth is a login, not a transport setting. The `BAO_CLIENT_*`
|
||
# variables above only decide which certificate the TLS handshake
|
||
# presents; without a token `bao` asks its token helper instead, and
|
||
# that is a `sh` this unit's `path` does not carry. `-token-only`
|
||
# answers on stdout and skips the helper on both sides.
|
||
#
|
||
# Fails LOUDLY, unlike the read below: the three states a login failure
|
||
# covers — store not up, sealed, role not written yet — are all things
|
||
# a retry fixes, and `Restart=on-failure` above is what retries. Exiting
|
||
# 0 here spends the whole boot on a condition that was seconds old.
|
||
if ! BAO_TOKEN="$(bao login -method=cert -token-only 2>"$err")"; then
|
||
echo "could not log in to swarm-bao with this host's certificate; keeping the token hive-matrix already has." >&2
|
||
if [ -s "$err" ]; then
|
||
cat "$err" >&2
|
||
else
|
||
echo "bao failed without writing a diagnostic." >&2
|
||
fi
|
||
exit 1
|
||
fi
|
||
export BAO_TOKEN
|
||
|
||
if ! token="$(bao kv get -field=value ${lib.escapeShellArg tokenPath} 2>"$err")"; then
|
||
echo "swarm-bao did not return ${tokenPath}; keeping the token hive-matrix already has." >&2
|
||
if [ -s "$err" ]; then
|
||
cat "$err" >&2
|
||
else
|
||
echo "bao failed without writing a diagnostic." >&2
|
||
fi
|
||
exit 0
|
||
fi
|
||
|
||
if [ -z "$token" ]; then
|
||
echo "swarm-bao returned an empty ${tokenPath}; keeping the local token." >&2
|
||
exit 0
|
||
fi
|
||
|
||
atomic_write_secret 0600 "" ${lib.escapeShellArg (toString deployCfg.matrix.appserviceTokenFile)} "$token"
|
||
|
||
# Re-stamp the registration file from the token just written. The
|
||
# token is half an agreement — the registration the homeserver loads
|
||
# has to carry the same value — so writing the file and stopping
|
||
# would leave the homeserver authenticating hive-c0re against
|
||
# whatever activation put there: a 401 on every request, naming
|
||
# nothing. Unconditional rather than on-change, because this unit
|
||
# has no way to know what the registration currently says.
|
||
#
|
||
# hive-matrix's own renderer rather than a `printf` here, so the
|
||
# registration's shape has one home.
|
||
${deployCfg.matrix.appserviceRegistrationScript}
|
||
'';
|
||
};
|
||
})
|
||
];
|
||
}
|