vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(nix): the matrix container gets the swarm-internal trust anchor

Browse source 
`security.pki.certificateFiles` is read when the system is BUILT and the
swarm root is deliberately a runtime file (its key must never enter the
world-readable store), so there was nothing build-time to name and the
container trusted no swarm-internal CA. Federation with a peer whose
cert chains to the swarm root did not validate.

The bridge already existed: `lib/hive-ca-trust.nix`, which `hive-ci` and
`hive-forge` both cross. This wires the matrix container to it -- read-only
bind of the trust bundle, container ordered after `hive-tls-ca.service` --
rather than inventing a second mechanism.

Consumption is the per-runtime half, and it is not what it looks like.
tuwunel links no openssl, which makes `SSL_CERT_FILE` appear inapplicable.
It isn't: the outbound client is `reqwest` with the `rustls` feature,
which builds a `rustls_platform_verifier::Verifier`, and because tuwunel
calls `tls_certs_merge` (additive) rather than `tls_certs_only`, the
platform roots stay alongside its compiled-in webpki set. On Linux that
verifier resolves through `rustls-native-certs` to `openssl-probe`, which
reads `SSL_CERT_FILE`. Established by reading tuwunel 1.8.0 and reqwest
0.13.4 source, not inferred from the feature list -- an earlier reading of
mine concluded the opposite from the absence of a `native-roots` feature
name and was wrong.

The variable therefore names a CONCATENATION (system CAs + the bundle),
never the anchor alone: `openssl-probe` uses it *instead of* the default
store, so pointing it at the bundle would drop every public CA and break
federation with the wider matrix network -- trading a small outage for a
much larger one.

`container@hive-matrix` needed `mkMerge`: it already assigned `after` for
the gateway ordering and the helper contributes its own `after`/`requires`,
so two bare assignments would have conflicted rather than combined.

⚠️ Gate honesty: `state/eval-3093.sh` proves the mount, the ordering, the
bundle service and the env var are RENDERED -- 6 probes, including two
separate absence probes (mount and consumption are wired at different
sites, and the gateway outage happened precisely because two of three
sites tolerated a missing cert and the third did not). It CANNOT prove
tuwunel loads them: a build that ignores the variable evaluates
identically to one that honours it, and the chain above is
version-specific. That half wants an executed check on a real deploy.
This commit is contained in:
atlas 2026-08-09 19:38:01 +02:00 • committed by mara
commit 038d085001
2 changed files with 89 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

25
docs/swarm/ca.md
View file

@ -148,11 +148,26 @@ Two consumers, and only one of them is fine:
`HIVE_TLS_CA_PATH`, and the meta-flake renderer embeds that one file
next to every agent's flake. The bundle is the runtime-to-build-time
bridge.
- **The Matrix container is not.** It has no equivalent bridge, so it
trusts no swarm-internal CA and federation with a self-signed peer
does not validate. Giving it the root needs a runtime mechanism —
bind-mount plus a bundle assembled at unit start, appending to the
system bundle rather than replacing it — which is tracked separately.
- **The Matrix container crosses the same bridge**, via the shared
`lib/hive-ca-trust.nix` helper that `hive-ci` and `hive-forge` already
use: the bundle is bind-mounted read-only into the container, and a
oneshot concatenates it with the system CAs before tuwunel starts.
The consumption differs per runtime and is the part worth knowing.
tuwunel links no openssl, which makes `SSL_CERT_FILE` look inapplicable
— it isn't. Its outbound client is `reqwest` with the `rustls` feature,
which builds a `rustls_platform_verifier::Verifier`; because tuwunel
calls `tls_certs_merge` (additive) rather than `tls_certs_only`, the
platform roots stay alongside its compiled-in webpki set. On Linux that
verifier resolves through `rustls-native-certs` → `openssl-probe`,
which reads `SSL_CERT_FILE`.
> ⚠️ **Concatenate; never point `SSL_CERT_FILE` at the anchor alone.**
> `openssl-probe` uses it *instead of* the default store, so naming
> just the bundle would drop every public CA and break federation with
> the wider matrix network — a much bigger outage than the one being
> fixed. The same caveat applies to `hive-forge` (Go) for the same
> reason.
To put the root on another host, copy the certificate to the same path
there (`scp <stateDir>/root.pem <host>:<stateDir>/root.pem`). One anchor

89
nix/host-modules/hive-matrix.nix
View file

@ -7,6 +7,19 @@
let
cfg = config.services.hyperhive.swarm.matrix;
networkCfg = config.services.hyperhive.network;
tlsCfg = config.services.hyperhive.tls;
gatewayCfg = config.services.hyperhive.gateway;
# Same runtime→build-time bridge hive-ci and hive-forge already cross:
# binds the hive trust bundle (which folds in the swarm root) into the
# container and orders the container after `hive-tls-ca.service`. The
# *consumption* is per-runtime and stays here — see the bundle service
# in the container config below.
caTrust = import ./lib/hive-ca-trust.nix { inherit lib tlsCfg gatewayCfg; };
useSelfSigned = caTrust.useSelfSigned;
# tuwunel's own combined bundle, assembled at start. /run is tmpfs, so
# it is rebuilt from the current CA every boot rather than going stale.
matrixCaBundle = "/run/hive-matrix-ca/ca-bundle.crt";
swarmDomain = config.services.hyperhive.swarm.domain;
# Falls back to the SWARM domain: a swarm runs one homeserver, so its
# identifier belongs to the swarm rather than to whichever hive happens
@ -426,10 +439,13 @@ in
privateNetwork = false;
# Read-only bind of the host-managed registration token; tuwunel
# reads it via systemd LoadCredential below (not directly).
bindMounts.${cfg.registrationTokenFile} = {
hostPath = cfg.registrationTokenFile;
isReadOnly = true;
};
bindMounts = {
${cfg.registrationTokenFile} = {
hostPath = cfg.registrationTokenFile;
isReadOnly = true;
};
}
// caTrust.bindMount;
config =
{ ... }:
{
@ -442,20 +458,12 @@ in
# all filtering; never run one in here.
networking.firewall.enable = false;
# ⚠️ This container trusts no swarm-internal CA. It used to take
# per-hive root CAs, hand-pinned as nix paths, so tuwunel could
# validate *federation* TLS from a self-signed peer hive; that
# field is gone, and the swarm root that replaces it cannot be
# substituted here. `security.pki.certificateFiles` is read when
# the system is BUILT, and the swarm root is a runtime file
# (`swarm.ca.stateDir`) precisely because its key must never
# reach the store — so there is nothing build-time to name.
#
# Giving the container the swarm root therefore needs a runtime
# mechanism (bind-mount + a bundle assembled at start), which is
# a different shape than this line and is tracked as its own
# issue. Federation with a peer whose cert chains to the swarm
# root does not validate until then.
# Swarm-internal trust reaches tuwunel at RUNTIME, not via
# `security.pki.certificateFiles`. That option is read when the
# system is BUILT, and the swarm root is deliberately a runtime
# file (`swarm.ca.stateDir`) because its key must never enter the
# store — so there is nothing build-time to name. The bind-mount
# above plus the bundle service below are what replaced it.
# tuwunel hard-fails to boot if `/etc/resolv.conf` has no
# `nameserver` line (`Failed to configure DNS resolver ... no
@ -527,6 +535,42 @@ in
systemd.services.tuwunel.serviceConfig.LoadCredential = [
"registration_token:${toString cfg.registrationTokenFile}"
];
# Federation TLS against a peer whose cert chains to the swarm
# root: tuwunel's outbound client is reqwest with the `rustls`
# feature, which builds a `rustls_platform_verifier::Verifier`
# and — because tuwunel calls `tls_certs_merge` rather than
# `tls_certs_only` — keeps the platform roots alongside its
# compiled-in webpki set. On Linux that verifier resolves through
# `rustls-native-certs` → `openssl-probe`, which reads
# `SSL_CERT_FILE`. So the openssl-shaped variable IS the lever
# here, despite tuwunel linking no openssl.
#
# ⚠️ CONCATENATE, never point at the anchor alone. `openssl-probe`
# uses `SSL_CERT_FILE` *instead of* the default location, so
# naming just the hive bundle would drop every public CA and
# break federation with the wider matrix network — trading a
# small outage for a much larger one.
systemd.services.hive-matrix-ca-bundle = lib.mkIf useSelfSigned {
description = "assemble tuwunel TLS trust bundle (system CAs + hive CA)";
wantedBy = [ "tuwunel.service" ];
before = [ "tuwunel.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SyslogIdentifier = "hive-matrix-ca-bundle";
};
path = [ pkgs.coreutils ];
script = ''
set -euo pipefail
install -d -m 0755 /run/hive-matrix-ca
cat /etc/ssl/certs/ca-certificates.crt ${caTrust.caContainerPath} \
> ${matrixCaBundle}
chmod 0644 ${matrixCaBundle}
'';
};
systemd.services.tuwunel.environment.SSL_CERT_FILE = lib.mkIf useSelfSigned matrixCaBundle;
environment.systemPackages = [ cfg.package ];
};
};
@ -550,8 +594,13 @@ in
# gateway always runs alongside hyperhive, so the gateway container
# unit always exists here. (Declarative `containers.<n>` →
# `container@<n>.service` — the nspawn template NixOS generates.)
systemd.services."container@hive-matrix".after = [
"container@hive-gateway.service"
# `mkMerge`, not a bare assignment: `caTrust.containerOrdering` also
# sets `after`/`requires` (so the bound trust bundle exists before
# nspawn wires the mount up), and two plain assignments to the same
# unit would conflict rather than combine.
systemd.services."container@hive-matrix" = lib.mkMerge [
{ after = [ "container@hive-gateway.service" ]; }
caTrust.containerOrdering
];
};
}