`serverName` is baked irrevocably into every user and room id, so a hive
that rebuilds onto a new default is a *different homeserver*, not a
renamed one: existing accounts and rooms are stranded, and reverting the
config does not undo it. Its neighbours (`gatewayHost`, the forge domain)
are routing, rediscovered through `.well-known` and fixed by editing them
back. Same diff shape, three orders of magnitude apart in blast radius --
which is an asymmetry a module should carry rather than an operator.
An activation script and not `warnings`, which is where this obviously
belongs and does not work: the condition needs the host filesystem, and
`nixos-rebuild switch --flake` evaluates purely, where
`builtins.pathExists "/var/lib/..."` answers false rather than throwing.
A `warnings` entry gated on it would evaluate, deploy, and print nothing
on every real deployment.
Rendered only when `serverName` is null, so a pinned hive has no script
rather than a script that stays quiet -- a guard that cries wolf at a
correctly-configured deployment makes the next real one read as noise.
Never fails the activation: it warns about a choice that cannot be
undone, and refusing the rebuild of a hive that already chose
deliberately is the opposite of helping.
The probed path is read out of the container's own evaluated config
rather than hardcoded. A guessed path resolves cleanly and silently never
matches, which is the same failure this guard exists to catch one level
up.
Both hand-rolled the same concat with wantedBy + before and no requires,
so a failed assembly let the consumer start against a missing file and
trust nothing at all -- which fails every outbound TLS call while the unit
looks healthy. The helper puts requires on the consumer and verifies the
assembled bundle before moving it into place.
Removes each module's now-dead useSelfSigned binding: its only use was the
mkIf on the deleted block, and nix does not warn about an unused let
binding. Also corrects two comments the deletion orphaned -- one described
the removed path binding, the other pointed at a bundle service that no
longer exists.
Three host units poll up to 120s for a secret authelia mints on its first
boot, and all three are `Type=oneshot` with no `TimeoutStartSec`. systemd's
`DefaultTimeoutStartSec` is 90s, so it kills them at 90 — before the script
reaches its own `exit 1` and names the file that never appeared.
The wait itself is fine; what's lost is the diagnosis. On a fresh hive the
operator gets a bare start-timeout instead of "authelia has not minted
<path>", several layers from the container that was actually slow.
Found while writing the same unit for Grafana, where the timeout is set —
so this is the existing three catching up with it, not a new pattern.
An SSO login whose derived localpart already exists silently gets a
random one instead: the candidate must be *available*, an existing
unlinked account is not, and `unique_id_fallbacks` defaults to true.
The operator's own login minted a stray account this way.
Fill the three identity_provider fields that decide which account a
login lands on: `trusted` (upstream's precondition -- self-hosted and
fully controlled -- is structural here, the issuer is always the swarm's
own authelia), `userid_claims` pinned to preferred_username so the
default ladder cannot fall through to an email local part, and
`unique_id_fallbacks = false` so a collision errors instead of
succeeding as the wrong user.
tuwunel authenticates at the token endpoint with the secret in the POST
body. Authelia enforces the method a client is REGISTERED with rather
than accepting whichever one arrives, and its default is
client_secret_basic — so the matrix login completed, consent was
granted, and the very last hop failed:
Client authentication failed ... The request was determined to be
using token_endpoint_auth_method client_secret_post, however the
OAuth 2.0 client registration does not allow this method.
The failure names neither the secret nor the redirect, and it lands
three layers from its cause, which is why it read as a credential
problem.
Adds a per-client tokenEndpointAuthMethod, null by default so every
existing client keeps authelia default (forgejo authenticates with
basic and is unaffected), and sets client_secret_post on the matrix
client only.
New `services.hyperhive.swarm.controller.links` option (listOf {label,
icon, url}, same shape as the per-agent hyperhive.dashboardLinks) plus
a new GET /api/links route serving it, same pattern as the existing
hives/GET /api/hives.
Rather than one central hardcoded list, each service's own module
contributes its own entry when actually enabled on the controller's
host: swarm-authelia.nix, hive-matrix.nix (gated on gui.enable too,
since / on that vhost only serves fluffychat then) and
hive-forge/default.nix (gated on behindGateway) each push one entry,
the same list-merge idiom services.hyperhive.gateway.localNames
already uses. swarm-ui.nix contributes a static entry for its own
same-origin swagger docs. Adding a future service's link is a nix-only
change to that service's own module.
Verified: cargo build/clippy/test -p swarm-controller clean, a
throwaway nixosSystem eval confirms all 4 entries merge correctly into
SWARM_CONTROLLER_LINKS, nix build .#swarm-controller succeeds.
Runs on the host: the two containers share a network namespace but not a
filesystem root, so this is the only place both trees are addressable.
A copy rather than a bindMounts entry. nixos-container refuses to start
when a bind source is missing, and the secret does not exist until
authelia's first boot has minted it. The registration token dodges that
with an activation script that pre-creates the file; that is unavailable
here, because tuwunel requires the secret to exist and be non-empty, so a
placeholder would satisfy the mount and then stop the homeserver.
Bounded wait then fail, never a silent skip: authelia's container can be
up while its generator is still minting.
Stage 1's actual login path. The provider entry is rendered only when
sso.enable is set, via optionalAttrs rather than an empty list, so a hive
that has not opted in renders byte-identical tuwunel settings.
The secret reaches tuwunel through LoadCredential rather than as a direct
path: upstream requires the file to survive systemd sandboxing and
suggests /etc/tuwunel/, which this container has no writable etc for. The
registration token two units below already solves it the same way, and
for the same reason -- DynamicUser and PrivateUsers stay intact.
Three assertions fail at eval instead of at boot. tuwunel reads identity
providers from its config file, so a half-configured one does not hide a
login button, it can stop the homeserver from starting.
Stage 1 of the staged rollout mara laid out on the issue: the options
and nothing that changes a running hive. `enable` defaults false, so
this commit is inert until someone opts in.
Three rules taken from hive-forge/default.nix rather than re-derived,
since it is the same pattern already merged and reviewed:
- `enable` ADDS a login path and leaves password login alone. An
identity provider that can take the homeserver offline when it
hiccups is worse than two ways in. Making authelia the only path is
tuwunel's `login_with_password`, a separate and reversible switch,
deliberately not folded in here.
- the client secret is a PATH, never a value: it has two holders in two
containers (authelia keeps a hash, the homeserver needs plaintext),
and a literal would be rendered into the world-readable store.
- required when enabled, no fallback. Worse here than for the forge:
tuwunel reads OIDC from its config file rather than a database row,
so a malformed block can stop the server outright instead of merely
hiding a button.
Also records the constraint that shapes the whole issue — matrix SSO
lives inside the homeserver, never behind a forward-auth proxy,
because the client-server API is spoken by non-browser clients holding
matrix access tokens plus federation.
Moves the matrix sub-domain vhost out of the gateway's vhosts.nix, its
`address=` rule out of dnsmasq.nix, and the Accept-header
`$matrix_spa_target` map out of the gateway's appendHttpConfig — all
three into hive-matrix.nix.
The map is the one that had no business being where it was: it exists
solely for the SPA fallback in the vhost's `/` location, and
`appendHttpConfig` is a `lines` option, so a module can contribute to
it without the gateway assembling it.
The `.well-known/matrix/*` delegation deliberately stays on the hive's
own vhost. The spec requires it at the SERVER NAME, which is the hive
domain: that is the hive answering "where is my homeserver", not the
homeserver answering for itself. Moving it would have been the obvious
symmetric thing and it would have been wrong.
Per mara on the PR: how a thing used to be deployed stops being
relevant the moment this is merged and the hives are rebuilt, so the
comments state the current constraint and nothing about the container
that used to hold it.
Removes the two ⚠️ blocks this PR added (hive-tls-ca's ordering, the
resign propagation), the matrix ordering's, dnsmasq's resolv.conf
paragraph, and the pre-existing 'REMOVED WITH THE CONTAINER' inventory
in the gateway module. The 🚨 do-not-simplify warning on the cert-copy
unit keeps both its reasons and loses the container framing.
Kept deliberately: the two operator-facing 'this used to default to
true' option notes (migration information for someone upgrading), and
the SupplementaryGroups block, which documents why an absence is
load-bearing rather than how a deployment used to look.
Removing the gateway container retired two things that were still
depended on: an ordering edge and a delivery path.
- hive-tls-ca ordered itself before/requiredBy
container@hive-gateway.service. That unit no longer exists, so the CA
was sequenced against nothing while its real consumer,
hive-gateway-self-signed-cert (which nginx Requires=), could win the
race and fail its copy under set -eu, blocking nginx.
- hive-tls-resign propagated a rotated leaf with
`systemctl -M hive-gateway … || true`. The machine is gone, so both
calls failed and both failures were swallowed: the unit logged
"propagating" and exited 0 while nginx served the stale copy until it
expired. Host units now, so no -M and no || true — a failed
propagation fails the timer.
- container@hive-matrix ordered after the gateway container to get the
resolver up first. dnsmasq is a host service now, so it orders after
dnsmasq.service, which is what the edge always meant.
`security.pki.certificateFiles` is read when the system is BUILT and the
swarm root is deliberately a runtime file (its key must never enter the
world-readable store), so there was nothing build-time to name and the
container trusted no swarm-internal CA. Federation with a peer whose
cert chains to the swarm root did not validate.
The bridge already existed: `lib/hive-ca-trust.nix`, which `hive-ci` and
`hive-forge` both cross. This wires the matrix container to it -- read-only
bind of the trust bundle, container ordered after `hive-tls-ca.service` --
rather than inventing a second mechanism.
Consumption is the per-runtime half, and it is not what it looks like.
tuwunel links no openssl, which makes `SSL_CERT_FILE` appear inapplicable.
It isn't: the outbound client is `reqwest` with the `rustls` feature,
which builds a `rustls_platform_verifier::Verifier`, and because tuwunel
calls `tls_certs_merge` (additive) rather than `tls_certs_only`, the
platform roots stay alongside its compiled-in webpki set. On Linux that
verifier resolves through `rustls-native-certs` to `openssl-probe`, which
reads `SSL_CERT_FILE`. Established by reading tuwunel 1.8.0 and reqwest
0.13.4 source, not inferred from the feature list -- an earlier reading of
mine concluded the opposite from the absence of a `native-roots` feature
name and was wrong.
The variable therefore names a CONCATENATION (system CAs + the bundle),
never the anchor alone: `openssl-probe` uses it *instead of* the default
store, so pointing it at the bundle would drop every public CA and break
federation with the wider matrix network -- trading a small outage for a
much larger one.
`container@hive-matrix` needed `mkMerge`: it already assigned `after` for
the gateway ordering and the helper contributes its own `after`/`requires`,
so two bare assignments would have conflicted rather than combined.
⚠️ Gate honesty: `state/eval-3093.sh` proves the mount, the ordering, the
bundle service and the env var are RENDERED -- 6 probes, including two
separate absence probes (mount and consumption are wired at different
sites, and the gateway outage happened precisely because two of three
sites tolerated a missing cert and the third did not). It CANNOT prove
tuwunel loads them: a build that ignores the variable evaluates
identically to one that honours it, and the chain above is
version-specific. That half wants an executed check on a real deploy.
A swarm runs one homeserver, so its identity belongs to the swarm and
not to whichever hive happens to host it -- otherwise relocating the
container between hives reads as a different homeserver rather than a
move. `gatewayHost` moved for that reason a commit ago; `serverName`
was left behind, which made the identity hive-shaped and the routing
swarm-shaped.
⚠️ This is the one value on the page that a rebuild cannot undo.
`gatewayHost` is a routing detail clients rediscover through
`.well-known`; `server_name` is embedded in every user and room id, so
adopting a new one strands the existing accounts and rooms rather than
renaming them -- their ids still name a homeserver that stops
answering. Moving the DEFAULT is safe; moving a RUNNING deployment is
not, and existing hives must pin the value they already minted ids
under before rebuilding.
The fallback stays total on a null swarm domain for the same reason as
its neighbours: the required-domain assertion in hive-network.nix is
what should fire, not a coercion error from an unrelated option
interpolating null.
The legacy-pin eval probe now covers `serverName` as well. It existed
to answer "what do I set so old deployments don't change", and was
proving that only for the two values that are cheap to change -- the
irreversible one was the one it did not assert.
Both halves change: the parent, because a swarm runs one homeserver and
every hive reaches it; and the label, because `chat` names the service
people use rather than the protocol it speaks.
⚠️ `serverName` is deliberately NOT touched, and the two are now
documented as the different things they are. `gatewayHost` is a routing
detail -- the API listener nginx proxies `/_matrix/*` to, which clients
rediscover through `.well-known`. `serverName` is the matrix identifier
baked into every user and room id: changing it is a different
homeserver, not a rename, so it still falls back to the bare hive
domain. A note at the fallback says so, since that binding is where a
future edit would most plausibly "fix" the inconsistency.
Old deployments pin `matrix.<hive domain>` -- exactly what the old
default rendered -- and dnsmasq already lists `gatewayHost` explicitly,
so a pinned or moved name stays routed either way.
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
mara: enableAllLocalDefaults is more of a deployment mode via settings
set, less a default setting.
That reframing is the change. A mode asserts values; an option declares
what it is when nobody asks. Written as default = <flag>, every service
option had to name a switch it has no relationship to, and the answer to
what does all-local turn on was spread across five files.
Two modules now hold the two tiers - local-defaults.nix for the mode and
what it asserts directly, swarm-required-services.nix for the
services-live-here switch and the per-service enables under it. Each
service module keeps its own declaration and loses only the line about
when a deployment wants it.
mkDefault is the right precedence rather than a compromise: it beats an
option default and yields to any explicit definition, so the mode fills
in for an operator who has not spoken and never argues with one who has.
Evaluated config is unchanged in both directions.
Matrix is a swarm-wide service — one homeserver, not one per hive — so
its `enable` defaults from `swarm.enableRequiredServices` rather than
being flipped on its own. That toggle is off by default, so matrix is
off by default exactly as before; what changes is that a host declaring
"the swarm's services live here" gets the homeserver with it.
The forge needs no equivalent: it is mandatory and deploys with
hyperhive itself, so it has no enable to derive.
The required-domain assertion in hive-network.nix could not be reached:
`forge.<domain>` and `matrix.<domain>` are evaluated while the
assertion list is, so an unset domain threw `cannot coerce null to a
string` naming one of those options instead of printing the message
that says which option to set.
Both defaults now fall back to a name under the reserved `.invalid`
TLD, which the assertion refuses to let out the door.
Second slice of the swarm-global service consolidation, same shape as
the forge move: the operator-facing host options become
services.hyperhive.swarm.matrix.*, and one mkRenamedOptionModule on the
namespace carries the whole subtree (nested gui.* included), so there is
no leaf list to forget an entry from.
The rename lives in hive-matrix.nix, the module that declares the
options, so each service's migration stays independent of its siblings.
The per-agent hyperhive.matrix.{enable,url} and hyperhive.matrixAccounts
are a different namespace -- a client pointer at the service, not the
service -- and deliberately do not move.
Nix half of the 4th layer mara found (47535: core cannot assume matrix
is on localhost). Rust half (matrix.rs MATRIX_HTTP) NOT done.
Parked here rather than left dirty: she has redirected me to jobq as
prio 1, and uncommitted files migrate across a checkout.
