The unit restricted RestrictAddressFamilies to AF_UNIX, which was correct
while the daemon only served its unix socket. It has since grown three
outbound clients -- authelia token minting and forge calls over HTTPS, and
the queue over NATS -- and every socket(AF_INET, ...) was refused by seccomp.
systemd surfaces that refusal as EAFNOSUPPORT, "Address family not supported
by protocol", so the failure names the protocol and never the sandbox. The
visible symptom was swarm agent creation failing while minting a bearer
token, with a connect error that reads like a network fault.
Permit AF_INET/AF_INET6 for those clients and AF_NETLINK, which glibc's
getaddrinfo needs to enumerate local addresses before returning one. The
rest of the unit's hardening is unchanged.
`swarm-controller.service` carried `SetCredential=queue-client.secret:` — an
empty value, which systemd's parser refuses:
/etc/systemd/system/swarm-controller.service:38:
Invalid syntax, ignoring: queue-client.secret:
So the line was dropped on every daemon-reload, and `LoadCredential=` was
fatal again — precisely the failure that default was added to prevent. On a
hive where authelia has not yet minted the secret, the controller refuses to
start rather than coming up with the queue unconfigured.
It looked correct for days because the credential file happened to exist, so
the fail-soft was never exercised.
Measured with `systemd-analyze verify`: an empty value is rejected, any
non-empty one is accepted. The placeholder is a real word rather than filler —
it reaches the token request as the client secret, so authelia refuses it and
the journal names something an operator can act on.
Gate: state/eval-setcredential.sh, with the parent commit as its mutation.
A systemd credential named by an absolute path is fatal when the file is
missing, and the co-located queue secret is minted by authelia's first
boot in another container — which a host unit cannot order against. On a
real boot the daemon spent three of systemd's five default starts losing
that race before the file appeared; two seconds more would have reached
start-limit-hit, which does not self-heal.
An empty SetCredential acts as the default that makes the load
non-fatal, so the controller starts and serves its HTTP surface with the
queue unconfigured — a shape it already reports.
That alone would be a regression, though: a credential is snapshotted at
unit start, so the placeholder would freeze and the daemon would sit
degraded forever instead of recovering the way the restart loop
accidentally did. A path unit on the secret file closes it, and closes a
second gap in the same stroke — mint_token reads the secret on every
call precisely so a rotation takes effect without a restart, and a
snapshot in %d silently defeats that.
PathChanged and not PathExists: the latter activates immediately when
the file is already present at unit start, which would restart a healthy
daemon on every boot.
The option being consumed already warned about this shape — its own
description says a consumer has to wait for the secret because a
missing source turns a fresh hive into a boot-order deadlock. That
warning was written on the producing side and did not fire while I was
writing the consumer.
Review point: the co-located defaults are an auto-deploy special case and
belong with the other ones, not inside each option's own default. An
option should describe itself; the mode describes what a deployment shape
implies. local-defaults.nix already says exactly this in its header.
Options now default to empty (= unset, which the assertions refuse), and
enableAllLocalDefaults fills in loopback + the minted secret path.
The controller's queue coordinates were gated on `autheliaCfg.enable &&
natsCfg.enable` -- i.e. on this host running both. Split the hosts and
the whole env block vanished, which the daemon's own rule reads as "no
queue configured": it started cleanly, served /api/hives, and silently
never connected or published.
Make the coordinates options instead, defaulted to the co-located values
so nobody types a path they didn't need, and assert each one so a split
deployment fails at eval naming the option rather than running blind.
The queue is required for a controller -- /api/hives/status reads the KV
-- but co-location with it is not.
Follow-up to the plumbing removal: docs/swarm/README.md gets the
biggest rewrite (drops the whole "Fingerprint format" section, fixes
the runtime-effects list, the WireGuard config example + "what the
mesh does" bullet), docs/gateway.md and hive-gateway/options.nix drop
4 "needs no certFingerprint" mentions, swarm-peers-removed.nix's
migration-warning text no longer tells an upgrading operator to carry
a field over that no longer exists, swarm.nix/swarm-wireguard.nix/
swarm-controller.nix/swarm-controller's main.rs get comment fixes
where they described the now-removed HYPERHIVE_PEERS shape. Also
caught one more stale "peer hives" mention in docs/web-ui/README.md's
SW4RM tab description that the first pass on this issue missed.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
New `services.hyperhive.swarm.controller.links` option (listOf {label,
icon, url}, same shape as the per-agent hyperhive.dashboardLinks) plus
a new GET /api/links route serving it, same pattern as the existing
hives/GET /api/hives.
Rather than one central hardcoded list, each service's own module
contributes its own entry when actually enabled on the controller's
host: swarm-authelia.nix, hive-matrix.nix (gated on gui.enable too,
since / on that vhost only serves fluffychat then) and
hive-forge/default.nix (gated on behindGateway) each push one entry,
the same list-merge idiom services.hyperhive.gateway.localNames
already uses. swarm-ui.nix contributes a static entry for its own
same-origin swagger docs. Adding a future service's link is a nix-only
change to that service's own module.
Verified: cargo build/clippy/test -p swarm-controller clean, a
throwaway nixosSystem eval confirms all 4 entries merge correctly into
SWARM_CONTROLLER_LINKS, nix build .#swarm-controller succeeds.
Fixes hyperhive#3223.
swarm-controller: GET /api/hives (utoipa-annotated same as /health),
serving the swarm's hive directory (name + domain) loaded once at
startup from a new SWARM_CONTROLLER_HIVES env var. The controller's
NixOS module sets it from services.hyperhive.swarm.hives, JSON-encoded
the same way hive-c0re already builds HYPERHIVE_PEERS for its own peer
list (environment.nix) — the full directory here rather than
peers-minus-self, since a swarm-level daemon has no 'self' hive to
exclude. Unset/malformed both fall back to an empty list with a
warning rather than failing startup, so /health stays answerable even
if this one env var is wrong.
swarm-ui: App.tsx's Home route fetches /api/hives and renders it
through the already-merged <Table>/<StatusChip>/<Panel> primitives —
name, domain (linking out to that hive's own gateway-routed
dashboard), and a static "configured" status chip until a real
online/stale/offline rollup exists server-side. Also gave swarm-ui a
base <a> color (theme's --blue) — base.css covers body/typography but
not links, and this is genuinely page-level rather than any one
component's concern.
Verified end to end, not just source-reading: ran the real
swarm-controller binary with SWARM_CONTROLLER_HIVES set, curled
/api/hives + /health over its actual unix socket; separately served
the real swarm-ui dist against a mock /api/hives and screenshotted the
rendered table. Also re-verified the nginx wiring evaluates (same
throwaway nixosSystem eval technique as #3212) — SWARM_CONTROLLER_HIVES
resolves to the expected JSON shape.
cargo test/clippy -p swarm-controller clean (2 tests, including a new
load_hives one covering missing/malformed/valid env var states). npm
run build + typecheck clean.
Prose-only sweep of every remaining claim that nginx or dnsmasq lives
in a container: the port comment (root in a container -> root on the
host), upstreamHost's netns rationale, the ACME state dir, the store
path reachability note, the vhost tree header, dnsmasq's resolv.conf
paragraph (there is no copy and no path unit watching it any more),
the two hive-network bridge comments, and swarm-controller's socket
access-control note, which described a bind-mount that no longer
exists.
No behaviour change; all of it was describing a mechanism that was
deleted.
`enableAllLocalDefaults` already asserts the swarm's shared services and
its CA; the controller was the one swarm-level thing it left off, so the
default deployment ran authelia, matrix and the forge with nothing
controlling them — and, until the previous commit in this area, without
`swarmctl` either.
The controller's own option stays `default = false`. Running it is a
statement about swarm topology rather than about hyperhive being
installed, and no single host can infer that on its own. But "this box
is the whole deployment" IS that statement, which is why the mode may
assert what `services.hyperhive.enable` never could.
Derived from the mode, not from `enableRequiredServices`: a hive in a
larger swarm can legitimately want the shared services without being the
host that controls them.
`mkDefault`, so `enableAllLocalDefaults = true` with an explicit
`controller.enable = false` still yields a controller-less box — the
mode fills in for an operator who hasn't spoken and never argues with
one who has.
The swarm-authelia module states that its users database is written by
swarm-controller, but nothing ever granted the means. This adds the tool
that does it.
swarmctl runs as root on the controller's host and acts directly. The
rootless alternative was examined and does not work: relocating the users
file into a directory the controller owns only turns a write problem into
a read problem, because authelia must then reach across the same boundary
in the other direction. Making that read work needs either a hand-pinned
gid or world-readable password hashes.
The user store is two files, one authoritative: users.json is canonical,
users.yml is a rendered artifact. That split is what lets the crate work
without a YAML parser -- the workspace has none, and adding one costs a
crates.io fetch, a lock update and a vendor hash for a schema we fully
control and only ever emit.
Passwords are generated by authelia rather than passed to it: argv is
world-readable, so a password on a command line is readable by any local
process for the lifetime of the call.
The three derived facts swarmctl needs about the authelia container --
machine, unit and the host-side users path -- become readOnly options on
the authelia module rather than literals repeated at the call site.
services.hyperhive.swarm.controller.{enable,package,socketPath} plus the
unprivileged swarm-controller user, its runtime and state directories,
and the unit itself.
enable is deliberately not derived from services.hyperhive.enable, unlike
c0re: a swarm has one controller, so turning it on is a statement about
swarm topology rather than about whether hyperhive is installed.
The socket gets its own RuntimeDirectory. nginx reaches a unix upstream
by having the socket's directory bind-mounted into the gateway
container, and the socket is 0666 because connect needs write -- so the
directory is the only access control there is. Sharing one with the host
admin socket would hand that socket to the gateway too. The constraint
is stated at both ends, in the option description and beside the bind,
because it is invisible from either site alone; a test pins the path so
a tidying edit fails rather than reviews cleanly.
RuntimeDirectoryPreserve and the daemon's stale-socket unlink are a
pair: preserving the directory without the unlink means bind fails with
EADDRINUSE after a restart.
