New GET /api/swarm on swarm-controller, backed by
services.hyperhive.swarm.name (SWARM_CONTROLLER_NAME env var, same
optionalAttrs-gated-on-option-resolving shape queueEnv/forgeEnv/etc.
already use). swarm-ui's <Shell> fetches it once and sets both
document.title and the header's brand text; falls back to the
existing static "hyperhive swarm" label when the operator never set
a name or the fetch fails.
Extracted the swarm-queue connect block out of main() into its own
connect_status_reader() fn to keep main() under clippy's line-count
lint after adding the new field wiring — no behavior change, same
comments moved as-is.
The controller's forge client speaks TLS to `https://<forge domain>`,
which the gateway serves with a leaf signed by the hive CA. That CA is
generated at runtime, so nothing build-time can name it and it is not in
the system store -- and `reqwest`/`rustls` resolves roots through
`rustls-native-certs`, whose `SSL_CERT_FILE` *replaces* the store rather
than adding to it. With no bundle wired, every forge call failed
`invalid peer certificate: UnknownIssuer` and agent creation died at its
first step.
`lib/hive-ca-trust.nix` already solved this, but only for containers: it
sources the CA through `/run/hive-ca/trust-bundle.pem`, a bind mount that
does not exist on the host. `hostUnit` makes it read the host copy and
wait on `hive-tls-ca.service` itself -- one flag driving both, because a
host source without that ordering is a race.
`enable` is the other half, and it is the sharp edge: a container caller
imports this into the container's module set, so it disappears with the
container. A host caller imports it at the host's top level, where
`imports` is unconditional -- without the flag, a hive with the
controller turned off would get a bundle oneshot and a `swarm-controller`
service conjured by `genAttrs`, holding an `SSL_CERT_FILE` and no
`ExecStart`.
mara asked whether SWARM_CONTROLLER_PUBLIC_URL should also be set when the
controller does not run on the same host, and then said the two read as one
service to her. Both are worth answering in the file rather than only in the
thread: that split is not representable today, and the reason is two hops
away from this line.
The UI's /api/ location proxies http://unix:<socketPath> -- a path that
resolves nowhere else -- and this daemon binds no TCP address at all. So the
vhost and the daemon are co-located by construction, and ui.enable is the
flag that declares the vhost rather than a guess about some host serving it.
Also names the tripwire: the day the daemon grows a TCP listener, this line
silently stops being right (env unset, registration quietly skipped, no
error). That is when an explicit publicUrl option becomes correct -- not
before, while there is exactly one derivable answer.
The endpoint landed inert: nothing pointed at it, so the only way to see
it work was to mint an HMAC by hand. Register the two swarm-wide hooks
at startup so a real forge event produces a journal line.
Registered ALONGSIDE the per-hive hooks, not instead of them. Every hive
keeps receiving and acting on its own deliveries; the controller gets a
copy and logs it. Moving the registration is a later step and has to be:
fan-out swarm->hive does not exist yet, so a hook moved now would point
at a receiver that forwards nowhere, silently on both sides.
Deliberately no stale-hook deletion arm, unlike the two per-hive
registrars this otherwise mirrors: theirs delete hooks matching their own
path with a foreign base, and the hives' hooks are not stale.
The route prefix is what keeps this safe. Both hive-side registrars
delete any hook ending in /webhook/knowledge or /webhook/config-pr with a
different base, so a swarm hook under those paths would be deleted by
every hive on every boot. Serving them under /webhook/forge/ avoids it,
and a test pins it -- there is nothing else that can.
SWARM_CONTROLLER_PUBLIC_URL is set only where the swarm vhost is served,
because a hook whose target_url nothing answers is worse than no hook.
Keeps why the setting exists (the bridge writes this file and cannot
restart authelia) and drops the incident narrative and the answered
objections -- that history belongs in the PR discussion, not in three
comment blocks a future reader has to scroll past.
authelia now watches the users file, so the restart is redundant -- and it
was the wrong shape twice over. It could fail: a login was refused for a
user whose record was already correct on disk, with nothing in either log
implicating the reload. And it only ever worked for this writer --
swarm-authelia-bridge writes the same file and cannot restart anything,
since running unprivileged inside the container is the whole reason it may
write it at all. A reload that depends on which process did the writing is
not a reload.
--machine/--unit and their two env vars existed solely to name a
systemctl -M target, so they go with it. That drops two required settings
from the operator surface.
The three objections previously recorded against watch are all answered
now, and are kept next to the decision rather than deleted: the key is
verified against the pinned build (validate-config accepts it and rejects
a misspelling), the watch is on the directory so a rename is observed, and
partial reads are structurally impossible because every writer of this
file goes through write_atomic.
The auth-callout responder decides what an admitted client may publish
from two strings: the prefix marking a hive client, and the client id
allowed to read every hive's key. Both were literals in three places --
swarm-authelia.nix mints "hive-${name}", swarm-controller.nix defines
"swarm-controller", and the responder carried its own copies as clap
defaults because swarm-nats.nix passed neither.
Each producer now publishes its value as a readOnly option and the
responder's ExecStart reads them, so the agreement is one evaluation
rather than three strings that happen to be equal. Same pattern the
module already uses for `--account`, and the same argument
swarm-authelia.nix gives for publishing `machine` and `unit`.
Worth the change because the failure is silent and misattributed:
rename either principal and the responder starts denying the one that
stopped matching, a denial reaches a NATS client as a timeout rather
than an error, and a hive that is refused looks exactly like a hive
that has not reported yet.
The unit restricted RestrictAddressFamilies to AF_UNIX, which was correct
while the daemon only served its unix socket. It has since grown three
outbound clients -- authelia token minting and forge calls over HTTPS, and
the queue over NATS -- and every socket(AF_INET, ...) was refused by seccomp.
systemd surfaces that refusal as EAFNOSUPPORT, "Address family not supported
by protocol", so the failure names the protocol and never the sandbox. The
visible symptom was swarm agent creation failing while minting a bearer
token, with a connect error that reads like a network fault.
Permit AF_INET/AF_INET6 for those clients and AF_NETLINK, which glibc's
getaddrinfo needs to enumerate local addresses before returning one. The
rest of the unit's hardening is unchanged.
`swarm-controller.service` carried `SetCredential=queue-client.secret:` — an
empty value, which systemd's parser refuses:
/etc/systemd/system/swarm-controller.service:38:
Invalid syntax, ignoring: queue-client.secret:
So the line was dropped on every daemon-reload, and `LoadCredential=` was
fatal again — precisely the failure that default was added to prevent. On a
hive where authelia has not yet minted the secret, the controller refuses to
start rather than coming up with the queue unconfigured.
It looked correct for days because the credential file happened to exist, so
the fail-soft was never exercised.
Measured with `systemd-analyze verify`: an empty value is rejected, any
non-empty one is accepted. The placeholder is a real word rather than filler —
it reaches the token request as the client secret, so authelia refuses it and
the journal names something an operator can act on.
Gate: state/eval-setcredential.sh, with the parent commit as its mutation.
A systemd credential named by an absolute path is fatal when the file is
missing, and the co-located queue secret is minted by authelia's first
boot in another container — which a host unit cannot order against. On a
real boot the daemon spent three of systemd's five default starts losing
that race before the file appeared; two seconds more would have reached
start-limit-hit, which does not self-heal.
An empty SetCredential acts as the default that makes the load
non-fatal, so the controller starts and serves its HTTP surface with the
queue unconfigured — a shape it already reports.
That alone would be a regression, though: a credential is snapshotted at
unit start, so the placeholder would freeze and the daemon would sit
degraded forever instead of recovering the way the restart loop
accidentally did. A path unit on the secret file closes it, and closes a
second gap in the same stroke — mint_token reads the secret on every
call precisely so a rotation takes effect without a restart, and a
snapshot in %d silently defeats that.
PathChanged and not PathExists: the latter activates immediately when
the file is already present at unit start, which would restart a healthy
daemon on every boot.
The option being consumed already warned about this shape — its own
description says a consumer has to wait for the secret because a
missing source turns a fresh hive into a boot-order deadlock. That
warning was written on the producing side and did not fire while I was
writing the consumer.
Review point: the co-located defaults are an auto-deploy special case and
belong with the other ones, not inside each option's own default. An
option should describe itself; the mode describes what a deployment shape
implies. local-defaults.nix already says exactly this in its header.
Options now default to empty (= unset, which the assertions refuse), and
enableAllLocalDefaults fills in loopback + the minted secret path.
The controller's queue coordinates were gated on `autheliaCfg.enable &&
natsCfg.enable` -- i.e. on this host running both. Split the hosts and
the whole env block vanished, which the daemon's own rule reads as "no
queue configured": it started cleanly, served /api/hives, and silently
never connected or published.
Make the coordinates options instead, defaulted to the co-located values
so nobody types a path they didn't need, and assert each one so a split
deployment fails at eval naming the option rather than running blind.
The queue is required for a controller -- /api/hives/status reads the KV
-- but co-location with it is not.
Follow-up to the plumbing removal: docs/swarm/README.md gets the
biggest rewrite (drops the whole "Fingerprint format" section, fixes
the runtime-effects list, the WireGuard config example + "what the
mesh does" bullet), docs/gateway.md and hive-gateway/options.nix drop
4 "needs no certFingerprint" mentions, swarm-peers-removed.nix's
migration-warning text no longer tells an upgrading operator to carry
a field over that no longer exists, swarm.nix/swarm-wireguard.nix/
swarm-controller.nix/swarm-controller's main.rs get comment fixes
where they described the now-removed HYPERHIVE_PEERS shape. Also
caught one more stale "peer hives" mention in docs/web-ui/README.md's
SW4RM tab description that the first pass on this issue missed.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
New `services.hyperhive.swarm.controller.links` option (listOf {label,
icon, url}, same shape as the per-agent hyperhive.dashboardLinks) plus
a new GET /api/links route serving it, same pattern as the existing
hives/GET /api/hives.
Rather than one central hardcoded list, each service's own module
contributes its own entry when actually enabled on the controller's
host: swarm-authelia.nix, hive-matrix.nix (gated on gui.enable too,
since / on that vhost only serves fluffychat then) and
hive-forge/default.nix (gated on behindGateway) each push one entry,
the same list-merge idiom services.hyperhive.gateway.localNames
already uses. swarm-ui.nix contributes a static entry for its own
same-origin swagger docs. Adding a future service's link is a nix-only
change to that service's own module.
Verified: cargo build/clippy/test -p swarm-controller clean, a
throwaway nixosSystem eval confirms all 4 entries merge correctly into
SWARM_CONTROLLER_LINKS, nix build .#swarm-controller succeeds.
Fixes hyperhive#3223.
swarm-controller: GET /api/hives (utoipa-annotated same as /health),
serving the swarm's hive directory (name + domain) loaded once at
startup from a new SWARM_CONTROLLER_HIVES env var. The controller's
NixOS module sets it from services.hyperhive.swarm.hives, JSON-encoded
the same way hive-c0re already builds HYPERHIVE_PEERS for its own peer
list (environment.nix) — the full directory here rather than
peers-minus-self, since a swarm-level daemon has no 'self' hive to
exclude. Unset/malformed both fall back to an empty list with a
warning rather than failing startup, so /health stays answerable even
if this one env var is wrong.
swarm-ui: App.tsx's Home route fetches /api/hives and renders it
through the already-merged <Table>/<StatusChip>/<Panel> primitives —
name, domain (linking out to that hive's own gateway-routed
dashboard), and a static "configured" status chip until a real
online/stale/offline rollup exists server-side. Also gave swarm-ui a
base <a> color (theme's --blue) — base.css covers body/typography but
not links, and this is genuinely page-level rather than any one
component's concern.
Verified end to end, not just source-reading: ran the real
swarm-controller binary with SWARM_CONTROLLER_HIVES set, curled
/api/hives + /health over its actual unix socket; separately served
the real swarm-ui dist against a mock /api/hives and screenshotted the
rendered table. Also re-verified the nginx wiring evaluates (same
throwaway nixosSystem eval technique as #3212) — SWARM_CONTROLLER_HIVES
resolves to the expected JSON shape.
cargo test/clippy -p swarm-controller clean (2 tests, including a new
load_hives one covering missing/malformed/valid env var states). npm
run build + typecheck clean.
Prose-only sweep of every remaining claim that nginx or dnsmasq lives
in a container: the port comment (root in a container -> root on the
host), upstreamHost's netns rationale, the ACME state dir, the store
path reachability note, the vhost tree header, dnsmasq's resolv.conf
paragraph (there is no copy and no path unit watching it any more),
the two hive-network bridge comments, and swarm-controller's socket
access-control note, which described a bind-mount that no longer
exists.
No behaviour change; all of it was describing a mechanism that was
deleted.
`enableAllLocalDefaults` already asserts the swarm's shared services and
its CA; the controller was the one swarm-level thing it left off, so the
default deployment ran authelia, matrix and the forge with nothing
controlling them — and, until the previous commit in this area, without
`swarmctl` either.
The controller's own option stays `default = false`. Running it is a
statement about swarm topology rather than about hyperhive being
installed, and no single host can infer that on its own. But "this box
is the whole deployment" IS that statement, which is why the mode may
assert what `services.hyperhive.enable` never could.
Derived from the mode, not from `enableRequiredServices`: a hive in a
larger swarm can legitimately want the shared services without being the
host that controls them.
`mkDefault`, so `enableAllLocalDefaults = true` with an explicit
`controller.enable = false` still yields a controller-less box — the
mode fills in for an operator who hasn't spoken and never argues with
one who has.
The swarm-authelia module states that its users database is written by
swarm-controller, but nothing ever granted the means. This adds the tool
that does it.
swarmctl runs as root on the controller's host and acts directly. The
rootless alternative was examined and does not work: relocating the users
file into a directory the controller owns only turns a write problem into
a read problem, because authelia must then reach across the same boundary
in the other direction. Making that read work needs either a hand-pinned
gid or world-readable password hashes.
The user store is two files, one authoritative: users.json is canonical,
users.yml is a rendered artifact. That split is what lets the crate work
without a YAML parser -- the workspace has none, and adding one costs a
crates.io fetch, a lock update and a vendor hash for a schema we fully
control and only ever emit.
Passwords are generated by authelia rather than passed to it: argv is
world-readable, so a password on a command line is readable by any local
process for the lifetime of the call.
The three derived facts swarmctl needs about the authelia container --
machine, unit and the host-side users path -- become readOnly options on
the authelia module rather than literals repeated at the call site.
services.hyperhive.swarm.controller.{enable,package,socketPath} plus the
unprivileged swarm-controller user, its runtime and state directories,
and the unit itself.
enable is deliberately not derived from services.hyperhive.enable, unlike
c0re: a swarm has one controller, so turning it on is a statement about
swarm topology rather than about whether hyperhive is installed.
The socket gets its own RuntimeDirectory. nginx reaches a unix upstream
by having the socket's directory bind-mounted into the gateway
container, and the socket is 0666 because connect needs write -- so the
directory is the only access control there is. Sharing one with the host
admin socket would hand that socket to the gateway too. The constraint
is stated at both ends, in the option description and beside the bind,
because it is invisible from either site alone; a test pins the path so
a tidying edit fails rather than reviews cleanly.
RuntimeDirectoryPreserve and the daemon's stale-socket unlink are a
pair: preserving the directory without the unlink means bind fails with
EADDRINUSE after a restart.
