Finishes the anyhow removal for the parts this branch adds: the status
bucket's open-or-create and the connected-client precondition. Two
variants, one of them behind the `kv` feature because the error type it
wraps does not exist without it — the error enum respects the same gate
the module does.
NotConnected is deliberately distinct from Connect: one is a connect that
was attempted and refused, the other is a request made before any
connection exists. The first is a deployment problem and the second is a
caller-ordering one, which is the whole reason a caller wants an enum
rather than a string.
The controller's `store` now returns the queue client's error rather than
an anyhow one: `OnceCell::get_or_try_init` takes its error type from the
closure, so widening there would mean converting inside the closure for
no gain. `view` `?`s it and anyhow converts at that boundary — the
library keeps a typed error, the binary keeps anyhow, and no call site
pays for the split.
An unconnected client does not fail a JetStream request, it hangs on it:
`retry_on_initial_connect` hands back a client before it is usable, and a
request made in that window waits (measured: still going at 15s against a
queue that refuses the credential). The controller guarded its read path
against that inline. Every consumer of the queue needs the same guard, so
it is not one daemon's to keep.
It matters more off a request path than on one. A hung request inside a
periodic task never reaches its `select!`, so the shutdown branch becomes
unreachable and the task cannot be stopped at all — where a request path
merely times a poll out.
The test is `!= Connected`, never `== Disconnected`: a client that has
never connected sits in `Pending`, so the `Disconnected` form passes it
straight through to the hang it was written to prevent — which is exactly
the boot-order case the guard exists for. Not feature-gated;
`connection_state()` is core async-nats.
The bucket has two ends in two crates: a hive writes its own key, the
controller reads every key. `swarm-controller` declared the name as a
private const with a doc comment arguing that "reader and writer must
name the same bucket" — an argument the writer, in another crate, could
not obey.
The name is the mild half. Both ends do get-or-create, because either may
come up first on a fresh swarm and neither can assume the other has run.
Two `Config`s that drift means whichever end created the bucket wins and
the other's `get_key_value` succeeds against a bucket it did not ask for:
no error, no log, just a retention policy nobody chose. Sharing the
constructor gives that race one outcome.
Behind a default-off `kv` feature, so the crate's other consumer — the
auth-callout responder, which speaks the connect and nothing else — still
pulls neither `jetstream` nor `kv`. That was the actual reason the
feature was excluded when this crate was extracted; the flag preserves
it. The surface is deliberately narrow: one bucket's name and creation
config, not a general KV facade.
Review catch: `anyhow::Error`'s Display special-cases `f.alternate()` to
walk the source chain; thiserror's derive does not, so `{e:#}` and `{e}`
render identically for the new error type. Every call site that held an
`anyhow::Error`, formatted it with `{:#}`, and now holds this crate's
error kept compiling, kept looking right, and silently dropped the cause.
`chain()` was written for exactly this and then left private, applied only
to the auth callback I happened to be editing. Its own doc comment argues
that dropping the source chain is wrong, which made it the one thing in
the PR that should not have had a scope of one.
The controller's "swarm queue unreachable" warning is the site this fixes
here; the stacked PR fixes the two boot-warning banners, which matter more
still — one-shot, no retry, and they leak until restart.
A hive publishing its own status needs the same connect the controller
already has - mint an authelia token, present it at CONNECT for the
callout responder, let async-nats re-run the callback per attempt. Only
the use differs: the controller reads, a hive writes.
Copying it would put credential handling in two places, and a
token-refresh fix would then have to be found twice. That is the same
reasoning that already put hive-sock-client in its own crate rather than
in each daemon that speaks to a unix socket.
`from_env` takes a prefix rather than hardcoding SWARM_CONTROLLER_*: the
variables belong to the consuming unit, since a NixOS module sets them
alongside its other options. What is shared is the RULE - all four
together or none at all - not the spelling. The half-set case gains a
test, because it is the case the rule exists for and it previously had
none.
No jetstream/kv feature on the crate: it ends at a connected client, and
what a consumer does with it should be visible in that consumer's own
Cargo.toml.
Behaviour-preserving, and proven that way rather than by inspection: the
full behavioural gate (real nats-server, credential rotation, mutation)
is 20/0 unchanged, and the controller's own tests still pass.
Follow-up to the plumbing removal: docs/swarm/README.md gets the
biggest rewrite (drops the whole "Fingerprint format" section, fixes
the runtime-effects list, the WireGuard config example + "what the
mesh does" bullet), docs/gateway.md and hive-gateway/options.nix drop
4 "needs no certFingerprint" mentions, swarm-peers-removed.nix's
migration-warning text no longer tells an upgrading operator to carry
a field over that no longer exists, swarm.nix/swarm-wireguard.nix/
swarm-controller.nix/swarm-controller's main.rs get comment fixes
where they described the now-removed HYPERHIVE_PEERS shape. Also
caught one more stale "peer hives" mention in docs/web-ui/README.md's
SW4RM tab description that the first pass on this issue missed.
Both from argus's review on the PR, both non-blocking, both real.
The HTTP client had no timeout, and it runs INSIDE the auth callback: a
token endpoint that accepts the connection and then never answers would
hang the callback and the connection attempt that invoked it, with no
retry and nothing in the log to say why. That is the same hang class the
status endpoint's connection-state check exists to prevent, one layer up.
Failing fast lets async-nats back off and try again, which it already
does well.
And the startup log said "connected to the swarm queue" at a point where
`retry_on_initial_connect` guarantees no connection has been established
yet - so the journal would read "connected", then 503 "not connected"
moments later, and a reader would rightly distrust the second line rather
than the first.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
New `services.hyperhive.swarm.controller.links` option (listOf {label,
icon, url}, same shape as the per-agent hyperhive.dashboardLinks) plus
a new GET /api/links route serving it, same pattern as the existing
hives/GET /api/hives.
Rather than one central hardcoded list, each service's own module
contributes its own entry when actually enabled on the controller's
host: swarm-authelia.nix, hive-matrix.nix (gated on gui.enable too,
since / on that vhost only serves fluffychat then) and
hive-forge/default.nix (gated on behindGateway) each push one entry,
the same list-merge idiom services.hyperhive.gateway.localNames
already uses. swarm-ui.nix contributes a static entry for its own
same-origin swagger docs. Adding a future service's link is a nix-only
change to that service's own module.
Verified: cargo build/clippy/test -p swarm-controller clean, a
throwaway nixosSystem eval confirms all 4 entries merge correctly into
SWARM_CONTROLLER_LINKS, nix build .#swarm-controller succeeds.
Nothing in the gate read doc-comments: clippy doesn't check intra-doc
links, cargo test doesn't, and no check built docs. So a [`Foo`] pointing
at a renamed, moved or deleted item rendered as plain text and had no
discoverer but a human happening to read the comment.
That matters here more than in most repos, because the convention is to
put a thing's authoritative description in one doc-comment and point at
it from everywhere else -- the design leans on the pointers being real,
and a dangling link is worse than no link since it names something and
sends the reader looking.
Adds `docs-rustdoc` to nix/checks.nix: craneLib.cargoDoc over
--workspace --no-deps --document-private-items, denying six rustdoc
lints. Listed explicitly rather than -D warnings so a new lint appearing
upstream cannot red the build on a class nobody has triaged.
--document-private-items is load-bearing rather than thoroughness for
its own sake: most of this workspace's doc-comments live on private
items and //! module headers, so without it rustdoc checks a small
fraction of the links and the gate sits green while the rot continues.
Then fixes every error it reports, 40 to 0 across nine crates. The
classes differ and so do the fixes:
- public item, wrong scope -> qualify. Node and Node::parent are both
public; the link failed only because scheduler.rs does not import
Node. Six sites become [`crate::Node::parent`].
- private item -> downgrade to backticks. Nothing was made public to
satisfy a lint; changing API surface to appease a doc check would be
the tail wagging the dog.
- genuinely dead -> [`JobBuilder::insert_into`] names a method that does
not exist. Insertion is Scheduler::insert_job.
- prose that looks like markup -> argv[0] parsed as a link, and
<args>/<hex>/<name> parsed as HTML tags.
Note for future fixes: pub(crate) resolves in an intra-doc link, a plain
private fn in a binary crate does not (wait_for_nodes resolved,
connect_hint did not, same crate, same shape).
The check does not ride the clippy/test artifact cache. It takes
cargoArtifacts, but rustdoc needs its own flavour of dependency
metadata, which cargo build does not produce, so a --no-deps docs build
still compiles dependencies it never documents. Measured at 6m47s cold;
that reasoning is recorded in the check's own comment so the next reader
does not re-derive it.
Verified by running the check's exact command against the pre-cleanup
tree first: 40 errors, build failed. A gate that cannot fail is not
evidence, and building it before the cleanup makes that proof free.
Fixes hyperhive#3223.
swarm-controller: GET /api/hives (utoipa-annotated same as /health),
serving the swarm's hive directory (name + domain) loaded once at
startup from a new SWARM_CONTROLLER_HIVES env var. The controller's
NixOS module sets it from services.hyperhive.swarm.hives, JSON-encoded
the same way hive-c0re already builds HYPERHIVE_PEERS for its own peer
list (environment.nix) — the full directory here rather than
peers-minus-self, since a swarm-level daemon has no 'self' hive to
exclude. Unset/malformed both fall back to an empty list with a
warning rather than failing startup, so /health stays answerable even
if this one env var is wrong.
swarm-ui: App.tsx's Home route fetches /api/hives and renders it
through the already-merged <Table>/<StatusChip>/<Panel> primitives —
name, domain (linking out to that hive's own gateway-routed
dashboard), and a static "configured" status chip until a real
online/stale/offline rollup exists server-side. Also gave swarm-ui a
base <a> color (theme's --blue) — base.css covers body/typography but
not links, and this is genuinely page-level rather than any one
component's concern.
Verified end to end, not just source-reading: ran the real
swarm-controller binary with SWARM_CONTROLLER_HIVES set, curled
/api/hives + /health over its actual unix socket; separately served
the real swarm-ui dist against a mock /api/hives and screenshotted the
rendered table. Also re-verified the nginx wiring evaluates (same
throwaway nixosSystem eval technique as #3212) — SWARM_CONTROLLER_HIVES
resolves to the expected JSON shape.
cargo test/clippy -p swarm-controller clean (2 tests, including a new
load_hives one covering missing/malformed/valid env var states). npm
run build + typecheck clean.
Rust half mirrors hive-c0re/src/dashboard/mod.rs's utoipa pattern
exactly: an ApiDoc root, #[utoipa::path(...)] on /health (the one
existing route), and a raw JSON route at /api/openapi.json served
via OpenApiRouter::split_for_parts(). Only annotated routes appear
in the spec.
Gateway wiring extends the swarm-UI vhost (the only vhost swarm-
controller is reachable from) with:
- /api/ — proxied to the controller's unix socket untouched (no URI
segment after the socket path), so a route swarm-controller
registers is the path nginx forwards, no prefix-stripping to keep
in sync by hand.
- /api/docs/ (+ the bare /api/docs redirect) — the same
swagger-ui-theme dist the per-hive dashboard already serves at its
own /api/docs/, reused as-is since it's generic.
Both new locations reuse the same auth_request block the vhost's own
'/' already applies, factored into a shared swarmAuthRequest string —
auth_request does not inherit across sibling nginx locations, so
without this the page itself would be gated while its own API and
API docs sat open.
cargo test -p swarm-controller + cargo clippy --all-targets both
clean. Verified the new nginx wiring evaluates correctly with a
throwaway nixosSystem eval (services.hyperhive.swarm.{controller,ui}
enabled): /api/ proxies to the socket, /api/docs redirects, and both
require auth_request the same as the vhost root.
Fixes hyperhive#3212
Per review: docs represent current state. Every "used to" / "no longer"
clause this branch introduced is gone — including the History section in
network.md, which was a whole subsection about a sync mechanism that
doesn't exist.
Where the removed clause was carrying a real constraint, the constraint
stays and is stated in the present tense instead of as a delta: nothing
narrows what the gateway's nginx can reach except the directory
permissions in front of a socket, and nothing bounds `ReloadGatewayNginx`
except the hard-coded unit name. Those read as rules now rather than as
the story of how they came to be rules.
The gateway's nginx + dnsmasq no longer run in their own nspawn container.
`nix/host-modules/hive-gateway/default.nix` loses the
`containers.hive-gateway` wrapper and everything that existed only to punch
holes in it: `privateNetwork = false`, `CAP_NET_ADMIN`, five bind mounts,
its own `stateVersion`, `networking.firewall.enable = false`,
`networking.resolvconf.enable = false`, and the `hive-gateway-resolv`
path+service pair. 465 -> 303 lines.
The container never bought isolation here. It shared the host netns by
necessity — nginx binds the host's :80/:443, dnsmasq answers on the bridge —
so each of those settings was undoing a boundary the gateway could not
afford in the first place.
Four things made it more than a deletion, none of them visible in the nix
diff:
- The self-signed cert service also imports the hive CA leaf, so removing it
with the container would have left nginx naming a missing cert file, which
it refuses to load at all.
- The nginx reload is a hive-priv verb. It still needs root, but no longer
for the reason its doc gave, and `--machine=` was both transport and
scope — so the unit name is now hard-coded in the helper as the
containment.
- The lifecycle verb named a container that stops existing.
- `journalctl -M hive-gateway` had no machine to enter.
Per the operator's ruling, the operator verb keeps working and agents lose
it. `InfraContainer` answered three questions that used to share an answer;
it now splits into `name()` (identity), `target()` (Container vs HostUnit),
`service_unit()` (the systemd unit), and `agent_restartable()`, which the
MCP restart path checks before the capability so the refusal cannot read as
"ask for infra_admin". `SIBLING_CONTAINERS` drops the gateway — it gates the
requests that name a container as a string — while `FromStr` still accepts
it, because that answers what a name is, not who may act on it. The
dashboard's gateway journal reads host journald filtered to `nginx.service`.
Prose was corrected where it only named a location, and re-argued where the
container was doing security work: a `0666` per-agent socket was safe
because only the gateway container had the directory bind-mounted. There is
no mount now, so the directory permissions are the whole of the access
control — the constraint holds, its mechanism doesn't.
Gate: nix fmt / clippy --all-targets -D warnings / cargo test all clean (710
tests); hivectl-cli.md regenerated from the clap tree. The nix eval was run
in both TLS shapes at this commit: every delta in the rendered
virtualHosts is one of the three intended path moves, dnsmasq settings are
byte-identical, and the absence probe flips true -> false with bindMounts
emptied.
services.hyperhive.swarm.controller.{enable,package,socketPath} plus the
unprivileged swarm-controller user, its runtime and state directories,
and the unit itself.
enable is deliberately not derived from services.hyperhive.enable, unlike
c0re: a swarm has one controller, so turning it on is a statement about
swarm topology rather than about whether hyperhive is installed.
The socket gets its own RuntimeDirectory. nginx reaches a unix upstream
by having the socket's directory bind-mounted into the gateway
container, and the socket is 0666 because connect needs write -- so the
directory is the only access control there is. Sharing one with the host
admin socket would hand that socket to the gateway too. The constraint
is stated at both ends, in the option description and beside the bind,
because it is invisible from either site alone; a test pins the path so
a tidying edit fails rather than reviews cleanly.
RuntimeDirectoryPreserve and the daemon's stale-socket unlink are a
pair: preserving the directory without the unlink means bind fails with
EADDRINUSE after a restart.
bind leaves a unix socket 0755 and connecting needs write, so the
gateway's nginx -- a different user -- would be locked out.
0666 is what hive-c0re already does for the per-agent sockets, and it
rests on the same argument: the containing directory is the access
control, not the socket mode. This directory holds one socket and is
bind-mounted into exactly one container.
That is also the sharper reason the socket does not live beside the host
admin socket. With a 0666 socket, a directory that carries more than it
should is not untidiness, it is the vulnerability.
First half of the swarm-controller slice: the crate, its workspace entry
and its daemonBins entry, so the systemd unit that follows has a binary
to point at.
It serves one health endpoint and owns no state. That is the whole
intent -- this makes the unit real (service user, runtime and state
directories, socket, nginx reachability) so the swarm-level surfaces
that follow have somewhere to land. Inventing those surfaces now would
bake in a shape nobody has agreed to.
The socket gets its own runtime directory rather than sharing
hive-c0re's. nginx reaches a unix upstream by having the socket's
directory bind-mounted into the gateway container, so co-locating this
socket with the host admin socket would hand the gateway that socket
too.
