Finishes the anyhow removal for the parts this branch adds: the status
bucket's open-or-create and the connected-client precondition. Two
variants, one of them behind the `kv` feature because the error type it
wraps does not exist without it — the error enum respects the same gate
the module does.
NotConnected is deliberately distinct from Connect: one is a connect that
was attempted and refused, the other is a request made before any
connection exists. The first is a deployment problem and the second is a
caller-ordering one, which is the whole reason a caller wants an enum
rather than a string.
The controller's `store` now returns the queue client's error rather than
an anyhow one: `OnceCell::get_or_try_init` takes its error type from the
closure, so widening there would mean converting inside the closure for
no gain. `view` `?`s it and anyhow converts at that boundary — the
library keeps a typed error, the binary keeps anyhow, and no call site
pays for the split.
An unconnected client does not fail a JetStream request, it hangs on it:
`retry_on_initial_connect` hands back a client before it is usable, and a
request made in that window waits (measured: still going at 15s against a
queue that refuses the credential). The controller guarded its read path
against that inline. Every consumer of the queue needs the same guard, so
it is not one daemon's to keep.
It matters more off a request path than on one. A hung request inside a
periodic task never reaches its `select!`, so the shutdown branch becomes
unreachable and the task cannot be stopped at all — where a request path
merely times a poll out.
The test is `!= Connected`, never `== Disconnected`: a client that has
never connected sits in `Pending`, so the `Disconnected` form passes it
straight through to the hang it was written to prevent — which is exactly
the boot-order case the guard exists for. Not feature-gated;
`connection_state()` is core async-nats.
The bucket has two ends in two crates: a hive writes its own key, the
controller reads every key. `swarm-controller` declared the name as a
private const with a doc comment arguing that "reader and writer must
name the same bucket" — an argument the writer, in another crate, could
not obey.
The name is the mild half. Both ends do get-or-create, because either may
come up first on a fresh swarm and neither can assume the other has run.
Two `Config`s that drift means whichever end created the bucket wins and
the other's `get_key_value` succeeds against a bucket it did not ask for:
no error, no log, just a retention policy nobody chose. Sharing the
constructor gives that race one outcome.
Behind a default-off `kv` feature, so the crate's other consumer — the
auth-callout responder, which speaks the connect and nothing else — still
pulls neither `jetstream` nor `kv`. That was the actual reason the
feature was excluded when this crate was extracted; the flag preserves
it. The surface is deliberately narrow: one bucket's name and creation
config, not a general KV facade.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
