feat(swarm-controller): aggregate per-hive status from the swarm queue

The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.

THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.

Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.

Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.

Three failure shapes are deliberate:

- A half-set environment is fatal; an absent one is not. Silently
  behaving like an unconfigured host is how every hive ends up reading
  `never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
  cannot be read. "I cannot reach the store" and "every hive is silent"
  are different answers, and rendering the second turns a local fault
  into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
  either order, and the status handler refuses when the client is not
  Connected rather than issuing a request into it - a request made in
  that window does not fail, it waits, so every poll would hang and
  learn nothing. `Pending` is the state a never-connected client is in,
  which is why the test is `!= Connected` and not `== Disconnected`.

The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
This commit is contained in:
atlas 2026-08-15 18:37:23 +02:00
commit 8891b46943
7 changed files with 1030 additions and 17 deletions

View file

@ -330,6 +330,65 @@ What it serves, why it is a unix socket rather than a port, and the
socket-directory constraint that governs where `socketPath` may point:
[`swarm-controller/README.md`](../../swarm-controller/README.md).
### Per-hive status (`GET /api/hives/status`)
What each hive last **offered** about itself. Hives publish upward; the
controller never reaches down to collect. That direction is deliberate:
during the #3097 gateway outage every recovery channel ran through the
one broken thing, so a status path that depended on the controller would
have gone dark exactly when it was needed to diagnose the controller's
own network. A hive computes its own status locally either way — this
endpoint is a view of what was published, never the source.
**The queue is the store.** A hive publishes into the `hive-status`
JetStream KV bucket (`history: 1` — the last thing each hive said), and
the controller reads that bucket per request, keeping no copy. A cache
here would be a second answer free to disagree with the first, and the
disagreement would surface as a hive reading healthy on a dashboard
while the bucket says otherwise. The bucket is created by whichever side
gets there first.
**Absence is what the endpoint is built around**:
| freshness | means |
|---|---|
| `fresh` | published within `staleAfterSeconds` |
| `stale` | published longer ago than that — the payload is still returned, because "old" and "absent" are different answers |
| `never_reported` | in the roster, has never published. Distinct from `stale`: went quiet is a fault, never spoke is usually a deployment that hasn't happened |
| `unknown` | published but not in `swarm.hives` — surfaced rather than dropped |
Rows come from the **roster**, not from the bucket, so a hive that has
never reported appears rather than not appearing, and an empty bucket
renders as a swarm nobody has heard from instead of a healthy one.
Freshness is derived at read time from a timestamp and never stored as a
flag — a stored `healthy` boolean goes stale silently the moment nothing
arrives, which is the failure this is designed against. Each row also
carries `last_seen_unix` and `age_seconds`, so a consumer that disagrees
with `staleAfterSeconds` can apply its own threshold.
`last_seen_unix` is the **bucket's** timestamp, applied by the NATS
server when the value landed, not a field inside the payload — a
publisher cannot make itself look fresher than it is, and a hive with a
wrong clock skews its own payload rather than its freshness.
Because the bucket outlives a controller restart, a restarted controller
reports what it reads: `stale, age_seconds: 10800` rather than
`never_reported`. That is the more honest of the two — it genuinely
knows when the hive last spoke. Losing the bucket degrades in the same
direction: every hive reads `never_reported` until its next publish,
which is the true answer and not a remembered "healthy".
The endpoint answers **503**, not an empty list, when no queue is
configured on this host or its store cannot be read. "I cannot reach the
store" and "every hive is silent" are different answers, and rendering
the second when the first is true would turn a local fault into an
apparent swarm-wide outage.
⚠️ **Nothing writes to the bucket yet.** The transport and the
controller's read path are in place; the hive-side publisher is a later
slice. Until one lands, every hive reads `never_reported` — the correct
answer for a controller that has been told nothing.
## Cross-references
- `docs/snapshot-store.md` — the swarm's `btrfs receive` endpoint, and