docs(swarm): make the status section operator-facing

Per mara on the PR: less implementation detail, more operator focus. The
section argued why the design is shaped as it is - a cache would
disagree with itself, freshness must be derived rather than stored, the
timestamp is the server's - which is reasoning a reader of this page does
not need and cannot act on.

It now answers what an operator actually asks: what each freshness value
means they should do, what to set the threshold to and why that number,
what a 503 tells them, and that nothing publishes yet so an empty board
is expected rather than broken.

The design rationale is not lost, it is just homed where it belongs: the
status module's own doc comment, next to the code it constrains.
This commit is contained in:
atlas 2026-08-15 18:48:51 +02:00
commit 602ae92a3e

View file

@ -332,62 +332,36 @@ socket-directory constraint that governs where `socketPath` may point:
### Per-hive status (`GET /api/hives/status`) ### Per-hive status (`GET /api/hives/status`)
What each hive last **offered** about itself. Hives publish upward; the One row per hive in `swarm.hives`, saying when it last reported and what
controller never reaches down to collect. That direction is deliberate: it said. Hives publish upward through the swarm queue; the controller
during the #3097 gateway outage every recovery channel ran through the never reaches down to collect, so a hive that cannot reach the swarm
one broken thing, so a status path that depended on the controller would still knows its own state — you just cannot see it from here.
have gone dark exactly when it was needed to diagnose the controller's
own network. A hive computes its own status locally either way — this
endpoint is a view of what was published, never the source.
**The queue is the store.** A hive publishes into the `hive-status` ⚠️ **Nothing publishes yet.** The read path is in place; the hive-side
JetStream KV bucket (`history: 1` — the last thing each hive said), and publisher lands in a later change. Until it does, every hive reads
the controller reads that bucket per request, keeping no copy. A cache `never_reported`.
here would be a second answer free to disagree with the first, and the
disagreement would surface as a hive reading healthy on a dashboard
while the bucket says otherwise. The bucket is created by whichever side
gets there first.
**Absence is what the endpoint is built around**: | freshness | what to do about it |
| freshness | means |
|---|---| |---|---|
| `fresh` | published within `staleAfterSeconds` | | `fresh` | nothing — reported within `staleAfterSeconds` |
| `stale` | published longer ago than that — the payload is still returned, because "old" and "absent" are different answers | | `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
| `never_reported` | in the roster, has never published. Distinct from `stale`: went quiet is a fault, never spoke is usually a deployment that hasn't happened | | `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
| `unknown` | published but not in `swarm.hives` — surfaced rather than dropped | | `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
Rows come from the **roster**, not from the bucket, so a hive that has Every row also carries `last_seen_unix` and `age_seconds` if you want to
never reported appears rather than not appearing, and an empty bucket apply your own threshold. The timestamp is the one the queue recorded on
renders as a swarm nobody has heard from instead of a healthy one. arrival, not one the hive put in its own payload.
Freshness is derived at read time from a timestamp and never stored as a
flag — a stored `healthy` boolean goes stale silently the moment nothing
arrives, which is the failure this is designed against. Each row also
carries `last_seen_unix` and `age_seconds`, so a consumer that disagrees
with `staleAfterSeconds` can apply its own threshold.
`last_seen_unix` is the **bucket's** timestamp, applied by the NATS Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
server when the value landed, not a field inside the payload — a `120`) **above the rate hives publish at**, or everything reads `stale`
publisher cannot make itself look fresher than it is, and a hive with a between reports. It takes effect on the next request; nothing has to
wrong clock skews its own payload rather than its freshness. re-publish.
Because the bucket outlives a controller restart, a restarted controller The endpoint answers **503** when this host has no swarm queue
reports what it reads: `stale, age_seconds: 10800` rather than configured, or has one and cannot read it — deliberately not an empty
`never_reported`. That is the more honest of the two — it genuinely list, which would look like a silent swarm rather than a controller that
knows when the hive last spoke. Losing the bucket degrades in the same cannot see. The body says which. Status survives a controller restart:
direction: every hive reads `never_reported` until its next publish, it is stored in the queue, not in the daemon.
which is the true answer and not a remembered "healthy".
The endpoint answers **503**, not an empty list, when no queue is
configured on this host or its store cannot be read. "I cannot reach the
store" and "every hive is silent" are different answers, and rendering
the second when the first is true would turn a local fault into an
apparent swarm-wide outage.
⚠️ **Nothing writes to the bucket yet.** The transport and the
controller's read path are in place; the hive-side publisher is a later
slice. Until one lands, every hive reads `never_reported` — the correct
answer for a controller that has been told nothing.
## Cross-references ## Cross-references