docs(swarm): make the status section operator-facing
Per mara on the PR: less implementation detail, more operator focus. The section argued why the design is shaped as it is - a cache would disagree with itself, freshness must be derived rather than stored, the timestamp is the server's - which is reasoning a reader of this page does not need and cannot act on. It now answers what an operator actually asks: what each freshness value means they should do, what to set the threshold to and why that number, what a 503 tells them, and that nothing publishes yet so an empty board is expected rather than broken. The design rationale is not lost, it is just homed where it belongs: the status module's own doc comment, next to the code it constrains.
This commit is contained in:
parent
465d68d5a7
commit
602ae92a3e
1 changed files with 29 additions and 55 deletions
|
|
@ -332,62 +332,36 @@ socket-directory constraint that governs where `socketPath` may point:
|
|||
|
||||
### Per-hive status (`GET /api/hives/status`)
|
||||
|
||||
What each hive last **offered** about itself. Hives publish upward; the
|
||||
controller never reaches down to collect. That direction is deliberate:
|
||||
during the #3097 gateway outage every recovery channel ran through the
|
||||
one broken thing, so a status path that depended on the controller would
|
||||
have gone dark exactly when it was needed to diagnose the controller's
|
||||
own network. A hive computes its own status locally either way — this
|
||||
endpoint is a view of what was published, never the source.
|
||||
One row per hive in `swarm.hives`, saying when it last reported and what
|
||||
it said. Hives publish upward through the swarm queue; the controller
|
||||
never reaches down to collect, so a hive that cannot reach the swarm
|
||||
still knows its own state — you just cannot see it from here.
|
||||
|
||||
**The queue is the store.** A hive publishes into the `hive-status`
|
||||
JetStream KV bucket (`history: 1` — the last thing each hive said), and
|
||||
the controller reads that bucket per request, keeping no copy. A cache
|
||||
here would be a second answer free to disagree with the first, and the
|
||||
disagreement would surface as a hive reading healthy on a dashboard
|
||||
while the bucket says otherwise. The bucket is created by whichever side
|
||||
gets there first.
|
||||
⚠️ **Nothing publishes yet.** The read path is in place; the hive-side
|
||||
publisher lands in a later change. Until it does, every hive reads
|
||||
`never_reported`.
|
||||
|
||||
**Absence is what the endpoint is built around**:
|
||||
|
||||
| freshness | means |
|
||||
| freshness | what to do about it |
|
||||
|---|---|
|
||||
| `fresh` | published within `staleAfterSeconds` |
|
||||
| `stale` | published longer ago than that — the payload is still returned, because "old" and "absent" are different answers |
|
||||
| `never_reported` | in the roster, has never published. Distinct from `stale`: went quiet is a fault, never spoke is usually a deployment that hasn't happened |
|
||||
| `unknown` | published but not in `swarm.hives` — surfaced rather than dropped |
|
||||
| `fresh` | nothing — reported within `staleAfterSeconds` |
|
||||
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
|
||||
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
|
||||
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
|
||||
|
||||
Rows come from the **roster**, not from the bucket, so a hive that has
|
||||
never reported appears rather than not appearing, and an empty bucket
|
||||
renders as a swarm nobody has heard from instead of a healthy one.
|
||||
Freshness is derived at read time from a timestamp and never stored as a
|
||||
flag — a stored `healthy` boolean goes stale silently the moment nothing
|
||||
arrives, which is the failure this is designed against. Each row also
|
||||
carries `last_seen_unix` and `age_seconds`, so a consumer that disagrees
|
||||
with `staleAfterSeconds` can apply its own threshold.
|
||||
Every row also carries `last_seen_unix` and `age_seconds` if you want to
|
||||
apply your own threshold. The timestamp is the one the queue recorded on
|
||||
arrival, not one the hive put in its own payload.
|
||||
|
||||
`last_seen_unix` is the **bucket's** timestamp, applied by the NATS
|
||||
server when the value landed, not a field inside the payload — a
|
||||
publisher cannot make itself look fresher than it is, and a hive with a
|
||||
wrong clock skews its own payload rather than its freshness.
|
||||
Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
|
||||
`120`) **above the rate hives publish at**, or everything reads `stale`
|
||||
between reports. It takes effect on the next request; nothing has to
|
||||
re-publish.
|
||||
|
||||
Because the bucket outlives a controller restart, a restarted controller
|
||||
reports what it reads: `stale, age_seconds: 10800` rather than
|
||||
`never_reported`. That is the more honest of the two — it genuinely
|
||||
knows when the hive last spoke. Losing the bucket degrades in the same
|
||||
direction: every hive reads `never_reported` until its next publish,
|
||||
which is the true answer and not a remembered "healthy".
|
||||
|
||||
The endpoint answers **503**, not an empty list, when no queue is
|
||||
configured on this host or its store cannot be read. "I cannot reach the
|
||||
store" and "every hive is silent" are different answers, and rendering
|
||||
the second when the first is true would turn a local fault into an
|
||||
apparent swarm-wide outage.
|
||||
|
||||
⚠️ **Nothing writes to the bucket yet.** The transport and the
|
||||
controller's read path are in place; the hive-side publisher is a later
|
||||
slice. Until one lands, every hive reads `never_reported` — the correct
|
||||
answer for a controller that has been told nothing.
|
||||
The endpoint answers **503** when this host has no swarm queue
|
||||
configured, or has one and cannot read it — deliberately not an empty
|
||||
list, which would look like a silent swarm rather than a controller that
|
||||
cannot see. The body says which. Status survives a controller restart:
|
||||
it is stored in the queue, not in the daemon.
|
||||
|
||||
## Cross-references
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue