docs(swarm): make the status section operator-facing
Per mara on the PR: less implementation detail, more operator focus. The section argued why the design is shaped as it is - a cache would disagree with itself, freshness must be derived rather than stored, the timestamp is the server's - which is reasoning a reader of this page does not need and cannot act on. It now answers what an operator actually asks: what each freshness value means they should do, what to set the threshold to and why that number, what a 503 tells them, and that nothing publishes yet so an empty board is expected rather than broken. The design rationale is not lost, it is just homed where it belongs: the status module's own doc comment, next to the code it constrains.
This commit is contained in:
parent
465d68d5a7
commit
602ae92a3e
1 changed files with 29 additions and 55 deletions
|
|
@ -332,62 +332,36 @@ socket-directory constraint that governs where `socketPath` may point:
|
||||||
|
|
||||||
### Per-hive status (`GET /api/hives/status`)
|
### Per-hive status (`GET /api/hives/status`)
|
||||||
|
|
||||||
What each hive last **offered** about itself. Hives publish upward; the
|
One row per hive in `swarm.hives`, saying when it last reported and what
|
||||||
controller never reaches down to collect. That direction is deliberate:
|
it said. Hives publish upward through the swarm queue; the controller
|
||||||
during the #3097 gateway outage every recovery channel ran through the
|
never reaches down to collect, so a hive that cannot reach the swarm
|
||||||
one broken thing, so a status path that depended on the controller would
|
still knows its own state — you just cannot see it from here.
|
||||||
have gone dark exactly when it was needed to diagnose the controller's
|
|
||||||
own network. A hive computes its own status locally either way — this
|
|
||||||
endpoint is a view of what was published, never the source.
|
|
||||||
|
|
||||||
**The queue is the store.** A hive publishes into the `hive-status`
|
⚠️ **Nothing publishes yet.** The read path is in place; the hive-side
|
||||||
JetStream KV bucket (`history: 1` — the last thing each hive said), and
|
publisher lands in a later change. Until it does, every hive reads
|
||||||
the controller reads that bucket per request, keeping no copy. A cache
|
`never_reported`.
|
||||||
here would be a second answer free to disagree with the first, and the
|
|
||||||
disagreement would surface as a hive reading healthy on a dashboard
|
|
||||||
while the bucket says otherwise. The bucket is created by whichever side
|
|
||||||
gets there first.
|
|
||||||
|
|
||||||
**Absence is what the endpoint is built around**:
|
| freshness | what to do about it |
|
||||||
|
|
||||||
| freshness | means |
|
|
||||||
|---|---|
|
|---|---|
|
||||||
| `fresh` | published within `staleAfterSeconds` |
|
| `fresh` | nothing — reported within `staleAfterSeconds` |
|
||||||
| `stale` | published longer ago than that — the payload is still returned, because "old" and "absent" are different answers |
|
| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say |
|
||||||
| `never_reported` | in the roster, has never published. Distinct from `stale`: went quiet is a fault, never spoke is usually a deployment that hasn't happened |
|
| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage |
|
||||||
| `unknown` | published but not in `swarm.hives` — surfaced rather than dropped |
|
| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running |
|
||||||
|
|
||||||
Rows come from the **roster**, not from the bucket, so a hive that has
|
Every row also carries `last_seen_unix` and `age_seconds` if you want to
|
||||||
never reported appears rather than not appearing, and an empty bucket
|
apply your own threshold. The timestamp is the one the queue recorded on
|
||||||
renders as a swarm nobody has heard from instead of a healthy one.
|
arrival, not one the hive put in its own payload.
|
||||||
Freshness is derived at read time from a timestamp and never stored as a
|
|
||||||
flag — a stored `healthy` boolean goes stale silently the moment nothing
|
|
||||||
arrives, which is the failure this is designed against. Each row also
|
|
||||||
carries `last_seen_unix` and `age_seconds`, so a consumer that disagrees
|
|
||||||
with `staleAfterSeconds` can apply its own threshold.
|
|
||||||
|
|
||||||
`last_seen_unix` is the **bucket's** timestamp, applied by the NATS
|
Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default
|
||||||
server when the value landed, not a field inside the payload — a
|
`120`) **above the rate hives publish at**, or everything reads `stale`
|
||||||
publisher cannot make itself look fresher than it is, and a hive with a
|
between reports. It takes effect on the next request; nothing has to
|
||||||
wrong clock skews its own payload rather than its freshness.
|
re-publish.
|
||||||
|
|
||||||
Because the bucket outlives a controller restart, a restarted controller
|
The endpoint answers **503** when this host has no swarm queue
|
||||||
reports what it reads: `stale, age_seconds: 10800` rather than
|
configured, or has one and cannot read it — deliberately not an empty
|
||||||
`never_reported`. That is the more honest of the two — it genuinely
|
list, which would look like a silent swarm rather than a controller that
|
||||||
knows when the hive last spoke. Losing the bucket degrades in the same
|
cannot see. The body says which. Status survives a controller restart:
|
||||||
direction: every hive reads `never_reported` until its next publish,
|
it is stored in the queue, not in the daemon.
|
||||||
which is the true answer and not a remembered "healthy".
|
|
||||||
|
|
||||||
The endpoint answers **503**, not an empty list, when no queue is
|
|
||||||
configured on this host or its store cannot be read. "I cannot reach the
|
|
||||||
store" and "every hive is silent" are different answers, and rendering
|
|
||||||
the second when the first is true would turn a local fault into an
|
|
||||||
apparent swarm-wide outage.
|
|
||||||
|
|
||||||
⚠️ **Nothing writes to the bucket yet.** The transport and the
|
|
||||||
controller's read path are in place; the hive-side publisher is a later
|
|
||||||
slice. Until one lands, every hive reads `never_reported` — the correct
|
|
||||||
answer for a controller that has been told nothing.
|
|
||||||
|
|
||||||
## Cross-references
|
## Cross-references
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue