diff --git a/docs/swarm/README.md b/docs/swarm/README.md index 89e9b1e4..87466635 100644 --- a/docs/swarm/README.md +++ b/docs/swarm/README.md @@ -332,62 +332,36 @@ socket-directory constraint that governs where `socketPath` may point: ### Per-hive status (`GET /api/hives/status`) -What each hive last **offered** about itself. Hives publish upward; the -controller never reaches down to collect. That direction is deliberate: -during the #3097 gateway outage every recovery channel ran through the -one broken thing, so a status path that depended on the controller would -have gone dark exactly when it was needed to diagnose the controller's -own network. A hive computes its own status locally either way — this -endpoint is a view of what was published, never the source. +One row per hive in `swarm.hives`, saying when it last reported and what +it said. Hives publish upward through the swarm queue; the controller +never reaches down to collect, so a hive that cannot reach the swarm +still knows its own state — you just cannot see it from here. -**The queue is the store.** A hive publishes into the `hive-status` -JetStream KV bucket (`history: 1` — the last thing each hive said), and -the controller reads that bucket per request, keeping no copy. A cache -here would be a second answer free to disagree with the first, and the -disagreement would surface as a hive reading healthy on a dashboard -while the bucket says otherwise. The bucket is created by whichever side -gets there first. +⚠️ **Nothing publishes yet.** The read path is in place; the hive-side +publisher lands in a later change. Until it does, every hive reads +`never_reported`. -**Absence is what the endpoint is built around**: - -| freshness | means | +| freshness | what to do about it | |---|---| -| `fresh` | published within `staleAfterSeconds` | -| `stale` | published longer ago than that — the payload is still returned, because "old" and "absent" are different answers | -| `never_reported` | in the roster, has never published. Distinct from `stale`: went quiet is a fault, never spoke is usually a deployment that hasn't happened | -| `unknown` | published but not in `swarm.hives` — surfaced rather than dropped | +| `fresh` | nothing — reported within `staleAfterSeconds` | +| `stale` | the hive stopped reporting. Its last payload is still shown, so check `age_seconds` and the payload for what it managed to say | +| `never_reported` | this hive has never reported at all — normally a deployment that hasn't happened, not an outage | +| `unknown` | something is publishing under a name that is not in `swarm.hives` — a typo in the roster, or a hive that was removed and is still running | -Rows come from the **roster**, not from the bucket, so a hive that has -never reported appears rather than not appearing, and an empty bucket -renders as a swarm nobody has heard from instead of a healthy one. -Freshness is derived at read time from a timestamp and never stored as a -flag — a stored `healthy` boolean goes stale silently the moment nothing -arrives, which is the failure this is designed against. Each row also -carries `last_seen_unix` and `age_seconds`, so a consumer that disagrees -with `staleAfterSeconds` can apply its own threshold. +Every row also carries `last_seen_unix` and `age_seconds` if you want to +apply your own threshold. The timestamp is the one the queue recorded on +arrival, not one the hive put in its own payload. -`last_seen_unix` is the **bucket's** timestamp, applied by the NATS -server when the value landed, not a field inside the payload — a -publisher cannot make itself look fresher than it is, and a hive with a -wrong clock skews its own payload rather than its freshness. +Set `services.hyperhive.swarm.controller.staleAfterSeconds` (default +`120`) **above the rate hives publish at**, or everything reads `stale` +between reports. It takes effect on the next request; nothing has to +re-publish. -Because the bucket outlives a controller restart, a restarted controller -reports what it reads: `stale, age_seconds: 10800` rather than -`never_reported`. That is the more honest of the two — it genuinely -knows when the hive last spoke. Losing the bucket degrades in the same -direction: every hive reads `never_reported` until its next publish, -which is the true answer and not a remembered "healthy". - -The endpoint answers **503**, not an empty list, when no queue is -configured on this host or its store cannot be read. "I cannot reach the -store" and "every hive is silent" are different answers, and rendering -the second when the first is true would turn a local fault into an -apparent swarm-wide outage. - -⚠️ **Nothing writes to the bucket yet.** The transport and the -controller's read path are in place; the hive-side publisher is a later -slice. Until one lands, every hive reads `never_reported` — the correct -answer for a controller that has been told nothing. +The endpoint answers **503** when this host has no swarm queue +configured, or has one and cannot read it — deliberately not an empty +list, which would look like a silent swarm rather than a controller that +cannot see. The body says which. Status survives a controller restart: +it is stored in the queue, not in the daemon. ## Cross-references