feat(swarm-controller): aggregate per-hive status from the swarm queue
The controller connects to the swarm queue as its own client and serves what each hive last said about itself at GET /api/hives/status. THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream KV bucket (history 1) and the controller reads it per request, keeping no copy. A cache here would be a second answer to the same question, free to disagree with the first, and the disagreement surfaces as a hive reading healthy on a dashboard while the bucket says otherwise. Whichever side arrives first creates the bucket; both want the same shape. Rows come from the roster rather than from the bucket, so an empty bucket renders as a swarm nobody has heard from instead of a healthy one, and `never_reported` stays distinct from `stale` - went quiet is a fault, never spoke is usually a deployment that has not happened. Freshness is derived at read time and never stored as a flag, because a stored `healthy` boolean goes stale silently the moment nothing arrives, which is the failure this endpoint is designed against. The timestamp is the NATS server's, applied when the value landed, so a publisher cannot make itself look fresher than it is. Authentication is per connection attempt, not per process. Authelia issues `client_credentials` tokens that expire in 3599s, and auth happens at CONNECT, so a long-lived connection is fine but a reconnect an hour later needs a token minted an hour later. `with_auth_callback` is re-run by async-nats for each attempt, which handles expiry by construction rather than by a timer - the alternative fails in the way this subsystem exists to prevent, with the controller still serving while its data quietly stops updating. Three failure shapes are deliberate: - A half-set environment is fatal; an absent one is not. Silently behaving like an unconfigured host is how every hive ends up reading `never_reported` with nothing to point at. - The endpoint answers 503 rather than an empty list when the store cannot be read. "I cannot reach the store" and "every hive is silent" are different answers, and rendering the second turns a local fault into an apparent swarm-wide outage. - `retry_on_initial_connect` makes the daemon and the queue bootable in either order, and the status handler refuses when the client is not Connected rather than issuing a request into it - a request made in that window does not fail, it waits, so every poll would hang and learn nothing. `Pending` is the state a never-connected client is in, which is why the test is `!= Connected` and not `== Disconnected`. The rendering rules are a pure function over a map, so the semantics are tested against a table rather than against a running server. The KV read, the credential rotation and the 503 paths are covered behaviourally instead: a real NATS server with a rotating token endpoint, asserting that the controller recovers only when the credential rotates, and mutation-tested by holding the credential wrong for the same window.
This commit is contained in:
parent
b3f46e4f43
commit
8891b46943
7 changed files with 1030 additions and 17 deletions
25
Cargo.lock
generated
25
Cargo.lock
generated
|
|
@ -213,14 +213,17 @@ dependencies = [
|
|||
"rustls-webpki",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"serde_nanos",
|
||||
"serde_repr",
|
||||
"thiserror 2.0.18",
|
||||
"time",
|
||||
"tokio",
|
||||
"tokio-rustls",
|
||||
"tokio-stream",
|
||||
"tokio-util",
|
||||
"tokio-websockets",
|
||||
"tracing",
|
||||
"tryhard",
|
||||
"url",
|
||||
]
|
||||
|
||||
|
|
@ -4310,6 +4313,15 @@ dependencies = [
|
|||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_nanos"
|
||||
version = "0.1.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a93142f0367a4cc53ae0fead1bcda39e85beccfad3dcd717656cacab94b12985"
|
||||
dependencies = [
|
||||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_path_to_error"
|
||||
version = "0.1.20"
|
||||
|
|
@ -4546,7 +4558,10 @@ name = "swarm-controller"
|
|||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-nats",
|
||||
"axum",
|
||||
"futures-util",
|
||||
"reqwest 0.13.1",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
|
|
@ -5070,6 +5085,16 @@ version = "0.2.5"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
|
||||
|
||||
[[package]]
|
||||
name = "tryhard"
|
||||
version = "0.5.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9fe58ebd5edd976e0fe0f8a14d2a04b7c81ef153ea9a54eebc42e67c2c23b4e5"
|
||||
dependencies = [
|
||||
"pin-project-lite",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tungstenite"
|
||||
version = "0.29.0"
|
||||
|
|
|
|||
Loading…
Reference in a new issue