feat(swarm-controller): aggregate per-hive status from the swarm queue

The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.

THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.

Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.

Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.

Three failure shapes are deliberate:

- A half-set environment is fatal; an absent one is not. Silently
  behaving like an unconfigured host is how every hive ends up reading
  `never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
  cannot be read. "I cannot reach the store" and "every hive is silent"
  are different answers, and rendering the second turns a local fault
  into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
  either order, and the status handler refuses when the client is not
  Connected rather than issuing a request into it - a request made in
  that window does not fail, it waits, so every poll would hang and
  learn nothing. `Pending` is the state a never-connected client is in,
  which is why the test is `!= Connected` and not `== Disconnected`.

The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.

swarm-controller/src/main.rs

@@ -31,6 +31,9 @@ use serde::{Deserialize, Serialize};
 use utoipa::{OpenApi, ToSchema};
 use utoipa_axum::{router::OpenApiRouter, routes};
 
+mod queue;
+mod status;
+
 /// Where the daemon binds, overridable via `SWARM_CONTROLLER_SOCKET`.
 ///
 /// A compiled-in default is legitimate here and is *not* the mistake that
@@ -101,6 +104,11 @@ struct AppState {
     /// Loaded once at startup (`load_links`); same synchronization story
     /// as `hives`.
     links: Arc<Vec<ServiceLink>>,
+    /// `None` when this deployment wired up no swarm queue — the only
+    /// state in which `/api/hives/status` cannot answer at all. A queue
+    /// that is merely *unreachable* still yields a reader, because
+    /// `async-nats` reconnects underneath it.
+    status: Option<Arc<status::StatusReader>>,
 }
 
 /// Env var the controller's NixOS module sets from
@@ -192,6 +200,59 @@ async fn get_links(State(state): State<AppState>) -> Json<Vec<ServiceLink>> {
     Json((*state.links).clone())
 }
 
+/// Why the status route answers 503 rather than an empty list.
+///
+/// "I cannot reach the store" and "every hive is silent" are different
+/// answers, and rendering the second when the first is true is exactly
+/// the smoothing this endpoint exists to avoid — a caller would draw a
+/// swarm-wide outage out of a local one. The cause is carried in the
+/// body because a bare 503 on an operator-facing diagnostic is how a
+/// misconfiguration costs an afternoon; it is a queue/JetStream error
+/// string, and this surface is already behind the swarm's SSO.
+struct StatusUnavailable(String);
+
+impl axum::response::IntoResponse for StatusUnavailable {
+    fn into_response(self) -> axum::response::Response {
+        (axum::http::StatusCode::SERVICE_UNAVAILABLE, self.0).into_response()
+    }
+}
+
+/// What each hive last said about itself, read from the swarm queue at
+/// request time.
+///
+/// Every hive in the roster gets a row whether or not it has ever
+/// reported — see the `status` module for why absence, not presence, is
+/// the case this is built around.
+#[utoipa::path(
+    get,
+    path = "/api/hives/status",
+    responses(
+        (status = 200, description = "a row per hive, freshness derived now", body = Vec<status::HiveStatus>),
+        (status = 503, description = "no swarm queue is configured here, or its store could not be read", body = String),
+    ),
+    tag = "hives"
+)]
+async fn get_hives_status(
+    State(state): State<AppState>,
+) -> Result<Json<Vec<status::HiveStatus>>, StatusUnavailable> {
+    let Some(reader) = state.status.as_ref() else {
+        return Err(StatusUnavailable(
+            "no swarm queue is configured on this host".to_owned(),
+        ));
+    };
+    match reader
+        .view(&state.hives, std::time::SystemTime::now())
+        .await
+    {
+        Ok(rows) => Ok(Json(rows)),
+        Err(e) => {
+            let detail = format!("{e:#}");
+            tracing::warn!(error = %detail, "reading the swarm status bucket failed");
+            Err(StatusUnavailable(detail))
+        }
+    }
+}
+
 #[tokio::main]
 async fn main() -> Result<()> {
     tracing_subscriber::fmt()
@@ -230,14 +291,44 @@ async fn main() -> Result<()> {
         .with_context(|| format!("chmod {}", path.display()))?;
     tracing::info!(socket = %path.display(), "swarm-controller listening");
 
+    // Connect to the swarm queue when this deployment wired one up.
+    //
+    // Deliberately NOT fatal on failure: the controller's HTTP surface is
+    // useful without the queue, and a hive that cannot be read from renders
+    // as `unknown` rather than as an outage of this daemon. What IS fatal is
+    // a half-set environment — `QueueConfig::from_env` refuses that, because
+    // silently behaving like an unconfigured host is how every hive ends up
+    // reading `never_reported` with nothing to point at.
+    let status = match queue::QueueConfig::from_env()? {
+        None => {
+            tracing::info!("no swarm queue configured; status aggregation is off");
+            None
+        }
+        Some(cfg) => match queue::connect(cfg).await {
+            Ok(client) => {
+                tracing::info!("connected to the swarm queue");
+                Some(Arc::new(status::StatusReader::new(
+                    client,
+                    status::StatusReader::stale_after_from_env(),
+                )))
+            }
+            Err(e) => {
+                tracing::warn!(error = format!("{e:#}"), "swarm queue unreachable");
+                None
+            }
+        },
+    };
+
     let state = AppState {
         hives: Arc::new(load_hives()),
         links: Arc::new(load_links()),
+        status,
     };
 
     let (router, api) = OpenApiRouter::<AppState>::with_openapi(ApiDoc::openapi())
         .routes(routes!(health))
         .routes(routes!(get_hives))
+        .routes(routes!(get_hives_status))
         .routes(routes!(get_links))
         .split_for_parts();
     // Just the JSON, not the UI — Swagger UI itself is nginx-hosted from