feat(swarm-controller): aggregate per-hive status from the swarm queue
The controller connects to the swarm queue as its own client and serves what each hive last said about itself at GET /api/hives/status. THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream KV bucket (history 1) and the controller reads it per request, keeping no copy. A cache here would be a second answer to the same question, free to disagree with the first, and the disagreement surfaces as a hive reading healthy on a dashboard while the bucket says otherwise. Whichever side arrives first creates the bucket; both want the same shape. Rows come from the roster rather than from the bucket, so an empty bucket renders as a swarm nobody has heard from instead of a healthy one, and `never_reported` stays distinct from `stale` - went quiet is a fault, never spoke is usually a deployment that has not happened. Freshness is derived at read time and never stored as a flag, because a stored `healthy` boolean goes stale silently the moment nothing arrives, which is the failure this endpoint is designed against. The timestamp is the NATS server's, applied when the value landed, so a publisher cannot make itself look fresher than it is. Authentication is per connection attempt, not per process. Authelia issues `client_credentials` tokens that expire in 3599s, and auth happens at CONNECT, so a long-lived connection is fine but a reconnect an hour later needs a token minted an hour later. `with_auth_callback` is re-run by async-nats for each attempt, which handles expiry by construction rather than by a timer - the alternative fails in the way this subsystem exists to prevent, with the controller still serving while its data quietly stops updating. Three failure shapes are deliberate: - A half-set environment is fatal; an absent one is not. Silently behaving like an unconfigured host is how every hive ends up reading `never_reported` with nothing to point at. - The endpoint answers 503 rather than an empty list when the store cannot be read. "I cannot reach the store" and "every hive is silent" are different answers, and rendering the second turns a local fault into an apparent swarm-wide outage. - `retry_on_initial_connect` makes the daemon and the queue bootable in either order, and the status handler refuses when the client is not Connected rather than issuing a request into it - a request made in that window does not fail, it waits, so every poll would hang and learn nothing. `Pending` is the state a never-connected client is in, which is why the test is `!= Connected` and not `== Disconnected`. The rendering rules are a pure function over a map, so the semantics are tested against a table rather than against a running server. The KV read, the credential rotation and the 503 paths are covered behaviourally instead: a real NATS server with a rotating token endpoint, asserting that the controller recovers only when the credential rotates, and mutation-tested by holding the credential wrong for the same window.
This commit is contained in:
atlas
parent
b3f46e4f43
commit
8891b46943
7 changed files with 1030 additions and 17 deletions
197
swarm-controller/src/queue.rs
Normal file
197
swarm-controller/src/queue.rs
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
//! The controller's client end of the swarm message queue.
|
||||
//!
|
||||
//! The queue admits every non-responder client through `auth_callout`: a
|
||||
//! client presents a token at CONNECT, the callout responder introspects it
|
||||
//! against authelia and mints a user JWT if it is good. So the controller is
|
||||
//! an ordinary client and needs an identity of its own — it is not a hive, and
|
||||
//! the per-hive clients issued from the roster are not its to use.
|
||||
//!
|
||||
//! Two things about that shape drive everything here:
|
||||
//!
|
||||
//! - **A token expires.** Authelia issues `client_credentials` access tokens
|
||||
//! with `expires_in: 3599`. Authentication happens at CONNECT, so a
|
||||
//! long-lived connection is fine — but a *reconnect* an hour later needs a
|
||||
//! token that was minted an hour later.
|
||||
//! - **`async-nats` re-runs an auth callback per connection attempt** (it is
|
||||
//! handed that attempt's nonce). So the refresh belongs in the callback and
|
||||
//! not in a timer: there is no window in which the client holds a token it
|
||||
//! minted for a previous connection.
|
||||
//!
|
||||
//! The alternative — mint once, pass a static `auth_token`, own the reconnect
|
||||
//! loop — fails in the way this subsystem exists to prevent: the controller
|
||||
//! keeps serving, its status data quietly stops updating, and nothing says so
|
||||
//! until someone reads a dashboard.
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use anyhow::{Context, Result, bail};
|
||||
|
||||
/// Only the one field this needs; authelia returns several.
|
||||
#[derive(serde::Deserialize)]
|
||||
struct TokenResponse {
|
||||
access_token: String,
|
||||
}
|
||||
|
||||
/// Where the controller finds the queue and what it authenticates with.
|
||||
///
|
||||
/// Every field comes from an environment variable the NixOS module sets, the
|
||||
/// same way `load_hives` takes the roster — a config change is a redeploy, and
|
||||
/// this process reads no file it was not pointed at.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct QueueConfig {
|
||||
/// `nats://host:port` for the swarm queue.
|
||||
pub url: String,
|
||||
/// Authelia's token endpoint, e.g. `https://auth.<swarm>/api/oidc/token`.
|
||||
pub token_endpoint: String,
|
||||
/// The controller's own `OAuth2` client id.
|
||||
pub client_id: String,
|
||||
/// File holding the client secret's PLAINTEXT.
|
||||
///
|
||||
/// A path and not a value: the secret is minted on the authelia host and
|
||||
/// read here, and putting it in the environment would publish it to
|
||||
/// anything that can read `/proc/<pid>/environ`.
|
||||
pub client_secret_file: PathBuf,
|
||||
}
|
||||
|
||||
impl QueueConfig {
|
||||
/// Read the config from the environment, or `None` when the queue was not
|
||||
/// wired up for this deployment.
|
||||
///
|
||||
/// `None` rather than an error on purpose: the controller serves its HTTP
|
||||
/// surface on hosts where the queue is not enabled, and refusing to start
|
||||
/// there would trade a missing feature for a dead daemon. What must NOT
|
||||
/// happen is a *half* configuration silently behaving like an absent one —
|
||||
/// hence the explicit partial check below.
|
||||
pub fn from_env() -> Result<Option<Self>> {
|
||||
let url = std::env::var("SWARM_CONTROLLER_NATS_URL").ok();
|
||||
let token_endpoint = std::env::var("SWARM_CONTROLLER_OIDC_TOKEN_ENDPOINT").ok();
|
||||
let client_id = std::env::var("SWARM_CONTROLLER_OIDC_CLIENT_ID").ok();
|
||||
let secret = std::env::var("SWARM_CONTROLLER_OIDC_CLIENT_SECRET_FILE").ok();
|
||||
|
||||
match (url, token_endpoint, client_id, secret) {
|
||||
(None, None, None, None) => Ok(None),
|
||||
(Some(url), Some(token_endpoint), Some(client_id), Some(secret)) => Ok(Some(Self {
|
||||
url,
|
||||
token_endpoint,
|
||||
client_id,
|
||||
client_secret_file: PathBuf::from(secret),
|
||||
})),
|
||||
// A partially-set environment is a deployment bug, and the failure
|
||||
// it would otherwise produce is the expensive kind: the controller
|
||||
// comes up "fine", never connects, and every hive reads as having
|
||||
// never reported. Naming the missing variables costs one line.
|
||||
_ => bail!(
|
||||
"swarm queue is half-configured: SWARM_CONTROLLER_NATS_URL, \
|
||||
_OIDC_TOKEN_ENDPOINT, _OIDC_CLIENT_ID and \
|
||||
_OIDC_CLIENT_SECRET_FILE must be set together or not at all"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Mint a fresh access token for the controller's own client.
|
||||
///
|
||||
/// `client_credentials`, because there is no user here: the controller
|
||||
/// authenticates as itself. Authelia refuses the `openid` scope for this grant
|
||||
/// (a machine client receives an access token and never an id-token), so no
|
||||
/// scope is requested.
|
||||
async fn mint_token(http: &reqwest::Client, cfg: &QueueConfig) -> Result<String> {
|
||||
// Read per call rather than caching: the file is small, and a cached
|
||||
// secret would survive a rotation that the operator believes took effect.
|
||||
let secret = tokio::fs::read_to_string(&cfg.client_secret_file)
|
||||
.await
|
||||
.with_context(|| {
|
||||
format!(
|
||||
"reading the queue client secret from {}",
|
||||
cfg.client_secret_file.display()
|
||||
)
|
||||
})?;
|
||||
|
||||
let response = http
|
||||
.post(&cfg.token_endpoint)
|
||||
.form(&[
|
||||
("grant_type", "client_credentials"),
|
||||
("client_id", cfg.client_id.as_str()),
|
||||
("client_secret", secret.trim()),
|
||||
])
|
||||
.send()
|
||||
.await
|
||||
.context("requesting an access token from authelia")?;
|
||||
|
||||
// The body carries authelia's own error description, and it is far more
|
||||
// useful than the status alone: a wrong grant says `unauthorized_client`,
|
||||
// a wrong secret says `invalid_client`, and those point at different
|
||||
// config.
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
if !status.is_success() {
|
||||
bail!("authelia refused the controller's token request ({status}): {body}");
|
||||
}
|
||||
|
||||
let parsed: TokenResponse =
|
||||
serde_json::from_str(&body).context("parsing authelia's token response")?;
|
||||
Ok(parsed.access_token)
|
||||
}
|
||||
|
||||
/// Connect to the swarm queue, minting a token for each connection attempt.
|
||||
pub async fn connect(cfg: QueueConfig) -> Result<async_nats::Client> {
|
||||
let http = reqwest::Client::new();
|
||||
let url = cfg.url.clone();
|
||||
|
||||
let client = async_nats::ConnectOptions::with_auth_callback(move |_nonce| {
|
||||
let http = http.clone();
|
||||
let cfg = cfg.clone();
|
||||
async move {
|
||||
let token = mint_token(&http, &cfg)
|
||||
.await
|
||||
// The callback's error type carries a string, so the context
|
||||
// chain would be lost; flatten it rather than dropping it.
|
||||
.map_err(|e| async_nats::AuthError::new(format!("{e:#}")))?;
|
||||
let mut auth = async_nats::Auth::new();
|
||||
auth.token = Some(token);
|
||||
Ok(auth)
|
||||
}
|
||||
})
|
||||
// The controller and the queue are separate units on (possibly)
|
||||
// separate hosts, and nothing orders them. Without this, a queue that
|
||||
// comes up one second later leaves the controller permanently
|
||||
// queue-less until someone restarts it — a boot-order race that
|
||||
// presents as "status has been unavailable since Tuesday".
|
||||
//
|
||||
// It also composes with the callback above rather than fighting it:
|
||||
// each background attempt is a connection attempt, so each one mints
|
||||
// its own token instead of retrying a stale one.
|
||||
.retry_on_initial_connect()
|
||||
.connect(&url)
|
||||
.await
|
||||
.with_context(|| format!("connecting to the swarm queue at {url}"))?;
|
||||
|
||||
Ok(client)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// The all-unset case is the common one — most hosts do not run the queue.
|
||||
#[test]
|
||||
fn an_absent_environment_is_not_an_error() {
|
||||
// Guard: this test would pass vacuously inside a configured
|
||||
// environment, so it asserts the variables really are unset first.
|
||||
for k in [
|
||||
"SWARM_CONTROLLER_NATS_URL",
|
||||
"SWARM_CONTROLLER_OIDC_TOKEN_ENDPOINT",
|
||||
"SWARM_CONTROLLER_OIDC_CLIENT_ID",
|
||||
"SWARM_CONTROLLER_OIDC_CLIENT_SECRET_FILE",
|
||||
] {
|
||||
if std::env::var(k).is_ok() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
QueueConfig::from_env()
|
||||
.expect("absent is not an error")
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue