Watch
0
0
Fork
You've already forked hyperhive
0

swarm-controller: an opt-in plain /healthz and /readyz listener a kubelet can probe

This commit is contained in:
maia 2026-11-19 09:43:02 +01:00 • committed by mara
commit ffbd2f4dea
4 changed files with 147 additions and 50 deletions

View file

@ -270,37 +270,40 @@ spells out. The settings, limits and reasons:
The queue daemons (`swarm-stt`, `swarm-review-classifier`, The queue daemons (`swarm-stt`, `swarm-review-classifier`,
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`), `swarm-notify-router`, `swarm-builder`, `swarm-claude-login`),
`swarm-nats-auth` and the three auth responders (`swarm-logs-auth`, `swarm-nats-auth`, `swarm-controller` and the three auth responders
`swarm-metrics-auth`, `swarm-authelia-bridge`) can serve two probe endpoints. (`swarm-logs-auth`, `swarm-metrics-auth`, `swarm-authelia-bridge`) can serve two
They answer 200 or 503 and send no body. probe endpoints. They answer 200 or 503 and send no body.
- `GET /healthz` is 200 while the daemon's main loop is running. The loop - `GET /healthz` is 200 while the daemon's main loop is running. The loop
beats as it takes messages, so a loop that stops polling turns the answer to 503 beats as it takes messages, so a loop that stops polling turns the answer to 503
after 30 seconds (5 minutes for `swarm-notify-router`, whose loop handles a after 30 seconds (5 minutes for `swarm-notify-router`, whose loop handles a
batch of events between beats). Before the loop's first beat the answer is batch of events between beats). Before the loop's first beat the answer is
200: waiting on the secret store at startup doesn't count as stuck. A 200: waiting on the secret store at startup doesn't count as stuck. A
daemon that only serves requests (the three auth responders) beats from a task daemon that only serves requests (`swarm-controller` and the three auth
on its runtime, so `/healthz` shows the runtime still runs tasks, not that responders) beats from a task on its runtime, so `/healthz` shows the runtime
its accept loop is healthy. still runs tasks, not that its accept loop is healthy.
- `GET /readyz` is 200 while the daemon has what it needs to work: its queue - `GET /readyz` is 200 while the daemon has what it needs to work: its queue
connection works (`swarm-stt`, `swarm-review-classifier`, connection works (`swarm-stt`, `swarm-review-classifier`,
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`, `swarm-notify-router`, `swarm-builder`, `swarm-claude-login`,
`swarm-nats-auth`), or what it answers from is there: `swarm-logs-auth` and `swarm-nats-auth`), or what it answers from is there: `swarm-logs-auth` and
`swarm-metrics-auth` are ready while the secret store answers, `swarm-metrics-auth` are ready while the secret store answers,
`swarm-authelia-bridge` while its users file loads. The mechanics are in `swarm-authelia-bridge` while its users file loads, and `swarm-controller`
while both its queue connection works and the store answers. The mechanics are in
[Readiness checks](../../4-implementation/swarm/readiness.md). [Readiness checks](../../4-implementation/swarm/readiness.md).
Nothing listens by default. Set `SWARM_STT_HEALTH_ADDR`, Nothing listens by default. Set `SWARM_STT_HEALTH_ADDR`,
`SWARM_REVIEW_CLASSIFIER_HEALTH_ADDR`, `SWARM_NOTIFY_ROUTER_HEALTH_ADDR`, `SWARM_REVIEW_CLASSIFIER_HEALTH_ADDR`, `SWARM_NOTIFY_ROUTER_HEALTH_ADDR`,
`SWARM_BUILDER_HEALTH_ADDR`, `SWARM_CLAUDE_LOGIN_HEALTH_ADDR` or `SWARM_BUILDER_HEALTH_ADDR`, `SWARM_CLAUDE_LOGIN_HEALTH_ADDR` or
`SWARM_AUTHELIA_BRIDGE_HEALTH_ADDR` to a `host:port` to bind the listener. For `SWARM_AUTHELIA_BRIDGE_HEALTH_ADDR` or `SWARM_CONTROLLER_HEALTH_ADDR` to a
`host:port` to bind the listener. For
`swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth`, pass `swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth`, pass
`--health-addr`, or set `SWARM_NATS_AUTH_HEALTH_ADDR`, `--health-addr`, or set `SWARM_NATS_AUTH_HEALTH_ADDR`,
`SWARM_LOGS_AUTH_HEALTH_ADDR` or `SWARM_METRICS_AUTH_HEALTH_ADDR`; the flag `SWARM_LOGS_AUTH_HEALTH_ADDR` or `SWARM_METRICS_AUTH_HEALTH_ADDR`; the flag
wins. No NixOS module sets one. The listener stops on the same SIGTERM as the wins. No NixOS module sets one. The listener stops on the same SIGTERM as the
daemon. daemon.
`swarm-controller` serves `GET /readyz` on its existing socket, next to `swarm-controller` also serves `GET /readyz` on its existing socket, next to
`/health`. It answers 200 while the queue connection works and the secret store `/health`, with or without the listener. The socket and the listener give the
answers the controller's login, and 503 otherwise (see same answer: 200 while the queue connection works and the secret store answers
the controller's login, and 503 otherwise (see
[Readiness checks](../../4-implementation/swarm/readiness.md)). [Readiness checks](../../4-implementation/swarm/readiness.md)).

View file

@ -29,4 +29,8 @@ ends when the last clone of the check drops.
The controller starts its check once, when it starts serving The controller starts its check once, when it starts serving
(`routes::watch_store`), and `/readyz` reads the last result. Before that, and (`routes::watch_store`), and `/readyz` reads the last result. Before that, and
in tests that don't start it, the store reads as not ready. in tests that don't start it, the store reads as not ready. The socket's
`/readyz` and the plain listener's (`SWARM_CONTROLLER_HEALTH_ADDR`) both ask
`routes::is_ready`, so they can't disagree. The listener binds at the top of
`main`, before the store is reachable, so a probe gets a 503 and not a refused
connection while the controller starts.

View file

@ -110,8 +110,7 @@ use startup::{
#[tokio::main] #[tokio::main]
async fn main() -> Result<()> { async fn main() -> Result<()> {
hive_log::init(); let (stop, health) = start_process().await?;
let stop = install_process_hooks();
let path = socket_path(); let path = socket_path();
@ -269,15 +268,7 @@ async fn main() -> Result<()> {
reviews: Arc::default(), reviews: Arc::default(),
alerts: alerts::from_env(), alerts: alerts::from_env(),
}; };
serve(listener, state, stop).await serve(listener, state, stop, health).await
}
/// The TLS provider, and the stop signal. The signal is installed first thing,
/// so a stop that arrives while the startup reads wait on a store that is down
/// ends them instead of being ignored until `serve`.
fn install_process_hooks() -> swarm_secret_client::Shutdown {
swarm_secret_client::install_crypto_provider();
swarm_secret_client::shutdown_signal()
} }
/// The value a startup step produced, or exit 0 when the stop signal ended the /// The value a startup step produced, or exit 0 when the stop signal ended the
@ -286,6 +277,32 @@ fn exit_if_stopped<T>(started: Option<T>) -> T {
started.unwrap_or_else(|| std::process::exit(0)) started.unwrap_or_else(|| std::process::exit(0))
} }
/// The environment variable that names the plain probe listener's address.
const HEALTH_ADDR_VAR: &str = "SWARM_CONTROLLER_HEALTH_ADDR";
/// What `main` sets up before anything else: logging, the TLS provider, the
/// stop signal (installed first thing, so a stop that arrives while the startup
/// reads wait on a store that is down ends them instead of being ignored until
/// `serve`), and the opt-in `/healthz` and `/readyz` listener on the address
/// [`HEALTH_ADDR_VAR`] names, which a kubelet can reach where the socket's
/// `/readyz` is out of its way. Nothing binds when the variable is unset. The
/// controller is a server with no loop of its own to beat, so the beat comes
/// from a task on its runtime. `/readyz` stays not ready until [`serve`] sets
/// its condition.
async fn start_process() -> Result<(
swarm_secret_client::Shutdown,
swarm_secret_client::health::Health,
)> {
hive_log::init();
swarm_secret_client::install_crypto_provider();
let stop = swarm_secret_client::shutdown_signal();
let health = swarm_secret_client::health::Health::from_env(HEALTH_ADDR_VAR, stop.clone())
.await
.context("starting the health listener")?;
health.beat_on_runtime();
Ok((stop, health))
}
/// Registers the `bao.client_cert.*` gauges for the controller's own store /// Registers the `bao.client_cert.*` gauges for the controller's own store
/// certificate (`controller.pem`) on the jobq exporter, the one provider built /// certificate (`controller.pem`) on the jobq exporter, the one provider built
/// at startup rather than on first use. Nothing when no exporter is configured. /// at startup rather than on first use. Nothing when no exporter is configured.
@ -302,13 +319,15 @@ fn register_cert_gauges(provider: Option<&opentelemetry_sdk::metrics::SdkMeterPr
/// Start taking the queued review requests and answering the notification /// Start taking the queued review requests and answering the notification
/// router's, then serve `state` on `listener` until `stop` resolves. The /// router's, then serve `state` on `listener` until `stop` resolves. The
/// requests in hand finish for at most the drain; the review queue takes no /// requests in hand finish for at most the drain; the review queue takes no
/// request after `stop`. /// request after `stop`. `health` answers `/readyz` as the socket's does.
async fn serve( async fn serve(
listener: tokio::net::UnixListener, listener: tokio::net::UnixListener,
state: AppState, state: AppState,
stop: impl Future<Output = ()> + Clone + Send + 'static, stop: impl Future<Output = ()> + Clone + Send + 'static,
health: swarm_secret_client::health::Health,
) -> Result<()> { ) -> Result<()> {
routes::watch_store(); let store = routes::watch_store();
health.ready_when(routes::probe_condition(state.status.clone(), store));
review_classifier::review::spawn_queue(&state, stop.clone()); review_classifier::review::spawn_queue(&state, stop.clone());
request_reviewer::spawn(&state); request_reviewer::spawn(&state);
let serving = axum::serve(listener, build_app(state)).with_graceful_shutdown(stop.clone()); let serving = axum::serve(listener, build_app(state)).with_graceful_shutdown(stop.clone());

View file

@ -11,6 +11,7 @@ use std::sync::{Arc, Mutex, OnceLock};
use axum::extract::State; use axum::extract::State;
use axum::http::StatusCode; use axum::http::StatusCode;
use swarm_secret_client::health::CachedCheck;
use crate::jobq::{Late, SwarmNodeKind, SwarmResourceKind}; use crate::jobq::{Late, SwarmNodeKind, SwarmResourceKind};
use crate::routes::hives::{HiveEntry, ServiceLink}; use crate::routes::hives::{HiveEntry, ServiceLink};
@ -38,38 +39,49 @@ pub(crate) async fn health() -> &'static str {
/// ///
/// Not in the `OpenAPI` document: it is for the supervisor, not for API clients. /// Not in the `OpenAPI` document: it is for the supervisor, not for API clients.
pub(crate) async fn readyz(State(state): State<AppState>) -> StatusCode { pub(crate) async fn readyz(State(state): State<AppState>) -> StatusCode {
let queue_connected = state if is_ready(state.status.as_deref(), STORE_READY.get()) {
.status
.as_ref()
.is_some_and(|status| swarm_queue_client::ensure_connected(&status.queue_client()).is_ok());
let store_fresh = STORE_READY
.get()
.is_some_and(swarm_secret_client::health::CachedCheck::is_fresh);
ready(queue_connected, store_fresh)
}
/// The controller's own store login, tried every 30 s once [`watch_store`] has
/// started it. Process-wide, as the controller has one store login to ask about.
static STORE_READY: OnceLock<swarm_secret_client::health::CachedCheck> = OnceLock::new();
/// Starts the store login `/readyz` reads. Called once, where serving starts.
pub(crate) fn watch_store() {
STORE_READY.get_or_init(|| {
swarm_secret_client::health::CachedCheck::every_30s(|| async {
crate::store::connect().await.map(drop)
})
});
}
/// 200 when the queue is connected and the store's last login is fresh.
fn ready(queue_connected: bool, store_fresh: bool) -> StatusCode {
if queue_connected && store_fresh {
StatusCode::OK StatusCode::OK
} else { } else {
StatusCode::SERVICE_UNAVAILABLE StatusCode::SERVICE_UNAVAILABLE
} }
} }
/// Whether the controller has what it needs to work: its queue connection and a
/// fresh store login. The one answer behind `/readyz` on the socket and on the
/// opt-in plain listener (`SWARM_CONTROLLER_HEALTH_ADDR`). Reads memory only.
pub(crate) fn is_ready(status: Option<&status::StatusReader>, store: Option<&CachedCheck>) -> bool {
let queue_connected = status
.is_some_and(|status| swarm_queue_client::ensure_connected(&status.queue_client()).is_ok());
ready(queue_connected, store.is_some_and(CachedCheck::is_fresh))
}
/// What the plain listener's `/readyz` asks: [`is_ready`], as the socket's does.
pub(crate) fn probe_condition(
status: Option<Arc<status::StatusReader>>,
store: CachedCheck,
) -> impl Fn() -> bool + Send + Sync + 'static {
move || is_ready(status.as_deref(), Some(&store))
}
/// The controller's own store login, tried every 30 s once [`watch_store`] has
/// started it. Process-wide, as the controller has one store login to ask about.
static STORE_READY: OnceLock<CachedCheck> = OnceLock::new();
/// Starts the store login `/readyz` reads, and returns it for the plain
/// listener to read too. Called once, where serving starts.
pub(crate) fn watch_store() -> CachedCheck {
STORE_READY
.get_or_init(|| {
CachedCheck::every_30s(|| async { crate::store::connect().await.map(drop) })
})
.clone()
}
/// Ready when the queue is connected and the store's last login is fresh.
fn ready(queue_connected: bool, store_fresh: bool) -> bool {
queue_connected && store_fresh
}
#[derive(Clone)] #[derive(Clone)]
pub(crate) struct AppState { pub(crate) struct AppState {
/// Loaded once at startup (`load_hives`); never mutated, so an /// Loaded once at startup (`load_hives`); never mutated, so an
@ -249,10 +261,69 @@ pub(crate) mod tests {
/// `/readyz` needs both the queue and a fresh store login. /// `/readyz` needs both the queue and a fresh store login.
#[test] #[test]
fn ready_needs_the_queue_and_the_store() { fn ready_needs_the_queue_and_the_store() {
assert_eq!(ready(true, true), StatusCode::OK); assert!(ready(true, true));
assert_eq!(ready(true, false), StatusCode::SERVICE_UNAVAILABLE); assert!(!ready(true, false));
assert_eq!(ready(false, true), StatusCode::SERVICE_UNAVAILABLE); assert!(!ready(false, true));
assert_eq!(ready(false, false), StatusCode::SERVICE_UNAVAILABLE); assert!(!ready(false, false));
}
/// A store check that has succeeded, so it reads fresh.
async fn fresh_store() -> CachedCheck {
let check = CachedCheck::every_30s(|| std::future::ready(Ok::<(), String>(())));
while !check.is_fresh() {
tokio::task::yield_now().await;
}
check
}
/// A store check that never succeeds, as a store that is down.
fn failing_store() -> CachedCheck {
CachedCheck::every_30s(|| std::future::ready(Err::<(), String>("store is down".into())))
}
/// Readiness is a connected queue *and* a fresh store login, whichever of
/// the two is missing.
#[tokio::test]
async fn is_ready_needs_a_connected_queue_and_a_fresh_store() {
let (_nats, state, _sched) = state_with_roster_queued().await;
let queue = state.status.as_deref();
let (fresh, failing) = (fresh_store().await, failing_store());
assert!(is_ready(queue, Some(&fresh)));
assert!(!is_ready(queue, Some(&failing)), "store login failing");
assert!(!is_ready(queue, None), "no store check started");
assert!(!is_ready(None, Some(&fresh)), "no queue wired up");
}
/// The plain listener's `/readyz` follows the same answer as the socket's,
/// and its `/healthz` is 200 whatever readiness says.
#[tokio::test]
async fn the_plain_listener_answers_readyz_like_the_socket() {
use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _};
let (_nats, state, _sched) = state_with_roster_queued().await;
let cases = [
(state.status.clone(), fresh_store().await, 200, "ready"),
(state.status.clone(), failing_store(), 503, "store failing"),
(None, fresh_store().await, 503, "no queue"),
];
for (status, store, expected, why) in cases {
let health =
swarm_secret_client::health::Health::new(std::time::Duration::from_mins(1));
health.ready_when(probe_condition(status, store));
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(health.serve(listener, std::future::pending()));
for (path, want) in [("/readyz", expected), ("/healthz", 200)] {
let mut conn = tokio::net::TcpStream::connect(addr).await.unwrap();
let request =
format!("GET {path} HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n");
conn.write_all(request.as_bytes()).await.unwrap();
let mut reply = String::new();
conn.read_to_string(&mut reply).await.unwrap();
let code = reply.split_whitespace().nth(1).unwrap_or_default();
assert_eq!(code, want.to_string(), "{path} ({why}): {reply:?}");
}
}
} }
/// A probe makes no store call: the login runs once at start and then every /// A probe makes no store call: the login runs once at start and then every