Watch
0
0
Fork
You've already forked hyperhive
0

swarm-controller: an opt-in plain /healthz and /readyz listener a kubelet can probe

This commit is contained in:
maia 2026-11-19 09:43:02 +01:00 • committed by mara
commit ffbd2f4dea
2 changed files with 147 additions and 50 deletions

View file

@ -270,37 +270,40 @@ spells out. The settings, limits and reasons:
The queue daemons (`swarm-stt`, `swarm-review-classifier`,
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`),
`swarm-nats-auth` and the three auth responders (`swarm-logs-auth`,
`swarm-metrics-auth`, `swarm-authelia-bridge`) can serve two probe endpoints.
They answer 200 or 503 and send no body.
`swarm-nats-auth`, `swarm-controller` and the three auth responders
(`swarm-logs-auth`, `swarm-metrics-auth`, `swarm-authelia-bridge`) can serve two
probe endpoints. They answer 200 or 503 and send no body.
- `GET /healthz` is 200 while the daemon's main loop is running. The loop
beats as it takes messages, so a loop that stops polling turns the answer to 503
after 30 seconds (5 minutes for `swarm-notify-router`, whose loop handles a
batch of events between beats). Before the loop's first beat the answer is
200: waiting on the secret store at startup doesn't count as stuck. A
daemon that only serves requests (the three auth responders) beats from a task
on its runtime, so `/healthz` shows the runtime still runs tasks, not that
its accept loop is healthy.
daemon that only serves requests (`swarm-controller` and the three auth
responders) beats from a task on its runtime, so `/healthz` shows the runtime
still runs tasks, not that its accept loop is healthy.
- `GET /readyz` is 200 while the daemon has what it needs to work: its queue
connection works (`swarm-stt`, `swarm-review-classifier`,
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`,
`swarm-nats-auth`), or what it answers from is there: `swarm-logs-auth` and
`swarm-metrics-auth` are ready while the secret store answers,
`swarm-authelia-bridge` while its users file loads. The mechanics are in
`swarm-authelia-bridge` while its users file loads, and `swarm-controller`
while both its queue connection works and the store answers. The mechanics are in
[Readiness checks](../../4-implementation/swarm/readiness.md).
Nothing listens by default. Set `SWARM_STT_HEALTH_ADDR`,
`SWARM_REVIEW_CLASSIFIER_HEALTH_ADDR`, `SWARM_NOTIFY_ROUTER_HEALTH_ADDR`,
`SWARM_BUILDER_HEALTH_ADDR`, `SWARM_CLAUDE_LOGIN_HEALTH_ADDR` or
`SWARM_AUTHELIA_BRIDGE_HEALTH_ADDR` to a `host:port` to bind the listener. For
`SWARM_AUTHELIA_BRIDGE_HEALTH_ADDR` or `SWARM_CONTROLLER_HEALTH_ADDR` to a
`host:port` to bind the listener. For
`swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth`, pass
`--health-addr`, or set `SWARM_NATS_AUTH_HEALTH_ADDR`,
`SWARM_LOGS_AUTH_HEALTH_ADDR` or `SWARM_METRICS_AUTH_HEALTH_ADDR`; the flag
wins. No NixOS module sets one. The listener stops on the same SIGTERM as the
daemon.
`swarm-controller` serves `GET /readyz` on its existing socket, next to
`/health`. It answers 200 while the queue connection works and the secret store
answers the controller's login, and 503 otherwise (see
`swarm-controller` also serves `GET /readyz` on its existing socket, next to
`/health`, with or without the listener. The socket and the listener give the
same answer: 200 while the queue connection works and the secret store answers
the controller's login, and 503 otherwise (see
[Readiness checks](../../4-implementation/swarm/readiness.md)).

View file

@ -29,4 +29,8 @@ ends when the last clone of the check drops.
The controller starts its check once, when it starts serving
(`routes::watch_store`), and `/readyz` reads the last result. Before that, and
in tests that don't start it, the store reads as not ready.
in tests that don't start it, the store reads as not ready. The socket's
`/readyz` and the plain listener's (`SWARM_CONTROLLER_HEALTH_ADDR`) both ask
`routes::is_ready`, so they can't disagree. The listener binds at the top of
`main`, before the store is reachable, so a probe gets a 503 and not a refused
connection while the controller starts.