swarm-controller: an opt-in plain /healthz and /readyz listener a kubelet can probe
This commit is contained in:
parent
c78d73ccd1
commit
ffbd2f4dea
2 changed files with 147 additions and 50 deletions
|
|
@ -270,37 +270,40 @@ spells out. The settings, limits and reasons:
|
|||
|
||||
The queue daemons (`swarm-stt`, `swarm-review-classifier`,
|
||||
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`),
|
||||
`swarm-nats-auth` and the three auth responders (`swarm-logs-auth`,
|
||||
`swarm-metrics-auth`, `swarm-authelia-bridge`) can serve two probe endpoints.
|
||||
They answer 200 or 503 and send no body.
|
||||
`swarm-nats-auth`, `swarm-controller` and the three auth responders
|
||||
(`swarm-logs-auth`, `swarm-metrics-auth`, `swarm-authelia-bridge`) can serve two
|
||||
probe endpoints. They answer 200 or 503 and send no body.
|
||||
|
||||
- `GET /healthz` is 200 while the daemon's main loop is running. The loop
|
||||
beats as it takes messages, so a loop that stops polling turns the answer to 503
|
||||
after 30 seconds (5 minutes for `swarm-notify-router`, whose loop handles a
|
||||
batch of events between beats). Before the loop's first beat the answer is
|
||||
200: waiting on the secret store at startup doesn't count as stuck. A
|
||||
daemon that only serves requests (the three auth responders) beats from a task
|
||||
on its runtime, so `/healthz` shows the runtime still runs tasks, not that
|
||||
its accept loop is healthy.
|
||||
daemon that only serves requests (`swarm-controller` and the three auth
|
||||
responders) beats from a task on its runtime, so `/healthz` shows the runtime
|
||||
still runs tasks, not that its accept loop is healthy.
|
||||
- `GET /readyz` is 200 while the daemon has what it needs to work: its queue
|
||||
connection works (`swarm-stt`, `swarm-review-classifier`,
|
||||
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`,
|
||||
`swarm-nats-auth`), or what it answers from is there: `swarm-logs-auth` and
|
||||
`swarm-metrics-auth` are ready while the secret store answers,
|
||||
`swarm-authelia-bridge` while its users file loads. The mechanics are in
|
||||
`swarm-authelia-bridge` while its users file loads, and `swarm-controller`
|
||||
while both its queue connection works and the store answers. The mechanics are in
|
||||
[Readiness checks](../../4-implementation/swarm/readiness.md).
|
||||
|
||||
Nothing listens by default. Set `SWARM_STT_HEALTH_ADDR`,
|
||||
`SWARM_REVIEW_CLASSIFIER_HEALTH_ADDR`, `SWARM_NOTIFY_ROUTER_HEALTH_ADDR`,
|
||||
`SWARM_BUILDER_HEALTH_ADDR`, `SWARM_CLAUDE_LOGIN_HEALTH_ADDR` or
|
||||
`SWARM_AUTHELIA_BRIDGE_HEALTH_ADDR` to a `host:port` to bind the listener. For
|
||||
`SWARM_AUTHELIA_BRIDGE_HEALTH_ADDR` or `SWARM_CONTROLLER_HEALTH_ADDR` to a
|
||||
`host:port` to bind the listener. For
|
||||
`swarm-nats-auth`, `swarm-logs-auth` and `swarm-metrics-auth`, pass
|
||||
`--health-addr`, or set `SWARM_NATS_AUTH_HEALTH_ADDR`,
|
||||
`SWARM_LOGS_AUTH_HEALTH_ADDR` or `SWARM_METRICS_AUTH_HEALTH_ADDR`; the flag
|
||||
wins. No NixOS module sets one. The listener stops on the same SIGTERM as the
|
||||
daemon.
|
||||
|
||||
`swarm-controller` serves `GET /readyz` on its existing socket, next to
|
||||
`/health`. It answers 200 while the queue connection works and the secret store
|
||||
answers the controller's login, and 503 otherwise (see
|
||||
`swarm-controller` also serves `GET /readyz` on its existing socket, next to
|
||||
`/health`, with or without the listener. The socket and the listener give the
|
||||
same answer: 200 while the queue connection works and the secret store answers
|
||||
the controller's login, and 503 otherwise (see
|
||||
[Readiness checks](../../4-implementation/swarm/readiness.md)).
|
||||
|
|
|
|||
|
|
@ -29,4 +29,8 @@ ends when the last clone of the check drops.
|
|||
|
||||
The controller starts its check once, when it starts serving
|
||||
(`routes::watch_store`), and `/readyz` reads the last result. Before that, and
|
||||
in tests that don't start it, the store reads as not ready.
|
||||
in tests that don't start it, the store reads as not ready. The socket's
|
||||
`/readyz` and the plain listener's (`SWARM_CONTROLLER_HEALTH_ADDR`) both ask
|
||||
`routes::is_ready`, so they can't disagree. The listener binds at the top of
|
||||
`main`, before the store is reachable, so a probe gets a 503 and not a refused
|
||||
connection while the controller starts.
|
||||
|
|
|
|||
Loading…
Reference in a new issue