readiness follows a cached store check instead of a constant or a login per probe
This commit is contained in:
parent
2eacb83b21
commit
b0239914b8
2 changed files with 369 additions and 40 deletions
|
|
@ -285,7 +285,10 @@ They answer 200 or 503 and send no body.
|
|||
- `GET /readyz` is 200 while the daemon has what it needs to work: its queue
|
||||
connection works (`swarm-stt`, `swarm-review-classifier`,
|
||||
`swarm-notify-router`, `swarm-builder`, `swarm-claude-login`,
|
||||
`swarm-nats-auth`), or it has bound its socket (the three auth responders).
|
||||
`swarm-nats-auth`), or what it answers from is there: `swarm-logs-auth` and
|
||||
`swarm-metrics-auth` are ready while the secret store answers,
|
||||
`swarm-authelia-bridge` while its users file loads. The mechanics are in
|
||||
[Readiness checks](../../4-implementation/swarm/readiness.md).
|
||||
|
||||
Nothing listens by default. Set `SWARM_STT_HEALTH_ADDR`,
|
||||
`SWARM_REVIEW_CLASSIFIER_HEALTH_ADDR`, `SWARM_NOTIFY_ROUTER_HEALTH_ADDR`,
|
||||
|
|
@ -299,4 +302,5 @@ daemon.
|
|||
|
||||
`swarm-controller` serves `GET /readyz` on its existing socket, next to
|
||||
`/health`. It answers 200 while the queue connection works and the secret store
|
||||
accepts the controller's login, and 503 otherwise.
|
||||
answers the controller's login, and 503 otherwise (see
|
||||
[Readiness checks](../../4-implementation/swarm/readiness.md)).
|
||||
|
|
|
|||
32
docs/4-implementation/swarm/readiness.md
Normal file
32
docs/4-implementation/swarm/readiness.md
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
# Readiness checks
|
||||
|
||||
The swarm daemons that serve `/readyz` (see the [services
|
||||
page](../../2-handbook/swarm/services.md)) answer it one of two ways. A daemon
|
||||
whose readiness is its queue connection reads that connection's state. A daemon
|
||||
whose readiness is "its store answers" reads a cached check, so a probe never
|
||||
waits on the store and never logs in to it.
|
||||
|
||||
## The cached check
|
||||
|
||||
`swarm_secret_client::health::CachedCheck` runs a check at start and then every
|
||||
30 s, each run bounded by the 30 s period. A success records its time. The
|
||||
check counts as fresh for 90 s after that, three periods, so one failed run
|
||||
doesn't flip readiness. Before the first success there is nothing fresh, so the
|
||||
daemon isn't ready. A failed or hung run logs a warning and changes nothing else. The task
|
||||
ends when the last clone of the check drops.
|
||||
|
||||
`CachedCheck::is_fresh` reads one atomic: the probe path does no I/O.
|
||||
`Health::ready_while_checked` wires a check to a daemon's `/readyz`.
|
||||
|
||||
## What each daemon checks
|
||||
|
||||
| Daemon | The check |
|
||||
| ----------------------- | ------------------------------------------------------------------------------------- |
|
||||
| `swarm-logs-auth` | a login to the store under its own certificate |
|
||||
| `swarm-metrics-auth` | the read of the stored bearer, the lookup it makes anyway |
|
||||
| `swarm-authelia-bridge` | the users file loads (the bridge has no secret store) |
|
||||
| `swarm-controller` | a login to the store; the probe reads the queue connection live, since it's in memory |
|
||||
|
||||
The controller starts its check once, when it starts serving
|
||||
(`routes::watch_store`), and `/readyz` reads the last result. Before that, and
|
||||
in tests that don't start it, the store reads as not ready.
|
||||
Loading…
Reference in a new issue