Watch
0
0
Fork
You've already forked hyperhive
0
hyperhive/docs/4-implementation/swarm/readiness.md

36 lines
2.1 KiB
Markdown

# Readiness checks
The swarm daemons that serve `/readyz` (see the [services
page](../../2-handbook/swarm/services.md)) answer it one of two ways. A daemon
whose readiness is its queue connection reads that connection's state. A daemon
whose readiness is "its store answers" reads a cached check, so a probe never
waits on the store and never logs in to it.
## The cached check
`swarm_secret_client::health::CachedCheck` runs a check at start and then every
30 s, each run bounded by the 30 s period. A success records its time. The
check counts as fresh for 90 s after that, three periods, so one failed run
doesn't flip readiness. Before the first success there is nothing fresh, so the
daemon isn't ready. A failed or hung run logs a warning and changes nothing else. The task
ends when the last clone of the check drops.
`CachedCheck::is_fresh` reads one atomic: the probe path does no I/O.
`Health::ready_while_checked` wires a check to a daemon's `/readyz`.
## What each daemon checks
| Daemon | The check |
| ----------------------- | ------------------------------------------------------------------------------------- |
| `swarm-logs-auth` | a login to the store under its own certificate |
| `swarm-metrics-auth` | the read of the stored bearer, the lookup it makes anyway |
| `swarm-authelia-bridge` | the users file loads (the bridge has no secret store) |
| `swarm-controller` | a login to the store; the probe reads the queue connection live, since it's in memory |
The controller starts its check once, when it starts serving
(`routes::watch_store`), and `/readyz` reads the last result. Before that, and
in tests that don't start it, the store reads as not ready. The socket's
`/readyz` and the plain listener's (`SWARM_CONTROLLER_HEALTH_ADDR`) both ask
`routes::is_ready`, so they can't disagree. The listener binds at the top of
`main`, before the store is reachable, so a probe gets a 503 and not a refused
connection while the controller starts.