Watch
0
0
Fork
You've already forked hyperhive
0
hyperhive/docs/4-implementation/swarm/readiness.md

2.1 KiB

Readiness checks

The swarm daemons that serve /readyz (see the services page) answer it one of two ways. A daemon whose readiness is its queue connection reads that connection's state. A daemon whose readiness is "its store answers" reads a cached check, so a probe never waits on the store and never logs in to it.

The cached check

swarm_secret_client::health::CachedCheck runs a check at start and then every 30 s, each run bounded by the 30 s period. A success records its time. The check counts as fresh for 90 s after that, three periods, so one failed run doesn't flip readiness. Before the first success there is nothing fresh, so the daemon isn't ready. A failed or hung run logs a warning and changes nothing else. The task ends when the last clone of the check drops.

CachedCheck::is_fresh reads one atomic: the probe path does no I/O. Health::ready_while_checked wires a check to a daemon's /readyz.

What each daemon checks

Daemon The check
swarm-logs-auth a login to the store under its own certificate
swarm-metrics-auth the read of the stored bearer, the lookup it makes anyway
swarm-authelia-bridge the users file loads (the bridge has no secret store)
swarm-controller a login to the store; the probe reads the queue connection live, since it's in memory

The controller starts its check once, when it starts serving (routes::watch_store), and /readyz reads the last result. Before that, and in tests that don't start it, the store reads as not ready. The socket's /readyz and the plain listener's (SWARM_CONTROLLER_HEALTH_ADDR) both ask routes::is_ready, so they can't disagree. The listener binds at the top of main, before the store is reachable, so a probe gets a 503 and not a refused connection while the controller starts.