swarm-controller: retry hive provisioning until the store is up

The read policy and cert-auth role for each hive were written once, at
startup. On the deploy that surfaced this, the store was still coming
up, the pass logged its warning and moved on, and no hive could log in
until someone restarted the daemon — while cert auth answered "no chain
matching all constraints", which reads like a certificate problem
rather than a role that was never created.

The bootstrap unit in swarm-bao.nix lost the same race and won on its
retry 30s later. A daemon that boots alongside its store loses that race
routinely; on a normal boot it is the ordinary case.

The two passes fold into one `provision()` that logs in once instead of
twice for two loops over the same list, keeping policy before role since
the role names the policy. `ensure_hive_access` still awaits the first
pass, so a store that is already up leaves nothing deferred, and only a
pass that could not reach the store at all spawns the retry.

The retry is `config_pr::spawn`'s idiom from this same crate: an
interval task whose first tick is immediate. Its cadence and bound match
the bootstrap unit's — 30s, ~a day — because the two halves of one race
should not disagree about how long a wait is worth.

`Error::MissingEnv` is what keeps it from spinning forever: no `BAO_*`
set means a deployment that runs no store, where asking again changes
nothing, so it returns Ok. Everything else is retryable, including an
authority file that is not placed yet — the unit that writes it starts
alongside this one. Both cases previously landed in the same "not
managed here" line, so a store that was late looked exactly like one
that was never configured.

Per-hive failures keep their old behaviour: logged, skipped, Ok. A store
that refuses one hive's write refuses it again, so the next start really
is the right retry for those, and the module doc still says so.

Closes #4176.
This commit is contained in:
atlas 2026-09-11 01:36:52 +02:00 committed by mara
commit b8c5840299
2 changed files with 189 additions and 55 deletions

View file

@ -1625,13 +1625,11 @@ async fn main() -> Result<()> {
let hives = load_hives();
// Before serving, because a hive whose role does not exist cannot log in,
// and one whose policy does not exist logs in able to read nothing —
// either way it cannot collect what this daemon writes for it. Policy
// first: the role names the policy, so writing it the other way round
// leaves a window where the role points at nothing. Same "log and carry
// on" shape as every connect above.
// either way it cannot collect what this daemon writes for it. A store
// that is not up yet is retried in the background instead of being
// abandoned, since the two of them boot together.
let hive_names: Vec<String> = hives.iter().map(|h| h.name.clone()).collect();
read_policy::ensure_hive_policies(&hive_names).await;
read_policy::ensure_hive_roles(&hive_names).await;
read_policy::ensure_hive_access(hive_names).await;
let state = AppState {
hives: Arc::new(hives),