swarm-controller: retry hive provisioning until the store is up
The read policy and cert-auth role for each hive were written once, at startup. On the deploy that surfaced this, the store was still coming up, the pass logged its warning and moved on, and no hive could log in until someone restarted the daemon — while cert auth answered "no chain matching all constraints", which reads like a certificate problem rather than a role that was never created. The bootstrap unit in swarm-bao.nix lost the same race and won on its retry 30s later. A daemon that boots alongside its store loses that race routinely; on a normal boot it is the ordinary case. The two passes fold into one `provision()` that logs in once instead of twice for two loops over the same list, keeping policy before role since the role names the policy. `ensure_hive_access` still awaits the first pass, so a store that is already up leaves nothing deferred, and only a pass that could not reach the store at all spawns the retry. The retry is `config_pr::spawn`'s idiom from this same crate: an interval task whose first tick is immediate. Its cadence and bound match the bootstrap unit's — 30s, ~a day — because the two halves of one race should not disagree about how long a wait is worth. `Error::MissingEnv` is what keeps it from spinning forever: no `BAO_*` set means a deployment that runs no store, where asking again changes nothing, so it returns Ok. Everything else is retryable, including an authority file that is not placed yet — the unit that writes it starts alongside this one. Both cases previously landed in the same "not managed here" line, so a store that was late looked exactly like one that was never configured. Per-hive failures keep their old behaviour: logged, skipped, Ok. A store that refuses one hive's write refuses it again, so the next start really is the right retry for those, and the module doc still says so. Closes #4176.
This commit is contained in:
parent
513554fe9a
commit
b8c5840299
2 changed files with 189 additions and 55 deletions
|
|
@ -1625,13 +1625,11 @@ async fn main() -> Result<()> {
|
|||
let hives = load_hives();
|
||||
// Before serving, because a hive whose role does not exist cannot log in,
|
||||
// and one whose policy does not exist logs in able to read nothing —
|
||||
// either way it cannot collect what this daemon writes for it. Policy
|
||||
// first: the role names the policy, so writing it the other way round
|
||||
// leaves a window where the role points at nothing. Same "log and carry
|
||||
// on" shape as every connect above.
|
||||
// either way it cannot collect what this daemon writes for it. A store
|
||||
// that is not up yet is retried in the background instead of being
|
||||
// abandoned, since the two of them boot together.
|
||||
let hive_names: Vec<String> = hives.iter().map(|h| h.name.clone()).collect();
|
||||
read_policy::ensure_hive_policies(&hive_names).await;
|
||||
read_policy::ensure_hive_roles(&hive_names).await;
|
||||
read_policy::ensure_hive_access(hive_names).await;
|
||||
|
||||
let state = AppState {
|
||||
hives: Arc::new(hives),
|
||||
|
|
|
|||
Loading…
Reference in a new issue