glue-matrix-bao-token: retry a failed login, keep degrading on an empty read

The unit treats every failure as permanent: it prints why and `exit 0`s,
with no `Restart=`, so one bad moment costs the whole boot. Two of its
failure modes deserve that and one does not.

`bao login` fails when the store is unreachable, sealed, or has not been
given this host's cert-auth role yet. All three are transient. Measured
on this morning's rebuild:

  11:24:05  Started Container 'swarm-bao'
  11:24:06  could not log in to swarm-bao with this host's certificate
  11:24:07  Success! Data written to: auth/cert/certs/swarm-secret-publisher

It lost by one second, and stayed degraded for the boot. The publisher
next to it hit the same race and recovered on its first retry, because it
has `Restart=on-failure`.

`bao kv get` returning nothing is the opposite: the store answered, and
holds no token at that path. A retry cannot improve it, so that branch
keeps `exit 0` and the local token.

The bound is sized for this race, not for an unseal.
`swarm-bao-controller-policy` waits 2880 x 30s because a shamir unseal is
a human action and that unit blocks nothing. This one is `Before=` the
homeserver's container, so every retry is time the homeserver may spend
waiting -- 4 x 15s covers a container-start race with margin, and a store
still sealed after it degrades exactly as it does today.

`StartLimit*` are `[Unit]` settings and are ignored under `[Service]`, so
they are top-level attrs here. The module-eval case asserts the window
outlasts `RestartSec x burst`, since a burst that cannot be reached is a
unit that looks like it retries and does not.

Refs #4303
This commit is contained in:
atlas 2026-09-12 15:19:17 +02:00 committed by mara
commit 35c3f724f1
2 changed files with 43 additions and 1 deletions

View file

@ -1127,6 +1127,27 @@ let
&& !(holdsTokenFirst "# bao login -method=cert -token-only goes here\nbao kv get secret/x")
&& holdsTokenFirst "BAO_TOKEN=\"$(bao login -method=cert -token-only)\"\nbao kv get secret/x";
}
{
# A login failure is the store being unreachable, sealed, or not yet
# holding this host's role — all of which a retry fixes. A read that
# answers "nothing there" is not, so only the first is allowed to fail
# the unit.
name = "the matrix token reader retries a failed login and still degrades on an empty read";
ok =
let
u = baoWithMatrix.systemd.services.swarm-bao-matrix-token;
# Everything between the login's failure branch and the read's, which
# is where the exit that decides "retry or give up" lives.
afterLogin = lib.last (lib.splitString "bao login" u.script);
loginBranch = lib.head (lib.splitString "bao kv get" afterLogin);
in
u.serviceConfig.Restart or null == "on-failure"
&& u.startLimitBurst or 0 > 0
# The window has to outlast every attempt, or the burst is unreachable.
&& u.startLimitIntervalSec or 0 > (u.serviceConfig.RestartSec or 0) * (u.startLimitBurst or 0)
&& lib.hasInfix "exit 1" loginBranch
&& lib.hasInfix "exit 0" (lib.last (lib.splitString "bao kv get" u.script));
}
{
# The doctrine three glue files state, as a property a rewrite has to
# keep: a client is defined by holding a certificate the store accepts,