credential units: 24h retry shape; start a failed nginx when the cert lands

Six credential-fetch units retried 4 times at 15s, so an apply during
which the store or gateway was down for more than about a minute left
them in start-limit-hit, and nothing started them again once the store
came back. The swarm-services leaf could also land after nginx had
already given up on it, and the hook that propagates a new leaf only
reloaded a running nginx, so a stopped one stayed down until a second
apply.

- nix/host-modules/lib/store-retry.nix: the 2880 x 30s / 25h window
  shape swarm-services-cert already had, as one attrset.
- swarm-services-cert, swarm-bao-otel-oidc, swarm-bao-forwarder-oidc,
  swarm-bao-matrix-token, swarm-bao-queue-agent, swarm-bao-grafana-oidc,
  hive-agent-bao-identity and hive-agent-forge-token use it.
  queue-identity.nix no longer has a fetch unit (ccb5bd3b), and
  forge-token.nix is a fetch unit with the same short budget that was
  added after the census in #4662.
- The swarm-services-cert propagation hook now reset-fails and starts
  (--no-block) a loaded nginx that is not active; an active nginx keeps
  the re-import + reload.
- module-eval-bao-grants: one case pinning the shape on every host-side
  fetch unit, swarm-services-cert included.

Refs #4662

nix/host-modules/swarm-otel.nix

@@ -310,6 +310,7 @@ let
   # store, without either crossing a network boundary that would need
   # its own trust material.
   privateNetwork = false;
+  storeRetry = import ./lib/store-retry.nix { };
 in
 {
   # `enable` moved to `services.hyperhive.deploy.swarm-otel.enable` — see ./deploy.nix.
@@ -790,27 +791,17 @@ in
         baoDeploy.package
         pkgs.coreutils
       ];
-      # Sized for the race this loses, not for an unseal: `swarm-bao` comes up
-      # seconds before this unit asks, and the cert-auth role it logs in
-      # against is written seconds after, so a few short attempts cover it. An
-      # hours-long window would be a bet on a sealed store, and the degrade
-      # below is already correct for that.
-      #
-      # `StartLimit*` are `[Unit]` settings, so they go here and not in
-      # `serviceConfig` — systemd ignores them under `[Service]`. The window
-      # has to exceed `RestartSec × burst`.
-      startLimitBurst = 4;
-      startLimitIntervalSec = 300;
-      serviceConfig = {
+      # ./lib/store-retry.nix. The collector's container is ordered after
+      # this unit, so it waits while the login below keeps failing.
+      inherit (storeRetry) startLimitBurst startLimitIntervalSec;
+      serviceConfig = storeRetry.serviceConfig // {
         Type = "oneshot";
         RemainAfterExit = true;
         SyslogIdentifier = "swarm-bao-otel-oidc";
-        # What actually bounds the read below. Stated here rather than left to
-        # systemd's default, so the number a boot waits on is in the file that
-        # waits.
+        # What actually bounds each attempt below. Stated here rather than left
+        # to systemd's default, so the number a boot waits on is in the file
+        # that waits.
         TimeoutStartSec = 30;
-        Restart = "on-failure";
-        RestartSec = 15;
       };
       environment = {
         BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";