vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

credential units: 24h retry shape; start a failed nginx when the cert lands

Browse source 
Six credential-fetch units retried 4 times at 15s, so an apply during
which the store or gateway was down for more than about a minute left
them in start-limit-hit, and nothing started them again once the store
came back. The swarm-services leaf could also land after nginx had
already given up on it, and the hook that propagates a new leaf only
reloaded a running nginx, so a stopped one stayed down until a second
apply.

- nix/host-modules/lib/store-retry.nix: the 2880 x 30s / 25h window
  shape swarm-services-cert already had, as one attrset.
- swarm-services-cert, swarm-bao-otel-oidc, swarm-bao-forwarder-oidc,
  swarm-bao-matrix-token, swarm-bao-queue-agent, swarm-bao-grafana-oidc,
  hive-agent-bao-identity and hive-agent-forge-token use it.
  queue-identity.nix no longer has a fetch unit (ccb5bd3b), and
  forge-token.nix is a fetch unit with the same short budget that was
  added after the census in #4662.
- The swarm-services-cert propagation hook now reset-fails and starts
  (--no-block) a loaded nginx that is not active; an active nginx keeps
  the re-import + reload.
- module-eval-bao-grants: one case pinning the shape on every host-side
  fetch unit, swarm-services-cert included.

Refs #4662
This commit is contained in:
atlas 2026-09-29 23:15:29 +02:00 • committed by mara
commit b3b42d3279
10 changed files with 137 additions and 125 deletions
Download patch file Download diff file Expand all files Collapse all files

35
nix/host-modules/glue-matrix-bao-token.nix
View file

@ -82,6 +82,7 @@ let
matrixMachine = "hive-matrix";
atomicWriteSecret = import ./lib/atomic-write-secret.nix { };
storeRetry = import ./lib/store-retry.nix { };
in
{
config = lib.mkMerge [
@ -151,30 +152,18 @@ in
deployCfg.bao.package
pkgs.coreutils
];
# Sized for the race this loses, not for an unseal. `swarm-bao` comes up
# seconds before this unit asks, and the cert-auth role it logs in
# against is written seconds after — so a few short attempts cover it.
# ⚠️ `swarm-bao-controller-policy`'s 2880 × 30s is NOT the model to copy.
# That unit blocks nothing; this one is `Before=` the homeserver's
# container, and whether that ordering waits across an auto-restart is
# unverified — so an hours-long window would be a bet on an unknown,
# where a minute is not. A store still sealed after it keeps the degrade
# below, as today.
#
# `StartLimit*` are `[Unit]` settings, so they go here and not in
# `serviceConfig` — systemd ignores them under `[Service]`. The window
# has to exceed `RestartSec × burst`.
startLimitBurst = 4;
startLimitIntervalSec = 300;
serviceConfig = {
# ./lib/store-retry.nix. ⚠️ This unit is `Before=` the homeserver's
# container, and an auto-restarting unit is still `activating`, so the
# homeserver waits for as long as the login below keeps failing — up to
# the full 24h on a store that stays sealed.
inherit (storeRetry) startLimitBurst startLimitIntervalSec;
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
RemainAfterExit = true;
# What actually bounds the read below. Stated here rather than
# What actually bounds each attempt below. Stated here rather than
# left to systemd's default, so the number a boot waits on is in
# the file that waits.
TimeoutStartSec = 30;
Restart = "on-failure";
RestartSec = 15;
};
environment = {
BAO_ADDR = "https://${baoCfg.domain}:${toString baoCfg.port}";
@ -193,9 +182,11 @@ in
# A sealed or uninitialised store answers on the port and never
# answers the read, so "the store is up" is not the same as "the
# store can answer". `TimeoutStartSec` above is the bound; the
# homeserver only `Wants=` this unit, so hitting it degrades to
# keeping the local token rather than holding up the container.
# store can answer". `TimeoutStartSec` above bounds each attempt and
# a timed-out attempt is retried like any other failure; the
# homeserver only `Wants=` this unit, so a retry budget spent
# degrades to keeping the local token rather than failing the
# container.
# `bao`'s own message is the only thing separating a missing value
# from a refused identity from an unreachable host. This unit's
# degraded mode is correct for all three, so it reports which one