diff --git a/docs/swarm/credentials-reference.md b/docs/swarm/credentials-reference.md index 6d10a8a5..3872d3a2 100644 --- a/docs/swarm/credentials-reference.md +++ b/docs/swarm/credentials-reference.md @@ -128,7 +128,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials. | the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) | | a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) | | the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives//` and drop the setting | -| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | +| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once. An attempt that fails or runs past 30 seconds retries after 5 seconds, then at growing intervals up to 5 minutes | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) | | the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path | | an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents//backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file | | an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents//proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it | diff --git a/docs/swarm/credentials.md b/docs/swarm/credentials.md index a2204efd..79b6afa1 100644 --- a/docs/swarm/credentials.md +++ b/docs/swarm/credentials.md @@ -89,7 +89,7 @@ The same unit mints the client secrets, straight into the store. It reads only when the store answers and holds none does it run `authelia crypto hash generate`, create the leaf and write the digest. Authelia keeps only the digest on disk, in `oidc-clients/.digest`, and starts from it while the store is down. A store that -can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries every 15 seconds, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a +can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest, with the password typed into its prompt through a pty, never in a command line. `swarm-authelia-bridge` fetches its own secret from diff --git a/nix/container-modules/swarm-container.nix b/nix/container-modules/swarm-container.nix index cbfecd35..fb7a379a 100644 --- a/nix/container-modules/swarm-container.nix +++ b/nix/container-modules/swarm-container.nix @@ -5,6 +5,17 @@ # They restate host-side facts the container cannot read on its own: its # evaluation is nested inside `containers.`, and reading the host side # back from in here recurses. +# +# ⚠️ The container is `Type=notify`: the host's start job lasts until this +# guest's job queue is empty and ends in a kill at the container's +# `timeoutStartSec`, one minute unless the host module sets more. A unit in that +# queue, wanted by `multi-user.target` or required by a unit that is, states a +# `TimeoutStartSec` below that limit; a fetch that needs longer fails fast, +# restarts with a growing delay, and starts what depends on it with +# `systemctl start --no-block` (./swarm-journal-forwarder.nix is the pattern). +# `module-eval-swarm-services-switch` fails on a unit whose budget, stated or the +# manager's 90 s default, reaches the limit, unless the unit is listed there with +# a reason. { config, lib, ... }: let cfg = config.services.hyperhive.swarmContainer; diff --git a/nix/host-modules/hive-forge/default.nix b/nix/host-modules/hive-forge/default.nix index f32e4f3e..2cbd39d1 100644 --- a/nix/host-modules/hive-forge/default.nix +++ b/nix/host-modules/hive-forge/default.nix @@ -1017,9 +1017,17 @@ in LoadCredential = lib.attrNames storeFiles; # The forge's answers pass through a temp file. PrivateTmp = true; - # Each forge call is bounded by curl's own timeouts, and a sealed - # store answers on the port and never answers the write. - TimeoutStartSec = "3min"; + # Bounds one attempt, and so the container's boot when the store + # accepts a connection and never answers: this unit is in the boot + # wait of a container with a one-minute start limit. A killed + # attempt is retried with the growing delay below; an attempt + # that minted a token and was killed leaves one the next + # attempt revokes. The account re-check (`accountRecheckSec`) runs + # inside this budget. + TimeoutStartSec = 30; + RestartSec = 5; + RestartSteps = 6; + RestartMaxDelaySec = 300; }; # `FORGEJO_CUSTOM`, same reason as `forgejo-sso-source` above: # every `forgejo admin` invocation needs it to find the app.ini diff --git a/nix/host-modules/hive-matrix.nix b/nix/host-modules/hive-matrix.nix index 878914d7..b164e984 100644 --- a/nix/host-modules/hive-matrix.nix +++ b/nix/host-modules/hive-matrix.nix @@ -1076,7 +1076,9 @@ in RemainAfterExit = true; ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice render"; # A sealed store answers on the port and never answers the read. - TimeoutStartSec = 60; + # The render and the sync run one after the other in the + # container's boot wait, whose start limit is a minute. + TimeoutStartSec = 20; RuntimeDirectory = swarmAppserviceRuntimeName; RuntimeDirectoryMode = "0700"; RuntimeDirectoryPreserve = "yes"; @@ -1110,7 +1112,7 @@ in serviceConfig = storeRetry.serviceConfig // { Type = "oneshot"; ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice sync"; - TimeoutStartSec = 60; + TimeoutStartSec = 20; RuntimeDirectory = swarmAppserviceRuntimeName; RuntimeDirectoryMode = "0700"; RuntimeDirectoryPreserve = "yes"; diff --git a/nix/host-modules/swarm-authelia.nix b/nix/host-modules/swarm-authelia.nix index 4a8d7a3f..5b28b4d9 100644 --- a/nix/host-modules/swarm-authelia.nix +++ b/nix/host-modules/swarm-authelia.nix @@ -1192,9 +1192,15 @@ in // lib.optionalAttrs publishes ( storeRetry.serviceConfig // { - RestartSec = 15; LoadCredential = lib.attrNames storeFiles; - TimeoutStartSec = 120; + # Bounds one run, and so the container's boot when the store + # accepts a connection and never answers: this unit is in the + # boot wait of a container with a one-minute start limit. + # What a killed run left is picked up by the next one. + TimeoutStartSec = 30; + RestartSec = 15; + RestartSteps = 3; + RestartMaxDelaySec = 120; } ); environment = lib.optionalAttrs publishes ( @@ -1281,6 +1287,11 @@ in // lib.optionalAttrs publishes { RuntimeDirectory = "swarm-authelia-bridge"; LoadCredential = lib.attrNames storeFiles; + # The secret fetch is part of the start job, which is in the + # container's boot wait: a store that never answers ends it here + # and not at the 90 s default, which is over the container's + # one-minute start limit. + TimeoutStartSec = 10; }; environment = { SWARM_AUTHELIA_BRIDGE_SOCKET = bridgeSocket; diff --git a/nix/module-eval/boot-wait.nix b/nix/module-eval/boot-wait.nix new file mode 100644 index 00000000..35a2c4bf --- /dev/null +++ b/nix/module-eval/boot-wait.nix @@ -0,0 +1,185 @@ +# Which units of a container's guest hold its boot open, and which of those can +# hold it past the host's start limit for the container. +# +# A swarm service container is `Type=notify`: the host's `container@` +# start job lasts until the guest's job queue is empty, and ends in a kill at +# `timeoutStartSec`. A guest unit that is pulled in by the boot targets and +# whose start job blocks therefore counts its start timeout against that limit. +# Pure functions over a guest's `config.systemd` and the host's +# `timeoutStartSec` string; ./swarm-services-switch.nix feeds them each named +# container. +{ lib }: +let + # A time span as seconds; `null` is infinity, which `TimeoutStartSec` spells + # `0`, `"infinity"` or leaves to `Type=oneshot`'s default. + spanSeconds = + v: + if builtins.isInt v then + if v == 0 then null else v + else if v == "infinity" then + null + else + lib.foldl' (acc: t: acc + tokenSeconds t) 0 (lib.splitString " " (lib.trim v)); + + unitSeconds = { + s = 1; + sec = 1; + second = 1; + seconds = 1; + m = 60; + min = 60; + minute = 60; + minutes = 60; + h = 3600; + hr = 3600; + hour = 3600; + hours = 3600; + }; + tokenSeconds = + token: + let + m = builtins.match "([0-9]+)([a-z]*)" token; + unit = if m == null then null else lib.elemAt m 1; + in + if m == null then + throw "boot-wait: cannot read '${token}' as a time span" + else if unit == "" then + lib.toInt (lib.head m) + else + lib.toInt (lib.head m) + * (unitSeconds.${unit} or (throw "boot-wait: unknown time unit '${unit}' in '${token}'")); + + # Where a boot starts: what `multi-user.target` requires, and the targets + # the stock units pull in alongside it. + rootTargets = [ + "sysinit.target" + "basic.target" + "multi-user.target" + "sockets.target" + "timers.target" + "paths.target" + "local-fs.target" + ]; + + kinds = { + services = "service"; + timers = "timer"; + sockets = "socket"; + paths = "path"; + targets = "target"; + }; + + # Every unit the guest configures, as `{ name = "."; unit; }`. + unitsOf = + systemd: + lib.concatLists ( + lib.mapAttrsToList ( + kind: suffix: + lib.mapAttrsToList (n: unit: { + name = "${n}.${suffix}"; + inherit unit; + }) (systemd.${kind} or { }) + ) kinds + ); + + # The units a boot starts: the closure of `rootTargets` over what a unit + # `wants`/`requires`/`bindsTo` and over the units naming it in + # `wantedBy`/`requiredBy`. `after`/`before` alone pull nothing in. + bootSet = + systemd: + let + units = unitsOf systemd; + byName = lib.listToAttrs (map (u: lib.nameValuePair u.name u.unit) units); + pulls = u: (u.wants or [ ]) ++ (u.requires or [ ]) ++ (u.bindsTo or [ ]); + pulledBy = u: (u.wantedBy or [ ]) ++ (u.requiredBy or [ ]); + children = + key: + (pulls (byName.${key} or { })) + ++ (map (u: u.name) (lib.filter (u: lib.elem key (pulledBy u.unit)) units)); + in + map (n: n.key) ( + builtins.genericClosure { + startSet = map (key: { inherit key; }) rootTargets; + operator = n: map (key: { inherit key; }) (children n.key); + } + ); + + # Whether the service's start job blocks on the unit's own work rather than + # ending at the fork: a oneshot, a notifying or forking daemon, or any unit + # with a start hook (which runs inside the job). + holdsJob = + unit: + let + sc = unit.serviceConfig or { }; + in + lib.elem (sc.Type or "simple") [ + "oneshot" + "notify" + "notify-reload" + "forking" + "dbus" + ] + || sc ? ExecStartPre + || sc ? ExecStartPost + || (unit.preStart or "") != "" + || (unit.postStart or "") != ""; + + # The services in `guest`'s boot wait whose start job can last `limit` seconds + # or more: `{ key; seconds; source; }`, `seconds = null` for infinity. A job + # that ends at `limit` or later can outlast the container's start, because the + # guest has already spent time booting. + # + # `source` says where the number comes from: `stated` is the unit's own + # `TimeoutStartSec`; `default` is the manager's `DefaultTimeoutStartSec` (90 s + # unless the guest sets one), which a unit that states none gets; `unbounded` + # is a oneshot that states none, which systemd leaves without a timeout. + overBudget = + { guest, limit }: + let + services = guest.systemd.services or { }; + managerDefault = (guest.systemd.settings.Manager or { }).DefaultTimeoutStartSec or 90; + in + lib.concatMap ( + key: + let + n = lib.removeSuffix ".service" key; + unit = services.${n}; + sc = unit.serviceConfig or { }; + budget = + if sc ? TimeoutStartSec then + { + source = "stated"; + seconds = spanSeconds sc.TimeoutStartSec; + } + else if (sc.Type or "simple") == "oneshot" then + { + source = "unbounded"; + seconds = null; + } + else + { + source = "default"; + seconds = spanSeconds managerDefault; + }; + in + lib.optional + ( + lib.hasSuffix ".service" key + && services ? ${n} + && holdsJob unit + && (budget.seconds == null || budget.seconds >= limit) + ) + { + inherit key; + inherit (budget) seconds source; + } + ) (bootSet guest.systemd); +in +{ + inherit + spanSeconds + bootSet + holdsJob + overBudget + ; +} diff --git a/nix/module-eval/forge-placement.nix b/nix/module-eval/forge-placement.nix index 81393dec..1c7a7ef2 100644 --- a/nix/module-eval/forge-placement.nix +++ b/nix/module-eval/forge-placement.nix @@ -409,6 +409,7 @@ let && lib.hasInfix "${toString budget} account_listed; then" s && budget > 0 && budget <= 20 + && budget < u.serviceConfig.TimeoutStartSec && allLocal.containers.hive-forge.timeoutStartSec == "1min" && lib.elem "multi-user.target" u.wantedBy && lib.hasInfix "list=\"$(forgejo admin user list \"$@\")\"" s diff --git a/nix/module-eval/swarm-services-switch.nix b/nix/module-eval/swarm-services-switch.nix index 839c7888..2092f3a0 100644 --- a/nix/module-eval/swarm-services-switch.nix +++ b/nix/module-eval/swarm-services-switch.nix @@ -183,6 +183,110 @@ let }; serviceContainers = serviceContainersWithCi.containers; + bootWait = import ./boot-wait.nix { inherit lib; }; + + # Every container hyperhive declares, by name: a generic walk over + # `containers` would also judge an operator's own. + hyperhiveContainers = [ + "hive-forge" + "hive-matrix" + "swarm-attic" + "swarm-authelia" + "swarm-bao" + "swarm-builder" + "swarm-ci-runner" + "swarm-claude-login" + "swarm-grafana" + "swarm-nats" + "swarm-notify-router" + "swarm-otel" + "swarm-review-classifier" + "swarm-victorialogs" + "swarm-victoriametrics" + ]; + + # A container's start limit, in seconds. + startLimitOf = name: bootWait.spanSeconds serviceContainers.${name}.timeoutStartSec; + + # The boot-wait units of container `name` whose start job can last `limit` + # (default: the container's own start limit) or more. + reachingLimit = + name: + { + limit ? startLimitOf name, + }: + bootWait.overBudget { + guest = serviceContainers.${name}.config; + inherit limit; + }; + + # Boot-wait units that state no `TimeoutStartSec` and are let through, each + # group with the reason. A unit that states a budget is judged on it whatever + # this says. A new unit that states none fails the check until it states one + # or is added here with its reason. + unstatedAllowed = [ + { + why = "NixOS's own units: they change local files, interfaces and rules, and wait on no peer"; + units = [ + "dhcpcd.service" + "firewall.service" + "linger-users.service" + "logrotate-checkconf.service" + "network-addresses-eth0.service" + "network-local-commands.service" + "nscd.service" + "resolvconf.service" + "suid-sgid-wrappers.service" + ]; + } + { + why = "hyperhive's local oneshots: they write the resolver and trust files, set up local state or initialise the store's own token, and wait on no peer"; + units = [ + "forgejo-gpg-init.service" + "forgejo-secrets.service" + "swarm-bao-token.service" + "swarm-bridge-dns.service" + "victoriametrics-state-migrate.service" + ]; + suffixes = [ + "-ca-bundle.service" + "-ca-bundle-refresh.service" + ]; + } + { + why = "the service's own start: a daemon that does not come up in time is a container that is not useful, whatever its budget"; + units = [ + "authelia-swarm.service" + "forgejo.service" + "gitea-runner-swarm.service" + "grafana.service" + "nats.service" + "opentelemetry-collector.service" + "tuwunel.service" + "victorialogs.service" + "victoriametrics.service" + ]; + } + { + why = "known gap: registering the login source makes an outbound HTTPS call to authelia, and the unit has neither a timeout nor a Restart="; + units = [ "forgejo-sso-source.service" ]; + } + ]; + allowedUnstated = + key: + lib.any ( + g: lib.elem key (g.units or [ ]) || lib.any (s: lib.hasSuffix s key) (g.suffixes or [ ]) + ) unstatedAllowed; + + # `: s () >= s`, over every named + # container. + bootWaitOffenders = lib.concatMap ( + name: + map ( + o: "${name}: ${o.key} ${toString o.seconds}s (${o.source}) >= ${toString (startLimitOf name)}s" + ) (lib.filter (o: o.source == "stated" || !allowedUnstated o.key) (reachingLimit name { })) + ) hyperhiveContainers; + # The metrics store on a build older than its socket needs. Only its # `assertions` are read, and the package is a stub carrying a version. oldMetricsStore = hive { @@ -1294,6 +1398,228 @@ let && lib.all namesNoStoreUnit readersElsewhere && !(lib.any (u: storeOnlyNoHive.systemd.services ? ${u}) readers); } + { + # A swarm service container is `Type=notify`: its start job lasts until + # the guest's job queue is empty and is killed at `timeoutStartSec`. A + # unit in that queue whose start job can last that long takes the whole + # container down with it. A unit that states no budget has the manager's + # 90 s default, or none if it is a oneshot, and counts too unless + # `unstatedAllowed` names it. Judged per unit, so a chain of units each + # under the limit is not caught. The offenders are in the name, which is + # printed on failure only. + name = "no unit in a service container's boot wait has a start budget that reaches the container's start limit${ + lib.optionalString (bootWaitOffenders != [ ]) ": ${lib.concatStringsSep "; " bootWaitOffenders}" + }"; + ok = bootWaitOffenders == [ ]; + } + { + # A name that no container's boot wait carries any more is a dead entry, + # and would let a unit of that name back in unseen. The presence arm is + # that each entry is found. + name = "every unit the boot-wait check lets through without a stated budget is in some container's boot wait"; + ok = + let + unstated = lib.concatMap ( + name: lib.filter (o: o.source != "stated") (reachingLimit name { }) + ) hyperhiveContainers; + keys = map (o: o.key) unstated; + found = key: lib.elem key keys; + foundSuffix = s: lib.any (k: lib.hasSuffix s k) keys; + in + lib.all ( + g: lib.all found (g.units or [ ]) && lib.all foundSuffix (g.suffixes or [ ]) + ) unstatedAllowed; + } + { + # Names, not `lib.attrNames containers`: a walk over every container + # would also judge an operator's own. Every container the fixture + # declares is in the list, so a new one cannot dodge the check. + name = "the boot-wait check names every service container the fixture declares"; + ok = + lib.sort lib.lessThan hyperhiveContainers + == lib.sort lib.lessThan (lib.attrNames serviceContainers); + } + { + # The presence arms for the check above: it reads real units and real + # limits. Each named unit states its budget (30 s, 30 s, 30 s, 20 s, 20 s), + # so it is found at a limit equal to it and not one second above (the + # boundary), and the container limits are the ones the check reads. + name = "the boot-wait check finds the units that state a budget, at the boundary, and reads each container's own limit"; + ok = + let + keys = name: limit: map (o: o.key) (reachingLimit name { inherit limit; }); + mint = "forgejo-swarm-controller-account.service"; + fetch = "swarm-journal-forwarder-oidc.service"; + secrets = "authelia-swarm-secrets.service"; + sync = "swarm-matrix-appservice-sync.service"; + render = "swarm-matrix-appservice-render.service"; + in + lib.elem mint (keys "hive-forge" 30) + && lib.elem fetch (keys "hive-forge" 30) + && !(lib.elem mint (keys "hive-forge" 31)) + && !(lib.elem fetch (keys "hive-forge" 31)) + && lib.elem secrets (keys "swarm-authelia" 30) + && !(lib.elem secrets (keys "swarm-authelia" 31)) + && lib.elem sync (keys "hive-matrix" 20) + && lib.elem render (keys "hive-matrix" 20) + && !(lib.elem sync (keys "hive-matrix" 21)) + && !(lib.elem render (keys "hive-matrix" 21)) + && startLimitOf "hive-forge" == 60 + && startLimitOf "swarm-bao" == 300; + } + { + # A unit held to a budget that fits the container's start retries with a + # growing delay, not at a fixed one, and the authelia bridge's secret + # fetch states a budget so that the check above judges it. + name = "the store-fetching units in a container's boot wait restart with a growing delay and the bridge's fetch states its budget"; + ok = + let + grows = + u: + let + sc = u.serviceConfig; + in + sc.Restart == "on-failure" && sc.RestartSteps > 0 && sc.RestartMaxDelaySec > sc.RestartSec; + authelia = serviceContainers.swarm-authelia.config.systemd.services; + in + grows serviceContainers.hive-forge.config.systemd.services.forgejo-swarm-controller-account + && grows authelia.authelia-swarm-secrets + && authelia.swarm-authelia-bridge.serviceConfig.TimeoutStartSec == 10; + } + { + # The forwarder's collector is started `--no-block` by the fetch and is + # not in the boot wait, whatever its own budget; its fetch is. The + # presence arm is the fetch, found by the same walk. + name = "the boot-wait walk includes a unit wanted by multi-user.target and leaves out one started on demand"; + ok = + let + booted = bootWait.bootSet serviceContainers.hive-forge.config.systemd; + in + lib.elem "swarm-journal-forwarder-oidc.service" booted + && lib.elem "forgejo-swarm-controller-account.service" booted + && !(lib.elem "opentelemetry-collector.service" booted); + } + { + # The walk and the comparison on a hand-made guest: each unit that must + # be found is, and each that must not be is not. Seconds are counted + # from every spelling a unit uses, infinity included. + name = "the boot-wait check flags a budget at or over the limit in the boot wait, pulled in any way, and nothing else"; + ok = + let + oneshot = budget: { + serviceConfig = { + Type = "oneshot"; + TimeoutStartSec = budget; + }; + }; + guest = { + systemd = { + targets.multi-user.wants = [ "viaTarget.service" ]; + services = { + wantedBy = oneshot "3min" // { + wantedBy = [ "multi-user.target" ]; + }; + viaRequires = oneshot 120; + parent = { + wantedBy = [ "multi-user.target" ]; + requires = [ "viaRequires.service" ]; + wants = [ "gate.target" ]; + }; + viaTarget = oneshot "1min 30s"; + viaRequiredBy = oneshot 100 // { + requiredBy = [ "multi-user.target" ]; + }; + viaGate = oneshot 100 // { + wantedBy = [ "gate.target" ]; + }; + atLimit = oneshot "1min" // { + wantedBy = [ "multi-user.target" ]; + }; + forever = oneshot "infinity" // { + wantedBy = [ "multi-user.target" ]; + }; + belowLimit = oneshot 59 // { + wantedBy = [ "multi-user.target" ]; + }; + notPulledIn = oneshot 900; + orderedOnly = oneshot 900 // { + after = [ "wantedBy.service" ]; + }; + forks = { + wantedBy = [ "multi-user.target" ]; + serviceConfig = { + Type = "simple"; + TimeoutStartSec = 900; + }; + }; + hook = { + wantedBy = [ "multi-user.target" ]; + preStart = "true"; + serviceConfig.TimeoutStartSec = "1h"; + }; + unstated = { + wantedBy = [ "multi-user.target" ]; + serviceConfig.Type = "oneshot"; + }; + defaulted = { + wantedBy = [ "multi-user.target" ]; + preStart = "true"; + }; + }; + }; + }; + in + lib.sort lib.lessThan ( + map (o: "${o.key}:${o.source}") ( + bootWait.overBudget { + inherit guest; + limit = 60; + } + ) + ) == [ + "atLimit.service:stated" + "defaulted.service:default" + "forever.service:stated" + "hook.service:stated" + "unstated.service:unbounded" + "viaGate.service:stated" + "viaRequiredBy.service:stated" + "viaRequires.service:stated" + "viaTarget.service:stated" + "wantedBy.service:stated" + ] + # The default is the manager's: over a limit above it, or with the guest + # setting a shorter one, the unit is not reached, and the unbounded + # oneshot still is. + && lib.elem "defaulted.service" ( + map (o: o.key) ( + bootWait.overBudget { + inherit guest; + limit = 90; + } + ) + ) + && !(lib.elem "defaulted.service" ( + map (o: o.key) ( + bootWait.overBudget { + inherit guest; + limit = 91; + } + ) + )) + && !(lib.elem "defaulted.service" ( + map (o: o.key) ( + bootWait.overBudget { + guest = guest // { + systemd = guest.systemd // { + settings.Manager.DefaultTimeoutStartSec = "45s"; + }; + }; + limit = 60; + } + ) + )); + } ]; in runGroup "swarm-services-switch" cases