mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

keep service containers' boot-wait budgets under their start limit, and check it

Browse source
This commit is contained in:
atlas 2026-11-02 01:07:26 +01:00 • committed by mara
commit 70b112058a
9 changed files with 553 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/swarm/credentials-reference.md
View file

@ -128,7 +128,7 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](credentials.
| the swarm controller's copy of its OIDC secret | `swarm-controller` reads it out of the swarm secret store at start, under its own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the controller's store leaf (`deploy.swarm-controller.baoClientCertFile`) |
| a hive's copy of its own OIDC secret | hive-c0re reads it out of the swarm secret store when it first connects to the queue, under the hive's own leaf | the daemon's memory only; never on disk | same read, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`) |
| the hive collector's copy of its hive's OIDC secret | `swarm-bao-hive-otel-oidc.service` reads it out of the swarm secret store under the hive's own leaf, **on every hive that runs the collector and holds that leaf**; the leaf is the source | `/run/hive-otel-oidc/hive-<name>.secret` on the hive's host tmpfs, 0400 root, loaded by `LoadCredential` — never on persistent disk, so a reboot fetches it again | same unit, same path, co-located or not. Nothing for an operator to place beyond the hive's store leaf (`deploy.bao.clientCertFile`). Naming a secret file with `otel.clientSecretFile` instead is legacy for a host without a leaf: the build warns, and the fix is to place the leaf from `/var/lib/swarm-bao-pki/hives/<hive>/` and drop the setting |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's forge token | the forge's `forgejo-swarm-controller-account`, in-container, once. An attempt that fails or runs past 30 seconds retries after 5 seconds, then at growing intervals up to 5 minutes | the swarm secret store at `swarm/controller/swarm-controller/forge/token`, written by the mint under the forge publisher's leaf, which the forge container loads as a credential; the controller holds it in memory only. `/var/lib/forgejo/swarm-controller-token-name` in the forge container holds its name, the minter's record that it has minted | nothing to place beyond the controller's store leaf. A forge host without the store needs the publisher's leaf placed (`deploy.bao.forgePublisherClientCertFile`) |
| the swarm controller's Grafana token | `swarm-bao-grafana-controller-token.service`, **on the host that runs Grafana**, as Grafana's built-in admin over its socket: it creates the `swarm-controller` service account as a Viewer (and sets it back to Viewer on every run), mints a token for it and writes it to the swarm secret store at `swarm/controller/swarm-controller/grafana/token` under Grafana's leaf. It waits up to 60 seconds for Grafana's socket to answer before its first call. Re-run every ten minutes: it keeps a stored token Grafana accepts, and replaces one Grafana refuses with 401 (or a missing one), deleting every other token of that account first | the swarm secret store only; the controller reads it for `GET /api/alerts` and holds it in memory, re-reading the store once when Grafana answers 401 | nothing to place: Grafana's leaf is already required to run Grafana. Until the unit has run, `/api/alerts` answers 503 naming the empty path |
| an agent's API-key environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`, an ACP provider key) | `hive-agent-backend-env.service`, in the agent's container, on an agent with a store that uses `useApiKey` or the `acp` runtime. When the store holds nothing at `swarm/agents/<agent>/backend-env` and `backendEnvironmentFile` is a non-empty file, it creates the leaf with the file's exact bytes and `-cas=0`, under the agent's own leaf, which may create that path and not update it. It never rotates a value and never rewrites or deletes the file. To set or replace a key, use **set the API-key environment** in the swarm UI (`PUT /api/hives/{hive}/agents/{agent}/backend-env`): `swarm-controller` overwrites the leaf with its own grant, and the store's value then wins over the file. Its **remove** (`DELETE` on the same path) deletes the leaf, and an agent whose file has content then creates it from that file again | the swarm secret store at that path; the unit writes it to `/run/hive-agent-backend-env/env` (tmpfs, `0600`, root), and `hive-agent` and `hive-subagent-daemon` load that after `backendEnvironmentFile` as `EnvironmentFile=` | the same unit, same path. A variable in the store wins over the same one in the file. With the store down at a start, the file alone carries the agent, as it did before the store held a copy; with an older hive build the agent ignores the leaf and keeps reading the file |
| an agent's proxy settings (`HTTPS_PROXY`, `HTTP_PROXY`, `NO_PROXY`, the credential in the proxy URL) | `hive-agent-proxy-credential.service`, in the agent's container, as the agent user, at start and every 10 minutes: it reads `swarm/agents/<agent>/proxy-credential` under the agent's own leaf and builds the file from it. It writes nothing while the path holds nothing or the hive's facts name no squid | `/run/hive-agent-proxy-credential/proxy.env` (tmpfs, `0600`, the agent user); the harness, its bash tool, the subagents, the MCP servers and the pollers load it as `EnvironmentFile=-` and read it when they start | With no credential, or on a hive that writes no `HIVE_SQUID_PROXY` and `HIVE_EGRESS_NO_PROXY` fact, nothing writes the file and a unit has no proxy variable and reaches the internet directly. A failed read keeps the file already in place. A unit started before the file appeared goes direct until its next start. The credential is also in the environment of every process those units start, so `env` in the agent's shell prints it |

2
docs/swarm/credentials.md
View file

@ -89,7 +89,7 @@ The same unit mints the client secrets, straight into the store. It reads
only when the store answers and holds none does it run `authelia crypto hash generate`,
create the leaf and write the digest. Authelia keeps only the digest on disk, in
`oidc-clients/<id>.digest`, and starts from it while the store is down. A store that
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries every 15 seconds, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
can't answer a read never mints, because a second secret would rotate the first. When it can't read the store, the unit still writes authelia's client file from the digests, then exits non-zero and retries after 15 seconds, then at growing intervals up to two minutes, so authelia starts meanwhile and `swarm-authelia-bridge` finds its secret once the store answers. A client with a
secret in the store but no digest gets the digest of that secret and keeps the secret; `authelia crypto hash generate` computes the digest,
with the password typed into its prompt through a pty, never in a command line.
`swarm-authelia-bridge` fetches its own secret from

11
nix/container-modules/swarm-container.nix
View file

@ -5,6 +5,17 @@
# They restate host-side facts the container cannot read on its own: its
# evaluation is nested inside `containers.<name>`, and reading the host side
# back from in here recurses.
#
# ⚠️ The container is `Type=notify`: the host's start job lasts until this
# guest's job queue is empty and ends in a kill at the container's
# `timeoutStartSec`, one minute unless the host module sets more. A unit in that
# queue, wanted by `multi-user.target` or required by a unit that is, states a
# `TimeoutStartSec` below that limit; a fetch that needs longer fails fast,
# restarts with a growing delay, and starts what depends on it with
# `systemctl start --no-block` (./swarm-journal-forwarder.nix is the pattern).
# `module-eval-swarm-services-switch` fails on a unit whose budget, stated or the
# manager's 90 s default, reaches the limit, unless the unit is listed there with
# a reason.
{ config, lib, ... }:
let
cfg = config.services.hyperhive.swarmContainer;

14
nix/host-modules/hive-forge/default.nix
View file

@ -1017,9 +1017,17 @@ in
LoadCredential = lib.attrNames storeFiles;
# The forge's answers pass through a temp file.
PrivateTmp = true;
# Each forge call is bounded by curl's own timeouts, and a sealed
# store answers on the port and never answers the write.
TimeoutStartSec = "3min";
# Bounds one attempt, and so the container's boot when the store
# accepts a connection and never answers: this unit is in the boot
# wait of a container with a one-minute start limit. A killed
# attempt is retried with the growing delay below; an attempt
# that minted a token and was killed leaves one the next
# attempt revokes. The account re-check (`accountRecheckSec`) runs
# inside this budget.
TimeoutStartSec = 30;
RestartSec = 5;
RestartSteps = 6;
RestartMaxDelaySec = 300;
};
# `FORGEJO_CUSTOM`, same reason as `forgejo-sso-source` above:
# every `forgejo admin` invocation needs it to find the app.ini

6
nix/host-modules/hive-matrix.nix
View file

@ -1076,7 +1076,9 @@ in
RemainAfterExit = true;
ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice render";
# A sealed store answers on the port and never answers the read.
TimeoutStartSec = 60;
# The render and the sync run one after the other in the
# container's boot wait, whose start limit is a minute.
TimeoutStartSec = 20;
RuntimeDirectory = swarmAppserviceRuntimeName;
RuntimeDirectoryMode = "0700";
RuntimeDirectoryPreserve = "yes";
@ -1110,7 +1112,7 @@ in
serviceConfig = storeRetry.serviceConfig // {
Type = "oneshot";
ExecStart = "${deployCfg.matrix.ctlPackage}/bin/swarm-matrix-ctl appservice sync";
TimeoutStartSec = 60;
TimeoutStartSec = 20;
RuntimeDirectory = swarmAppserviceRuntimeName;
RuntimeDirectoryMode = "0700";
RuntimeDirectoryPreserve = "yes";

15
nix/host-modules/swarm-authelia.nix
View file

@ -1192,9 +1192,15 @@ in
// lib.optionalAttrs publishes (
storeRetry.serviceConfig
// {
RestartSec = 15;
LoadCredential = lib.attrNames storeFiles;
TimeoutStartSec = 120;
# Bounds one run, and so the container's boot when the store
# accepts a connection and never answers: this unit is in the
# boot wait of a container with a one-minute start limit.
# What a killed run left is picked up by the next one.
TimeoutStartSec = 30;
RestartSec = 15;
RestartSteps = 3;
RestartMaxDelaySec = 120;
}
);
environment = lib.optionalAttrs publishes (
@ -1281,6 +1287,11 @@ in
// lib.optionalAttrs publishes {
RuntimeDirectory = "swarm-authelia-bridge";
LoadCredential = lib.attrNames storeFiles;
# The secret fetch is part of the start job, which is in the
# container's boot wait: a store that never answers ends it here
# and not at the 90 s default, which is over the container's
# one-minute start limit.
TimeoutStartSec = 10;
};
environment = {
SWARM_AUTHELIA_BRIDGE_SOCKET = bridgeSocket;

185
nix/module-eval/boot-wait.nix Normal file
View file

@ -0,0 +1,185 @@
# Which units of a container's guest hold its boot open, and which of those can
# hold it past the host's start limit for the container.
#
# A swarm service container is `Type=notify`: the host's `container@<name>`
# start job lasts until the guest's job queue is empty, and ends in a kill at
# `timeoutStartSec`. A guest unit that is pulled in by the boot targets and
# whose start job blocks therefore counts its start timeout against that limit.
# Pure functions over a guest's `config.systemd` and the host's
# `timeoutStartSec` string; ./swarm-services-switch.nix feeds them each named
# container.
{ lib }:
let
# A time span as seconds; `null` is infinity, which `TimeoutStartSec` spells
# `0`, `"infinity"` or leaves to `Type=oneshot`'s default.
spanSeconds =
v:
if builtins.isInt v then
if v == 0 then null else v
else if v == "infinity" then
null
else
lib.foldl' (acc: t: acc + tokenSeconds t) 0 (lib.splitString " " (lib.trim v));
unitSeconds = {
s = 1;
sec = 1;
second = 1;
seconds = 1;
m = 60;
min = 60;
minute = 60;
minutes = 60;
h = 3600;
hr = 3600;
hour = 3600;
hours = 3600;
};
tokenSeconds =
token:
let
m = builtins.match "([0-9]+)([a-z]*)" token;
unit = if m == null then null else lib.elemAt m 1;
in
if m == null then
throw "boot-wait: cannot read '${token}' as a time span"
else if unit == "" then
lib.toInt (lib.head m)
else
lib.toInt (lib.head m)
* (unitSeconds.${unit} or (throw "boot-wait: unknown time unit '${unit}' in '${token}'"));
# Where a boot starts: what `multi-user.target` requires, and the targets
# the stock units pull in alongside it.
rootTargets = [
"sysinit.target"
"basic.target"
"multi-user.target"
"sockets.target"
"timers.target"
"paths.target"
"local-fs.target"
];
kinds = {
services = "service";
timers = "timer";
sockets = "socket";
paths = "path";
targets = "target";
};
# Every unit the guest configures, as `{ name = "<n>.<type>"; unit; }`.
unitsOf =
systemd:
lib.concatLists (
lib.mapAttrsToList (
kind: suffix:
lib.mapAttrsToList (n: unit: {
name = "${n}.${suffix}";
inherit unit;
}) (systemd.${kind} or { })
) kinds
);
# The units a boot starts: the closure of `rootTargets` over what a unit
# `wants`/`requires`/`bindsTo` and over the units naming it in
# `wantedBy`/`requiredBy`. `after`/`before` alone pull nothing in.
bootSet =
systemd:
let
units = unitsOf systemd;
byName = lib.listToAttrs (map (u: lib.nameValuePair u.name u.unit) units);
pulls = u: (u.wants or [ ]) ++ (u.requires or [ ]) ++ (u.bindsTo or [ ]);
pulledBy = u: (u.wantedBy or [ ]) ++ (u.requiredBy or [ ]);
children =
key:
(pulls (byName.${key} or { }))
++ (map (u: u.name) (lib.filter (u: lib.elem key (pulledBy u.unit)) units));
in
map (n: n.key) (
builtins.genericClosure {
startSet = map (key: { inherit key; }) rootTargets;
operator = n: map (key: { inherit key; }) (children n.key);
}
);
# Whether the service's start job blocks on the unit's own work rather than
# ending at the fork: a oneshot, a notifying or forking daemon, or any unit
# with a start hook (which runs inside the job).
holdsJob =
unit:
let
sc = unit.serviceConfig or { };
in
lib.elem (sc.Type or "simple") [
"oneshot"
"notify"
"notify-reload"
"forking"
"dbus"
]
|| sc ? ExecStartPre
|| sc ? ExecStartPost
|| (unit.preStart or "") != ""
|| (unit.postStart or "") != "";
# The services in `guest`'s boot wait whose start job can last `limit` seconds
# or more: `{ key; seconds; source; }`, `seconds = null` for infinity. A job
# that ends at `limit` or later can outlast the container's start, because the
# guest has already spent time booting.
#
# `source` says where the number comes from: `stated` is the unit's own
# `TimeoutStartSec`; `default` is the manager's `DefaultTimeoutStartSec` (90 s
# unless the guest sets one), which a unit that states none gets; `unbounded`
# is a oneshot that states none, which systemd leaves without a timeout.
overBudget =
{ guest, limit }:
let
services = guest.systemd.services or { };
managerDefault = (guest.systemd.settings.Manager or { }).DefaultTimeoutStartSec or 90;
in
lib.concatMap (
key:
let
n = lib.removeSuffix ".service" key;
unit = services.${n};
sc = unit.serviceConfig or { };
budget =
if sc ? TimeoutStartSec then
{
source = "stated";
seconds = spanSeconds sc.TimeoutStartSec;
}
else if (sc.Type or "simple") == "oneshot" then
{
source = "unbounded";
seconds = null;
}
else
{
source = "default";
seconds = spanSeconds managerDefault;
};
in
lib.optional
(
lib.hasSuffix ".service" key
&& services ? ${n}
&& holdsJob unit
&& (budget.seconds == null || budget.seconds >= limit)
)
{
inherit key;
inherit (budget) seconds source;
}
) (bootSet guest.systemd);
in
{
inherit
spanSeconds
bootSet
holdsJob
overBudget
;
}

1
nix/module-eval/forge-placement.nix
View file

@ -409,6 +409,7 @@ let
&& lib.hasInfix "${toString budget} account_listed; then" s
&& budget > 0
&& budget <= 20
&& budget < u.serviceConfig.TimeoutStartSec
&& allLocal.containers.hive-forge.timeoutStartSec == "1min"
&& lib.elem "multi-user.target" u.wantedBy
&& lib.hasInfix "list=\"$(forgejo admin user list \"$@\")\"" s

326
nix/module-eval/swarm-services-switch.nix
View file

@ -183,6 +183,110 @@ let
};
serviceContainers = serviceContainersWithCi.containers;
bootWait = import ./boot-wait.nix { inherit lib; };
# Every container hyperhive declares, by name: a generic walk over
# `containers` would also judge an operator's own.
hyperhiveContainers = [
"hive-forge"
"hive-matrix"
"swarm-attic"
"swarm-authelia"
"swarm-bao"
"swarm-builder"
"swarm-ci-runner"
"swarm-claude-login"
"swarm-grafana"
"swarm-nats"
"swarm-notify-router"
"swarm-otel"
"swarm-review-classifier"
"swarm-victorialogs"
"swarm-victoriametrics"
];
# A container's start limit, in seconds.
startLimitOf = name: bootWait.spanSeconds serviceContainers.${name}.timeoutStartSec;
# The boot-wait units of container `name` whose start job can last `limit`
# (default: the container's own start limit) or more.
reachingLimit =
name:
{
limit ? startLimitOf name,
}:
bootWait.overBudget {
guest = serviceContainers.${name}.config;
inherit limit;
};
# Boot-wait units that state no `TimeoutStartSec` and are let through, each
# group with the reason. A unit that states a budget is judged on it whatever
# this says. A new unit that states none fails the check until it states one
# or is added here with its reason.
unstatedAllowed = [
{
why = "NixOS's own units: they change local files, interfaces and rules, and wait on no peer";
units = [
"dhcpcd.service"
"firewall.service"
"linger-users.service"
"logrotate-checkconf.service"
"network-addresses-eth0.service"
"network-local-commands.service"
"nscd.service"
"resolvconf.service"
"suid-sgid-wrappers.service"
];
}
{
why = "hyperhive's local oneshots: they write the resolver and trust files, set up local state or initialise the store's own token, and wait on no peer";
units = [
"forgejo-gpg-init.service"
"forgejo-secrets.service"
"swarm-bao-token.service"
"swarm-bridge-dns.service"
"victoriametrics-state-migrate.service"
];
suffixes = [
"-ca-bundle.service"
"-ca-bundle-refresh.service"
];
}
{
why = "the service's own start: a daemon that does not come up in time is a container that is not useful, whatever its budget";
units = [
"authelia-swarm.service"
"forgejo.service"
"gitea-runner-swarm.service"
"grafana.service"
"nats.service"
"opentelemetry-collector.service"
"tuwunel.service"
"victorialogs.service"
"victoriametrics.service"
];
}
{
why = "known gap: registering the login source makes an outbound HTTPS call to authelia, and the unit has neither a timeout nor a Restart=";
units = [ "forgejo-sso-source.service" ];
}
];
allowedUnstated =
key:
lib.any (
g: lib.elem key (g.units or [ ]) || lib.any (s: lib.hasSuffix s key) (g.suffixes or [ ])
) unstatedAllowed;
# `<container>: <unit> <seconds>s (<source>) >= <limit>s`, over every named
# container.
bootWaitOffenders = lib.concatMap (
name:
map (
o: "${name}: ${o.key} ${toString o.seconds}s (${o.source}) >= ${toString (startLimitOf name)}s"
) (lib.filter (o: o.source == "stated" || !allowedUnstated o.key) (reachingLimit name { }))
) hyperhiveContainers;
# The metrics store on a build older than its socket needs. Only its
# `assertions` are read, and the package is a stub carrying a version.
oldMetricsStore = hive {
@ -1294,6 +1398,228 @@ let
&& lib.all namesNoStoreUnit readersElsewhere
&& !(lib.any (u: storeOnlyNoHive.systemd.services ? ${u}) readers);
}
{
# A swarm service container is `Type=notify`: its start job lasts until
# the guest's job queue is empty and is killed at `timeoutStartSec`. A
# unit in that queue whose start job can last that long takes the whole
# container down with it. A unit that states no budget has the manager's
# 90 s default, or none if it is a oneshot, and counts too unless
# `unstatedAllowed` names it. Judged per unit, so a chain of units each
# under the limit is not caught. The offenders are in the name, which is
# printed on failure only.
name = "no unit in a service container's boot wait has a start budget that reaches the container's start limit${
lib.optionalString (bootWaitOffenders != [ ]) ": ${lib.concatStringsSep "; " bootWaitOffenders}"
}";
ok = bootWaitOffenders == [ ];
}
{
# A name that no container's boot wait carries any more is a dead entry,
# and would let a unit of that name back in unseen. The presence arm is
# that each entry is found.
name = "every unit the boot-wait check lets through without a stated budget is in some container's boot wait";
ok =
let
unstated = lib.concatMap (
name: lib.filter (o: o.source != "stated") (reachingLimit name { })
) hyperhiveContainers;
keys = map (o: o.key) unstated;
found = key: lib.elem key keys;
foundSuffix = s: lib.any (k: lib.hasSuffix s k) keys;
in
lib.all (
g: lib.all found (g.units or [ ]) && lib.all foundSuffix (g.suffixes or [ ])
) unstatedAllowed;
}
{
# Names, not `lib.attrNames containers`: a walk over every container
# would also judge an operator's own. Every container the fixture
# declares is in the list, so a new one cannot dodge the check.
name = "the boot-wait check names every service container the fixture declares";
ok =
lib.sort lib.lessThan hyperhiveContainers
== lib.sort lib.lessThan (lib.attrNames serviceContainers);
}
{
# The presence arms for the check above: it reads real units and real
# limits. Each named unit states its budget (30 s, 30 s, 30 s, 20 s, 20 s),
# so it is found at a limit equal to it and not one second above (the
# boundary), and the container limits are the ones the check reads.
name = "the boot-wait check finds the units that state a budget, at the boundary, and reads each container's own limit";
ok =
let
keys = name: limit: map (o: o.key) (reachingLimit name { inherit limit; });
mint = "forgejo-swarm-controller-account.service";
fetch = "swarm-journal-forwarder-oidc.service";
secrets = "authelia-swarm-secrets.service";
sync = "swarm-matrix-appservice-sync.service";
render = "swarm-matrix-appservice-render.service";
in
lib.elem mint (keys "hive-forge" 30)
&& lib.elem fetch (keys "hive-forge" 30)
&& !(lib.elem mint (keys "hive-forge" 31))
&& !(lib.elem fetch (keys "hive-forge" 31))
&& lib.elem secrets (keys "swarm-authelia" 30)
&& !(lib.elem secrets (keys "swarm-authelia" 31))
&& lib.elem sync (keys "hive-matrix" 20)
&& lib.elem render (keys "hive-matrix" 20)
&& !(lib.elem sync (keys "hive-matrix" 21))
&& !(lib.elem render (keys "hive-matrix" 21))
&& startLimitOf "hive-forge" == 60
&& startLimitOf "swarm-bao" == 300;
}
{
# A unit held to a budget that fits the container's start retries with a
# growing delay, not at a fixed one, and the authelia bridge's secret
# fetch states a budget so that the check above judges it.
name = "the store-fetching units in a container's boot wait restart with a growing delay and the bridge's fetch states its budget";
ok =
let
grows =
u:
let
sc = u.serviceConfig;
in
sc.Restart == "on-failure" && sc.RestartSteps > 0 && sc.RestartMaxDelaySec > sc.RestartSec;
authelia = serviceContainers.swarm-authelia.config.systemd.services;
in
grows serviceContainers.hive-forge.config.systemd.services.forgejo-swarm-controller-account
&& grows authelia.authelia-swarm-secrets
&& authelia.swarm-authelia-bridge.serviceConfig.TimeoutStartSec == 10;
}
{
# The forwarder's collector is started `--no-block` by the fetch and is
# not in the boot wait, whatever its own budget; its fetch is. The
# presence arm is the fetch, found by the same walk.
name = "the boot-wait walk includes a unit wanted by multi-user.target and leaves out one started on demand";
ok =
let
booted = bootWait.bootSet serviceContainers.hive-forge.config.systemd;
in
lib.elem "swarm-journal-forwarder-oidc.service" booted
&& lib.elem "forgejo-swarm-controller-account.service" booted
&& !(lib.elem "opentelemetry-collector.service" booted);
}
{
# The walk and the comparison on a hand-made guest: each unit that must
# be found is, and each that must not be is not. Seconds are counted
# from every spelling a unit uses, infinity included.
name = "the boot-wait check flags a budget at or over the limit in the boot wait, pulled in any way, and nothing else";
ok =
let
oneshot = budget: {
serviceConfig = {
Type = "oneshot";
TimeoutStartSec = budget;
};
};
guest = {
systemd = {
targets.multi-user.wants = [ "viaTarget.service" ];
services = {
wantedBy = oneshot "3min" // {
wantedBy = [ "multi-user.target" ];
};
viaRequires = oneshot 120;
parent = {
wantedBy = [ "multi-user.target" ];
requires = [ "viaRequires.service" ];
wants = [ "gate.target" ];
};
viaTarget = oneshot "1min 30s";
viaRequiredBy = oneshot 100 // {
requiredBy = [ "multi-user.target" ];
};
viaGate = oneshot 100 // {
wantedBy = [ "gate.target" ];
};
atLimit = oneshot "1min" // {
wantedBy = [ "multi-user.target" ];
};
forever = oneshot "infinity" // {
wantedBy = [ "multi-user.target" ];
};
belowLimit = oneshot 59 // {
wantedBy = [ "multi-user.target" ];
};
notPulledIn = oneshot 900;
orderedOnly = oneshot 900 // {
after = [ "wantedBy.service" ];
};
forks = {
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "simple";
TimeoutStartSec = 900;
};
};
hook = {
wantedBy = [ "multi-user.target" ];
preStart = "true";
serviceConfig.TimeoutStartSec = "1h";
};
unstated = {
wantedBy = [ "multi-user.target" ];
serviceConfig.Type = "oneshot";
};
defaulted = {
wantedBy = [ "multi-user.target" ];
preStart = "true";
};
};
};
};
in
lib.sort lib.lessThan (
map (o: "${o.key}:${o.source}") (
bootWait.overBudget {
inherit guest;
limit = 60;
}
)
) == [
"atLimit.service:stated"
"defaulted.service:default"
"forever.service:stated"
"hook.service:stated"
"unstated.service:unbounded"
"viaGate.service:stated"
"viaRequiredBy.service:stated"
"viaRequires.service:stated"
"viaTarget.service:stated"
"wantedBy.service:stated"
]
# The default is the manager's: over a limit above it, or with the guest
# setting a shorter one, the unit is not reached, and the unbounded
# oneshot still is.
&& lib.elem "defaulted.service" (
map (o: o.key) (
bootWait.overBudget {
inherit guest;
limit = 90;
}
)
)
&& !(lib.elem "defaulted.service" (
map (o: o.key) (
bootWait.overBudget {
inherit guest;
limit = 91;
}
)
))
&& !(lib.elem "defaulted.service" (
map (o: o.key) (
bootWait.overBudget {
guest = guest // {
systemd = guest.systemd // {
settings.Manager.DefaultTimeoutStartSec = "45s";
};
};
limit = 60;
}
)
));
}
];
in
runGroup "swarm-services-switch" cases