mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

collectors: retry a failed start forever with a growing delay instead of hitting start-limit

Browse source
This commit is contained in:
atlas 2026-11-11 00:42:27 +01:00 • committed by mara
commit d48cc83dcb
1 changed files with 101 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

41
nix/host-modules/swarm-otel.nix
View file

@ -730,7 +730,7 @@ in
fi
# `LoadCredential` copies the file at start only, and a collector that
# started without it is in its start limit.
# started without it is in its growing-delay retry loop.
refresh_consumer ${lib.escapeShellArg cfg.machine} opentelemetry-collector.service ${lib.escapeShellArg collectorSecretInContainer}
'';
};
@ -1745,26 +1745,29 @@ in
};
systemd.services.opentelemetry-collector = lib.mkMerge [
# ⚠️ `StartLimit*` are `[Unit]` settings; systemd IGNORES them
# under `[Service]` — in `serviceConfig` they render, deploy and
# do nothing. `checks.module-eval-swarm-otel-core` pins them in
# `unitConfig`, same shape as `checks.module-eval-hive-otel` for
# the sibling host-tier collector. `Restart` is deliberately NOT
# set: nixpkgs' own module defines it at normal priority, so a
# second definition is a module-system conflict.
# The `oidc/*` extensions fetch authelia's discovery document
# through the gateway at startup, and the collector exits 1 when
# that fails. A gateway or authelia that is briefly away must not
# end the retrying: any start limit leaves the unit failed, with no
# receiver sockets, until someone starts it by hand.
#
# This collector's `oidc/*` extensions call out to authelia at
# startup, so a start that races authelia's own restart fails —
# and with systemd's defaults (100ms `RestartSec`, 5-in-10s
# start limit) that burns the whole allowance before authelia is
# back, landing in `start-limit-hit`. The window here (12 × 5s =
# 60s) has to exceed `RestartSec × burst` with margin for that
# drift, same values as `otel.nix`'s host-tier collector, which
# shares the same dependency.
# ⚠️ `StartLimitIntervalSec` is a `[Unit]` setting; systemd IGNORES
# it under `[Service]`, so it is the unit-level option here and
# `checks.module-eval-swarm-otel-core` pins it in `unitConfig`.
# `Restart` is deliberately NOT set: nixpkgs' own module defines it
# as `"always"` at normal priority, so a second definition is a
# module-system conflict.
#
# The delay grows from `RestartSec` to `RestartMaxDelaySec` over
# `RestartSteps` restarts (systemd.service(5)): a long outage costs
# one start a minute.
{
startLimitBurst = 12;
startLimitIntervalSec = 120;
serviceConfig.RestartSec = 5;
startLimitIntervalSec = 0;
serviceConfig = {
RestartSec = 2;
RestartSteps = 6;
RestartMaxDelaySec = 60;
};
}
{