vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-otel: bound the collector's restart backoff

Browse source 
The collector's oidc/* authenticators call out to authelia at startup, so a
restart that races authelia's own (a redeploy that touches both, a store
outage) can fail immediately. nixpkgs' upstream opentelemetry-collector
module sets Restart=always with no RestartSec, so systemd's defaults
(100ms RestartSec, 5-in-10s start limit) burn the whole allowance in well
under a second and leave the unit in start-limit-hit, dead until someone
resets it by hand.

Sets RestartSec=5 plus an explicit startLimitBurst/startLimitIntervalSec
window (12/120s) sized so the burst can never trip while authelia comes
back — same values host-modules/otel.nix already uses for the sibling
host-tier collector, which depends on authelia the same way. Pins the
[Unit]-vs-[Service] placement and the window relation in
module-eval-swarm-otel-core, mirroring module-eval-hive-otel's existing
case for the host tier.
This commit is contained in:
atlas 2026-09-29 00:02:24 +02:00 • committed by mara
commit 916c441e82
2 changed files with 57 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

46
nix/host-modules/swarm-otel.nix
View file

@ -1757,14 +1757,37 @@ in
};
};
# The credential file is already `NAME=value`, systemd's
# EnvironmentFile format — so the secret reaches the process as an
# environment variable without being read by nix, written to the
# store, or passed in argv.
systemd.services.opentelemetry-collector.serviceConfig =
lib.optionalAttrs (otelCfg.headersCredential != null) {
EnvironmentFile = otelCfg.headersCredential;
systemd.services.opentelemetry-collector = lib.mkMerge [
# ⚠️ `StartLimit*` are `[Unit]` settings; systemd IGNORES them
# under `[Service]` — in `serviceConfig` they render, deploy and
# do nothing. `checks.module-eval-swarm-otel-core` pins them in
# `unitConfig`, same shape as `checks.module-eval-hive-otel` for
# the sibling host-tier collector. `Restart` is deliberately NOT
# set: nixpkgs' own module defines it at normal priority, so a
# second definition is a module-system conflict.
#
# This collector's `oidc/*` extensions call out to authelia at
# startup, so a start that races authelia's own restart fails —
# and with systemd's defaults (100ms `RestartSec`, 5-in-10s
# start limit) that burns the whole allowance before authelia is
# back, landing in `start-limit-hit`. The window here (12 × 5s =
# 60s) has to exceed `RestartSec × burst` with margin for that
# drift, same values as `otel.nix`'s host-tier collector, which
# shares the same dependency.
{
startLimitBurst = 12;
startLimitIntervalSec = 120;
serviceConfig.RestartSec = 5;
}
# The credential file is already `NAME=value`, systemd's
# EnvironmentFile format — so the secret reaches the process as
# an environment variable without being read by nix, written to
# the store, or passed in argv.
(lib.optionalAttrs (otelCfg.headersCredential != null) {
serviceConfig.EnvironmentFile = otelCfg.headersCredential;
})
# The other credential, and the other direction: the one above
# authenticates this collector's export onward, this one
# authenticates it to a service it scrapes.
@ -1776,9 +1799,12 @@ in
# exactly rather than restating a narrower one. Where it is false
# no authenticator is rendered either, so the collector starts and
# is refused by the stores rather than failing to start.
// lib.optionalAttrs haveCollectorSecret {
LoadCredential = [ "${collectorCredentialId}:${deployCfg.swarm-otel.clientSecretFile}" ];
};
(lib.optionalAttrs haveCollectorSecret {
serviceConfig.LoadCredential = [
"${collectorCredentialId}:${deployCfg.swarm-otel.clientSecretFile}"
];
})
];
};
};
};

21
nix/module-eval/swarm-otel-core.nix
View file

@ -223,6 +223,27 @@ let
name = "the swarm collector's journald receiver carries the shared PRIORITY mapping";
ok = carriesJournaldSeverity (otelSettings otelNoStores).receivers.journald;
}
{
# Sibling of ./hive-otel.nix's identical case for the host-tier
# collector: this one's `oidc/*` extensions call out to authelia at
# startup, so it fails the same way when that restart races its own.
# nixpkgs ships `Restart = "always"` with no `RestartSec`, so without
# this the unit burns its five default attempts inside two seconds and
# lands in `start-limit-hit`, where it stops retrying. The third clause
# is the one that has to hold — `StartLimit*` are `[Unit]` settings
# that systemd ignores under `[Service]`, so a bound written into
# `serviceConfig` renders, deploys and does nothing.
name = "the swarm collector backs off a failed bind from [Unit], not [Service]";
ok =
let
u = otelNoStores.containers.swarm-otel.config.systemd.services.opentelemetry-collector;
in
u.serviceConfig.RestartSec or 0 > 0
&& toString u.unitConfig.StartLimitBurst == "12"
&& !(u.serviceConfig ? StartLimitBurst)
# The window has to outlast every attempt, or the burst is unreachable.
&& u.startLimitIntervalSec or 0 > u.serviceConfig.RestartSec * u.startLimitBurst;
}
];
in
runGroup "swarm-otel-core" cases