otel: back the hive collector off a failed bind instead of burning its start limit

Every deploy on a hive host, the replacement opentelemetry-collector
reaches bind() while the outgoing process still holds
127.0.0.1:8888 (its self-scrape endpoint). nixpkgs sets
Restart = "always" with no RestartSec, so the unit spends its five
default attempts in under two seconds, hits start-limit-hit and stops
retrying — ~27s of telemetry blackout per deploy.

RestartSec = 5 with a 12-attempt burst over a 120s window rides the
race out instead: the blackout ends within one interval of the port
coming free, and 55s of it being held is survivable where 2s was not.

StartLimitBurst/StartLimitIntervalSec go at the systemd.services attr
level, which NixOS renders into [Unit]; under serviceConfig systemd
ignores them silently. module-eval-hive-otel asserts the placement.
This commit is contained in:
atlas 2026-09-23 12:55:56 +02:00 committed by mara
commit 596c0c17bc
2 changed files with 73 additions and 15 deletions

View file

@ -111,6 +111,27 @@ let
in
!(m ? address) && (lib.head m.readers).pull.exporter.prometheus.port == 8888;
}
{
# A bind that loses the race with the outgoing process is the whole
# failure: nixpkgs ships `Restart = "always"` with no `RestartSec`, so
# without this the unit spends its five default attempts inside two
# seconds and lands in `start-limit-hit`, where it stops retrying. The
# third clause is the one that has to hold — `StartLimit*` are `[Unit]`
# settings that systemd ignores under `[Service]`, so a bound written
# into `serviceConfig` renders, deploys and does nothing. Asserted
# where nixpkgs puts it rather than where it was written, same as
# ./bao-grants.nix.
name = "the hive collector backs off a failed bind from [Unit], not [Service]";
ok =
let
u = hiveOtel.systemd.services.opentelemetry-collector;
in
u.serviceConfig.RestartSec or 0 > 0
&& toString u.unitConfig.StartLimitBurst == "12"
&& !(u.serviceConfig ? StartLimitBurst)
# The window has to outlast every attempt, or the burst is unreachable.
&& u.startLimitIntervalSec or 0 > u.serviceConfig.RestartSec * u.startLimitBurst;
}
{
# Both collectors share a network namespace whenever they are
# co-located, and this port appears in no config the port-collision