otel: back the hive collector off a failed bind instead of burning its start limit
Every deploy on a hive host, the replacement opentelemetry-collector reaches bind() while the outgoing process still holds 127.0.0.1:8888 (its self-scrape endpoint). nixpkgs sets Restart = "always" with no RestartSec, so the unit spends its five default attempts in under two seconds, hits start-limit-hit and stops retrying — ~27s of telemetry blackout per deploy. RestartSec = 5 with a 12-attempt burst over a 120s window rides the race out instead: the blackout ends within one interval of the port coming free, and 55s of it being held is survivable where 2s was not. StartLimitBurst/StartLimitIntervalSec go at the systemd.services attr level, which NixOS renders into [Unit]; under serviceConfig systemd ignores them silently. module-eval-hive-otel asserts the placement.
This commit is contained in:
parent
2c066cc871
commit
596c0c17bc
2 changed files with 73 additions and 15 deletions
|
|
@ -111,6 +111,27 @@ let
|
|||
in
|
||||
!(m ? address) && (lib.head m.readers).pull.exporter.prometheus.port == 8888;
|
||||
}
|
||||
{
|
||||
# A bind that loses the race with the outgoing process is the whole
|
||||
# failure: nixpkgs ships `Restart = "always"` with no `RestartSec`, so
|
||||
# without this the unit spends its five default attempts inside two
|
||||
# seconds and lands in `start-limit-hit`, where it stops retrying. The
|
||||
# third clause is the one that has to hold — `StartLimit*` are `[Unit]`
|
||||
# settings that systemd ignores under `[Service]`, so a bound written
|
||||
# into `serviceConfig` renders, deploys and does nothing. Asserted
|
||||
# where nixpkgs puts it rather than where it was written, same as
|
||||
# ./bao-grants.nix.
|
||||
name = "the hive collector backs off a failed bind from [Unit], not [Service]";
|
||||
ok =
|
||||
let
|
||||
u = hiveOtel.systemd.services.opentelemetry-collector;
|
||||
in
|
||||
u.serviceConfig.RestartSec or 0 > 0
|
||||
&& toString u.unitConfig.StartLimitBurst == "12"
|
||||
&& !(u.serviceConfig ? StartLimitBurst)
|
||||
# The window has to outlast every attempt, or the burst is unreachable.
|
||||
&& u.startLimitIntervalSec or 0 > u.serviceConfig.RestartSec * u.startLimitBurst;
|
||||
}
|
||||
{
|
||||
# Both collectors share a network namespace whenever they are
|
||||
# co-located, and this port appears in no config the port-collision
|
||||
|
|
|
|||
Loading…
Reference in a new issue