otel: back the hive collector off a failed bind instead of burning its start limit
Every deploy on a hive host, the replacement opentelemetry-collector reaches bind() while the outgoing process still holds 127.0.0.1:8888 (its self-scrape endpoint). nixpkgs sets Restart = "always" with no RestartSec, so the unit spends its five default attempts in under two seconds, hits start-limit-hit and stops retrying — ~27s of telemetry blackout per deploy. RestartSec = 5 with a 12-attempt burst over a 120s window rides the race out instead: the blackout ends within one interval of the port coming free, and 55s of it being held is survivable where 2s was not. StartLimitBurst/StartLimitIntervalSec go at the systemd.services attr level, which NixOS renders into [Unit]; under serviceConfig systemd ignores them silently. module-eval-hive-otel asserts the placement.
This commit is contained in:
parent
2c066cc871
commit
596c0c17bc
2 changed files with 73 additions and 15 deletions
|
|
@ -626,21 +626,58 @@ in
|
|||
};
|
||||
};
|
||||
|
||||
# `LoadCredential` and not a copy-oneshot: this collector is a HOST
|
||||
# unit, so there is no container boundary to cross and therefore no
|
||||
# reason for a second on-disk copy of the secret. systemd hands it to
|
||||
# the process in a private tmpfs and exports the directory, which is
|
||||
# what the config above names.
|
||||
systemd.services.opentelemetry-collector = lib.optionalAttrs senderAuth {
|
||||
serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
|
||||
systemd.services.opentelemetry-collector = lib.mkMerge [
|
||||
# Survive a restart that races the outgoing process off
|
||||
# `telemetryPort`. Every deploy, the replacement collector has been
|
||||
# reaching `bind()` while the old one still holds `127.0.0.1:8888`,
|
||||
# and nixpkgs' `Restart = "always"` with no `RestartSec` turns that
|
||||
# into the worst possible response: five retries inside two seconds,
|
||||
# `start-limit-hit`, and no telemetry at all until something else
|
||||
# pokes the unit. Measured at ~27s of blackout per deploy.
|
||||
#
|
||||
# 🔑 Same inversion the credential wait below records — `always`
|
||||
# reads like it makes this self-healing and does the opposite. A
|
||||
# FAST-failing unit exhausts its allowance before the thing it is
|
||||
# waiting for (here, an exiting process letting go of a socket) can
|
||||
# possibly have happened. Nothing here removes that race; it makes
|
||||
# the unit outlast it instead.
|
||||
#
|
||||
# Sized for a socket held by a process already on its way out —
|
||||
# seconds, not an unseal's hours. 5s ends the blackout within one
|
||||
# interval of the port coming free; 12 attempts ride out 55s of it
|
||||
# being held, twice the worst gap observed. `Restart` itself is
|
||||
# deliberately NOT set, same as both container collectors: nixpkgs'
|
||||
# own module defines it at the normal priority, so a second
|
||||
# definition is a conflict the module system refuses.
|
||||
#
|
||||
# ⚠️ `StartLimit*` are `[Unit]` settings and systemd IGNORES them
|
||||
# under `[Service]` — written into `serviceConfig` they render,
|
||||
# deploy and do nothing. These two options are the `[Unit]`
|
||||
# spelling; `checks.module-eval-hive-otel` asserts they land in
|
||||
# `unitConfig` and not beside `RestartSec`. The window has to
|
||||
# exceed `RestartSec × burst`.
|
||||
{
|
||||
startLimitBurst = 12;
|
||||
startLimitIntervalSec = 120;
|
||||
serviceConfig.RestartSec = 5;
|
||||
}
|
||||
|
||||
# `requires`, not merely `after`: ordering without gating leaves the
|
||||
# collector starting anyway and failing on its own `LoadCredential`,
|
||||
# which is the failure the wait exists to prevent. Same reasoning
|
||||
# `lib/hive-ca-trust.nix` records for its bundle consumers.
|
||||
requires = [ "${secretWaitUnit}.service" ];
|
||||
after = [ "${secretWaitUnit}.service" ];
|
||||
};
|
||||
# `LoadCredential` and not a copy-oneshot: this collector is a HOST
|
||||
# unit, so there is no container boundary to cross and therefore no
|
||||
# reason for a second on-disk copy of the secret. systemd hands it to
|
||||
# the process in a private tmpfs and exports the directory, which is
|
||||
# what the config above names.
|
||||
(lib.mkIf senderAuth {
|
||||
serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
|
||||
|
||||
# `requires`, not merely `after`: ordering without gating leaves the
|
||||
# collector starting anyway and failing on its own `LoadCredential`,
|
||||
# which is the failure the wait exists to prevent. Same reasoning
|
||||
# `lib/hive-ca-trust.nix` records for its bundle consumers.
|
||||
requires = [ "${secretWaitUnit}.service" ];
|
||||
after = [ "${secretWaitUnit}.service" ];
|
||||
})
|
||||
];
|
||||
|
||||
# Wait for the credential rather than racing it.
|
||||
#
|
||||
|
|
|
|||
Loading…
Reference in a new issue