Watch
0
0
Fork
You've already forked hyperhive
0

Compare commits

..
Author SHA1 Message Date
atlas
5cec69bd21 otel.nix: trim the StartLimit comment block to the load-bearing points
Cut ~27 lines of blackout-measurement and cross-reference narrative
(already in the PR body / issue) down to the three things a reader
actually needs at this call site: the [Unit]-vs-[Service] trap, why
Restart is absent, and the window-vs-burst constraint.
2026-09-23 17:22:51 +02:00
atlas
596c0c17bc otel: back the hive collector off a failed bind instead of burning its start limit
Every deploy on a hive host, the replacement opentelemetry-collector
reaches bind() while the outgoing process still holds
127.0.0.1:8888 (its self-scrape endpoint). nixpkgs sets
Restart = "always" with no RestartSec, so the unit spends its five
default attempts in under two seconds, hits start-limit-hit and stops
retrying — ~27s of telemetry blackout per deploy.

RestartSec = 5 with a 12-attempt burst over a 120s window rides the
race out instead: the blackout ends within one interval of the port
coming free, and 55s of it being held is survivable where 2s was not.

StartLimitBurst/StartLimitIntervalSec go at the systemd.services attr
level, which NixOS renders into [Unit]; under serviceConfig systemd
ignores them silently. module-eval-hive-otel asserts the placement.
2026-09-23 17:22:51 +02:00
2 changed files with 50 additions and 15 deletions

View file

@ -626,21 +626,35 @@ in
};
};
# `LoadCredential` and not a copy-oneshot: this collector is a HOST
# unit, so there is no container boundary to cross and therefore no
# reason for a second on-disk copy of the secret. systemd hands it to
# the process in a private tmpfs and exports the directory, which is
# what the config above names.
systemd.services.opentelemetry-collector = lib.optionalAttrs senderAuth {
serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
systemd.services.opentelemetry-collector = lib.mkMerge [
# ⚠️ `StartLimit*` are `[Unit]` settings; systemd IGNORES them under
# `[Service]` — in `serviceConfig` they render, deploy and do
# nothing. `checks.module-eval-hive-otel` pins them in `unitConfig`.
# `Restart` is deliberately NOT set: nixpkgs' own module defines it
# at normal priority, so a second definition is a module-system
# conflict. The window must exceed `RestartSec × burst`.
{
startLimitBurst = 12;
startLimitIntervalSec = 120;
serviceConfig.RestartSec = 5;
}
# `requires`, not merely `after`: ordering without gating leaves the
# collector starting anyway and failing on its own `LoadCredential`,
# which is the failure the wait exists to prevent. Same reasoning
# `lib/hive-ca-trust.nix` records for its bundle consumers.
requires = [ "${secretWaitUnit}.service" ];
after = [ "${secretWaitUnit}.service" ];
};
# `LoadCredential` and not a copy-oneshot: this collector is a HOST
# unit, so there is no container boundary to cross and therefore no
# reason for a second on-disk copy of the secret. systemd hands it to
# the process in a private tmpfs and exports the directory, which is
# what the config above names.
(lib.mkIf senderAuth {
serviceConfig.LoadCredential = [ "${credName}:${otel.clientSecretFile}" ];
# `requires`, not merely `after`: ordering without gating leaves the
# collector starting anyway and failing on its own `LoadCredential`,
# which is the failure the wait exists to prevent. Same reasoning
# `lib/hive-ca-trust.nix` records for its bundle consumers.
requires = [ "${secretWaitUnit}.service" ];
after = [ "${secretWaitUnit}.service" ];
})
];
# Wait for the credential rather than racing it.
#

View file

@ -111,6 +111,27 @@ let
in
!(m ? address) && (lib.head m.readers).pull.exporter.prometheus.port == 8888;
}
{
# A bind that loses the race with the outgoing process is the whole
# failure: nixpkgs ships `Restart = "always"` with no `RestartSec`, so
# without this the unit spends its five default attempts inside two
# seconds and lands in `start-limit-hit`, where it stops retrying. The
# third clause is the one that has to hold — `StartLimit*` are `[Unit]`
# settings that systemd ignores under `[Service]`, so a bound written
# into `serviceConfig` renders, deploys and does nothing. Asserted
# where nixpkgs puts it rather than where it was written, same as
# ./bao-grants.nix.
name = "the hive collector backs off a failed bind from [Unit], not [Service]";
ok =
let
u = hiveOtel.systemd.services.opentelemetry-collector;
in
u.serviceConfig.RestartSec or 0 > 0
&& toString u.unitConfig.StartLimitBurst == "12"
&& !(u.serviceConfig ? StartLimitBurst)
# The window has to outlast every attempt, or the burst is unreachable.
&& u.startLimitIntervalSec or 0 > u.serviceConfig.RestartSec * u.startLimitBurst;
}
{
# Both collectors share a network namespace whenever they are
# co-located, and this port appears in no config the port-collision