mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

collectors: retry a failed start forever with a growing delay instead of hitting start-limit

Browse source
This commit is contained in:
atlas 2026-11-11 00:42:27 +01:00 • committed by mara
commit d48cc83dcb
1 changed files with 101 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/4-implementation/swarm/telemetry.md
View file

@ -231,6 +231,18 @@ the collector's 4xx included. A switch leaves a window of about 20 seconds in wh
nginx and the container are out of step: the container stops before nginx
restarts and starts about 18 seconds after it. Senders retry through that window.
The collector fetches authelia's discovery document through the gateway when
it starts and exits if that fails. systemd restarts it with no start limit and a
delay that grows from 2 to 60 seconds, so it comes back by itself once the
gateway and authelia answer. If ingest stops, `systemctl status
swarm-otel-sockets` on the host shows whether the receiver sockets are there;
`machinectl shell swarm-otel /run/current-system/sw/bin/journalctl -u
opentelemetry-collector` shows why the collector didn't start. A collector that
keeps retrying never reaches `failed`, so `systemctl --failed` doesn't list it.
The signals are `swarm-otel-sockets` failing and the Grafana alert
`no log records from <hive>`, which fires after about 15 minutes without a log
record from a hive.
Every hive's own collector reaches this one by its gateway name,
`swarm.otel.domain` (default `otel.<swarm domain>`) — the same
by-domain-through-the-gateway shape every other swarm service uses, not a