vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert

Browse source
This commit is contained in:
damocles 2026-10-12 22:21:04 +02:00 • committed by mara
commit 02831fb5e9
13 changed files with 214 additions and 224 deletions
Download patch file Download diff file Expand all files Collapse all files

18
nix/host-modules/swarm-grafana.nix
View file

@ -1176,6 +1176,24 @@ in
for = "0s";
summary = "{{ $labels._HOSTNAME }} refused to deploy merged config commit {{ $labels.F_REV }} of agent {{ $labels.F_AGENT }}; the reason is the `F_ERROR` field of the hive-c0re log line";
})
# Logged once per detected crash, by `emit_crash_transitions`
# in hive-c0re's crash_watch.rs (10s poll interval). Replaces
# the old per-hive dashboard `agents_crashing` banner — see
# that function's module doc for why the banner went away
# in favour of this. Host journal records carry the
# tracing fields as `F_<NAME>` and no `hive` attribute,
# so the machine is `_HOSTNAME`
# (same shape as `hh-config-rev-refused` above). The unit
# filter keeps out an agent's own log line if one ever
# quotes the same message.
(logCountRule {
uid = "hh-container-crashed";
title = "container crashed";
windowSeconds = 900;
expr = ''_time:15m _SYSTEMD_UNIT:="hive-c0re.service" _msg:"container crash detected" | stats by (_HOSTNAME, F_AGENT) count() as crashes'';
for = "0s";
summary = "{{ $labels._HOSTNAME }} detected agent {{ $labels.F_AGENT }}'s container crash (not an operator-initiated stop)";
})
]
# One rule per declared hive rather than one grouped by
# `hive`, so a hive that sends nothing at all still has