vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

log the queue-connect and identity-mint failures the alert rules match

Browse source 
hh-swarm-queue-connect-failed matched "swarm queue connect failed; this
agent publishes nothing upward", which nothing has logged since the
harness started retrying its first connect and exiting (235bc481b). The
rule now matches the error the harness logs when every attempt has
failed, CONNECT_EXHAUSTED in swarm_queue.rs, and its comment and summary
describe the retry-then-exit behaviour.

A failed MintAgentIdentity node logged nothing: the error reached only
the job graph. mint_identity now warns with the agent and the error, and
hh-agent-identity-mint-failed fires on that line.

Refs #4651
This commit is contained in:
atlas 2026-10-17 01:13:12 +02:00 • committed by mara
commit 9258563f9c
6 changed files with 166 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

27
nix/host-modules/swarm-grafana.nix
View file

@ -1497,17 +1497,19 @@ in
for = "5m";
summary = "{{ $labels.hive }}/{{ $labels.agent }} is dropping swarm terminal rows: the harness is not connected to the queue";
})
# Logged once per agent process: the connection is made
# once and never retried, so the agent stays unconnected
# until it restarts. The window is how long the rule
# stays firing after that one line.
# `CONNECT_EXHAUSTED` in hive-agent's swarm_queue.rs,
# logged when the harness's first-connect retries
# (about five minutes) all fail, just before it exits.
# systemd restarts it 30s later, so an agent that never
# connects logs it about every six minutes. The window
# is how long the rule stays firing after the last one.
(logCountRule {
uid = "hh-swarm-queue-connect-failed";
title = "swarm queue connect failed";
windowSeconds = 3600;
expr = ''_time:1h _msg:"swarm queue connect failed; this agent publishes nothing upward" | stats by (hive, agent) count() as failures'';
expr = ''_time:1h _msg:"no swarm queue connection; exiting, since this agent cannot run without one" | stats by (hive, agent) count() as failures'';
for = "0s";
summary = "{{ $labels.hive }}/{{ $labels.agent }} could not connect to the swarm queue and publishes nothing upward until it restarts";
summary = "{{ $labels.hive }}/{{ $labels.agent }} could not connect to the swarm queue and its harness exited; systemd starts it again 30s later";
})
(logCountRule {
uid = "hh-swarm-agent-state-publish-skipped";
@ -1553,6 +1555,19 @@ in
for = "15m";
summary = "swarm-controller has failed the agent credential renewal pass on consecutive ticks";
})
# `MINT_IDENTITY_FAILED` in swarm-controller's main.rs,
# logged once per failed `MintAgentIdentity` job node, at
# agent creation or on a certificate renewal; the window
# is how long the rule stays firing after it. Host
# journal records carry the tracing fields as `F_<NAME>`.
(logCountRule {
uid = "hh-agent-identity-mint-failed";
title = "agent identity mint failed";
windowSeconds = 900;
expr = ''_time:15m _SYSTEMD_UNIT:="swarm-controller.service" _msg:"swarm jobq: mint_agent_identity failed" | stats by (F_AGENT) count() as failures'';
for = "0s";
summary = "swarm-controller failed to mint agent {{ $labels.F_AGENT }}'s store identity; the reason is the `F_ERROR` field of the swarm-controller log line";
})
# Logged once per refused deploy request, by
# `apply_rev` in hive-c0re's swarm_status.rs; the window is
# how long the rule stays firing after it. Host journal