vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

log the queue-connect and identity-mint failures the alert rules match

Browse source 
hh-swarm-queue-connect-failed matched "swarm queue connect failed; this
agent publishes nothing upward", which nothing has logged since the
harness started retrying its first connect and exiting (235bc481b). The
rule now matches the error the harness logs when every attempt has
failed, CONNECT_EXHAUSTED in swarm_queue.rs, and its comment and summary
describe the retry-then-exit behaviour.

A failed MintAgentIdentity node logged nothing: the error reached only
the job graph. mint_identity now warns with the agent and the error, and
hh-agent-identity-mint-failed fires on that line.

Refs #4651
This commit is contained in:
atlas 2026-10-17 01:13:12 +02:00 • committed by mara
commit 9258563f9c
6 changed files with 166 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

9
docs/swarm/README.md
View file

@ -53,7 +53,10 @@ grafana, the controller, authelia, …) →
the agent roster and their wanted state, and the job graph. Creating an agent —
from the swarm UI or `swarmctl agent create --hive <h>` — queues its SSO
identity, forge user, config repo, store identity and matrix account, then
sends the hive a deploy message. A new agent starts `paused`.
sends the hive a deploy message. A new agent starts `paused`. A store identity
that fails to mint logs `swarm jobq: mint_agent_identity failed` with the agent
and the error, which the Grafana alert rule `hh-agent-identity-mint-failed`
fires on. The agent is still created, without a store identity.
`services.hyperhive.deploy.swarm-controller.enable` runs it on this host;
`singleHostSwarm` turns it on. **Otherwise off: set it on the one host that
@ -184,7 +187,9 @@ show the agent nor send it anything, so a harness without one stops:
certificate) exits at boot with an error naming what's missing.
- A harness whose first connect fails tries again after 5 seconds, doubling the
wait each time, and logs a warning for each attempt that fails. After the
seventh, about five minutes in, it logs an error and exits.
seventh, about five minutes in, it logs an error and exits. The error reads
`no swarm queue connection; exiting, since this agent cannot run without one`,
and the Grafana alert rule `hh-swarm-queue-connect-failed` fires on it.
Either way the harness exits and comes back until the queue accepts it. Once
connected, the client reconnects by itself after a dropped connection, reading