vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert

Browse source
This commit is contained in:
damocles 2026-10-12 22:21:04 +02:00 • committed by mara
commit 02831fb5e9
13 changed files with 214 additions and 224 deletions
Download patch file Download diff file Expand all files Collapse all files

31
docs/scheduler/observability.md
View file

@ -5,8 +5,10 @@ swarm's metrics and log stores and, if you name one, exports upstream; each
hive runs a collector of its own that forwards its agents' telemetry there.
What flows through it: each agent's Claude Code statistics (token usage,
cost, tool call counts) via Claude Code's built-in OpenTelemetry
integration, each agent container's journal, and the hyperhive metrics
catalogued below.
integration, each agent container's journal, the hive tier's own
`hive-c0re.service` journal line for a detected container crash (see
[Container-crash alerting](#container-crash-alerting-hive-c0re)), and the
hyperhive metrics catalogued below.
## The two collectors
@ -327,6 +329,31 @@ credential stays on the swarm tier.
<!-- vale write-good.Passive = YES -->
## Container-crash alerting (hive-c0re)
The crash watcher (`crash_watch.rs`, a 10s poll of every managed container)
classifies a container that stopped without an operator-initiated action as
a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
no broker event, no per-hive dashboard banner. Both of those used to exist;
they were removed in favour of this.
When `otel.enable` is set, the hive tier's own collector runs a `journald`
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
agent's journal, this collector already shares a filesystem with the unit it
reads, being a host systemd service rather than a container) and forwards
that one log line alongside the rest of the hive's telemetry. A
swarm-level grafana alert, `hh-container-crashed` in
[`nix/host-modules/swarm-grafana.nix`](../../nix/host-modules/swarm-grafana.nix),
matches it directly and pages the operator — no agent, and no per-hive
dashboard, in the loop.
Requires `services.journald.storage = "persistent"` (the NixOS default) on
every hive host with `otel.enable` set — asserted at eval time.
⚠️ **A hive with `otel.enable` off has no operator-visible crash signal at
all** — the per-hive dashboard banner this replaces is gone, and there is no
fallback path. Turning on `otel.enable` is what makes a crash reach anyone.
## Agent-emitted per-turn metrics (`hive-agent`)
<!-- vale write-good.Passive = NO -->