Watch
0
0
Fork
You've already forked hyperhive
0

hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert

This commit is contained in:
damocles 2026-10-12 22:21:04 +02:00 • committed by mara
commit 02831fb5e9
13 changed files with 214 additions and 224 deletions

View file

@ -359,42 +359,42 @@ the system prompt template.
## Helper events
`Coordinator::notify_agent(agent, &HelperEvent)` pushes the event into
the agent's inbox, and the harness delivers it as a regular `system`
inbox message so it drives a normal claude turn. A "FYI, check when
convenient" event doesn't need a message — hive-c0re publishes those as
swarm lifecycle notices instead (`swarm_notices::notify`). Variants
(`hive_sh4re::manager::HelperEvent`):
`hive_sh4re::manager::HelperEvent` is the wire type for an event urgent
enough to drive an immediate claude turn (delivered as a regular
`system` inbox message, same `recv` path as any other message). It
currently has one variant, `NeedsUpdate { agent }` (sub-agent's
recorded flake rev is stale, operator rebuilds from the dashboard) —
and nothing in the tree constructs it; the push path
(`Coordinator::notify_agent`) was removed once its last caller
(`notify_manager`, used only for the crash event below) went away.
`HelperEvent` is kept as the type for whatever comes along that
genuinely needs to interrupt an agent's turn loop, not because
anything live uses it today.
- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
running container went away with no operator-initiated transient
state (Stopping / Restarting / Destroying / Rebuilding) AND nothing
cleared that transient in the last 30s (`RECENT_TRANSIENT_GRACE`
tombstone, three `POLL_INTERVAL`s — closes the race where a
lifecycle op finishes between two crash-watch polls and the
container shows briefly as "stopped without transient" before
the next start). When `systemctl is-active` fails or runs past its
10s bound on a poll, the container keeps its previous state for that
poll, so an unreadable unit never counts as a crash.
The root agent escalates to the operator, who starts it again from
the dashboard.
- `NeedsUpdate { agent }` — sub-agent's recorded flake rev is
stale. The operator rebuilds it from the dashboard.
**A previously-running container crashing** used to be exactly that
case (`ContainerCrash`, routed to the root agent via
`Coordinator::notify_manager`) — it no longer drives any agent turn at
all. `crash_watch` now only `tracing::warn!`s `"container crash
detected"`; a swarm-level grafana alert matches that log line directly
and pages the operator, with no agent (and no per-hive dashboard
banner) in the loop. See
[`docs/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re).
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
when convenient" events with no reason to drive an immediate turn, so
they go out via `swarm_notices::notify` (see above) instead
they go out via `swarm_notices::notify` instead
of a `HelperEvent`: `subsystem = "core"`, `key = "<event>:<agent>"`,
and a single free-text `summary` (`rebuilt_todo_summary` renders
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
To add a new lifecycle notice: if it needs to drive an immediate turn
(something genuinely urgent, like `ContainerCrash`), add a
`HelperEvent` variant + call sites + update `prompts/system.md`'s
message-event list. If it's "FYI, check when convenient," call
`swarm_notices::notify` instead — no new wire type
needed.
To add a new lifecycle notice: if it's "FYI, check when convenient,"
call `swarm_notices::notify` — no new wire type needed. If it needs to
drive an immediate turn, decide first whether it actually needs an
*agent* in the loop at all (the crash case above didn't, once a
swarm-level alert could carry the same signal) — only reach for a new
`HelperEvent` variant + `notify_agent`-equivalent call site +
`prompts/system.md` update when the answer is yes.
## Autoupdate on startup