hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert
This commit is contained in:
parent
6c9c2cef45
commit
02831fb5e9
13 changed files with 214 additions and 224 deletions
|
|
@ -359,42 +359,42 @@ the system prompt template.
|
|||
|
||||
## Helper events
|
||||
|
||||
`Coordinator::notify_agent(agent, &HelperEvent)` pushes the event into
|
||||
the agent's inbox, and the harness delivers it as a regular `system`
|
||||
inbox message so it drives a normal claude turn. A "FYI, check when
|
||||
convenient" event doesn't need a message — hive-c0re publishes those as
|
||||
swarm lifecycle notices instead (`swarm_notices::notify`). Variants
|
||||
(`hive_sh4re::manager::HelperEvent`):
|
||||
`hive_sh4re::manager::HelperEvent` is the wire type for an event urgent
|
||||
enough to drive an immediate claude turn (delivered as a regular
|
||||
`system` inbox message, same `recv` path as any other message). It
|
||||
currently has one variant, `NeedsUpdate { agent }` (sub-agent's
|
||||
recorded flake rev is stale, operator rebuilds from the dashboard) —
|
||||
and nothing in the tree constructs it; the push path
|
||||
(`Coordinator::notify_agent`) was removed once its last caller
|
||||
(`notify_manager`, used only for the crash event below) went away.
|
||||
`HelperEvent` is kept as the type for whatever comes along that
|
||||
genuinely needs to interrupt an agent's turn loop, not because
|
||||
anything live uses it today.
|
||||
|
||||
- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
|
||||
running container went away with no operator-initiated transient
|
||||
state (Stopping / Restarting / Destroying / Rebuilding) AND nothing
|
||||
cleared that transient in the last 30s (`RECENT_TRANSIENT_GRACE`
|
||||
tombstone, three `POLL_INTERVAL`s — closes the race where a
|
||||
lifecycle op finishes between two crash-watch polls and the
|
||||
container shows briefly as "stopped without transient" before
|
||||
the next start). When `systemctl is-active` fails or runs past its
|
||||
10s bound on a poll, the container keeps its previous state for that
|
||||
poll, so an unreadable unit never counts as a crash.
|
||||
The root agent escalates to the operator, who starts it again from
|
||||
the dashboard.
|
||||
- `NeedsUpdate { agent }` — sub-agent's recorded flake rev is
|
||||
stale. The operator rebuilds it from the dashboard.
|
||||
**A previously-running container crashing** used to be exactly that
|
||||
case (`ContainerCrash`, routed to the root agent via
|
||||
`Coordinator::notify_manager`) — it no longer drives any agent turn at
|
||||
all. `crash_watch` now only `tracing::warn!`s `"container crash
|
||||
detected"`; a swarm-level grafana alert matches that log line directly
|
||||
and pages the operator, with no agent (and no per-hive dashboard
|
||||
banner) in the loop. See
|
||||
[`docs/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re).
|
||||
|
||||
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
||||
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
||||
when convenient" events with no reason to drive an immediate turn, so
|
||||
they go out via `swarm_notices::notify` (see above) instead
|
||||
they go out via `swarm_notices::notify` instead
|
||||
of a `HelperEvent`: `subsystem = "core"`, `key = "<event>:<agent>"`,
|
||||
and a single free-text `summary` (`rebuilt_todo_summary` renders
|
||||
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
|
||||
|
||||
To add a new lifecycle notice: if it needs to drive an immediate turn
|
||||
(something genuinely urgent, like `ContainerCrash`), add a
|
||||
`HelperEvent` variant + call sites + update `prompts/system.md`'s
|
||||
message-event list. If it's "FYI, check when convenient," call
|
||||
`swarm_notices::notify` instead — no new wire type
|
||||
needed.
|
||||
To add a new lifecycle notice: if it's "FYI, check when convenient,"
|
||||
call `swarm_notices::notify` — no new wire type needed. If it needs to
|
||||
drive an immediate turn, decide first whether it actually needs an
|
||||
*agent* in the loop at all (the crash case above didn't, once a
|
||||
swarm-level alert could carry the same signal) — only reach for a new
|
||||
`HelperEvent` variant + `notify_agent`-equivalent call site +
|
||||
`prompts/system.md` update when the answer is yes.
|
||||
|
||||
## Autoupdate on startup
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue