hive-c0re: replace the agents_crashing banner with a crash-watcher log line + grafana alert
This commit is contained in:
damocles
mara
parent
6c9c2cef45
commit
02831fb5e9
13 changed files with 214 additions and 224 deletions
|
|
@ -359,42 +359,42 @@ the system prompt template.
|
|||
|
||||
## Helper events
|
||||
|
||||
`Coordinator::notify_agent(agent, &HelperEvent)` pushes the event into
|
||||
the agent's inbox, and the harness delivers it as a regular `system`
|
||||
inbox message so it drives a normal claude turn. A "FYI, check when
|
||||
convenient" event doesn't need a message — hive-c0re publishes those as
|
||||
swarm lifecycle notices instead (`swarm_notices::notify`). Variants
|
||||
(`hive_sh4re::manager::HelperEvent`):
|
||||
`hive_sh4re::manager::HelperEvent` is the wire type for an event urgent
|
||||
enough to drive an immediate claude turn (delivered as a regular
|
||||
`system` inbox message, same `recv` path as any other message). It
|
||||
currently has one variant, `NeedsUpdate { agent }` (sub-agent's
|
||||
recorded flake rev is stale, operator rebuilds from the dashboard) —
|
||||
and nothing in the tree constructs it; the push path
|
||||
(`Coordinator::notify_agent`) was removed once its last caller
|
||||
(`notify_manager`, used only for the crash event below) went away.
|
||||
`HelperEvent` is kept as the type for whatever comes along that
|
||||
genuinely needs to interrupt an agent's turn loop, not because
|
||||
anything live uses it today.
|
||||
|
||||
- `ContainerCrash { agent, note }` — `crash_watch`: a previously-
|
||||
running container went away with no operator-initiated transient
|
||||
state (Stopping / Restarting / Destroying / Rebuilding) AND nothing
|
||||
cleared that transient in the last 30s (`RECENT_TRANSIENT_GRACE`
|
||||
tombstone, three `POLL_INTERVAL`s — closes the race where a
|
||||
lifecycle op finishes between two crash-watch polls and the
|
||||
container shows briefly as "stopped without transient" before
|
||||
the next start). When `systemctl is-active` fails or runs past its
|
||||
10s bound on a poll, the container keeps its previous state for that
|
||||
poll, so an unreadable unit never counts as a crash.
|
||||
The root agent escalates to the operator, who starts it again from
|
||||
the dashboard.
|
||||
- `NeedsUpdate { agent }` — sub-agent's recorded flake rev is
|
||||
stale. The operator rebuilds it from the dashboard.
|
||||
**A previously-running container crashing** used to be exactly that
|
||||
case (`ContainerCrash`, routed to the root agent via
|
||||
`Coordinator::notify_manager`) — it no longer drives any agent turn at
|
||||
all. `crash_watch` now only `tracing::warn!`s `"container crash
|
||||
detected"`; a swarm-level grafana alert matches that log line directly
|
||||
and pages the operator, with no agent (and no per-hive dashboard
|
||||
banner) in the loop. See
|
||||
[`docs/scheduler/observability.md`](../scheduler/observability.md#container-crash-alerting-hive-c0re).
|
||||
|
||||
The remaining lower-urgency lifecycle notices — `Rebuilt`, `Killed`,
|
||||
`Destroyed`, `NeedsLogin`, `LoggedIn` — are "FYI, check
|
||||
when convenient" events with no reason to drive an immediate turn, so
|
||||
they go out via `swarm_notices::notify` (see above) instead
|
||||
they go out via `swarm_notices::notify` instead
|
||||
of a `HelperEvent`: `subsystem = "core"`, `key = "<event>:<agent>"`,
|
||||
and a single free-text `summary` (`rebuilt_todo_summary` renders
|
||||
`Rebuilt`'s `ok`/`note`/`sha`/`tag` fields into that string).
|
||||
|
||||
To add a new lifecycle notice: if it needs to drive an immediate turn
|
||||
(something genuinely urgent, like `ContainerCrash`), add a
|
||||
`HelperEvent` variant + call sites + update `prompts/system.md`'s
|
||||
message-event list. If it's "FYI, check when convenient," call
|
||||
`swarm_notices::notify` instead — no new wire type
|
||||
needed.
|
||||
To add a new lifecycle notice: if it's "FYI, check when convenient,"
|
||||
call `swarm_notices::notify` — no new wire type needed. If it needs to
|
||||
drive an immediate turn, decide first whether it actually needs an
|
||||
*agent* in the loop at all (the crash case above didn't, once a
|
||||
swarm-level alert could carry the same signal) — only reach for a new
|
||||
`HelperEvent` variant + `notify_agent`-equivalent call site +
|
||||
`prompts/system.md` update when the answer is yes.
|
||||
|
||||
## Autoupdate on startup
|
||||
|
||||
|
|
|
|||
|
|
@ -5,8 +5,10 @@ swarm's metrics and log stores and, if you name one, exports upstream; each
|
|||
hive runs a collector of its own that forwards its agents' telemetry there.
|
||||
What flows through it: each agent's Claude Code statistics (token usage,
|
||||
cost, tool call counts) via Claude Code's built-in OpenTelemetry
|
||||
integration, each agent container's journal, and the hyperhive metrics
|
||||
catalogued below.
|
||||
integration, each agent container's journal, the hive tier's own
|
||||
`hive-c0re.service` journal line for a detected container crash (see
|
||||
[Container-crash alerting](#container-crash-alerting-hive-c0re)), and the
|
||||
hyperhive metrics catalogued below.
|
||||
|
||||
## The two collectors
|
||||
|
||||
|
|
@ -327,6 +329,31 @@ credential stays on the swarm tier.
|
|||
|
||||
<!-- vale write-good.Passive = YES -->
|
||||
|
||||
## Container-crash alerting (hive-c0re)
|
||||
|
||||
The crash watcher (`crash_watch.rs`, a 10s poll of every managed container)
|
||||
classifies a container that stopped without an operator-initiated action as
|
||||
a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
|
||||
no broker event, no per-hive dashboard banner. Both of those used to exist;
|
||||
they were removed in favour of this.
|
||||
|
||||
When `otel.enable` is set, the hive tier's own collector runs a `journald`
|
||||
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
|
||||
agent's journal, this collector already shares a filesystem with the unit it
|
||||
reads, being a host systemd service rather than a container) and forwards
|
||||
that one log line alongside the rest of the hive's telemetry. A
|
||||
swarm-level grafana alert, `hh-container-crashed` in
|
||||
[`nix/host-modules/swarm-grafana.nix`](../../nix/host-modules/swarm-grafana.nix),
|
||||
matches it directly and pages the operator — no agent, and no per-hive
|
||||
dashboard, in the loop.
|
||||
|
||||
Requires `services.journald.storage = "persistent"` (the NixOS default) on
|
||||
every hive host with `otel.enable` set — asserted at eval time.
|
||||
|
||||
⚠️ **A hive with `otel.enable` off has no operator-visible crash signal at
|
||||
all** — the per-hive dashboard banner this replaces is gone, and there is no
|
||||
fallback path. Turning on `otel.enable` is what makes a crash reach anyone.
|
||||
|
||||
## Agent-emitted per-turn metrics (`hive-agent`)
|
||||
|
||||
<!-- vale write-good.Passive = NO -->
|
||||
|
|
|
|||
|
|
@ -62,12 +62,13 @@ is opt-in via the P3RM1SS10NS tab.
|
|||
Transient pings (sentinel id 0) have nothing to ack and show no marker.
|
||||
|
||||
**System messages** (from sender `system`): the broker delivers
|
||||
`hive_sh4re::manager::HelperEvent` (two variants: `ContainerCrash`,
|
||||
`NeedsUpdate` — declared, but no call site constructs it) as regular
|
||||
inbox messages (same `recv` path; body is a JSON object with an `event`
|
||||
discriminant field).
|
||||
`ContainerCrash` always goes to `ruth` (`Coordinator::notify_manager`,
|
||||
hardcoded — `hive-c0re/src/workers/crash_watch.rs`). Lifecycle
|
||||
`hive_sh4re::manager::HelperEvent` (one variant, `NeedsUpdate` —
|
||||
declared, but no call site constructs it) as regular inbox messages
|
||||
(same `recv` path; body is a JSON object with an `event` discriminant
|
||||
field). A container crash used to be the other variant
|
||||
(`ContainerCrash`, routed to `ruth`) but that path is gone — a crash
|
||||
now only produces a log line + swarm-level grafana alert, no agent
|
||||
turn at all (`docs/scheduler/observability.md`). Lifecycle
|
||||
transitions the job-queue scheduler or crash watcher drive directly —
|
||||
stop/kill, destroy, a flake-rev login or logout state change — reach
|
||||
no individual agent: they publish onto a swarm-wide NATS
|
||||
|
|
|
|||
|
|
@ -40,9 +40,13 @@ from the dashboard tab strip.
|
|||
that registry. Producers today include the host disk-pressure check
|
||||
(a `statvfs` probe of `/nix`: ≥85% used → `warn`, ≥95% → `crit`, e.g.
|
||||
`⚠ host nix store N% full (G GiB free) — garbage-collect …`), forge
|
||||
provisioning/CI-runner boot failures, and agent-state warnings
|
||||
(`pending_logins`, `agents_crashing`) computed alongside the container
|
||||
snapshot. Hidden when there are no warnings.
|
||||
provisioning/CI-runner boot failures, and an agent-state warning
|
||||
(`pending_logins`) computed alongside the container snapshot. A
|
||||
container crash used to have a sibling `agents_crashing` entry here —
|
||||
removed in favour of a swarm-level grafana alert on the crash
|
||||
watcher's own log line (`docs/scheduler/observability.md`), so a
|
||||
hive with no otel export now has no operator-visible crash signal at
|
||||
all. Hidden when there are no warnings.
|
||||
- **Browser tab title** — `hyperhive // h1ve-c0re` by default; updated
|
||||
to `<swarm> / <hive> // h1ve-c0re` once `hive_name` / `swarm_name`
|
||||
arrive in the state snapshot.
|
||||
|
|
|
|||
Loading…
Reference in a new issue