diff --git a/docs/agent-lifecycle/config-changes.md b/docs/agent-lifecycle/config-changes.md index c22725a1..0cf83813 100644 --- a/docs/agent-lifecycle/config-changes.md +++ b/docs/agent-lifecycle/config-changes.md @@ -364,10 +364,10 @@ enough to drive an immediate claude turn (delivered as a regular `system` inbox message, same `recv` path as any other message). It currently has one variant, `NeedsUpdate { agent }` (sub-agent's recorded flake rev is stale, operator rebuilds from the dashboard) — -and nothing in the tree constructs it; the push path -(`Coordinator::notify_agent`) was removed once its last caller +and nothing in the tree constructs it; this removal dropped the push path +(`Coordinator::notify_agent`) once its last caller (`notify_manager`, used only for the crash event below) went away. -`HelperEvent` is kept as the type for whatever comes along that +`HelperEvent` stays as the type for whatever comes along that genuinely needs to interrupt an agent's turn loop, not because anything live uses it today. diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index bad51fd5..73d3b60a 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -327,17 +327,15 @@ Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP first hop is unauthenticated for every producer on this host, and the upstream credential stays on the swarm tier. - - ## Container-crash alerting (hive-c0re) The crash watcher (`crash_watch.rs`, a 10s poll of every managed container) classifies a container that stopped without an operator-initiated action as a crash and `tracing::warn!`s `"container crash detected"` — nothing else: no broker event, no per-hive dashboard banner. Both of those used to exist; -they were removed in favour of this. +this change removed them in favour of the log line. -When `otel.enable` is set, the hive tier's own collector runs a `journald` +When `otel.enable` is on, the hive tier's own collector runs a `journald` receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an agent's journal, this collector already shares a filesystem with the unit it reads, being a host systemd service rather than a container) and forwards @@ -351,8 +349,9 @@ Requires `services.journald.storage = "persistent"` (the NixOS default) on every hive host with `otel.enable` set — asserted at eval time. ⚠️ **A hive with `otel.enable` off has no operator-visible crash signal at -all** — the per-hive dashboard banner this replaces is gone, and there is no -fallback path. Turning on `otel.enable` is what makes a crash reach anyone. +all** — the per-hive dashboard banner this replaces no longer exists, and +there is no fallback path. Turning on `otel.enable` is what makes a crash +reach anyone. ## Agent-emitted per-turn metrics (`hive-agent`) diff --git a/docs/turn-loop/mcp.md b/docs/turn-loop/mcp.md index 146d7802..bd6c04fe 100644 --- a/docs/turn-loop/mcp.md +++ b/docs/turn-loop/mcp.md @@ -66,9 +66,9 @@ is opt-in via the P3RM1SS10NS tab. declared, but no call site constructs it) as regular inbox messages (same `recv` path; body is a JSON object with an `event` discriminant field). A container crash used to be the other variant -(`ContainerCrash`, routed to `ruth`) but that path is gone — a crash -now only produces a log line + swarm-level grafana alert, no agent -turn at all (`docs/scheduler/observability.md`). Lifecycle +(`ContainerCrash`, routed to `ruth`) but this tree no longer has that +path — a crash now only produces a log line + swarm-level grafana +alert, no agent turn at all (`docs/scheduler/observability.md`). Lifecycle transitions the job-queue scheduler or crash watcher drive directly — stop/kill, destroy, a flake-rev login or logout state change — reach no individual agent: they publish onto a swarm-wide NATS diff --git a/nix/module-eval/hive-otel.nix b/nix/module-eval/hive-otel.nix index 773eaaa6..99a379b3 100644 --- a/nix/module-eval/hive-otel.nix +++ b/nix/module-eval/hive-otel.nix @@ -80,14 +80,21 @@ let # The tier in the middle. Its OTLP receiver takes both signals on one # port, so without this pipeline an agent's push is answered 404 on # `/v1/logs` — and a forwarder retrying into a 404 is indistinguish- - # able from one with nothing to send. Compared against the metrics - # pipeline's exporters rather than a name, so the two signals cannot - # drift to different destinations. + # able from one with nothing to send. `journald` joined the receiver + # list for this collector's own host journal (hive-c0re's crash-watch + # log line, see ./otel.nix) — both have to be present, in either + # order, so this stays a set check rather than a list-equality one. + # Compared against the metrics pipeline's exporters rather than a + # name, so the two signals cannot drift to different destinations. name = "the hive collector forwards logs upstream, not only metrics"; ok = (hiveOtelPipelines ? logs) && hiveOtelPipelines.metrics.exporters != [ ] - && hiveOtelPipelines.logs.receivers == [ "otlp" ] + && + lib.sort builtins.lessThan hiveOtelPipelines.logs.receivers == [ + "journald" + "otlp" + ] && hiveOtelPipelines.logs.exporters == hiveOtelPipelines.metrics.exporters; } {