fix passive-voice vale errors in docs + hive-otel module-eval test for the new journald receiver
This commit is contained in:
parent
02831fb5e9
commit
8e58ad8717
4 changed files with 22 additions and 16 deletions
|
|
@ -364,10 +364,10 @@ enough to drive an immediate claude turn (delivered as a regular
|
||||||
`system` inbox message, same `recv` path as any other message). It
|
`system` inbox message, same `recv` path as any other message). It
|
||||||
currently has one variant, `NeedsUpdate { agent }` (sub-agent's
|
currently has one variant, `NeedsUpdate { agent }` (sub-agent's
|
||||||
recorded flake rev is stale, operator rebuilds from the dashboard) —
|
recorded flake rev is stale, operator rebuilds from the dashboard) —
|
||||||
and nothing in the tree constructs it; the push path
|
and nothing in the tree constructs it; this removal dropped the push path
|
||||||
(`Coordinator::notify_agent`) was removed once its last caller
|
(`Coordinator::notify_agent`) once its last caller
|
||||||
(`notify_manager`, used only for the crash event below) went away.
|
(`notify_manager`, used only for the crash event below) went away.
|
||||||
`HelperEvent` is kept as the type for whatever comes along that
|
`HelperEvent` stays as the type for whatever comes along that
|
||||||
genuinely needs to interrupt an agent's turn loop, not because
|
genuinely needs to interrupt an agent's turn loop, not because
|
||||||
anything live uses it today.
|
anything live uses it today.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -327,17 +327,15 @@ Cadence follows `metricIntervalMs` (default 60s). Transport is OTLP/HTTP
|
||||||
first hop is unauthenticated for every producer on this host, and the upstream
|
first hop is unauthenticated for every producer on this host, and the upstream
|
||||||
credential stays on the swarm tier.
|
credential stays on the swarm tier.
|
||||||
|
|
||||||
<!-- vale write-good.Passive = YES -->
|
|
||||||
|
|
||||||
## Container-crash alerting (hive-c0re)
|
## Container-crash alerting (hive-c0re)
|
||||||
|
|
||||||
The crash watcher (`crash_watch.rs`, a 10s poll of every managed container)
|
The crash watcher (`crash_watch.rs`, a 10s poll of every managed container)
|
||||||
classifies a container that stopped without an operator-initiated action as
|
classifies a container that stopped without an operator-initiated action as
|
||||||
a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
|
a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
|
||||||
no broker event, no per-hive dashboard banner. Both of those used to exist;
|
no broker event, no per-hive dashboard banner. Both of those used to exist;
|
||||||
they were removed in favour of this.
|
this change removed them in favour of the log line.
|
||||||
|
|
||||||
When `otel.enable` is set, the hive tier's own collector runs a `journald`
|
When `otel.enable` is on, the hive tier's own collector runs a `journald`
|
||||||
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
|
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
|
||||||
agent's journal, this collector already shares a filesystem with the unit it
|
agent's journal, this collector already shares a filesystem with the unit it
|
||||||
reads, being a host systemd service rather than a container) and forwards
|
reads, being a host systemd service rather than a container) and forwards
|
||||||
|
|
@ -351,8 +349,9 @@ Requires `services.journald.storage = "persistent"` (the NixOS default) on
|
||||||
every hive host with `otel.enable` set — asserted at eval time.
|
every hive host with `otel.enable` set — asserted at eval time.
|
||||||
|
|
||||||
⚠️ **A hive with `otel.enable` off has no operator-visible crash signal at
|
⚠️ **A hive with `otel.enable` off has no operator-visible crash signal at
|
||||||
all** — the per-hive dashboard banner this replaces is gone, and there is no
|
all** — the per-hive dashboard banner this replaces no longer exists, and
|
||||||
fallback path. Turning on `otel.enable` is what makes a crash reach anyone.
|
there is no fallback path. Turning on `otel.enable` is what makes a crash
|
||||||
|
reach anyone.
|
||||||
|
|
||||||
## Agent-emitted per-turn metrics (`hive-agent`)
|
## Agent-emitted per-turn metrics (`hive-agent`)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -66,9 +66,9 @@ is opt-in via the P3RM1SS10NS tab.
|
||||||
declared, but no call site constructs it) as regular inbox messages
|
declared, but no call site constructs it) as regular inbox messages
|
||||||
(same `recv` path; body is a JSON object with an `event` discriminant
|
(same `recv` path; body is a JSON object with an `event` discriminant
|
||||||
field). A container crash used to be the other variant
|
field). A container crash used to be the other variant
|
||||||
(`ContainerCrash`, routed to `ruth`) but that path is gone — a crash
|
(`ContainerCrash`, routed to `ruth`) but this tree no longer has that
|
||||||
now only produces a log line + swarm-level grafana alert, no agent
|
path — a crash now only produces a log line + swarm-level grafana
|
||||||
turn at all (`docs/scheduler/observability.md`). Lifecycle
|
alert, no agent turn at all (`docs/scheduler/observability.md`). Lifecycle
|
||||||
transitions the job-queue scheduler or crash watcher drive directly —
|
transitions the job-queue scheduler or crash watcher drive directly —
|
||||||
stop/kill, destroy, a flake-rev login or logout state change — reach
|
stop/kill, destroy, a flake-rev login or logout state change — reach
|
||||||
no individual agent: they publish onto a swarm-wide NATS
|
no individual agent: they publish onto a swarm-wide NATS
|
||||||
|
|
|
||||||
|
|
@ -80,14 +80,21 @@ let
|
||||||
# The tier in the middle. Its OTLP receiver takes both signals on one
|
# The tier in the middle. Its OTLP receiver takes both signals on one
|
||||||
# port, so without this pipeline an agent's push is answered 404 on
|
# port, so without this pipeline an agent's push is answered 404 on
|
||||||
# `/v1/logs` — and a forwarder retrying into a 404 is indistinguish-
|
# `/v1/logs` — and a forwarder retrying into a 404 is indistinguish-
|
||||||
# able from one with nothing to send. Compared against the metrics
|
# able from one with nothing to send. `journald` joined the receiver
|
||||||
# pipeline's exporters rather than a name, so the two signals cannot
|
# list for this collector's own host journal (hive-c0re's crash-watch
|
||||||
# drift to different destinations.
|
# log line, see ./otel.nix) — both have to be present, in either
|
||||||
|
# order, so this stays a set check rather than a list-equality one.
|
||||||
|
# Compared against the metrics pipeline's exporters rather than a
|
||||||
|
# name, so the two signals cannot drift to different destinations.
|
||||||
name = "the hive collector forwards logs upstream, not only metrics";
|
name = "the hive collector forwards logs upstream, not only metrics";
|
||||||
ok =
|
ok =
|
||||||
(hiveOtelPipelines ? logs)
|
(hiveOtelPipelines ? logs)
|
||||||
&& hiveOtelPipelines.metrics.exporters != [ ]
|
&& hiveOtelPipelines.metrics.exporters != [ ]
|
||||||
&& hiveOtelPipelines.logs.receivers == [ "otlp" ]
|
&&
|
||||||
|
lib.sort builtins.lessThan hiveOtelPipelines.logs.receivers == [
|
||||||
|
"journald"
|
||||||
|
"otlp"
|
||||||
|
]
|
||||||
&& hiveOtelPipelines.logs.exporters == hiveOtelPipelines.metrics.exporters;
|
&& hiveOtelPipelines.logs.exporters == hiveOtelPipelines.metrics.exporters;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue