diff --git a/docs/scheduler/observability.md b/docs/scheduler/observability.md index d4b7d66e..b7d17749 100644 --- a/docs/scheduler/observability.md +++ b/docs/scheduler/observability.md @@ -216,6 +216,38 @@ Append additional labels via `extraResourceAttributes` (see option reference above); pass custom per-data-point labels with `hive-metric --labels` (see below). +## Log severity + +Every journald receiver in the pipeline maps the journal's `PRIORITY` onto an +OpenTelemetry severity, so a line arrives in the log store already carrying its +level and `severity_text:ERROR` is a query you can write: + +| `PRIORITY` | syslog | severity | +| ---------- | ------------------ | -------- | +| 0, 1, 2 | emerg, alert, crit | `FATAL` | +| 3 | err | `ERROR` | +| 4 | warning | `WARN` | +| 5, 6 | notice, info | `INFO` | +| 7 | debug | `DEBUG` | + +Both tiers that read a journal do this — the agent container's forwarder over +its own journal, and the swarm collector over the host's — and both import the +same mapping from `nix/journald-severity.nix`, which holds the table once. + +Note the direction. Syslog's priority counts **down** in urgency, where +OpenTelemetry's severity number counts **up**. Neither is a passthrough of the +other, and a pipeline that hands one straight to the other renders debug lines +as critical without erroring anywhere. + +A record that reaches the store from somewhere other than a journal has no +priority to map. Claude Code's own telemetry is the live example: it exports +OTLP log records directly (`OTEL_LOGS_EXPORTER`, see +`nix/agent-modules/claude-settings.nix`) and sets no severity on them, so they +store as `Unspecified`. The **Log rows with no severity** panel on the +`hyperhive · logs` dashboard counts both cases and keeps them apart: lines that +had a priority and arrived without a severity anyway signal a broken mapping, +while lines that never had one signal nothing of the sort. + ## Host-emitted container-resource metrics (hive-c0re) When OTEL is enabled, **hive-c0re itself** also exports each agent