Records reached VictoriaLogs carrying the journal's raw PRIORITY and
severity_text "Unspecified" — every line in the store, at every tier, with
no level a query or a dashboard could read. VictoriaLogs has no ingest
parameter naming a level field; it auto-detects one by field name, so the
mapping has to happen in the collector.
A stanza severity_parser on each journald receiver, from one shared file
rather than a copy per tier: the two receivers are unrelated config (a
fixed stanza in the agent container, a parameterised block inside the
swarm-otel container) and a drifted copy fails silently — every line still
arrives, labelled as the wrong thing.
Two details that are easy to get wrong and quiet when wrong. PRIORITY
counts down in urgency where the OTEL severity counts up, so the table is
written as a table. And overwrite_text is required: without it the parser
sets the severity number and leaves the text as the raw digit, so
severity_text arrives as the literal "6" — populated, and not a level
anything renders.
Every per-agent harness option lived at the top-level `hyperhive.*` while
the host tier has always been `services.hyperhive.*`. Move all 52 agent-tier
option leaves (33 top-level names across 16 modules) to
`services.hyperhive.agent.*`, repoint every read, and keep existing agent
configs evaluating through one `mkRenamedOptionModule` per old leaf path in
the new nix/agent-modules/renamed-options.nix.
The shims are per leaf rather than per namespace: `user`, `mcp`, `otel`,
`queue`, `docs`, `forge`, `frontend`, `github`, `gui`, `logs`, `matrix` and
`cargo` are plain attrsets of declarations, not submodule-typed options, so
a parent-path rename would not reach their children. Three read-only
options (`frontend.mergedDist`, `queue.clientIdFile`,
`queue.clientSecretFile`) deliberately get no shim — a rename contributes a
definition, which a read-only option refuses; the exclusions are commented
in place.
Refs #4473
validateConfigFile runs `otelcol validate` at nix build time, in a pure
sandbox where systemd's StateDirectory= has not run yet, so the
directory named by extensions.file_storage.directory does not exist.
Setting create_directory = true lets the validator create it itself,
matching what happens at runtime once StateDirectory= has acted.
Refs #4375
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The journald receiver runs with --lines=0, so every collector start
only ships what's written after it starts, and a restart silently
loses whatever landed while it was down. Point it at a file_storage
extension so the read cursor survives a restart; start_at stays at
its 'end' default since the cursor now covers everything after the
first run.
Refs #3818
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An agent container writes a complete journal — 991 MB and nine days deep
on this hive — that nothing outside it can read: the host-side
per-container journal directory is an id-mapped bind mount, and journald
writes nothing into it. So the reader has to run inside the container,
and the path it would push to did not exist.
Three tiers, one vertical slice, because any two of them alone are
silent:
- the agent container gains an `opentelemetry-collector` with a
`journald` receiver aimed at its own journal and an exporter aimed at
the same base address every in-process producer already exports to.
- the hive collector gains `service.pipelines.logs`. Without it the
`otlp` receiver answers 404 on `/v1/logs` — measured, and
indistinguishable from a route that was never meant to exist.
- the swarm collector gains a per-hive `logs/<hive>` pipeline beside
`metrics/<hive>`. Without it the push is accepted, answered 200, and
routed nowhere.
The receiver's `directory` is stated rather than inherited, and that is
the load-bearing line: its default is the RUNTIME journal
(`/run/log/journal`), which in an agent container is empty. Left at the
default this whole path validates, starts, reports healthy and forwards
nothing. The assertion beside it covers the same silence from the other
end — a `volatile` or `none` journald storage empties the directory the
receiver reads.
Attribution follows the tier that can prove it. The forwarder stamps
`agent`, which no host-side reader could supply; `hive` is deliberately
left to the swarm tier, which upserts it from whichever receiver
accepted the sample, precisely so the label comes from something the
sender cannot write.
No `units` allowlist, unlike the swarm tier's journald receiver. That
one needs one because the host's journal also holds an operator's own
session; a container's journal is the harness and what the harness
spawns. Measured volume is 20827 entries / 6.3 MB per agent per day,
with nothing logging below `info` — so the receiver's `info` default
filters nothing and there is no bill to justify a knob.
Agent containers only, per the ruling on the issue: swarm services need
one forwarder per service container and get re-measured once this works.
Part of #3940.
The endpoint, protocol, temporality preference and resource labels were
shipped only inside claude's managed settings json, so they applied to
claude's own process. hive-bash-daemon, hive-mcp-http, hive-matrix-daemon
and hive-forge-notify are systemd *siblings* of claude rather than its
children, so nothing shipped there could ever reach them: `hive-metric`
invoked from a tool call exited with "OTEL_EXPORTER_OTLP_ENDPOINT not
set", which is the honest failure of a value it structurally could not
see.
Declare those variables container-wide in a new agent module instead —
systemd.globalEnvironment for every unit PID 1 starts, environment.variables
for login shells. Both are needed and neither implies the other; NIX_REMOTE
is set both ways for the same reason.
Claude keeps only what is genuinely its own: the telemetry master flag,
the feedback-survey flag, the version label, and which signals it
exports. A different producer in the same container may legitimately
emit only metrics.
The hyperhive.otel.* options move across with them. They have more than
one consumer now, so their home is the OTEL module rather than the
claude one.
