The nix-store gauge comment claimed a semconv name it deliberately does
not use; it now says why the name is hyperhive-scoped. Two doc pages
still described the removed banner for the limits-read and
knowledge-pull failures. The warning alert's comment now says
pending_logins pages on purpose.
The dashboard's top-of-page warning banner is gone. The warning registry
stays, because /health/ready, the swarm status view and the OTEL gauges
read it. What it feeds is now alerted on in Grafana:
- hyperhive.hive.warning{kind,level}: one series per raised warning, so an
alert names the cause. Raising a warning also logs one `hive warning
raised` line with the message.
- hyperhive.hive.nix_store.utilization: the used fraction of the nix
store's filesystem, probed in the gauge callback.
- Alerts: hh-hive-degraded on the existing degraded gauge, hh-hive-warning
per kind (disk excluded), hh-host-disk-high at 85% and
hh-host-disk-critical at 95%.
- pending_logins is now a registry warning, refreshed from the container
rescan, so it reaches the per-kind gauge.
Frontend: server_warnings leaves the state snapshot, and the dashboard,
flow and stats pages keep only the sticky chrome wrapper. The shared
hive-warn and WarnBanner components stay, since other views use them.
A module-eval case checks each alert's metric name against the exporter's
source, so a rename on either side fails the build.
These three are per-host choices that hive-c0re reads and renders into
its agents, so they belong under the hive controller's deploy namespace
rather than swarm-wide:
- c0re.dashboardPort -> deploy.hive-controller.dashboardPort
- c0re.claudeCodePackage -> deploy.hive-controller.claudeCodePackage
(the claude-login and review-classifier containers on the same host
keep following it)
- otel.extraResourceAttributes -> deploy.hive-controller.otel.extraResourceAttributes
The old paths stay as mkRenamedOptionModule aliases, now in deploy.nix
with the namespace's other renames. github.enable and the collector port
stay swarm-wide.
Both are ignored with a warning through the same shim, now
nix/host-modules/hive-options-removed.nix.
- otel.metricIntervalMs: claude in every agent and hive-c0re's
container-resource exporter run at the 60s default. The agent-level
services.hyperhive.agent.otel.metricIntervalMs, set only by the meta
render, goes too (warning shim in nix/agent-modules/otel-removed.nix).
- otel.debug: services.hyperhive.agent.otel.debug stays, documented and
set in an agent's own config; the meta render no longer emits it.
Agents are handed `http://hive-collector.<swarm domain>:<port>` as their
otel endpoint instead of `http://<bridge IP>:<port>`. Every hive running
its collector contributes the name to `gateway.localNames`, so its own
dnsmasq answers it with its bridge IP, the way the queue's name already
works. The rendered meta flake then carries no hive-specific address,
which a swarm-level render needs (#4857, operator ruling 100744: "b").
hive-c0re's own exporter keeps the bridge address: on the host the name
may resolve to loopback through `gateway.localHostsEntry`, where the
receiver does not listen.
The name sits under the swarm domain and starts with `hive`, a fragment
no hive name may contain, so no hive's default domain can take it.
No behaviour change. Casts replaced by try_from / Duration math; schedules.rs
reuses agent_renewal::unix_now; the five swarm-queue-client KV open-or-create
copies share one private kv::open_or_create.
Every hive-labelled series in the store also carries an agent label, so a
hive is only ever visible as the sum of its agents — and a hive whose c0re
has stopped is indistinguishable from one that simply hosts none.
Adds three instruments to the exporter hive-c0re already runs, each a
projection of a value the process computes anyway: process.uptime (the
semconv name — the spec defines it as a double gauge in seconds, which is
exactly this instrument), hyperhive.hive.degraded, and
hyperhive.hive.warnings split by level. None carries an agent attribute;
that absence is what makes them selectable as hive-scoped.
The health pair reads warnings::readiness() rather than deriving its own
verdict, and degraded ships as a series instead of being left for a
dashboard query to compute from warnings{level="crit"} — either would put
the "what counts as unhealthy" rule in a second place that disagrees
silently the first time a degrading condition is added.
Review finding from argus. The new endpoint test carried a SAFETY comment
claiming no other test in its module asserts on the variables it perturbs —
the wrong boundary. The module is not the unit that shares the environment,
the process is: meta.rs's render_flake_injects_otel_when_signalled mutates
the same HYPERHIVE_OTEL_ENDPOINT, both land in the one hive-c0re test binary,
and cargo runs it at default parallelism with no serialisation anywhere in
the crate. Each test independently claimed exclusive ownership of shared
global state, which is the instrument-that-looks-solid class the endpoint
change's own gate reasoning warns about.
Adds test_env with a single ENV_LOCK, taken by both. No new dependency: this
is the pattern hive-bash-mcp and hive-agent already use, and hive-bash-mcp's
helper records why it has to be crate-wide rather than per-module — two
per-module mutexes serialise nothing against each other, which produced a
CI-only flake there.
The asymmetry that makes this hard to see locally is worth stating: an agent
container has the hyperhive variables ambient-set, so a losing race still
finds a plausible value and the test passes; the nix sandbox strips them, so
only there can one thread delete a variable out from under another. Verified
in that shape with `env -u HYPERHIVE_OTEL_ENDPOINT -u
OTEL_EXPORTER_OTLP_ENDPOINT`, five consecutive runs green — a sanity check,
not a proof, since a race cannot be shown absent by running. What makes it
correct is structural: both tests take the same lock.
Deliberately scoped to the pair that overlaps. meta.rs has three further
env-mutating tests (HIVE_FORGE_URL twice, the TLS CA pair) that race with
each other, untouched here and tracked separately, because the fix is not
the mechanical one it looks like: std::sync::Mutex is not reentrant, so
adding a lock to a test whose helpers also lock deadlocks. That needs
reading per test rather than a sweep.
hive-c0re's container-resource exporter has POSTed to a 404 for as long as
it has existed, silently: it passed the collector's base address to
`with_endpoint`, which the SDK takes verbatim, so every export went to `/`
instead of `/v1/metrics`. Nothing reported it — OTLP export failures go to
an error handler no binary here installs — so the daemon logged "exporter
enabled" and delivered nothing. VictoriaMetrics has never held a sample
under `service.name=hyperhive-c0re`.
Fix the way the rest of the repo already resolves an endpoint: an endpoint
option names a BASE, and the layer that knows the signal appends to it.
`hive-metric` — same SDK, same collector — never calls `with_endpoint`, and
`docs/observability.md` documents the append as system behaviour; the one
place a full path is spelled out is the VictoriaMetrics exporter, because
its far end is not a standard OTLP path.
So drop the call. The builder is now byte-identical to hive-metric's, and
hive-c0re's unit carries the standard `OTEL_EXPORTER_OTLP_ENDPOINT` for the
SDK to read. The address is bound once in nix and consumed twice, so what a
hive hands its agents and what it exports to itself cannot drift.
The enable signal moves to that same standard variable: "configured" and
"where it actually goes" become one string rather than two that agree by
convention. `HYPERHIVE_OTEL_*` keeps its own job, the agent-config
transport meta.rs reads — a name the SDK has never known, which is the bug.
The test changes shape with the fix. The old one asserted a URL this module
built; the new one pins that the exporter is gated on the variable the SDK
itself reads, because the fix is now an absence and an absence is what a
later "the endpoint is right there, just pass it" edit puts back.
Refs #3402
hive-c0re's container-resource exporter already targets this hive's own
collector (environment.nix derives the bridge address), so the upstream
header it was loaded with has nowhere to be presented: that hop is
unauthenticated for every producer on the host, and the credential
belongs to the swarm tier, which is the one that leaves the swarm.
Drop the LoadCredential entry and the auth_headers() reader with it.
The option itself stays -- swarm-otel.nix is its real consumer, via
EnvironmentFile on the collector unit.
Also corrects three descriptions that this makes false, or that were
already false: the module doc claimed to reuse the config "Claude Code's
in-container SDK export uses", which stopped being true when agents
moved off that path; the nix comment claimed the secret is "the same one
the agent containers get, forwarded via nspawn --load-credential", which
lost its last producer earlier; and docs/observability.md described an
Authorization header on a hop that will no longer send one. The
headersCredential option's own docs already said it reaches "neither an
agent container nor a hive's own collector" -- this makes that true
rather than aspirational.
Takes the crate from 26 rustdoc warnings to 1, on top of the ten in the
previous commit.
argus's review findings:
- agent_sockets.rs: [`write`] was still ambiguous (function vs macro).
The previous change narrowed the qualifier and left the ambiguity;
[`write()`] is what resolves it.
- forge/users.rs <hex> and stats/container_stats.rs <name>: unclosed
HTML tags in prose, now backticked.
The rest of the crate, so the count actually reaches zero:
- job_queue/mod.rs: Queue::graph_snapshot -> JobQueue::graph_snapshot
(there is no Queue type), and super::scheduler -> scheduler (mod.rs
*is* job_queue, so super:: pointed outside it)
- job_queue/resource.rs: NodeKind -> super::model::NodeKind
- matrix.rs: password_path(name) -> password_path; and
forge::provision_user_token -> crate::forge::provision_user_token.
Note the path has no `users` segment: forge/mod.rs declares `mod
users` private and re-exports it, so the canonical path comes from the
re-export rather than the directory tree.
- socket_server/lifecycle_handlers.rs: InfraContainer ->
hive_priv_sock::InfraContainer
- stats/otel_metrics.rs: crate::meta::otel_config is a private fn no
path can name from another module, so it becomes prose
- main.rs: redundant explicit link target dropped
coordinator.rs:405 (CrashWatchGuard) is deliberately untouched: #3244
deletes that doc block, so fixing it here would conflict with an open PR
and repair a symbol that is about to stop existing.
The container-resource exporter identified the samples it sent by
container and by hive, but not by machine or by build — so a sample
could not be attributed to the host it came from or the deploy that
produced it.
Both go on the OTEL resource rather than on each data point, for the
same reason `hive` / `swarm` already do: they are constant across one
hive-c0re. Only per-container facts stay per-data-point.
- `host.arch` — mapped to the semconv spelling, not forwarded from
rust's. The two disagree on exactly the architectures this runs on
(`x86_64` / `aarch64` vs `amd64` / `arm64`), and the failure mode is
silent: a dashboard filtering the standard value matches nothing.
A test pins this, since nothing else would catch it.
- `service.version` — the running flake rev, via
`auto_update::current_flake_rev`, the same source the dashboard
snapshot and `get_agent_meta` already use. NOT the crate version:
that's a workspace constant that never moves between deploys, so it
could not answer "which build produced this sample?". Omitted rather
than guessed when the flake ref carries no rev.
`spawn_exporter` takes the flake ref to reach the rev — the string it
needs, not the whole `Coordinator`, so the module's coupling doesn't
widen for one attribute.
The issue's third item, `container`, needs no change: `attrs()` has
emitted `container.name` per data point since this exporter landed.
Checked with clippy (`--all-targets -D warnings`), `cargo test -p
hive-c0re` (321 passed) and `nix fmt`. No option surface is touched, so
no nix-eval gate.
hive-c0re already samples each agent container's cgroup load for the
dashboard (stats/container_stats.rs); this rides those gauges out to the
configured OTLP endpoint, reusing the existing services.hyperhive.otel
config (endpoint + auth header) — no new toggle.
- New stats/otel_metrics.rs: exports via the OpenTelemetry Rust SDK (same
crates as hive-metric) with the semconv container.* metric names +
container.name attribute so off-the-shelf OTel/Grafana dashboards work,
plus the hive agent label. container.cpu.time (counter, s, from cumulative
cpu.stat usage_usec), container.memory.usage, container.memory.usage.limit;
memory peak / on-disk storage / instantaneous cpu percent stay hyperhive.*
custom (no semconv equivalent). Observable instruments read a shared
snapshot an async task refreshes (gather() is async; SDK callbacks sync).
- container_stats: expose cpu_time_usec (cumulative) on ContainerResource.
- The OTLP auth header is loaded onto hive-c0re's own unit via systemd
LoadCredential and read from $CREDENTIALS_DIRECTORY/otel-headers.
- docs/observability.md documents the host-emitted semconv metrics.
Host-side export, so it covers containers even when their agent is idle.
