vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

every hive runs its otel collector, and agents export to it by default

Browse source
This commit is contained in:
atlas 2026-10-18 23:17:28 +02:00 • committed by mara
commit 13daf4c7dc
28 changed files with 245 additions and 303 deletions
Download patch file Download diff file Expand all files Collapse all files

119
nix/host-modules/otel.nix
View file

@ -1,11 +1,8 @@
# Hive-wide OTEL stats export, and the HIVE tier of the collector pair.
# Set ONCE here at host level; the meta-flake renderer
# (`hive-c0re/src/meta.rs::otel_config`) reads the HYPERHIVE_OTEL_* env
# exported off hive-c0re's unit (see ./hive-c0re) and injects the matching
# `services.hyperhive.agent.otel.*` build-time config into EVERY agent (mirroring the
# CA-cert injection), so each agent's harness exports its own Claude Code
# stats directly to the collector. There is no per-agent opt-in — this is
# the single switch for the whole hive.
# The HIVE tier of the collector pair: every hive runs one, on the host that
# runs hive-c0re (`deploy.hive-controller.enable`). Agents export to it at
# the fixed name `services.hyperhive.otel.collector.host`, which this hive's
# resolver answers with its own bridge IP, so an agent's build carries
# nothing hive-specific.
#
# This tier receives from this hive's agents and forwards to the swarm's
# collector (./swarm-otel.nix). It holds no credential and picks no
@ -41,6 +38,8 @@ let
# The hive's own store identity. The collector authenticates to the swarm
# as the hive, so it reads its secret as the hive: the hive leaf's policy
# (`swarm-secret-client`'s `policy::render`) grants `swarm/hives/<hive>/*`.
runsHive = config.services.hyperhive.otel.collector.enable;
hiveLeaf =
config.services.hyperhive.deploy.bao.clientCertFile != null
&& config.services.hyperhive.deploy.bao.clientKeyFile != null;
@ -55,39 +54,19 @@ in
name = "hive-otel";
consumers = [ "opentelemetry-collector" ];
hostUnit = true;
enable = config.services.hyperhive.otel.enable;
enable = runsHive;
})
(lib.mkRenamedOptionModule
[ "services" "hyperhive" "otel" "collector" "port" ]
[ "services" "hyperhive" "swarm" "hiveOtel" "collectorPort" ]
)
(lib.mkRemovedOptionModule [ "services" "hyperhive" "otel" "enable" ] ''
Every hive runs its own telemetry collector, and every agent exports to
it by default, so there is no switch. Remove this definition.
'')
(lib.mkRemovedOptionModule [ "services" "hyperhive" "otel" "collector" "port" ] ''
The hive collector's receiver port is fixed at 4318: agents dial that
literal. Remove this definition.
'')
];
options.services.hyperhive.otel = {
enable = lib.mkEnableOption ''
hive-wide export of every agent's Claude Code stats (token usage,
cost, tool calls) to an OTLP endpoint via Claude Code's built-in
OpenTelemetry. One switch for all agents.
Enabling this also runs this hive's collector on the host: there is
exactly one way telemetry leaves this hive, and it is through that
collector. Agents export unauthenticated to
{option}`services.hyperhive.otel.collector.host`, a name this hive's
resolver answers with a bridge address only its own containers can
reach.
That collector forwards to the swarm's
({option}`services.hyperhive.deploy.swarm-otel.enable`), which holds the
upstream credential and writes the swarm's store. So an agent never
sees the credential, and neither does this tier.
⚠️ The collector is therefore in the path of all telemetry. It runs
on the same host as the agents and restarts on failure, and
telemetry is not the control plane, so degraded telemetry is not
degraded operation — but the export does not survive independently
of anything host-side
'';
endpoint = lib.mkOption {
type = lib.types.str;
default = "";
@ -171,25 +150,51 @@ in
'';
};
collector.enable = lib.mkOption {
type = lib.types.bool;
readOnly = true;
default =
config.services.hyperhive.deploy.hive-controller.enable
|| config.services.hyperhive.deploy.allSwarmServices;
defaultText = lib.literalExpression "deploy.hive-controller.enable || deploy.allSwarmServices";
description = ''
Whether this host runs the hive tier's collector. Not a switch:
every host that runs a hive
({option}`services.hyperhive.deploy.hive-controller.enable`) runs
one, and so does a host carrying all the swarm's shared services
({option}`services.hyperhive.deploy.allSwarmServices`), whose own
containers export to it.
Agents export to it unauthenticated at
{option}`services.hyperhive.otel.collector.host`, a name this
hive's resolver answers with a bridge address only its own
containers can reach. It forwards to the swarm's collector
({option}`services.hyperhive.deploy.swarm-otel.enable`), which holds
the upstream credential and writes the swarm's store, so neither an
agent nor this tier ever sees the credential.
⚠️ The collector is in the path of all telemetry. It runs on the
same host as the agents and restarts on failure; telemetry is not
the control plane, so degraded telemetry is not degraded
operation, but the export does not survive independently of
anything host-side.
'';
};
collector.host = lib.mkOption {
type = lib.types.str;
readOnly = true;
# Total on a null swarm domain, as in ./swarm-otel-service.nix: the
# required-domain assertion should be what an operator sees.
default = "hive-collector.${
if config.services.hyperhive.swarm.domain == null then
"invalid"
else
config.services.hyperhive.swarm.domain
}";
defaultText = lib.literalExpression ''"hive-collector.''${services.hyperhive.swarm.domain}"'';
default = "hive-collector.hive";
description = ''
The name agents reach this hive's collector on. The same string on
every hive: each hive's own resolver answers it with its own bridge
IP, so an agent's build carries no hive-specific address.
The name agents reach this hive's collector on. A literal, the same
on every hive and in every swarm: each hive's own resolver answers
it with its own bridge IP, so an agent's build carries neither a
swarm domain nor a hive-specific address. The agent module's
`services.hyperhive.agent.otel.endpoint` default spells the same
name.
No hive can claim this name as its domain:
`nix/reserved-hive-fragments.nix` forbids `hive` in any hive name.
It sits outside every domain a hive or the swarm serves: a hive's
domain is `<hiveName>.<swarm.domain>`, so it never ends in `.hive`.
'';
};
@ -274,12 +279,14 @@ in
};
};
# Shared by every hive's otel tier and its agents, so an agent's build
# carries the same port whichever hive renders it.
# Shared by every hive's otel tier and its agents. Read-only because the agent
# module's `otel.endpoint` default spells the same literal, and a host that
# moved the port would leave its agents dialling the old one.
options.services.hyperhive.swarm.hiveOtel = {
collectorPort = lib.mkOption {
type = lib.types.port;
default = 4318;
readOnly = true;
description = ''
Port of every hive collector's OTLP/HTTP receiver, at that hive's
`services.hyperhive.network.bridgeIp`: the port agents dial at
@ -307,7 +314,7 @@ in
'';
};
config = lib.mkIf config.services.hyperhive.otel.enable (
config = lib.mkIf runsHive (
let
otel = config.services.hyperhive.otel;
autheliaCfg = config.services.hyperhive.swarm.authelia;
@ -913,7 +920,7 @@ in
# cleanly, retries forever, and reports nothing to anyone.
assertion = senderAuth;
message = ''
services.hyperhive.otel.enable is true but this hive has no
This hive runs a telemetry collector (every hive does) but has no
identity to present to the swarm's collector:
services.hyperhive.otel.clientSecretFile = ${
@ -948,7 +955,7 @@ in
"none"
]);
message = ''
services.hyperhive.otel.enable is true, which runs a journald
This hive runs a telemetry collector, which runs a journald
receiver reading /var/log/journal for hive-c0re's own crash-watch
log line — but services.journald.storage is
"${config.services.journald.storage}" on this host.