every hive runs its otel collector, and agents export to it by default
This commit is contained in:
parent
839efd1642
commit
13daf4c7dc
28 changed files with 245 additions and 303 deletions
|
|
@ -117,8 +117,8 @@ dnsmasq is **authoritative** for the hive's own zone (`<hive-domain>`)
|
||||||
plus whatever swarm-service names this host contributes via
|
plus whatever swarm-service names this host contributes via
|
||||||
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
|
`gateway.localNames` — `forge.<swarm-domain>` and `chat.<swarm-domain>`
|
||||||
(matrix) when this host runs those services, `auth.<swarm-domain>`
|
(matrix) when this host runs those services, `auth.<swarm-domain>`
|
||||||
when it runs authelia, and `hive-collector.<swarm-domain>` on every hive
|
when it runs authelia, and `hive-collector.hive` on every hive
|
||||||
with `otel.enable` — answering each with the bridge IP (where nginx
|
— answering each with the bridge IP (where nginx
|
||||||
is reachable, and where the hive's own collector listens). Note forge and
|
is reachable, and where the hive's own collector listens). Note forge and
|
||||||
matrix are swarm-domain names, not sub-domains of `<hive-domain>`: a swarm
|
matrix are swarm-domain names, not sub-domains of `<hive-domain>`: a swarm
|
||||||
runs one forge and one homeserver, so their names belong to the swarm rather
|
runs one forge and one homeserver, so their names belong to the swarm rather
|
||||||
|
|
@ -182,7 +182,7 @@ By default agents can only reach the host on 80/443 (+53 DNS), so a
|
||||||
host-side service on another port — for example a dev OTLP collector you want
|
host-side service on another port — for example a dev OTLP collector you want
|
||||||
agents to reach directly — is unreachable. (hyperhive's own services
|
agents to reach directly — is unreachable. (hyperhive's own services
|
||||||
need none of this: a module that means to be reachable from agents opens
|
need none of this: a module that means to be reachable from agents opens
|
||||||
its own port here, which `otel.enable` and `nats.enable` both do. Note
|
its own port here, which the hive collector and `nats.enable` both do. Note
|
||||||
that `otel.endpoint` is the _upstream_, which no agent ever dials. See
|
that `otel.endpoint` is the _upstream_, which no agent ever dials. See
|
||||||
`docs/scheduler/observability.md`.)
|
`docs/scheduler/observability.md`.)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -16,15 +16,14 @@ Telemetry crosses two collectors, and which one you configure depends on what
|
||||||
the host is:
|
the host is:
|
||||||
|
|
||||||
| | runs where | receives from | does |
|
| | runs where | receives from | does |
|
||||||
| ------------------------------------ | ---------------------- | --------------------------------- | ----------------------------------------------- |
|
| ------------------------------------ | -------------- | --------------------------------- | ----------------------------------------------- |
|
||||||
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's stores and exports upstream |
|
| **swarm tier** — `deploy.swarm-otel` | once per swarm | every hive's collector | writes the swarm's stores and exports upstream |
|
||||||
| **hive tier** — `otel.enable` | every hive with agents | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential |
|
| **hive tier** — always on | every hive | that hive's agents, on the bridge | forwards to the swarm tier. Holds no credential |
|
||||||
|
|
||||||
An all-local host runs both; the hop between them configures itself.
|
An all-local host runs both; the hop between them configures itself.
|
||||||
|
|
||||||
```nix
|
```nix
|
||||||
services.hyperhive.otel = {
|
services.hyperhive.otel = {
|
||||||
enable = true;
|
|
||||||
endpoint = "https://collector.example.com/otel"; # the upstream
|
endpoint = "https://collector.example.com/otel"; # the upstream
|
||||||
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
|
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
|
||||||
};
|
};
|
||||||
|
|
@ -32,20 +31,20 @@ services.hyperhive.otel = {
|
||||||
|
|
||||||
## Enabling export
|
## Enabling export
|
||||||
|
|
||||||
`otel.enable` is the single gate on a hive: one switch in the host config
|
Every hive runs its collector and every agent exports to it, with nothing to
|
||||||
covers every agent container on it.
|
switch on: there is no per-hive or per-agent opt-in.
|
||||||
`endpoint` is where telemetry ends up after it leaves the swarm — optional,
|
`endpoint` is where telemetry ends up after it leaves the swarm — optional,
|
||||||
because the swarm's own metrics store (`deploy.victoriametrics`) is a
|
because the swarm's own metrics store (`deploy.victoriametrics`) is a
|
||||||
destination in its own right. With both, telemetry goes to both. See
|
destination in its own right. With both, telemetry goes to both. See
|
||||||
[`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana).
|
[`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana).
|
||||||
|
|
||||||
**Telemetry leaves a hive exactly one way: through the collector that
|
**Telemetry leaves a hive exactly one way: through the collector that
|
||||||
`enable` starts on the host.** Agents never talk to `endpoint` themselves —
|
runs on the host.** Agents never talk to `endpoint` themselves —
|
||||||
they export unauthenticated to a collector address only their own containers
|
they export unauthenticated to a collector address only their own containers
|
||||||
can reach. They dial it as
|
can reach. They dial it as
|
||||||
`http://hive-collector.<swarm domain>:<collector port>`, a name identical on
|
`http://hive-collector.hive:4318`, a name identical on
|
||||||
every hive, which each hive's own dnsmasq answers with its bridge IP, so an
|
every hive and every swarm, which each hive's own dnsmasq answers with its bridge IP, so an
|
||||||
agent's build carries no hive-specific address. The otel module opens that
|
agent's build carries no hive-specific address and no swarm domain. The otel module opens that
|
||||||
port on the bridge itself. That collector forwards to the swarm's
|
port on the bridge itself. That collector forwards to the swarm's
|
||||||
([`swarm/services.md`](../swarm/services.md#telemetry-collector-otel)), the
|
([`swarm/services.md`](../swarm/services.md#telemetry-collector-otel)), the
|
||||||
single process holding the upstream credential and the only writer to the
|
single process holding the upstream credential and the only writer to the
|
||||||
|
|
@ -170,7 +169,7 @@ a build error naming the reason rather than telemetry silently going nowhere.
|
||||||
## Network access
|
## Network access
|
||||||
|
|
||||||
Agent containers reach the host only on ports 80 and 443 (plus DNS). The
|
Agent containers reach the host only on ports 80 and 443 (plus DNS). The
|
||||||
collector port needs nothing from you: `otel.enable` opens it on the bridge.
|
collector port needs nothing from you: the hive collector opens it on the bridge.
|
||||||
To let agents reach some other host-local service of your own →
|
To let agents reach some other host-local service of your own →
|
||||||
[`networking/network.md`](../networking/network.md#reaching-host-services-exposehostports).
|
[`networking/network.md`](../networking/network.md#reaching-host-services-exposehostports).
|
||||||
|
|
||||||
|
|
@ -180,7 +179,7 @@ The nix module (`nix/host-modules/otel.nix`) generates every
|
||||||
`services.hyperhive.otel.*` option's full type, default, description and
|
`services.hyperhive.otel.*` option's full type, default, description and
|
||||||
example into the [options reference](https://hyperhive.darkest.space/options/)
|
example into the [options reference](https://hyperhive.darkest.space/options/)
|
||||||
(host options — `nix build .#docs-host` for a local render). The same module
|
(host options — `nix build .#docs-host` for a local render). The same module
|
||||||
declares the collector port every hive and its agents share,
|
fixes the collector port every hive and its agents share,
|
||||||
`services.hyperhive.swarm.hiveOtel.collectorPort`, and each hive's own
|
`services.hyperhive.swarm.hiveOtel.collectorPort`, and each hive's own
|
||||||
`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`.
|
`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`.
|
||||||
That's the reference; this page covers what a per-option listing can't: the
|
That's the reference; this page covers what a per-option listing can't: the
|
||||||
|
|
@ -296,7 +295,7 @@ cleanly and the affected level button goes on returning zero rows:
|
||||||
|
|
||||||
<!-- vale write-good.Passive = NO -->
|
<!-- vale write-good.Passive = NO -->
|
||||||
|
|
||||||
When OTEL is enabled, **hive-c0re itself** also exports each agent
|
**hive-c0re itself** also exports each agent
|
||||||
container's resource load — the same cgroup gauges —
|
container's resource load — the same cgroup gauges —
|
||||||
to this hive's own collector, exactly like an agent does and with
|
to this hive's own collector, exactly like an agent does and with
|
||||||
no separate toggle. These come from the host, not the in-container Claude SDK,
|
no separate toggle. These come from the host, not the in-container Claude SDK,
|
||||||
|
|
@ -342,7 +341,7 @@ a crash and `tracing::warn!`s `"container crash detected"` — nothing else:
|
||||||
no broker event, no per-hive dashboard banner. Both of those used to exist;
|
no broker event, no per-hive dashboard banner. Both of those used to exist;
|
||||||
this change removed them in favour of the log line.
|
this change removed them in favour of the log line.
|
||||||
|
|
||||||
When `otel.enable` is on, the hive tier's own collector runs a `journald`
|
The hive tier's own collector runs a `journald`
|
||||||
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
|
receiver scoped to `hive-c0re.service` (no bind mount needed — unlike an
|
||||||
agent's journal, this collector already shares a filesystem with the unit it
|
agent's journal, this collector already shares a filesystem with the unit it
|
||||||
reads, being a host systemd service rather than a container) and forwards
|
reads, being a host systemd service rather than a container) and forwards
|
||||||
|
|
@ -353,18 +352,13 @@ matches it directly and pages the operator — no agent, and no per-hive
|
||||||
dashboard, in the loop.
|
dashboard, in the loop.
|
||||||
|
|
||||||
Requires `services.journald.storage = "persistent"` (the NixOS default) on
|
Requires `services.journald.storage = "persistent"` (the NixOS default) on
|
||||||
every hive host with `otel.enable` set — asserted at eval time.
|
every hive host — asserted at eval time.
|
||||||
|
|
||||||
⚠️ **A hive with `otel.enable` off has no operator-visible crash signal at
|
|
||||||
all** — the per-hive dashboard banner this replaces no longer exists, and
|
|
||||||
there is no fallback path. Turning on `otel.enable` is what makes a crash
|
|
||||||
reach anyone.
|
|
||||||
|
|
||||||
## Agent-emitted per-turn metrics (`hive-agent`)
|
## Agent-emitted per-turn metrics (`hive-agent`)
|
||||||
|
|
||||||
<!-- vale write-good.Passive = NO -->
|
<!-- vale write-good.Passive = NO -->
|
||||||
|
|
||||||
When OTEL is enabled, the harness itself (`hive-agent`) exports one small set
|
The harness itself (`hive-agent`) exports one small set
|
||||||
of metrics per claude turn, recorded the moment the turn ends (not polled).
|
of metrics per claude turn, recorded the moment the turn ends (not polled).
|
||||||
These are deliberately the fields Claude Code's own built-in export (see
|
These are deliberately the fields Claude Code's own built-in export (see
|
||||||
above) can't know about — the harness's own wall-clock timing, what woke the
|
above) can't know about — the harness's own wall-clock timing, what woke the
|
||||||
|
|
@ -508,8 +502,7 @@ and the start of the response body. It repeats a persisting failure every
|
||||||
## Agent-emitted custom metrics (`hive-metric`)
|
## Agent-emitted custom metrics (`hive-metric`)
|
||||||
|
|
||||||
Agents can push arbitrary labeled metrics to the same OTEL collector via the
|
Agents can push arbitrary labeled metrics to the same OTEL collector via the
|
||||||
`hive-metric` CLI tool, available in every agent container when
|
`hive-metric` CLI tool, available in every agent container.
|
||||||
`services.hyperhive.otel.enable = true`.
|
|
||||||
|
|
||||||
### Usage
|
### Usage
|
||||||
|
|
||||||
|
|
@ -544,11 +537,11 @@ hive-metric queue_depth 17 --type gauge
|
||||||
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
|
hive-metric api_latency_ms 142.5 --type gauge --labels model=sonnet --labels tier=api
|
||||||
```
|
```
|
||||||
|
|
||||||
### Error when OTEL isn't configured
|
### Error when an agent turns OTEL off
|
||||||
|
|
||||||
When `services.hyperhive.otel.enable = false` (the default), the
|
An agent whose own config sets `services.hyperhive.agent.otel.enable = false`
|
||||||
`OTEL_EXPORTER_OTLP_ENDPOINT` env var isn't set and `hive-metric` exits
|
has no `OTEL_EXPORTER_OTLP_ENDPOINT` env var, and `hive-metric` exits with an
|
||||||
with an informative error message. No silently dropped metrics.
|
informative error message. No silently dropped metrics.
|
||||||
|
|
||||||
### Wire format
|
### Wire format
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -561,42 +561,17 @@ fn hive_ca_source() -> Option<String> {
|
||||||
Some(path)
|
Some(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Hive-wide OTEL config injected into every agent's build, read off
|
/// Extra `OTEL_RESOURCE_ATTRIBUTES` entries for every agent, read off
|
||||||
/// hive-c0re's own unit env (set from `services.hyperhive.otel.*`,
|
/// hive-c0re's own unit env
|
||||||
/// `services.hyperhive.swarm.hiveOtel.*` and
|
/// (`services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes`).
|
||||||
/// `services.hyperhive.deploy.hive-controller.otel.*`). A present, non-empty
|
/// `None` when unset or empty. This is the only OTEL value the renderer
|
||||||
/// `HYPERHIVE_OTEL_ENDPOINT` is the enable signal — the host module
|
/// writes into the flake: agents export to the hive collector by default and
|
||||||
/// asserts the endpoint is set whenever `otel.enable` is true, so
|
/// find it by name, so there is no enable flag, endpoint or protocol to
|
||||||
/// "endpoint present" == "OTEL on". The optional fields map to the
|
/// carry.
|
||||||
/// matching host options and are only carried when set.
|
fn otel_extra_resource_attributes() -> Option<String> {
|
||||||
struct OtelConfig {
|
std::env::var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES")
|
||||||
endpoint: String,
|
|
||||||
protocol: String,
|
|
||||||
extra_resource_attributes: Option<String>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Read the hive-wide OTEL config from env, or `None` when OTEL is off.
|
|
||||||
/// Mirrors `hive_ca_source` — host state surfaced to the meta renderer
|
|
||||||
/// so it can bake build-time `services.hyperhive.agent.otel.*` config into each agent
|
|
||||||
/// (the per-agent options the harness modules consume). Returns `None`
|
|
||||||
/// when the endpoint signal is absent so the renderer emits no
|
|
||||||
/// `services.hyperhive.agent.otel.*` lines and agents keep the disabled default.
|
|
||||||
fn otel_config() -> Option<OtelConfig> {
|
|
||||||
let endpoint = std::env::var("HYPERHIVE_OTEL_ENDPOINT")
|
|
||||||
.ok()
|
|
||||||
.filter(|v| !v.is_empty())?;
|
|
||||||
let protocol = std::env::var("HYPERHIVE_OTEL_PROTOCOL")
|
|
||||||
.ok()
|
.ok()
|
||||||
.filter(|v| !v.is_empty())
|
.filter(|v| !v.is_empty())
|
||||||
.unwrap_or_else(|| "http/protobuf".to_owned());
|
|
||||||
let extra_resource_attributes = std::env::var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES")
|
|
||||||
.ok()
|
|
||||||
.filter(|v| !v.is_empty());
|
|
||||||
Some(OtelConfig {
|
|
||||||
endpoint,
|
|
||||||
protocol,
|
|
||||||
extra_resource_attributes,
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The ordered set of CA certs embedded next to the meta flake, as
|
/// The ordered set of CA certs embedded next to the meta flake, as
|
||||||
|
|
@ -907,36 +882,20 @@ fn render_flake(
|
||||||
ca_refs.join(" ")
|
ca_refs.join(" ")
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
// Hive-wide OTEL stats export (`services.hyperhive.otel.*`): inject the
|
// Extra OTEL resource labels (`deploy.hive-controller.otel.*`) are the one
|
||||||
// build-time `services.hyperhive.agent.otel.*` config the harness modules consume (its
|
// OTEL value rendered here; enablement and the collector endpoint are
|
||||||
// otelEnv + otelExecStart wrapper + LoadCredential). Host-driven, so
|
// agent-module defaults.
|
||||||
// the same config lands on every agent; emitted only when enabled.
|
if let Some(attrs) = otel_extra_resource_attributes() {
|
||||||
// Mirrors the CA-cert injection above — host state -> build-time agent
|
|
||||||
// module config.
|
|
||||||
if let Some(otel) = otel_config() {
|
|
||||||
let esc = |s: &str| s.replace('\\', "\\\\").replace('"', "\\\"");
|
let esc = |s: &str| s.replace('\\', "\\\\").replace('"', "\\\"");
|
||||||
out.push_str(" services.hyperhive.agent.otel.enable = true;\n");
|
|
||||||
let _ = writeln!(
|
|
||||||
out,
|
|
||||||
" services.hyperhive.agent.otel.endpoint = \"{}\";",
|
|
||||||
esc(&otel.endpoint)
|
|
||||||
);
|
|
||||||
let _ = writeln!(
|
|
||||||
out,
|
|
||||||
" services.hyperhive.agent.otel.protocol = \"{}\";",
|
|
||||||
esc(&otel.protocol)
|
|
||||||
);
|
|
||||||
if let Some(attrs) = &otel.extra_resource_attributes {
|
|
||||||
let _ = writeln!(
|
let _ = writeln!(
|
||||||
out,
|
out,
|
||||||
" services.hyperhive.agent.otel.extraResourceAttributes = \"{}\";",
|
" services.hyperhive.agent.otel.extraResourceAttributes = \"{}\";",
|
||||||
esc(attrs)
|
esc(&attrs)
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
|
||||||
// Agent-facing service URLs (`services.hyperhive.agent.forge.url`, `services.hyperhive.agent.matrix.url`):
|
// Agent-facing service URLs (`services.hyperhive.agent.forge.url`, `services.hyperhive.agent.matrix.url`):
|
||||||
// emit the host's real values as build-time agent config, the same
|
// emit the host's real values as build-time agent config, the same
|
||||||
// host-state -> agent-module shape as the otel block above.
|
// host-state -> agent-module shape as the CA block above.
|
||||||
//
|
//
|
||||||
// These options exist already, but until now *nothing set them*, so every
|
// These options exist already, but until now *nothing set them*, so every
|
||||||
// agent silently fell back to their `localhost:<port>` defaults while the
|
// agent silently fell back to their `localhost:<port>` defaults while the
|
||||||
|
|
@ -2140,16 +2099,16 @@ mod tests {
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn render_flake_injects_otel_when_signalled() {
|
fn render_flake_carries_no_otel_enablement() {
|
||||||
// services.hyperhive.otel.* -> HYPERHIVE_OTEL_* on hive-c0re's unit
|
// The agent module defaults otel on and finds the collector by name, so
|
||||||
// -> injected as build-time services.hyperhive.agent.otel.* into every agent. With
|
// the flake carries no enable, endpoint or protocol line however the
|
||||||
// no endpoint signal, no services.hyperhive.agent.otel lines are emitted (agents
|
// host's OTEL env is set. Only the per-hive resource labels are
|
||||||
// keep the the harness modules disabled default).
|
// rendered, and only when set.
|
||||||
//
|
//
|
||||||
// Serialised against every other env-mutating test in the crate.
|
// Serialised against every other env-mutating test in the crate.
|
||||||
// `stats::otel_metrics::tests` perturbs HYPERHIVE_OTEL_ENDPOINT too,
|
// `stats::otel_metrics::tests` perturbs OTEL env too, and lands in this
|
||||||
// and lands in this same test binary — "no other test asserts on
|
// same test binary — "no other test asserts on these" was true of this
|
||||||
// these" was true of this module and false of the process.
|
// module and false of the process.
|
||||||
let _env = crate::test_env::lock();
|
let _env = crate::test_env::lock();
|
||||||
// SAFETY: serialised by the guard above; restored before returning.
|
// SAFETY: serialised by the guard above; restored before returning.
|
||||||
let render = || {
|
let render = || {
|
||||||
|
|
@ -2165,53 +2124,49 @@ mod tests {
|
||||||
std::env::set_var("HYPERHIVE_OTEL_ENDPOINT", "https://c.example/otel");
|
std::env::set_var("HYPERHIVE_OTEL_ENDPOINT", "https://c.example/otel");
|
||||||
std::env::set_var("HYPERHIVE_OTEL_PROTOCOL", "grpc");
|
std::env::set_var("HYPERHIVE_OTEL_PROTOCOL", "grpc");
|
||||||
}
|
}
|
||||||
let on_minimal = render();
|
let plain = render();
|
||||||
unsafe {
|
unsafe {
|
||||||
std::env::set_var(
|
std::env::set_var(
|
||||||
"HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES",
|
"HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES",
|
||||||
"deployment.environment=prod",
|
"deployment.environment=prod",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
let on_full = render();
|
let labelled = render();
|
||||||
unsafe {
|
unsafe {
|
||||||
std::env::remove_var("HYPERHIVE_OTEL_ENDPOINT");
|
std::env::remove_var("HYPERHIVE_OTEL_ENDPOINT");
|
||||||
std::env::remove_var("HYPERHIVE_OTEL_PROTOCOL");
|
std::env::remove_var("HYPERHIVE_OTEL_PROTOCOL");
|
||||||
std::env::remove_var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES");
|
std::env::remove_var("HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES");
|
||||||
}
|
}
|
||||||
let off = render();
|
|
||||||
|
|
||||||
|
// Presence control: the render is the real flake, carrying the pin.
|
||||||
assert!(
|
assert!(
|
||||||
on_minimal.contains("services.hyperhive.agent.otel.enable = true;"),
|
plain.contains("rev=0123abcd"),
|
||||||
"otel enable must be injected:\n{on_minimal}"
|
"the hyperhive pin line must be rendered:\n{plain}"
|
||||||
);
|
);
|
||||||
|
for (name, flake) in [("plain", &plain), ("labelled", &labelled)] {
|
||||||
|
for needle in [
|
||||||
|
"agent.otel.enable",
|
||||||
|
"agent.otel.endpoint",
|
||||||
|
"agent.otel.protocol",
|
||||||
|
"c.example",
|
||||||
|
] {
|
||||||
assert!(
|
assert!(
|
||||||
on_minimal
|
!flake.contains(needle),
|
||||||
.contains("services.hyperhive.agent.otel.endpoint = \"https://c.example/otel\";"),
|
"{name}: `{needle}` must not be rendered:\n{flake}"
|
||||||
"otel endpoint must be injected:\n{on_minimal}"
|
|
||||||
);
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
assert!(
|
assert!(
|
||||||
on_minimal.contains("services.hyperhive.agent.otel.protocol = \"grpc\";"),
|
!plain.contains("agent.otel.extraResourceAttributes"),
|
||||||
"otel protocol must be injected:\n{on_minimal}"
|
"extraResourceAttributes must not appear when unset:\n{plain}"
|
||||||
);
|
);
|
||||||
// Optional fields absent when unset.
|
|
||||||
assert!(
|
assert!(
|
||||||
!on_minimal.contains("services.hyperhive.agent.otel.extraResourceAttributes"),
|
labelled.contains(
|
||||||
"extraResourceAttributes must not appear when unset:\n{on_minimal}"
|
|
||||||
);
|
|
||||||
|
|
||||||
assert!(
|
|
||||||
on_full.contains(
|
|
||||||
"services.hyperhive.agent.otel.extraResourceAttributes = \"deployment.environment=prod\";"
|
"services.hyperhive.agent.otel.extraResourceAttributes = \"deployment.environment=prod\";"
|
||||||
),
|
),
|
||||||
"extraResourceAttributes must be injected when set:\n{on_full}"
|
"extraResourceAttributes must be rendered when set:\n{labelled}"
|
||||||
);
|
|
||||||
|
|
||||||
assert!(
|
|
||||||
!off.contains("services.hyperhive.agent.otel"),
|
|
||||||
"no otel lines when disabled:\n{off}"
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn render_flake_injects_github_disable_only_when_signalled() {
|
fn render_flake_injects_github_disable_only_when_signalled() {
|
||||||
// services.hyperhive.swarm.github.enable = false -> HYPERHIVE_GITHUB_DISABLED
|
// services.hyperhive.swarm.github.enable = false -> HYPERHIVE_GITHUB_DISABLED
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
//! hive's own collector — the same first hop the agents use, named by the
|
//! hive's own collector — the same first hop the agents use, named by the
|
||||||
//! standard `OTEL_EXPORTER_OTLP_ENDPOINT` (see `nix/host-modules/hive-c0re`,
|
//! standard `OTEL_EXPORTER_OTLP_ENDPOINT` (see `nix/host-modules/hive-c0re`,
|
||||||
//! which sets it to the collector's bridge address where agents get its
|
//! which sets it to the collector's bridge address where agents get its
|
||||||
//! swarm-wide name). No toggle of
|
//! fixed name). No toggle of
|
||||||
//! its own, and no credential: a hive's collector takes unauthenticated OTLP
|
//! its own, and no credential: a hive's collector takes unauthenticated OTLP
|
||||||
//! on the bridge, and the only hop that presents anything is the swarm tier's,
|
//! on the bridge, and the only hop that presents anything is the swarm tier's,
|
||||||
//! which is the one that leaves the swarm.
|
//! which is the one that leaves the swarm.
|
||||||
|
|
@ -100,8 +100,7 @@ fn build_provider(
|
||||||
hyperhive_flake: &str,
|
hyperhive_flake: &str,
|
||||||
) -> Result<SdkMeterProvider> {
|
) -> Result<SdkMeterProvider> {
|
||||||
// http/json — the only OTLP transport this crate enables (matching
|
// http/json — the only OTLP transport this crate enables (matching
|
||||||
// hive-metric). The Claude SDK path honours `HYPERHIVE_OTEL_PROTOCOL` for
|
// hive-metric); this exporter is always http/json.
|
||||||
// its own export; this exporter is always http/json.
|
|
||||||
//
|
//
|
||||||
// The address is deliberately NOT passed here. The SDK reads
|
// The address is deliberately NOT passed here. The SDK reads
|
||||||
// `OTEL_EXPORTER_OTLP_ENDPOINT` itself and appends the signal path
|
// `OTEL_EXPORTER_OTLP_ENDPOINT` itself and appends the signal path
|
||||||
|
|
|
||||||
|
|
@ -12,15 +12,15 @@
|
||||||
//! sibling helper records why: two separate per-module mutexes serialise
|
//! sibling helper records why: two separate per-module mutexes serialise
|
||||||
//! nothing against each other, and that produced a CI-only flake. The pair that
|
//! nothing against each other, and that produced a CI-only flake. The pair that
|
||||||
//! motivated this one is `meta::tests` (which renders a flake from
|
//! motivated this one is `meta::tests` (which renders a flake from
|
||||||
//! `HYPERHIVE_OTEL_*`) and `stats::otel_metrics::tests` (which asserts which
|
//! `HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES`) and `stats::otel_metrics::tests` (which asserts which
|
||||||
//! variable enables the exporter) — different modules, same variable, same
|
//! variable enables the exporter) — different modules, same process env, same
|
||||||
//! binary.
|
//! binary.
|
||||||
//!
|
//!
|
||||||
//! ⚠️ **A green local run is weak evidence for this class.** An agent container
|
//! ⚠️ **A green local run is weak evidence for this class.** An agent container
|
||||||
//! has the hyperhive variables ambient-set, so a losing race still finds a
|
//! has the hyperhive variables ambient-set, so a losing race still finds a
|
||||||
//! plausible value; the nix sandbox strips them, so there the race can delete a
|
//! plausible value; the nix sandbox strips them, so there the race can delete a
|
||||||
//! variable out from under another thread. Reproduce the sandbox shape with
|
//! variable out from under another thread. Reproduce the sandbox shape with
|
||||||
//! `env -u HYPERHIVE_OTEL_ENDPOINT cargo test -p hive-c0re`.
|
//! `env -u OTEL_EXPORTER_OTLP_ENDPOINT cargo test -p hive-c0re`.
|
||||||
|
|
||||||
use std::sync::{Mutex, MutexGuard, PoisonError};
|
use std::sync::{Mutex, MutexGuard, PoisonError};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,6 @@ The SDK reads standard OTEL environment variables automatically:
|
||||||
- `OTEL_EXPORTER_OTLP_ENDPOINT` — collector URL (**required**)
|
- `OTEL_EXPORTER_OTLP_ENDPOINT` — collector URL (**required**)
|
||||||
- `OTEL_RESOURCE_ATTRIBUTES` — resource labels (`k=v,...`)
|
- `OTEL_RESOURCE_ATTRIBUTES` — resource labels (`k=v,...`)
|
||||||
|
|
||||||
The harness populates both per-agent when
|
The harness populates both in every agent container: an agent exports straight to
|
||||||
`services.hyperhive.otel.enable = true`: an agent exports straight to
|
|
||||||
its own hive's collector. See `docs/scheduler/observability.md` for
|
its own hive's collector. See `docs/scheduler/observability.md` for
|
||||||
the collector setup and the metrics hyperhive exports.
|
the collector setup and the metrics hyperhive exports.
|
||||||
|
|
|
||||||
|
|
@ -7,8 +7,7 @@
|
||||||
//! - `OTEL_EXPORTER_OTLP_HEADERS` — auth headers (`Key=Value,...`)
|
//! - `OTEL_EXPORTER_OTLP_HEADERS` — auth headers (`Key=Value,...`)
|
||||||
//! - `OTEL_RESOURCE_ATTRIBUTES` — resource labels (`k=v,...`)
|
//! - `OTEL_RESOURCE_ATTRIBUTES` — resource labels (`k=v,...`)
|
||||||
//!
|
//!
|
||||||
//! The harness populates all of these per-agent when
|
//! The harness populates all of these in every agent container.
|
||||||
//! `services.hyperhive.otel.enable = true`.
|
|
||||||
//!
|
//!
|
||||||
//! # Usage
|
//! # Usage
|
||||||
//! ```text
|
//! ```text
|
||||||
|
|
@ -25,7 +24,7 @@ use opentelemetry_sdk::metrics::Temporality;
|
||||||
#[derive(Parser)]
|
#[derive(Parser)]
|
||||||
#[command(
|
#[command(
|
||||||
name = "hive-metric",
|
name = "hive-metric",
|
||||||
about = "Push a labeled metric to the OTEL collector (requires services.hyperhive.otel.enable = true)",
|
about = "Push a labeled metric to the hive's OTEL collector",
|
||||||
// Let a leading-minus value parse as the `value` positional rather than
|
// Let a leading-minus value parse as the `value` positional rather than
|
||||||
// being rejected as an unknown flag — negative gauges are legitimate
|
// being rejected as an unknown flag — negative gauges are legitimate
|
||||||
// (e.g. a delta), and it lets the counter-sign check below produce a clear
|
// (e.g. a delta), and it lets the counter-sign check below produce a clear
|
||||||
|
|
@ -90,7 +89,7 @@ fn main() -> Result<()> {
|
||||||
if std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").is_err() {
|
if std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").is_err() {
|
||||||
bail!(
|
bail!(
|
||||||
"OTEL_EXPORTER_OTLP_ENDPOINT not set — \
|
"OTEL_EXPORTER_OTLP_ENDPOINT not set — \
|
||||||
enable OTEL in NixOS config (services.hyperhive.otel.enable = true)"
|
an agent that sets services.hyperhive.agent.otel.enable = false has no collector"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,8 +15,8 @@
|
||||||
let
|
let
|
||||||
userName = config.services.hyperhive.agent.user.name;
|
userName = config.services.hyperhive.agent.user.name;
|
||||||
homeDir = "/home/${userName}";
|
homeDir = "/home/${userName}";
|
||||||
# Hive-wide OpenTelemetry config (host-driven; baked in per-agent by
|
# OpenTelemetry config (on by default; extra labels baked in per-agent by
|
||||||
# meta.rs `otel_config`). Options declared in `otel.nix`, which also
|
# meta.rs). Options declared in `otel.nix`, which also
|
||||||
# exports the generic OTLP environment this container's producers read.
|
# exports the generic OTLP environment this container's producers read.
|
||||||
otelCfg = config.services.hyperhive.agent.otel;
|
otelCfg = config.services.hyperhive.agent.otel;
|
||||||
# Base claude-code environment applied to every agent regardless of OTEL.
|
# Base claude-code environment applied to every agent regardless of OTEL.
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
# OpenTelemetry wiring for an agent container: the per-agent options the
|
# OpenTelemetry wiring for an agent container: the per-agent options (on by
|
||||||
# meta flake injects (host-driven from `services.hyperhive.otel.*`) plus
|
# default, exporting to the hive's collector by name) plus
|
||||||
# `debug`, which the agent's own config sets, the
|
# `debug`, which the agent's own config sets, the
|
||||||
# environment every OTLP producer inside the container reads, and the
|
# environment every OTLP producer inside the container reads, and the
|
||||||
# collector that forwards this container's journal to the hive.
|
# collector that forwards this container's journal to the hive.
|
||||||
|
|
@ -71,35 +71,35 @@ let
|
||||||
logExporter = if cfg.protocol == "grpc" then "otlp" else "otlphttp";
|
logExporter = if cfg.protocol == "grpc" then "otlp" else "otlphttp";
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
# OTEL stats export is configured ONCE at host level via
|
# On by default: every hive runs a collector and answers
|
||||||
# `services.hyperhive.otel.*` (see nix/host-modules/otel.nix) and
|
# `hive-collector.hive` with its own bridge address, so the endpoint below
|
||||||
# injected into every agent's build by the meta-flake renderer
|
# is the same string in every agent on every hive and nothing is rendered
|
||||||
# (`hive-c0re/src/meta.rs::otel_config`). These per-agent options are
|
# into the meta flake. The port is the literal the host fixes at
|
||||||
# the build-time implementation surface that injection writes into;
|
# `services.hyperhive.swarm.hiveOtel.collectorPort` (read-only there).
|
||||||
# they are not meant to be set directly in an agent.nix. Marked
|
# Marked `internal`: these are not meant to be set in an agent.nix, except
|
||||||
# `internal` so the host option is the only documented operator knob.
|
# `debug`.
|
||||||
options.services.hyperhive.agent.otel = {
|
options.services.hyperhive.agent.otel = {
|
||||||
enable = lib.mkOption {
|
enable = lib.mkOption {
|
||||||
type = lib.types.bool;
|
type = lib.types.bool;
|
||||||
default = false;
|
default = true;
|
||||||
internal = true;
|
internal = true;
|
||||||
description = ''
|
description = ''
|
||||||
Export this agent's telemetry — Claude Code's stats (token
|
Export this agent's telemetry — Claude Code's stats (token
|
||||||
usage, cost, tool calls) and anything pushed with
|
usage, cost, tool calls) and anything pushed with
|
||||||
`hive-metric` — to an OTLP endpoint. Each agent exports
|
`hive-metric` — to the hive's collector. Each agent exports
|
||||||
directly to the hive's collector, so it keeps working even when
|
directly to it, so it keeps working even when hive-c0re is
|
||||||
hive-c0re is down. Host-driven: set
|
down.
|
||||||
`services.hyperhive.otel.enable` instead.
|
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
endpoint = lib.mkOption {
|
endpoint = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "";
|
default = "http://hive-collector.hive:4318";
|
||||||
internal = true;
|
internal = true;
|
||||||
description = ''
|
description = ''
|
||||||
OTLP collector endpoint, set as `OTEL_EXPORTER_OTLP_ENDPOINT`.
|
OTLP collector endpoint, set as `OTEL_EXPORTER_OTLP_ENDPOINT`.
|
||||||
Host-driven via `services.hyperhive.otel.endpoint`.
|
The name is the one `services.hyperhive.otel.collector.host`
|
||||||
|
holds, which the hive's resolver answers.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -113,7 +113,8 @@ in
|
||||||
internal = true;
|
internal = true;
|
||||||
description = ''
|
description = ''
|
||||||
OTLP wire protocol, set as `OTEL_EXPORTER_OTLP_PROTOCOL`.
|
OTLP wire protocol, set as `OTEL_EXPORTER_OTLP_PROTOCOL`.
|
||||||
Host-driven via `services.hyperhive.otel.protocol`.
|
The hive collector's receiver speaks OTLP/HTTP protobuf, so
|
||||||
|
the default is the only value that reaches it.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -297,7 +297,7 @@ in
|
||||||
|
|
||||||
Named `swarm-otel` rather than `otel` because there are two
|
Named `swarm-otel` rather than `otel` because there are two
|
||||||
collectors and the tier is the whole distinction:
|
collectors and the tier is the whole distinction:
|
||||||
{option}`services.hyperhive.otel.enable` is the **hive-tier** one,
|
{option}`services.hyperhive.otel.collector.enable` is the **hive-tier** one,
|
||||||
which every hive runs. A bare `deploy.otel` would not say which
|
which every hive runs. A bare `deploy.otel` would not say which
|
||||||
it meant. A hive that does not run the swarm collector still runs
|
it meant. A hive that does not run the swarm collector still runs
|
||||||
its own, and reaches this one by name at
|
its own, and reaches this one by name at
|
||||||
|
|
|
||||||
|
|
@ -41,7 +41,7 @@ let
|
||||||
# ./otel.nix: `fetches`, the collector loading the file this unit writes
|
# ./otel.nix: `fetches`, the collector loading the file this unit writes
|
||||||
# under the hive's own leaf.
|
# under the hive's own leaf.
|
||||||
swarm-bao-hive-otel-oidc =
|
swarm-bao-hive-otel-oidc =
|
||||||
otel.enable
|
otel.collector.enable
|
||||||
&& hyperhiveCfg.hiveName != null
|
&& hyperhiveCfg.hiveName != null
|
||||||
&& havePair baoDeploy.clientCertFile baoDeploy.clientKeyFile
|
&& havePair baoDeploy.clientCertFile baoDeploy.clientKeyFile
|
||||||
&&
|
&&
|
||||||
|
|
|
||||||
|
|
@ -31,7 +31,7 @@ let
|
||||||
waits = senderAuth && !fetches;
|
waits = senderAuth && !fetches;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
config = lib.mkIf (hyperhiveCfg.deploy.authelia.enable && otel.enable && waits) {
|
config = lib.mkIf (hyperhiveCfg.deploy.authelia.enable && otel.collector.enable && waits) {
|
||||||
systemd.services.hive-otel-client-secret = {
|
systemd.services.hive-otel-client-secret = {
|
||||||
after = [ "container@${autheliaMachine}.service" ];
|
after = [ "container@${autheliaMachine}.service" ];
|
||||||
requires = [ "container@${autheliaMachine}.service" ];
|
requires = [ "container@${autheliaMachine}.service" ];
|
||||||
|
|
|
||||||
|
|
@ -71,64 +71,34 @@ in
|
||||||
# every agent. See services.hyperhive.swarm.github.enable.
|
# every agent. See services.hyperhive.swarm.github.enable.
|
||||||
HYPERHIVE_GITHUB_DISABLED = "1";
|
HYPERHIVE_GITHUB_DISABLED = "1";
|
||||||
}
|
}
|
||||||
// lib.optionalAttrs config.services.hyperhive.otel.enable (
|
// {
|
||||||
# OTEL config (`swarm.hiveOtel`, `deploy.hive-controller.otel`, gated on
|
# hive-c0re's OWN container-resource exporter (stats/otel_metrics.rs) reads
|
||||||
# this hive's `otel.enable`) -> read by meta.rs::otel_config and injected
|
# the STANDARD OTLP variable — the same one hive-metric and every agent
|
||||||
# as build-time `services.hyperhive.agent.otel.*` into every agent.
|
# read — and lets the SDK resolve the URL, which appends the signal path
|
||||||
# Endpoint presence is the enable signal on the meta side; the
|
# (`/v1/metrics`). Handing the SDK an address programmatically instead
|
||||||
# optional fields are only emitted when set so absent values
|
# takes it verbatim: it POSTs to the collector's root, gets a 404 on every
|
||||||
# don't render no-op env lines.
|
# export, and says nothing, because OTLP export failures go to an error
|
||||||
let
|
# handler no binary here installs.
|
||||||
otel = config.services.hyperhive.otel;
|
|
||||||
hiveOtel = config.services.hyperhive.swarm.hiveOtel;
|
|
||||||
controllerOtel = config.services.hyperhive.deploy.hive-controller.otel;
|
|
||||||
# The first hop, this hive's own collector, under the two spellings its
|
|
||||||
# two kinds of caller need. Agents get the swarm-wide name, so their
|
|
||||||
# builds carry nothing hive-specific; this daemon gets the bridge
|
|
||||||
# address, because on the host the name may resolve to loopback
|
|
||||||
# (`gateway.localHostsEntry`), where the receiver does not listen.
|
|
||||||
agentFirstHop = "http://${otel.collector.host}:${toString hiveOtel.collectorPort}";
|
|
||||||
hostFirstHop = "http://${config.services.hyperhive.network.bridgeIp}:${toString hiveOtel.collectorPort}";
|
|
||||||
in
|
|
||||||
{
|
|
||||||
# `otel.endpoint` means "where telemetry ultimately goes" and keeps
|
|
||||||
# that meaning; what agents are handed is the *first hop*, which is
|
|
||||||
# always this hive's own collector. Deriving it rather than
|
|
||||||
# redefining `endpoint` is what lets every existing deployment keep
|
|
||||||
# its configured value untouched.
|
|
||||||
HYPERHIVE_OTEL_ENDPOINT = agentFirstHop;
|
|
||||||
# hive-c0re's OWN container-resource exporter (stats/otel_metrics.rs)
|
|
||||||
# reads the STANDARD OTLP variable — the same one hive-metric and every
|
|
||||||
# agent read — and lets the SDK resolve the URL, which appends the
|
|
||||||
# signal path (`/v1/metrics`). Handing the SDK an address
|
|
||||||
# programmatically instead takes it verbatim: it POSTs to the
|
|
||||||
# collector's root, gets a 404 on every export, and says nothing,
|
|
||||||
# because OTLP export failures go to an error handler no binary here
|
|
||||||
# installs — the daemon logged "exporter enabled" and never delivered a
|
|
||||||
# sample, for as long as it had that exporter. The variable above cannot
|
|
||||||
# replace this one:
|
|
||||||
# it is the agent-config transport meta.rs reads, and the SDK does not
|
|
||||||
# know that name.
|
|
||||||
OTEL_EXPORTER_OTLP_ENDPOINT = hostFirstHop;
|
|
||||||
# The first hop is the collector's OTLP/HTTP receiver, which speaks
|
|
||||||
# protobuf regardless of what the upstream wants — `otel.protocol`
|
|
||||||
# describes the *upstream* link, and the collector's own exporter is
|
|
||||||
# what has to honour it (see nix/host-modules/otel.nix).
|
|
||||||
HYPERHIVE_OTEL_PROTOCOL = "http/protobuf";
|
|
||||||
}
|
|
||||||
// lib.optionalAttrs (controllerOtel.extraResourceAttributes != "") {
|
|
||||||
HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES = controllerOtel.extraResourceAttributes;
|
|
||||||
}
|
|
||||||
# HYPERHIVE_OTEL_HEADERS_CREDENTIAL is deliberately NOT emitted, and
|
|
||||||
# its absence is the security half of this design: that variable would
|
|
||||||
# put the upstream token into an agent's own settings.json. Nothing
|
|
||||||
# delivers the token into the container — no nspawn credential, no
|
|
||||||
# agent-readable header file.
|
|
||||||
#
|
#
|
||||||
# The rule: the collector holding the credential achieves nothing while
|
# The bridge address, not `otel.collector.host`: on the host that name may
|
||||||
# anything else hands out a copy, so there is exactly one holder and it
|
# resolve to loopback (`gateway.localHostsEntry`), where the receiver does
|
||||||
# is on the host.
|
# not listen. Agents are not told an address at all; their module defaults
|
||||||
)
|
# to the collector's name.
|
||||||
|
OTEL_EXPORTER_OTLP_ENDPOINT = "http://${config.services.hyperhive.network.bridgeIp}:${toString config.services.hyperhive.swarm.hiveOtel.collectorPort}";
|
||||||
|
}
|
||||||
|
//
|
||||||
|
lib.optionalAttrs
|
||||||
|
(config.services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes != "")
|
||||||
|
{
|
||||||
|
HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES =
|
||||||
|
config.services.hyperhive.deploy.hive-controller.otel.extraResourceAttributes;
|
||||||
|
}
|
||||||
|
# No upstream credential is emitted here, and its absence is the security half
|
||||||
|
# of this design: it would put the upstream token where an agent could read
|
||||||
|
# it. Nothing delivers the token into a container — no nspawn credential, no
|
||||||
|
# agent-readable header file. The collector holding the credential achieves
|
||||||
|
# nothing while anything else hands out a copy, so there is exactly one
|
||||||
|
# holder and it is on the host.
|
||||||
// {
|
// {
|
||||||
# In-cluster forge URL — the gateway vhost (`forge.<domain>`), which
|
# In-cluster forge URL — the gateway vhost (`forge.<domain>`), which
|
||||||
# nginx proxies to forgejo. Used both for internal API calls in
|
# nginx proxies to forgejo. Used both for internal API calls in
|
||||||
|
|
|
||||||
|
|
@ -320,7 +320,7 @@ in
|
||||||
description = ''
|
description = ''
|
||||||
Loopback TCP port nginx serves its `stub_status` counters on, at
|
Loopback TCP port nginx serves its `stub_status` counters on, at
|
||||||
`http://127.0.0.1:''${statusPort}/stub_status`. The hive's collector
|
`http://127.0.0.1:''${statusPort}/stub_status`. The hive's collector
|
||||||
({option}`services.hyperhive.otel.enable`) reads it with its `nginx`
|
(every hive runs one) reads it with its `nginx`
|
||||||
receiver.
|
receiver.
|
||||||
|
|
||||||
TCP rather than a unix socket because that receiver can only dial
|
TCP rather than a unix socket because that receiver can only dial
|
||||||
|
|
|
||||||
|
|
@ -55,7 +55,6 @@ let
|
||||||
h.gateway.enable
|
h.gateway.enable
|
||||||
h.gateway.dns.enable
|
h.gateway.dns.enable
|
||||||
h.network.enable
|
h.network.enable
|
||||||
h.otel.enable
|
|
||||||
h.snapshotStore.enable
|
h.snapshotStore.enable
|
||||||
];
|
];
|
||||||
in
|
in
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,8 @@
|
||||||
# Hive-wide OTEL stats export, and the HIVE tier of the collector pair.
|
# The HIVE tier of the collector pair: every hive runs one, on the host that
|
||||||
# Set ONCE here at host level; the meta-flake renderer
|
# runs hive-c0re (`deploy.hive-controller.enable`). Agents export to it at
|
||||||
# (`hive-c0re/src/meta.rs::otel_config`) reads the HYPERHIVE_OTEL_* env
|
# the fixed name `services.hyperhive.otel.collector.host`, which this hive's
|
||||||
# exported off hive-c0re's unit (see ./hive-c0re) and injects the matching
|
# resolver answers with its own bridge IP, so an agent's build carries
|
||||||
# `services.hyperhive.agent.otel.*` build-time config into EVERY agent (mirroring the
|
# nothing hive-specific.
|
||||||
# CA-cert injection), so each agent's harness exports its own Claude Code
|
|
||||||
# stats directly to the collector. There is no per-agent opt-in — this is
|
|
||||||
# the single switch for the whole hive.
|
|
||||||
#
|
#
|
||||||
# This tier receives from this hive's agents and forwards to the swarm's
|
# This tier receives from this hive's agents and forwards to the swarm's
|
||||||
# collector (./swarm-otel.nix). It holds no credential and picks no
|
# collector (./swarm-otel.nix). It holds no credential and picks no
|
||||||
|
|
@ -41,6 +38,8 @@ let
|
||||||
# The hive's own store identity. The collector authenticates to the swarm
|
# The hive's own store identity. The collector authenticates to the swarm
|
||||||
# as the hive, so it reads its secret as the hive: the hive leaf's policy
|
# as the hive, so it reads its secret as the hive: the hive leaf's policy
|
||||||
# (`swarm-secret-client`'s `policy::render`) grants `swarm/hives/<hive>/*`.
|
# (`swarm-secret-client`'s `policy::render`) grants `swarm/hives/<hive>/*`.
|
||||||
|
runsHive = config.services.hyperhive.otel.collector.enable;
|
||||||
|
|
||||||
hiveLeaf =
|
hiveLeaf =
|
||||||
config.services.hyperhive.deploy.bao.clientCertFile != null
|
config.services.hyperhive.deploy.bao.clientCertFile != null
|
||||||
&& config.services.hyperhive.deploy.bao.clientKeyFile != null;
|
&& config.services.hyperhive.deploy.bao.clientKeyFile != null;
|
||||||
|
|
@ -55,39 +54,19 @@ in
|
||||||
name = "hive-otel";
|
name = "hive-otel";
|
||||||
consumers = [ "opentelemetry-collector" ];
|
consumers = [ "opentelemetry-collector" ];
|
||||||
hostUnit = true;
|
hostUnit = true;
|
||||||
enable = config.services.hyperhive.otel.enable;
|
enable = runsHive;
|
||||||
})
|
})
|
||||||
(lib.mkRenamedOptionModule
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "otel" "enable" ] ''
|
||||||
[ "services" "hyperhive" "otel" "collector" "port" ]
|
Every hive runs its own telemetry collector, and every agent exports to
|
||||||
[ "services" "hyperhive" "swarm" "hiveOtel" "collectorPort" ]
|
it by default, so there is no switch. Remove this definition.
|
||||||
)
|
'')
|
||||||
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "otel" "collector" "port" ] ''
|
||||||
|
The hive collector's receiver port is fixed at 4318: agents dial that
|
||||||
|
literal. Remove this definition.
|
||||||
|
'')
|
||||||
];
|
];
|
||||||
|
|
||||||
options.services.hyperhive.otel = {
|
options.services.hyperhive.otel = {
|
||||||
enable = lib.mkEnableOption ''
|
|
||||||
hive-wide export of every agent's Claude Code stats (token usage,
|
|
||||||
cost, tool calls) to an OTLP endpoint via Claude Code's built-in
|
|
||||||
OpenTelemetry. One switch for all agents.
|
|
||||||
|
|
||||||
Enabling this also runs this hive's collector on the host: there is
|
|
||||||
exactly one way telemetry leaves this hive, and it is through that
|
|
||||||
collector. Agents export unauthenticated to
|
|
||||||
{option}`services.hyperhive.otel.collector.host`, a name this hive's
|
|
||||||
resolver answers with a bridge address only its own containers can
|
|
||||||
reach.
|
|
||||||
|
|
||||||
That collector forwards to the swarm's
|
|
||||||
({option}`services.hyperhive.deploy.swarm-otel.enable`), which holds the
|
|
||||||
upstream credential and writes the swarm's store. So an agent never
|
|
||||||
sees the credential, and neither does this tier.
|
|
||||||
|
|
||||||
⚠️ The collector is therefore in the path of all telemetry. It runs
|
|
||||||
on the same host as the agents and restarts on failure, and
|
|
||||||
telemetry is not the control plane, so degraded telemetry is not
|
|
||||||
degraded operation — but the export does not survive independently
|
|
||||||
of anything host-side
|
|
||||||
'';
|
|
||||||
|
|
||||||
endpoint = lib.mkOption {
|
endpoint = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
default = "";
|
default = "";
|
||||||
|
|
@ -171,25 +150,51 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
collector.enable = lib.mkOption {
|
||||||
|
type = lib.types.bool;
|
||||||
|
readOnly = true;
|
||||||
|
default =
|
||||||
|
config.services.hyperhive.deploy.hive-controller.enable
|
||||||
|
|| config.services.hyperhive.deploy.allSwarmServices;
|
||||||
|
defaultText = lib.literalExpression "deploy.hive-controller.enable || deploy.allSwarmServices";
|
||||||
|
description = ''
|
||||||
|
Whether this host runs the hive tier's collector. Not a switch:
|
||||||
|
every host that runs a hive
|
||||||
|
({option}`services.hyperhive.deploy.hive-controller.enable`) runs
|
||||||
|
one, and so does a host carrying all the swarm's shared services
|
||||||
|
({option}`services.hyperhive.deploy.allSwarmServices`), whose own
|
||||||
|
containers export to it.
|
||||||
|
|
||||||
|
Agents export to it unauthenticated at
|
||||||
|
{option}`services.hyperhive.otel.collector.host`, a name this
|
||||||
|
hive's resolver answers with a bridge address only its own
|
||||||
|
containers can reach. It forwards to the swarm's collector
|
||||||
|
({option}`services.hyperhive.deploy.swarm-otel.enable`), which holds
|
||||||
|
the upstream credential and writes the swarm's store, so neither an
|
||||||
|
agent nor this tier ever sees the credential.
|
||||||
|
|
||||||
|
⚠️ The collector is in the path of all telemetry. It runs on the
|
||||||
|
same host as the agents and restarts on failure; telemetry is not
|
||||||
|
the control plane, so degraded telemetry is not degraded
|
||||||
|
operation, but the export does not survive independently of
|
||||||
|
anything host-side.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
collector.host = lib.mkOption {
|
collector.host = lib.mkOption {
|
||||||
type = lib.types.str;
|
type = lib.types.str;
|
||||||
readOnly = true;
|
readOnly = true;
|
||||||
# Total on a null swarm domain, as in ./swarm-otel-service.nix: the
|
default = "hive-collector.hive";
|
||||||
# required-domain assertion should be what an operator sees.
|
|
||||||
default = "hive-collector.${
|
|
||||||
if config.services.hyperhive.swarm.domain == null then
|
|
||||||
"invalid"
|
|
||||||
else
|
|
||||||
config.services.hyperhive.swarm.domain
|
|
||||||
}";
|
|
||||||
defaultText = lib.literalExpression ''"hive-collector.''${services.hyperhive.swarm.domain}"'';
|
|
||||||
description = ''
|
description = ''
|
||||||
The name agents reach this hive's collector on. The same string on
|
The name agents reach this hive's collector on. A literal, the same
|
||||||
every hive: each hive's own resolver answers it with its own bridge
|
on every hive and in every swarm: each hive's own resolver answers
|
||||||
IP, so an agent's build carries no hive-specific address.
|
it with its own bridge IP, so an agent's build carries neither a
|
||||||
|
swarm domain nor a hive-specific address. The agent module's
|
||||||
|
`services.hyperhive.agent.otel.endpoint` default spells the same
|
||||||
|
name.
|
||||||
|
|
||||||
No hive can claim this name as its domain:
|
It sits outside every domain a hive or the swarm serves: a hive's
|
||||||
`nix/reserved-hive-fragments.nix` forbids `hive` in any hive name.
|
domain is `<hiveName>.<swarm.domain>`, so it never ends in `.hive`.
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -274,12 +279,14 @@ in
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
# Shared by every hive's otel tier and its agents, so an agent's build
|
# Shared by every hive's otel tier and its agents. Read-only because the agent
|
||||||
# carries the same port whichever hive renders it.
|
# module's `otel.endpoint` default spells the same literal, and a host that
|
||||||
|
# moved the port would leave its agents dialling the old one.
|
||||||
options.services.hyperhive.swarm.hiveOtel = {
|
options.services.hyperhive.swarm.hiveOtel = {
|
||||||
collectorPort = lib.mkOption {
|
collectorPort = lib.mkOption {
|
||||||
type = lib.types.port;
|
type = lib.types.port;
|
||||||
default = 4318;
|
default = 4318;
|
||||||
|
readOnly = true;
|
||||||
description = ''
|
description = ''
|
||||||
Port of every hive collector's OTLP/HTTP receiver, at that hive's
|
Port of every hive collector's OTLP/HTTP receiver, at that hive's
|
||||||
`services.hyperhive.network.bridgeIp`: the port agents dial at
|
`services.hyperhive.network.bridgeIp`: the port agents dial at
|
||||||
|
|
@ -307,7 +314,7 @@ in
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
config = lib.mkIf config.services.hyperhive.otel.enable (
|
config = lib.mkIf runsHive (
|
||||||
let
|
let
|
||||||
otel = config.services.hyperhive.otel;
|
otel = config.services.hyperhive.otel;
|
||||||
autheliaCfg = config.services.hyperhive.swarm.authelia;
|
autheliaCfg = config.services.hyperhive.swarm.authelia;
|
||||||
|
|
@ -913,7 +920,7 @@ in
|
||||||
# cleanly, retries forever, and reports nothing to anyone.
|
# cleanly, retries forever, and reports nothing to anyone.
|
||||||
assertion = senderAuth;
|
assertion = senderAuth;
|
||||||
message = ''
|
message = ''
|
||||||
services.hyperhive.otel.enable is true but this hive has no
|
This hive runs a telemetry collector (every hive does) but has no
|
||||||
identity to present to the swarm's collector:
|
identity to present to the swarm's collector:
|
||||||
|
|
||||||
services.hyperhive.otel.clientSecretFile = ${
|
services.hyperhive.otel.clientSecretFile = ${
|
||||||
|
|
@ -948,7 +955,7 @@ in
|
||||||
"none"
|
"none"
|
||||||
]);
|
]);
|
||||||
message = ''
|
message = ''
|
||||||
services.hyperhive.otel.enable is true, which runs a journald
|
This hive runs a telemetry collector, which runs a journald
|
||||||
receiver reading /var/log/journal for hive-c0re's own crash-watch
|
receiver reading /var/log/journal for hive-c0re's own crash-watch
|
||||||
log line — but services.journald.storage is
|
log line — but services.journald.storage is
|
||||||
"${config.services.journald.storage}" on this host.
|
"${config.services.journald.storage}" on this host.
|
||||||
|
|
|
||||||
|
|
@ -60,7 +60,7 @@ let
|
||||||
# Whether the swarm collector's client is actually registered here.
|
# Whether the swarm collector's client is actually registered here.
|
||||||
#
|
#
|
||||||
# Asked of the client list rather than re-derived from the conditions
|
# Asked of the client list rather than re-derived from the conditions
|
||||||
# that produce it (`otel.enable`, a non-empty audience set, authelia
|
# that produce it (a non-empty audience set, authelia
|
||||||
# being co-located). A second copy of that predicate is a second thing
|
# being co-located). A second copy of that predicate is a second thing
|
||||||
# to keep in step, and the two drifting is not a build failure: a rule
|
# to keep in step, and the two drifting is not a build failure: a rule
|
||||||
# naming an unregistered client is refused by authelia's *startup*
|
# naming an unregistered client is refused by authelia's *startup*
|
||||||
|
|
|
||||||
|
|
@ -1446,7 +1446,7 @@ let
|
||||||
#
|
#
|
||||||
# "Does the swarm have a collector" is not a question that can be false —
|
# "Does the swarm have a collector" is not a question that can be false —
|
||||||
# the swarm always has at least one instance of all the swarm level
|
# the swarm always has at least one instance of all the swarm level
|
||||||
# services — so gating the forwarder on `otel.enable` or
|
# services — so gating the forwarder on `otel.collector.enable` or
|
||||||
# `deploy.swarm-otel.enable` would ask about the wrong host and render no
|
# `deploy.swarm-otel.enable` would ask about the wrong host and render no
|
||||||
# forwarder wherever the answer happened to be no.
|
# forwarder wherever the answer happened to be no.
|
||||||
#
|
#
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,7 @@ in
|
||||||
{
|
{
|
||||||
# `enable` is `services.hyperhive.deploy.swarm-otel.enable` — see ./deploy.nix.
|
# `enable` is `services.hyperhive.deploy.swarm-otel.enable` — see ./deploy.nix.
|
||||||
# ⚠️ That is the SWARM collector. The per-hive one keeps its own
|
# ⚠️ That is the SWARM collector. The per-hive one keeps its own
|
||||||
# `services.hyperhive.otel.enable` (./otel.nix) and is a different
|
# `services.hyperhive.otel.collector.enable` (./otel.nix) and is a different
|
||||||
# option entirely — every hive runs that one.
|
# option entirely — every hive runs that one.
|
||||||
options.services.hyperhive.swarm.otel = {
|
options.services.hyperhive.swarm.otel = {
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -49,10 +49,6 @@ in
|
||||||
# host. Same switch, same rule, one attribute path.
|
# host. Same switch, same rule, one attribute path.
|
||||||
config.services.hyperhive.deploy.matrix.enable = lib.mkDefault deployCfg.allSwarmServices;
|
config.services.hyperhive.deploy.matrix.enable = lib.mkDefault deployCfg.allSwarmServices;
|
||||||
|
|
||||||
# The collector that feeds the pair above (note: no `swarm.` prefix,
|
|
||||||
# this is ./otel.nix's existing per-hive option).
|
|
||||||
config.services.hyperhive.otel.enable = lib.mkDefault deployCfg.allSwarmServices;
|
|
||||||
|
|
||||||
# The rest of the shared services, from the same switch and for the same
|
# The rest of the shared services, from the same switch and for the same
|
||||||
# reason.
|
# reason.
|
||||||
#
|
#
|
||||||
|
|
|
||||||
|
|
@ -97,7 +97,7 @@ let
|
||||||
# where this host runs a hive-tier collector on the bridge. The daemon adds
|
# where this host runs a hive-tier collector on the bridge. The daemon adds
|
||||||
# the repo and the PR to each `claude` run's resource attributes, and to
|
# the repo and the PR to each `claude` run's resource attributes, and to
|
||||||
# each cost record's attributes.
|
# each cost record's attributes.
|
||||||
otelEnv = lib.optionalAttrs otelCfg.enable {
|
otelEnv = lib.optionalAttrs otelCfg.collector.enable {
|
||||||
CLAUDE_CODE_ENABLE_TELEMETRY = "1";
|
CLAUDE_CODE_ENABLE_TELEMETRY = "1";
|
||||||
OTEL_METRICS_EXPORTER = "otlp";
|
OTEL_METRICS_EXPORTER = "otlp";
|
||||||
OTEL_EXPORTER_OTLP_ENDPOINT = "http://${networkCfg.bridgeIp}:${toString hyperhiveCfg.swarm.hiveOtel.collectorPort}";
|
OTEL_EXPORTER_OTLP_ENDPOINT = "http://${networkCfg.bridgeIp}:${toString hyperhiveCfg.swarm.hiveOtel.collectorPort}";
|
||||||
|
|
|
||||||
|
|
@ -47,7 +47,11 @@ let
|
||||||
otel.metricIntervalMs = 10000;
|
otel.metricIntervalMs = 10000;
|
||||||
};
|
};
|
||||||
|
|
||||||
agentNoOtel = shared.agentBare;
|
# Sets nothing: what an agent gets on a hive, whose meta flake renders no
|
||||||
|
# otel line.
|
||||||
|
agentDefault = shared.agentBare;
|
||||||
|
|
||||||
|
agentNoOtel = agent { otel.enable = false; };
|
||||||
|
|
||||||
agentSettings = machine: machine.services.opentelemetry-collector.settings;
|
agentSettings = machine: machine.services.opentelemetry-collector.settings;
|
||||||
cases = [
|
cases = [
|
||||||
|
|
@ -100,11 +104,30 @@ let
|
||||||
&& p.exporters != [ ]
|
&& p.exporters != [ ]
|
||||||
&& lib.all (e: (s.exporters ? ${e}) && s.exporters.${e}.endpoint == agentBridge) p.exporters;
|
&& lib.all (e: (s.exporters ? ${e}) && s.exporters.${e}.endpoint == agentBridge) p.exporters;
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
# 🎯 What the meta flake no longer says: an agent that sets nothing
|
||||||
|
# exports, to a name rather than an address. The name carries no swarm
|
||||||
|
# domain and no hive, so the same build is correct on every hive; the
|
||||||
|
# resolver of the hive it runs on answers it. `agentNoOtel` above is the
|
||||||
|
# control that these lookups find nothing where the switch is off.
|
||||||
|
name = "an agent with no otel config exports to the fixed hive-collector name by default";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
endpoint = "http://hive-collector.hive:4318";
|
||||||
|
env = agentDefault.systemd.globalEnvironment;
|
||||||
|
s = agentSettings agentDefault;
|
||||||
|
in
|
||||||
|
agentDefault.services.hyperhive.agent.otel.enable
|
||||||
|
&& env.OTEL_EXPORTER_OTLP_ENDPOINT == endpoint
|
||||||
|
&& agentDefault.environment.variables.OTEL_EXPORTER_OTLP_ENDPOINT == endpoint
|
||||||
|
&& lib.all (e: s.exporters.${e}.endpoint == endpoint) s.service.pipelines.logs.exporters
|
||||||
|
&& !(agentNoOtel.systemd.globalEnvironment ? OTEL_EXPORTER_OTLP_ENDPOINT);
|
||||||
|
}
|
||||||
{
|
{
|
||||||
# Presence control for the two cases above: with the switch off there
|
# Presence control for the two cases above: with the switch off there
|
||||||
# is no collector in the container at all, so their passing is about
|
# is no collector in the container at all, so their passing is about
|
||||||
# the wiring rather than about a unit that renders regardless.
|
# the wiring rather than about a unit that renders regardless.
|
||||||
name = "an agent that has not opted into telemetry runs no collector";
|
name = "an agent that switched telemetry off runs no collector";
|
||||||
ok = !agentNoOtel.services.opentelemetry-collector.enable;
|
ok = !agentNoOtel.services.opentelemetry-collector.enable;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
|
|
|
||||||
|
|
@ -283,11 +283,11 @@ let
|
||||||
{
|
{
|
||||||
# The case the old hive-tier gate got wrong: this host runs the swarm's
|
# The case the old hive-tier gate got wrong: this host runs the swarm's
|
||||||
# collector and no hive collector at all, and the forwarder still exists
|
# collector and no hive collector at all, and the forwarder still exists
|
||||||
# with an address that resolves. Gated on `otel.enable` it rendered
|
# with an address that resolves. Gated on `otel.collector.enable` it rendered
|
||||||
# nothing here, so the store's journal left no trace anywhere.
|
# nothing here, so the store's journal left no trace anywhere.
|
||||||
name = "a store on a host with no hive collector still forwards";
|
name = "a store on a host with no hive collector still forwards";
|
||||||
ok =
|
ok =
|
||||||
!baoWithCollector.services.hyperhive.otel.enable
|
!baoWithCollector.services.hyperhive.otel.collector.enable
|
||||||
&& (baoForwarder baoWithCollector).enable
|
&& (baoForwarder baoWithCollector).enable
|
||||||
&& (baoForwarder baoWithCollector).settings.exporters.otlphttp.endpoint != "";
|
&& (baoForwarder baoWithCollector).settings.exporters.otlphttp.endpoint != "";
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -233,18 +233,22 @@ let
|
||||||
&& e.HIVE_AGENT_NATS_URL == e.HIVE_C0RE_NATS_URL;
|
&& e.HIVE_AGENT_NATS_URL == e.HIVE_C0RE_NATS_URL;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# What agents are handed lands in their builds, so it has to be the same
|
# Agents are handed no address at all: their module defaults to a
|
||||||
# on every hive: a name, which this hive's resolver answers with the
|
# literal name, the same on every hive and in every swarm, which this
|
||||||
# bridge IP its collector binds. hive-c0re itself keeps the address.
|
# hive's resolver answers with the bridge IP its collector binds.
|
||||||
name = "the agents' collector address is a swarm-wide name this hive resolves to its bridge";
|
# hive-c0re itself keeps the address. `HYPERHIVE_OTEL_ENDPOINT` is gone,
|
||||||
|
# and the bridge address next to it is the control that the lookup on
|
||||||
|
# the same attrset finds what is there.
|
||||||
|
name = "the agents' collector address is a fixed name this hive resolves to its bridge";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
e = allLocal.systemd.services.hive-c0re.environment;
|
e = allLocal.systemd.services.hive-c0re.environment;
|
||||||
bridge = allLocal.services.hyperhive.network.bridgeIp;
|
bridge = allLocal.services.hyperhive.network.bridgeIp;
|
||||||
in
|
in
|
||||||
e.HYPERHIVE_OTEL_ENDPOINT == "http://hive-collector.t.local:4318"
|
!(e ? HYPERHIVE_OTEL_ENDPOINT)
|
||||||
&& e.OTEL_EXPORTER_OTLP_ENDPOINT == "http://${bridge}:4318"
|
&& e.OTEL_EXPORTER_OTLP_ENDPOINT == "http://${bridge}:4318"
|
||||||
&& lib.elem "/hive-collector.t.local/${bridge}" allLocal.services.dnsmasq.settings.address;
|
&& allLocal.services.hyperhive.otel.collector.host == "hive-collector.hive"
|
||||||
|
&& lib.elem "/hive-collector.hive/${bridge}" allLocal.services.dnsmasq.settings.address;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The agents mint against the swarm's IdP, the same endpoint the hive's
|
# The agents mint against the swarm's IdP, the same endpoint the hive's
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,7 @@ let
|
||||||
;
|
;
|
||||||
})
|
})
|
||||||
hive
|
hive
|
||||||
|
hiveWithController
|
||||||
runGroup
|
runGroup
|
||||||
otelSettings
|
otelSettings
|
||||||
;
|
;
|
||||||
|
|
@ -27,24 +28,22 @@ let
|
||||||
# tier's, which lives in a container and is read through `otelSettings`.
|
# tier's, which lives in a container and is read through `otelSettings`.
|
||||||
# The IdP is here too, with a hand-named secret file: the collector's wait
|
# The IdP is here too, with a hand-named secret file: the collector's wait
|
||||||
# for that file exists beside authelia's container.
|
# for that file exists beside authelia's container.
|
||||||
hiveOtel = hive {
|
hiveOtel = hiveWithController {
|
||||||
otel.enable = true;
|
|
||||||
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
|
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
|
||||||
deploy.authelia.enable = true;
|
deploy.authelia.enable = true;
|
||||||
};
|
};
|
||||||
|
|
||||||
# A hive holding its own store leaf, placed by hand, and naming no secret
|
# A hive holding its own store leaf, placed by hand, and naming no secret
|
||||||
# file: the collector's secret comes from the store under that leaf.
|
# file: the collector's secret comes from the store under that leaf.
|
||||||
hiveOtelLeaf = hive {
|
hiveOtelLeaf = hiveWithController {
|
||||||
otel.enable = true;
|
gateway.enable = false;
|
||||||
deploy.bao.clientCertFile = "/etc/pki/hive.pem";
|
deploy.bao.clientCertFile = "/etc/pki/hive.pem";
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/hive-key.pem";
|
deploy.bao.clientKeyFile = "/etc/pki/hive-key.pem";
|
||||||
};
|
};
|
||||||
|
|
||||||
# The IdP, the store and the collector on one host: the publisher that
|
# The IdP, the store and the collector on one host: the publisher that
|
||||||
# writes the secret and the unit that reads it, in one config.
|
# writes the secret and the unit that reads it, in one config.
|
||||||
hiveOtelAllLocal = hive {
|
hiveOtelAllLocal = hiveWithController {
|
||||||
otel.enable = true;
|
|
||||||
deploy.bao.enable = true;
|
deploy.bao.enable = true;
|
||||||
deploy.authelia.enable = true;
|
deploy.authelia.enable = true;
|
||||||
};
|
};
|
||||||
|
|
@ -144,7 +143,7 @@ let
|
||||||
{
|
{
|
||||||
# The receiver, the pipeline entry and the nginx server it reads have to
|
# The receiver, the pipeline entry and the nginx server it reads have to
|
||||||
# agree on one address, and none of them may exist without a gateway on
|
# agree on one address, and none of them may exist without a gateway on
|
||||||
# the host. `hiveOtelLeaf` is the control: it runs no gateway, and the
|
# the host. `hiveOtelLeaf` is the control: it opts out of the gateway, and the
|
||||||
# clause asserting that keeps the control from going vacuous.
|
# clause asserting that keeps the control from going vacuous.
|
||||||
name = "the hive collector reads the gateway's stub_status, and only where a gateway runs";
|
name = "the hive collector reads the gateway's stub_status, and only where a gateway runs";
|
||||||
ok =
|
ok =
|
||||||
|
|
|
||||||
|
|
@ -221,7 +221,7 @@ let
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
matrix = h.deploy.matrix.enable;
|
matrix = h.deploy.matrix.enable;
|
||||||
otel = h.otel.enable;
|
otel = h.otel.collector.enable;
|
||||||
authelia = h.deploy.authelia.enable;
|
authelia = h.deploy.authelia.enable;
|
||||||
nats = h.deploy.nats.enable;
|
nats = h.deploy.nats.enable;
|
||||||
swarm-otel = h.deploy.swarm-otel.enable;
|
swarm-otel = h.deploy.swarm-otel.enable;
|
||||||
|
|
|
||||||
|
|
@ -99,10 +99,9 @@ in
|
||||||
};
|
};
|
||||||
|
|
||||||
# `clientSecretFile` is what ../host-modules/otel.nix's identity assertion
|
# `clientSecretFile` is what ../host-modules/otel.nix's identity assertion
|
||||||
# demands of any hive with the tier on.
|
# demands of any host running a hive.
|
||||||
baoWithHiveOtel = hive {
|
baoWithHiveOtel = hiveWithController {
|
||||||
deploy.bao.enable = true;
|
deploy.bao.enable = true;
|
||||||
otel.enable = true;
|
|
||||||
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
|
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -50,7 +50,6 @@ let
|
||||||
deploy.hive-controller.enable = false;
|
deploy.hive-controller.enable = false;
|
||||||
deploy.allSwarmServices = true;
|
deploy.allSwarmServices = true;
|
||||||
deploy.matrix.enable = false;
|
deploy.matrix.enable = false;
|
||||||
otel.enable = false;
|
|
||||||
deploy.authelia.enable = false;
|
deploy.authelia.enable = false;
|
||||||
deploy.nats.enable = false;
|
deploy.nats.enable = false;
|
||||||
deploy.swarm-otel.enable = false;
|
deploy.swarm-otel.enable = false;
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue