Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
41a7974b9e | ||
|
|
2977dc91d1 |
7 changed files with 280 additions and 124 deletions
|
|
@ -183,7 +183,7 @@ host-side service on another port — for example a dev OTLP collector you want
|
||||||
agents to reach directly — is unreachable. (hyperhive's own services
|
agents to reach directly — is unreachable. (hyperhive's own services
|
||||||
need none of this: a module that means to be reachable from agents opens
|
need none of this: a module that means to be reachable from agents opens
|
||||||
its own port here, which the hive collector and `nats.enable` both do. Note
|
its own port here, which the hive collector and `nats.enable` both do. Note
|
||||||
that `otel.endpoint` is the _upstream_, which no agent ever dials. See
|
that `deploy.swarm-otel.upstream.endpoint` is the _upstream_, which no agent ever dials. See
|
||||||
`docs/scheduler/observability.md`.)
|
`docs/scheduler/observability.md`.)
|
||||||
|
|
||||||
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
|
`services.hyperhive.network.exposeHostPorts = [ 4318 ];` opens each
|
||||||
|
|
|
||||||
|
|
@ -23,7 +23,7 @@ the host is:
|
||||||
An all-local host runs both; the hop between them configures itself.
|
An all-local host runs both; the hop between them configures itself.
|
||||||
|
|
||||||
```nix
|
```nix
|
||||||
services.hyperhive.otel = {
|
services.hyperhive.deploy.swarm-otel.upstream = {
|
||||||
endpoint = "https://collector.example.com/otel"; # the upstream
|
endpoint = "https://collector.example.com/otel"; # the upstream
|
||||||
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
|
headersCredential = "/run/secrets/otel-headers"; # only the swarm tier reads it
|
||||||
};
|
};
|
||||||
|
|
@ -33,13 +33,13 @@ services.hyperhive.otel = {
|
||||||
|
|
||||||
Every hive runs its collector and every agent exports to it, with nothing to
|
Every hive runs its collector and every agent exports to it, with nothing to
|
||||||
switch on: there is no per-hive or per-agent opt-in.
|
switch on: there is no per-hive or per-agent opt-in.
|
||||||
`endpoint` is where telemetry ends up after it leaves the swarm — optional,
|
`upstream.endpoint` is where telemetry ends up after it leaves the swarm — optional,
|
||||||
because the swarm's own metrics store (`deploy.victoriametrics`) is a
|
because the swarm's own metrics store (`deploy.victoriametrics`) is a
|
||||||
destination in its own right. With both, telemetry goes to both. See
|
destination in its own right. With both, telemetry goes to both. See
|
||||||
[`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana).
|
[`swarm/services.md`](../swarm/services.md#metrics-victoriametrics--grafana).
|
||||||
|
|
||||||
**Telemetry leaves a hive exactly one way: through the collector that
|
**Telemetry leaves a hive exactly one way: through the collector that
|
||||||
runs on the host.** Agents never talk to `endpoint` themselves —
|
runs on the host.** Agents never talk to `upstream.endpoint` themselves —
|
||||||
they export unauthenticated to a collector address only their own containers
|
they export unauthenticated to a collector address only their own containers
|
||||||
can reach. They dial it as
|
can reach. They dial it as
|
||||||
`http://hive-collector.hive:4318`, a name identical on
|
`http://hive-collector.hive:4318`, a name identical on
|
||||||
|
|
@ -62,7 +62,7 @@ the control plane, so degraded telemetry isn't degraded operation.
|
||||||
|
|
||||||
### Why two tiers
|
### Why two tiers
|
||||||
|
|
||||||
**Agents export only to the hive tier.** Exporting straight to `endpoint` would mean
|
**Agents export only to the hive tier.** Exporting straight to `upstream.endpoint` would mean
|
||||||
every agent needs the credential — and the only place to hand it to an agent
|
every agent needs the credential — and the only place to hand it to an agent
|
||||||
container is somewhere the agent itself can read, its own claude settings
|
container is somewhere the agent itself can read, its own claude settings
|
||||||
among them. `0600` protects a secret from other containers, not from the
|
among them. `0600` protects a secret from other containers, not from the
|
||||||
|
|
@ -159,8 +159,8 @@ belongs to exactly one hive.
|
||||||
Getting the secret wrong shows up as the hive's collector logging 401s from the
|
Getting the secret wrong shows up as the hive's collector logging 401s from the
|
||||||
swarm tier and no metrics appearing for that hive.
|
swarm tier and no metrics appearing for that hive.
|
||||||
|
|
||||||
⚠️ **`endpoint` must be valid for `protocol`.** The upstream exporter follows
|
⚠️ **`upstream.endpoint` must be valid for `upstream.protocol`.** The upstream exporter follows
|
||||||
`otel.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
|
`upstream.protocol` (`grpc` → the gRPC exporter, otherwise OTLP/HTTP), and the gRPC
|
||||||
exporter takes an _address_: `https://host/path` is a legal
|
exporter takes an _address_: `https://host/path` is a legal
|
||||||
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
|
`OTEL_EXPORTER_OTLP_ENDPOINT` for HTTP but fails as gRPC with _"missing port in
|
||||||
address"_. nix validates the collector's config at build time, so a mismatch is
|
address"_. nix validates the collector's config at build time, so a mismatch is
|
||||||
|
|
@ -175,7 +175,8 @@ To let agents reach some other host-local service of your own →
|
||||||
|
|
||||||
## Options reference
|
## Options reference
|
||||||
|
|
||||||
The nix module (`nix/host-modules/otel.nix`) generates every
|
The nix modules (`nix/host-modules/otel.nix`, and `nix/host-modules/swarm-otel.nix`
|
||||||
|
for the `services.hyperhive.deploy.swarm-otel.upstream.*` export) generate every
|
||||||
`services.hyperhive.otel.*` option's full type, default, description and
|
`services.hyperhive.otel.*` option's full type, default, description and
|
||||||
example into the [options reference](https://hyperhive.darkest.space/options/)
|
example into the [options reference](https://hyperhive.darkest.space/options/)
|
||||||
(host options — `nix build .#docs-host` for a local render). The same module
|
(host options — `nix build .#docs-host` for a local render). The same module
|
||||||
|
|
|
||||||
|
|
@ -313,7 +313,9 @@ read side stays open to any authenticated caller.
|
||||||
|
|
||||||
The swarm's collector receives from every hive's own collector and is the
|
The swarm's collector receives from every hive's own collector and is the
|
||||||
only process that decides where telemetry goes: it always writes the stores
|
only process that decides where telemetry goes: it always writes the stores
|
||||||
above, and exports to `otel.endpoint` as well when that's set. It holds
|
above, and exports to `deploy.swarm-otel.upstream.endpoint` as well when that's set. Which signals
|
||||||
|
go there is `deploy.swarm-otel.upstream.signals` (default `[ "metrics" "logs" ]`); the stores
|
||||||
|
always get both. It holds
|
||||||
the upstream credential, which is why no hive and no agent needs one.
|
the upstream credential, which is why no hive and no agent needs one.
|
||||||
|
|
||||||
It runs in a `swarm-otel` container. Its `swarm.otel.port` defaults to
|
It runs in a `swarm-otel` container. Its `swarm.otel.port` defaults to
|
||||||
|
|
|
||||||
|
|
@ -145,7 +145,8 @@ in
|
||||||
#
|
#
|
||||||
# ⚠️ `SSL_CERT_FILE` REPLACES the trust store, and unlike the other
|
# ⚠️ `SSL_CERT_FILE` REPLACES the trust store, and unlike the other
|
||||||
# consumers of this helper, hive-c0re already makes a *public*-TLS call:
|
# consumers of this helper, hive-c0re already makes a *public*-TLS call:
|
||||||
# the OTEL exporter reaches whatever `otel.endpoint` names, typically a
|
# the OTEL exporter reaches whatever
|
||||||
|
# `deploy.swarm-otel.upstream.endpoint` names, typically a
|
||||||
# normal internet host. The bundle is system CAs + the store's root
|
# normal internet host. The bundle is system CAs + the store's root
|
||||||
# precisely so that path keeps working — narrowing it to the store's root
|
# precisely so that path keeps working — narrowing it to the store's root
|
||||||
# alone would fix a case nobody hits yet and break one that runs today.
|
# alone would fix a case nobody hits yet and break one that runs today.
|
||||||
|
|
|
||||||
|
|
@ -8,8 +8,9 @@
|
||||||
# collector (./swarm-otel.nix). It holds no credential and picks no
|
# collector (./swarm-otel.nix). It holds no credential and picks no
|
||||||
# destination: an agent's samples cross a hive boundary exactly once, and
|
# destination: an agent's samples cross a hive boundary exactly once, and
|
||||||
# what happens after that is the swarm's decision, not a hive's. The
|
# what happens after that is the swarm's decision, not a hive's. The
|
||||||
# upstream options declared below describe that far end and are read one
|
# upstream export is a swarm-collector deploy option,
|
||||||
# tier up — they stay here because they mean what they have always meant.
|
# `services.hyperhive.deploy.swarm-otel.upstream` (./swarm-otel.nix); the
|
||||||
|
# old `otel.*` paths forward to it below.
|
||||||
{
|
{
|
||||||
pkgs,
|
pkgs,
|
||||||
lib,
|
lib,
|
||||||
|
|
@ -56,6 +57,22 @@ in
|
||||||
hostUnit = true;
|
hostUnit = true;
|
||||||
enable = runsHive;
|
enable = runsHive;
|
||||||
})
|
})
|
||||||
|
(lib.mkRenamedOptionModule
|
||||||
|
[ "services" "hyperhive" "otel" "endpoint" ]
|
||||||
|
[ "services" "hyperhive" "deploy" "swarm-otel" "upstream" "endpoint" ]
|
||||||
|
)
|
||||||
|
(lib.mkRenamedOptionModule
|
||||||
|
[ "services" "hyperhive" "otel" "protocol" ]
|
||||||
|
[ "services" "hyperhive" "deploy" "swarm-otel" "upstream" "protocol" ]
|
||||||
|
)
|
||||||
|
(lib.mkRenamedOptionModule
|
||||||
|
[ "services" "hyperhive" "otel" "headersCredential" ]
|
||||||
|
[ "services" "hyperhive" "deploy" "swarm-otel" "upstream" "headersCredential" ]
|
||||||
|
)
|
||||||
|
(lib.mkRenamedOptionModule
|
||||||
|
[ "services" "hyperhive" "otel" "collector" "upstreamHeaderName" ]
|
||||||
|
[ "services" "hyperhive" "deploy" "swarm-otel" "upstream" "headerName" ]
|
||||||
|
)
|
||||||
(lib.mkRemovedOptionModule [ "services" "hyperhive" "otel" "enable" ] ''
|
(lib.mkRemovedOptionModule [ "services" "hyperhive" "otel" "enable" ] ''
|
||||||
Every hive runs its own telemetry collector, and every agent exports to
|
Every hive runs its own telemetry collector, and every agent exports to
|
||||||
it by default, so there is no switch. Remove this definition.
|
it by default, so there is no switch. Remove this definition.
|
||||||
|
|
@ -67,89 +84,6 @@ in
|
||||||
];
|
];
|
||||||
|
|
||||||
options.services.hyperhive.otel = {
|
options.services.hyperhive.otel = {
|
||||||
endpoint = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "";
|
|
||||||
example = "https://collector.example.com/otel";
|
|
||||||
description = ''
|
|
||||||
Upstream OTLP endpoint: where telemetry ultimately goes, after it
|
|
||||||
has left the swarm.
|
|
||||||
|
|
||||||
Read by the swarm's collector
|
|
||||||
({option}`services.hyperhive.deploy.swarm-otel.enable`), which is the only
|
|
||||||
tier that holds the upstream credential. An agent is handed the
|
|
||||||
*first* hop instead — this hive's own collector — so this value is
|
|
||||||
never given to a container.
|
|
||||||
|
|
||||||
Optional. Leave it empty and the swarm's own metrics store
|
|
||||||
({option}`services.hyperhive.deploy.victoriametrics.enable`) is the
|
|
||||||
destination; that is a complete deployment, not a degraded one.
|
|
||||||
Set both and telemetry goes to both.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
protocol = lib.mkOption {
|
|
||||||
type = lib.types.enum [
|
|
||||||
"http/protobuf"
|
|
||||||
"http/json"
|
|
||||||
"grpc"
|
|
||||||
];
|
|
||||||
default = "http/protobuf";
|
|
||||||
description = ''
|
|
||||||
OTLP wire protocol for the **upstream** link, honoured by the
|
|
||||||
swarm collector's exporter.
|
|
||||||
|
|
||||||
Not what agents speak: their first hop is this hive's collector,
|
|
||||||
whose OTLP/HTTP receiver takes protobuf whatever the upstream
|
|
||||||
wants (see `hive-c0re/environment.nix`).
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
headersCredential = lib.mkOption {
|
|
||||||
# `str`, not `path`: a `path`-typed relative literal is hash-copied
|
|
||||||
# into the world-readable nix store at eval time, defeating the
|
|
||||||
# point. Keep it a string + require an absolute runtime path so the
|
|
||||||
# secret is only ever read from disk by systemd at start.
|
|
||||||
type = lib.types.nullOr lib.types.str;
|
|
||||||
default = null;
|
|
||||||
example = "/run/secrets/otel-headers";
|
|
||||||
description = ''
|
|
||||||
Absolute path to an operator-provided secret file holding the
|
|
||||||
upstream auth header as `NAME=value` (e.g.
|
|
||||||
`Authorization=Bearer <token>`).
|
|
||||||
|
|
||||||
**Only the swarm's collector reads this** — the one tier that
|
|
||||||
talks to the upstream. It arrives as an `EnvironmentFile`, so the
|
|
||||||
value is never read by nix, never copied into the store or the
|
|
||||||
generated config, and never passed in argv; and it reaches
|
|
||||||
neither an agent container nor this hive's own collector, which
|
|
||||||
is the point of the tiers existing. Must be absolute.
|
|
||||||
|
|
||||||
Leave null if the upstream needs no auth header; the collector
|
|
||||||
then sends none rather than an empty one.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
collector.upstreamHeaderName = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "Authorization";
|
|
||||||
description = ''
|
|
||||||
Name of the HTTP header the collector sends upstream, whose
|
|
||||||
*value* comes from `headersCredential`.
|
|
||||||
|
|
||||||
The name is here and the value is not, and that split is forced
|
|
||||||
rather than chosen: the collector models exporter headers as a
|
|
||||||
static map, so rendering them means nix reading the value — the
|
|
||||||
one thing `headersCredential` being a path exists to prevent.
|
|
||||||
A name is public, a value is not.
|
|
||||||
|
|
||||||
⇒ exactly one header is expressible this way. A credential
|
|
||||||
carrying several (`a=1,b=2`) would be read as a single value,
|
|
||||||
which is why the shape is a named header rather than an opaque
|
|
||||||
blob: a second header has to be *declared*, not smuggled.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
collector.enable = lib.mkOption {
|
collector.enable = lib.mkOption {
|
||||||
type = lib.types.bool;
|
type = lib.types.bool;
|
||||||
readOnly = true;
|
readOnly = true;
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,7 @@
|
||||||
let
|
let
|
||||||
cfg = config.services.hyperhive.swarm.otel;
|
cfg = config.services.hyperhive.swarm.otel;
|
||||||
deployCfg = config.services.hyperhive.deploy;
|
deployCfg = config.services.hyperhive.deploy;
|
||||||
otelCfg = config.services.hyperhive.otel;
|
upstream = deployCfg.swarm-otel.upstream;
|
||||||
vmCfg = config.services.hyperhive.swarm.victoriametrics;
|
vmCfg = config.services.hyperhive.swarm.victoriametrics;
|
||||||
vlCfg = config.services.hyperhive.swarm.victorialogs;
|
vlCfg = config.services.hyperhive.swarm.victorialogs;
|
||||||
hyperhiveCfg = config.services.hyperhive;
|
hyperhiveCfg = config.services.hyperhive;
|
||||||
|
|
@ -273,10 +273,17 @@ let
|
||||||
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
|
storeSecretPath = "secret/swarm/services/${cfg.clientId}/oidc/client";
|
||||||
|
|
||||||
# The operator-configured upstream, named once: the same exporter carries
|
# The operator-configured upstream, named once: the same exporter carries
|
||||||
# every signal, so metrics and logs both reach it without a second
|
# every signal listed in `upstream.signals`, so metrics and logs both reach
|
||||||
# definition.
|
# it without a second definition.
|
||||||
upstreamExporterName = if otelCfg.protocol == "grpc" then "otlp" else "otlphttp";
|
upstreamExporterName = if upstream.protocol == "grpc" then "otlp" else "otlphttp";
|
||||||
upstreamExporters = lib.optional (otelCfg.endpoint != "") upstreamExporterName;
|
upstreamFor =
|
||||||
|
signal:
|
||||||
|
lib.optional (
|
||||||
|
upstream.endpoint != "" && builtins.elem signal upstream.signals
|
||||||
|
) upstreamExporterName;
|
||||||
|
# The collector refuses an exporter that no pipeline uses, so the exporter
|
||||||
|
# is only defined when at least one signal goes to it.
|
||||||
|
haveUpstream = upstreamFor "metrics" ++ upstreamFor "logs" != [ ];
|
||||||
|
|
||||||
# One list, read by every pipeline: the per-hive pipelines fan out to
|
# One list, read by every pipeline: the per-hive pipelines fan out to
|
||||||
# exactly the same destinations as the single pipeline they replace.
|
# exactly the same destinations as the single pipeline they replace.
|
||||||
|
|
@ -285,10 +292,10 @@ let
|
||||||
# ⚠️ Unconditional: `deploy.victoriametrics.enable` says "this host RUNS the
|
# ⚠️ Unconditional: `deploy.victoriametrics.enable` says "this host RUNS the
|
||||||
# store", and a swarm has one either way. Gating on it left a collector
|
# store", and a swarm has one either way. Gating on it left a collector
|
||||||
# elsewhere with no exporter at all, dropping everything silently.
|
# elsewhere with no exporter at all, dropping everything silently.
|
||||||
exporterNames = upstreamExporters ++ [ "otlphttp/victoriametrics" ];
|
exporterNames = upstreamFor "metrics" ++ [ "otlphttp/victoriametrics" ];
|
||||||
|
|
||||||
# Same fan-out for logs, unconditional for the same reason.
|
# Same fan-out for logs, unconditional for the same reason.
|
||||||
logExporterNames = upstreamExporters ++ [ "otlphttp/victorialogs" ];
|
logExporterNames = upstreamFor "logs" ++ [ "otlphttp/victorialogs" ];
|
||||||
|
|
||||||
# An empty exporter list is not a quiet no-op — the collector rejects it.
|
# An empty exporter list is not a quiet no-op — the collector rejects it.
|
||||||
# Now always satisfied, as a consequence of the log store always existing.
|
# Now always satisfied, as a consequence of the log store always existing.
|
||||||
|
|
@ -303,11 +310,126 @@ let
|
||||||
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
storeRefreshTimer = import ./lib/store-refresh-timer.nix { };
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
# What the collector IS to the swarm — the client it is registered as, where
|
# What the collector IS to the swarm — the client it is registered as — is
|
||||||
# it exports — is `swarm.otel` in ./swarm-otel-service.nix. The secret is a
|
# `swarm.otel` in ./swarm-otel-service.nix. The secret is a path on the
|
||||||
# path on the machine that runs it, so it hangs off the deployment. `enable`
|
# machine that runs it, and the external export (`upstream`) is the
|
||||||
# already lives in ./deploy.nix, which also carries the rename.
|
# operator's choice for this deployment, so both hang off the deployment.
|
||||||
|
# `enable` already lives in ./deploy.nix, which also carries the rename.
|
||||||
options.services.hyperhive.deploy.swarm-otel = {
|
options.services.hyperhive.deploy.swarm-otel = {
|
||||||
|
upstream = {
|
||||||
|
endpoint = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "";
|
||||||
|
example = "https://collector.example.com/otel";
|
||||||
|
description = ''
|
||||||
|
Upstream OTLP endpoint: where telemetry ultimately goes, after it
|
||||||
|
has left the swarm.
|
||||||
|
|
||||||
|
Read by the swarm's collector
|
||||||
|
({option}`services.hyperhive.deploy.swarm-otel.enable`), which is the only
|
||||||
|
tier that holds the upstream credential. An agent is handed the
|
||||||
|
*first* hop instead — this hive's own collector — so this value is
|
||||||
|
never given to a container.
|
||||||
|
|
||||||
|
Optional. Leave it empty and the swarm's own metrics store
|
||||||
|
({option}`services.hyperhive.deploy.victoriametrics.enable`) is the
|
||||||
|
destination; that is a complete deployment, not a degraded one.
|
||||||
|
Set both and telemetry goes to both.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
signals = lib.mkOption {
|
||||||
|
type = lib.types.listOf (
|
||||||
|
lib.types.enum [
|
||||||
|
"metrics"
|
||||||
|
"logs"
|
||||||
|
]
|
||||||
|
);
|
||||||
|
default = [
|
||||||
|
"metrics"
|
||||||
|
"logs"
|
||||||
|
];
|
||||||
|
example = [ "metrics" ];
|
||||||
|
description = ''
|
||||||
|
Which signals the swarm's collector sends to
|
||||||
|
{option}`services.hyperhive.deploy.swarm-otel.upstream.endpoint`. The
|
||||||
|
swarm's own metrics and log stores always receive everything,
|
||||||
|
whatever this lists.
|
||||||
|
|
||||||
|
Use it when the upstream does not accept a signal, for example one
|
||||||
|
that answers 404 on `/v1/logs`: list only `"metrics"` and the logs
|
||||||
|
pipelines stop exporting upstream. An empty list defines no upstream
|
||||||
|
exporter at all. No pipeline exports traces, so they have no entry.
|
||||||
|
|
||||||
|
Has no effect while
|
||||||
|
{option}`services.hyperhive.deploy.swarm-otel.upstream.endpoint` is empty.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
protocol = lib.mkOption {
|
||||||
|
type = lib.types.enum [
|
||||||
|
"http/protobuf"
|
||||||
|
"http/json"
|
||||||
|
"grpc"
|
||||||
|
];
|
||||||
|
default = "http/protobuf";
|
||||||
|
description = ''
|
||||||
|
OTLP wire protocol for the **upstream** link, honoured by the
|
||||||
|
swarm collector's exporter.
|
||||||
|
|
||||||
|
Not what agents speak: their first hop is this hive's collector,
|
||||||
|
whose OTLP/HTTP receiver takes protobuf whatever the upstream
|
||||||
|
wants (see `hive-c0re/environment.nix`).
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
headersCredential = lib.mkOption {
|
||||||
|
# `str`, not `path`: a `path`-typed relative literal is hash-copied
|
||||||
|
# into the world-readable nix store at eval time, defeating the
|
||||||
|
# point. Keep it a string + require an absolute runtime path so the
|
||||||
|
# secret is only ever read from disk by systemd at start.
|
||||||
|
type = lib.types.nullOr lib.types.str;
|
||||||
|
default = null;
|
||||||
|
example = "/run/secrets/otel-headers";
|
||||||
|
description = ''
|
||||||
|
Absolute path to an operator-provided secret file holding the
|
||||||
|
upstream auth header as `NAME=value` (e.g.
|
||||||
|
`Authorization=Bearer <token>`).
|
||||||
|
|
||||||
|
**Only the swarm's collector reads this** — the one tier that
|
||||||
|
talks to the upstream. It arrives as an `EnvironmentFile`, so the
|
||||||
|
value is never read by nix, never copied into the store or the
|
||||||
|
generated config, and never passed in argv; and it reaches
|
||||||
|
neither an agent container nor this hive's own collector, which
|
||||||
|
is the point of the tiers existing. Must be absolute.
|
||||||
|
|
||||||
|
Leave null if the upstream needs no auth header; the collector
|
||||||
|
then sends none rather than an empty one.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
headerName = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "Authorization";
|
||||||
|
description = ''
|
||||||
|
Name of the HTTP header the collector sends upstream, whose
|
||||||
|
*value* comes from
|
||||||
|
{option}`services.hyperhive.deploy.swarm-otel.upstream.headersCredential`.
|
||||||
|
|
||||||
|
The name is here and the value is not, and that split is forced
|
||||||
|
rather than chosen: the collector models exporter headers as a
|
||||||
|
static map, so rendering them means nix reading the value — the
|
||||||
|
one thing `headersCredential` being a path exists to prevent.
|
||||||
|
A name is public, a value is not.
|
||||||
|
|
||||||
|
⇒ exactly one header is expressible this way. A credential
|
||||||
|
carrying several (`a=1,b=2`) would be read as a single value,
|
||||||
|
which is why the shape is a named header rather than an opaque
|
||||||
|
blob: a second header has to be *declared*, not smuggled.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
clientSecretFile = lib.mkOption {
|
clientSecretFile = lib.mkOption {
|
||||||
type = lib.types.nullOr lib.types.str;
|
type = lib.types.nullOr lib.types.str;
|
||||||
default = null;
|
default = null;
|
||||||
|
|
@ -809,9 +931,9 @@ in
|
||||||
isReadOnly = true;
|
isReadOnly = true;
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
// lib.optionalAttrs (otelCfg.headersCredential != null) {
|
// lib.optionalAttrs (upstream.headersCredential != null) {
|
||||||
${otelCfg.headersCredential} = {
|
${upstream.headersCredential} = {
|
||||||
hostPath = otelCfg.headersCredential;
|
hostPath = upstream.headersCredential;
|
||||||
isReadOnly = true;
|
isReadOnly = true;
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
@ -846,7 +968,7 @@ in
|
||||||
{ ... }:
|
{ ... }:
|
||||||
{
|
{
|
||||||
# This tier is the one that resolves an operator-configured
|
# This tier is the one that resolves an operator-configured
|
||||||
# hostname: `otel.endpoint` is an external URL, and reaching it
|
# hostname: `upstream.endpoint` is an external URL, and reaching it
|
||||||
# is the entire reason this container holds a credential. The
|
# is the entire reason this container holds a credential. The
|
||||||
# `/etc/resolv.conf` nixos-containers copies in is a snapshot
|
# `/etc/resolv.conf` nixos-containers copies in is a snapshot
|
||||||
# taken once at boot, so without this the upstream export
|
# taken once at boot, so without this the upstream export
|
||||||
|
|
@ -1068,17 +1190,17 @@ in
|
||||||
auth.authenticator = pushAuthenticator "victoriametrics";
|
auth.authenticator = pushAuthenticator "victoriametrics";
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
// lib.optionalAttrs (otelCfg.endpoint != "") {
|
// lib.optionalAttrs haveUpstream {
|
||||||
${if otelCfg.protocol == "grpc" then "otlp" else "otlphttp"} = {
|
${if upstream.protocol == "grpc" then "otlp" else "otlphttp"} = {
|
||||||
endpoint = otelCfg.endpoint;
|
endpoint = upstream.endpoint;
|
||||||
}
|
}
|
||||||
// lib.optionalAttrs (otelCfg.headersCredential != null) {
|
// lib.optionalAttrs (upstream.headersCredential != null) {
|
||||||
# Interpolated by the collector from its environment at
|
# Interpolated by the collector from its environment at
|
||||||
# runtime, never by nix: `EnvironmentFile` below is what
|
# runtime, never by nix: `EnvironmentFile` below is what
|
||||||
# puts it there, so the value is not read into the store.
|
# puts it there, so the value is not read into the store.
|
||||||
headers.${otelCfg.collector.upstreamHeaderName} = "\${env:${otelCfg.collector.upstreamHeaderName}}";
|
headers.${upstream.headerName} = "\${env:${upstream.headerName}}";
|
||||||
}
|
}
|
||||||
// lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; };
|
// lib.optionalAttrs (upstream.protocol == "http/json") { encoding = "json"; };
|
||||||
}
|
}
|
||||||
// {
|
// {
|
||||||
# `logs_endpoint`, NOT `endpoint`: the latter is a base the
|
# `logs_endpoint`, NOT `endpoint`: the latter is a base the
|
||||||
|
|
@ -1505,8 +1627,8 @@ in
|
||||||
# EnvironmentFile format — so the secret reaches the process as
|
# EnvironmentFile format — so the secret reaches the process as
|
||||||
# an environment variable without being read by nix, written to
|
# an environment variable without being read by nix, written to
|
||||||
# the store, or passed in argv.
|
# the store, or passed in argv.
|
||||||
(lib.optionalAttrs (otelCfg.headersCredential != null) {
|
(lib.optionalAttrs (upstream.headersCredential != null) {
|
||||||
serviceConfig.EnvironmentFile = otelCfg.headersCredential;
|
serviceConfig.EnvironmentFile = upstream.headersCredential;
|
||||||
})
|
})
|
||||||
|
|
||||||
# The other credential, and the other direction: the one above
|
# The other credential, and the other direction: the one above
|
||||||
|
|
|
||||||
|
|
@ -28,20 +28,45 @@ let
|
||||||
# shape from docs/swarm/services.md, and the one the old per-host gates made
|
# shape from docs/swarm/services.md, and the one the old per-host gates made
|
||||||
# inexpressible. It is the whole point of the cases below that this hive is
|
# inexpressible. It is the whole point of the cases below that this hive is
|
||||||
# not a degenerate configuration but a supported one.
|
# not a degenerate configuration but a supported one.
|
||||||
otelNoStores = hive {
|
mkOtelNoStores =
|
||||||
|
upstreamSignals:
|
||||||
|
hive {
|
||||||
|
deploy.swarm-otel.enable = true;
|
||||||
|
deploy.authelia.enable = true;
|
||||||
|
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||||
|
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||||
|
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
|
||||||
|
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
|
||||||
|
deploy.victoriametrics.enable = false;
|
||||||
|
deploy.victorialogs.enable = false;
|
||||||
|
# An upstream makes `logExporterNames` longer than the store alone, which
|
||||||
|
# is what the self-journal case below needs to tell the two apart.
|
||||||
|
deploy.swarm-otel.upstream.endpoint = "https://upstream.example/otel";
|
||||||
|
deploy.swarm-otel.upstream.signals = upstreamSignals;
|
||||||
|
};
|
||||||
|
otelNoStores = mkOtelNoStores [
|
||||||
|
"metrics"
|
||||||
|
"logs"
|
||||||
|
];
|
||||||
|
|
||||||
|
# A host config still on the `otel.*` upstream paths, every shipped one.
|
||||||
|
otelOldUpstreamPaths = hive {
|
||||||
deploy.swarm-otel.enable = true;
|
deploy.swarm-otel.enable = true;
|
||||||
deploy.authelia.enable = true;
|
deploy.authelia.enable = true;
|
||||||
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
deploy.bao.clientCertFile = "/etc/pki/bao-client.pem";
|
||||||
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
deploy.bao.clientKeyFile = "/etc/pki/bao-client-key.pem";
|
||||||
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
|
deploy.bao.otelOidcClientCertFile = "/etc/pki/bao-otel-oidc.pem";
|
||||||
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
|
deploy.bao.otelOidcClientKeyFile = "/etc/pki/bao-otel-oidc-key.pem";
|
||||||
deploy.victoriametrics.enable = false;
|
otel.endpoint = "https://old.example/otel";
|
||||||
deploy.victorialogs.enable = false;
|
otel.protocol = "http/json";
|
||||||
# An upstream makes `logExporterNames` longer than the store alone, which
|
otel.headersCredential = "/run/secrets/old-otel-headers";
|
||||||
# is what the self-journal case below needs to tell the two apart.
|
otel.collector.upstreamHeaderName = "X-Old-Token";
|
||||||
otel.endpoint = "https://upstream.example/otel";
|
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# The same collector with the upstream limited to one signal, and with none.
|
||||||
|
otelUpstreamMetricsOnly = mkOtelNoStores [ "metrics" ];
|
||||||
|
otelUpstreamNone = mkOtelNoStores [ ];
|
||||||
|
|
||||||
# The collector beside authelia, reading its own OIDC secret out of the
|
# The collector beside authelia, reading its own OIDC secret out of the
|
||||||
# store like every other collector — the cert pair here is not scenery, it
|
# store like every other collector — the cert pair here is not scenery, it
|
||||||
# is the arm that would catch the deleted co-located copy unit coming back.
|
# is the arm that would catch the deleted co-located copy unit coming back.
|
||||||
|
|
@ -313,6 +338,77 @@ let
|
||||||
&& lib.hasInfix "Exporting failed" (lib.head f.logs.log_record)
|
&& lib.hasInfix "Exporting failed" (lib.head f.logs.log_record)
|
||||||
&& lib.hasInfix "otlphttp/victorialogs" (lib.head f.logs.log_record);
|
&& lib.hasInfix "otlphttp/victorialogs" (lib.head f.logs.log_record);
|
||||||
}
|
}
|
||||||
|
{
|
||||||
|
# Presence control: the default sends both signals to the upstream, so
|
||||||
|
# the negative arms below cannot pass on a misspelled exporter name.
|
||||||
|
name = "the upstream receives metrics and logs by default";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = otelSettings otelNoStores;
|
||||||
|
ps = s.service.pipelines;
|
||||||
|
names = pre: lib.filter (lib.hasPrefix pre) (lib.attrNames ps);
|
||||||
|
in
|
||||||
|
s.exporters ? otlphttp
|
||||||
|
&& names "metrics/" != [ ]
|
||||||
|
&& names "logs/" != [ ]
|
||||||
|
&& lib.all (n: builtins.elem "otlphttp" ps.${n}.exporters) (names "metrics/")
|
||||||
|
&& lib.all (n: n == "logs/self" || builtins.elem "otlphttp" ps.${n}.exporters) (names "logs/");
|
||||||
|
}
|
||||||
|
{
|
||||||
|
name = "upstreamSignals limits which pipelines export upstream, and the stores keep everything";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = otelSettings otelUpstreamMetricsOnly;
|
||||||
|
ps = s.service.pipelines;
|
||||||
|
names = pre: lib.filter (lib.hasPrefix pre) (lib.attrNames ps);
|
||||||
|
in
|
||||||
|
s.exporters ? otlphttp
|
||||||
|
&& names "metrics/" != [ ]
|
||||||
|
&& names "logs/" != [ ]
|
||||||
|
&& lib.all (n: builtins.elem "otlphttp" ps.${n}.exporters) (names "metrics/")
|
||||||
|
&& lib.all (n: builtins.elem "otlphttp/victoriametrics" ps.${n}.exporters) (names "metrics/")
|
||||||
|
&& !(lib.any (n: builtins.elem "otlphttp" ps.${n}.exporters) (names "logs/"))
|
||||||
|
&& lib.all (n: builtins.elem "otlphttp/victorialogs" ps.${n}.exporters) (names "logs/");
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# The control is `otelNoStores`: it sets the new paths and warns about
|
||||||
|
# none of the old ones, so the warnings below come from the renames.
|
||||||
|
name = "the old otel.* upstream paths set deploy.swarm-otel.upstream and warn";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
u = otelOldUpstreamPaths.services.hyperhive.deploy.swarm-otel.upstream;
|
||||||
|
warnsAbout = m: path: lib.any (lib.hasInfix "services.hyperhive.${path}") m.warnings;
|
||||||
|
in
|
||||||
|
u.endpoint == "https://old.example/otel"
|
||||||
|
&& u.protocol == "http/json"
|
||||||
|
&& u.headersCredential == "/run/secrets/old-otel-headers"
|
||||||
|
&& u.headerName == "X-Old-Token"
|
||||||
|
&& warnsAbout otelOldUpstreamPaths "otel.endpoint"
|
||||||
|
&& warnsAbout otelOldUpstreamPaths "otel.protocol"
|
||||||
|
&& warnsAbout otelOldUpstreamPaths "otel.headersCredential"
|
||||||
|
&& warnsAbout otelOldUpstreamPaths "otel.collector.upstreamHeaderName"
|
||||||
|
&& !(warnsAbout otelNoStores "otel.endpoint")
|
||||||
|
&&
|
||||||
|
otelNoStores.services.hyperhive.deploy.swarm-otel.upstream.endpoint
|
||||||
|
== "https://upstream.example/otel"
|
||||||
|
&& (otelSettings otelOldUpstreamPaths).exporters ? otlphttp;
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# The collector rejects an exporter no pipeline uses, so with nothing
|
||||||
|
# going upstream the exporter must not be defined.
|
||||||
|
name = "an empty upstream.signals defines no upstream exporter and keeps the stores";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
s = otelSettings otelUpstreamNone;
|
||||||
|
ps = lib.attrValues s.service.pipelines;
|
||||||
|
in
|
||||||
|
!(s.exporters ? otlphttp)
|
||||||
|
&& !(lib.any (p: builtins.elem "otlphttp" p.exporters) ps)
|
||||||
|
&& s.exporters ? "otlphttp/victoriametrics"
|
||||||
|
&& s.exporters ? "otlphttp/victorialogs"
|
||||||
|
&& lib.any (p: builtins.elem "otlphttp/victorialogs" p.exporters) ps
|
||||||
|
&& lib.any (p: builtins.elem "otlphttp/victoriametrics" p.exporters) ps;
|
||||||
|
}
|
||||||
];
|
];
|
||||||
in
|
in
|
||||||
runGroup "swarm-otel-core" cases
|
runGroup "swarm-otel-core" cases
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue