2e06957b32
The scrape config names a client_secret_file; this is what puts a file there. A host oneshot copies authelia's minted secret between the two container trees — a copy and not a bind mount, because the secret does not exist until authelia's first boot and nixos-container refuses to start on a missing bind source, which on a fresh swarm is a permanent stall presenting as broken metrics. The collector runs under DynamicUser and the prometheus receiver opens client_secret_file itself, as that user, so there is no uid to hand the file to. LoadCredential reads it as root before the sandbox exists and re-exposes it under a path that does not depend on which uid the unit got; the scrape config points there. Both spellings derive from one binding, since a mismatch is a file the collector cannot open and nothing but a runtime 401 would say so.
959 lines
47 KiB
Nix
959 lines
47 KiB
Nix
# The swarm's telemetry collector: one per swarm, in a `swarm-otel`
|
||
# nixos-container beside the swarm's other shared services.
|
||
#
|
||
# Two tiers, and they are separate on purpose:
|
||
#
|
||
# - `otel.nix` is the **hive** tier. It receives from this hive's agents
|
||
# on the bridge and forwards, and it holds no upstream credential.
|
||
# - this is the **swarm** tier. It is the only holder of the upstream
|
||
# credential, the only writer to the swarm's metrics store, and the
|
||
# place that will stamp `hive=` from the authenticated connection
|
||
# rather than from anything a sender can choose.
|
||
#
|
||
# On a host that runs both, both processes run. They are not collapsed:
|
||
# all-local is a statement about *where* processes run, not about what
|
||
# shape the deployment has, and a local tier boundary that disappears is
|
||
# one the local deployment stops testing. `hive=` attribution is the
|
||
# property that would differ, and the ingest auth that makes it
|
||
# unforgeable is built on this boundary existing.
|
||
#
|
||
# A container rather than a second host unit, for the same reason every
|
||
# sibling swarm service is one — and because `services.opentelemetry-collector`
|
||
# is a singleton NixOS option, already spoken for on the host by the hive
|
||
# tier. A container gets its own evaluation and therefore its own collector.
|
||
{
|
||
pkgs,
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
cfg = config.services.hyperhive.swarm.otel;
|
||
swarmCfg = config.services.hyperhive.swarm;
|
||
otelCfg = config.services.hyperhive.otel;
|
||
vmCfg = config.services.hyperhive.swarm.victoriametrics;
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
gatewayCfg = hyperhiveCfg.gateway;
|
||
swarmDomain = hyperhiveCfg.swarm.domain;
|
||
|
||
# Total on a null swarm domain for the same reason every sibling module is:
|
||
# the required-domain assertion in hive-network.nix should be what an
|
||
# operator sees, not a coercion error from here.
|
||
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
|
||
|
||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||
|
||
# The collector names its components `<kind>/<owner>`, where `<owner>` is a
|
||
# hive name for the per-hive pipelines and this literal for the swarm tier's
|
||
# own. The two share one namespace and are merged with `//`, so a hive named
|
||
# this would silently REPLACE the swarm tier's parts — and lose its own
|
||
# pipeline in the process, while its receiver keeps accepting pushes.
|
||
#
|
||
# Bound once and interpolated at every swarm-tier use below, so the assertion
|
||
# that reserves it is checking the same string the config emits. A literal
|
||
# repeated at each site would let the guard and the config drift apart, which
|
||
# is the failure this guard exists to prevent.
|
||
swarmTierName = "swarm";
|
||
# Every `<owner>` this module claims for itself. One entry today; a second
|
||
# swarm-tier pipeline would be added here and inherit the check for free.
|
||
reservedOwners = [ swarmTierName ];
|
||
|
||
# A published target is declared as ONE url, because that url is also the
|
||
# audience its token is minted for — but prometheus wants the same fact in
|
||
# three fields. Split it here rather than asking a service to state it
|
||
# twice: two spellings of one address is a mismatch waiting to happen, and
|
||
# the failure is a valid token refused at the target.
|
||
#
|
||
# `null` when the shape is wrong, which the assertion below reports by name.
|
||
# ⚠️ The pattern requires `https`. A published endpoint reached over plain
|
||
# http would ship a bearer token in clear text, and that is worth an eval
|
||
# error rather than a warning nobody reads.
|
||
parsePublished = url: builtins.match "https://([^/]+)(/.*)" url;
|
||
|
||
# The client secret takes three names, and the reason is `DynamicUser`.
|
||
#
|
||
# Upstream's collector unit runs with `DynamicUser = true`, and the
|
||
# prometheus receiver opens `client_secret_file` ITSELF, at runtime, as that
|
||
# user — so there is no stable uid to hand a file to, and the root-owned
|
||
# 0400 shape `hive-matrix-oidc-secret` delivers to would be unreadable.
|
||
# (Matrix gets away with it because `LoadCredential` reads the file as root
|
||
# before the sandbox exists, and tuwunel never opens that path itself.)
|
||
#
|
||
# `LoadCredential` solves both halves: systemd reads the file as root and
|
||
# re-exposes it to the dynamic user under a path that does not depend on
|
||
# which uid it turned out to be.
|
||
#
|
||
# At rest in the container's tree — written by the host oneshot below.
|
||
# Under /var/lib and not /run because the collector may start before the
|
||
# delivery unit on a later boot, and a secret that evaporates on reboot
|
||
# turns a working scrape into an intermittent one.
|
||
collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret";
|
||
collectorCredentialId = "oidc-client-secret";
|
||
# What the scrape config points at. ⚠️ This path and the `LoadCredential`
|
||
# id below are one fact spelled twice by systemd's design — both derive from
|
||
# `collectorCredentialId` so they cannot drift; a mismatch is a file the
|
||
# collector cannot open, discovered at runtime and nowhere else.
|
||
collectorSecretPath = "/run/credentials/opentelemetry-collector.service/${collectorCredentialId}";
|
||
|
||
# `attrNames` is sorted, so this is a function of the hive SET and not of
|
||
# the order anyone wrote it in.
|
||
#
|
||
# These ports are internal and appear in no URL: a hive addresses its own
|
||
# receiver as a PATH on this collector's single gateway name, and nginx —
|
||
# rendered from this same evaluation — is the only thing that ever names
|
||
# the port. That is what makes deriving them safe here and unsafe in the
|
||
# obvious other place: were a hive told a port, inserting a hive would
|
||
# renumber the ones after it and silently move a port a running hive was
|
||
# already sending to.
|
||
hivePorts = lib.listToAttrs (
|
||
lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||
);
|
||
|
||
# The swarm's authelia is reached by its gateway name, whose leaf is
|
||
# issued by the swarm services sub-CA — so this container needs the same
|
||
# runtime CA trust every other consumer of a swarm-service name needs.
|
||
# The CA is generated at runtime and cannot be baked into a derivation,
|
||
# which is why it arrives as a bind mount rather than
|
||
# `security.pki.certificateFiles`.
|
||
caTrust = import ./lib/hive-ca-trust.nix {
|
||
inherit lib;
|
||
tlsCfg = hyperhiveCfg.tls;
|
||
inherit gatewayCfg;
|
||
};
|
||
|
||
# `unknown` rather than omitting the label, copying `agent-modules/otel.nix`
|
||
# deliberately: a producer that cannot name its swarm should say so in the
|
||
# same vocabulary as every other producer, so a query never has to handle
|
||
# both "the label is absent" and "the label says unknown".
|
||
swarmDisplayName = if hyperhiveCfg.swarm.name == null then "unknown" else hyperhiveCfg.swarm.name;
|
||
|
||
# One list, read by every pipeline: the per-hive pipelines fan out to
|
||
# exactly the same destinations as the single pipeline they replace.
|
||
# Written once because "which exporters" is a property of this tier, not
|
||
# of which hive a sample came from.
|
||
exporterNames =
|
||
lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
|
||
++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
|
||
in
|
||
{
|
||
options.services.hyperhive.swarm.otel = {
|
||
enable = lib.mkOption {
|
||
type = lib.types.bool;
|
||
default = false;
|
||
description = ''
|
||
Run the swarm's telemetry collector on this host.
|
||
|
||
Asserted from `swarm.enableRequiredServices` in
|
||
./swarm-required-services.nix, with the metrics pair this
|
||
collector feeds: a swarm has one of these, and it belongs
|
||
wherever the shared services live rather than on every hive.
|
||
|
||
A hive that does not run it still runs its own hive-tier collector
|
||
(`services.hyperhive.otel.enable`) and reaches this one by name, at
|
||
{option}`services.hyperhive.swarm.otel.domain`.
|
||
'';
|
||
};
|
||
|
||
machine = lib.mkOption {
|
||
type = lib.types.str;
|
||
readOnly = true;
|
||
default = "swarm-otel";
|
||
description = ''
|
||
Name of the nixos-container this collector runs in — also the
|
||
`machinectl` name, so other modules may read it rather than
|
||
repeating the literal.
|
||
'';
|
||
};
|
||
|
||
port = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 4319;
|
||
description = ''
|
||
First port of this collector's receiver range. Every hive in
|
||
{option}`services.hyperhive.swarm.hives` gets its **own**
|
||
authenticated receiver — that is what makes the `hive` label
|
||
unforgeable — so the range is one port per hive, starting here, in
|
||
sorted-name order.
|
||
|
||
⚠️ Internal. No client is ever told a port: a hive reaches its own
|
||
receiver as `https://''${domain}/<hive>`, and the gateway routes on
|
||
that path. So adding a hive, which renumbers the ones after it, is
|
||
harmless — nginx is rendered from this same evaluation and moves
|
||
with it.
|
||
|
||
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
|
||
hive tier already uses it (`services.hyperhive.otel.collector.port`)
|
||
and every swarm container shares the host's network namespace. Two
|
||
listeners claiming one port on one host is not a build failure —
|
||
it is a runtime coin toss over which one gets it, with nothing in
|
||
any log saying so. The same collision cost a release when grafana
|
||
and the forge both defaulted to 3000. The assertions below check
|
||
the whole derived range against every port this module and the hive
|
||
tier declare, which is as far as a module can see.
|
||
'';
|
||
};
|
||
|
||
telemetryPort = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 8889;
|
||
description = ''
|
||
Port this collector serves its **own** metrics on — queue depth,
|
||
refused and dropped samples, exporter failures. How you find out
|
||
that telemetry is being lost, so it is worth keeping rather than
|
||
switching off.
|
||
|
||
⚠️ **Deliberately not 8889's neighbour 8888**, which is the
|
||
collector's built-in default and therefore what the hive tier
|
||
already binds. Two collectors share a network namespace whenever
|
||
they are co-located, and unlike the OTLP port this one appears
|
||
nowhere in either config — it is a default inside the binary, so
|
||
nothing that compares configured ports can see the clash. The
|
||
second collector to start simply dies with
|
||
`bind: address already in use`.
|
||
'';
|
||
};
|
||
|
||
domain = lib.mkOption {
|
||
type = lib.types.str;
|
||
default = "otel.${domainBase}";
|
||
defaultText = lib.literalExpression ''"otel.''${services.hyperhive.swarm.domain}"'';
|
||
description = ''
|
||
Name the gateway serves this on. A sibling of the swarm's other
|
||
service names, so the swarm-services sub-CA can issue for it — see
|
||
`hive-tls.nix` for why a service name being a sibling rather than a
|
||
child decides which CA may sign it.
|
||
|
||
This is what the **hive** tier's exporter reaches — the hive
|
||
collector is a plain producer against this name exactly like every
|
||
other client of a swarm service, resolved locally by dnsmasq on a
|
||
co-located host and over the real network otherwise. There is no
|
||
separate loopback-vs-remote knob to get wrong: `swarm-nats` is the
|
||
deliberate exception to this pattern (its cross-hive reach is the
|
||
wireguard mesh, not the gateway), everything else in this swarm
|
||
addresses its siblings by name.
|
||
'';
|
||
};
|
||
|
||
scrapeTargets = lib.mkOption {
|
||
type = lib.types.attrsOf lib.types.str;
|
||
default = { };
|
||
example = lib.literalExpression ''{ forgejo = "127.0.0.1:3000"; }'';
|
||
description = ''
|
||
Prometheus exposition endpoints this collector scrapes, as
|
||
`<job name> = "<host>:<port>"`.
|
||
|
||
**A service declares its own entry, from its own module, under its
|
||
own `enable`.** That is what puts the scraper and the target on the
|
||
same host by construction rather than by luck: an entry exists only
|
||
where the service that named it runs. Do not assemble the list here.
|
||
Every swarm service being co-located is a property of the all-local
|
||
deployment, not a guarantee — and that is precisely the case where
|
||
the difference is invisible until a swarm splits across hosts.
|
||
|
||
Samples land in a swarm-level pipeline that stamps `swarm` and
|
||
**never** `hive`: a swarm service does not belong to a hive, and
|
||
`hive` stays a property of which authenticated receiver accepted a
|
||
push, not something a scrape can acquire.
|
||
|
||
Empty by default, in which case no scrape receiver, processor or
|
||
pipeline is emitted at all — an enabled scraper with nothing to
|
||
scrape is the inert configuration this option exists to avoid.
|
||
'';
|
||
};
|
||
|
||
publishedScrapeTargets = lib.mkOption {
|
||
type = lib.types.attrsOf lib.types.str;
|
||
default = { };
|
||
example = lib.literalExpression ''{ forge = "https://forge.example.com/metrics"; }'';
|
||
description = ''
|
||
Prometheus endpoints this collector scrapes **by name, with a
|
||
credential**, as `<job name> = "<url>"`.
|
||
|
||
A service module declares its own entry, from its own module, the
|
||
same way it does for `scrapeTargets` — and the collector's OAuth2
|
||
client derives its permitted audiences from these URLs, so a
|
||
target and the authorisation to reach it are **one declaration**.
|
||
Two lists that must agree would be a drift to maintain, and its
|
||
failure mode is the bad one: a target whose audience was forgotten
|
||
authenticates against nothing and reads as a scrape failure rather
|
||
than a config mistake.
|
||
|
||
⚠️ A **full URL**, not `host:port`. Authelia validates a bearer
|
||
token against the address being requested, so the string here is
|
||
also the audience the token is minted for; a near miss (a trailing
|
||
slash, `http` for `https`) presents as a valid token rejected at
|
||
the target, several layers from its cause.
|
||
|
||
⚠️ Deliberately separate from `scrapeTargets`. An entry there is
|
||
trusted because the scraper and the target share a host — that
|
||
option is loopback-and-unauthenticated by contract. An entry here
|
||
is trusted because it presents a credential. One shape for both
|
||
would leave a reader unable to tell which of those a given target
|
||
relies on.
|
||
'';
|
||
};
|
||
|
||
clientId = lib.mkOption {
|
||
type = lib.types.str;
|
||
readOnly = true;
|
||
default = "swarm-collector";
|
||
description = ''
|
||
OAuth2 client id this collector authenticates as. Published so
|
||
authelia's `access_control` rules can name it without carrying a
|
||
second copy of the string, exactly as
|
||
`services.hyperhive.swarm.authelia.hiveClientPrefix` is published
|
||
for the queue's responder.
|
||
|
||
Two spellings drifting apart is not a build failure: authelia
|
||
refuses a rule naming an unregistered client in its startup
|
||
validator, so the swarm's SSO service fails to *restart* — long
|
||
after the change that caused it evaluated cleanly.
|
||
'';
|
||
};
|
||
};
|
||
|
||
config = lib.mkIf (config.services.hyperhive.enable && cfg.enable) {
|
||
# The gateway name, inside `cfg.enable` — that guard is the load-bearing
|
||
# part. Every hive in a swarm may know this collector exists, but only
|
||
# the host that RUNS it may claim the name; a client hive declaring the
|
||
# vhost would answer for a service it does not have.
|
||
services.hyperhive.gateway.localNames = [ cfg.domain ];
|
||
|
||
# OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
|
||
# every sibling swarm service: TLS terminates here, then plain http to
|
||
# the co-located container over loopback (shared netns, like the store
|
||
# this collector writes to).
|
||
services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
|
||
listen = gatewayCfg.lib.listen;
|
||
extraConfig = gatewayCfg.lib.securityHeaders;
|
||
locations =
|
||
# One name for the whole collector, and the hive is a path under
|
||
# it. The alternative — a vhost per hive — needs a certificate,
|
||
# a DNS name and a `localNames` entry per hive to express the
|
||
# same routing the gateway already does for free.
|
||
#
|
||
# ⚠️ The trailing slash on both sides is load-bearing: it is what
|
||
# strips `/<hive>` before the request reaches the receiver, which
|
||
# serves `/v1/metrics` and knows nothing about hives. Without it
|
||
# the receiver sees `/<hive>/v1/metrics` and answers 404 to a
|
||
# request that authenticated perfectly.
|
||
lib.mapAttrs' (
|
||
h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; }
|
||
) hivePorts
|
||
// {
|
||
# There is no swarm-wide inbox, and a closed door is the honest
|
||
# description of that. Every route into this collector belongs to
|
||
# exactly one hive.
|
||
"/".return = "404";
|
||
};
|
||
};
|
||
|
||
# Turning on the identities this tier authenticates against, which the
|
||
# option exists to allow: its own description names this module as the
|
||
# second consumer, so the queue is not a prerequisite for authenticated
|
||
# telemetry.
|
||
services.hyperhive.swarm.authelia.oidc.hiveIdentities = true;
|
||
|
||
# The collector's own identity, for the other direction: the hive
|
||
# identities above are how this collector authenticates its *callers*,
|
||
# this is how it authenticates *itself* to a service published behind
|
||
# the gateway.
|
||
#
|
||
# Only where authelia is co-located. A client is a row in this host's
|
||
# provider config, so declaring one against a remote provider would
|
||
# render nothing while reading as done; a swarm whose authelia lives
|
||
# elsewhere registers it there.
|
||
#
|
||
# ⚠️ Also conditional on there being a published target, and that is
|
||
# the shipped case rather than an edge — nothing declares one until
|
||
# some service publishes behind the gateway. Authelia refuses a
|
||
# bearer-authz client with no audience, so an unconditional
|
||
# declaration would break every hive that runs a collector and
|
||
# publishes nothing.
|
||
services.hyperhive.swarm.authelia.oidc.clients =
|
||
lib.mkIf (autheliaCfg.enable && cfg.publishedScrapeTargets != { })
|
||
[
|
||
{
|
||
id = cfg.clientId;
|
||
description = "HyperHive swarm collector";
|
||
kind = "machine";
|
||
# Grants `authelia.bearer.authz`, without which the authz
|
||
# endpoint refuses an otherwise valid token and blames the
|
||
# token rather than the missing grant.
|
||
bearerAuthz = true;
|
||
# DERIVED from the targets rather than contributed alongside
|
||
# them. A service declares a URL once and this is the
|
||
# permission to reach it; two lists that had to agree would be
|
||
# a drift to maintain, and the failure mode is the quiet one —
|
||
# a target whose audience was forgotten authenticates against
|
||
# nothing and looks like a broken scrape.
|
||
audience = lib.attrValues cfg.publishedScrapeTargets;
|
||
# Stated rather than left on authelia's default, because the
|
||
# two agreeing today is not the same as this being the
|
||
# required value: authelia permits only basic / JWT methods
|
||
# for a confidential client holding that scope, and enforces
|
||
# it in the startup validator.
|
||
tokenEndpointAuthMethod = "client_secret_basic";
|
||
}
|
||
];
|
||
|
||
# Deliver the collector's client secret from authelia's container into
|
||
# this one. On the HOST because that is the only place both container
|
||
# trees are addressable: they share this host's network namespace, which
|
||
# makes them feel co-located, but their filesystem roots are separate.
|
||
#
|
||
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
|
||
# refuses to start when a bind source is missing, and this secret does not
|
||
# exist until authelia's first boot has minted it — so binding it would
|
||
# make the collector wait on a file that waits on a container that starts
|
||
# after it. On a fresh swarm that is a permanent stall presenting as
|
||
# "metrics are broken", several layers from its cause.
|
||
systemd.services.swarm-otel-oidc-secret =
|
||
lib.mkIf (autheliaCfg.enable && cfg.publishedScrapeTargets != { })
|
||
{
|
||
description = "deliver the swarm collector's OIDC client secret from authelia";
|
||
after = [ "container@${autheliaCfg.machine}.service" ];
|
||
requires = [ "container@${autheliaCfg.machine}.service" ];
|
||
before = [ "container@${cfg.machine}.service" ];
|
||
wantedBy = [ "container@${cfg.machine}.service" ];
|
||
serviceConfig = {
|
||
Type = "oneshot";
|
||
RemainAfterExit = true;
|
||
SyslogIdentifier = "swarm-otel-oidc-secret";
|
||
# Longer than the bounded wait below, and that is the point:
|
||
# `DefaultTimeoutStartSec` is 90s, so without this systemd kills
|
||
# the unit before it can emit the message naming the file it was
|
||
# waiting for — the failure then reads as a timeout with no cause.
|
||
TimeoutStartSec = "180s";
|
||
};
|
||
path = [ pkgs.coreutils ];
|
||
script = ''
|
||
set -euo pipefail
|
||
|
||
src=${lib.escapeShellArg "${autheliaCfg.hostClientSecretDir}/${cfg.clientId}.secret"}
|
||
dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"}
|
||
|
||
# authelia's container is up, but its first-boot generator may
|
||
# still be minting. Bounded wait, then fail: skipping silently
|
||
# produces a collector whose scrape gets a 401 forever, which is
|
||
# the failure this whole design exists to make impossible.
|
||
for _ in $(seq 1 60); do
|
||
[ -s "$src" ] && break
|
||
sleep 2
|
||
done
|
||
if [ ! -s "$src" ]; then
|
||
echo "authelia has not minted $src after 120s" >&2
|
||
exit 1
|
||
fi
|
||
|
||
# root-owned 0400. The collector runs under `DynamicUser`, so
|
||
# there is no uid to give it to — `LoadCredential` reads this as
|
||
# root before the sandbox exists and re-exposes it to whichever
|
||
# uid the unit got.
|
||
install -D -m 0400 -o root -g root "$src" "$dst"
|
||
'';
|
||
};
|
||
|
||
# The CA bind source is written at runtime by a host unit, so the
|
||
# container has to start after it — otherwise nspawn sets up a mount
|
||
# over a file that does not exist yet.
|
||
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
|
||
|
||
assertions = [
|
||
{
|
||
# The tier exists to hold the upstream credential and to write the
|
||
# swarm's store. With neither, it is a process that receives
|
||
# samples and drops them — which looks healthy and loses data.
|
||
assertion = otelCfg.endpoint != "" || vmCfg.enable;
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but this collector
|
||
has nowhere to send what it receives:
|
||
services.hyperhive.otel.endpoint is empty and
|
||
services.hyperhive.swarm.victoriametrics.enable is false.
|
||
|
||
Set the endpoint to export upstream, or enable the swarm's
|
||
metrics store.
|
||
'';
|
||
}
|
||
{
|
||
# Without a roster there are no receivers at all, so this
|
||
# collector would listen on nothing while looking configured.
|
||
assertion = hyperhiveCfg.swarm.hives != { };
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but
|
||
services.hyperhive.swarm.hives is empty: ingest is authenticated
|
||
per hive, so an empty roster means this collector accepts nothing
|
||
from anyone.
|
||
|
||
List the swarm's hives.
|
||
'';
|
||
}
|
||
{
|
||
# A hive whose name is one this module claims for itself collides in
|
||
# the collector's component namespace, and `//` resolves it silently:
|
||
# the swarm tier's parts win, that hive's `metrics/<name>` pipeline
|
||
# disappears, and its `resource/<name>` stamp goes with it — so the
|
||
# hive still connects and pushes into nothing, unlabelled.
|
||
#
|
||
# Checked rather than documented for the same reason the port range
|
||
# is: the names are all known at evaluation time, and the runtime
|
||
# symptom is a hive that looks healthy and reports no metrics, with
|
||
# nothing in any log naming the cause.
|
||
assertion = !lib.any (h: lib.elem h reservedOwners) (lib.attrNames hyperhiveCfg.swarm.hives);
|
||
message = ''
|
||
services.hyperhive.swarm.hives contains ${
|
||
lib.concatMapStringsSep ", " (h: "'${h}'") (
|
||
lib.filter (h: lib.elem h reservedOwners) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||
)
|
||
}, which the swarm collector reserves for its own pipelines.
|
||
|
||
The collector names components `<kind>/<owner>` and uses the hive
|
||
name as the owner, so such a hive would silently replace the
|
||
swarm tier's parts and lose its own — it would keep accepting
|
||
pushes into a pipeline that routes nowhere.
|
||
|
||
Rename the hive. Reserved: ${lib.concatMapStringsSep ", " (n: "'${n}'") reservedOwners}.
|
||
'';
|
||
}
|
||
{
|
||
# A published target that is not an `https://host/path` url. Without
|
||
# this the split returns null and the failure surfaces as
|
||
# `attempt to call elemAt on null` from inside the renderer, naming
|
||
# neither the option nor the offending value.
|
||
#
|
||
# It also enforces the scheme: a published endpoint scraped over plain
|
||
# http would put a bearer token on the wire in clear text.
|
||
assertion = lib.all (u: parsePublished u != null) (lib.attrValues cfg.publishedScrapeTargets);
|
||
message = ''
|
||
services.hyperhive.swarm.otel.publishedScrapeTargets has ${
|
||
lib.concatMapStringsSep ", " (kv: "${kv.name} = \"${kv.value}\"") (
|
||
lib.filter (kv: parsePublished kv.value == null) (lib.attrsToList cfg.publishedScrapeTargets)
|
||
)
|
||
}, which is not of the form https://<host>/<path>.
|
||
|
||
The value is both the address scraped and the audience the
|
||
collector's token is minted for, so it has to be the exact url a
|
||
request goes to. https is required: this hop carries a bearer
|
||
token, and plain http would put it on the wire in clear text.
|
||
'';
|
||
}
|
||
{
|
||
# A job name used by BOTH scrape options. Within one option this
|
||
# cannot happen — the module system refuses two definitions of the
|
||
# same key with different values, measured — but the two options
|
||
# are separate, so nothing arbitrates between them, and they
|
||
# render into a single `scrape_configs` LIST where nothing
|
||
# overwrites anything: both entries ship under one `job_name`.
|
||
#
|
||
# The message names which side each collision came from, which is
|
||
# the part a rendered-config error could not tell an operator.
|
||
assertion =
|
||
lib.intersectLists (lib.attrNames cfg.scrapeTargets) (lib.attrNames cfg.publishedScrapeTargets)
|
||
== [ ];
|
||
message = ''
|
||
services.hyperhive.swarm.otel: ${
|
||
lib.concatMapStringsSep ", " (j: "'${j}'") (
|
||
lib.intersectLists (lib.attrNames cfg.scrapeTargets) (lib.attrNames cfg.publishedScrapeTargets)
|
||
)
|
||
} is declared as both a loopback
|
||
scrapeTargets job and a publishedScrapeTargets job.
|
||
|
||
They render into one prometheus scrape_configs list, so both
|
||
entries would ship under the same job_name — nothing overwrites
|
||
anything, and the two are scraped by different rules with
|
||
different trust.
|
||
|
||
Rename one. A target is either reachable on loopback because it
|
||
shares this host, or published and reached with a credential; it
|
||
should not be described as both.
|
||
'';
|
||
}
|
||
{
|
||
# A hive proves who it is with a token this provider mints, so
|
||
# there is no version of this collector that runs without one.
|
||
# Stated as an assertion rather than a fallback because a guessed
|
||
# issuer URL evaluates cleanly and refuses every hive at runtime.
|
||
assertion = autheliaCfg.url != null;
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but
|
||
services.hyperhive.swarm.authelia.url is null: every hive
|
||
authenticates to this collector as itself, and the token comes
|
||
from the swarm's identity provider.
|
||
|
||
Point authelia.url at the swarm's provider, or enable
|
||
services.hyperhive.swarm.authelia on the host that runs it.
|
||
'';
|
||
}
|
||
{
|
||
# A port collision between two listeners on one host is a runtime
|
||
# coin toss with nothing in any log — the failure this whole
|
||
# comment budget exists to prevent. Checked against every port
|
||
# reachable from here; a port some other module picks is not.
|
||
#
|
||
# ⚠️ `cfg.port` is deliberately absent from `others`: it is the
|
||
# FIRST element of the derived range, so listing it would make this
|
||
# assertion fire on every config.
|
||
assertion =
|
||
let
|
||
derived = lib.attrValues hivePorts;
|
||
others = [
|
||
cfg.telemetryPort
|
||
otelCfg.collector.port
|
||
]
|
||
++ lib.optional vmCfg.enable vmCfg.port;
|
||
all = derived ++ others;
|
||
in
|
||
lib.length (lib.unique all) == lib.length all;
|
||
message = ''
|
||
services.hyperhive.swarm.otel: the receiver range starting at
|
||
port (${toString cfg.port}, one port per hive in
|
||
services.hyperhive.swarm.hives) overlaps another port on this
|
||
host.
|
||
|
||
Every swarm container shares the host's network namespace, so
|
||
two listeners claiming one port is not a build failure — it is
|
||
whichever process started first, silently. Move
|
||
services.hyperhive.swarm.otel.port to a free range.
|
||
'';
|
||
}
|
||
];
|
||
|
||
containers.${cfg.machine} = {
|
||
autoStart = true;
|
||
ephemeral = false;
|
||
# Shared host netns, like every sibling swarm service: the hive tier
|
||
# reaches this collector, and this collector reaches the metrics
|
||
# store, without either crossing a network boundary that would need
|
||
# its own trust material.
|
||
privateNetwork = false;
|
||
|
||
# The upstream credential is operator-provided and lives on the host.
|
||
# Read-only, and only when one is configured — binding a path that
|
||
# does not exist makes nixos-container refuse to start the container,
|
||
# which is a stall several layers from its cause.
|
||
bindMounts =
|
||
lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
${otelCfg.headersCredential} = {
|
||
hostPath = otelCfg.headersCredential;
|
||
isReadOnly = true;
|
||
};
|
||
}
|
||
# The public hive CA, read-only — only when something in here
|
||
# actually verifies a swarm-service name.
|
||
// caTrust.bindMount;
|
||
|
||
config =
|
||
{ ... }:
|
||
{
|
||
# This tier is the one that resolves an operator-configured
|
||
# hostname: `otel.endpoint` is an external URL, and reaching it
|
||
# is the entire reason this container holds a credential. The
|
||
# `/etc/resolv.conf` nixos-containers copies in is a snapshot
|
||
# taken once at boot, so without this the upstream export
|
||
# depends on the host's file having been right at that instant.
|
||
imports = [
|
||
(import ./swarm-container-resolver.nix {
|
||
inherit (config.services.hyperhive.network) bridgeIp;
|
||
dnsConsumers = [ "opentelemetry-collector.service" ];
|
||
})
|
||
]
|
||
# `SSL_CERT_FILE` REPLACES the trust store rather than adding to
|
||
# it, so a failed assembly yields an empty pool and every TLS
|
||
# call fails while the unit looks healthy. That is why this is
|
||
# the shared helper — it carries the `Requires` and the
|
||
# non-empty check — and not a local `cat`.
|
||
++ [
|
||
(caTrust.trustBundle {
|
||
inherit pkgs;
|
||
name = cfg.machine;
|
||
consumers = [ "opentelemetry-collector" ];
|
||
})
|
||
];
|
||
|
||
system.stateVersion = config.system.stateVersion;
|
||
networking.firewall.enable = false;
|
||
# Keep the host-copied /etc/resolv.conf intact — same reasoning
|
||
# as the sibling swarm containers.
|
||
networking.resolvconf.enable = lib.mkForce false;
|
||
|
||
services.opentelemetry-collector = {
|
||
enable = true;
|
||
package = pkgs.opentelemetry-collector-contrib;
|
||
# Runs `otelcol validate` at build time. ⚠️ A parser, not a
|
||
# wiring check: it accepts a receiver naming an absent
|
||
# extension, and the collector then dies at startup. A green
|
||
# build does not prove this config starts, never mind that a
|
||
# sample arrives — which is why this module's gate pushes a
|
||
# real sample through both tiers into the store.
|
||
validateConfigFile = true;
|
||
settings = {
|
||
# One receiver per hive, and that multiplicity is forced
|
||
# rather than chosen. The `hive`
|
||
# label has to come from something the sender cannot write,
|
||
# and the only such thing here is WHICH RECEIVER accepted
|
||
# the sample: a processor cannot read the token's claims
|
||
# (`from_context` reads request metadata, and asking it for
|
||
# an auth claim yields nothing — silently, with a healthy
|
||
# startup), and one receiver holding many credentials never
|
||
# reveals which one matched.
|
||
receivers =
|
||
lib.mapAttrs' (
|
||
h: p:
|
||
lib.nameValuePair "otlp/${h}" {
|
||
protocols.http = {
|
||
endpoint = "127.0.0.1:${toString p}";
|
||
auth.authenticator = "oidc/${h}";
|
||
};
|
||
}
|
||
) hivePorts
|
||
# MERGED with the per-hive receivers, never assigned over
|
||
# them. A plain assignment here would drop every hive's
|
||
# receiver and still render a valid config that starts
|
||
# cleanly — the collector has no opinion about how many
|
||
# pipelines it was supposed to have.
|
||
#
|
||
# Only emitted when a service has actually declared a
|
||
# target: a `prometheus` receiver with nothing to scrape is
|
||
# the shape this whole issue is about, a config that renders
|
||
# and deploys perfectly while adding no data.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { } || cfg.publishedScrapeTargets != { }) {
|
||
prometheus.config.scrape_configs =
|
||
lib.mapAttrsToList (job: target: {
|
||
job_name = job;
|
||
static_configs = [ { targets = [ target ]; } ];
|
||
}) cfg.scrapeTargets
|
||
# ⚠️ `++`, so the two kinds of target land in ONE list —
|
||
# which is exactly why a job name may not appear in both
|
||
# options. A list concatenation does not resolve a
|
||
# collision the way an attrset would: both entries ship
|
||
# under the same `job_name`. The assertion above is what
|
||
# stands between that and a deploy.
|
||
++ lib.mapAttrsToList (
|
||
job: url:
|
||
let
|
||
parts = parsePublished url;
|
||
in
|
||
{
|
||
job_name = job;
|
||
# Split from the declared url — see `parsePublished`.
|
||
scheme = "https";
|
||
static_configs = [ { targets = [ (lib.elemAt parts 0) ]; } ];
|
||
metrics_path = lib.elemAt parts 1;
|
||
# Prometheus-native oauth2, not the collector's
|
||
# `oauth2client` extension: the prometheus receiver
|
||
# takes upstream's scrape config verbatim and does not
|
||
# accept an `auth` block naming a collector extension.
|
||
oauth2 = {
|
||
client_id = cfg.clientId;
|
||
# A path, never a value — nothing here may read the
|
||
# secret, or it lands in the store world-readable.
|
||
client_secret_file = collectorSecretPath;
|
||
token_url = "${autheliaCfg.url}/api/oidc/token";
|
||
# The audience is the target's own url, and authelia
|
||
# checks it against the address being requested.
|
||
# Registered ≠ requested: a client that does not ASK
|
||
# for an audience gets a token with `aud: []` however
|
||
# complete its registration looks.
|
||
endpoint_params.audience = url;
|
||
};
|
||
}
|
||
) cfg.publishedScrapeTargets;
|
||
};
|
||
|
||
exporters =
|
||
lib.optionalAttrs vmCfg.enable {
|
||
# `metrics_endpoint`, NOT `endpoint`: the latter is a
|
||
# base that otlphttp appends `/v1/metrics` to, while
|
||
# VictoriaMetrics serves OTLP at
|
||
# `/opentelemetry/api/v1/push`. With `endpoint` the
|
||
# collector answers 200 to its own clients and posts the
|
||
# samples to a path that does not exist. Measured
|
||
# end-to-end, not read — `state/probe-3265-collector-to-vm.sh`.
|
||
"otlphttp/victoriametrics".metrics_endpoint =
|
||
"http://127.0.0.1:${toString vmCfg.port}/opentelemetry/api/v1/push";
|
||
}
|
||
// lib.optionalAttrs (otelCfg.endpoint != "") {
|
||
${if otelCfg.protocol == "grpc" then "otlp" else "otlphttp"} = {
|
||
endpoint = otelCfg.endpoint;
|
||
}
|
||
// lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
# Interpolated by the collector from its environment at
|
||
# runtime, never by nix: `EnvironmentFile` below is what
|
||
# puts it there, so the value is not read into the store.
|
||
headers.${otelCfg.collector.upstreamHeaderName} = "\${env:${otelCfg.collector.upstreamHeaderName}}";
|
||
}
|
||
// lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; };
|
||
};
|
||
|
||
# Moves this collector's self-metrics off the built-in
|
||
# default of `localhost:8888`, which the hive tier holds.
|
||
#
|
||
# ⚠️ `metrics.address` is the spelling that looks right and
|
||
# is REJECTED by this collector version — measured, not
|
||
# read: `'migration.MetricsConfigV030' has invalid keys:
|
||
# address`. `readers` is the schema it accepts, and the
|
||
# difference is a startup failure rather than a warning.
|
||
service.telemetry.metrics.readers = [
|
||
{
|
||
pull.exporter.prometheus = {
|
||
host = "127.0.0.1";
|
||
port = cfg.telemetryPort;
|
||
};
|
||
}
|
||
];
|
||
|
||
# ⚠️ An extension that is configured but not listed here is
|
||
# INERT — the collector starts clean and the receiver
|
||
# naming it authenticates nothing. Derived from the same
|
||
# attrset as the receivers so the two cannot disagree.
|
||
service.extensions = map (h: "oidc/${h}") (lib.attrNames hivePorts);
|
||
|
||
# Fan-out, not a choice: with both configured the same
|
||
# samples go upstream AND into the swarm's store. The store
|
||
# is for looking at this swarm; the upstream is for whoever
|
||
# aggregates across swarms, and neither replaces the other.
|
||
# `exporterNames` is shared by every pipeline — where a
|
||
# sample goes is a property of this tier, not of the hive
|
||
# that sent it.
|
||
service.pipelines =
|
||
lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "metrics/${h}" {
|
||
receivers = [ "otlp/${h}" ];
|
||
processors = [ "resource/${h}" ];
|
||
exporters = exporterNames;
|
||
}
|
||
) hivePorts
|
||
# Its OWN pipeline, and that separation is the ruling, not a
|
||
# tidiness choice: every `resource/<hive>` above UPSERTS a
|
||
# `hive` key, so a scraped swarm sample routed through any of
|
||
# them would acquire the one label a swarm-level service must
|
||
# not have. Keeping it out of them makes the absence
|
||
# structural rather than something to remember to strip.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
|
||
"metrics/${swarmTierName}" = {
|
||
receivers = [ "prometheus" ];
|
||
processors = [ "resource/${swarmTierName}" ];
|
||
exporters = exporterNames;
|
||
};
|
||
};
|
||
}
|
||
// {
|
||
extensions = lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "oidc/${h}" {
|
||
issuer_url = autheliaCfg.url;
|
||
# The audience this hive's client is registered to
|
||
# request, and the reason one hive's token is refused by
|
||
# another hive's receiver. Same expression authelia
|
||
# registers it under — a second spelling here would deny
|
||
# every hive, as a 401 that blames the token.
|
||
audience = "${autheliaCfg.hiveClientPrefix}${h}";
|
||
# ⛔ DO NOT ADD `issuer_ca_path` HERE. It took this
|
||
# collector down for forty minutes once, and the failure
|
||
# is invisible to every check we have.
|
||
#
|
||
# It loads only the FIRST certificate in the file it
|
||
# names. The bundle assembled for this container is
|
||
# `system CAs ++ hive trust bundle`, so the anchor sits
|
||
# ~123rd and is never in the pool: the extension then
|
||
# cannot verify authelia and the whole collector exits
|
||
# `x509: certificate signed by unknown authority`, on
|
||
# every start, with 125 valid certificates in the file.
|
||
#
|
||
# Leaving it unset makes the extension use the process
|
||
# trust store, which `trustBundle` already populates via
|
||
# `SSL_CERT_FILE` — and *that* consumer reads every
|
||
# certificate regardless of order. One file, two
|
||
# consumers, opposite parsing: the fix is to stop naming
|
||
# it twice, not to reorder the bundle.
|
||
#
|
||
# ⚠️ Nor is pointing it at the hive trust bundle a fix:
|
||
# `hive-tls.nix` writes that leading with the *hive* CA
|
||
# (`nameConstraints` = this hive's domain), which cannot
|
||
# issue a swarm-level name at all.
|
||
#
|
||
# (Kept from the original note, still true and still
|
||
# worth not re-deriving: `issuer_ca_file`, `ca_file` and
|
||
# `tls.ca_file` are INVALID KEYS for this extension.)
|
||
}
|
||
) hivePorts;
|
||
|
||
# `upsert`, not `insert`: a sender that stamps its own
|
||
# `hive` must be OVERWRITTEN, not deferred to. This
|
||
# processor is the whole attribution boundary — the value
|
||
# is a constant per receiver, so it says which hive
|
||
# authenticated, not which hive claimed to be sending.
|
||
processors =
|
||
lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "resource/${h}" {
|
||
attributes = [
|
||
{
|
||
key = "hive";
|
||
value = h;
|
||
action = "upsert";
|
||
}
|
||
# Stamped here rather than on a separate upstream-only
|
||
# pipeline, which would double the pipeline count to
|
||
# withhold one constant label from the local store. It
|
||
# is redundant there — one VictoriaMetrics per swarm, so
|
||
# every series in it already belongs to this swarm — but
|
||
# a constant label multiplies no series, and it means
|
||
# what LEAVES and what STAYS have the same shape.
|
||
#
|
||
# Upstream is where it stops being redundant: that is the
|
||
# one hop where several swarms can land in one store, and
|
||
# samples that cannot name their swarm collide there
|
||
# exactly as hives collided here before per-hive
|
||
# receivers existed.
|
||
{
|
||
key = "swarm";
|
||
value = swarmDisplayName;
|
||
action = "upsert";
|
||
}
|
||
];
|
||
}
|
||
) hivePorts
|
||
# The swarm tier's own stamp: `swarm` and deliberately NO
|
||
# `hive`. A scraped swarm service belongs to the swarm, not to
|
||
# any one hive, so there is no honest value to put there — and
|
||
# an invented one (a sentinel, the local hive's name) would be
|
||
# queried as though it meant something.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
|
||
"resource/${swarmTierName}".attributes = [
|
||
{
|
||
# The metric LABEL, a different namespace from the
|
||
# component name above — deliberately not interpolated.
|
||
key = "swarm";
|
||
value = swarmDisplayName;
|
||
action = "upsert";
|
||
}
|
||
];
|
||
};
|
||
};
|
||
};
|
||
|
||
# The credential file is already `NAME=value`, systemd's
|
||
# EnvironmentFile format — so the secret reaches the process as an
|
||
# environment variable without being read by nix, written to the
|
||
# store, or passed in argv.
|
||
systemd.services.opentelemetry-collector.serviceConfig =
|
||
lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
EnvironmentFile = otelCfg.headersCredential;
|
||
}
|
||
# The other credential, and the other direction: the one above
|
||
# authenticates this collector's export onward, this one
|
||
# authenticates it to a service it scrapes.
|
||
#
|
||
# ⚠️ Only where a published target exists. `LoadCredential` on a
|
||
# missing source is a unit that refuses to start, so declaring it
|
||
# unconditionally would take the collector down on every hive that
|
||
# scrapes nothing published — the empty case is the shipped one.
|
||
// lib.optionalAttrs (cfg.publishedScrapeTargets != { }) {
|
||
LoadCredential = [ "${collectorCredentialId}:${collectorSecretInContainer}" ];
|
||
};
|
||
};
|
||
};
|
||
};
|
||
}
|