d2de3e8a2a
The option's description claimed declaring an entry from the service's own module put 'the scraper and the target on the same host by construction rather than by luck'. It does not. It constrains where the target is; nothing in it places the collector, and the two enable flags are co-located by a shared lib.mkDefault rather than by construction. Split across hosts, a target is silently never scraped — the service's host declares an entry no local collector reads, the collector's host never enabled the service. No error surfaces, and no assertion can catch it: separate hosts are separate evaluations with no shared context, so the doc telling the truth is the only mechanism there is. The same paragraph already warned co-location was not a guarantee, four lines below the sentence claiming it was; a reader arriving for permission stopped at the permission. This one did. swarm-nats carries the concrete caveat for its own contribution.
973 lines
48 KiB
Nix
973 lines
48 KiB
Nix
# The swarm's telemetry collector: one per swarm, in a `swarm-otel`
|
||
# nixos-container beside the swarm's other shared services.
|
||
#
|
||
# Two tiers, and they are separate on purpose:
|
||
#
|
||
# - `otel.nix` is the **hive** tier. It receives from this hive's agents
|
||
# on the bridge and forwards, and it holds no upstream credential.
|
||
# - this is the **swarm** tier. It is the only holder of the upstream
|
||
# credential, the only writer to the swarm's metrics store, and the
|
||
# place that will stamp `hive=` from the authenticated connection
|
||
# rather than from anything a sender can choose.
|
||
#
|
||
# On a host that runs both, both processes run. They are not collapsed:
|
||
# all-local is a statement about *where* processes run, not about what
|
||
# shape the deployment has, and a local tier boundary that disappears is
|
||
# one the local deployment stops testing. `hive=` attribution is the
|
||
# property that would differ, and the ingest auth that makes it
|
||
# unforgeable is built on this boundary existing.
|
||
#
|
||
# A container rather than a second host unit, for the same reason every
|
||
# sibling swarm service is one — and because `services.opentelemetry-collector`
|
||
# is a singleton NixOS option, already spoken for on the host by the hive
|
||
# tier. A container gets its own evaluation and therefore its own collector.
|
||
{
|
||
pkgs,
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
cfg = config.services.hyperhive.swarm.otel;
|
||
swarmCfg = config.services.hyperhive.swarm;
|
||
otelCfg = config.services.hyperhive.otel;
|
||
vmCfg = config.services.hyperhive.swarm.victoriametrics;
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
gatewayCfg = hyperhiveCfg.gateway;
|
||
swarmDomain = hyperhiveCfg.swarm.domain;
|
||
|
||
# Total on a null swarm domain for the same reason every sibling module is:
|
||
# the required-domain assertion in hive-network.nix should be what an
|
||
# operator sees, not a coercion error from here.
|
||
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
|
||
|
||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||
|
||
# The collector names its components `<kind>/<owner>`, where `<owner>` is a
|
||
# hive name for the per-hive pipelines and this literal for the swarm tier's
|
||
# own. The two share one namespace and are merged with `//`, so a hive named
|
||
# this would silently REPLACE the swarm tier's parts — and lose its own
|
||
# pipeline in the process, while its receiver keeps accepting pushes.
|
||
#
|
||
# Bound once and interpolated at every swarm-tier use below, so the assertion
|
||
# that reserves it is checking the same string the config emits. A literal
|
||
# repeated at each site would let the guard and the config drift apart, which
|
||
# is the failure this guard exists to prevent.
|
||
swarmTierName = "swarm";
|
||
# Every `<owner>` this module claims for itself. One entry today; a second
|
||
# swarm-tier pipeline would be added here and inherit the check for free.
|
||
reservedOwners = [ swarmTierName ];
|
||
|
||
# A published target is declared as ONE url, because that url is also the
|
||
# audience its token is minted for — but prometheus wants the same fact in
|
||
# three fields. Split it here rather than asking a service to state it
|
||
# twice: two spellings of one address is a mismatch waiting to happen, and
|
||
# the failure is a valid token refused at the target.
|
||
#
|
||
# `null` when the shape is wrong, which the assertion below reports by name.
|
||
# ⚠️ The pattern requires `https`. A published endpoint reached over plain
|
||
# http would ship a bearer token in clear text, and that is worth an eval
|
||
# error rather than a warning nobody reads.
|
||
parsePublished = url: builtins.match "https://([^/]+)(/.*)" url;
|
||
|
||
# The client secret takes three names, and the reason is `DynamicUser`.
|
||
#
|
||
# Upstream's collector unit runs with `DynamicUser = true`, and the
|
||
# prometheus receiver opens `client_secret_file` ITSELF, at runtime, as that
|
||
# user — so there is no stable uid to hand a file to, and the root-owned
|
||
# 0400 shape `hive-matrix-oidc-secret` delivers to would be unreadable.
|
||
# (Matrix gets away with it because `LoadCredential` reads the file as root
|
||
# before the sandbox exists, and tuwunel never opens that path itself.)
|
||
#
|
||
# `LoadCredential` solves both halves: systemd reads the file as root and
|
||
# re-exposes it to the dynamic user under a path that does not depend on
|
||
# which uid it turned out to be.
|
||
#
|
||
# At rest in the container's tree — written by the host oneshot below.
|
||
# Under /var/lib and not /run because the collector may start before the
|
||
# delivery unit on a later boot, and a secret that evaporates on reboot
|
||
# turns a working scrape into an intermittent one.
|
||
collectorSecretInContainer = "/var/lib/swarm-otel-oidc/${cfg.clientId}.secret";
|
||
collectorCredentialId = "oidc-client-secret";
|
||
# What the scrape config points at. ⚠️ This path and the `LoadCredential`
|
||
# id below are one fact spelled twice by systemd's design — both derive from
|
||
# `collectorCredentialId` so they cannot drift; a mismatch is a file the
|
||
# collector cannot open, discovered at runtime and nowhere else.
|
||
collectorSecretPath = "/run/credentials/opentelemetry-collector.service/${collectorCredentialId}";
|
||
|
||
# `attrNames` is sorted, so this is a function of the hive SET and not of
|
||
# the order anyone wrote it in.
|
||
#
|
||
# These ports are internal and appear in no URL: a hive addresses its own
|
||
# receiver as a PATH on this collector's single gateway name, and nginx —
|
||
# rendered from this same evaluation — is the only thing that ever names
|
||
# the port. That is what makes deriving them safe here and unsafe in the
|
||
# obvious other place: were a hive told a port, inserting a hive would
|
||
# renumber the ones after it and silently move a port a running hive was
|
||
# already sending to.
|
||
hivePorts = lib.listToAttrs (
|
||
lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||
);
|
||
|
||
# The swarm's authelia is reached by its gateway name, whose leaf is
|
||
# issued by the swarm services sub-CA — so this container needs the same
|
||
# runtime CA trust every other consumer of a swarm-service name needs.
|
||
# The CA is generated at runtime and cannot be baked into a derivation,
|
||
# which is why it arrives as a bind mount rather than
|
||
# `security.pki.certificateFiles`.
|
||
caTrust = import ./lib/hive-ca-trust.nix {
|
||
inherit lib;
|
||
tlsCfg = hyperhiveCfg.tls;
|
||
inherit gatewayCfg;
|
||
};
|
||
|
||
# `unknown` rather than omitting the label, copying `agent-modules/otel.nix`
|
||
# deliberately: a producer that cannot name its swarm should say so in the
|
||
# same vocabulary as every other producer, so a query never has to handle
|
||
# both "the label is absent" and "the label says unknown".
|
||
swarmDisplayName = if hyperhiveCfg.swarm.name == null then "unknown" else hyperhiveCfg.swarm.name;
|
||
|
||
# One list, read by every pipeline: the per-hive pipelines fan out to
|
||
# exactly the same destinations as the single pipeline they replace.
|
||
# Written once because "which exporters" is a property of this tier, not
|
||
# of which hive a sample came from.
|
||
exporterNames =
|
||
lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
|
||
++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
|
||
in
|
||
{
|
||
options.services.hyperhive.swarm.otel = {
|
||
enable = lib.mkOption {
|
||
type = lib.types.bool;
|
||
default = false;
|
||
description = ''
|
||
Run the swarm's telemetry collector on this host.
|
||
|
||
Asserted from `swarm.enableRequiredServices` in
|
||
./swarm-required-services.nix, with the metrics pair this
|
||
collector feeds: a swarm has one of these, and it belongs
|
||
wherever the shared services live rather than on every hive.
|
||
|
||
A hive that does not run it still runs its own hive-tier collector
|
||
(`services.hyperhive.otel.enable`) and reaches this one by name, at
|
||
{option}`services.hyperhive.swarm.otel.domain`.
|
||
'';
|
||
};
|
||
|
||
machine = lib.mkOption {
|
||
type = lib.types.str;
|
||
readOnly = true;
|
||
default = "swarm-otel";
|
||
description = ''
|
||
Name of the nixos-container this collector runs in — also the
|
||
`machinectl` name, so other modules may read it rather than
|
||
repeating the literal.
|
||
'';
|
||
};
|
||
|
||
port = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 4319;
|
||
description = ''
|
||
First port of this collector's receiver range. Every hive in
|
||
{option}`services.hyperhive.swarm.hives` gets its **own**
|
||
authenticated receiver — that is what makes the `hive` label
|
||
unforgeable — so the range is one port per hive, starting here, in
|
||
sorted-name order.
|
||
|
||
⚠️ Internal. No client is ever told a port: a hive reaches its own
|
||
receiver as `https://''${domain}/<hive>`, and the gateway routes on
|
||
that path. So adding a hive, which renumbers the ones after it, is
|
||
harmless — nginx is rendered from this same evaluation and moves
|
||
with it.
|
||
|
||
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
|
||
hive tier already uses it (`services.hyperhive.otel.collector.port`)
|
||
and every swarm container shares the host's network namespace. Two
|
||
listeners claiming one port on one host is not a build failure —
|
||
it is a runtime coin toss over which one gets it, with nothing in
|
||
any log saying so. The same collision cost a release when grafana
|
||
and the forge both defaulted to 3000. The assertions below check
|
||
the whole derived range against every port this module and the hive
|
||
tier declare, which is as far as a module can see.
|
||
'';
|
||
};
|
||
|
||
telemetryPort = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 8889;
|
||
description = ''
|
||
Port this collector serves its **own** metrics on — queue depth,
|
||
refused and dropped samples, exporter failures. How you find out
|
||
that telemetry is being lost, so it is worth keeping rather than
|
||
switching off.
|
||
|
||
⚠️ **Deliberately not 8889's neighbour 8888**, which is the
|
||
collector's built-in default and therefore what the hive tier
|
||
already binds. Two collectors share a network namespace whenever
|
||
they are co-located, and unlike the OTLP port this one appears
|
||
nowhere in either config — it is a default inside the binary, so
|
||
nothing that compares configured ports can see the clash. The
|
||
second collector to start simply dies with
|
||
`bind: address already in use`.
|
||
'';
|
||
};
|
||
|
||
domain = lib.mkOption {
|
||
type = lib.types.str;
|
||
default = "otel.${domainBase}";
|
||
defaultText = lib.literalExpression ''"otel.''${services.hyperhive.swarm.domain}"'';
|
||
description = ''
|
||
Name the gateway serves this on. A sibling of the swarm's other
|
||
service names, so the swarm-services sub-CA can issue for it — see
|
||
`hive-tls.nix` for why a service name being a sibling rather than a
|
||
child decides which CA may sign it.
|
||
|
||
This is what the **hive** tier's exporter reaches — the hive
|
||
collector is a plain producer against this name exactly like every
|
||
other client of a swarm service, resolved locally by dnsmasq on a
|
||
co-located host and over the real network otherwise. There is no
|
||
separate loopback-vs-remote knob to get wrong: `swarm-nats` is the
|
||
deliberate exception to this pattern (its cross-hive reach is the
|
||
wireguard mesh, not the gateway), everything else in this swarm
|
||
addresses its siblings by name.
|
||
'';
|
||
};
|
||
|
||
scrapeTargets = lib.mkOption {
|
||
type = lib.types.attrsOf lib.types.str;
|
||
default = { };
|
||
example = lib.literalExpression ''{ forgejo = "127.0.0.1:3000"; }'';
|
||
description = ''
|
||
Prometheus exposition endpoints this collector scrapes, as
|
||
`<job name> = "<host>:<port>"`.
|
||
|
||
**A service declares its own entry, from its own module, under its
|
||
own `enable`.** Do not assemble the list here: an entry then exists
|
||
only where the service that named it runs, so a target is never
|
||
declared on a host that does not serve it.
|
||
|
||
⚠️ **That constrains the TARGET, not the SCRAPER, and the difference
|
||
is a silent gap.** Nothing here puts the collector on the same host.
|
||
Services are co-located by *default* — `swarm-required-services.nix`
|
||
derives their `enable` flags from one `lib.mkDefault` — not by
|
||
construction, and an operator may split them.
|
||
|
||
When they are split the target is simply never scraped: the
|
||
service's host declares an entry no local collector reads, and the
|
||
collector's host never enabled that service so has no entry at all.
|
||
No error, no eval failure, no warning.
|
||
|
||
**No assertion can catch this.** Two hosts are separate NixOS
|
||
evaluations with no shared context, so neither can see what the
|
||
other runs. A service that would be seriously wrong to lose should
|
||
say so in its own contribution, because saying it is the only
|
||
mechanism available.
|
||
|
||
Samples land in a swarm-level pipeline that stamps `swarm` and
|
||
**never** `hive`: a swarm service does not belong to a hive, and
|
||
`hive` stays a property of which authenticated receiver accepted a
|
||
push, not something a scrape can acquire.
|
||
|
||
Empty by default, in which case no scrape receiver, processor or
|
||
pipeline is emitted at all — an enabled scraper with nothing to
|
||
scrape is the inert configuration this option exists to avoid.
|
||
'';
|
||
};
|
||
|
||
publishedScrapeTargets = lib.mkOption {
|
||
type = lib.types.attrsOf lib.types.str;
|
||
default = { };
|
||
example = lib.literalExpression ''{ forge = "https://forge.example.com/metrics"; }'';
|
||
description = ''
|
||
Prometheus endpoints this collector scrapes **by name, with a
|
||
credential**, as `<job name> = "<url>"`.
|
||
|
||
A service module declares its own entry, from its own module, the
|
||
same way it does for `scrapeTargets` — and the collector's OAuth2
|
||
client derives its permitted audiences from these URLs, so a
|
||
target and the authorisation to reach it are **one declaration**.
|
||
Two lists that must agree would be a drift to maintain, and its
|
||
failure mode is the bad one: a target whose audience was forgotten
|
||
authenticates against nothing and reads as a scrape failure rather
|
||
than a config mistake.
|
||
|
||
⚠️ A **full URL**, not `host:port`. Authelia validates a bearer
|
||
token against the address being requested, so the string here is
|
||
also the audience the token is minted for; a near miss (a trailing
|
||
slash, `http` for `https`) presents as a valid token rejected at
|
||
the target, several layers from its cause.
|
||
|
||
⚠️ Deliberately separate from `scrapeTargets`. An entry there is
|
||
trusted because the scraper and the target share a host — that
|
||
option is loopback-and-unauthenticated by contract. An entry here
|
||
is trusted because it presents a credential. One shape for both
|
||
would leave a reader unable to tell which of those a given target
|
||
relies on.
|
||
'';
|
||
};
|
||
|
||
clientId = lib.mkOption {
|
||
type = lib.types.str;
|
||
readOnly = true;
|
||
default = "swarm-collector";
|
||
description = ''
|
||
OAuth2 client id this collector authenticates as. Published so
|
||
authelia's `access_control` rules can name it without carrying a
|
||
second copy of the string, exactly as
|
||
`services.hyperhive.swarm.authelia.hiveClientPrefix` is published
|
||
for the queue's responder.
|
||
|
||
Two spellings drifting apart is not a build failure: authelia
|
||
refuses a rule naming an unregistered client in its startup
|
||
validator, so the swarm's SSO service fails to *restart* — long
|
||
after the change that caused it evaluated cleanly.
|
||
'';
|
||
};
|
||
};
|
||
|
||
config = lib.mkIf (config.services.hyperhive.enable && cfg.enable) {
|
||
# The gateway name, inside `cfg.enable` — that guard is the load-bearing
|
||
# part. Every hive in a swarm may know this collector exists, but only
|
||
# the host that RUNS it may claim the name; a client hive declaring the
|
||
# vhost would answer for a service it does not have.
|
||
services.hyperhive.gateway.localNames = [ cfg.domain ];
|
||
|
||
# OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
|
||
# every sibling swarm service: TLS terminates here, then plain http to
|
||
# the co-located container over loopback (shared netns, like the store
|
||
# this collector writes to).
|
||
services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
|
||
listen = gatewayCfg.lib.listen;
|
||
extraConfig = gatewayCfg.lib.securityHeaders;
|
||
locations =
|
||
# One name for the whole collector, and the hive is a path under
|
||
# it. The alternative — a vhost per hive — needs a certificate,
|
||
# a DNS name and a `localNames` entry per hive to express the
|
||
# same routing the gateway already does for free.
|
||
#
|
||
# ⚠️ The trailing slash on both sides is load-bearing: it is what
|
||
# strips `/<hive>` before the request reaches the receiver, which
|
||
# serves `/v1/metrics` and knows nothing about hives. Without it
|
||
# the receiver sees `/<hive>/v1/metrics` and answers 404 to a
|
||
# request that authenticated perfectly.
|
||
lib.mapAttrs' (
|
||
h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; }
|
||
) hivePorts
|
||
// {
|
||
# There is no swarm-wide inbox, and a closed door is the honest
|
||
# description of that. Every route into this collector belongs to
|
||
# exactly one hive.
|
||
"/".return = "404";
|
||
};
|
||
};
|
||
|
||
# Turning on the identities this tier authenticates against, which the
|
||
# option exists to allow: its own description names this module as the
|
||
# second consumer, so the queue is not a prerequisite for authenticated
|
||
# telemetry.
|
||
services.hyperhive.swarm.authelia.oidc.hiveIdentities = true;
|
||
|
||
# The collector's own identity, for the other direction: the hive
|
||
# identities above are how this collector authenticates its *callers*,
|
||
# this is how it authenticates *itself* to a service published behind
|
||
# the gateway.
|
||
#
|
||
# Only where authelia is co-located. A client is a row in this host's
|
||
# provider config, so declaring one against a remote provider would
|
||
# render nothing while reading as done; a swarm whose authelia lives
|
||
# elsewhere registers it there.
|
||
#
|
||
# ⚠️ Also conditional on there being a published target, and that is
|
||
# the shipped case rather than an edge — nothing declares one until
|
||
# some service publishes behind the gateway. Authelia refuses a
|
||
# bearer-authz client with no audience, so an unconditional
|
||
# declaration would break every hive that runs a collector and
|
||
# publishes nothing.
|
||
services.hyperhive.swarm.authelia.oidc.clients =
|
||
lib.mkIf (autheliaCfg.enable && cfg.publishedScrapeTargets != { })
|
||
[
|
||
{
|
||
id = cfg.clientId;
|
||
description = "HyperHive swarm collector";
|
||
kind = "machine";
|
||
# Grants `authelia.bearer.authz`, without which the authz
|
||
# endpoint refuses an otherwise valid token and blames the
|
||
# token rather than the missing grant.
|
||
bearerAuthz = true;
|
||
# DERIVED from the targets rather than contributed alongside
|
||
# them. A service declares a URL once and this is the
|
||
# permission to reach it; two lists that had to agree would be
|
||
# a drift to maintain, and the failure mode is the quiet one —
|
||
# a target whose audience was forgotten authenticates against
|
||
# nothing and looks like a broken scrape.
|
||
audience = lib.attrValues cfg.publishedScrapeTargets;
|
||
# Stated rather than left on authelia's default, because the
|
||
# two agreeing today is not the same as this being the
|
||
# required value: authelia permits only basic / JWT methods
|
||
# for a confidential client holding that scope, and enforces
|
||
# it in the startup validator.
|
||
tokenEndpointAuthMethod = "client_secret_basic";
|
||
}
|
||
];
|
||
|
||
# Deliver the collector's client secret from authelia's container into
|
||
# this one. On the HOST because that is the only place both container
|
||
# trees are addressable: they share this host's network namespace, which
|
||
# makes them feel co-located, but their filesystem roots are separate.
|
||
#
|
||
# ⚠️ Deliberately a copy and not a `bindMounts` entry. nixos-container
|
||
# refuses to start when a bind source is missing, and this secret does not
|
||
# exist until authelia's first boot has minted it — so binding it would
|
||
# make the collector wait on a file that waits on a container that starts
|
||
# after it. On a fresh swarm that is a permanent stall presenting as
|
||
# "metrics are broken", several layers from its cause.
|
||
systemd.services.swarm-otel-oidc-secret =
|
||
lib.mkIf (autheliaCfg.enable && cfg.publishedScrapeTargets != { })
|
||
{
|
||
description = "deliver the swarm collector's OIDC client secret from authelia";
|
||
after = [ "container@${autheliaCfg.machine}.service" ];
|
||
requires = [ "container@${autheliaCfg.machine}.service" ];
|
||
before = [ "container@${cfg.machine}.service" ];
|
||
wantedBy = [ "container@${cfg.machine}.service" ];
|
||
serviceConfig = {
|
||
Type = "oneshot";
|
||
RemainAfterExit = true;
|
||
SyslogIdentifier = "swarm-otel-oidc-secret";
|
||
# Longer than the bounded wait below, and that is the point:
|
||
# `DefaultTimeoutStartSec` is 90s, so without this systemd kills
|
||
# the unit before it can emit the message naming the file it was
|
||
# waiting for — the failure then reads as a timeout with no cause.
|
||
TimeoutStartSec = "180s";
|
||
};
|
||
path = [ pkgs.coreutils ];
|
||
script = ''
|
||
set -euo pipefail
|
||
|
||
src=${lib.escapeShellArg "${autheliaCfg.hostClientSecretDir}/${cfg.clientId}.secret"}
|
||
dst=${lib.escapeShellArg "/var/lib/nixos-containers/${cfg.machine}${collectorSecretInContainer}"}
|
||
|
||
# authelia's container is up, but its first-boot generator may
|
||
# still be minting. Bounded wait, then fail: skipping silently
|
||
# produces a collector whose scrape gets a 401 forever, which is
|
||
# the failure this whole design exists to make impossible.
|
||
for _ in $(seq 1 60); do
|
||
[ -s "$src" ] && break
|
||
sleep 2
|
||
done
|
||
if [ ! -s "$src" ]; then
|
||
echo "authelia has not minted $src after 120s" >&2
|
||
exit 1
|
||
fi
|
||
|
||
# root-owned 0400. The collector runs under `DynamicUser`, so
|
||
# there is no uid to give it to — `LoadCredential` reads this as
|
||
# root before the sandbox exists and re-exposes it to whichever
|
||
# uid the unit got.
|
||
install -D -m 0400 -o root -g root "$src" "$dst"
|
||
'';
|
||
};
|
||
|
||
# The CA bind source is written at runtime by a host unit, so the
|
||
# container has to start after it — otherwise nspawn sets up a mount
|
||
# over a file that does not exist yet.
|
||
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
|
||
|
||
assertions = [
|
||
{
|
||
# The tier exists to hold the upstream credential and to write the
|
||
# swarm's store. With neither, it is a process that receives
|
||
# samples and drops them — which looks healthy and loses data.
|
||
assertion = otelCfg.endpoint != "" || vmCfg.enable;
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but this collector
|
||
has nowhere to send what it receives:
|
||
services.hyperhive.otel.endpoint is empty and
|
||
services.hyperhive.swarm.victoriametrics.enable is false.
|
||
|
||
Set the endpoint to export upstream, or enable the swarm's
|
||
metrics store.
|
||
'';
|
||
}
|
||
{
|
||
# Without a roster there are no receivers at all, so this
|
||
# collector would listen on nothing while looking configured.
|
||
assertion = hyperhiveCfg.swarm.hives != { };
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but
|
||
services.hyperhive.swarm.hives is empty: ingest is authenticated
|
||
per hive, so an empty roster means this collector accepts nothing
|
||
from anyone.
|
||
|
||
List the swarm's hives.
|
||
'';
|
||
}
|
||
{
|
||
# A hive whose name is one this module claims for itself collides in
|
||
# the collector's component namespace, and `//` resolves it silently:
|
||
# the swarm tier's parts win, that hive's `metrics/<name>` pipeline
|
||
# disappears, and its `resource/<name>` stamp goes with it — so the
|
||
# hive still connects and pushes into nothing, unlabelled.
|
||
#
|
||
# Checked rather than documented for the same reason the port range
|
||
# is: the names are all known at evaluation time, and the runtime
|
||
# symptom is a hive that looks healthy and reports no metrics, with
|
||
# nothing in any log naming the cause.
|
||
assertion = !lib.any (h: lib.elem h reservedOwners) (lib.attrNames hyperhiveCfg.swarm.hives);
|
||
message = ''
|
||
services.hyperhive.swarm.hives contains ${
|
||
lib.concatMapStringsSep ", " (h: "'${h}'") (
|
||
lib.filter (h: lib.elem h reservedOwners) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||
)
|
||
}, which the swarm collector reserves for its own pipelines.
|
||
|
||
The collector names components `<kind>/<owner>` and uses the hive
|
||
name as the owner, so such a hive would silently replace the
|
||
swarm tier's parts and lose its own — it would keep accepting
|
||
pushes into a pipeline that routes nowhere.
|
||
|
||
Rename the hive. Reserved: ${lib.concatMapStringsSep ", " (n: "'${n}'") reservedOwners}.
|
||
'';
|
||
}
|
||
{
|
||
# A published target that is not an `https://host/path` url. Without
|
||
# this the split returns null and the failure surfaces as
|
||
# `attempt to call elemAt on null` from inside the renderer, naming
|
||
# neither the option nor the offending value.
|
||
#
|
||
# It also enforces the scheme: a published endpoint scraped over plain
|
||
# http would put a bearer token on the wire in clear text.
|
||
assertion = lib.all (u: parsePublished u != null) (lib.attrValues cfg.publishedScrapeTargets);
|
||
message = ''
|
||
services.hyperhive.swarm.otel.publishedScrapeTargets has ${
|
||
lib.concatMapStringsSep ", " (kv: "${kv.name} = \"${kv.value}\"") (
|
||
lib.filter (kv: parsePublished kv.value == null) (lib.attrsToList cfg.publishedScrapeTargets)
|
||
)
|
||
}, which is not of the form https://<host>/<path>.
|
||
|
||
The value is both the address scraped and the audience the
|
||
collector's token is minted for, so it has to be the exact url a
|
||
request goes to. https is required: this hop carries a bearer
|
||
token, and plain http would put it on the wire in clear text.
|
||
'';
|
||
}
|
||
{
|
||
# A job name used by BOTH scrape options. Within one option this
|
||
# cannot happen — the module system refuses two definitions of the
|
||
# same key with different values, measured — but the two options
|
||
# are separate, so nothing arbitrates between them, and they
|
||
# render into a single `scrape_configs` LIST where nothing
|
||
# overwrites anything: both entries ship under one `job_name`.
|
||
#
|
||
# The message names which side each collision came from, which is
|
||
# the part a rendered-config error could not tell an operator.
|
||
assertion =
|
||
lib.intersectLists (lib.attrNames cfg.scrapeTargets) (lib.attrNames cfg.publishedScrapeTargets)
|
||
== [ ];
|
||
message = ''
|
||
services.hyperhive.swarm.otel: ${
|
||
lib.concatMapStringsSep ", " (j: "'${j}'") (
|
||
lib.intersectLists (lib.attrNames cfg.scrapeTargets) (lib.attrNames cfg.publishedScrapeTargets)
|
||
)
|
||
} is declared as both a loopback
|
||
scrapeTargets job and a publishedScrapeTargets job.
|
||
|
||
They render into one prometheus scrape_configs list, so both
|
||
entries would ship under the same job_name — nothing overwrites
|
||
anything, and the two are scraped by different rules with
|
||
different trust.
|
||
|
||
Rename one. A target is either reachable on loopback because it
|
||
shares this host, or published and reached with a credential; it
|
||
should not be described as both.
|
||
'';
|
||
}
|
||
{
|
||
# A hive proves who it is with a token this provider mints, so
|
||
# there is no version of this collector that runs without one.
|
||
# Stated as an assertion rather than a fallback because a guessed
|
||
# issuer URL evaluates cleanly and refuses every hive at runtime.
|
||
assertion = autheliaCfg.url != null;
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but
|
||
services.hyperhive.swarm.authelia.url is null: every hive
|
||
authenticates to this collector as itself, and the token comes
|
||
from the swarm's identity provider.
|
||
|
||
Point authelia.url at the swarm's provider, or enable
|
||
services.hyperhive.swarm.authelia on the host that runs it.
|
||
'';
|
||
}
|
||
{
|
||
# A port collision between two listeners on one host is a runtime
|
||
# coin toss with nothing in any log — the failure this whole
|
||
# comment budget exists to prevent. Checked against every port
|
||
# reachable from here; a port some other module picks is not.
|
||
#
|
||
# ⚠️ `cfg.port` is deliberately absent from `others`: it is the
|
||
# FIRST element of the derived range, so listing it would make this
|
||
# assertion fire on every config.
|
||
assertion =
|
||
let
|
||
derived = lib.attrValues hivePorts;
|
||
others = [
|
||
cfg.telemetryPort
|
||
otelCfg.collector.port
|
||
]
|
||
++ lib.optional vmCfg.enable vmCfg.port;
|
||
all = derived ++ others;
|
||
in
|
||
lib.length (lib.unique all) == lib.length all;
|
||
message = ''
|
||
services.hyperhive.swarm.otel: the receiver range starting at
|
||
port (${toString cfg.port}, one port per hive in
|
||
services.hyperhive.swarm.hives) overlaps another port on this
|
||
host.
|
||
|
||
Every swarm container shares the host's network namespace, so
|
||
two listeners claiming one port is not a build failure — it is
|
||
whichever process started first, silently. Move
|
||
services.hyperhive.swarm.otel.port to a free range.
|
||
'';
|
||
}
|
||
];
|
||
|
||
containers.${cfg.machine} = {
|
||
autoStart = true;
|
||
ephemeral = false;
|
||
# Shared host netns, like every sibling swarm service: the hive tier
|
||
# reaches this collector, and this collector reaches the metrics
|
||
# store, without either crossing a network boundary that would need
|
||
# its own trust material.
|
||
privateNetwork = false;
|
||
|
||
# The upstream credential is operator-provided and lives on the host.
|
||
# Read-only, and only when one is configured — binding a path that
|
||
# does not exist makes nixos-container refuse to start the container,
|
||
# which is a stall several layers from its cause.
|
||
bindMounts =
|
||
lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
${otelCfg.headersCredential} = {
|
||
hostPath = otelCfg.headersCredential;
|
||
isReadOnly = true;
|
||
};
|
||
}
|
||
# The public hive CA, read-only — only when something in here
|
||
# actually verifies a swarm-service name.
|
||
// caTrust.bindMount;
|
||
|
||
config =
|
||
{ ... }:
|
||
{
|
||
# This tier is the one that resolves an operator-configured
|
||
# hostname: `otel.endpoint` is an external URL, and reaching it
|
||
# is the entire reason this container holds a credential. The
|
||
# `/etc/resolv.conf` nixos-containers copies in is a snapshot
|
||
# taken once at boot, so without this the upstream export
|
||
# depends on the host's file having been right at that instant.
|
||
imports = [
|
||
(import ./swarm-container-resolver.nix {
|
||
inherit (config.services.hyperhive.network) bridgeIp;
|
||
dnsConsumers = [ "opentelemetry-collector.service" ];
|
||
})
|
||
]
|
||
# `SSL_CERT_FILE` REPLACES the trust store rather than adding to
|
||
# it, so a failed assembly yields an empty pool and every TLS
|
||
# call fails while the unit looks healthy. That is why this is
|
||
# the shared helper — it carries the `Requires` and the
|
||
# non-empty check — and not a local `cat`.
|
||
++ [
|
||
(caTrust.trustBundle {
|
||
inherit pkgs;
|
||
name = cfg.machine;
|
||
consumers = [ "opentelemetry-collector" ];
|
||
})
|
||
];
|
||
|
||
system.stateVersion = config.system.stateVersion;
|
||
networking.firewall.enable = false;
|
||
# Keep the host-copied /etc/resolv.conf intact — same reasoning
|
||
# as the sibling swarm containers.
|
||
networking.resolvconf.enable = lib.mkForce false;
|
||
|
||
services.opentelemetry-collector = {
|
||
enable = true;
|
||
package = pkgs.opentelemetry-collector-contrib;
|
||
# Runs `otelcol validate` at build time. ⚠️ A parser, not a
|
||
# wiring check: it accepts a receiver naming an absent
|
||
# extension, and the collector then dies at startup. A green
|
||
# build does not prove this config starts, never mind that a
|
||
# sample arrives — which is why this module's gate pushes a
|
||
# real sample through both tiers into the store.
|
||
validateConfigFile = true;
|
||
settings = {
|
||
# One receiver per hive, and that multiplicity is forced
|
||
# rather than chosen. The `hive`
|
||
# label has to come from something the sender cannot write,
|
||
# and the only such thing here is WHICH RECEIVER accepted
|
||
# the sample: a processor cannot read the token's claims
|
||
# (`from_context` reads request metadata, and asking it for
|
||
# an auth claim yields nothing — silently, with a healthy
|
||
# startup), and one receiver holding many credentials never
|
||
# reveals which one matched.
|
||
receivers =
|
||
lib.mapAttrs' (
|
||
h: p:
|
||
lib.nameValuePair "otlp/${h}" {
|
||
protocols.http = {
|
||
endpoint = "127.0.0.1:${toString p}";
|
||
auth.authenticator = "oidc/${h}";
|
||
};
|
||
}
|
||
) hivePorts
|
||
# MERGED with the per-hive receivers, never assigned over
|
||
# them. A plain assignment here would drop every hive's
|
||
# receiver and still render a valid config that starts
|
||
# cleanly — the collector has no opinion about how many
|
||
# pipelines it was supposed to have.
|
||
#
|
||
# Only emitted when a service has actually declared a
|
||
# target: a `prometheus` receiver with nothing to scrape is
|
||
# the shape this whole issue is about, a config that renders
|
||
# and deploys perfectly while adding no data.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { } || cfg.publishedScrapeTargets != { }) {
|
||
prometheus.config.scrape_configs =
|
||
lib.mapAttrsToList (job: target: {
|
||
job_name = job;
|
||
static_configs = [ { targets = [ target ]; } ];
|
||
}) cfg.scrapeTargets
|
||
# ⚠️ `++`, so the two kinds of target land in ONE list —
|
||
# which is exactly why a job name may not appear in both
|
||
# options. A list concatenation does not resolve a
|
||
# collision the way an attrset would: both entries ship
|
||
# under the same `job_name`. The assertion above is what
|
||
# stands between that and a deploy.
|
||
++ lib.mapAttrsToList (
|
||
job: url:
|
||
let
|
||
parts = parsePublished url;
|
||
in
|
||
{
|
||
job_name = job;
|
||
# Split from the declared url — see `parsePublished`.
|
||
scheme = "https";
|
||
static_configs = [ { targets = [ (lib.elemAt parts 0) ]; } ];
|
||
metrics_path = lib.elemAt parts 1;
|
||
# Prometheus-native oauth2, not the collector's
|
||
# `oauth2client` extension: the prometheus receiver
|
||
# takes upstream's scrape config verbatim and does not
|
||
# accept an `auth` block naming a collector extension.
|
||
oauth2 = {
|
||
client_id = cfg.clientId;
|
||
# A path, never a value — nothing here may read the
|
||
# secret, or it lands in the store world-readable.
|
||
client_secret_file = collectorSecretPath;
|
||
token_url = "${autheliaCfg.url}/api/oidc/token";
|
||
# The audience is the target's own url, and authelia
|
||
# checks it against the address being requested.
|
||
# Registered ≠ requested: a client that does not ASK
|
||
# for an audience gets a token with `aud: []` however
|
||
# complete its registration looks.
|
||
endpoint_params.audience = url;
|
||
};
|
||
}
|
||
) cfg.publishedScrapeTargets;
|
||
};
|
||
|
||
exporters =
|
||
lib.optionalAttrs vmCfg.enable {
|
||
# `metrics_endpoint`, NOT `endpoint`: the latter is a
|
||
# base that otlphttp appends `/v1/metrics` to, while
|
||
# VictoriaMetrics serves OTLP at
|
||
# `/opentelemetry/api/v1/push`. With `endpoint` the
|
||
# collector answers 200 to its own clients and posts the
|
||
# samples to a path that does not exist. Measured
|
||
# end-to-end, not read — `state/probe-3265-collector-to-vm.sh`.
|
||
"otlphttp/victoriametrics".metrics_endpoint =
|
||
"http://127.0.0.1:${toString vmCfg.port}/opentelemetry/api/v1/push";
|
||
}
|
||
// lib.optionalAttrs (otelCfg.endpoint != "") {
|
||
${if otelCfg.protocol == "grpc" then "otlp" else "otlphttp"} = {
|
||
endpoint = otelCfg.endpoint;
|
||
}
|
||
// lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
# Interpolated by the collector from its environment at
|
||
# runtime, never by nix: `EnvironmentFile` below is what
|
||
# puts it there, so the value is not read into the store.
|
||
headers.${otelCfg.collector.upstreamHeaderName} = "\${env:${otelCfg.collector.upstreamHeaderName}}";
|
||
}
|
||
// lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; };
|
||
};
|
||
|
||
# Moves this collector's self-metrics off the built-in
|
||
# default of `localhost:8888`, which the hive tier holds.
|
||
#
|
||
# ⚠️ `metrics.address` is the spelling that looks right and
|
||
# is REJECTED by this collector version — measured, not
|
||
# read: `'migration.MetricsConfigV030' has invalid keys:
|
||
# address`. `readers` is the schema it accepts, and the
|
||
# difference is a startup failure rather than a warning.
|
||
service.telemetry.metrics.readers = [
|
||
{
|
||
pull.exporter.prometheus = {
|
||
host = "127.0.0.1";
|
||
port = cfg.telemetryPort;
|
||
};
|
||
}
|
||
];
|
||
|
||
# ⚠️ An extension that is configured but not listed here is
|
||
# INERT — the collector starts clean and the receiver
|
||
# naming it authenticates nothing. Derived from the same
|
||
# attrset as the receivers so the two cannot disagree.
|
||
service.extensions = map (h: "oidc/${h}") (lib.attrNames hivePorts);
|
||
|
||
# Fan-out, not a choice: with both configured the same
|
||
# samples go upstream AND into the swarm's store. The store
|
||
# is for looking at this swarm; the upstream is for whoever
|
||
# aggregates across swarms, and neither replaces the other.
|
||
# `exporterNames` is shared by every pipeline — where a
|
||
# sample goes is a property of this tier, not of the hive
|
||
# that sent it.
|
||
service.pipelines =
|
||
lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "metrics/${h}" {
|
||
receivers = [ "otlp/${h}" ];
|
||
processors = [ "resource/${h}" ];
|
||
exporters = exporterNames;
|
||
}
|
||
) hivePorts
|
||
# Its OWN pipeline, and that separation is the ruling, not a
|
||
# tidiness choice: every `resource/<hive>` above UPSERTS a
|
||
# `hive` key, so a scraped swarm sample routed through any of
|
||
# them would acquire the one label a swarm-level service must
|
||
# not have. Keeping it out of them makes the absence
|
||
# structural rather than something to remember to strip.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
|
||
"metrics/${swarmTierName}" = {
|
||
receivers = [ "prometheus" ];
|
||
processors = [ "resource/${swarmTierName}" ];
|
||
exporters = exporterNames;
|
||
};
|
||
};
|
||
}
|
||
// {
|
||
extensions = lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "oidc/${h}" {
|
||
issuer_url = autheliaCfg.url;
|
||
# The audience this hive's client is registered to
|
||
# request, and the reason one hive's token is refused by
|
||
# another hive's receiver. Same expression authelia
|
||
# registers it under — a second spelling here would deny
|
||
# every hive, as a 401 that blames the token.
|
||
audience = "${autheliaCfg.hiveClientPrefix}${h}";
|
||
# ⛔ DO NOT ADD `issuer_ca_path` HERE. It took this
|
||
# collector down for forty minutes once, and the failure
|
||
# is invisible to every check we have.
|
||
#
|
||
# It loads only the FIRST certificate in the file it
|
||
# names. The bundle assembled for this container is
|
||
# `system CAs ++ hive trust bundle`, so the anchor sits
|
||
# ~123rd and is never in the pool: the extension then
|
||
# cannot verify authelia and the whole collector exits
|
||
# `x509: certificate signed by unknown authority`, on
|
||
# every start, with 125 valid certificates in the file.
|
||
#
|
||
# Leaving it unset makes the extension use the process
|
||
# trust store, which `trustBundle` already populates via
|
||
# `SSL_CERT_FILE` — and *that* consumer reads every
|
||
# certificate regardless of order. One file, two
|
||
# consumers, opposite parsing: the fix is to stop naming
|
||
# it twice, not to reorder the bundle.
|
||
#
|
||
# ⚠️ Nor is pointing it at the hive trust bundle a fix:
|
||
# `hive-tls.nix` writes that leading with the *hive* CA
|
||
# (`nameConstraints` = this hive's domain), which cannot
|
||
# issue a swarm-level name at all.
|
||
#
|
||
# (Kept from the original note, still true and still
|
||
# worth not re-deriving: `issuer_ca_file`, `ca_file` and
|
||
# `tls.ca_file` are INVALID KEYS for this extension.)
|
||
}
|
||
) hivePorts;
|
||
|
||
# `upsert`, not `insert`: a sender that stamps its own
|
||
# `hive` must be OVERWRITTEN, not deferred to. This
|
||
# processor is the whole attribution boundary — the value
|
||
# is a constant per receiver, so it says which hive
|
||
# authenticated, not which hive claimed to be sending.
|
||
processors =
|
||
lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "resource/${h}" {
|
||
attributes = [
|
||
{
|
||
key = "hive";
|
||
value = h;
|
||
action = "upsert";
|
||
}
|
||
# Stamped here rather than on a separate upstream-only
|
||
# pipeline, which would double the pipeline count to
|
||
# withhold one constant label from the local store. It
|
||
# is redundant there — one VictoriaMetrics per swarm, so
|
||
# every series in it already belongs to this swarm — but
|
||
# a constant label multiplies no series, and it means
|
||
# what LEAVES and what STAYS have the same shape.
|
||
#
|
||
# Upstream is where it stops being redundant: that is the
|
||
# one hop where several swarms can land in one store, and
|
||
# samples that cannot name their swarm collide there
|
||
# exactly as hives collided here before per-hive
|
||
# receivers existed.
|
||
{
|
||
key = "swarm";
|
||
value = swarmDisplayName;
|
||
action = "upsert";
|
||
}
|
||
];
|
||
}
|
||
) hivePorts
|
||
# The swarm tier's own stamp: `swarm` and deliberately NO
|
||
# `hive`. A scraped swarm service belongs to the swarm, not to
|
||
# any one hive, so there is no honest value to put there — and
|
||
# an invented one (a sentinel, the local hive's name) would be
|
||
# queried as though it meant something.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
|
||
"resource/${swarmTierName}".attributes = [
|
||
{
|
||
# The metric LABEL, a different namespace from the
|
||
# component name above — deliberately not interpolated.
|
||
key = "swarm";
|
||
value = swarmDisplayName;
|
||
action = "upsert";
|
||
}
|
||
];
|
||
};
|
||
};
|
||
};
|
||
|
||
# The credential file is already `NAME=value`, systemd's
|
||
# EnvironmentFile format — so the secret reaches the process as an
|
||
# environment variable without being read by nix, written to the
|
||
# store, or passed in argv.
|
||
systemd.services.opentelemetry-collector.serviceConfig =
|
||
lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
EnvironmentFile = otelCfg.headersCredential;
|
||
}
|
||
# The other credential, and the other direction: the one above
|
||
# authenticates this collector's export onward, this one
|
||
# authenticates it to a service it scrapes.
|
||
#
|
||
# ⚠️ Only where a published target exists. `LoadCredential` on a
|
||
# missing source is a unit that refuses to start, so declaring it
|
||
# unconditionally would take the collector down on every hive that
|
||
# scrapes nothing published — the empty case is the shipped one.
|
||
// lib.optionalAttrs (cfg.publishedScrapeTargets != { }) {
|
||
LoadCredential = [ "${collectorCredentialId}:${collectorSecretInContainer}" ];
|
||
};
|
||
};
|
||
};
|
||
};
|
||
}
|