ff5c76c42f
Nothing in this deployment read a Prometheus endpoint, so enabling /metrics on a managed service added an endpoint and no data. The swarm collector receives OTLP pushes and does not scrape; VictoriaMetrics stores what is pushed and has no scrape config. The pipeline was entirely push-based and every such service is pull-based. Adds a prometheus receiver, a resource/swarm processor and a metrics/swarm pipeline alongside the per-hive ones. The pipeline is separate because that is the ruling, not for tidiness: every resource/<hive> processor UPSERTS a hive key, so a scraped swarm sample routed through one would acquire the single label a swarm-level service must not have. Keeping it out makes the absence structural rather than something to remember to strip, the same way hive stays a property of which authenticated receiver accepted a push. Targets come from an option each service fills in from its own module, under its own enable, rather than a list assembled here. That is what puts the scraper and the target on the same host by construction: an entry exists only where the service that named it runs. Co-location is true of the all-local deployment and is not a guarantee, and that is exactly the case where assuming it is invisible. All three additions MERGE with the per-hive attrsets rather than replacing them. A plain assignment would drop every hive's receiver, processor and pipeline and still render a config the collector starts cleanly on. Empty target set emits no receiver, no processor and no pipeline — an enabled scraper with nothing to scrape is the inert configuration this issue is about, and the target set ships empty here because the targets themselves are separate issues. Formatting verified with nix fmt. The evaluation gate is not written yet; nothing here has been evaluated against a fixture.
628 lines
30 KiB
Nix
628 lines
30 KiB
Nix
# The swarm's telemetry collector: one per swarm, in a `swarm-otel`
|
||
# nixos-container beside the swarm's other shared services.
|
||
#
|
||
# Two tiers, and they are separate on purpose:
|
||
#
|
||
# - `otel.nix` is the **hive** tier. It receives from this hive's agents
|
||
# on the bridge and forwards, and it holds no upstream credential.
|
||
# - this is the **swarm** tier. It is the only holder of the upstream
|
||
# credential, the only writer to the swarm's metrics store, and the
|
||
# place that will stamp `hive=` from the authenticated connection
|
||
# rather than from anything a sender can choose.
|
||
#
|
||
# On a host that runs both, both processes run. They are not collapsed:
|
||
# all-local is a statement about *where* processes run, not about what
|
||
# shape the deployment has, and a local tier boundary that disappears is
|
||
# one the local deployment stops testing. `hive=` attribution is the
|
||
# property that would differ, and the ingest auth that makes it
|
||
# unforgeable is built on this boundary existing.
|
||
#
|
||
# A container rather than a second host unit, for the same reason every
|
||
# sibling swarm service is one — and because `services.opentelemetry-collector`
|
||
# is a singleton NixOS option, already spoken for on the host by the hive
|
||
# tier. A container gets its own evaluation and therefore its own collector.
|
||
{
|
||
pkgs,
|
||
lib,
|
||
config,
|
||
...
|
||
}:
|
||
let
|
||
cfg = config.services.hyperhive.swarm.otel;
|
||
swarmCfg = config.services.hyperhive.swarm;
|
||
otelCfg = config.services.hyperhive.otel;
|
||
vmCfg = config.services.hyperhive.swarm.victoriametrics;
|
||
hyperhiveCfg = config.services.hyperhive;
|
||
gatewayCfg = hyperhiveCfg.gateway;
|
||
swarmDomain = hyperhiveCfg.swarm.domain;
|
||
|
||
# Total on a null swarm domain for the same reason every sibling module is:
|
||
# the required-domain assertion in hive-network.nix should be what an
|
||
# operator sees, not a coercion error from here.
|
||
domainBase = if swarmDomain == null then "invalid" else swarmDomain;
|
||
|
||
autheliaCfg = hyperhiveCfg.swarm.authelia;
|
||
|
||
# `attrNames` is sorted, so this is a function of the hive SET and not of
|
||
# the order anyone wrote it in.
|
||
#
|
||
# These ports are internal and appear in no URL: a hive addresses its own
|
||
# receiver as a PATH on this collector's single gateway name, and nginx —
|
||
# rendered from this same evaluation — is the only thing that ever names
|
||
# the port. That is what makes deriving them safe here and unsafe in the
|
||
# obvious other place: were a hive told a port, inserting a hive would
|
||
# renumber the ones after it and silently move a port a running hive was
|
||
# already sending to.
|
||
hivePorts = lib.listToAttrs (
|
||
lib.imap0 (i: h: lib.nameValuePair h (cfg.port + i)) (lib.attrNames hyperhiveCfg.swarm.hives)
|
||
);
|
||
|
||
# The swarm's authelia is reached by its gateway name, whose leaf is
|
||
# issued by the swarm services sub-CA — so this container needs the same
|
||
# runtime CA trust every other consumer of a swarm-service name needs.
|
||
# The CA is generated at runtime and cannot be baked into a derivation,
|
||
# which is why it arrives as a bind mount rather than
|
||
# `security.pki.certificateFiles`.
|
||
caTrust = import ./lib/hive-ca-trust.nix {
|
||
inherit lib;
|
||
tlsCfg = hyperhiveCfg.tls;
|
||
inherit gatewayCfg;
|
||
};
|
||
|
||
# `unknown` rather than omitting the label, copying `agent-modules/otel.nix`
|
||
# deliberately: a producer that cannot name its swarm should say so in the
|
||
# same vocabulary as every other producer, so a query never has to handle
|
||
# both "the label is absent" and "the label says unknown".
|
||
swarmDisplayName = if hyperhiveCfg.swarm.name == null then "unknown" else hyperhiveCfg.swarm.name;
|
||
|
||
# One list, read by every pipeline: the per-hive pipelines fan out to
|
||
# exactly the same destinations as the single pipeline they replace.
|
||
# Written once because "which exporters" is a property of this tier, not
|
||
# of which hive a sample came from.
|
||
exporterNames =
|
||
lib.optional (otelCfg.endpoint != "") (if otelCfg.protocol == "grpc" then "otlp" else "otlphttp")
|
||
++ lib.optional vmCfg.enable "otlphttp/victoriametrics";
|
||
in
|
||
{
|
||
options.services.hyperhive.swarm.otel = {
|
||
enable = lib.mkOption {
|
||
type = lib.types.bool;
|
||
default = false;
|
||
description = ''
|
||
Run the swarm's telemetry collector on this host.
|
||
|
||
Asserted from `swarm.enableRequiredServices` in
|
||
./swarm-required-services.nix, with the metrics pair this
|
||
collector feeds: a swarm has one of these, and it belongs
|
||
wherever the shared services live rather than on every hive.
|
||
|
||
A hive that does not run it still runs its own hive-tier collector
|
||
(`services.hyperhive.otel.enable`) and reaches this one by name, at
|
||
{option}`services.hyperhive.swarm.otel.domain`.
|
||
'';
|
||
};
|
||
|
||
machine = lib.mkOption {
|
||
type = lib.types.str;
|
||
readOnly = true;
|
||
default = "swarm-otel";
|
||
description = ''
|
||
Name of the nixos-container this collector runs in — also the
|
||
`machinectl` name, so other modules may read it rather than
|
||
repeating the literal.
|
||
'';
|
||
};
|
||
|
||
port = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 4319;
|
||
description = ''
|
||
First port of this collector's receiver range. Every hive in
|
||
{option}`services.hyperhive.swarm.hives` gets its **own**
|
||
authenticated receiver — that is what makes the `hive` label
|
||
unforgeable — so the range is one port per hive, starting here, in
|
||
sorted-name order.
|
||
|
||
⚠️ Internal. No client is ever told a port: a hive reaches its own
|
||
receiver as `https://''${domain}/<hive>`, and the gateway routes on
|
||
that path. So adding a hive, which renumbers the ones after it, is
|
||
harmless — nginx is rendered from this same evaluation and moves
|
||
with it.
|
||
|
||
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
|
||
hive tier already uses it (`services.hyperhive.otel.collector.port`)
|
||
and every swarm container shares the host's network namespace. Two
|
||
listeners claiming one port on one host is not a build failure —
|
||
it is a runtime coin toss over which one gets it, with nothing in
|
||
any log saying so. The same collision cost a release when grafana
|
||
and the forge both defaulted to 3000. The assertions below check
|
||
the whole derived range against every port this module and the hive
|
||
tier declare, which is as far as a module can see.
|
||
'';
|
||
};
|
||
|
||
telemetryPort = lib.mkOption {
|
||
type = lib.types.port;
|
||
default = 8889;
|
||
description = ''
|
||
Port this collector serves its **own** metrics on — queue depth,
|
||
refused and dropped samples, exporter failures. How you find out
|
||
that telemetry is being lost, so it is worth keeping rather than
|
||
switching off.
|
||
|
||
⚠️ **Deliberately not 8889's neighbour 8888**, which is the
|
||
collector's built-in default and therefore what the hive tier
|
||
already binds. Two collectors share a network namespace whenever
|
||
they are co-located, and unlike the OTLP port this one appears
|
||
nowhere in either config — it is a default inside the binary, so
|
||
nothing that compares configured ports can see the clash. The
|
||
second collector to start simply dies with
|
||
`bind: address already in use`.
|
||
'';
|
||
};
|
||
|
||
domain = lib.mkOption {
|
||
type = lib.types.str;
|
||
default = "otel.${domainBase}";
|
||
defaultText = lib.literalExpression ''"otel.''${services.hyperhive.swarm.domain}"'';
|
||
description = ''
|
||
Name the gateway serves this on. A sibling of the swarm's other
|
||
service names, so the swarm-services sub-CA can issue for it — see
|
||
`hive-tls.nix` for why a service name being a sibling rather than a
|
||
child decides which CA may sign it.
|
||
|
||
This is what the **hive** tier's exporter reaches — the hive
|
||
collector is a plain producer against this name exactly like every
|
||
other client of a swarm service, resolved locally by dnsmasq on a
|
||
co-located host and over the real network otherwise. There is no
|
||
separate loopback-vs-remote knob to get wrong: `swarm-nats` is the
|
||
deliberate exception to this pattern (its cross-hive reach is the
|
||
wireguard mesh, not the gateway), everything else in this swarm
|
||
addresses its siblings by name.
|
||
'';
|
||
};
|
||
|
||
scrapeTargets = lib.mkOption {
|
||
type = lib.types.attrsOf lib.types.str;
|
||
default = { };
|
||
example = lib.literalExpression ''{ forgejo = "127.0.0.1:3000"; }'';
|
||
description = ''
|
||
Prometheus exposition endpoints this collector scrapes, as
|
||
`<job name> = "<host>:<port>"`.
|
||
|
||
**A service declares its own entry, from its own module, under its
|
||
own `enable`.** That is what puts the scraper and the target on the
|
||
same host by construction rather than by luck: an entry exists only
|
||
where the service that named it runs. Do not assemble the list here.
|
||
Every swarm service being co-located is a property of the all-local
|
||
deployment, not a guarantee — and that is precisely the case where
|
||
the difference is invisible until a swarm splits across hosts.
|
||
|
||
Samples land in a swarm-level pipeline that stamps `swarm` and
|
||
**never** `hive`: a swarm service does not belong to a hive, and
|
||
`hive` stays a property of which authenticated receiver accepted a
|
||
push, not something a scrape can acquire.
|
||
|
||
Empty by default, in which case no scrape receiver, processor or
|
||
pipeline is emitted at all — an enabled scraper with nothing to
|
||
scrape is the inert configuration this option exists to avoid.
|
||
'';
|
||
};
|
||
};
|
||
|
||
config = lib.mkIf (config.services.hyperhive.enable && cfg.enable) {
|
||
# The gateway name, inside `cfg.enable` — that guard is the load-bearing
|
||
# part. Every hive in a swarm may know this collector exists, but only
|
||
# the host that RUNS it may claim the name; a client hive declaring the
|
||
# vhost would answer for a service it does not have.
|
||
services.hyperhive.gateway.localNames = [ cfg.domain ];
|
||
|
||
# OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
|
||
# every sibling swarm service: TLS terminates here, then plain http to
|
||
# the co-located container over loopback (shared netns, like the store
|
||
# this collector writes to).
|
||
services.nginx.virtualHosts."${cfg.domain}" = (gatewayCfg.lib.tlsFor cfg.domain) // {
|
||
listen = gatewayCfg.lib.listen;
|
||
extraConfig = gatewayCfg.lib.securityHeaders;
|
||
locations =
|
||
# One name for the whole collector, and the hive is a path under
|
||
# it. The alternative — a vhost per hive — needs a certificate,
|
||
# a DNS name and a `localNames` entry per hive to express the
|
||
# same routing the gateway already does for free.
|
||
#
|
||
# ⚠️ The trailing slash on both sides is load-bearing: it is what
|
||
# strips `/<hive>` before the request reaches the receiver, which
|
||
# serves `/v1/metrics` and knows nothing about hives. Without it
|
||
# the receiver sees `/<hive>/v1/metrics` and answers 404 to a
|
||
# request that authenticated perfectly.
|
||
lib.mapAttrs' (
|
||
h: p: lib.nameValuePair "/${h}/" { proxyPass = "http://127.0.0.1:${toString p}/"; }
|
||
) hivePorts
|
||
// {
|
||
# There is no swarm-wide inbox, and a closed door is the honest
|
||
# description of that. Every route into this collector belongs to
|
||
# exactly one hive.
|
||
"/".return = "404";
|
||
};
|
||
};
|
||
|
||
# Turning on the identities this tier authenticates against, which the
|
||
# option exists to allow: its own description names this module as the
|
||
# second consumer, so the queue is not a prerequisite for authenticated
|
||
# telemetry.
|
||
services.hyperhive.swarm.authelia.oidc.hiveIdentities = true;
|
||
|
||
# The CA bind source is written at runtime by a host unit, so the
|
||
# container has to start after it — otherwise nspawn sets up a mount
|
||
# over a file that does not exist yet.
|
||
systemd.services."container@${cfg.machine}" = caTrust.containerOrdering;
|
||
|
||
assertions = [
|
||
{
|
||
# The tier exists to hold the upstream credential and to write the
|
||
# swarm's store. With neither, it is a process that receives
|
||
# samples and drops them — which looks healthy and loses data.
|
||
assertion = otelCfg.endpoint != "" || vmCfg.enable;
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but this collector
|
||
has nowhere to send what it receives:
|
||
services.hyperhive.otel.endpoint is empty and
|
||
services.hyperhive.swarm.victoriametrics.enable is false.
|
||
|
||
Set the endpoint to export upstream, or enable the swarm's
|
||
metrics store.
|
||
'';
|
||
}
|
||
{
|
||
# Without a roster there are no receivers at all, so this
|
||
# collector would listen on nothing while looking configured.
|
||
assertion = hyperhiveCfg.swarm.hives != { };
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but
|
||
services.hyperhive.swarm.hives is empty: ingest is authenticated
|
||
per hive, so an empty roster means this collector accepts nothing
|
||
from anyone.
|
||
|
||
List the swarm's hives.
|
||
'';
|
||
}
|
||
{
|
||
# A hive proves who it is with a token this provider mints, so
|
||
# there is no version of this collector that runs without one.
|
||
# Stated as an assertion rather than a fallback because a guessed
|
||
# issuer URL evaluates cleanly and refuses every hive at runtime.
|
||
assertion = autheliaCfg.url != null;
|
||
message = ''
|
||
services.hyperhive.swarm.otel.enable is true but
|
||
services.hyperhive.swarm.authelia.url is null: every hive
|
||
authenticates to this collector as itself, and the token comes
|
||
from the swarm's identity provider.
|
||
|
||
Point authelia.url at the swarm's provider, or enable
|
||
services.hyperhive.swarm.authelia on the host that runs it.
|
||
'';
|
||
}
|
||
{
|
||
# A port collision between two listeners on one host is a runtime
|
||
# coin toss with nothing in any log — the failure this whole
|
||
# comment budget exists to prevent. Checked against every port
|
||
# reachable from here; a port some other module picks is not.
|
||
#
|
||
# ⚠️ `cfg.port` is deliberately absent from `others`: it is the
|
||
# FIRST element of the derived range, so listing it would make this
|
||
# assertion fire on every config.
|
||
assertion =
|
||
let
|
||
derived = lib.attrValues hivePorts;
|
||
others = [
|
||
cfg.telemetryPort
|
||
otelCfg.collector.port
|
||
]
|
||
++ lib.optional vmCfg.enable vmCfg.port;
|
||
all = derived ++ others;
|
||
in
|
||
lib.length (lib.unique all) == lib.length all;
|
||
message = ''
|
||
services.hyperhive.swarm.otel: the receiver range starting at
|
||
port (${toString cfg.port}, one port per hive in
|
||
services.hyperhive.swarm.hives) overlaps another port on this
|
||
host.
|
||
|
||
Every swarm container shares the host's network namespace, so
|
||
two listeners claiming one port is not a build failure — it is
|
||
whichever process started first, silently. Move
|
||
services.hyperhive.swarm.otel.port to a free range.
|
||
'';
|
||
}
|
||
];
|
||
|
||
containers.${cfg.machine} = {
|
||
autoStart = true;
|
||
ephemeral = false;
|
||
# Shared host netns, like every sibling swarm service: the hive tier
|
||
# reaches this collector, and this collector reaches the metrics
|
||
# store, without either crossing a network boundary that would need
|
||
# its own trust material.
|
||
privateNetwork = false;
|
||
|
||
# The upstream credential is operator-provided and lives on the host.
|
||
# Read-only, and only when one is configured — binding a path that
|
||
# does not exist makes nixos-container refuse to start the container,
|
||
# which is a stall several layers from its cause.
|
||
bindMounts =
|
||
lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
${otelCfg.headersCredential} = {
|
||
hostPath = otelCfg.headersCredential;
|
||
isReadOnly = true;
|
||
};
|
||
}
|
||
# The public hive CA, read-only — only when something in here
|
||
# actually verifies a swarm-service name.
|
||
// caTrust.bindMount;
|
||
|
||
config =
|
||
{ ... }:
|
||
{
|
||
# This tier is the one that resolves an operator-configured
|
||
# hostname: `otel.endpoint` is an external URL, and reaching it
|
||
# is the entire reason this container holds a credential. The
|
||
# `/etc/resolv.conf` nixos-containers copies in is a snapshot
|
||
# taken once at boot, so without this the upstream export
|
||
# depends on the host's file having been right at that instant.
|
||
imports = [
|
||
(import ./swarm-container-resolver.nix {
|
||
inherit (config.services.hyperhive.network) bridgeIp;
|
||
dnsConsumers = [ "opentelemetry-collector.service" ];
|
||
})
|
||
]
|
||
# `SSL_CERT_FILE` REPLACES the trust store rather than adding to
|
||
# it, so a failed assembly yields an empty pool and every TLS
|
||
# call fails while the unit looks healthy. That is why this is
|
||
# the shared helper — it carries the `Requires` and the
|
||
# non-empty check — and not a local `cat`.
|
||
++ [
|
||
(caTrust.trustBundle {
|
||
inherit pkgs;
|
||
name = cfg.machine;
|
||
consumers = [ "opentelemetry-collector" ];
|
||
})
|
||
];
|
||
|
||
system.stateVersion = config.system.stateVersion;
|
||
networking.firewall.enable = false;
|
||
# Keep the host-copied /etc/resolv.conf intact — same reasoning
|
||
# as the sibling swarm containers.
|
||
networking.resolvconf.enable = lib.mkForce false;
|
||
|
||
services.opentelemetry-collector = {
|
||
enable = true;
|
||
package = pkgs.opentelemetry-collector-contrib;
|
||
# Runs `otelcol validate` at build time. ⚠️ A parser, not a
|
||
# wiring check: it accepts a receiver naming an absent
|
||
# extension, and the collector then dies at startup. A green
|
||
# build does not prove this config starts, never mind that a
|
||
# sample arrives — which is why this module's gate pushes a
|
||
# real sample through both tiers into the store.
|
||
validateConfigFile = true;
|
||
settings = {
|
||
# One receiver per hive, and that multiplicity is forced
|
||
# rather than chosen. The `hive`
|
||
# label has to come from something the sender cannot write,
|
||
# and the only such thing here is WHICH RECEIVER accepted
|
||
# the sample: a processor cannot read the token's claims
|
||
# (`from_context` reads request metadata, and asking it for
|
||
# an auth claim yields nothing — silently, with a healthy
|
||
# startup), and one receiver holding many credentials never
|
||
# reveals which one matched.
|
||
receivers =
|
||
lib.mapAttrs' (
|
||
h: p:
|
||
lib.nameValuePair "otlp/${h}" {
|
||
protocols.http = {
|
||
endpoint = "127.0.0.1:${toString p}";
|
||
auth.authenticator = "oidc/${h}";
|
||
};
|
||
}
|
||
) hivePorts
|
||
# MERGED with the per-hive receivers, never assigned over
|
||
# them. A plain assignment here would drop every hive's
|
||
# receiver and still render a valid config that starts
|
||
# cleanly — the collector has no opinion about how many
|
||
# pipelines it was supposed to have.
|
||
#
|
||
# Only emitted when a service has actually declared a
|
||
# target: a `prometheus` receiver with nothing to scrape is
|
||
# the shape this whole issue is about, a config that renders
|
||
# and deploys perfectly while adding no data.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
|
||
prometheus.config.scrape_configs = lib.mapAttrsToList (job: target: {
|
||
job_name = job;
|
||
static_configs = [ { targets = [ target ]; } ];
|
||
}) cfg.scrapeTargets;
|
||
};
|
||
|
||
exporters =
|
||
lib.optionalAttrs vmCfg.enable {
|
||
# `metrics_endpoint`, NOT `endpoint`: the latter is a
|
||
# base that otlphttp appends `/v1/metrics` to, while
|
||
# VictoriaMetrics serves OTLP at
|
||
# `/opentelemetry/api/v1/push`. With `endpoint` the
|
||
# collector answers 200 to its own clients and posts the
|
||
# samples to a path that does not exist. Measured
|
||
# end-to-end, not read — `state/probe-3265-collector-to-vm.sh`.
|
||
"otlphttp/victoriametrics".metrics_endpoint =
|
||
"http://127.0.0.1:${toString vmCfg.port}/opentelemetry/api/v1/push";
|
||
}
|
||
// lib.optionalAttrs (otelCfg.endpoint != "") {
|
||
${if otelCfg.protocol == "grpc" then "otlp" else "otlphttp"} = {
|
||
endpoint = otelCfg.endpoint;
|
||
}
|
||
// lib.optionalAttrs (otelCfg.headersCredential != null) {
|
||
# Interpolated by the collector from its environment at
|
||
# runtime, never by nix: `EnvironmentFile` below is what
|
||
# puts it there, so the value is not read into the store.
|
||
headers.${otelCfg.collector.upstreamHeaderName} = "\${env:${otelCfg.collector.upstreamHeaderName}}";
|
||
}
|
||
// lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; };
|
||
};
|
||
|
||
# Moves this collector's self-metrics off the built-in
|
||
# default of `localhost:8888`, which the hive tier holds.
|
||
#
|
||
# ⚠️ `metrics.address` is the spelling that looks right and
|
||
# is REJECTED by this collector version — measured, not
|
||
# read: `'migration.MetricsConfigV030' has invalid keys:
|
||
# address`. `readers` is the schema it accepts, and the
|
||
# difference is a startup failure rather than a warning.
|
||
service.telemetry.metrics.readers = [
|
||
{
|
||
pull.exporter.prometheus = {
|
||
host = "127.0.0.1";
|
||
port = cfg.telemetryPort;
|
||
};
|
||
}
|
||
];
|
||
|
||
# ⚠️ An extension that is configured but not listed here is
|
||
# INERT — the collector starts clean and the receiver
|
||
# naming it authenticates nothing. Derived from the same
|
||
# attrset as the receivers so the two cannot disagree.
|
||
service.extensions = map (h: "oidc/${h}") (lib.attrNames hivePorts);
|
||
|
||
# Fan-out, not a choice: with both configured the same
|
||
# samples go upstream AND into the swarm's store. The store
|
||
# is for looking at this swarm; the upstream is for whoever
|
||
# aggregates across swarms, and neither replaces the other.
|
||
# `exporterNames` is shared by every pipeline — where a
|
||
# sample goes is a property of this tier, not of the hive
|
||
# that sent it.
|
||
service.pipelines =
|
||
lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "metrics/${h}" {
|
||
receivers = [ "otlp/${h}" ];
|
||
processors = [ "resource/${h}" ];
|
||
exporters = exporterNames;
|
||
}
|
||
) hivePorts
|
||
# Its OWN pipeline, and that separation is the ruling, not a
|
||
# tidiness choice: every `resource/<hive>` above UPSERTS a
|
||
# `hive` key, so a scraped swarm sample routed through any of
|
||
# them would acquire the one label a swarm-level service must
|
||
# not have. Keeping it out of them makes the absence
|
||
# structural rather than something to remember to strip.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
|
||
"metrics/swarm" = {
|
||
receivers = [ "prometheus" ];
|
||
processors = [ "resource/swarm" ];
|
||
exporters = exporterNames;
|
||
};
|
||
};
|
||
}
|
||
// {
|
||
extensions = lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "oidc/${h}" {
|
||
issuer_url = autheliaCfg.url;
|
||
# The audience this hive's client is registered to
|
||
# request, and the reason one hive's token is refused by
|
||
# another hive's receiver. Same expression authelia
|
||
# registers it under — a second spelling here would deny
|
||
# every hive, as a 401 that blames the token.
|
||
audience = "${autheliaCfg.hiveClientPrefix}${h}";
|
||
# ⛔ DO NOT ADD `issuer_ca_path` HERE. It took this
|
||
# collector down for forty minutes once, and the failure
|
||
# is invisible to every check we have.
|
||
#
|
||
# It loads only the FIRST certificate in the file it
|
||
# names. The bundle assembled for this container is
|
||
# `system CAs ++ hive trust bundle`, so the anchor sits
|
||
# ~123rd and is never in the pool: the extension then
|
||
# cannot verify authelia and the whole collector exits
|
||
# `x509: certificate signed by unknown authority`, on
|
||
# every start, with 125 valid certificates in the file.
|
||
#
|
||
# Leaving it unset makes the extension use the process
|
||
# trust store, which `trustBundle` already populates via
|
||
# `SSL_CERT_FILE` — and *that* consumer reads every
|
||
# certificate regardless of order. One file, two
|
||
# consumers, opposite parsing: the fix is to stop naming
|
||
# it twice, not to reorder the bundle.
|
||
#
|
||
# ⚠️ Nor is pointing it at the hive trust bundle a fix:
|
||
# `hive-tls.nix` writes that leading with the *hive* CA
|
||
# (`nameConstraints` = this hive's domain), which cannot
|
||
# issue a swarm-level name at all.
|
||
#
|
||
# (Kept from the original note, still true and still
|
||
# worth not re-deriving: `issuer_ca_file`, `ca_file` and
|
||
# `tls.ca_file` are INVALID KEYS for this extension.)
|
||
}
|
||
) hivePorts;
|
||
|
||
# `upsert`, not `insert`: a sender that stamps its own
|
||
# `hive` must be OVERWRITTEN, not deferred to. This
|
||
# processor is the whole attribution boundary — the value
|
||
# is a constant per receiver, so it says which hive
|
||
# authenticated, not which hive claimed to be sending.
|
||
processors =
|
||
lib.mapAttrs' (
|
||
h: _:
|
||
lib.nameValuePair "resource/${h}" {
|
||
attributes = [
|
||
{
|
||
key = "hive";
|
||
value = h;
|
||
action = "upsert";
|
||
}
|
||
# Stamped here rather than on a separate upstream-only
|
||
# pipeline, which would double the pipeline count to
|
||
# withhold one constant label from the local store. It
|
||
# is redundant there — one VictoriaMetrics per swarm, so
|
||
# every series in it already belongs to this swarm — but
|
||
# a constant label multiplies no series, and it means
|
||
# what LEAVES and what STAYS have the same shape.
|
||
#
|
||
# Upstream is where it stops being redundant: that is the
|
||
# one hop where several swarms can land in one store, and
|
||
# samples that cannot name their swarm collide there
|
||
# exactly as hives collided here before per-hive
|
||
# receivers existed.
|
||
{
|
||
key = "swarm";
|
||
value = swarmDisplayName;
|
||
action = "upsert";
|
||
}
|
||
];
|
||
}
|
||
) hivePorts
|
||
# The swarm tier's own stamp: `swarm` and deliberately NO
|
||
# `hive`. A scraped swarm service belongs to the swarm, not to
|
||
# any one hive, so there is no honest value to put there — and
|
||
# an invented one (a sentinel, the local hive's name) would be
|
||
# queried as though it meant something.
|
||
// lib.optionalAttrs (cfg.scrapeTargets != { }) {
|
||
"resource/swarm".attributes = [
|
||
{
|
||
key = "swarm";
|
||
value = swarmDisplayName;
|
||
action = "upsert";
|
||
}
|
||
];
|
||
};
|
||
};
|
||
};
|
||
|
||
# The credential file is already `NAME=value`, systemd's
|
||
# EnvironmentFile format — so the secret reaches the process as an
|
||
# environment variable without being read by nix, written to the
|
||
# store, or passed in argv.
|
||
systemd.services.opentelemetry-collector.serviceConfig =
|
||
lib.optionalAttrs (otelCfg.headersCredential != null)
|
||
{
|
||
EnvironmentFile = otelCfg.headersCredential;
|
||
};
|
||
};
|
||
};
|
||
};
|
||
}
|