vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

nix: the store's own collector scrapes its metrics listener

Browse source 
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.

What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.

Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.

The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.

Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.

The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
This commit is contained in:
atlas 2026-09-20 18:19:06 +02:00 • committed by mara
commit bb0afcd256
8 changed files with 815 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

238
nix/module-eval/bao-otel-collector.nix
View file

@ -26,7 +26,10 @@ let
deploy.bao.enable = true;
deploy.bao.extraListenAddresses = [ "10.0.0.1" ];
# Pinned, not incidental: the case counting these listeners is about the
# declared addresses, and a collector on this host would add one of its own.
# declared API addresses. ⚠️ Since the store got its own collector, this no longer keeps the METRICS
# listener out of the count — that one is a function of the store, not of
# a collector's placement — so the case below subtracts it by name rather
# than by leaving the swarm tier off.
deploy.swarm-otel.enable = false;
};
@ -72,10 +75,20 @@ let
# The scrape list prometheus is handed, not the option a service declared:
# the address, the path and the query are one string on the way in and three
# fields on the way out, and only the second shape is what gets requested.
scrapeJob =
machine: job:
lib.findFirst (c: c.job_name == job) null
machine.containers.swarm-otel.config.services.opentelemetry-collector.settings.receivers.prometheus.config.scrape_configs;
#
# ⚠️ Two of these now, aimed at two different containers, and the split is
# the point: the store's own job left the swarm collector for the
# one inside its container, so a case that looked in the old place would
# find `null` and a case that only looked in the new one would not notice
# the store still being declared to both.
scrapeJobIn =
container: machine: job:
lib.findFirst (c: c.job_name == job) null (
machine.containers.${container}.config.services.opentelemetry-collector.settings.receivers.prometheus.config.scrape_configs
or [ ]
);
scrapeJob = scrapeJobIn "swarm-otel";
baoScrapeJob = scrapeJobIn "swarm-bao";
cases = [
{
# Same gap one tier up, and it needs its own arm: this collector
@ -102,9 +115,14 @@ let
}
{
# Control for the case above: these settings are rendered per deployment,
# not constants a passing case could be indifferent to.
# not constants a passing case could be indifferent to. The metrics
# listener is excluded by name — it answers on a port of its own and is
# not one of the API addresses this case is counting.
name = "a declared extra address renders a second listener beside loopback";
ok = builtins.length (builtins.attrNames (baoSettings baoTwoAddresses).listener) == 2;
ok =
builtins.length (
builtins.filter (n: n != "metrics") (builtins.attrNames (baoSettings baoTwoAddresses).listener)
) == 2;
}
{
# Retention is what serves the endpoint at all, so the listener alone
@ -119,13 +137,56 @@ let
{
# openbao serves no `/metrics` at all, so a scrape of the default path
# 404s: the store looks like a dead exporter, and every panel built on
# it renders empty rather than erroring.
# it renders empty rather than erroring. The query is the other half —
# `/v1/sys/metrics` answers JSON without it, which parses as no series
# rather than as an error.
name = "the store's scrape asks for the path openbao serves";
ok =
let
j = scrapeJob baoWithCollector "bao";
j = baoScrapeJob baoWithCollector "bao";
in
(j.metrics_path or "") == "/v1/sys/metrics" && (j.params.format or [ ]) == [ "prometheus" ];
j != null
&& (j.metrics_path or "") == "/v1/sys/metrics"
&& (j.params.format or [ ]) == [ "prometheus" ];
}
{
# 🎯 This move's deliverable, and it needs BOTH halves or it is not a move:
# the job is in the store's own container, and it is no longer declared
# to the swarm collector. Only the second half can catch a "move" that
# left the original behind — two scrapers of one endpoint, double-counted
# and neither obviously wrong.
name = "the store's metrics are scraped by its own collector, not the swarm's";
ok = baoScrapeJob baoWithCollector "bao" != null && scrapeJob baoWithCollector "bao" == null;
}
{
# The scrape target is the loopback listener, not the API one: dialing
# the API listener would need a client certificate the collector has no
# way to present, and the fix for that would be relaxing
# `tls_require_and_verify_client_cert` — which is the trade this address
# exists to avoid. Pinned so a later edit has to argue with it.
name = "the store's collector scrapes loopback, never a wider address";
ok =
let
j = baoScrapeJob baoWithCollector "bao";
s = baoSettings baoWithCollector;
in
j.static_configs == [ { targets = [ "127.0.0.1:8202" ]; } ]
&& s.listener.metrics.address == "127.0.0.1:8202";
}
{
# A pipeline is typed: a prometheus receiver named in `logs` is rejected
# at startup, and metrics with no pipeline at all are collected and
# dropped. The exporter is pinned alongside because "no otel bypass"
# means both signals leave by the one hop.
name = "the store's collector has a metrics pipeline out the same hop as its logs";
ok =
let
s = (baoForwarder baoWithCollector).settings;
m = s.service.pipelines.metrics;
in
m.receivers == [ "prometheus" ]
&& m.exporters == [ "otlphttp" ]
&& m.exporters == s.service.pipelines.logs.exporters;
}
{
# Presence control for the case above: both fields are omitted rather
@ -139,14 +200,28 @@ let
j != null && !(j ? metrics_path) && !(j ? params);
}
{
# Absence arm. Unauthenticated by design, so it must not exist where
# nothing reads it.
name = "a store with no collector beside it serves no metrics";
# 🩸 This case USED to be the absence arm — "a store with no collector
# beside it serves no metrics" — and this inverts it deliberately
# rather than deleting it, because the condition it asserted is the bug.
#
# The old reasoning was sound for the old reader: the endpoint is
# unauthenticated, so it should not exist where nothing reads it, and
# the only reader was a swarm collector that had to be on this host.
# What that rendered on every OTHER host was a store whose metrics went
# nowhere, indistinguishably from a store nobody had asked to scrape.
#
# The reader now ships inside the container, so "no collector beside it"
# describes no host that exists. The endpoint is still not exposed to
# anything new: it is loopback in a netns whose only other occupants are
# the store and its collector.
name = "a store whose swarm collector is elsewhere still serves and scrapes its metrics";
ok =
let
s = baoSettings baoOtelElsewhere;
in
!(s.listener ? metrics) && !(s ? telemetry);
s.listener.metrics.address == "127.0.0.1:8202"
&& (s.telemetry.prometheus_retention_time or "0s") != "0s"
&& baoScrapeJob baoOtelElsewhere "bao" != null;
}
{
# The store's journal reaches a reader through a collector of its own,
@ -162,7 +237,11 @@ let
in
s.receivers.journald.directory == "/var/log/journal"
&& s.receivers.journald.storage == "file_storage"
&& s.service.extensions == [ "file_storage" ]
&&
s.service.extensions == [
"file_storage"
"oauth2client/swarm-bao"
]
&& p.receivers == [ "journald" ]
&& p.exporters == [ "otlphttp" ]
&& s ? exporters.otlphttp;
@ -188,8 +267,8 @@ let
vhost = baoWithCollector.services.nginx.virtualHosts.${otel.domain};
in
(baoForwarder baoWithCollector).settings.exporters.otlphttp.endpoint
== "https://${otel.domain}/${otel.producerName}"
&& vhost.locations ? "/${otel.producerName}/";
== "https://${otel.domain}/${otel.storeProducerName}"
&& vhost.locations ? "/${otel.storeProducerName}/";
}
{
# The case the old hive-tier gate got wrong: this host runs the swarm's
@ -219,7 +298,7 @@ let
ok =
let
otel = baoOtelElsewhere.services.hyperhive.swarm.otel;
route = "https://${otel.domain}/${otel.producerName}";
route = "https://${otel.domain}/${otel.storeProducerName}";
in
!(baoOtelElsewhere.services.nginx.virtualHosts ? ${otel.domain})
&& (baoForwarder baoOtelElsewhere).enable
@ -229,6 +308,129 @@ let
&& (baoForwarder baoWithHiveOtel).enable
&& (baoForwarder baoWithHiveOtel).settings.exporters.otlphttp.endpoint == route;
}
{
# 🎯 The credential half of this move, and the arm that says the hop
# DELIVERS rather than merely resolves. The receiver at the far end
# takes an OIDC token and nothing else, so an exporter with no
# authenticator is a forwarder that retries into 401s while every other
# case here passes. Both halves: the extension has to exist AND the
# exporter has to name it.
name = "the store's forwarder presents a credential on the hop it exports over";
ok =
let
s = (baoForwarder baoWithCollector).settings;
auth = "oauth2client/swarm-bao";
in
s.exporters.otlphttp.auth.authenticator == auth
&& s.extensions ? ${auth}
&& lib.elem auth s.service.extensions;
}
{
# Its OWN client, never the swarm collector's and never the
# controller's: one identity per principal, and this one runs on a host
# neither of those two does. The audience is the same id because the
# receiver checks exactly that — a token minted without ASKING for it
# carries `aud: []` and is refused with a config that reads correctly at
# both ends.
name = "the forwarder authenticates as a principal of its own";
ok =
let
bao = baoWithCollector.services.hyperhive.swarm.bao;
e = (baoForwarder baoWithCollector).settings.extensions."oauth2client/swarm-bao";
in
e.client_id == bao.otel.clientId
&& e.endpoint_params.audience == bao.otel.clientId
&& bao.otel.clientId != baoWithCollector.services.hyperhive.swarm.otel.clientId
&& bao.otel.clientId != baoWithCollector.services.hyperhive.swarm.controller.queueClientId;
}
{
# A path systemd resolves at runtime, never a value and never a store
# path: the collector runs under `DynamicUser` and opens this file
# itself, so the credential arrives through `LoadCredential` and the
# config names only the directory systemd exports.
name = "the forwarder's client secret reaches it as a credential, not a value";
ok =
let
container = baoWithCollector.containers.swarm-bao.config;
e = (baoForwarder baoWithCollector).settings.extensions."oauth2client/swarm-bao";
unit = container.systemd.services.opentelemetry-collector.serviceConfig;
in
e.client_secret_file == "\${env:CREDENTIALS_DIRECTORY}/oidc-client-secret"
&&
unit.LoadCredential == [
"oidc-client-secret:/var/lib/swarm-bao-otel-oidc/swarm-bao-collector.secret"
];
}
{
# Same 403-not-a-miss property the grafana and matrix readers are pinned
# for: the reader's grant covers the `services` prefix, so a secret
# filed under the hive that happens to run the store would be refused
# rather than missing, however correct the path reads.
name = "the forwarder's secret is read from the prefix the publisher writes";
ok =
let
s = baoWithCollector.systemd.services.swarm-bao-forwarder-oidc.script;
in
lib.hasInfix "secret/swarm/services/swarm-bao-collector/oidc/client" s
&& !(lib.hasInfix "secret/swarm/hives/" s);
}
{
# The delivery unit is ordered AFTER the store's container, not before
# it: the store it reads lives in the container it delivers into, so a
# `before` edge is a wait on a process that cannot start until this
# finishes. Pinned because the unit it was copied from does the
# opposite, and copying that line too would deadlock every cold boot.
name = "the delivery unit waits for the store rather than blocking it";
ok =
let
u = baoWithCollector.systemd.services.swarm-bao-forwarder-oidc;
in
lib.elem "container@swarm-bao.service" u.after
&& !(lib.elem "container@swarm-bao.service" (u.before or [ ]))
&& lib.elem "swarm-bao-pki.service" u.requires;
}
{
# The far end of the credential. An authenticator checks ONE audience,
# so the store's forwarder gets a receiver of its own rather than a
# second sender into the route whose audience belongs to
# `swarm-controller` — and every one of these four is silent when
# wrong: an unlisted extension is inert, a receiver in no pipeline
# drops what it accepts, and a mismatched audience is a healthy 401.
name = "the swarm collector admits this principal on a receiver of its own";
ok =
let
otel = baoWithCollector.services.hyperhive.swarm.otel;
s = baoWithCollector.containers.swarm-otel.config.services.opentelemetry-collector.settings;
name = otel.storeProducerName;
in
s.receivers."otlp/${name}".protocols.http.auth.authenticator == "oidc/${name}"
&&
s.extensions."oidc/${name}".audience == baoWithCollector.services.hyperhive.swarm.bao.otel.clientId
&& lib.elem "oidc/${name}" s.service.extensions
&& lib.elem "otlp/${name}" s.service.pipelines."metrics/${otel.producerName}".receivers
&& lib.elem "otlp/${name}" s.service.pipelines."logs/${otel.producerName}".receivers;
}
{
# A client authelia has never heard of mints nothing, so the publisher
# has nothing to copy and the delivery unit reads an empty path
# forever. Registered where AUTHELIA runs, which is why it is a glue
# module and not a line in the store's own config — this fixture runs
# no store at all and still has to register it.
name = "the forwarder's client is registered wherever authelia runs";
ok =
let
autheliaOnly = hive { deploy.authelia.enable = true; };
clients = autheliaOnly.services.hyperhive.swarm.authelia.oidc.clients;
c = lib.findFirst (c: c.id == "swarm-bao-collector") null clients;
in
!autheliaOnly.services.hyperhive.deploy.bao.enable
&& c != null
&& c.audience == [ "swarm-bao-collector" ]
# ⚠️ What makes the token READABLE by the receiver at all: authelia's
# default is an opaque handle, and an `oidc` extension verifies
# offline against the provider's JWKS.
&& c.accessTokenSignedResponseAlg == "RS256";
}
];
in
runGroup "bao-otel-collector" cases

11
nix/module-eval/secret-publisher.nix
View file

@ -132,6 +132,17 @@ let
secretPublisherHere.systemd.services.swarm-secret-publish.script
);
}
{
# The third swarm service on that list, and the one whose absence is
# hardest to see: the store's own forwarder holds a client authelia
# registers and mints for, so every layer above looks complete while
# the one hop that CARRIES the value skips it and the reader on the
# store's host waits forever on a path nothing writes.
name = "the publisher carries the store forwarder's secret too";
ok = lib.hasInfix "secret/swarm/services/swarm-bao-collector/oidc/client" (
secretPublisherHere.systemd.services.swarm-secret-publish.script
);
}
{
# The same hole the controller's case above names, open a second time: the
# PKI script grew a third leaf and no case read it.