vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

nix: the store's own collector scrapes its metrics listener

Browse source 
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.

What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.

Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.

The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.

Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.

The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
This commit is contained in:
atlas 2026-09-20 18:19:06 +02:00 • committed by mara
commit bb0afcd256
8 changed files with 815 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

133
nix/host-modules/swarm-otel.nix
View file

@ -47,6 +47,24 @@ let
# bound once here (rather than copy-pasted at both sites) is the only way
# the two can't drift apart.
swarmTierName = cfg.producerName;
# The SECOND swarm-tier producer: the collector inside the secret store's
# container (`swarm-bao.nix`). It gets an owner of its own rather than
# pushing into `swarmTierName`'s route, because the audience that route
# checks is `swarm-controller`'s own client id — one identity per principal,
# so a second principal brings its own client, its own audience and
# therefore its own authenticator and receiver.
#
# Named after the store's container, so `otlp/swarm-bao` reads as the thing
# that pushes into it. No `reserved-names.nix` entry is needed for it the
# way `swarm` has one: the name CONTAINS `swarm`, which
# `nix/reserved-hive-fragments.nix` forbids as a substring of any hive name,
# so no hive can ever own these components. The assertion below checks that
# fragment is still listed rather than trusting it.
storeProducerName = baoCfg.machine;
storeClientId = baoCfg.otel.clientId;
reservedHiveFragments = import ../reserved-hive-fragments.nix;
# Every `<owner>` no hive may take. Read from `nix/reserved-names.nix`, the
# same file the daemons are handed as `HIVE_RESERVED_NAMES`, because agent
# names and hive names are ONE namespace going forward — a locally-owned
@ -363,6 +381,50 @@ in
'';
};
storeProducerPort = lib.mkOption {
type = lib.types.port;
default = 4391;
description = ''
Port the receiver for the **secret store's own forwarder** listens
on, at `127.0.0.1`. Reached through the gateway at
`https://''${domain}/''${storeProducerName}/`, the same
path-per-producer shape a hive and the swarm-tier producer above
both use.
Its own receiver, and not a second sender into
{option}`services.hyperhive.swarm.otel.producerPort`, because an
`oidc` authenticator checks exactly ONE audience: that receiver
admits the audience `swarm-controller` registers for itself, and
the store's forwarder is a different principal with a client of its
own. Sharing the route would mean sharing that client — which is
the thing the store exists to make unnecessary.
⚠️ A reserved value rather than an offset from
{option}`services.hyperhive.swarm.otel.port`, for the reason stated
on `producerPort`: that range grows with the hive count and would
eventually walk into any fixed offset from it. The collision
assertion below covers this port too.
'';
};
storeProducerName = lib.mkOption {
type = lib.types.str;
readOnly = true;
default = baoCfg.machine;
defaultText = lib.literalExpression "services.hyperhive.swarm.bao.machine";
description = ''
Path and component name of the store forwarder's own receiver
(`otlp/''${storeProducerName}`, `oidc/''${storeProducerName}`).
Published so `swarm-bao.nix` addresses the receiver by name rather
than repeating the string, exactly as `producerName` above is
published for `swarm-controller.nix`.
Read-only and derived from the store's container name: two
spellings of a name both ends have to agree on is a 404 on a
request that authenticated perfectly.
'';
};
producerName = lib.mkOption {
type = lib.types.str;
readOnly = true;
@ -619,6 +681,9 @@ in
# request reaches the receiver, which knows nothing about the
# path it was found at.
"/${swarmTierName}/".proxyPass = "http://127.0.0.1:${toString cfg.producerPort}/";
# The secret store's forwarder, on its own route for its own
# audience — same trailing slashes, same reason.
"/${storeProducerName}/".proxyPass = "http://127.0.0.1:${toString cfg.storeProducerPort}/";
# There is no swarm-wide inbox, and a closed door is the honest
# description of that. Every other route into this collector
# belongs to exactly one hive or the swarm-tier producer above.
@ -873,6 +938,25 @@ in
Put it back, or give this module a different swarmTierName.
'';
}
{
# The store forwarder's components are named after a container whose
# name merely CONTAINS the reserved word, so what protects them is
# the substring list rather than the equality one. Same reason the
# assertion above exists: the list lives in another file, and a guard
# whose subject can be edited elsewhere has to assert its own case is
# still covered.
assertion = lib.any (f: lib.hasInfix f storeProducerName) reservedHiveFragments;
message = ''
nix/reserved-hive-fragments.nix forbids no substring of
'${storeProducerName}', which the swarm collector needs kept out of
the hive namespace: it names components `<kind>/<owner>`, so a hive
called '${storeProducerName}' would replace the secret store
forwarder's own receiver and pipeline entry.
Put the fragment back, or give the store's container a name one of
them covers.
'';
}
{
# A published target that is not an `https://host/path` url. Without
# this the split returns null and the failure surfaces as
@ -974,6 +1058,7 @@ in
others = [
cfg.telemetryPort
cfg.producerPort
cfg.storeProducerPort
otelCfg.collector.port
]
++ lib.optional deployCfg.victoriametrics.enable vmCfg.port;
@ -1119,6 +1204,15 @@ in
auth.authenticator = "oidc/${swarmTierName}";
};
}
# The secret store's forwarder, which is a swarm-tier
# producer with a principal of its own — a receiver per
# audience, because an `oidc` authenticator checks one.
// {
"otlp/${storeProducerName}".protocols.http = {
endpoint = "127.0.0.1:${toString cfg.storeProducerPort}";
auth.authenticator = "oidc/${storeProducerName}";
};
}
# MERGED with the per-hive receivers, never assigned over
# them. A plain assignment here would drop every hive's
# receiver and still render a valid config that starts
@ -1303,7 +1397,13 @@ in
# same reasoning as its receiver above: `otlp/${swarmTierName}`
# always exists, so the authenticator it names must too, or an
# extension-not-listed startup failure follows every deploy.
++ [ "oidc/${swarmTierName}" ]
++ [
"oidc/${swarmTierName}"
# Unconditional for the same reason, and one level sharper:
# the store runs in every swarm, so its forwarder's
# receiver is as permanent as the swarm tier's own.
"oidc/${storeProducerName}"
]
# The push side's authenticators. Same rule as above: one an
# exporter names but this list omits is INERT — the collector
# starts clean and pushes unauthenticated.
@ -1347,6 +1447,16 @@ in
"metrics/${swarmTierName}" = {
receivers = [
"otlp/${swarmTierName}"
# The store forwarder's receiver joins the swarm
# tier's pipeline rather than bringing one of its own:
# the stamp it needs is `resource/${swarmTierName}`'s
# exact stamp — the swarm's name and deliberately no
# `hive` — and a second pipeline would be that same
# processor under a second name to keep in step. What
# a producer is separated FOR is the credential it
# proves, and that is the receiver's job, not the
# pipeline's.
"otlp/${storeProducerName}"
]
++ lib.optional (cfg.scrapeTargets != { } || cfg.publishedScrapeTargets != { }) "prometheus";
processors = [ "resource/${swarmTierName}" ];
@ -1386,7 +1496,14 @@ in
# `hive` value to put on them.
// lib.optionalAttrs collectLogs {
"logs/${swarmTierName}" = {
receivers = [ "journald" ];
# This host's own journal, and the store container's,
# which arrives over the receiver above rather than off
# a disk — the store's forwarder reads a journal no
# reader on another host can see.
receivers = [
"journald"
"otlp/${storeProducerName}"
];
processors = [ "resource/${swarmTierName}" ];
exporters = logExporterNames;
};
@ -1450,6 +1567,18 @@ in
audience = config.services.hyperhive.swarm.controller.queueClientId;
};
}
# The secret store forwarder's own authenticator. Same shape
# again, and the audience is that principal's own client id
# — the self-referential form `swarm-authelia.nix` registers
# a hive client under and `swarm-controller` uses above,
# read from the option `glue-swarm-bao-otel-oidc-client.nix`
# registers so the two ends cannot spell it differently.
// {
"oidc/${storeProducerName}" = {
issuer_url = hyperhiveCfg.swarm.authelia.url;
audience = storeClientId;
};
}
# The push side. Opposite direction to every `oidc/*` above —
# those VALIDATE a token arriving; these OBTAIN one to send.
# Hence `oauth2client` rather than `oidc`, and hence a client