vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-otel: run in its own network namespace, receivers on unix sockets

Browse source 
Refs #4815
This commit is contained in:
atlas 2026-10-22 03:22:49 +02:00 • committed by mara
commit e226e65954
6 changed files with 196 additions and 221 deletions
Download patch file Download diff file Expand all files Collapse all files

136
nix/host-modules/swarm-otel-service.nix
View file

@ -50,34 +50,6 @@ in
'';
};
port = lib.mkOption {
type = lib.types.port;
default = 4319;
description = ''
First port of this collector's receiver range. Every hive in
{option}`services.hyperhive.swarm.hives` gets its **own**
authenticated receiver — that is what makes the `hive` label
unforgeable — so the range is one port per hive, starting here, in
sorted-name order.
⚠️ Internal. No client is ever told a port: a hive reaches its own
receiver as `https://''${domain}/<hive>`, and the gateway routes on
that path. So adding a hive, which renumbers the ones after it, is
harmless — nginx is rendered from this same evaluation and moves
with it.
⚠️ **Deliberately not 4318**, the OTLP/HTTP default, because the
hive tier already uses it (`services.hyperhive.swarm.hiveOtel.collectorPort`)
and every swarm container shares the host's network namespace. Two
listeners claiming one port on one host is not a build failure —
it is a runtime coin toss over which one gets it, with nothing in
any log saying so. On the host running this collector, an
assertion rejects a derived range that overlaps any other port
this module or the hive tier declares, which is as far as a module
can see.
'';
};
telemetryPort = lib.mkOption {
type = lib.types.port;
default = 8889;
@ -87,73 +59,8 @@ in
that telemetry is being lost, so it is worth keeping rather than
switching off.
⚠️ **Deliberately not 8889's neighbour 8888**, which is the
collector's built-in default and therefore what the hive tier
already binds. Two collectors share a network namespace whenever
they are co-located, and unlike the OTLP port this one appears
nowhere in either config — it is a default inside the binary, so
nothing that compares configured ports can see the clash. The
second collector to start simply dies with
`bind: address already in use`.
'';
};
producerPort = lib.mkOption {
type = lib.types.port;
default = 4390;
description = ''
Port the swarm tier's own OTLP/HTTP receiver for a
**swarm-level** producer (`swarm-controller`'s vcs/jobq counters
today) listens on, at `127.0.0.1`. Reached through the gateway at
`https://''${domain}/''${producerName}/`, same shape as a hive's
own receiver — a swarm-level producer may not share a host with
this collector (the swarm services need not run on the same host
as the swarm controller), so loopback-only reachability is not a
supported shape here, as it is not for a hive.
Authenticated by `oidc/''${producerName}`, checking for the
audience `services.hyperhive.swarm.controller.queueClientId`
requests — not the per-hive `hiveClientPrefix` namespace, because
a swarm-level producer is explicitly not a hive (see that
option's own doc comment). The `swarm` label this receiver's
samples get is still the one the receiver's own *existence*
establishes, same as before — there is no hive identity being
forged or attributed here, just a caller proving it is the one
principal allowed to push into this pipeline.
⚠️ Deliberately NOT derived from `port + (number of hives)`: that
range grows every time a hive is added, and a fixed offset from
it would silently start colliding once the hive count caught up.
Kept as its own reserved value instead, and the port-collision
assertion still rejects a real collision (including one hive
growth eventually causes) rather than starting a collector that
quietly drops one receiver's samples.
'';
};
storeProducerPort = lib.mkOption {
type = lib.types.port;
default = 4391;
description = ''
Port the receiver for the **secret store's own forwarder** listens
on, at `127.0.0.1`. Reached through the gateway at
`https://''${domain}/''${storeProducerName}/`, the same
path-per-producer shape a hive and the swarm-tier producer above
both use.
Its own receiver, and not a second sender into
{option}`services.hyperhive.swarm.otel.producerPort`, because an
`oidc` authenticator checks exactly ONE audience: that receiver
admits the audience `swarm-controller` registers for itself, and
the store's forwarder is a different principal with a client of its
own. Sharing the route would mean sharing that client — which is
the thing the store exists to make unnecessary.
⚠️ A reserved value rather than an offset from
{option}`services.hyperhive.swarm.otel.port`, for the reason stated
on `producerPort`: that range grows with the hive count and would
eventually walk into any fixed offset from it. The port-collision
assertion covers this port too.
Bound on the container's own loopback, so no other collector
shares it; only this collector scrapes it.
'';
};
@ -170,14 +77,6 @@ in
the forwarder requests.
'';
};
port = lib.mkOption {
type = lib.types.port;
description = ''
Loopback port of the producer's receiver on the collector's
host. In the port-collision assertion with every other
receiver there.
'';
};
};
}
);
@ -256,13 +155,15 @@ in
scrapeTargets = lib.mkOption {
type = lib.types.attrsOf lib.types.str;
default = { };
example = lib.literalExpression ''{ forgejo = "127.0.0.1:3000"; }'';
example = lib.literalExpression ''{ collector = "127.0.0.1:8889"; }'';
description = ''
Prometheus exposition endpoints this collector scrapes, as
`<job name> = "<host>:<port>"`.
`<job name> = "<host>:<port>"`. The collector runs in a network namespace of
its own, so `127.0.0.1` is the collector container's loopback: a target
here is a listener inside that container, such as its own telemetry.
A path and query may follow the port —
`"127.0.0.1:8202/v1/sys/metrics?format=prometheus"` — for an
`"127.0.0.1:9100/stats?format=prometheus"` — for an
exporter that does not serve `/metrics`. Both are optional and
omitted when absent, so a bare `host:port` is scraped exactly as
before.
@ -270,11 +171,10 @@ in
**A service declares its own entry, from its own module, under its
own `enable`.** Do not assemble the list here: an entry then exists
only where the service that named it runs, so a target is never
declared on a host that does not serve it. An entry that also needs
the collector on the same host (the loopback address is only honest
then) is declared from a glue module instead, which reads both
switches. Every swarm service is a `publishedScrapeTargets` entry
instead, which a collector on any host scrapes.
declared on a host that does not serve it. A swarm service on the
host is not reachable here, since its loopback is not the collector's:
it is a `publishedScrapeTargets` entry instead, which a collector on
any host scrapes.
⚠️ **That constrains the TARGET, not the SCRAPER, and the difference
is a silent gap.** Nothing here puts the collector on the same host.
@ -383,73 +283,59 @@ in
# The forwarder in ./swarm-bao.nix.
${cfg.storeProducerName} = {
clientId = baoCfg.otel.clientId;
port = cfg.storeProducerPort;
};
# The forwarder in ./swarm-nats.nix, which addresses this route by the
# queue container's name.
swarm-nats = {
clientId = hyperhiveCfg.swarm.nats.clientId;
port = lib.mkDefault 4392;
};
# The forwarders in ./swarm-builder.nix, ./swarm-claude-login.nix,
# ./swarm-notify-router.nix and ./swarm-review-classifier.nix, which
# address their routes by container name.
swarm-builder = {
clientId = hyperhiveCfg.swarm.builder.clientId;
port = lib.mkDefault 4393;
};
swarm-claude-login = {
clientId = hyperhiveCfg.swarm.claude-login.clientId;
port = lib.mkDefault 4394;
};
swarm-review-classifier = {
clientId = hyperhiveCfg.swarm.review-classifier.clientId;
port = lib.mkDefault 4395;
};
swarm-notify-router = {
clientId = hyperhiveCfg.swarm.notify-router.clientId;
port = lib.mkDefault 4404;
};
# The forwarders in ./swarm-ci-runner.nix, ./swarm-grafana.nix and
# ./swarm-victorialogs.nix, each authenticated as a client of its own.
swarm-ci-runner = {
clientId = hyperhiveCfg.swarm.ci-runner.otel.clientId;
port = lib.mkDefault 4396;
};
swarm-grafana = {
clientId = hyperhiveCfg.swarm.grafana.otel.clientId;
port = lib.mkDefault 4397;
};
swarm-victorialogs = {
clientId = hyperhiveCfg.swarm.victorialogs.otel.clientId;
port = lib.mkDefault 4398;
};
# The forwarders in ./hive-forge/default.nix and ./hive-matrix.nix, each
# authenticated as a client of its own.
hive-forge = {
clientId = hyperhiveCfg.swarm.forge.otel.clientId;
port = lib.mkDefault 4399;
};
hive-matrix = {
clientId = hyperhiveCfg.swarm.matrix.otel.clientId;
port = lib.mkDefault 4400;
};
# The forwarder in ./swarm-victoriametrics.nix, which reaches this route
# through its host's nginx.
swarm-victoriametrics = {
clientId = hyperhiveCfg.swarm.victoriametrics.otel.clientId;
port = lib.mkDefault 4401;
};
# The CI run poller in ./swarm-ci-metrics.nix, a host unit that pushes
# its run samples here directly.
swarm-ci-metrics = {
clientId = hyperhiveCfg.swarm.ci-metrics.clientId;
port = lib.mkDefault 4403;
};
# The forwarder in ./swarm-authelia.nix.
swarm-authelia = {
clientId = hyperhiveCfg.swarm.authelia.otel.clientId;
port = lib.mkDefault 4402;
};
};