bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.
What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.
Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.
The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.
Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.
The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
Both earlier versions asked the wrong host. `hyperhive.otel.enable` asked
whether this host runs a HIVE collector; `deploy.swarm-otel.enable` asked
whether this host runs the SWARM one. Neither answers the question the
forwarder actually has — "is there a collector to forward to" — and that
question cannot be false: a swarm always runs at least one instance of
every swarm-level service. So the forwarder renders under the condition
already enclosing it, that the store is deployed here, and nothing else.
`scrapeHere` deliberately keeps its `deploy.*` gate one line up. It is a
loopback metrics listener, which genuinely only works where the scraper
is — the two are different tiers, and the name says so.
The module-eval case that pins it is the split topology: the swarm
collector on another host, nothing local naming it, and the forwarder
still enabled and still addressed at `swarm.otel.domain`'s route. Both
removed gates render nothing in that fixture, which the co-located ones
they shipped with could not show.
The forwarder pointed at the hive bridge address and was gated on the
hive's `otel.enable`, so it existed only where a hive collector stood
beside it. It now exports to `swarm.otel.domain` — the gateway-served
name that resolves locally when co-located and over the network
otherwise — on the swarm tier's own producer route, and is gated on
`deploy.swarm-otel.enable` like its sibling `scrapeHere`.
Refs #4526
Every container is supposed to run a collector that passes its logs and
metrics to the next hop. swarm-bao did not: its journal reached the store
only because `--link-journal=host` puts it in the host tree, where the
swarm collector — a different container — reads it through a unit
allowlist. That is the topology being retired, and in this deployment it
delivers nothing: no `_SYSTEMD_UNIT` value in the seven-day store mentions
openbao at all.
So the store's container now runs its own journal forwarder, copied from
an agent container's (nix/agent-modules/otel.nix): the whole journal, no
unit allowlist, pushed to the same first hop every agent on the host
already exports to. A local collector reads the local journal, so there is
nothing for a list of unit names to disagree with.
The `swarm.otel.journaldUnits` entry and `--link-journal=host` both stay.
Every sibling swarm container still rides the shared collector, and they
come out once each of them has a forwarder of its own.
Closes#4526
The single module-eval derivation forced ~62 full nixosSystem
fixtures live at once to compute its cases list: 10.6GB peak RSS /
5m25s to evaluate, by far the dominant cost in nix flake check.
Splits it into 21 independent checks.module-eval-* derivations
(1-7 fixtures each) sharing builders/helpers via module-eval/lib.nix,
so no single derivation needs more than a handful of fixtures live
at once. A few cases spanning two clusters carry a small duplicated
fixture rather than threading shared state through lib.nix.
