vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

nix: the store's own collector scrapes its metrics listener

Browse source 
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.

What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.

Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.

The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.

Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.

The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
This commit is contained in:
atlas 2026-09-20 18:19:06 +02:00 • committed by mara
commit bb0afcd256
8 changed files with 815 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

34
docs/swarm/secrets.md
View file

@ -64,7 +64,8 @@ because nothing outside that container ever reads them. **That's the test worth
applying to any secret added here** — and the client secret's plaintext half is
the one row that fails it, which is the entire reason a delivery step exists.
Two telemetry collectors exist, and they land on opposite sides of that test.
Three telemetry collectors exist, and they don't land on the same side of
that test.
The **hive's** collector needs no delivery step. It authenticates to the swarm's
collector as its own hive, and it's a host unit rather than a container, so on
@ -94,6 +95,27 @@ uses for its non-optional one. The operator can still deliver the secret by
hand, at `services.hyperhive.deploy.swarm-otel.clientSecretFile`, on a
collector this unit never reaches.
The **secret store's own** collector — the forwarder inside the `swarm-bao`
container — needs a delivery step too, and it takes the same route with one
principal of its own: `swarm-bao-forwarder-oidc.service` reads
`swarm/services/<client-id>/oidc/client` under this host's certificate and
lands it in the container's tree, where `LoadCredential` hands it to the
collector. The client id is its own
(`services.hyperhive.swarm.bao.otel.clientId`), registered by
`glue-swarm-bao-otel-oidc-client.nix` wherever authelia runs, and the swarm
collector admits it on a receiver of its own — one identity per principal, so
this forwarder borrows neither the swarm collector's client nor
`swarm-controller`'s.
⚠️ **This one has no degraded mode, and the difference is structural.** The
route it exports over accepts an authenticated request and nothing else, so a
forwarder without the secret has nothing to fall back to: the unit fails and
retries, and the collector refuses to start until the file arrives rather than
exporting into a 401 while reporting healthy. Its ordering differs from the
unit above it for the same structural reason — it reads the store inside the
container it delivers into, so it runs **after** that container rather than
before it, and only a first boot ever waits.
### Minting the queue's callout nkeys
`deploy.nats.autoGenerateCallout` mints both keypairs on the host before the queue
@ -254,6 +276,16 @@ doesn't render the reading unit without them, the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use for
their own optional reads.
The **fifth reader** is the secret store forwarder's OIDC client secret:
`swarm-bao-forwarder-oidc.service`, on the store's own host, registered by
`glue-swarm-bao-otel-oidc-client.nix` on authelia's. It differs from the
fourth in both directions. It renders unconditionally, because the export it
authenticates has no unauthenticated mode to degrade into — and it orders
itself **after** `container@swarm-bao`, because the store it reads runs in the
container it delivers into. Nothing circular sits behind that: the identity it
logs in with is this host's static `swarm-bao-pki` leaf, not anything the store
mints.
A service's secret is one value for the whole swarm rather than one per hive, so
it lives under the `services` prefix, and a hive's read policy grants that prefix
whole. That's because nothing in a swarm's configuration records which hive runs