Commit graph hyperhive/nix/host-modules/swarm-otel.nix
Author SHA1 Message Date
atlas
08faa0970e swarm-otel: authenticate ingest per hive, and stamp the hive from the receiver
The swarm collector accepted OTLP from anyone who could reach it, and took
the `hive` resource attribute from the payload. So any writer on the swarm
network could attribute metrics to any hive, and nothing downstream could
tell.

The label now comes from which receiver accepted the sample: one receiver
per hive, each behind an `oidc` extension verifying a token minted for that
hive's audience, each feeding a pipeline whose `resource` processor upserts
a constant. A sender cannot influence it, because the only input is which
authenticated port the bytes arrived on.

That multiplicity is forced rather than preferred. A processor cannot read
the token's claims — `from_context` reads request metadata, and asking it
for an auth claim yields nothing, silently, with a healthy startup — and
one receiver holding several credentials never reveals which one matched.

The per-hive ports are internal: a hive reaches its receiver as a path
under this collector's existing gateway name, so nginx (rendered from this
same evaluation) is the only thing that names a port. Fronting each hive
with its own vhost would need a certificate, a DNS name and a gateway entry
per hive to express routing the gateway already does.

Turning this on removes the unauthenticated receiver. While an open port
still accepts samples the per-hive receivers are decoration, so this is the
switch itself rather than a hardening layer beside it; a swarm that wants
the open receiver says so.

`hive-ca-trust.nix` grows `bundlePathFor`, because a consumer taking its own
CA argument has to name the bundle rather than just have `SSL_CERT_FILE`
exported at it.
2026-08-19 15:27:09 +02:00
atlas
d2fb4bff79 feat(#3125): reshape the hive-to-swarm OTEL hop by domain
Drops swarm.otel.url (a loopback default an operator had to override on a
split host) in favor of swarm.otel.domain -- the same
gateway.localNames + nginx-vhost-through-the-gateway shape every other
swarm service (authelia, grafana, victoriametrics, ui) already uses. The
hive tier's exporter now reaches it as https://<domain> unconditionally,
resolved locally by dnsmasq on a co-located host and over the real network
otherwise, instead of a config knob nobody sets until they hit the silent
drop.

Costs CA trust on the hive tier: otel.nix wires
lib/hive-ca-trust.nix's trustBundle with hostUnit = true on the
opentelemetry-collector host unit, the same flag #3441/#3442 added for
swarm-controller and hive-c0re.

mara, #3125 comment 58363: "go c".
2026-08-18 21:02:18 +02:00
atlas
28623e5eff fix(#3125): the swarm collector writes its own resolver, like its siblings
All four sibling swarm containers import swarm-container-resolver.nix;
this one did not. It matters more here than most: otel.endpoint is an
operator-configured external hostname, and reaching it is the entire
reason this container holds a credential.

Also aligns two details with those siblings - the enable default is
asserted from swarm-required-services.nix with the metrics pair it
feeds, so that file remains the one place a service host is declared,
and machine is readOnly since its description already calls it a fact
rather than a knob.
2026-08-18 21:02:18 +02:00
atlas
98ab0ad59f style: drop tracker tags from the collector modules' comments
The pre-push lint refuses them, and rightly: a comment that names an
issue number ages into a pointer at a closed thread. The constraint each
one carried is stated directly instead.
2026-08-18 21:02:18 +02:00
atlas
636dba8d61 fix(#3125): the swarm tier's self-metrics must not fight the hive tier's
A collector serves its own metrics on localhost:8888 unless told
otherwise, and co-located tiers share a network namespace, so the second
one to start dies with 'bind: address already in use'.

The port appears in neither config - it is a default inside the binary -
so comparing the ports the configs name reports them distinct. A
behavioural probe found it by being unable to start the chain.

metrics.address is the spelling that looks right and is rejected by this
version ('migration.MetricsConfigV030' has invalid keys: address);
readers is the schema it accepts.
2026-08-18 21:02:18 +02:00
atlas
f61c927310 feat(#3125): a swarm-tier OTEL collector, in its own container
The swarm tier is the only holder of the upstream credential, the only
writer to the swarm's metrics store, and (once #3283 lands) the place
that stamps hive= from the authenticated connection rather than from
anything a sender can choose. Today one collector does both tiers' jobs,
which works only because they land on one box.

A container rather than a second host unit, for two reasons that agree:
every sibling swarm service is one, and `services.opentelemetry-collector`
is a singleton NixOS option already spoken for on the host by the hive
tier. A container gets its own evaluation and therefore its own
collector.

Port defaults to 4319, deliberately not the OTLP default 4318 the hive
tier uses: swarm containers share the host netns, and two listeners
claiming one port is not a build failure but a runtime coin toss with
nothing in any log saying so -- the same collision grafana and the forge
hit on 3000.

`url` is an option with a co-located default rather than a loopback
literal in the exporter, so a split-host deployment is a config change
instead of a code change.

Wires nothing yet: the hive tier still exports directly, and switching it
over is the next commit.
2026-08-18 21:02:18 +02:00