vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix(#3554): push to the swarm's stores by domain, authenticated

Browse source 
The collector's store exporters were gated on deploy.victoriametrics.enable /
deploy.victorialogs.enable — options that say "this host RUNS the store", not
"the swarm has one". A collector that did not share a host with the stores
rendered no exporter at all and dropped everything it received, from every
hive, silently: an absent exporter is not an error.

mara ruled the premise rather than the mechanism ("the swarm always has a
store"), so there is no gate and no new option for that. Both exporters are
unconditional and address the stores by domain, per the cross-host rule from
the OIDC client and secret-delivery unit #3517 already built. The logs exporter
had the identical bug and is fixed here too.

Both stores gained a machine ingest route, required in the same change: the
exporter now targets https://logs.<domain>/insert/..., and that vhost is
browser-shaped, so shipping the collector half alone would have regressed logs
ingestion that works today. Neither ingest location carries
`error_page 401 =302` — a pusher handed a redirect follows it and POSTs at a
login page, which answers 200.

Whether the collector authenticates follows the CREDENTIAL, never another
service's placement: `clientSecretFile` is a nullable option, and the delivery
unit — the one thing here that may know where authelia runs, since it copies
out of its container — sets it by mkDefault. An earlier revision gated this on
deploy.authelia.enable directly, which put a different service's co-location in
the collector's own config.

Also removed rather than relaxed: the assertion that this collector has
"somewhere to send". It read the store's per-host enable, so it rejected at
eval exactly the deployment reaching the stores by domain exists for.
Deliberately not replaced with an authentication assertion — a collector on a
host of its own is a supported shape, and refusing to build it would make this
fix illegal where the bug bites hardest.

Knock-on worth review: collectLogs is now always satisfied, so journald
collection is unconditional.

Config shape validated against otelcol-contrib 0.151.0 `validate`, with a
bogus-key control confirming the validator checks the extension schema.
module-eval: 31 properties.
This commit is contained in:
atlas 2026-08-31 23:14:41 +02:00
commit 5478e0bf67
4 changed files with 446 additions and 181 deletions
Download patch file Download diff file Expand all files Collapse all files

129
nix/module-eval.nix
View file

@ -145,6 +145,28 @@ let
lib.findFirst (c: c.job_name == job) null
machine.containers.swarm-otel.config.services.opentelemetry-collector.settings.receivers.prometheus.config.scrape_configs;
# A swarm collector on a host that runs NEITHER store — the fully-spread
# shape from docs/swarm/services.md, and the one the old per-host gates made
# inexpressible. It is the whole point of the cases below that this hive is
# not a degenerate configuration but a supported one.
otelNoStores = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = true;
deploy.victoriametrics.enable = false;
deploy.victorialogs.enable = false;
};
otelSettings =
machine: machine.containers.swarm-otel.config.services.opentelemetry-collector.settings;
# authelia somewhere else, the credential delivered by hand. Whether this
# collector authenticates must follow the credential, never another
# service's placement.
otelRemoteAuthelia = hive {
deploy.swarm-otel.enable = true;
deploy.authelia.enable = false;
swarm.otel.clientSecretFile = "/var/lib/swarm-otel-oidc/by-hand.secret";
};
# A priority collision is a property of the *option*, not
# of the merged value's interior — nix throws the moment the value is
# demanded at all, so `seq`-ing each `serviceConfig` value to WHNF is
@ -208,6 +230,113 @@ let
name = "the CI container's unit definitions merge without a priority collision";
ok = forceCiServiceConfigs;
}
{
# The defect itself. These exporters used to be gated on the stores'
# PER-HOST enables, so a collector that did not share a host with them
# rendered none at all and dropped everything it received, from every
# hive — silently, because an absent exporter is not an error.
name = "a collector that hosts neither store still exports to both";
ok =
let
e = (otelSettings otelNoStores).exporters;
in
(e ? "otlphttp/victoriametrics") && (e ? "otlphttp/victorialogs");
}
{
# A swarm has one of each store, so the address is a swarm-level name.
# A loopback literal here is the co-location assumption written back in,
# and it renders, deploys and reports healthy while reaching nothing.
name = "the store exporters address the stores by name, never by loopback";
ok =
let
e = (otelSettings otelNoStores).exporters;
m = e."otlphttp/victoriametrics".metrics_endpoint;
l = e."otlphttp/victorialogs".logs_endpoint;
in
!(lib.hasInfix "127.0.0.1" m)
&& !(lib.hasInfix "127.0.0.1" l)
&& lib.hasInfix "metrics.t.local" m
&& lib.hasInfix "logs.t.local" l;
}
{
# The collector reaches these routes through the gateway now, so each
# store needs an ingest location of its own. Without one the write rides
# the `/` catch-all: unauthenticated on the metrics store, and into a
# browser redirect on the log store.
name = "each store's vhost has an authenticated ingest location";
ok =
let
v = allLocal.services.nginx.virtualHosts;
m = v."metrics.t.local".locations."= /opentelemetry/api/v1/push" or null;
l = v."logs.t.local".locations."= /insert/opentelemetry/v1/logs" or null;
in
m != null
&& l != null
&& lib.hasInfix "auth_request" m.extraConfig
&& lib.hasInfix "auth_request" l.extraConfig;
}
{
# The arm that actually protects something. A pusher handed
# `error_page 401 =302` FOLLOWS it and POSTs its batch at a login page,
# which answers 200 — ingest reporting healthy while storing nothing.
# The third clause is the positive control: the log store's browser
# location really does redirect, so this says the machine routes differ
# rather than that the string is absent from the whole file.
name = "the ingest locations answer 401 instead of redirecting a pusher";
ok =
let
v = allLocal.services.nginx.virtualHosts;
m = v."metrics.t.local".locations."= /opentelemetry/api/v1/push".extraConfig;
l = v."logs.t.local".locations."= /insert/opentelemetry/v1/logs".extraConfig;
browser = v."logs.t.local".locations."/".extraConfig;
in
!(lib.hasInfix "error_page" m)
&& !(lib.hasInfix "error_page" l)
&& lib.hasInfix "error_page" browser;
}
{
# Defining an exporter and REFERENCING it are two separate lists, and
# the second is where the original gate also lived. An exporter no
# pipeline names is as silent as one that does not exist — this case
# exists because a mutation that restored only the reference-side gate
# left every other case here green.
name = "every pipeline that has a store exporter defined actually sends to it";
ok =
let
s = otelSettings otelNoStores;
used = lib.unique (lib.concatMap (p: p.exporters) (lib.attrValues s.service.pipelines));
in
builtins.elem "otlphttp/victoriametrics" used && builtins.elem "otlphttp/victorialogs" used;
}
{
# The collector authenticates because it HOLDS a credential, not because
# authelia happens to share its host. Gating on the other service's
# placement renders a collector that pushes unauthenticated wherever
# authelia lives elsewhere — one of the supported shapes.
name = "a collector with a hand-delivered secret authenticates without authelia beside it";
ok =
let
s = otelSettings otelRemoteAuthelia;
in
(s.exporters."otlphttp/victoriametrics" ? auth)
&& builtins.elem "oauth2client/victoriametrics" s.service.extensions;
}
{
# An authenticator an exporter names but `service.extensions` omits is
# INERT — the collector starts clean and pushes unauthenticated until
# something at the far end refuses it. Checked as a set relation rather
# than by naming the two, so it keeps holding for exporters not written
# yet.
name = "every exporter authenticator is listed in service.extensions";
ok =
let
s = otelSettings otelNoStores;
named = lib.filter (v: v != null) (
lib.mapAttrsToList (_: e: e.auth.authenticator or null) s.exporters
);
in
named != [ ] && lib.all (a: builtins.elem a s.service.extensions) named;
}
{
# The store's seal is spread over six gates — the stanza, the
# provisioning unit, two bind mounts, a device and an EnvironmentFile.