nix: give swarm-bao its own otel collector

Every container is supposed to run a collector that passes its logs and
metrics to the next hop. swarm-bao did not: its journal reached the store
only because `--link-journal=host` puts it in the host tree, where the
swarm collector — a different container — reads it through a unit
allowlist. That is the topology being retired, and in this deployment it
delivers nothing: no `_SYSTEMD_UNIT` value in the seven-day store mentions
openbao at all.

So the store's container now runs its own journal forwarder, copied from
an agent container's (nix/agent-modules/otel.nix): the whole journal, no
unit allowlist, pushed to the same first hop every agent on the host
already exports to. A local collector reads the local journal, so there is
nothing for a list of unit names to disagree with.

The `swarm.otel.journaldUnits` entry and `--link-journal=host` both stay.
Every sibling swarm container still rides the shared collector, and they
come out once each of them has a forwarder of its own.

Closes #4526
This commit is contained in:
atlas 2026-09-19 15:39:24 +02:00 committed by mara
commit e5224a6725
2 changed files with 200 additions and 1 deletions

View file

@ -47,6 +47,24 @@ let
deploy.swarm-otel.enable = false;
};
# The store on a host whose HIVE collector is running — the only state in
# which the container forwards its own journal, since that collector is the
# hop it forwards to. `clientSecretFile` is what ../host-modules/otel.nix's
# identity assertion demands of any hive with the tier on. `swarm-otel` is
# deliberately left off: the forwarder is a function of the first hop
# existing, and pairing the two fixtures would make a case that passed on the
# wrong condition.
baoWithHiveOtel = hive {
deploy.bao.enable = true;
otel.enable = true;
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
};
# The forwarder INSIDE the store's container, not the host's collector and
# not the swarm tier's — three collectors in this tree, and only this one can
# see the store's journal.
baoForwarder = machine: machine.containers.swarm-bao.config.services.opentelemetry-collector;
# The scrape list prometheus is handed, not the option a service declared:
# the address, the path and the query are one string on the way in and three
# fields on the way out, and only the second shape is what gets requested.
@ -126,6 +144,49 @@ let
in
!(s.listener ? metrics) && !(s ? telemetry);
}
{
# The store's journal reaches a reader through a collector of its own,
# and every one of these fields is silent when wrong: the runtime journal
# is the receiver's own default and is empty here, an unlisted extension
# is inert so the cursor silently stops persisting, and a pipeline is
# free to name none of it.
name = "the store's container forwards its own journal";
ok =
let
s = (baoForwarder baoWithHiveOtel).settings;
p = s.service.pipelines.logs;
in
s.receivers.journald.directory == "/var/log/journal"
&& s.receivers.journald.storage == "file_storage"
&& s.service.extensions == [ "file_storage" ]
&& p.receivers == [ "journald" ]
&& p.exporters == [ "otlphttp" ]
&& s ? exporters.otlphttp;
}
{
# The whole journal, which is what the shared collector's unit allowlist
# is not. A `units` list here would render and deploy perfectly while
# shipping only the units someone remembered to name — the failure this
# forwarder exists to end.
name = "the store's forwarder filters no units";
ok = !((baoForwarder baoWithHiveOtel).settings.receivers.journald ? units);
}
{
# Both ends of the first hop, on ONE host, because a mismatch between
# them is silent in both directions: the exporter retries into nothing
# and the receiver never hears from it.
name = "the store's forwarder exports to this hive's own collector";
ok =
(baoForwarder baoWithHiveOtel).settings.exporters.otlphttp.endpoint
== "http://${baoWithHiveOtel.services.opentelemetry-collector.settings.receivers.otlp.protocols.http.endpoint}";
}
{
# Absence arm — `baoNoCollector` runs neither tier, so there is no first
# hop on this host at all. A forwarder rendered anyway would start, find
# nothing listening, and retry forever while reporting healthy.
name = "a store with no hive collector renders no forwarder";
ok = !(baoForwarder baoNoCollector).enable;
}
];
in
runGroup "bao-otel-collector" cases