nix: give swarm-bao its own otel collector
Every container is supposed to run a collector that passes its logs and metrics to the next hop. swarm-bao did not: its journal reached the store only because `--link-journal=host` puts it in the host tree, where the swarm collector — a different container — reads it through a unit allowlist. That is the topology being retired, and in this deployment it delivers nothing: no `_SYSTEMD_UNIT` value in the seven-day store mentions openbao at all. So the store's container now runs its own journal forwarder, copied from an agent container's (nix/agent-modules/otel.nix): the whole journal, no unit allowlist, pushed to the same first hop every agent on the host already exports to. A local collector reads the local journal, so there is nothing for a list of unit names to disagree with. The `swarm.otel.journaldUnits` entry and `--link-journal=host` both stay. Every sibling swarm container still rides the shared collector, and they come out once each of them has a forwarder of its own. Closes #4526
This commit is contained in:
parent
8d257e9362
commit
e5224a6725
2 changed files with 200 additions and 1 deletions
|
|
@ -47,6 +47,24 @@ let
|
|||
deploy.swarm-otel.enable = false;
|
||||
};
|
||||
|
||||
# The store on a host whose HIVE collector is running — the only state in
|
||||
# which the container forwards its own journal, since that collector is the
|
||||
# hop it forwards to. `clientSecretFile` is what ../host-modules/otel.nix's
|
||||
# identity assertion demands of any hive with the tier on. `swarm-otel` is
|
||||
# deliberately left off: the forwarder is a function of the first hop
|
||||
# existing, and pairing the two fixtures would make a case that passed on the
|
||||
# wrong condition.
|
||||
baoWithHiveOtel = hive {
|
||||
deploy.bao.enable = true;
|
||||
otel.enable = true;
|
||||
otel.clientSecretFile = "/var/lib/hive-otel-oidc/client.secret";
|
||||
};
|
||||
|
||||
# The forwarder INSIDE the store's container, not the host's collector and
|
||||
# not the swarm tier's — three collectors in this tree, and only this one can
|
||||
# see the store's journal.
|
||||
baoForwarder = machine: machine.containers.swarm-bao.config.services.opentelemetry-collector;
|
||||
|
||||
# The scrape list prometheus is handed, not the option a service declared:
|
||||
# the address, the path and the query are one string on the way in and three
|
||||
# fields on the way out, and only the second shape is what gets requested.
|
||||
|
|
@ -126,6 +144,49 @@ let
|
|||
in
|
||||
!(s.listener ? metrics) && !(s ? telemetry);
|
||||
}
|
||||
{
|
||||
# The store's journal reaches a reader through a collector of its own,
|
||||
# and every one of these fields is silent when wrong: the runtime journal
|
||||
# is the receiver's own default and is empty here, an unlisted extension
|
||||
# is inert so the cursor silently stops persisting, and a pipeline is
|
||||
# free to name none of it.
|
||||
name = "the store's container forwards its own journal";
|
||||
ok =
|
||||
let
|
||||
s = (baoForwarder baoWithHiveOtel).settings;
|
||||
p = s.service.pipelines.logs;
|
||||
in
|
||||
s.receivers.journald.directory == "/var/log/journal"
|
||||
&& s.receivers.journald.storage == "file_storage"
|
||||
&& s.service.extensions == [ "file_storage" ]
|
||||
&& p.receivers == [ "journald" ]
|
||||
&& p.exporters == [ "otlphttp" ]
|
||||
&& s ? exporters.otlphttp;
|
||||
}
|
||||
{
|
||||
# The whole journal, which is what the shared collector's unit allowlist
|
||||
# is not. A `units` list here would render and deploy perfectly while
|
||||
# shipping only the units someone remembered to name — the failure this
|
||||
# forwarder exists to end.
|
||||
name = "the store's forwarder filters no units";
|
||||
ok = !((baoForwarder baoWithHiveOtel).settings.receivers.journald ? units);
|
||||
}
|
||||
{
|
||||
# Both ends of the first hop, on ONE host, because a mismatch between
|
||||
# them is silent in both directions: the exporter retries into nothing
|
||||
# and the receiver never hears from it.
|
||||
name = "the store's forwarder exports to this hive's own collector";
|
||||
ok =
|
||||
(baoForwarder baoWithHiveOtel).settings.exporters.otlphttp.endpoint
|
||||
== "http://${baoWithHiveOtel.services.opentelemetry-collector.settings.receivers.otlp.protocols.http.endpoint}";
|
||||
}
|
||||
{
|
||||
# Absence arm — `baoNoCollector` runs neither tier, so there is no first
|
||||
# hop on this host at all. A forwarder rendered anyway would start, find
|
||||
# nothing listening, and retry forever while reporting healthy.
|
||||
name = "a store with no hive collector renders no forwarder";
|
||||
ok = !(baoForwarder baoNoCollector).enable;
|
||||
}
|
||||
];
|
||||
in
|
||||
runGroup "bao-otel-collector" cases
|
||||
|
|
|
|||
Loading…
Reference in a new issue