vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

swarm-otel: ship the whole host journal, drop user sessions after it

Browse source 
The swarm collector's journald receiver read only the units listed in
`services.hyperhive.swarm.otel.journaldUnits`. A unit nobody listed
never reached the store, and a misspelt entry shipped nothing without
an error. The list existed to keep an operator's desktop session out of
a store every swarm operator can read, but the receiver can only match
positively, so the only way to express "not user sessions" was to name
every service instead.

The receiver now reads the whole host journal, and a new
`filter/exclude-user-sessions` processor in the `logs/<swarm>` pipeline
drops records whose `_SYSTEMD_SLICE` is `user-<uid>.slice` (session
scopes and `user@<uid>.service`). The per-hive `logs/<hive>` pipelines
carry agent-container journals only and get no filter.

`journaldUnits` is removed with `mkRemovedOptionModule`, together with
its non-empty assertion and the entry each host module added. The four
module-eval membership checks go with it, replaced by one structural
case in swarm-otel-core.

Closes #3646
This commit is contained in:
atlas 2026-09-30 22:51:52 +02:00 • committed by mara
commit 6b1e825c0a
25 changed files with 72 additions and 324 deletions
Download patch file Download diff file Expand all files Collapse all files

126
nix/host-modules/swarm-otel.nix
View file

@ -314,6 +314,15 @@ let
storeRetry = import ./lib/store-retry.nix { };
in
{
imports = [
(lib.mkRemovedOptionModule [ "services" "hyperhive" "swarm" "otel" "journaldUnits" ] ''
The swarm collector no longer takes a unit allowlist: it ships the
journal of every system unit on the host and drops only user
sessions (records under a `user-<uid>.slice`). Delete the setting;
nothing replaces it.
'')
];
# `enable` moved to `services.hyperhive.deploy.swarm-otel.enable` — see ./deploy.nix.
# ⚠️ That is the SWARM collector. The per-hive one keeps its own
# `services.hyperhive.otel.enable` (./otel.nix) and is a different
@ -572,32 +581,6 @@ in
'';
};
journaldUnits = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ ];
example = [ "nginx" ];
description = ''
systemd units whose journal this collector ships to the swarm's
log store. Nothing outside this list is collected.
Every module that defines a unit worth reading swarm-wide adds
its own names here, rather than one list naming them all: a
service that is not running contributes nothing, and a service
added later arrives with its units already declared. A central
list would be a second place that has to know which services
exist, and it would go stale in the direction that hides a
service's logs rather than the one that shows too many.
Names are matched as journald `_SYSTEMD_UNIT` values, so an
in-container unit is named exactly as it is inside its container
— `authelia`, not `container@swarm-authelia`.
⚠️ A name that matches nothing is not an error anywhere. The unit
simply never appears in the store, which looks the same as a
service that had nothing to say.
'';
};
clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
@ -667,28 +650,12 @@ in
services.hyperhive.gateway.enable = lib.mkDefault true;
services.hyperhive.gateway.dns.enable = lib.mkDefault true;
# This collector reads its own journal, so a pipeline that stops
# delivering says so in the store it stopped delivering to. That is
# less circular than it sounds: the failure that matters here is a
# single hive's receiver refusing pushes, not the process dying.
#
# The container's own host unit is what says the collector never started
# — a failed bind source or a `Requires=` that did not come up — which
# the process inside cannot report. The activation has no module to name
# it: nixos-rebuild runs switch-to-configuration as this transient unit,
# and its syslog lines are the deploy's start, finish or failure status.
services.hyperhive.swarm.otel.journaldUnits = [
"opentelemetry-collector"
"swarm-bao-otel-oidc"
"container@${cfg.machine}"
"nixos-rebuild-switch-to-configuration"
];
# The metrics counterpart to the journal line above, closing the same
# gap from the other side: the journal says the process is alive, these
# counters say whether it is dropping what it receives. Loopback works
# here because `privateNetwork = false` — this container shares the
# host's netns, so `127.0.0.1` is where `telemetryPort` is bound.
# The metrics counterpart to this collector's own journal, which its
# receiver ships like any other unit's: the journal says the process is
# alive, these counters say whether it is dropping what it receives.
# Loopback works here because `privateNetwork = false` — this container
# shares the host's netns, so `127.0.0.1` is where `telemetryPort` is
# bound.
services.hyperhive.swarm.otel.scrapeTargets.collector = "127.0.0.1:${toString cfg.telemetryPort}";
# OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
@ -936,26 +903,7 @@ in
# rejected at eval the very deployment the stores are reached by domain
# for — a collector on its own host never built.
{
# An empty allowlist is not an empty collection: the receiver
# renders no unit filter at all and reads the host's entire
# journal, an operator's desktop session included. Fail-open, and
# silent — so the one configuration that must not be reachable is
# "logs are being shipped and nothing said which".
assertion = !collectLogs || cfg.journaldUnits != [ ];
message = ''
The swarm collector has a log destination but
services.hyperhive.swarm.otel.journaldUnits is empty. An empty
list is not "collect nothing" — it is no filter at all, and this
collector would ship every unit on the host, including any
operator user session, to a store the whole swarm can read.
Name the units worth collecting, or turn off log collection by
leaving both log destinations unset.
'';
}
{
# The assertion above covers *which* units to collect; this one
# covers whether there is a journal on disk to collect them from.
# Whether there is a journal on disk to collect from.
# A `bindMounts` entry never creates its `hostPath`, and unlike
# the CA bind source above there is no unit to order after — the
# directory exists because journald was told to store
@ -1332,8 +1280,7 @@ in
}
) cfg.publishedScrapeTargets;
}
# The host's whole journal directory, read through a unit
# allowlist.
# The host's whole journal directory, every unit in it.
#
# The directory is the host's rather than the swarm
# containers', because the host units are where the incidents
@ -1341,13 +1288,11 @@ in
# none of them is a swarm container. A source restricted to
# `swarm-*` would exclude the most-needed one.
#
# But that directory also holds every other unit on the host,
# including an operator's `user@.service` session on a hive
# that runs its services on a workstation. The allowlist is
# what separates "this host's services" from "this host", and
# only a per-unit one can: the receiver has no system-only
# switch, and its `matches` field is an allowlist too, so
# "everything except user sessions" is not expressible.
# ⚠️ That directory also holds an operator's desktop session
# on a hive that runs its services on a workstation. The
# receiver cannot exclude it — `units`, `matches` and the rest
# are all positive matches — so `filter/exclude-user-sessions`
# in this receiver's pipeline is what keeps it out of the store.
#
# Attribution is journald's either way — these fields are
# written by it rather than by the logging process. ⚠️ Use
@ -1357,7 +1302,6 @@ in
// lib.optionalAttrs collectLogs {
journald = {
directory = hostJournalDir;
units = cfg.journaldUnits;
# The SAME operator list the agent tier attaches to its own
# receiver (nix/agent-modules/otel.nix), and it transfers
# without adjustment: the entry shape is the receiver's,
@ -1365,8 +1309,8 @@ in
# `journalctl -o json`, so `PRIORITY` is spelled and typed
# identically whether the directory it was pointed at holds
# a host's journal or a container's. What differs between
# the two receivers is which journal and which units —
# neither of which the parser reads.
# the two receivers is which journal, and the parser does
# not read that.
operators = import ../journald-severity.nix;
# Persists the read cursor, so a restart resumes where
# the last run stopped. Without it the receiver starts
@ -1577,7 +1521,10 @@ in
"journald"
"otlp/${storeProducerName}"
];
processors = [ "resource/${swarmTierName}" ];
processors = [
"filter/exclude-user-sessions"
"resource/${swarmTierName}"
];
exporters = logExporterNames;
};
};
@ -1754,6 +1701,23 @@ in
action = "upsert";
}
];
}
# Drops an operator's desktop session from the host journal:
# journald stamps `_SYSTEMD_SLICE=user-<uid>.slice` on every
# record from a logind session scope and from `user@<uid>.service`.
# System units, containers and `user.slice` itself do not match.
# The store's forwarder shares the pipeline, and its container
# runs no login session, so nothing of it is dropped.
#
# `ignore`: a record whose body is not a map cannot be indexed,
# and is kept (with a warning) rather than dropped.
// lib.optionalAttrs collectLogs {
"filter/exclude-user-sessions" = {
error_mode = "ignore";
logs.log_record = [
''IsMatch(body["_SYSTEMD_SLICE"], "^user-[0-9]+\\.slice$")''
];
};
};
};
};