vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

swarm-otel: collect only the units the swarm's services declare

Browse source 
The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.

The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.

An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
This commit is contained in:
atlas 2026-08-24 21:54:45 +02:00 • committed by mara
commit 4336436457
11 changed files with 145 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

87
nix/host-modules/swarm-otel.nix
View file

@ -336,6 +336,32 @@ in
'';
};
journaldUnits = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ ];
example = [ "nginx" ];
description = ''
systemd units whose journal this collector ships to the swarm's
log store. Nothing outside this list is collected.
Every module that defines a unit worth reading swarm-wide adds
its own names here, rather than one list naming them all: a
service that is not running contributes nothing, and a service
added later arrives with its units already declared. A central
list would be a second place that has to know which services
exist, and it would go stale in the direction that hides a
service's logs rather than the one that shows too many.
Names are matched as journald `_SYSTEMD_UNIT` values, so an
in-container unit is named exactly as it is inside its container
— `authelia`, not `container@swarm-authelia`.
⚠️ A name that matches nothing is not an error anywhere. The unit
simply never appears in the store, which looks the same as a
service that had nothing to say.
'';
};
clientId = lib.mkOption {
type = lib.types.str;
readOnly = true;
@ -362,6 +388,12 @@ in
# vhost would answer for a service it does not have.
services.hyperhive.gateway.localNames = [ cfg.domain ];
# This collector reads its own journal, so a pipeline that stops
# delivering says so in the store it stopped delivering to. That is
# less circular than it sounds: the failure that matters here is a
# single hive's receiver refusing pushes, not the process dying.
services.hyperhive.swarm.otel.journaldUnits = [ "opentelemetry-collector" ];
# OTLP/HTTP, not a browsable UI, but the same reverse-proxy shape as
# every sibling swarm service: TLS terminates here, then plain http to
# the co-located container over loopback (shared netns, like the store
@ -518,6 +550,24 @@ in
metrics store.
'';
}
{
# An empty allowlist is not an empty collection: the receiver
# renders no unit filter at all and reads the host's entire
# journal, an operator's desktop session included. Fail-open, and
# silent — so the one configuration that must not be reachable is
# "logs are being shipped and nothing said which".
assertion = !collectLogs || cfg.journaldUnits != [ ];
message = ''
The swarm collector has a log destination but
services.hyperhive.swarm.otel.journaldUnits is empty. An empty
list is not "collect nothing" — it is no filter at all, and this
collector would ship every unit on the host, including any
operator user session, to a store the whole swarm can read.
Name the units worth collecting, or turn off log collection by
leaving both log destinations unset.
'';
}
{
# Without a roster there are no receivers at all, so this
# collector would listen on nothing while looking configured.
@ -825,23 +875,32 @@ in
}
) cfg.publishedScrapeTargets;
}
# The whole journal directory, deliberately unfiltered.
# The host's whole journal directory, read through a unit
# allowlist.
#
# A filter restricted to the swarm's own containers was the
# obvious shape and is the wrong one: the host units are where
# the incidents live — the gateway's nginx, the core daemon,
# dnsmasq — and none of them is a swarm container. Filtering to
# `swarm-*` would exclude the single most-needed source.
# The directory is the host's rather than the swarm
# containers', because the host units are where the incidents
# live — the gateway's nginx, the core daemon, dnsmasq — and
# none of them is a swarm container. A source restricted to
# `swarm-*` would exclude the most-needed one.
#
# Nor would a list of container names stay true: it is a second
# place that has to know which services exist, and it goes stale
# silently the next time one is added. Attribution does not need
# it — journald's `_HOSTNAME` / `_SYSTEMD_UNIT` / `_MACHINE_ID`
# are written by journald rather than by the logging process, so
# a reader can tell the origins apart without this collector
# deciding for them.
# But that directory also holds every other unit on the host,
# including an operator's `user@.service` session on a hive
# that runs its services on a workstation. The allowlist is
# what separates "this host's services" from "this host", and
# only a per-unit one can: the receiver has no system-only
# switch, and its `matches` field is an allowlist too, so
# "everything except user sessions" is not expressible.
#
# Attribution is journald's either way — `_HOSTNAME`,
# `_SYSTEMD_UNIT` and `_MACHINE_ID` are written by journald
# rather than by the logging process, so a reader can still
# tell the origins apart.
// lib.optionalAttrs collectLogs {
journald.directory = hostJournalDir;
journald = {
directory = hostJournalDir;
units = cfg.journaldUnits;
};
};
exporters =