vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

Compare commits

base: vinzenz:3c6c2ed27228cf2c93465173bf03f7c7d18bb91c
Branches Tags
vinzenz:main
...
compare: vinzenz:57aedc82aede46ebc252ac0fc59584a9f924397d
Branches Tags
vinzenz:main
Author SHA1 Message Date
atlas
57aedc82ae swarm-otel: say what is measured about _HOSTNAME, not a mechanism that is wrong 
The previous commit's comments explained `_HOSTNAME` being `nixos` with
"no container sets `networking.hostName`", read off a `git grep` of this
tree. That grep answers a question about our source; the default is
nixpkgs': `nixos/modules/virtualisation/nixos-containers.nix` sets
`networking.hostName = mkDefault name` for every `containers.<name>`
guest, which is how every swarm service container here is declared.

So the stated cause is wrong, and why the default does not reach these
guests is still open. What is measured is narrower: an agent container
reports `nixos`, and `_MACHINE_ID` is per-machine by construction. The
comments now claim only that, and say the rest is unresolved.

The code is unchanged — `_MACHINE_ID` in `_stream_fields` is correct
regardless of which explanation turns out to be true.

Refs #4304
 2026-09-12 14:38:35 +02:00
atlas
bb38e1967c swarm-otel: key the log stream by machine, not by a hostname every container shares 
`_stream_fields=_HOSTNAME,_SYSTEMD_UNIT` was chosen to give "one stream
per unit per machine", which is what the comment above it claims. It does
not: no container sets `networking.hostName`, so `_HOSTNAME` is the NixOS
default `nixos` in every one of them, and every container's stream for a
given unit name merges into a single series.

Measured from inside an agent container:

  journalctl -n1 -o json -> "_MACHINE_ID":"5d1427ea…", "_HOSTNAME":"nixos"
  cat /etc/machine-id    ->  5d1427ea…
  git grep -i hostname -- nix/  -> 20 hits, all public vhost names,
                                   zero `networking.hostName =`

`_MACHINE_ID` is written by journald per machine and is already on every
entry, so adding it to the stream key makes the partition what its own
comment says it is. That merge is also why the host collector and the
containerised one were one series: both log to
`opentelemetry-collector.service`, and the field meant to separate them
was a constant.

The second comment touched here asserted a reader "can still tell the
origins apart" from `_HOSTNAME`/`_SYSTEMD_UNIT`/`_MACHINE_ID`. True only
of the last one; it now says so.

Not a full fix for attribution: `_MACHINE_ID` is opaque hex and nothing
maps it to a container name. Naming is the other half and is a separate
change.

Refs #4304
 2026-09-12 14:14:11 +02:00
2 changed files with 29 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

24
nix/host-modules/swarm-otel.nix
View file

@ -1106,10 +1106,13 @@ in
# switch, and its `matches` field is an allowlist too, so
# "everything except user sessions" is not expressible.
#
# Attribution is journald's either way — `_HOSTNAME`,
# `_SYSTEMD_UNIT` and `_MACHINE_ID` are written by journald
# rather than by the logging process, so a reader can still
# tell the origins apart.
# Attribution is journald's either way — these fields are
# written by it rather than by the logging process. ⚠️ Use
# `_MACHINE_ID` to tell machines apart, not `_HOSTNAME`: an
# agent container reports the NixOS default `nixos`. Whether
# every container does, and why the `networking.hostName`
# nixos-containers sets does not reach them, is unresolved —
# so rely on the field that is per-machine by construction.
// lib.optionalAttrs collectLogs {
journald = {
directory = hostJournalDir;
@ -1156,10 +1159,17 @@ in
# fields, so without `_msg_field` every record stores the
# literal `missing _msg field` as its message and text search
# finds nothing, while ingest keeps answering 200.
# `_stream_fields` is one stream per unit per machine instead
# of one for the whole host.
# `_stream_fields` is one stream per unit per machine.
# `_MACHINE_ID` is what makes that true — it is per-machine by
# construction, where `_HOSTNAME` is only as distinct as the
# hostnames happen to be, and an agent container's is the
# NixOS default `nixos`. Keying on the hostname alone merges
# the streams of any two machines sharing one, which is what
# the host and containerised `opentelemetry-collector.service`
# would do.
"otlphttp/victorialogs" = {
logs_endpoint = logsPushUrl + "?_msg_field=MESSAGE&_stream_fields=_HOSTNAME,_SYSTEMD_UNIT";
logs_endpoint =
logsPushUrl + "?_msg_field=MESSAGE&_stream_fields=_HOSTNAME,_MACHINE_ID,_SYSTEMD_UNIT";
}
// lib.optionalAttrs haveCollectorSecret {
auth.authenticator = pushAuthenticator "victorialogs";

12
nix/module-eval.nix
View file

@ -1437,6 +1437,18 @@ let
&& lib.hasInfix "metrics.t.local" m
&& lib.hasInfix "logs.t.local" l;
}
{
# `_HOSTNAME` cannot separate containers: none sets
# `networking.hostName`, so it is `nixos` for all of them and every
# container's stream for a unit name merges into one.
name = "the log stream is keyed by machine, not only by a hostname every container shares";
ok =
let
l = (otelSettings otelNoStores).exporters."otlphttp/victorialogs".logs_endpoint;
field = f: lib.hasInfix ("_stream_fields=" + f) l || lib.hasInfix ("," + f) l;
in
field "_MACHINE_ID" && field "_SYSTEMD_UNIT" && !(field "_NOSUCHFIELD");
}
{
# The collector reaches these routes through the gateway now, so each
# store needs an ingest location of its own. Without one the write rides