vinzenz/hyperhive
vinzenz/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

swarm-otel: stop restating what docs/observability.md already says

Browse source 
Two named anti-patterns from the comment-budget issue, applied to the file
that carries the most comment lines in nix/host-modules.

The header re-explained the two-tier model and the co-location argument that
`docs/observability.md` already owns — "a boundary that disappears locally is
one the local deployment stops testing" appears in both, nearly word for word.
The doc is the right home for it; what stays here is the one fact the doc does
not carry, that this is a container because `services.opentelemetry-collector`
is a singleton option already taken by the hive tier.

The `logs_endpoint` block keeps every instruction and loses the narration:
use `logs_endpoint` not `endpoint`, both spellings pass `otelcol validate`
and every wrong path answers 400, `_msg_field` is required or text search
finds nothing while ingest still answers 200, `_stream_fields` sets stream
cardinality.

Six smaller trims of "measured"/"verified"/"once". One pointed at a probe
script under a single agent's state directory — a path no other reader can
open, in source every reader loads.

What stays: `validateConfigFile`'s "a parser, not a wiring check", the
`issuer_ca_path` warning, the swarm-tier stamp. Non-obvious, in no doc, and
the target is a share of the whole tree rather than a per-file quota.

604 comment lines to 574. Comment-only: the module still evaluates to the
same 36 module-eval properties.
This commit is contained in:
atlas 2026-09-02 01:39:13 +02:00
commit e3121c1c81
1 changed files with 25 additions and 55 deletions
Download patch file Download diff file Expand all files Collapse all files

80
nix/host-modules/swarm-otel.nix
View file

@ -1,21 +1,8 @@
# The swarm's telemetry collector: one per swarm, in a `swarm-otel` # The swarm's telemetry collector: one per swarm, in a `swarm-otel`
# nixos-container beside the swarm's other shared services. # nixos-container beside the swarm's other shared services. This is the
# # **swarm** tier; `otel.nix` is the hive tier that forwards into it.
# Two tiers, and they are separate on purpose: # `docs/observability.md` owns the tier model and why the two stay separate
# # when co-located.
# - `otel.nix` is the **hive** tier. It receives from this hive's agents
# on the bridge and forwards, and it holds no upstream credential.
# - this is the **swarm** tier. It is the only holder of the upstream
# credential, the only writer to the swarm's metrics store, and the
# place that will stamp `hive=` from the authenticated connection
# rather than from anything a sender can choose.
#
# On a host that runs both, both processes run. They are not collapsed:
# all-local is a statement about *where* processes run, not about what
# shape the deployment has, and a local tier boundary that disappears is
# one the local deployment stops testing. `hive=` attribution is the
# property that would differ, and the ingest auth that makes it
# unforgeable is built on this boundary existing.
# #
# A container rather than a second host unit, for the same reason every # A container rather than a second host unit, for the same reason every
# sibling swarm service is one — and because `services.opentelemetry-collector` # sibling swarm service is one — and because `services.opentelemetry-collector`
@ -196,7 +183,7 @@ let
# invoked with `--link-journal=try-guest` for every non-ephemeral container, # invoked with `--link-journal=try-guest` for every non-ephemeral container,
# so a container's journal FILES live here, under its own machine-id # so a container's journal FILES live here, under its own machine-id
# subdirectory, and are bind-mounted into the guest rather than the other way # subdirectory, and are bind-mounted into the guest rather than the other way
# round. Measured: `journalctl -D` on this parent directory descends into # round. `journalctl -D` on this parent directory descends into
# those subdirectories, so one reader covers the host and every container. # those subdirectories, so one reader covers the host and every container.
hostJournalDir = "/var/log/journal"; hostJournalDir = "/var/log/journal";
@ -809,7 +796,7 @@ in
{ {
# A job name used by BOTH scrape options. Within one option this # A job name used by BOTH scrape options. Within one option this
# cannot happen — the module system refuses two definitions of the # cannot happen — the module system refuses two definitions of the
# same key with different values, measured — but the two options # same key with different values — but the two options
# are separate, so nothing arbitrates between them, and they # are separate, so nothing arbitrates between them, and they
# render into a single `scrape_configs` LIST where nothing # render into a single `scrape_configs` LIST where nothing
# overwrites anything: both entries ship under one `job_name`. # overwrites anything: both entries ship under one `job_name`.
@ -1107,8 +1094,7 @@ in
# VictoriaMetrics serves OTLP at # VictoriaMetrics serves OTLP at
# `/opentelemetry/api/v1/push`. With `endpoint` the # `/opentelemetry/api/v1/push`. With `endpoint` the
# collector answers 200 to its own clients and posts the # collector answers 200 to its own clients and posts the
# samples to a path that does not exist. Measured # samples to a path that does not exist.
# end-to-end, not read — `state/probe-3265-collector-to-vm.sh`.
"otlphttp/victoriametrics" = { "otlphttp/victoriametrics" = {
metrics_endpoint = metricsPushUrl; metrics_endpoint = metricsPushUrl;
} }
@ -1129,35 +1115,20 @@ in
// lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; }; // lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; };
} }
// { // {
# `logs_endpoint`, NOT `endpoint`, for exactly the reason the # `logs_endpoint`, NOT `endpoint`: the latter is a base the
# metrics exporter above spells out — and the trap is worse # exporter appends `/v1/logs` to, and VictoriaLogs serves
# here, because the two stores' OTLP routes differ. `endpoint` # `/insert/opentelemetry/v1/logs` — a different route from the
# is a base the exporter appends `/v1/logs` to; VictoriaLogs # metrics store's. Both spellings pass `otelcol validate`, and
# serves `/insert/opentelemetry/v1/logs`, which is also not # the store answers 400 to every wrong path, so neither end
# the metrics store's `/opentelemetry/api/v1/push`. Both # tells you which one you sent.
# spellings pass `otelcol validate`.
# #
# ⚠️ And the store cannot tell you either: a right path, a # 🔑 The query parameters are not tuning. The journald receiver
# wrong path and a nonsense path all answer 400. Only its log # leaves the OTLP body empty and carries the entry as journal
# distinguishes them — the real route complains about the # fields, so without `_msg_field` every record stores the
# encoding, everything else says "unsupported path requested". # literal `missing _msg field` as its message and text search
# 🔑 THE TWO QUERY PARAMETERS ARE NOT TUNING — without the # finds nothing, while ingest keeps answering 200.
# first one this pipeline fills the store with records that # `_stream_fields` is one stream per unit per machine instead
# cannot be searched by message text, which is the only # of one for the whole host.
# reason to collect logs at all.
#
# The journald receiver leaves the OTLP *body* empty and
# carries the entry as a map of journal fields, so the store
# has no message to index and writes the literal placeholder
# `missing _msg field` into `_msg` on EVERY record. Nothing
# errors: ingest returns 200, the data is all present, and a
# plain search for a line that is sitting right there returns
# nothing. `_msg_field` tells the store which field carries
# the message. Measured, both with and without.
#
# `_stream_fields` is the difference between one enormous
# stream for the whole host and one per unit per machine —
# both low-cardinality, and both fields journald sets itself.
"otlphttp/victorialogs" = { "otlphttp/victorialogs" = {
logs_endpoint = logsPushUrl + "?_msg_field=MESSAGE&_stream_fields=_HOSTNAME,_SYSTEMD_UNIT"; logs_endpoint = logsPushUrl + "?_msg_field=MESSAGE&_stream_fields=_HOSTNAME,_SYSTEMD_UNIT";
} }
@ -1170,8 +1141,8 @@ in
# default of `localhost:8888`, which the hive tier holds. # default of `localhost:8888`, which the hive tier holds.
# #
# ⚠️ `metrics.address` is the spelling that looks right and # ⚠️ `metrics.address` is the spelling that looks right and
# is REJECTED by this collector version — measured, not # is REJECTED by this collector version:
# read: `'migration.MetricsConfigV030' has invalid keys: # `'migration.MetricsConfigV030' has invalid keys:
# address`. `readers` is the schema it accepts, and the # address`. `readers` is the schema it accepts, and the
# difference is a startup failure rather than a warning. # difference is a startup failure rather than a warning.
service.telemetry.metrics.readers = [ service.telemetry.metrics.readers = [
@ -1265,9 +1236,8 @@ in
# registers it under — a second spelling here would deny # registers it under — a second spelling here would deny
# every hive, as a 401 that blames the token. # every hive, as a 401 that blames the token.
audience = "${autheliaCfg.hiveClientPrefix}${h}"; audience = "${autheliaCfg.hiveClientPrefix}${h}";
# ⛔ DO NOT ADD `issuer_ca_path` HERE. It took this # ⛔ DO NOT ADD `issuer_ca_path` HERE. The failure it
# collector down for forty minutes once, and the failure # causes is invisible to every check we have.
# is invisible to every check we have.
# #
# It loads only the FIRST certificate in the file it # It loads only the FIRST certificate in the file it
# names. The bundle assembled for this container is # names. The bundle assembled for this container is
@ -1376,7 +1346,7 @@ in
# queried as though it meant something. # queried as though it meant something.
# #
# ⚠️ Unconditional, not gated on `scrapeTargets != {} || # ⚠️ Unconditional, not gated on `scrapeTargets != {} ||
# collectLogs` (as it once was): a processor a pipeline names # collectLogs`: a processor a pipeline names
# but the config does not define is a collector that refuses # but the config does not define is a collector that refuses
# to start, and `metrics/${swarmTierName}` now ALWAYS exists # to start, and `metrics/${swarmTierName}` now ALWAYS exists
# (its `otlp/${swarmTierName}` receiver is unconditional too, # (its `otlp/${swarmTierName}` receiver is unconditional too,