From e3121c1c814cdf567b038c24d7bcd9cd5b953753 Mon Sep 17 00:00:00 2001 From: atlas Date: Wed, 2 Sep 2026 01:39:13 +0200 Subject: [PATCH] swarm-otel: stop restating what docs/observability.md already says MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two named anti-patterns from the comment-budget issue, applied to the file that carries the most comment lines in nix/host-modules. The header re-explained the two-tier model and the co-location argument that `docs/observability.md` already owns — "a boundary that disappears locally is one the local deployment stops testing" appears in both, nearly word for word. The doc is the right home for it; what stays here is the one fact the doc does not carry, that this is a container because `services.opentelemetry-collector` is a singleton option already taken by the hive tier. The `logs_endpoint` block keeps every instruction and loses the narration: use `logs_endpoint` not `endpoint`, both spellings pass `otelcol validate` and every wrong path answers 400, `_msg_field` is required or text search finds nothing while ingest still answers 200, `_stream_fields` sets stream cardinality. Six smaller trims of "measured"/"verified"/"once". One pointed at a probe script under a single agent's state directory — a path no other reader can open, in source every reader loads. What stays: `validateConfigFile`'s "a parser, not a wiring check", the `issuer_ca_path` warning, the swarm-tier stamp. Non-obvious, in no doc, and the target is a share of the whole tree rather than a per-file quota. 604 comment lines to 574. Comment-only: the module still evaluates to the same 36 module-eval properties. --- nix/host-modules/swarm-otel.nix | 80 +++++++++++---------------------- 1 file changed, 25 insertions(+), 55 deletions(-) diff --git a/nix/host-modules/swarm-otel.nix b/nix/host-modules/swarm-otel.nix index 48b5a4c4..2493f91c 100644 --- a/nix/host-modules/swarm-otel.nix +++ b/nix/host-modules/swarm-otel.nix @@ -1,21 +1,8 @@ # The swarm's telemetry collector: one per swarm, in a `swarm-otel` -# nixos-container beside the swarm's other shared services. -# -# Two tiers, and they are separate on purpose: -# -# - `otel.nix` is the **hive** tier. It receives from this hive's agents -# on the bridge and forwards, and it holds no upstream credential. -# - this is the **swarm** tier. It is the only holder of the upstream -# credential, the only writer to the swarm's metrics store, and the -# place that will stamp `hive=` from the authenticated connection -# rather than from anything a sender can choose. -# -# On a host that runs both, both processes run. They are not collapsed: -# all-local is a statement about *where* processes run, not about what -# shape the deployment has, and a local tier boundary that disappears is -# one the local deployment stops testing. `hive=` attribution is the -# property that would differ, and the ingest auth that makes it -# unforgeable is built on this boundary existing. +# nixos-container beside the swarm's other shared services. This is the +# **swarm** tier; `otel.nix` is the hive tier that forwards into it. +# `docs/observability.md` owns the tier model and why the two stay separate +# when co-located. # # A container rather than a second host unit, for the same reason every # sibling swarm service is one — and because `services.opentelemetry-collector` @@ -196,7 +183,7 @@ let # invoked with `--link-journal=try-guest` for every non-ephemeral container, # so a container's journal FILES live here, under its own machine-id # subdirectory, and are bind-mounted into the guest rather than the other way - # round. Measured: `journalctl -D` on this parent directory descends into + # round. `journalctl -D` on this parent directory descends into # those subdirectories, so one reader covers the host and every container. hostJournalDir = "/var/log/journal"; @@ -809,7 +796,7 @@ in { # A job name used by BOTH scrape options. Within one option this # cannot happen — the module system refuses two definitions of the - # same key with different values, measured — but the two options + # same key with different values — but the two options # are separate, so nothing arbitrates between them, and they # render into a single `scrape_configs` LIST where nothing # overwrites anything: both entries ship under one `job_name`. @@ -1107,8 +1094,7 @@ in # VictoriaMetrics serves OTLP at # `/opentelemetry/api/v1/push`. With `endpoint` the # collector answers 200 to its own clients and posts the - # samples to a path that does not exist. Measured - # end-to-end, not read — `state/probe-3265-collector-to-vm.sh`. + # samples to a path that does not exist. "otlphttp/victoriametrics" = { metrics_endpoint = metricsPushUrl; } @@ -1129,35 +1115,20 @@ in // lib.optionalAttrs (otelCfg.protocol == "http/json") { encoding = "json"; }; } // { - # `logs_endpoint`, NOT `endpoint`, for exactly the reason the - # metrics exporter above spells out — and the trap is worse - # here, because the two stores' OTLP routes differ. `endpoint` - # is a base the exporter appends `/v1/logs` to; VictoriaLogs - # serves `/insert/opentelemetry/v1/logs`, which is also not - # the metrics store's `/opentelemetry/api/v1/push`. Both - # spellings pass `otelcol validate`. + # `logs_endpoint`, NOT `endpoint`: the latter is a base the + # exporter appends `/v1/logs` to, and VictoriaLogs serves + # `/insert/opentelemetry/v1/logs` — a different route from the + # metrics store's. Both spellings pass `otelcol validate`, and + # the store answers 400 to every wrong path, so neither end + # tells you which one you sent. # - # ⚠️ And the store cannot tell you either: a right path, a - # wrong path and a nonsense path all answer 400. Only its log - # distinguishes them — the real route complains about the - # encoding, everything else says "unsupported path requested". - # 🔑 THE TWO QUERY PARAMETERS ARE NOT TUNING — without the - # first one this pipeline fills the store with records that - # cannot be searched by message text, which is the only - # reason to collect logs at all. - # - # The journald receiver leaves the OTLP *body* empty and - # carries the entry as a map of journal fields, so the store - # has no message to index and writes the literal placeholder - # `missing _msg field` into `_msg` on EVERY record. Nothing - # errors: ingest returns 200, the data is all present, and a - # plain search for a line that is sitting right there returns - # nothing. `_msg_field` tells the store which field carries - # the message. Measured, both with and without. - # - # `_stream_fields` is the difference between one enormous - # stream for the whole host and one per unit per machine — - # both low-cardinality, and both fields journald sets itself. + # 🔑 The query parameters are not tuning. The journald receiver + # leaves the OTLP body empty and carries the entry as journal + # fields, so without `_msg_field` every record stores the + # literal `missing _msg field` as its message and text search + # finds nothing, while ingest keeps answering 200. + # `_stream_fields` is one stream per unit per machine instead + # of one for the whole host. "otlphttp/victorialogs" = { logs_endpoint = logsPushUrl + "?_msg_field=MESSAGE&_stream_fields=_HOSTNAME,_SYSTEMD_UNIT"; } @@ -1170,8 +1141,8 @@ in # default of `localhost:8888`, which the hive tier holds. # # ⚠️ `metrics.address` is the spelling that looks right and - # is REJECTED by this collector version — measured, not - # read: `'migration.MetricsConfigV030' has invalid keys: + # is REJECTED by this collector version: + # `'migration.MetricsConfigV030' has invalid keys: # address`. `readers` is the schema it accepts, and the # difference is a startup failure rather than a warning. service.telemetry.metrics.readers = [ @@ -1265,9 +1236,8 @@ in # registers it under — a second spelling here would deny # every hive, as a 401 that blames the token. audience = "${autheliaCfg.hiveClientPrefix}${h}"; - # ⛔ DO NOT ADD `issuer_ca_path` HERE. It took this - # collector down for forty minutes once, and the failure - # is invisible to every check we have. + # ⛔ DO NOT ADD `issuer_ca_path` HERE. The failure it + # causes is invisible to every check we have. # # It loads only the FIRST certificate in the file it # names. The bundle assembled for this container is @@ -1376,7 +1346,7 @@ in # queried as though it meant something. # # ⚠️ Unconditional, not gated on `scrapeTargets != {} || - # collectLogs` (as it once was): a processor a pipeline names + # collectLogs`: a processor a pipeline names # but the config does not define is a collector that refuses # to start, and `metrics/${swarmTierName}` now ALWAYS exists # (its `otlp/${swarmTierName}` receiver is unconditional too,