The swarm-tier collector's journald receiver had no storage extension, so
it started each run with no cursor: journalctl --follow --lines=0 ships
only what arrives after the receiver starts. Every collector restart
therefore dropped whatever was written to the journal while it was down,
silently — no error and no replay.
Wire the receiver to a file_storage extension, matching the agent-tier
collector in nix/agent-modules/otel.nix, so a restart resumes from the
persisted cursor instead.
Refs #4527.
The four-way client-cert split gives each store reader its own leaf, and
three of the four readers render only where their own leaf exists. On a
host that mints its own PKI glue-bao-tls.nix defaults all eight, so there
is nothing to do; on a hand-configured remote-store hive, omitting one
pair used to mean that unit silently did not render — a privilege-
narrowing unit absent from a green build, with the missing unit as the
only evidence.
Each of the three now asserts its own pair, shaped after
swarm-grafana.nix's haveClientIdentity assertion and named to the pair it
needs. What differs from Grafana's is the gate: these fire only where the
host demonstrably reads the store (it holds deploy.bao.clientCertFile and
clientKeyFile) and the consumer is on. A host with no store identity is
the supported no-store deployment and still evaluates; the collector's
no-secret degrade is untouched, because that host holds no clientCertFile
either.
Also rewords three passive-voice sentences in docs/swarm/secrets.md that
vale flagged, and documents what the refusal costs and where it stays
silent.
Four units read one path each out of the store, and all four logged in
holding `deploy.bao.clientCertFile` — the hive's own leaf. Bao identifies
a principal by the subject of the certificate it presents, so four
readers behind one certificate were ONE principal, and the only grant
expressible was the union of what the four need: read on
`swarm/agents/*`, `swarm/hives/<hive>/*` and `swarm/services/*`. The unit
fetching Grafana's OIDC client secret could fetch every agent credential
in the swarm; the one fetching this hive's matrix token could fetch
Grafana's. Least privilege was not misconfigured here, it was
unrepresentable.
Each now holds a leaf, a cert-auth role and a policy of its own, and each
policy is the single `secret/data/…` path that unit's own script names —
spelled to the leaf, not to a prefix, the way matrix-ctl's already is.
Following the four exemplars in-tree rather than building a mechanism:
`signLeaf` mints the leaves, `swarm-bao.nix` writes the roles from the
bootstrap token, the consumers name their own pair.
Two of the four are written PER HIVE and two are not, which is the shape
of the paths rather than a preference. A matrix appservice token and a
queue credential live under `swarm/hives/<name>/` and every hive runs a
reader for its own, so one role for all of them would have to be granted
`hives/*` — letting one hive read another's, a reach no hive has today.
An OIDC client secret lives under `swarm/services/<client-id>/` and a
swarm registers each exactly once, so one role each is enough. The
per-hive subjects are `<prefix>-<hive>` and swarm.nix reserves every
composed spelling as a hive name, so a hive cannot be named into another
hive's role.
The shared leaf stays: hive-c0re still passes it into its container, the
`bao` CLI wrapper still defaults to it, and the three
`glue-*-bao-identity.nix` files derive the PKI directory from it.
module-eval-bao-grants gains a negative arm per principal — each pins the
three stanzas the hive's leaf carried and the two wildcards a later
widening would reach for, so a policy that grows fails here rather than
in a store. Plus the consuming side: repointing a unit back at the hive's
leaf would evaluate, deploy and log in, and silently restore the union.
A hive that reads a store on another machine now places one leaf per
principal instead of one shared by four. That cost is the point, and
docs/swarm/secrets.md lists the pairs.
Review response on #4620: not having SSO is not a supported
deployment, so the type should not permit it, and the docs paragraph
explaining why SSO is always present is redundant once the type says
so.
- swarm.authelia.url drops types.nullOr.
- Every consumer's null-arm is gone: two option defaults
(swarm-controller's and swarm's own statusPublish.tokenEndpoint)
that produced an empty/null placeholder when the URL was null now
unconditionally compute the real derived URL. Five now-dead
"assertion = ... != null" guards (swarm-authelia's bridge,
swarm-grafana, swarm-otel, swarm-nats, hive-forge, hive-matrix) are
removed as unreachable — in every case the same URL was already
interpolated unconditionally a few lines below the guard.
- grafanaNoSso, the module-eval fixture whose sole purpose was
exercising the now-unsupported no-IdP refusal, is removed along
with its dedicated test case; swarm.authelia.url = null is a type
error now, not a value that reaches that assertion.
- docs/swarm/services.md: cut the clause about setting the option to
null and the sentence explaining why the URL is co-location-
independent — both redundant now that the type enforces it.
bao's metrics were scraped by the SWARM collector over loopback, via a
`swarm.otel.scrapeTargets.bao` entry gated on `deploy.swarm-otel.enable`
— "does the swarm's collector run on THIS host". It had to be: loopback
only reaches a reader that landed on the same host.
What that rendered everywhere else was nothing at all. Off that host the
metrics listener was not emitted, so the store's metrics reached the
store nowhere, and a host with no entry is indistinguishable from a host
nobody asked to scrape.
Moves the scrape into the collector this container already runs, per
mara on #4537: "move the existing scraper to the local collector". The
container shares the host netns (privateNetwork = false), so the scrape
still dials 127.0.0.1 — the listener keeps its address, its
`metrics_only` narrowing and its loopback-only bind, and the API
listener's `tls_require_and_verify_client_cert` is untouched.
The listener and its `prometheus_retention_time` lose their gate: the
reader ships with the store now, so there is no host where the endpoint
has none. The metrics pipeline reuses the logs pipeline's `resource`
processor and `otlphttp` exporter, so both signals carry the same
`service.name` and leave by the one hop.
Logs are unaffected: `journaldUnits` and --link-journal=host stay until
every sibling swarm container has a collector of its own.
The module-eval absence arm "a store with no collector beside it serves
no metrics" is inverted rather than dropped — the condition it asserted
is the bug. Three cases join it: the job is in swarm-bao AND gone from
swarm-otel (a move, not a copy), the scrape target and listener are both
pinned to loopback, and the metrics pipeline shares its exporter with
the logs one.
Records reached VictoriaLogs carrying the journal's raw PRIORITY and
severity_text "Unspecified" — every line in the store, at every tier, with
no level a query or a dashboard could read. VictoriaLogs has no ingest
parameter naming a level field; it auto-detects one by field name, so the
mapping has to happen in the collector.
A stanza severity_parser on each journald receiver, from one shared file
rather than a copy per tier: the two receivers are unrelated config (a
fixed stanza in the agent container, a parameterised block inside the
swarm-otel container) and a drifted copy fails silently — every line still
arrives, labelled as the wrong thing.
Two details that are easy to get wrong and quiet when wrong. PRIORITY
counts down in urgency where the OTEL severity counts up, so the table is
written as a table. And overwrite_text is required: without it the parser
sets the severity number and leaves the text as the raw digit, so
severity_text arrives as the literal "6" — populated, and not a level
anything renders.
Two swarm-wide facts were being read off this machine's deploy set, so the
answer differed between two hosts of one swarm:
- `swarm.authelia.oidc.hiveIdentities` defaulted to `deploy.nats.enable`,
so whether a hive gets an identity at all depended on whether the IdP
host happened to also run the queue. It is on by default now: a swarm's
hives have identities, and the clients are inert until used.
- `swarm.statusPublish.tokenEndpoint` defaulted through `queueLocal`
(`deploy.nats.enable && deploy.authelia.enable`), so a hive that was not
the swarm host had no token endpoint even when the swarm's IdP was
reachable and named. It follows `swarm.authelia.url` now — the same
derivation `swarm-controller.nix`'s own `queue.tokenEndpoint` already
uses, which is correct for a remote provider.
`deploy.nix:1-30` is what makes this a rule rather than a preference:
`swarm.*` is "identical on every host, byte for byte" and `deploy.*` is
"necessarily different on every host". A swarm value derived from a deploy
value cannot satisfy both.
The all-or-nothing status-publish assertion follows: the token endpoint is
no longer one of the coordinates that says this hive publishes — every hive
in a swarm with an IdP has one — so the two per-host coordinates are what
must agree, and they now require the endpoint rather than being counted
beside it.
`queueLocal` itself stays for the three remaining host-local addresses
(`natsUrl`, `clientSecretFile`, `agentNatsUrl`): each of those is a
`deploy.*` value that genuinely differs per host.
Closes#4048
`services.hyperhive.gateway.enable`, `gateway.dns.enable` and
`network.enable` replace the `hyperhive.enable` gate on all three. Each
defaults to false; the modules that need one assert it with `mkDefault
true` from inside the guard their own deployment already carries, and
`swarm-required-services.nix` — the module that owns what the
swarm-services toggle implies — asserts all three explicitly.
hive-c0re asserts all three unconditionally, so an ordinary hive keeps
getting them with no opt-in: it is the host's only knowledge that agent
containers exist.
The resolver moves to its own `hive-gateway/dns.nix` so it can be gated
without reindenting the nginx half of the module.
Reinstates `network.enable`, dropping its `mkRemovedOptionModule` shim.
A config still carrying `network.enable = false` from before the removal
now switches the bridge off instead of failing eval.
Also deletes a duplicate `centralToggleOff` fixture in nix/module-eval.nix.
Two sibling slices added it independently (c5f60fd5, ce3b3d94); the merge
was textually clean and left `main` failing to evaluate at all, so this
file could not be gated without removing one.
The receiver was defined for `scrapeTargets != {} || publishedScrapeTargets
!= {}`, but the swarm-tier pipeline named it only for `scrapeTargets != {}`.
A collector with published targets and no loopback ones therefore rendered
scrape configs that belonged to no pipeline: requested, parsed, delivered
nowhere, and valid enough to deploy.
Latent, not live: the module seeds `scrapeTargets.collector` under its own
`enable`, so the loopback set is never empty in a real deploy and the
disagreeing arm is unreachable today. Nothing would have noticed if that
seeding became conditional, which is why the module-eval case pins the
pipeline rather than trusting the seed to keep masking it.
Refs #4511
`services.hyperhive.enable` is going away (Refs #4500). These four blocks
ANDed it with a second condition that was already the load-bearing one:
the bridge-firewall hole asks whether an operator named any
`network.exposeHostPorts`, and the three swarm-level services ask their
own `deploy.*.enable`, each of which defaults false (swarm-ui derives
from the controller's toggle, a sibling deployment decision) and none of
which is derived from the hive toggle. So dropping the conjunct turns
nothing on by itself — the remaining condition still decides.
The way that claim fails is by something becoming unconditional, so
module-eval gets absence arms for all four, each asserted on a host with
the hive ON and on one with it OFF, plus the two controls that make the
absences mean something.
The swarm-controller arm is probed by the credential oneshot and by the
daemon's `ExecStart` rather than by the unit name: hive-tls defines an
environment key on that name, which leaves an inert fragment behind on
any hive with a CA whether or not the controller runs there.
The swarm collector's OIDC client secret only existed where authelia
did: `swarm-otel-oidc-secret.service` copied the minted plaintext out
of authelia's container tree, reachable only because the two share a
host's network namespace. A swarm that placed authelia elsewhere
delivered nothing, and the option's own description said so —
"a deployment that places authelia elsewhere points this at a file it
delivers itself." Same gap as #3853 and #4234, and this is the
swarm-otel twin of #4234's fix for Grafana.
Mirrors PR #4361 (Grafana) almost exactly:
- `swarm-bao-otel-oidc.service` reads
`swarm/services/<client-id>/oidc/client` out of the store, in every
deployment, replacing the co-located copy unit outright — one
delivery route, not two, per the ruling that landed under #4234.
- Client registration moved out of `swarm-otel.nix`'s own `config`
block (gated on this host running the collector) into
`glue-swarm-otel-oidc-client.nix` (gated on this host running
authelia), the same split `glue-grafana-oidc-client.nix` made. It
was broken the same way: a split deployment registered the client
nowhere at all, so authelia never minted a secret for the publisher
to send on.
- The publisher's `services` prefix (write grant in `swarm-bao.nix`,
hive read grant in `policy::render`) already covers any service's
path — nothing to add there. `swarm-secret-publisher.nix` only grew
`serviceClientIds` by one entry.
One judgement call, stated rather than buried: the store-reading unit
renders only where this host holds a client identity
(`deploy.bao.clientCertFile`/`clientKeyFile`), rather than asserting
it the way `swarm-grafana.nix` does. Grafana's local login form is
disabled unconditionally, so a Grafana with no OIDC secret has no way
in at all — that earns a hard refusal. This collector without a
credential still receives every hive's telemetry; only its own pushes
to the stores go out unauthenticated and get refused there, an
already-supported degrade the module's own `haveCollectorSecret` flag
named before this change. So the reading unit follows the shape
`glue-matrix-bao-token.nix` and `glue-queue-agent-credential.nix` use
for their own optional readers: no unit when the identity is absent,
not a build refusal.
Fixtures mirror #4361's: `otelBaoWithAuthelia`/`otelBaoRemoteAuthelia`
are the positive pair (co-located and split, both reading through the
store), `otelNoIdentity` is the negative — no reading unit, no
assertion firing, `clientSecretFile` left null.
Refs #4258
autheliaCfg/forgeCfg/vmCfg/vlCfg were local re-exports of
hyperhiveCfg.swarm.<subpath> — a shape that hides the full option
path from grep, the same hazard #4356 fixed for the tls alias.
Removed the six let-bindings and spelled the full
hyperhiveCfg.swarm.<subpath>.<field> path at every use site instead.
None of the read fields (url, machine, hiveClientPrefix,
agentClientSuffix, domain, port) sit on the old side of a
mkRenamedOptionModule in deploy.nix, so inlining the alias's own
current path is correct as-is.
Refs #4363
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every agent container reports the hostname `nixos`, so every log line it
ships carries that as its `_HOSTNAME`. Measured: host `muede-lpt2`,
`hive-matrix` (declared as `containers.hive-matrix`) `hive-matrix`, and
`h-atlas` `nixos`.
nixpkgs sets the hostname in the merge function of the
`containers.<name>.config` option (nixos-containers.nix:524), so it reaches
a guest evaluated through that option and nothing else. Agent containers are
`nixos-container create --flake meta#<name>` — an independent `nixosSystem`
off the meta flake, which never evaluates that submodule.
nspawn also names a container's hostname after the machine by default; that
is ruled out as the source here, because `h-atlas`'s machine name is
`h-atlas` and it reports `nixos`.
The machine name rather than the logical one: `stats/otel_metrics.rs:345`
already labels metrics `container.name = "h-<name>"`, so the logical name
would make logs say `atlas` while metrics say `h-atlas` — a prefix transform
on every join between the two signals. Declarative containers and nspawn
both use the machine name too, so this is one rule with no exception for
agents.
The emission sits next to `hyperhive.user.name = name;`, which already
derives the container's unix user from the agent name; the hostname was the
one identity attr nobody wired.
Also drops the prose in swarm-otel.nix and module-eval.nix that explained
`_MACHINE_ID` by "every container is `nixos`" — that motivating example is
what this commit removes, and the argument for `_MACHINE_ID` never depended
on it.
Checked before editing: nothing in the tree assumes the hostname is
"nixos" (0 hits across *.rs and *.nix), and nothing reads the hostname at
runtime.
Gate: cargo fmt 0, clippy -D warnings 0, cargo test -p hive-c0re meta::
0 (23 run, 22 passed, 1 ignored), nix fmt 0 changed.
Refs #4304
The previous commit's comments explained `_HOSTNAME` being `nixos` with
"no container sets `networking.hostName`", read off a `git grep` of this
tree. That grep answers a question about our source; the default is
nixpkgs': `nixos/modules/virtualisation/nixos-containers.nix` sets
`networking.hostName = mkDefault name` for every `containers.<name>`
guest, which is how every swarm service container here is declared.
So the stated cause is wrong, and why the default does not reach these
guests is still open. What is measured is narrower: an agent container
reports `nixos`, and `_MACHINE_ID` is per-machine by construction. The
comments now claim only that, and say the rest is unresolved.
The code is unchanged — `_MACHINE_ID` in `_stream_fields` is correct
regardless of which explanation turns out to be true.
Refs #4304
`_stream_fields=_HOSTNAME,_SYSTEMD_UNIT` was chosen to give "one stream
per unit per machine", which is what the comment above it claims. It does
not: no container sets `networking.hostName`, so `_HOSTNAME` is the NixOS
default `nixos` in every one of them, and every container's stream for a
given unit name merges into a single series.
Measured from inside an agent container:
journalctl -n1 -o json -> "_MACHINE_ID":"5d1427ea…", "_HOSTNAME":"nixos"
cat /etc/machine-id -> 5d1427ea…
git grep -i hostname -- nix/ -> 20 hits, all public vhost names,
zero `networking.hostName =`
`_MACHINE_ID` is written by journald per machine and is already on every
entry, so adding it to the stream key makes the partition what its own
comment says it is. That merge is also why the host collector and the
containerised one were one series: both log to
`opentelemetry-collector.service`, and the field meant to separate them
was a constant.
The second comment touched here asserted a reader "can still tell the
origins apart" from `_HOSTNAME`/`_SYSTEMD_UNIT`/`_MACHINE_ID`. True only
of the last one; it now says so.
Not a full fix for attribution: `_MACHINE_ID` is opaque hex and nothing
maps it to a container name. Naming is the other half and is a separate
change.
Refs #4304
A collector's `refused` / `failed` / queue-depth counters are the only
signal that says telemetry is being dropped, and nothing read them at
any tier — so a collector losing records looked exactly like a quiet
system.
The hive tier could not be scraped without first naming its port. 8888
is the collector's built-in default and appeared in no config, which is
also why nothing comparing configured ports could see it clash with a
co-located collector — swarm-otel.nix sidesteps 8888 by hand for that
reason, and says so. Declaring the port and binding it explicitly makes
the value comparable; wiring the scrape is then one entry per tier.
Extending the port-collision assertion to cover it is deliberately left
out: that belongs with the other port work, and coupling a collision
fix to a scraping fix makes both harder to review.
Gate: 101 module properties hold, was 95. The six cases pin the rendered
scrape job rather than the option; the metrics pipeline naming the
prometheus receiver, a path never emitted on any hive before this since
the hive tier's scrapeTargets was empty everywhere; the `readers`
spelling, with a control so a missing telemetry block cannot pass the
port check vacuously; the swarm tier's own entry; the two tiers not
claiming the same port; and the absence arm, a hive with no collector
declaring no target.
scrapeTargets' description said "Empty by default, and that is the
shipped case". This makes that false, so the paragraph moves with it.
The two hive-name guards lived in swarm-otel.nix, inside its
`config = lib.mkIf (… && deployCfg.swarm-otel.enable)`. A swarm running the
secret store and the controller but no collector therefore had no hive-name
check at all, while the names were still composed into OIDC client ids, bao
policies and cert-auth roles exactly the same way. They move to swarm.nix,
which declares `swarm.hives` and is unconditional. swarm-otel keeps the
assertion that its own entry is still in the shared list — that one is about
this module's stake in a file it no longer controls.
The equality guard also takes the store's cert-auth subjects now. Cert auth
trusts the CA, so `allowed_common_names` is the whole of what narrows a role
to one identity, and the same CA signs every hive's leaf with the hive's name
as its CN. A hive named after a role's subject presents a certificate that
role accepts, which for the controller is write access to every hive's
credentials and policies.
A list rather than the one string, because the next role added beside it
widens what a hive name must not collide with, and because the subject is an
option an operator sets — a literal deny entry covers the default and nothing
else.
Four module-eval cases, two of them controls. The fixture overrides the
subject to `ctl` on purpose: the default contains `swarm`, which the substring
guard catches whatever the new arm does, so a fixture using it could not tell
the two apart. The controls are that a legal roster trips neither guard, and
that all three fixtures really do have the collector disabled — without the
second, every case would pass while testing the arrangement they exist to
rule out.
An agent container writes a complete journal — 991 MB and nine days deep
on this hive — that nothing outside it can read: the host-side
per-container journal directory is an id-mapped bind mount, and journald
writes nothing into it. So the reader has to run inside the container,
and the path it would push to did not exist.
Three tiers, one vertical slice, because any two of them alone are
silent:
- the agent container gains an `opentelemetry-collector` with a
`journald` receiver aimed at its own journal and an exporter aimed at
the same base address every in-process producer already exports to.
- the hive collector gains `service.pipelines.logs`. Without it the
`otlp` receiver answers 404 on `/v1/logs` — measured, and
indistinguishable from a route that was never meant to exist.
- the swarm collector gains a per-hive `logs/<hive>` pipeline beside
`metrics/<hive>`. Without it the push is accepted, answered 200, and
routed nowhere.
The receiver's `directory` is stated rather than inherited, and that is
the load-bearing line: its default is the RUNTIME journal
(`/run/log/journal`), which in an agent container is empty. Left at the
default this whole path validates, starts, reports healthy and forwards
nothing. The assertion beside it covers the same silence from the other
end — a `volatile` or `none` journald storage empties the directory the
receiver reads.
Attribution follows the tier that can prove it. The forwarder stamps
`agent`, which no host-side reader could supply; `hive` is deliberately
left to the swarm tier, which upserts it from whichever receiver
accepted the sample, precisely so the label comes from something the
sender cannot write.
No `units` allowlist, unlike the swarm tier's journald receiver. That
one needs one because the host's journal also holds an operator's own
session; a container's journal is the harness and what the harness
spawns. Measured volume is 20827 entries / 6.3 MB per agent per day,
with nothing logging below `info` — so the receiver's `info` default
filters nothing and there is no bill to justify a knob.
Agent containers only, per the ruling on the issue: swarm services need
one forwarder per service container and get re-measured once this works.
Part of #3940.
Slices 8 and 9 of the swarm/deploy split, and the last two.
statusPublish had three coordinates under one namespace. Two of them
are this machine's — where the queue listens *as seen from here*, and
where its client secret sits on this disk — so they move to
`deploy.hive-controller.statusPublish.*`, the namespace of the daemon
that is their only reader. `tokenEndpoint` is the swarm's one address,
so it stays. That leaves `swarm.statusPublish` holding a single option:
a legitimate split, not a botched move.
The all-or-nothing assertion now spans both namespaces. It is repointed
in both its condition and its message, and the message spells all three
paths in full so an operator is never told to set two options under a
path that only has one. `environment.nix`'s guard and the value beside
it likewise read different namespaces on purpose.
The collector's secret moves the same way, for the same reason, to
`deploy.swarm-otel.*` — `enable` already lives there. That also retires
one of the eight cross-namespace assignments tracked in #4048: the
delivery unit set a `swarm.*` value under a `deploy.*` gate, and now
sets a `deploy.*` value under one.
module-eval gets a fixture per slice. `otelRemoteAuthelia` already set
the collector secret through its pre-rename path, so it becomes slice
9's old-path case as it stands — left spelled that way deliberately,
with a comment, so it is not read later as a missed site.
That fixture also turned out to be describing an impossible hive: it
said authelia lives elsewhere without saying where, so the
authenticator interpolated a null `swarm.authelia.url` into its
`token_url`. Nothing to do with the rename, and invisible to the
existing case over the same fixture, which reads `? auth` and `elem` —
both stop at names and never force the extension's value. Given the
address a remote-IdP deployment has.
Verified: 49 -> 51 properties, all holding.
`usersFile`, `hostClientSecretDir` and `hostUsersFile` are filesystem
paths that only exist on the machine running the `swarm-authelia`
container. A hive that does not run it has nothing at any of them, so
they fail the swarm-wide test the namespace's header states.
`usersFile` is a path *inside* the container and still belongs on the
deploy side: a path's scope is the scope of the filesystem it names, and
that filesystem is this host's container root.
The two `host*` options are `readOnly` and derived, so nothing can set
them through the rename shims — those serve readers. The shims are still
required: `mkRenamedOptionModule` is what keeps an out-of-tree module
reading the old path resolving at all.
Seven reads went through an alias rather than a full path
(`autheliaCfg.hostClientSecretDir` in five modules, `.hostUsersFile` in a
sixth, `swarmCfg.authelia.hostClientSecretDir` in a seventh). Every one
of those files already binds `deployCfg`, so the repoint needed no new
alias. Four more sites were prose, including one in `docs/`, which no
grep restricted to `nix/` would have found.
The swarm collector reads /var/log/journal and has never seen a single
container unit. mara's count-by-unit against VictoriaLogs returns four
units, all host-tier; openbao -- which runs inside the swarm-bao
container -- is absent.
Cause: nixos-containers.nix hardcodes `--link-journal=try-guest` for
every non-ephemeral container. With `guest`, the host's
/var/log/journal/<machine-id> is a SYMLINK into the container's transient
root; a reader in the host namespace cannot follow it, and it dangles as
soon as the container stops. `ls -la /var/log/journal/` on the host shows
one real directory and a pile of `-> /tmp/nspawn-root-*` links dating
back to May.
swarm-otel.nix asserted the opposite, and that assertion is why the
receiver's path was considered sufficient: it said the files "live here"
and are "bind-mounted into the guest rather than the other way round".
That describes `--link-journal=host`. The same sentence names the flag we
actually use. The flag was right and the behaviour it described was not,
so grepping for the flag confirmed the comment and taught nothing.
`containers.<name>.extraFlags` feeds EXTRA_NSPAWN_FLAGS, which the
invocation expands after the hardcoded flag, so `--link-journal=host`
wins. The comment now describes what the code does instead of the other
way round.
Two payoffs, and the smaller one is the one the issue is about:
container logs become collectable, and -- independently -- they become
durable at all, rather than dying with the container.
Ten identical edits because ten host-modules hand-roll their own
container block; that duplication is #3773, not something to invent an
abstraction for here.
NOT VERIFIED: that systemd-nspawn honours the last `--link-journal` of
two. Everything else here is read out of nixpkgs; that step is a claim
about its argument parsing which cannot be exercised without starting a
container. It is settled by deploying one and re-running the `ls`: the
machine-id entry becomes a real directory instead of a symlink.
Refs #3849
`docs/` was reorganised into topic subdirectories and the references to it
were not moved with it. Thirteen distinct paths and three relative links no
longer resolved, spread across nix, css, html, js, markdown and
.prettierignore — a stale pointer is not a markdown problem, it is wherever
someone wrote a path down.
Each mapping resolved to exactly one target. `docs/matrix.md` was the sole
ambiguous basename: .prettierignore lists `docs/tools/matrix.md` separately
and that entry still resolves, so the stale one is the integrations doc.
The three relative links were each one `../` too deep — from `<crate>/src/`
two levels reach the repo root. `hive-agent/src/login.rs` already had the
correct form, in the same crate, at the same depth.
.prettierignore is repointed rather than dropped, though nothing in the tree
runs prettier: no treefmt entry, no CI job, no package script. Whether that
config should exist at all is a separate question from whether it names
files that do.
The collector bind-mounts /var/log/journal into its container and points
its journald receiver at that directory. journald only writes there when
it stores persistently: with storage = "volatile" the journal lives in
/run/log/journal, and with "none" there is none at all. Nothing in the
tree sets or asserts services.journald.storage, so a host with either
value deploys a collector that cannot start — nixos-container refuses to
start when a bind source is missing, which is the same trap the
swarm-bao-certs and hostClientSecretDir comments document.
A bindMounts entry never creates its hostPath, and unlike every other
runtime-written bind source in the tree there is no unit to order the
container after: each of those declares before + requiredBy on its
producing unit, and this directory has no producer. journald was told to
store persistently or it does not exist. That leaves an assertion as the
only place to say so.
"auto" is deliberately accepted: it uses the directory when it exists,
and evaluation cannot see whether it does. Certainty is what separates
an assertion from a warning, and only "volatile" and "none" are certain.
Verified by reading config.assertions for all three values rather than
system.build.toplevel, which raises the first failing assertion in the
whole config and so reports a minimal stub's unrelated ones instead:
persistent -> mine [] others 3
volatile -> mine [the message, quoting "volatile"] others 3
none -> mine [the message, quoting "none"] others 3
The others count is identical across the three, so the storage value is
the only thing that differs between them, and a populated list is what
makes the empty one mean "this assertion passed".
Two named anti-patterns from the comment-budget issue, applied to the file
that carries the most comment lines in nix/host-modules.
The header re-explained the two-tier model and the co-location argument that
`docs/observability.md` already owns — "a boundary that disappears locally is
one the local deployment stops testing" appears in both, nearly word for word.
The doc is the right home for it; what stays here is the one fact the doc does
not carry, that this is a container because `services.opentelemetry-collector`
is a singleton option already taken by the hive tier.
The `logs_endpoint` block keeps every instruction and loses the narration:
use `logs_endpoint` not `endpoint`, both spellings pass `otelcol validate`
and every wrong path answers 400, `_msg_field` is required or text search
finds nothing while ingest still answers 200, `_stream_fields` sets stream
cardinality.
Six smaller trims of "measured"/"verified"/"once". One pointed at a probe
script under a single agent's state directory — a path no other reader can
open, in source every reader loads.
What stays: `validateConfigFile`'s "a parser, not a wiring check", the
`issuer_ca_path` warning, the swarm-tier stamp. Non-obvious, in no doc, and
the target is a share of the whole tree rather than a per-file quota.
604 comment lines to 574. Comment-only: the module still evaluates to the
same 36 module-eval properties.
The collector's store exporters were gated on deploy.victoriametrics.enable /
deploy.victorialogs.enable — options that say "this host RUNS the store", not
"the swarm has one". A collector that did not share a host with the stores
rendered no exporter at all and dropped everything it received, from every
hive, silently: an absent exporter is not an error.
mara ruled the premise rather than the mechanism ("the swarm always has a
store"), so there is no gate and no new option for that. Both exporters are
unconditional and address the stores by domain, per the cross-host rule from
the OIDC client and secret-delivery unit #3517 already built. The logs exporter
had the identical bug and is fixed here too.
Both stores gained a machine ingest route, required in the same change: the
exporter now targets https://logs.<domain>/insert/..., and that vhost is
browser-shaped, so shipping the collector half alone would have regressed logs
ingestion that works today. Neither ingest location carries
`error_page 401 =302` — a pusher handed a redirect follows it and POSTs at a
login page, which answers 200.
Whether the collector authenticates follows the CREDENTIAL, never another
service's placement: `clientSecretFile` is a nullable option, and the delivery
unit — the one thing here that may know where authelia runs, since it copies
out of its container — sets it by mkDefault. An earlier revision gated this on
deploy.authelia.enable directly, which put a different service's co-location in
the collector's own config.
Also removed rather than relaxed: the assertion that this collector has
"somewhere to send". It read the store's per-host enable, so it rejected at
eval exactly the deployment reaching the stores by domain exists for.
Deliberately not replaced with an authentication assertion — a collector on a
host of its own is a supported shape, and refusing to build it would make this
fix illegal where the bug bites hardest.
Knock-on worth review: collectLogs is now always satisfied, so journald
collection is unconditional.
Config shape validated against otelcol-contrib 0.151.0 `validate`, with a
bogus-key control confirming the validator checks the extension schema.
module-eval: 31 properties.
The store's scrape target rendered as `host:port` alone, so the collector
requested `/metrics` — a path openbao does not serve on any listener. Every
scrape since the target landed has 404'd, which reads as a dead exporter
rather than a wrong address, and no bao sample has ever reached the store.
`scrapeTargets` values may now carry a path and query. Both are optional and
omitted when absent, so every existing target renders the config it rendered
before. The query cannot ride in `metrics_path`: prometheus percent-encodes
the `?`, so it has to become `params`.
Measured against openbao 2.6.2 and otelcol-contrib 0.151.0, the pinned
versions: `/metrics` 404s ("this listener only serves the metrics endpoint"),
`/v1/sys/metrics` answers JSON, and only `?format=prometheus` — or an Accept
header the collector happens to send today — returns exposition text. The
query param is the half that does not depend on content negotiation.
Her review: too much text, and an assertion helper instead of that much code
where the file should just say "this option cannot contain that".
`lib/name-guards.nix` gets `mustNotEqual` / `mustNotContain`; both call sites in
swarm-otel become four fields each — the option, the names, the list, and the
sentence explaining the failure. The message plumbing (find every offender,
quote them, name the list) is one shape shared by both.
Offenders are printed after their label rather than before it, because no
English verb agrees with both one name and five.
Converted the pre-existing equality assertion too. Leaving one hand-rolled
beside one helper-built is the worse of the two outcomes, and it is the same
guard.
Verified by evaluating both guards directly rather than only parsing: clean
rosters pass, `foo-agent`/`beehive` fail containment, `swarm` fails equality,
and `forgeworks`/`operator-hq` still pass — the control that keeps the two
lists from being merged.
Hive-scoped identifiers are built by joining a hive name with fixed words —
`hive-<name>`, `hive-<name>-agent` — so a hive called `foo-agent` produces
exactly the id hive `foo`'s agent containers produce. The queue's responder
resolves it as the agents, and that hive silently receives an agent grant
instead of its own; a NATS denial reaches a client as a timeout, so nothing
names the cause. Equality cannot see this: the two NAMES differ, only the
composed strings collide.
Deliberately a second file rather than three more entries in
`reserved-names.nix`. That list is matched by equality against a whole name and
every entry is a value some component emits; applying containment to it would
refuse `forgeworks` and `operator-hq` for failures that do not exist. The two
lists differ in subject (hive-only vs both), matcher (substring vs equality)
and admission rule, and each file's header says so.
No Rust change: `HIVE_RESERVED_NAMES` answers "may an AGENT be called this",
which stays an equality question — an agent name is never composed into these
identifiers.
Measured against the predicate rather than assumed:
pr1ma=ok alpha=ok forgeworks=ok operator-hq=ok
foo-agent=REFUSED beehive=REFUSED swarm=REFUSED myswarmx=REFUSED
The four `ok` cases are the control: `forgeworks` and `operator-hq` are exactly
what a merged list would have refused.
Per review: `deploy.otel` does not imply swarm level, and there is a
hive-tier collector too -- `services.hyperhive.otel.enable`, which every
hive runs. The tier is the entire distinction between the two, so the
name has to carry it, matching `deploy.swarm-controller` and
`deploy.swarm-ui`.
15 swarm-tier references renamed across 8 files. The hive-tier
collector's 16 references are deliberately untouched, verified as a
control on the same command.
Three spellings needed three different patterns, all inside this one
rename: the dotted path; the `mkRenamedOptionModule` target written as a
nix LIST (`[ ... "deploy" "otel" "enable" ]`), which no dotted grep can
match; and prose in docs/observability.md spelled `deploy.otel` with
neither `.enable` nor a leading dot. Unanchored `deploy\.otel\b` is the
only pattern that finds all three.
Two corrections from review, applied forward on this branch rather than
by rewriting it.
`deploy.<service>` was a bare bool, which makes
`deploy.forgejo = { enable; ci; }` unrepresentable -- the nested
CI-runner sub-option this namespace was designed around. Every entry is
now an attrset with an `enable`, so a second per-host deployment
decision becomes an ordinary addition rather than a migration.
`deploy.controller` is now `deploy.swarm-controller`, consistent with
`deploy.swarm-ui`, which was introduced in the same commit.
89 references rewritten across 24 files -- nix, Rust, docs, and the
repo's own CLAUDE.md.
The prefix-anchored sweep missed exactly one, and it was live code:
hive-tls.nix spells it `hyperhiveCfg.deploy.controller` -- the only
`hyperhiveCfg` prefix among 45 references. A suffix grep
(`\.deploy\.<name>`) finds it; a path-anchored one cannot, because the
head of a reference is whatever alias the reading file happens to bind.
The one move where the suffix grep is the wrong instrument. There are two
otel options a word apart — `swarm.otel.enable` (one per swarm) and
`otel.enable` (one per hive, every hive runs it) — so `\.otel\.enable`
matches twenty-five references of which most must not change. The module
already carries a comment warning about exactly this, on a line that
names `swarm.otel` in full rather than through a binding.
Triaged by hand and confirmed the other way round: after the move, a grep
for the per-hive option still finds it in the files that should keep it.
Also worth recording what the alias sweep cannot do. `swarm.nix` reads
this as `swarmCfg.otel.enable`, where `swarmCfg = cfg.swarm` and `cfg =
config.services.hyperhive` — an alias bound to an alias, two hops from
the option path. No syntactic sweep resolves that chain, which is why the
module system's own evaluation is the only complete check here and the
static sweeps are a way to narrow the work, not to finish it.
The largest of these moves: sixteen references spelled through `let`
aliases across eight modules, plus eight more spelled as a path, plus
five documentation pages.
authelia is also the clearest case for why the two namespaces exist.
`swarm.authelia.url` is needed by *every* hive in the swarm — it says
where to send a browser to authenticate — while running the container is
the business of exactly one host. The client half and the server half
were sharing a namespace whose whole contract is "identical everywhere",
and only one of them could honour it.
`swarm.authelia.oidc.clients` stays where it is for the same reason:
several modules register a client there, gated on authelia running here,
and the registry itself is what the service *is* rather than a decision
about this machine.
One sweep note worth recording: a grep for `swarm.authelia.enable` misses
`swarmCfg.authelia.enable`, because the prefix is whatever the reading
file bound. Grepping the suffix `.authelia.enable` finds both, and found
a reference in swarm.nix that the path-shaped pattern did not.
Same move as grafana, and the three belong together: they derive from one
switch and a store with no UI is as useless as a UI with no store.
`victorialogs` is the case that shows why the option-path sweep is not
enough on its own. It has **zero** references spelled
`swarm.victorialogs.enable` anywhere in the tree, and four spelled
through `let` aliases (`vlCfg.enable` in the collector, `cfg.enable` in
its own module). A sweep for the path would have reported nothing to do
and left every reader broken.
Prose moved with the code rather than being left behind: the comments in
swarm-required-services.nix that explained why the pair derives together
now sit above the assignments that do it, instead of above the gap where
they used to be.
One list, in nix/reserved-names.nix, handed to everything that needs it
as HIVE_RESERVED_NAMES. Keeping it current becomes a config change
rather than a rebuild, and hive names and agent names -- one namespace
going forward -- are checked against the same file: swarm-otel.nix's
hand-written reservedOwners is gone.
Whitespace-separated rather than JSON, deliberately, unlike the
structured env vars beside it. Every entry is an Ident ([a-z0-9-]), so
whitespace cannot occur inside a name and the encoding is provably
lossless; JSON would mean either a parser dependency in a crate whose
purpose is to have none, or a copy of the parse in every consumer.
An UNSET variable is not "nothing is reserved". Both creation sites log
an error and return a warning saying the check did not run, so a
misconfigured deployment says so instead of silently accepting every
name. A blank value folds into unset: nix always renders a non-empty
list, so present-but-empty is a rendering fault, not a declaration.
Two guards whose subject moved out of their own file now assert their
own case is still in it, because a guard that can be retired by an edit
elsewhere is not a guard:
- swarm-otel.nix asserts reserved-names.nix still contains its
swarmTierName.
- hive-sh4re's sentinel drift test PANICS when the variable is missing
rather than skipping -- a drift test that quietly does nothing still
reports green. checks.nix and devshell.nix both export it so CI and a
local cargo test agree. Verified as a pair: with the variable set, 8
tests pass; with it unset, exactly the 4 drift tests fail and the
unrelated ones still pass.
The collector's client is registered for authelia.bearer.authz, but the
prometheus scrape asked for no scopes, so every token came back carrying
none and authelia refused it at introspection with 'the requested scope
is invalid, unknown, or malformed'. Forgejo metrics were never scraped.
The rule was already stated one field below, for the audience: registered
is not requested. The two travel together, and a config read cannot see
the one that is missing -- every inspection of the rendered config came
back correct, because what was present was correct.
The journald receiver was configured with a directory and no filter, so
the swarm's log store received every unit on the host that runs the
collector. On a hive whose services live on a workstation that includes
the operator's desktop session, in a store every swarm operator can read.
The receiver has no system-only switch and its `matches` field is an
allowlist too, so what the swarm collects has to be stated rather than
excluded. Each service module names its own units: a service that is not
running contributes nothing, and one added later arrives declared.
An empty list is fail-open — the receiver renders no filter at all and
reads everything — so it is asserted against.
nixpkgs' collector unit already sets SupplementaryGroups to systemd-journal
unconditionally, with a comment saying why. Systemd list options concatenate,
so this module's copy rendered ["systemd-journal" "systemd-journal"] and made
this a second owner of a fact upstream may later change.
The bind mount stays, since that half is genuinely ours.
The logs half was gated on the swarm's log store being enabled, so turning
that store off stopped collection entirely rather than leaving the upstream
export. Logs now fan out exactly as metrics do: the store when it runs, the
operator's upstream when one is configured, both when both.
The receiver, the journal mount and the group grant follow whether there is
anywhere to send logs, not whether the local store exists. Tying the mount to
the store instead would render a receiver that can read nothing.
The journald receiver leaves the OTLP body empty and carries the entry as a
map of journal fields, so VictoriaLogs had no message to index and wrote a
placeholder into _msg on every record. Ingest returned 200, every field was
present, and a plain search for a line sitting in the store found nothing.
_msg_field names the field that holds the text. _stream_fields is the
difference between one stream for the whole host and one per unit per
machine; both are set by journald itself and both are low-cardinality.
A journald receiver reading the host's journal directory, a logs pipeline
stamped with the swarm tier's own resource processor, and an otlphttp
exporter pointed at VictoriaLogs. All four parts are conditional on the log
store being enabled, so a swarm without one renders exactly as before.
The host's directory is enough to see every container: nspawn links a
non-ephemeral container's journal guest-side, so the files live on the host
under the container's machine-id, and journalctl descends into those
subdirectories. Measured, along with the fixed systemd-journal gid that makes
the group grant meaningful across the bind mount.
The option's description claimed declaring an entry from the service's
own module put 'the scraper and the target on the same host by
construction rather than by luck'. It does not. It constrains where the
target is; nothing in it places the collector, and the two enable flags
are co-located by a shared lib.mkDefault rather than by construction.
Split across hosts, a target is silently never scraped — the service's
host declares an entry no local collector reads, the collector's host
never enabled the service. No error surfaces, and no assertion can
catch it: separate hosts are separate evaluations with no shared
context, so the doc telling the truth is the only mechanism there is.
The same paragraph already warned co-location was not a guarantee, four
lines below the sentence claiming it was; a reader arriving for
permission stopped at the permission. This one did.
swarm-nats carries the concrete caveat for its own contribution.
The scrape config names a client_secret_file; this is what puts a file
there. A host oneshot copies authelia's minted secret between the two
container trees — a copy and not a bind mount, because the secret does
not exist until authelia's first boot and nixos-container refuses to
start on a missing bind source, which on a fresh swarm is a permanent
stall presenting as broken metrics.
The collector runs under DynamicUser and the prometheus receiver opens
client_secret_file itself, as that user, so there is no uid to hand the
file to. LoadCredential reads it as root before the sandbox exists and
re-exposes it under a path that does not depend on which uid the unit
got; the scrape config points there. Both spellings derive from one
binding, since a mismatch is a file the collector cannot open and
nothing but a runtime 401 would say so.
The published targets now render as prometheus scrape jobs, so the
option's name is true: the collector reaches them by name over https,
using prometheus-native oauth2 with the audience set to the target's own
url. Registered is not the same as requested — a client that does not
ask for an audience gets a token with an empty one however complete its
registration looks.
The url is split into scheme, target and metrics_path rather than asked
for three times: two spellings of one address is a mismatch waiting to
happen, and the failure it produces is a valid token refused at the
target. A malformed or non-https url is an assertion rather than a null
dereference from inside the renderer.
`collectorAudiences` was a list of URLs services contributed so the
collector's client could be registered for them. Slice C needs the same
URLs as scrape jobs, and a job needs a name the audience list has no
room for — so services would have contributed to two options that must
agree.
They now contribute `publishedScrapeTargets` once, as `<job> = "<url>"`,
and the client's audiences derive from it. The drift that would have
needed maintaining is gone, and its failure mode was the quiet one: a
target whose audience was forgotten authenticates against nothing and
reads as a broken scrape rather than a missing registration.
Adds an assertion for the one collision the module system cannot catch.
Two definitions of the same key within one option are already refused
(measured); across the two scrape options nothing arbitrates, and both
entries would render into a single scrape_configs list under one
job_name.
The forge's `/metrics` is published behind the gateway and denied to
everyone, waiting on a client to allow. This is that client.
An audience is a URL: authelia validates a bearer token against the
address being requested, and a client may only request an audience it is
registered for, so registration is the authorisation. The URLs are owned
by the services that publish them while audiences attach to one client,
so services contribute to a list and this module builds the single entry
— the `gateway.localNames` split, forced here by client definitions
concatenating rather than merging into a shared entry.
The access-control rule asks the client list whether the collector is
registered rather than re-deriving the conditions that register it. The
two drifting is not a build failure: authelia refuses a rule naming an
unknown client in its startup validator, so SSO fails to restart.
The collector names components `<kind>/<owner>` — a hive name for the
per-hive pipelines, the literal `swarm` for the swarm tier's own. Both
land in one attrset via `//`, so a hive named `swarm` replaced the swarm
tier's parts and lost its own: its receiver kept accepting pushes into a
pipeline that routed nowhere, and its samples lost the `hive` stamp that
makes attribution unforgeable. Zero failed assertions.
The reserved name is now bound once and interpolated at each swarm-tier
use, so the guard checks the same string the config emits rather than a
copy of it. A second swarm-tier pipeline joins the list and inherits the
check without touching the assertion.
Nothing in this deployment read a Prometheus endpoint, so enabling
/metrics on a managed service added an endpoint and no data. The swarm
collector receives OTLP pushes and does not scrape; VictoriaMetrics
stores what is pushed and has no scrape config. The pipeline was entirely
push-based and every such service is pull-based.
Adds a prometheus receiver, a resource/swarm processor and a
metrics/swarm pipeline alongside the per-hive ones.
The pipeline is separate because that is the ruling, not for tidiness:
every resource/<hive> processor UPSERTS a hive key, so a scraped swarm
sample routed through one would acquire the single label a swarm-level
service must not have. Keeping it out makes the absence structural rather
than something to remember to strip, the same way hive stays a property
of which authenticated receiver accepted a push.
Targets come from an option each service fills in from its own module,
under its own enable, rather than a list assembled here. That is what
puts the scraper and the target on the same host by construction: an
entry exists only where the service that named it runs. Co-location is
true of the all-local deployment and is not a guarantee, and that is
exactly the case where assuming it is invisible.
All three additions MERGE with the per-hive attrsets rather than
replacing them. A plain assignment would drop every hive's receiver,
processor and pipeline and still render a config the collector starts
cleanly on.
Empty target set emits no receiver, no processor and no pipeline — an
enabled scraper with nothing to scrape is the inert configuration this
issue is about, and the target set ships empty here because the targets
themselves are separate issues.
Formatting verified with nix fmt. The evaluation gate is not written yet;
nothing here has been evaluated against a fixture.
