The three values that differ per host (deploy.hive-controller.dashboardPort,
claudeCodePackage and otel.extraResourceAttributes) no longer go into the
meta flake. hive-priv writes them into the container's
/etc/hyperhive-hive-facts with the hive name and domain, and the agent
reads them at boot:
- HIVE_DASHBOARD_PORT reaches every unit through the hive-facts generator;
the web UI still defaults to 7000 without it.
- `claude` on PATH is a wrapper that runs HYPERHIVE_CLAUDE_CODE_PATH's
claude when it is there, and the container's own claude-code otherwise.
The host's serve.json keeps gc-rooting the pinned build.
- OTEL_RESOURCE_ATTRIBUTES moves out of the units' Environment= into a
boot-time generator, which appends HYPERHIVE_OTEL_EXTRA_RESOURCE_ATTRIBUTES.
The agent options services.hyperhive.agent.claudeCodePath (now a removed
option) and otel.extraResourceAttributes go; only the meta render set them.
render_flake takes no HiveEnv any more, so the memory-cap render test and
render_hive_flake_at, which only existed to vary HiveEnv into the render,
go with it.
Refs #4857
The external export is a setting of the swarm collector, so it moves to
services.hyperhive.deploy.swarm-otel.upstream. Renamed:
- otel.endpoint -> upstream.endpoint
- otel.protocol -> upstream.protocol
- otel.headersCredential -> upstream.headersCredential
- otel.collector.upstreamHeaderName -> upstream.headerName
The unreleased otel.upstreamSignals is declared as upstream.signals.
Hosts keep working through the rename warnings. Eval test covers the old
paths reaching the new options. Docs and stale comments point at the
new paths.
Adds services.hyperhive.otel.upstreamSignals, a list of "metrics" and/or
"logs" naming the signals the swarm collector sends to otel.endpoint. The
swarm's own stores always receive everything. The upstream exporter is
only defined when at least one signal uses it, since otelcol refuses an
exporter no pipeline references.
On deploy: nothing changes until the option is set; the default is
[ "metrics" "logs" ], today's behaviour. To stop the 404 flood from an
upstream that has no /v1/logs, set services.hyperhive.otel.upstreamSignals
= [ "metrics" ].
These three are per-host choices that hive-c0re reads and renders into
its agents, so they belong under the hive controller's deploy namespace
rather than swarm-wide:
- c0re.dashboardPort -> deploy.hive-controller.dashboardPort
- c0re.claudeCodePackage -> deploy.hive-controller.claudeCodePackage
(the claude-login and review-classifier containers on the same host
keep following it)
- otel.extraResourceAttributes -> deploy.hive-controller.otel.extraResourceAttributes
The old paths stay as mkRenamedOptionModule aliases, now in deploy.nix
with the namespace's other renames. github.enable and the collector port
stay swarm-wide.
Five host options whose values hive-c0re renders into every agent's
build move under services.hyperhive.swarm, so the swarm states each of
them once instead of every hive carrying its own copy:
c0re.dashboardPort -> swarm.dashboardPort
c0re.claudeCodePackage -> swarm.claudeCodePackage
github.enable -> swarm.github.enable
otel.collector.port -> swarm.hiveOtel.collectorPort
otel.extraResourceAttributes -> swarm.hiveOtel.extraResourceAttributes
Every old path is a mkRenamedOptionModule alias, so a host that sets one
keeps its value and gets an eval warning naming the new path. Types and
defaults are unchanged, so the rendered meta flake, serve.json and
hive-c0re's environment are the same as before on a host that sets
none of them.
The readers move with them: serve.json and its gc-root assertion, the
hive-c0re environment, the hive collector's listen address and bridge
port, the swarm collector's port-collision assertion, and the
claude-login and review-classifier containers' claude pin. The
swarm-services-switch fixture sets the swarm path and now also checks
that serve.json carries the pinned claude.
Part of #4857 (slice 2).
Both are ignored with a warning through the same shim, now
nix/host-modules/hive-options-removed.nix.
- otel.metricIntervalMs: claude in every agent and hive-c0re's
container-resource exporter run at the 60s default. The agent-level
services.hyperhive.agent.otel.metricIntervalMs, set only by the meta
render, goes too (warning shim in nix/agent-modules/otel-removed.nix).
- otel.debug: services.hyperhive.agent.otel.debug stays, documented and
set in an agent's own config; the meta render no longer emits it.
swarm-secret-publish and its path unit now run inside authelia's container,
beside the mint, and read each client secret at the path authelia writes it.
The host unit and host path unit, which read the secrets through the
container's root on the host, are gone, and so is
deploy.authelia.hostClientSecretDir, which only they read. The publisher keeps
its store identity, the secret-publisher leaf. The host hands the leaf and the
store's CA into the container as nspawn system credentials, and the unit loads
them; where this host mints the leaf, the container waits for swarm-bao-pki.
Authelia does not wait for the publisher, which retries every 30 seconds while
the store is down.
The container also runs the journal forwarder the other service containers
run, pushing to a receiver of its own on the swarm collector (serviceProducers
swarm-authelia) as a new machine OIDC client (swarm.authelia.otel.clientId)
that authelia registers for itself and the publisher carries to the store. The
fetch unit reads it back under the publisher's leaf, also handed to it as a
system credential, so the swarm-secret-publisher policy gains read on that one
leaf; its stanza keeps create and update, which the exact path would otherwise
take from the services/* prefix. Without the leaf the forwarder is off.
The container drops --link-journal=host, and the host collector stops reading
it. The services' "nothing at <path>" errors name the publisher in authelia's
container.
The gateway nginx serves stub_status on a loopback server of its own,
127.0.0.1:8085 (services.hyperhive.gateway.statusPort). The hive-tier
collector reads it with the contrib nginx receiver, which the pinned
otelcol-contrib 0.151.0 ships, so no exporter daemon is added. The
receiver dials http:// only (a unix: or http+unix: endpoint fails with
"unsupported protocol scheme"), so the listener is loopback TCP with
allow 127.0.0.1 / deny all rather than a unix socket.
The new gateway dashboard plots nginx.requests, nginx.connections_current
by state, and nginx.connections_accepted against handled, per hive.
Part of #5342.
The otel enable description and the exposeHostPorts example in network.md
still described agents dialing a bridge address. Rewrap the two long lines
the earlier edit left.
Agents are handed `http://hive-collector.<swarm domain>:<port>` as their
otel endpoint instead of `http://<bridge IP>:<port>`. Every hive running
its collector contributes the name to `gateway.localNames`, so its own
dnsmasq answers it with its bridge IP, the way the queue's name already
works. The rendered meta flake then carries no hive-specific address,
which a swarm-level render needs (#4857, operator ruling 100744: "b").
hive-c0re's own exporter keeps the bridge address: on the host the name
may resolve to loopback through `gateway.localHostsEntry`, where the
receiver does not listen.
The name sits under the swarm domain and starts with `hive`, a fragment
no hive name may contain, so no hive's default domain can take it.
The hive collector's `settings` was `{ extensions.file_storage = …; … }
// lib.optionalAttrs senderAuth { extensions.oauth2client/swarm = …; }`.
`//` is shallow, so whenever senderAuth held (every valid hive config)
the right-hand `extensions` replaced the left one and `file_storage`
vanished, while `service.extensions` still listed it. `otelcol validate`
then failed the config.yaml build:
Error: service::extensions: references extension "file_storage" which is not configured
Broken by 02831fb5e (PR #5178), which added `extensions.file_storage`
to the left-hand attrset; the `//` block for the auth extension predates
it (cb787997bd). Both extensions now live in one `extensions` attrset.
module-eval-hive-otel gains a case that every name in
`service.extensions` is defined under `extensions`; it evaluates false
on the parent commit and true here.
Fixes#5240
Each store reader (forge's OIDC fetch and token publisher, matrix's OIDC
fetch, the hive collector's fetch, Grafana's OIDC fetch and admin password,
the queue's callout-xkey unit, the CI runner's token fetch, the swarm
collector's OIDC fetch) used to read `deploy.bao.enable` in its own module
to add `after`/`wants` on `container@<store machine>` and
`after`/`requires` on `swarm-bao-pki.service`. Per the #3742 ruling, that
cross-service read moves into glue-bao-readers-store-order.nix, which sets
the same edges when the store and the reader are both on this host and
restates each reader's own gate so no stub unit is rendered.
Rendered `after`/`wants`/`requires` are identical as sets to main on 16
fixtures, store here and elsewhere. New module-eval cases in existing
fixtures pin both arms.
Part of #3742.
The hive collector's secret-wait unit read deploy.authelia.enable to
order itself after authelia's container. That ordering is now set by
glue-hive-otel-authelia.nix, which restates the unit's own gate (otel's
`waits`). The rendered unit is unchanged.
The hive-otel module-eval fixture `hiveOtel` gains authelia, and a case
pins the edge with authelia here and its absence without.
Part of #3742.
Seven host oneshots fetch a secret from the swarm secret store into a file
their consumer loads at start: swarm-bao-grafana-oidc, swarm-bao-otel-oidc,
swarm-bao-matrix-oidc, swarm-bao-ci-runner, swarm-bao-forge-oidc,
swarm-bao-hive-otel-oidc and swarm-bao-forwarder-oidc. Each was
RemainAfterExit with no timer, so it ran at boot or deploy and never again:
a secret re-minted in the store reached its consumer only through an
operator restarting the fetch.
lib/store-refresh-timer.nix gives each one a timer (OnUnitInactiveSec 10min,
RandomizedDelaySec 1min, the agent's own store re-read interval), and each
drops RemainAfterExit so a timer start is not a no-op. Each script already
writes the file only when secret_differs, and refresh_consumer/refresh_unit
restart the consumer only when the file is newer than its last start, so a
run reading an unchanged value restarts nothing.
Two docs said these units do not run on a timer; credentials.md's
ci-runner row said the runner token is not re-pulled.
Part of #5017 (slice T).
The forge's login source read its OIDC client secret from a file that
`hive-forge-oidc-secret` copied out of authelia's container tree. That
unit only existed where authelia ran on the forge's host (`ssoLocal`),
so a forge anywhere else needed the file placed by hand.
The publisher on authelia's host now carries the forge's client secret
to `secret/swarm/services/<forge client id>/oidc/client`, like every
other swarm service's. On the forge's host `swarm-bao-forge-oidc`, a copy
of `swarm-bao-matrix-oidc`, reads it under the forge publisher's leaf
(the forge host's one store identity, whose policy gains read on that
one path) and writes the file `forgejo-sso-source` reads, re-running the
login source when the value changes. It retries on the shared
store-retry shape while the store is down or the secret is not yet
published.
`hive-forge-oidc-secret` is removed, `clientSecretFile` defaults to the
in-container path wherever the forge runs, and the `ssoLocal` gate on
the login source's resolver ordering goes with it: the login source
fetches the issuer's discovery document on every forge host.
Refs #5015
The previous commit dropped `hive-otel-client-secret` and the collector's
`requires`/`after` on it for every hive. Only the store fetch restarts the
collector when its file lands; an operator-named `otel.clientSecretFile`
has nothing doing that, so a late file failed `LoadCredential` straight into
the collector's start limit with no way back.
The wait unit and the ordering are back for `senderAuth && !fetches`, and
absent only where `swarm-bao-hive-otel-oidc` delivers the file. The
module-eval case for a hand-named secret now also pins the wait unit and
the collector's `requires`/`after` on it, on both of its fixtures.
Refs #5015
The hive collector's `oauth2client` secret came from a `clientSecretFile`
defaulting to authelia's own minted file on the IdP's host, with a
`hive-otel-client-secret` oneshot that only waited up to 120s for that
file to exist; any other hive had to copy the file across by hand.
`swarm-bao-hive-otel-oidc` now logs in with the hive's leaf
(`deploy.bao.clientCertFile`), reads `swarm/hives/<hive>/oidc/client`
(written by the publisher, granted by the hive leaf's
`swarm/hives/<hive>/*` read), writes it root 0400 when it differs, and
restarts the collector when the file is newer than its last start.
Retries every 30s for 24h (`lib/store-retry.nix`), like the matrix and
forge fetches. `clientSecretFile` defaults to that file wherever the hive
leaf is set and stays as an override; an overridden file renders no fetch.
otelcol-contrib 0.151.0's oauth2client re-reads `client_secret_file` on
every token fetch, but the path is `LoadCredential`'s start-time copy, so a
changed secret still needs a restart. `lib/refresh-consumer.nix` gains
`refresh_unit` for a host unit; `refresh_consumer` now calls it with
`--machine`.
nats-authelia's "consumer of the host client-secret dir" case pointed at
the collector's old default; it now reads the publisher's path unit.
Refs #5015
The docs, option descriptions and comments that named the swarm root, the
swarm CA or the hive CA as an existing trust anchor now name the secret
store's pki root or the trust bundle that holds it:
- docs/swarm/README.md, secrets.md, bao.md
- deploy.bao.clientCaFile's and gateway.verifiedProxyTo's descriptions
- comments in hive-c0re/src/meta.rs, hive-agent/src/swarm_queue.rs,
swarm-logs/src/auth.rs, swarm-controller/src/vcs_metrics.rs,
hive-forge, swarm-authelia, swarm-ci-runner, glue-bao-tls, swarm-bao,
swarm-grafana, swarm-otel, otel, hive-c0re, lib/hive-ca-trust.nix,
hive-forge/Cargo.toml and the flake-update workflow
Nothing behaves differently: every edit is a nix, Rust or TOML comment, an
option description, or a test assertion message. The Rust edits change the
workspace source, which this PR already rebuilds for hivectl.
Refs #4588
Rewrite comments and option descriptions under nix/ that narrate how the
code got here ("used to", "new name", "sat at the TOP", "no longer",
renamed/moved narration, attributions) to say what the code does now and
why. Comments and description strings only; no logic changes.
Files in open PR #4970's diff are left alone. Refs #3902
Four comments pointed at ./swarm-otel.nix for something the split moved
to ./swarm-otel-service.nix: `domain` (otel.nix), `domainBase`
(swarm-ui.nix), the `clientId`/`audience` options
(glue-swarm-otel-oidc-client.nix), and `producerName` (swarm-otel.nix's
own "Read-only option below"). Every other pointer to ./swarm-otel.nix
names its `config` block, units, exporters, authenticators or
assertions, which stayed.
Refs #3742
Cut ~27 lines of blackout-measurement and cross-reference narrative
(already in the PR body / issue) down to the three things a reader
actually needs at this call site: the [Unit]-vs-[Service] trap, why
Restart is absent, and the window-vs-burst constraint.
Every deploy on a hive host, the replacement opentelemetry-collector
reaches bind() while the outgoing process still holds
127.0.0.1:8888 (its self-scrape endpoint). nixpkgs sets
Restart = "always" with no RestartSec, so the unit spends its five
default attempts in under two seconds, hits start-limit-hit and stops
retrying — ~27s of telemetry blackout per deploy.
RestartSec = 5 with a 12-attempt burst over a 120s window rides the
race out instead: the blackout ends within one interval of the port
coming free, and 55s of it being held is survivable where 2s was not.
StartLimitBurst/StartLimitIntervalSec go at the systemd.services attr
level, which NixOS renders into [Unit]; under serviceConfig systemd
ignores them silently. module-eval-hive-otel asserts the placement.
`services.hyperhive.gateway.enable`, `gateway.dns.enable` and
`network.enable` replace the `hyperhive.enable` gate on all three. Each
defaults to false; the modules that need one assert it with `mkDefault
true` from inside the guard their own deployment already carries, and
`swarm-required-services.nix` — the module that owns what the
swarm-services toggle implies — asserts all three explicitly.
hive-c0re asserts all three unconditionally, so an ordinary hive keeps
getting them with no opt-in: it is the host's only knowledge that agent
containers exist.
The resolver moves to its own `hive-gateway/dns.nix` so it can be gated
without reindenting the nginx half of the module.
Reinstates `network.enable`, dropping its `mkRemovedOptionModule` shim.
A config still carrying `network.enable = false` from before the removal
now switches the bridge off instead of failing eval.
Also deletes a duplicate `centralToggleOff` fixture in nix/module-eval.nix.
Two sibling slices added it independently (c5f60fd5, ce3b3d94); the merge
was textually clean and left `main` failing to evaluate at all, so this
file could not be gated without removing one.
A collector's `refused` / `failed` / queue-depth counters are the only
signal that says telemetry is being dropped, and nothing read them at
any tier — so a collector losing records looked exactly like a quiet
system.
The hive tier could not be scraped without first naming its port. 8888
is the collector's built-in default and appeared in no config, which is
also why nothing comparing configured ports could see it clash with a
co-located collector — swarm-otel.nix sidesteps 8888 by hand for that
reason, and says so. Declaring the port and binding it explicitly makes
the value comparable; wiring the scrape is then one entry per tier.
Extending the port-collision assertion to cover it is deliberately left
out: that belongs with the other port work, and coupling a collision
fix to a scraping fix makes both harder to review.
Gate: 101 module properties hold, was 95. The six cases pin the rendered
scrape job rather than the option; the metrics pipeline naming the
prometheus receiver, a path never emitted on any hive before this since
the hive tier's scrapeTargets was empty everywhere; the `readers`
spelling, with a control so a missing telemetry block cannot pass the
port check vacuously; the swarm tier's own entry; the two tiers not
claiming the same port; and the absence arm, a hive with no collector
declaring no target.
scrapeTargets' description said "Empty by default, and that is the
shipped case". This makes that false, so the paragraph moves with it.
An agent container writes a complete journal — 991 MB and nine days deep
on this hive — that nothing outside it can read: the host-side
per-container journal directory is an id-mapped bind mount, and journald
writes nothing into it. So the reader has to run inside the container,
and the path it would push to did not exist.
Three tiers, one vertical slice, because any two of them alone are
silent:
- the agent container gains an `opentelemetry-collector` with a
`journald` receiver aimed at its own journal and an exporter aimed at
the same base address every in-process producer already exports to.
- the hive collector gains `service.pipelines.logs`. Without it the
`otlp` receiver answers 404 on `/v1/logs` — measured, and
indistinguishable from a route that was never meant to exist.
- the swarm collector gains a per-hive `logs/<hive>` pipeline beside
`metrics/<hive>`. Without it the push is accepted, answered 200, and
routed nowhere.
The receiver's `directory` is stated rather than inherited, and that is
the load-bearing line: its default is the RUNTIME journal
(`/run/log/journal`), which in an agent container is empty. Left at the
default this whole path validates, starts, reports healthy and forwards
nothing. The assertion beside it covers the same silence from the other
end — a `volatile` or `none` journald storage empties the directory the
receiver reads.
Attribution follows the tier that can prove it. The forwarder stamps
`agent`, which no host-side reader could supply; `hive` is deliberately
left to the swarm tier, which upserts it from whichever receiver
accepted the sample, precisely so the label comes from something the
sender cannot write.
No `units` allowlist, unlike the swarm tier's journald receiver. That
one needs one because the host's journal also holds an operator's own
session; a container's journal is the harness and what the harness
spawns. Measured volume is 20827 entries / 6.3 MB per agent per day,
with nothing logging below `info` — so the receiver's `info` default
filters nothing and there is no bill to justify a knob.
Agent containers only, per the ruling on the issue: swarm services need
one forwarder per service container and get re-measured once this works.
Part of #3940.
`usersFile`, `hostClientSecretDir` and `hostUsersFile` are filesystem
paths that only exist on the machine running the `swarm-authelia`
container. A hive that does not run it has nothing at any of them, so
they fail the swarm-wide test the namespace's header states.
`usersFile` is a path *inside* the container and still belongs on the
deploy side: a path's scope is the scope of the filesystem it names, and
that filesystem is this host's container root.
The two `host*` options are `readOnly` and derived, so nothing can set
them through the rename shims — those serve readers. The shims are still
required: `mkRenamedOptionModule` is what keeps an out-of-tree module
reading the old path resolving at all.
Seven reads went through an alias rather than a full path
(`autheliaCfg.hostClientSecretDir` in five modules, `.hostUsersFile` in a
sixth, `swarmCfg.authelia.hostClientSecretDir` in a seventh). Every one
of those files already binds `deployCfg`, so the repoint needed no new
alias. Four more sites were prose, including one in `docs/`, which no
grep restricted to `nix/` would have found.
`services.hyperhive.tls.{stateDir,caValidityDays,leafValidityDays}` sat at
the top of `services.hyperhive`, which is meant to be everything about
hyperhive rather than the settings of one hive. Where the hive CA lives,
how long it lasts and how long the leaves it signs last are decisions of
the host holding the key — `deploy.*`, by the same rule as the switches
that moved before them.
`hive-controller` is hive-c0re's new name (mara on the issue), so the
knobs hang off the daemon that owns the CA rather than off a bare `tls`
at the root. mkRenamedOptionModule entries carry existing configs.
⚠️ Unlike the two switch renames, these names are NOT unique, so this was
swept by ALIAS BINDING rather than by identifier: hive-tls.nix alone holds
two options spelled `stateDir` — its own `cfg.stateDir` and the swarm CA's
`swarmCaCfg.stateDir`, four sites that must not move. Nine files bind an
alias to this config; the rename followed those bindings.
Two sites were invisible to the obvious check, and an unanchored sweep for
`hyperhive\.tls\b` is what found them: the option declaration (`= {` after
the path, so no trailing `.` or `;`) and the alias convention documented in
a comment in lib/hive-ca-trust.nix.
Also renamed the `<tls.stateDir>` shorthand in four docs and two Rust doc
comments, anchored on its delimiters — the new path contains the old one
as a substring, so an unanchored replace would have doubled the prefix.
hive-metric counters are delta by default; VictoriaMetrics has no
delta-to-cumulative processor of its own and official guidance is to
feed it cumulative. Converting at the hive tier, closest to the
origin, means every downstream consumer (swarm tier, VictoriaMetrics,
upstream) sees ordinary cumulative counters and existing PromQL idioms
(rate()/increase()) keep working unchanged. Default config is a no-op
passthrough for anything already cumulative (Claude Code's own export,
hostmetrics, Prometheus scrapes) -- verified against the actual
collector binary (otelcol-contrib validate), not assumed: an empty
config validates clean, an invalid key is rejected.
Per review: `deploy.otel` does not imply swarm level, and there is a
hive-tier collector too -- `services.hyperhive.otel.enable`, which every
hive runs. The tier is the entire distinction between the two, so the
name has to carry it, matching `deploy.swarm-controller` and
`deploy.swarm-ui`.
15 swarm-tier references renamed across 8 files. The hive-tier
collector's 16 references are deliberately untouched, verified as a
control on the same command.
Three spellings needed three different patterns, all inside this one
rename: the dotted path; the `mkRenamedOptionModule` target written as a
nix LIST (`[ ... "deploy" "otel" "enable" ]`), which no dotted grep can
match; and prose in docs/observability.md spelled `deploy.otel` with
neither `.enable` nor a leading dot. Unanchored `deploy\.otel\b` is the
only pattern that finds all three.
Two corrections from review, applied forward on this branch rather than
by rewriting it.
`deploy.<service>` was a bare bool, which makes
`deploy.forgejo = { enable; ci; }` unrepresentable -- the nested
CI-runner sub-option this namespace was designed around. Every entry is
now an attrset with an `enable`, so a second per-host deployment
decision becomes an ordinary addition rather than a migration.
`deploy.controller` is now `deploy.swarm-controller`, consistent with
`deploy.swarm-ui`, which was introduced in the same commit.
89 references rewritten across 24 files -- nix, Rust, docs, and the
repo's own CLAUDE.md.
The prefix-anchored sweep missed exactly one, and it was live code:
hive-tls.nix spells it `hyperhiveCfg.deploy.controller` -- the only
`hyperhiveCfg` prefix among 45 references. A suffix grep
(`\.deploy\.<name>`) finds it; a path-anchored one cannot, because the
head of a reference is whatever alias the reading file happens to bind.
The one move where the suffix grep is the wrong instrument. There are two
otel options a word apart — `swarm.otel.enable` (one per swarm) and
`otel.enable` (one per hive, every hive runs it) — so `\.otel\.enable`
matches twenty-five references of which most must not change. The module
already carries a comment warning about exactly this, on a line that
names `swarm.otel` in full rather than through a binding.
Triaged by hand and confirmed the other way round: after the move, a grep
for the per-hive option still finds it in the files that should keep it.
Also worth recording what the alias sweep cannot do. `swarm.nix` reads
this as `swarmCfg.otel.enable`, where `swarmCfg = cfg.swarm` and `cfg =
config.services.hyperhive` — an alias bound to an alias, two hops from
the option path. No syntactic sweep resolves that chain, which is why the
module system's own evaluation is the only complete check here and the
static sweeps are a way to narrow the work, not to finish it.
The largest of these moves: sixteen references spelled through `let`
aliases across eight modules, plus eight more spelled as a path, plus
five documentation pages.
authelia is also the clearest case for why the two namespaces exist.
`swarm.authelia.url` is needed by *every* hive in the swarm — it says
where to send a browser to authenticate — while running the container is
the business of exactly one host. The client half and the server half
were sharing a namespace whose whole contract is "identical everywhere",
and only one of them could honour it.
`swarm.authelia.oidc.clients` stays where it is for the same reason:
several modules register a client there, gated on authelia running here,
and the registry itself is what the service *is* rather than a decision
about this machine.
One sweep note worth recording: a grep for `swarm.authelia.enable` misses
`swarmCfg.authelia.enable`, because the prefix is whatever the reading
file bound. Grepping the suffix `.authelia.enable` finds both, and found
a reference in swarm.nix that the path-shaped pattern did not.
Same move as grafana, and the three belong together: they derive from one
switch and a store with no UI is as useless as a UI with no store.
`victorialogs` is the case that shows why the option-path sweep is not
enough on its own. It has **zero** references spelled
`swarm.victorialogs.enable` anywhere in the tree, and four spelled
through `let` aliases (`vlCfg.enable` in the collector, `cfg.enable` in
its own module). A sweep for the path would have reported nothing to do
and left every reader broken.
Prose moved with the code rather than being left behind: the comments in
swarm-required-services.nix that explained why the pair derives together
now sit above the assignments that do it, instead of above the gap where
they used to be.
Host metrics arrived carrying `hive` and no host attribute, so "which
host is out of memory" was answerable only as "which hive" -- true only
while a hive is one machine, which is the framing the swarm work exists
to end.
Adds the collector's own `resourcedetection` processor to the metrics
pipeline, emitting both `host.name` and `host.id`.
`resourcedetection` rather than a value picked in nix: this is the
canonical OTel mechanism for host identity, and choosing our own would
answer a question the tool already answers, differently from every other
deployment. A resource attribute is captured into the stored series, so
a private naming scheme is expensive in the way that lasts.
Both attributes, deliberately: `host.name` is readable and what a panel
groups by, but splits the series silently on a rename with nothing
linking old to new; `host.id` survives that and is unreadable alone.
Hosts get renamed and repurposed exactly when services move off one box.
The processor sits in the pipeline, so it applies to every receiver --
agent-pushed OTLP and scraped targets included, not just `hostmetrics`.
Keys verified against otelcol-contrib 0.151.0 with two deliberate
mutations rejected in the same run (a bogus resource attribute and a
bogus detector key), so "accepted" is distinguishable from a validator
that ignores what it does not recognise.
They differ by one letter and a grep for the shorter name matches both.
The summary is what ships; the per-process scraper is deliberately absent,
and a reader (human or grep) cannot tell from the diff alone.
Nothing measured the host itself. hive-c0re emits per-container metrics
and each service exports its own, so the store could answer 'is this
container using more memory than last week' and could not answer 'is the
machine under them out of memory' -- the question every one of those
readings is implicitly relative to.
Hive tier rather than swarm: a host is owned by the hive running on it,
and the swarm tier runs one collector for the whole swarm, so it would
have to reach other machines to see theirs.
Every scraper the pinned receiver offers except process, which adds a
series set per running process -- unbounded on a hive host, where the
nine shipped are a fixed handful. Coverage and cardinality are different
axes and only the second can hurt the store.
Refs #3649
`LoadCredential` naming a missing path is fatal at unit start, and this hive's
secret is minted by authelia's first-boot generator inside its own container —
nothing orders a host unit against that.
nixpkgs sets `Restart = "always"` on the collector with no `RestartSec`, so
that failure is instant: the unit burns systemd's 5-starts-in-10s allowance in
well under a second, lands in `start-limit-hit`, and stops retrying entirely.
`Restart = always` reads like it makes this self-healing and does the opposite
— a slow-failing unit retries until the secret appears, a fast-failing one
exhausts its limit before the thing it waits for can exist.
A oneshot converts the fast failure into a slow one, which is what that restart
policy is actually good at. Copied from `hive-forge-oidc-secret.service`, which
already solves this for the forge: bounded wait, then fail loudly naming the
file — never skip, because a skip yields a collector that starts and ships
nothing.
`TimeoutStartSec` exceeds the wait on purpose: `DefaultTimeoutStartSec` is 90s
and would kill the unit before it could emit that message.
The ordering against authelia's container is conditional — on a hive that does
not host the provider the secret is operator-provided, and naming a unit that
does not exist orders nothing, silently. The wait itself still applies there,
so a file that arrives late is tolerated rather than fatal.
mara, reviewing this PR: "hives always require an identity, swarm controller
and auth is not optional."
So `requireHiveIdentity` is gone rather than defaulted, and with it every
branch that had to describe an unauthenticated collector. The swarm tier now
serves per-hive receivers only, and `/` answers 404 because there is no
swarm-wide inbox to route to. A hive with no credential is a build error, not
a quieter mode.
`hivePortBase` goes too: with per-hive receivers unconditional, `port` IS the
base of the range. That keeps one documented knob instead of adding a second,
and its advice ("move it if something else claims that range") still holds.
Two assertions replace the toggle — an empty hive roster, and a null
`authelia.url`. The second matters because a guessed issuer URL evaluates
cleanly, deploys cleanly, and then refuses every hive at runtime.
⚠️ `cfg.port` is deliberately no longer compared against the derived range in
the collision assertion: it is now the range's first element, so listing it
would make that assertion fire on every config.
This also retires the asymmetry guard added earlier in review — the state it
protected against (auth off on one side, credential still set on the other)
is no longer representable.
Turning ingest auth off without clearing a hive's credential leaves that
hive's collector authenticating and addressing its own path, while an
unauthenticated swarm tier serves one catch-all and forwards the URI
unchanged. The receiver is asked for a path it does not serve, so telemetry
stops with 404s and retries — no 401, no assertion, nothing in any log
naming auth.
Only reachable by overriding one side without the other, since both defaults
derive from the same flag. That is what makes it worth a build error rather
than a caveat: an operator who flips the documented escape hatch has no
reason to suspect the sending half.
Found in review by argus.
The receiving half authenticates per hive, so this half has to prove which
hive it is. It mints a token against the swarm's authelia with this hive's
client and posts to that hive's path on the collector's gateway name.
Holding a credential is what decides whether this tier authenticates —
`clientSecretFile` non-null — rather than a second switch that could
disagree with it. The default is the secret this host's own authelia
minted, which is right exactly when the IdP runs here; a hive that is not
that host names wherever the file landed, the same manual-copy shape the
identities option already documents as unsolved.
Two things that a diff will not explain:
`endpoint_params.audience` is not redundant with the client's registered
audience. Registering only makes an audience permissible; a token minted
without asking for one carries `aud: []` and every receiver refuses it,
with a config that reads correctly at both ends.
`client_secret_file` keeps the secret out of nix altogether — the
collector opens the file itself. It is a real key of this extension,
checked against the shipped binary with a deliberate typo rejected in the
same run, so "accepted" is distinguishable from "ignores everything". The
path comes from systemd's `CREDENTIALS_DIRECTORY`, so nothing hardcodes a
`/run/credentials` layout.
An assertion covers the one deployment where this can go wrong silently:
a host running both tiers with ingest authenticated and no credential to
present would 401 against a collector on the same machine.
The hive tier has to *present* a credential to the swarm tier, and the
upstream collector build ships no auth extensions at all — `oauth2client`
exists only in contrib. So this is what makes authenticated egress
expressible, not a preference between two equivalent packages.
It also matches the swarm tier, which has been on contrib since it was
written. Two tiers of the same pipeline built from different component
sets is a difference nobody would predict from reading either module.
Not a build-farm cost: contrib is fetched, not compiled.
No behaviour change on its own — nothing names an extension yet. The
config that does lands separately, because that is the commit whose
failure mode needs a running collector to detect: `otelcol validate`
accepts a receiver naming an absent extension and the process then dies
at startup, so a green build proves nothing about it.
Drops swarm.otel.url (a loopback default an operator had to override on a
split host) in favor of swarm.otel.domain -- the same
gateway.localNames + nginx-vhost-through-the-gateway shape every other
swarm service (authelia, grafana, victoriametrics, ui) already uses. The
hive tier's exporter now reaches it as https://<domain> unconditionally,
resolved locally by dnsmasq on a co-located host and over the real network
otherwise, instead of a config knob nobody sets until they hit the silent
drop.
Costs CA trust on the hive tier: otel.nix wires
lib/hive-ca-trust.nix's trustBundle with hostUnit = true on the
opentelemetry-collector host unit, the same flag #3441/#3442 added for
swarm-controller and hive-c0re.
mara, #3125 comment 58363: "go c".
The pre-push lint refuses them, and rightly: a comment that names an
issue number ages into a pointer at a closed thread. The constraint each
one carried is stated directly instead.
The hive collector's only exporter becomes the swarm's collector, and
the upstream credential, the metrics-store exporter and the choice of
destination all move one tier up.
Its assertion goes with them: 'endpoint or a local store' was the right
rule while this tier picked the destination, and is the wrong one now.
A hive that runs no swarm services has neither, forwards to a swarm
collector elsewhere, and is correctly configured — the rule that
replaces it lives in swarm-otel.nix, where the destinations are.
The option descriptions here described a topology with one collector in
it: endpoint and protocol are not what agents are handed (they get the
derived first hop, see hive-c0re/environment.nix), and the credential is
not read by this tier.
Review catch: this PR relaxed the `otel.endpoint` assertion and staled the
canonical OTEL reference in the same stroke — `docs/observability.md` is
what CLAUDE.md points readers at for "what OTEL options are available",
and it still said required-full-stop while the new swarm/services.md
section said a local store satisfies it.
Also corrects the option's own description in otel.nix, which said the
same thing and renders into the generated options doc. Grepping the
reviewer's phrasing did not find that one; grepping the claim did.
Records the second destination where the "endpoint is where telemetry
ultimately goes" paragraph makes its claim, rather than only in the new
section a reader may not reach.
Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
