hive-c0re's container-resource exporter has POSTed to a 404 for as long as
it has existed, silently: it passed the collector's base address to
`with_endpoint`, which the SDK takes verbatim, so every export went to `/`
instead of `/v1/metrics`. Nothing reported it — OTLP export failures go to
an error handler no binary here installs — so the daemon logged "exporter
enabled" and delivered nothing. VictoriaMetrics has never held a sample
under `service.name=hyperhive-c0re`.
Fix the way the rest of the repo already resolves an endpoint: an endpoint
option names a BASE, and the layer that knows the signal appends to it.
`hive-metric` — same SDK, same collector — never calls `with_endpoint`, and
`docs/observability.md` documents the append as system behaviour; the one
place a full path is spelled out is the VictoriaMetrics exporter, because
its far end is not a standard OTLP path.
So drop the call. The builder is now byte-identical to hive-metric's, and
hive-c0re's unit carries the standard `OTEL_EXPORTER_OTLP_ENDPOINT` for the
SDK to read. The address is bound once in nix and consumed twice, so what a
hive hands its agents and what it exports to itself cannot drift.
The enable signal moves to that same standard variable: "configured" and
"where it actually goes" become one string rather than two that agree by
convention. `HYPERHIVE_OTEL_*` keeps its own job, the agent-config
transport meta.rs reads — a name the SDK has never known, which is the bug.
The test changes shape with the fix. The old one asserted a URL this module
built; the new one pins that the exporter is gated on the variable the SDK
itself reads, because the fix is now an absence and an absence is what a
later "the endpoint is right there, just pass it" edit puts back.
Refs #3402
hive-c0re's container-resource exporter already targets this hive's own
collector (environment.nix derives the bridge address), so the upstream
header it was loaded with has nowhere to be presented: that hop is
unauthenticated for every producer on the host, and the credential
belongs to the swarm tier, which is the one that leaves the swarm.
Drop the LoadCredential entry and the auth_headers() reader with it.
The option itself stays -- swarm-otel.nix is its real consumer, via
EnvironmentFile on the collector unit.
Also corrects three descriptions that this makes false, or that were
already false: the module doc claimed to reuse the config "Claude Code's
in-container SDK export uses", which stopped being true when agents
moved off that path; the nix comment claimed the secret is "the same one
the agent containers get, forwarded via nspawn --load-credential", which
lost its last producer earlier; and docs/observability.md described an
Authorization header on a hop that will no longer send one. The
headersCredential option's own docs already said it reaches "neither an
agent container nor a hive's own collector" -- this makes that true
rather than aspirational.
The endpoint, protocol, temporality preference and resource labels were
shipped only inside claude's managed settings json, so they applied to
claude's own process. hive-bash-daemon, hive-mcp-http, hive-matrix-daemon
and hive-forge-notify are systemd *siblings* of claude rather than its
children, so nothing shipped there could ever reach them: `hive-metric`
invoked from a tool call exited with "OTEL_EXPORTER_OTLP_ENDPOINT not
set", which is the honest failure of a value it structurally could not
see.
Declare those variables container-wide in a new agent module instead —
systemd.globalEnvironment for every unit PID 1 starts, environment.variables
for login shells. Both are needed and neither implies the other; NIX_REMOTE
is set both ways for the same reason.
Claude keeps only what is genuinely its own: the telemetry master flag,
the feedback-survey flag, the version label, and which signals it
exports. A different producer in the same container may legitimately
emit only metrics.
The hyperhive.otel.* options move across with them. They have more than
one consumer now, so their home is the OTEL module rather than the
claude one.
Setting declarativePlugins to any list -- the empty one included -- also
sets upstream's plugins.preinstall_disabled, which nixpkgs defaults to
declarativePlugins != null. Preinstall is what fetches the Drilldown apps
on first start, so claiming the plugin directory removed them without
touching anything a person had installed.
The comment above that line reasoned about the plugin path and the update
check and never about preinstall, and its closing claim -- nothing is
taken over on a hive with no plugins -- was true only of plugins a PERSON
installs. Upstream installs some itself, and those are the ones that went
missing.
Listing it as a package is better than the preinstall it replaces: store
pinned, no runtime fetch from grafana.com, survives a state reset. Its
Logs/Traces/Profiles siblings stay off -- they front Loki, Tempo and
Pyroscope, none of which this swarm runs.
environment.nix explains why HYPERHIVE_OTEL_HEADERS_CREDENTIAL is not
emitted by naming the machinery that used to consume it -- machinery this
PR deletes. Left as written it would describe a removed nspawn credential
and a removed oneshot as if both still ran, in the same paragraph that
justifies withholding the variable.
That is this PR's own defect inverted: it exists so an auditor asking
'can an agent obtain the OTEL token?' finds nothing misleading, and a
comment claiming live delivery machinery is exactly the kind of thing
that costs an auditor a reconstruction.
Keeps the rule rather than the history -- one holder, on the host -- since
that is the part still worth reading.
The host-side collector is the only path telemetry leaves a hive, so
HYPERHIVE_OTEL_HEADERS_CREDENTIAL is never emitted and everything
downstream of it is unreachable. What made it worth removing rather than
leaving inert is what it looked like to a reader: a complete,
well-commented mechanism for writing the hive's upstream credential into
a file the agent can read, described in the present tense. Anyone auditing
"can an agent obtain the OTEL token?" had to reconstruct the whole env-var
chain to find out the answer is no.
Gone: the per-agent `hyperhive.otel.headersCredential` option, the
`hive-otel-header` oneshot that merged OTEL_EXPORTER_OTLP_HEADERS into the
agent's own settings.json, and meta.rs's field, env read and render.
⚠️ Scoped by NAMESPACE, not by name. `hyperhive.otel.headersCredential`
(per-agent) and `services.hyperhive.otel.headersCredential` (host) are
different options sharing a leaf name — the host one is read by
`stats/otel_metrics.rs` for c0re's own container-resource exporter and
stays. Sweeping the string would have taken out working code.
The comment above `otelSettingsEnv` now states the property rather than
the absence: there is no auth header and no mechanism to add one, because
an agent exports to the hive's own collector and nothing an agent can read
is a secret to the swarm. The old behaviour is named in the past tense so
it reads as removed rather than overlooked.
meta.rs's assertions that pinned the injection are deleted rather than
adjusted; the surrounding test keeps covering extraResourceAttributes and
the endpoint/protocol injection, which are live.
hive-tls-ca re-signs at service activation -- the rebuild itself --
while hive-tls-resign only fires from a weekly timer. The previous commit
put the coverage check in the timer unit, so a corrected serviceDomains
would not have taken effect until up to a week after the deploy that
changed it. Same bug one level along: found a trigger, not the trigger.
Two guards now share one definition rather than each carrying their own,
because a rule enforced in one and not the other is worse than one
enforced in neither -- it looks fixed and only fires on whichever path
you did not take.
Also widens hive-tls-ca's condition to the services leaf. Both leaves are
signed inside that block but only the hive leaf gated it, so a fresh
gateway.pem suppressed the re-signing of a swarm-services.pem that was
missing or stale.
Expiry was the only re-sign trigger, so a leaf signed when the name set
was smaller stayed valid -- and wrong -- for its whole lifetime. Adding a
service to swarm.serviceDomains reissues the sub-CA (its own .names
reconciliation) but nothing regenerated the leaf nginx actually serves,
which left the previous commit's config change unable to fix anything on
a hive whose gateway leaf was not near expiry.
covers() reads the DNS names back out of the certificate rather than a
sidecar file: the pem is what nginx serves, and a bookkeeping file drifts
from it the moment a leaf is replaced by hand. Applied to both leaves --
the hive leaf has the same defect if the hive domain ever changes.
swarm.serviceDomains is what gateway.lib.tlsFor consults to pick the
services leaf over the hive leaf. grafana, victoriametrics and otel each
claim a gateway name under the swarm apex but were absent from that list,
so their vhosts were served the HIVE certificate -- which cannot cover a
name under a different apex.
Invisible until a machine client hit it: a name mismatch is a
click-through warning in a browser and an outright refusal in an OTLP
exporter. The metrics UI and store looked healthy while the collector
failed every POST and dropped the samples.
tls: failed to verify certificate: x509: certificate is valid for
probe.example, *.probe.example, not otel.swarm.example
Drops swarm.otel.url (a loopback default an operator had to override on a
split host) in favor of swarm.otel.domain -- the same
gateway.localNames + nginx-vhost-through-the-gateway shape every other
swarm service (authelia, grafana, victoriametrics, ui) already uses. The
hive tier's exporter now reaches it as https://<domain> unconditionally,
resolved locally by dnsmasq on a co-located host and over the real network
otherwise, instead of a config knob nobody sets until they hit the silent
drop.
Costs CA trust on the hive tier: otel.nix wires
lib/hive-ca-trust.nix's trustBundle with hostUnit = true on the
opentelemetry-collector host unit, the same flag #3441/#3442 added for
swarm-controller and hive-c0re.
mara, #3125 comment 58363: "go c".
All four sibling swarm containers import swarm-container-resolver.nix;
this one did not. It matters more here than most: otel.endpoint is an
operator-configured external hostname, and reaching it is the entire
reason this container holds a credential.
Also aligns two details with those siblings - the enable default is
asserted from swarm-required-services.nix with the metrics pair it
feeds, so that file remains the one place a service host is declared,
and machine is readOnly since its description already calls it a fact
rather than a knob.
The pre-push lint refuses them, and rightly: a comment that names an
issue number ages into a pointer at a closed thread. The constraint each
one carried is stated directly instead.
A collector serves its own metrics on localhost:8888 unless told
otherwise, and co-located tiers share a network namespace, so the second
one to start dies with 'bind: address already in use'.
The port appears in neither config - it is a default inside the binary -
so comparing the ports the configs name reports them distinct. A
behavioural probe found it by being unable to start the chain.
metrics.address is the spelling that looks right and is rejected by this
version ('migration.MetricsConfigV030' has invalid keys: address);
readers is the schema it accepts.
The hive collector's only exporter becomes the swarm's collector, and
the upstream credential, the metrics-store exporter and the choice of
destination all move one tier up.
Its assertion goes with them: 'endpoint or a local store' was the right
rule while this tier picked the destination, and is the wrong one now.
A hive that runs no swarm services has neither, forwards to a swarm
collector elsewhere, and is correctly configured — the rule that
replaces it lives in swarm-otel.nix, where the destinations are.
The option descriptions here described a topology with one collector in
it: endpoint and protocol are not what agents are handed (they get the
derived first hop, see hive-c0re/environment.nix), and the credential is
not read by this tier.
The swarm tier is the only holder of the upstream credential, the only
writer to the swarm's metrics store, and (once #3283 lands) the place
that stamps hive= from the authenticated connection rather than from
anything a sender can choose. Today one collector does both tiers' jobs,
which works only because they land on one box.
A container rather than a second host unit, for two reasons that agree:
every sibling swarm service is one, and `services.opentelemetry-collector`
is a singleton NixOS option already spoken for on the host by the hive
tier. A container gets its own evaluation and therefore its own
collector.
Port defaults to 4319, deliberately not the OTLP default 4318 the hive
tier uses: swarm containers share the host netns, and two listeners
claiming one port is not a build failure but a runtime coin toss with
nothing in any log saying so -- the same collision grafana and the forge
hit on 3000.
`url` is an option with a co-located default rather than a loopback
literal in the exporter, so a split-host deployment is a config change
instead of a code change.
Wires nothing yet: the hive tier still exports directly, and switching it
over is the next commit.
hive-c0re is a host unit that wired no CA trust at all. Harmless today
-- every endpoint it is handed defaults to plain http or loopback, so
nothing verifies a certificate -- and latent for the split-host shape the
options invite: `swarm.matrix.apiUrl`'s own example is an https URL, and
pointing it or `statusPublish.natsUrl` at another hive's gateway means
verifying a leaf signed by a CA generated at runtime.
Unlike the other consumers of this helper, hive-c0re already makes a
public-TLS call -- the OTEL exporter reaches whatever `otel.endpoint`
names. `SSL_CERT_FILE` replaces the trust store rather than adding to it,
so the bundle being system CAs + hive CA is what keeps that path working;
narrowing it to the hive CA alone would fix a case nobody hits yet and
break one that runs today.
New GET /api/swarm on swarm-controller, backed by
services.hyperhive.swarm.name (SWARM_CONTROLLER_NAME env var, same
optionalAttrs-gated-on-option-resolving shape queueEnv/forgeEnv/etc.
already use). swarm-ui's <Shell> fetches it once and sets both
document.title and the header's brand text; falls back to the
existing static "hyperhive swarm" label when the operator never set
a name or the fetch fails.
Extracted the swarm-queue connect block out of main() into its own
connect_status_reader() fn to keep main() under clippy's line-count
lint after adding the new field wiring — no behavior change, same
comments moved as-is.
The controller's forge client speaks TLS to `https://<forge domain>`,
which the gateway serves with a leaf signed by the hive CA. That CA is
generated at runtime, so nothing build-time can name it and it is not in
the system store -- and `reqwest`/`rustls` resolves roots through
`rustls-native-certs`, whose `SSL_CERT_FILE` *replaces* the store rather
than adding to it. With no bundle wired, every forge call failed
`invalid peer certificate: UnknownIssuer` and agent creation died at its
first step.
`lib/hive-ca-trust.nix` already solved this, but only for containers: it
sources the CA through `/run/hive-ca/trust-bundle.pem`, a bind mount that
does not exist on the host. `hostUnit` makes it read the host copy and
wait on `hive-tls-ca.service` itself -- one flag driving both, because a
host source without that ordering is a race.
`enable` is the other half, and it is the sharp edge: a container caller
imports this into the container's module set, so it disappears with the
container. A host caller imports it at the host's top level, where
`imports` is unconditional -- without the flag, a hive with the
controller turned off would get a bundle oneshot and a `swarm-controller`
service conjured by `genAttrs`, holding an `SSL_CERT_FILE` and no
`ExecStart`.
`serverName` is baked irrevocably into every user and room id, so a hive
that rebuilds onto a new default is a *different homeserver*, not a
renamed one: existing accounts and rooms are stranded, and reverting the
config does not undo it. Its neighbours (`gatewayHost`, the forge domain)
are routing, rediscovered through `.well-known` and fixed by editing them
back. Same diff shape, three orders of magnitude apart in blast radius --
which is an asymmetry a module should carry rather than an operator.
An activation script and not `warnings`, which is where this obviously
belongs and does not work: the condition needs the host filesystem, and
`nixos-rebuild switch --flake` evaluates purely, where
`builtins.pathExists "/var/lib/..."` answers false rather than throwing.
A `warnings` entry gated on it would evaluate, deploy, and print nothing
on every real deployment.
Rendered only when `serverName` is null, so a pinned hive has no script
rather than a script that stays quiet -- a guard that cries wolf at a
correctly-configured deployment makes the next real one read as noise.
Never fails the activation: it warns about a choice that cannot be
undone, and refusing the rebuild of a hive that already chose
deliberately is the opposite of helping.
The probed path is read out of the container's own evaluated config
rather than hardcoded. A guessed path resolves cleanly and silently never
matches, which is the same failure this guard exists to catch one level
up.
mara asked whether SWARM_CONTROLLER_PUBLIC_URL should also be set when the
controller does not run on the same host, and then said the two read as one
service to her. Both are worth answering in the file rather than only in the
thread: that split is not representable today, and the reason is two hops
away from this line.
The UI's /api/ location proxies http://unix:<socketPath> -- a path that
resolves nowhere else -- and this daemon binds no TCP address at all. So the
vhost and the daemon are co-located by construction, and ui.enable is the
flag that declares the vhost rather than a guess about some host serving it.
Also names the tripwire: the day the daemon grows a TCP listener, this line
silently stops being right (env unset, registration quietly skipped, no
error). That is when an explicit publicUrl option becomes correct -- not
before, while there is exactly one derivable answer.
The endpoint landed inert: nothing pointed at it, so the only way to see
it work was to mint an HMAC by hand. Register the two swarm-wide hooks
at startup so a real forge event produces a journal line.
Registered ALONGSIDE the per-hive hooks, not instead of them. Every hive
keeps receiving and acting on its own deliveries; the controller gets a
copy and logs it. Moving the registration is a later step and has to be:
fan-out swarm->hive does not exist yet, so a hook moved now would point
at a receiver that forwards nowhere, silently on both sides.
Deliberately no stale-hook deletion arm, unlike the two per-hive
registrars this otherwise mirrors: theirs delete hooks matching their own
path with a foreign base, and the hives' hooks are not stale.
The route prefix is what keeps this safe. Both hive-side registrars
delete any hook ending in /webhook/knowledge or /webhook/config-pr with a
different base, so a swarm hook under those paths would be deleted by
every hive on every boot. Serving them under /webhook/forge/ avoids it,
and a test pins it -- there is nothing else that can.
SWARM_CONTROLLER_PUBLIC_URL is set only where the swarm vhost is served,
because a hook whose target_url nothing answers is worse than no hook.
A forge webhook is a machine POST carrying an HMAC and no session cookie,
so it cannot pass the authelia auth-request subrequest every other location
on this vhost uses. This location deliberately omits it; the HMAC check in
the controller is what guards the path.
Scoped to /webhook/forge/ rather than /webhook/ so a future endpoint under
the same prefix does not inherit the bypass.
The bridge no longer has a private canonical store, so the env var
naming one is gone rather than repointed. What is left is the single
users file it now reads and writes directly.
That line is what made `swarm agent create` fail on this hive: the
bridge found no store at its own JSON path, saw users already in
`users.yml`, and refused to overwrite a file it had not written --
correctly, given two things claimed to be canonical for one file.
Plugin management is server-admin scoped, and on an SSO hive nobody holds that
role: auto_assign_org_role grants an org role, and the built-in local admin
that does hold server admin cannot log in because the login form is disabled
whenever SSO is configured. Two individually-correct decisions leaving no path
to the plugin UI at all.
Declarative is the way through rather than a workaround for it -- plugins land
in the store and in git, survive a rebuild and a state reset, and the container
needs no runtime egress to grafana.com.
mkIf rather than passing the list through: upstream's default is null, while an
empty list is a real value pointing the plugin path at an empty store dir, so a
hive that sets nothing must keep seeing null.
Last of the three modules that hand-rolled the same concat with wantedBy +
before and no requires, so a failed assembly left the consumer running against
a missing file and trusting nothing -- every outbound TLS call fails while the
unit looks healthy.
The forge is the one with two consumers: forgejo-sso-source fetches the
issuer's discovery document over the swarm CA and once shipped without the
trust its sibling had. It only exists when SSO is on, so the consumer list is
conditional -- naming an absent unit would define a serviceless one and order
nothing.
Removes the now-dead useSelfSigned and caContainerPath bindings (nix does not
warn) and retargets three comments the deletion orphaned, including the helper
header that still named this module as the per-call-site concat.
Both hand-rolled the same concat with wantedBy + before and no requires,
so a failed assembly let the consumer start against a missing file and
trust nothing at all -- which fails every outbound TLS call while the unit
looks healthy. The helper puts requires on the consumer and verifies the
assembled bundle before moving it into place.
Removes each module's now-dead useSelfSigned binding: its only use was the
mkIf on the deleted block, and nix does not warn about an unused let
binding. Also corrects two comments the deletion orphaned -- one described
the removed path binding, the other pointed at a bundle service that no
longer exists.
A loopback literal encodes 'authelia is in my netns' at the call site,
and authelia's OIDC endpoints are https-only in effect: reached directly
they answer 400, because the forwarded headers nginx injects for every
other consumer are what let it determine its own issuer. Going by name
deletes the need for those headers rather than reproducing them, and
converges on the URL swarm-nats-auth already uses.
Depends on the CA trust added for the same container in #3407 -- without
it this swaps a 400 for an UnknownIssuer.
The null-url assertion joins the module's existing list: interpolating a
null would surface as a nix coercion error several files from its cause.
Same class-B defect as the queue responder: swarm-authelia-bridge
introspects authelia by name over https and the container trusted no swarm
CA, so the call could only ever fail UnknownIssuer.
This is also the prerequisite for the by-domain introspection change on
#3391 -- flipping that URL without the trust half would swap one failure
for another.
The responder introspects authelia over https by name. It had no CA trust
at all, so the handshake failed UnknownIssuer, introspection failed, and it
denied every client -- surfacing at the controller as a 60s
authorization-violation loop, two layers from the cause.
Adds a shared trustBundle helper to lib/hive-ca-trust.nix rather than a
fifth hand-rolled concat. Four containers were each assembling this
themselves, which is how they came to share one defect: wantedBy + before
express ordering but not success, so a failed assembly let the consumer
start against a missing file and trust nothing at all.
The helper fixes both halves of that. requires goes on the consumer, so a
failed bundle stops it and the dependency is visible in systemctl status
where someone debugging a TLS failure looks. And the script assembles to a
temp path, checks the result actually contains a certificate, and only then
moves it into place -- cat of an empty bind exits 0, so set -e does not
catch it and a partial bundle must never appear under the final name.
Returns a module rather than bare services: a caller that already writes
systemd.services.<consumer> cannot also write systemd.services in the same
attrset.
Keeps why the setting exists (the bridge writes this file and cannot
restart authelia) and drops the incident narrative and the answered
objections -- that history belongs in the PR discussion, not in three
comment blocks a future reader has to scroll past.
authelia now watches the users file, so the restart is redundant -- and it
was the wrong shape twice over. It could fail: a login was refused for a
user whose record was already correct on disk, with nothing in either log
implicating the reload. And it only ever worked for this writer --
swarm-authelia-bridge writes the same file and cannot restart anything,
since running unprivileged inside the container is the whole reason it may
write it at all. A reload that depends on which process did the writing is
not a reload.
--machine/--unit and their two env vars existed solely to name a
systemctl -M target, so they go with it. That drops two required settings
from the operator surface.
The three objections previously recorded against watch are all answered
now, and are kept next to the decision rather than deleted: the key is
verified against the pinned build (validate-config accepts it and rejects
a misspelling), the watch is on the directory so a rename is observed, and
partial reads are structurally impossible because every writer of this
file goes through write_atomic.
Authelia reads authentication_backend.file.path once at startup. Without
watch, an identity the bridge creates is real on disk and invisible to the
running authelia until something unrelated bounces the unit.
swarm-authelia-bridge cannot restart authelia -- running unprivileged
inside the container is the whole reason it may write that file, so
restarting the unit is exactly the privilege it was designed not to hold.
swarmctl does restart it, but that is a systemctl -M shellout that can
fail, and when it did the symptom was a login refused for a user whose
record was already correct on disk. This makes correctness stop depending
on that restart succeeding.
The auth-callout responder decides what an admitted client may publish
from two strings: the prefix marking a hive client, and the client id
allowed to read every hive's key. Both were literals in three places --
swarm-authelia.nix mints "hive-${name}", swarm-controller.nix defines
"swarm-controller", and the responder carried its own copies as clap
defaults because swarm-nats.nix passed neither.
Each producer now publishes its value as a readOnly option and the
responder's ExecStart reads them, so the agreement is one evaluation
rather than three strings that happen to be equal. Same pattern the
module already uses for `--account`, and the same argument
swarm-authelia.nix gives for publishing `machine` and `unit`.
Worth the change because the failure is silent and misattributed:
rename either principal and the responder starts denying the one that
stopped matching, a denial reaches a NATS client as a timeout rather
than an error, and a hive that is refused looks exactly like a hive
that has not reported yet.
A swarm service container shares the host netns and force-disables
resolvconf, so it inherits the /etc/resolv.conf nixos-containers copies in
at start (cp --remove-destination, host-side preStart, once per start) and
nothing ever refreshes it. That makes the container's resolver a snapshot of
the host's file at its boot instant.
When that snapshot is wrong the container can never recover, and the symptom
appears arbitrarily far from the cause: swarm-nats-auth cannot resolve
authelia's name, so it denies every auth-callout request and the queue
refuses every client with an authorization violation.
Give each of the four swarm containers a oneshot that writes the resolver
file itself, from the bridge IP, ordered before that container's first DNS
consumer. The shape is the one every agent container already uses.
networking.nameservers cannot do this: resolvconf is its only consumer and
these containers disable it, so setting it renders no file while still
evaluating cleanly. A static environment.etc entry cannot either -- it would
have to survive etc activation landing on the regular file the host already
copied there, which no eval can show.
The unit restricted RestrictAddressFamilies to AF_UNIX, which was correct
while the daemon only served its unix socket. It has since grown three
outbound clients -- authelia token minting and forge calls over HTTPS, and
the queue over NATS -- and every socket(AF_INET, ...) was refused by seccomp.
systemd surfaces that refusal as EAFNOSUPPORT, "Address family not supported
by protocol", so the failure names the protocol and never the sandbox. The
visible symptom was swarm agent creation failing while minting a bearer
token, with a connect error that reads like a network fault.
Permit AF_INET/AF_INET6 for those clients and AF_NETLINK, which glibc's
getaddrinfo needs to enumerate local addresses before returning one. The
rest of the unit's hardening is unchanged.
Per the operator's call on #3372: rather than moving Grafana off the
forge's 3000 to another number, take it off TCP entirely.
The collision was possible because every swarm service container shares
the host's network namespace, which makes a port a swarm-wide resource
two modules can each claim believing it free — Grafana took upstream's
3000, so does the forge, and `grafana.<swarm-domain>` served the forge
with no bind error and nothing in any log. A socket has a path, and a
path collision is a build-time conflict rather than a runtime coin toss.
Three parts, none of which works alone:
- `protocol = "socket"` with `socket_gid` = nginx's static gid 60.
- Grafana joins that gid inside the container. A non-root process may
only chgrp to a group it belongs to, and a container has its own user
database — without the membership Grafana starts, the chown fails, and
the socket is simply unreachable.
- The socket dir is created host-side by tmpfiles and bind-mounted in. A
container's /run is an nspawn tmpfs, so it is not visible from the host
at /var/lib/nixos-containers/<name>/run; the same shape the per-agent
web.sock already uses.
The `port` option is gone rather than deprecated — nothing can set it to
a colliding value if it does not exist.
`swarm-controller.service` carried `SetCredential=queue-client.secret:` — an
empty value, which systemd's parser refuses:
/etc/systemd/system/swarm-controller.service:38:
Invalid syntax, ignoring: queue-client.secret:
So the line was dropped on every daemon-reload, and `LoadCredential=` was
fatal again — precisely the failure that default was added to prevent. On a
hive where authelia has not yet minted the secret, the controller refuses to
start rather than coming up with the queue unconfigured.
It looked correct for days because the credential file happened to exist, so
the fail-soft was never exercised.
Measured with `systemd-analyze verify`: an empty value is rejected, any
non-empty one is accepted. The placeholder is a real word rather than filler —
it reaches the token request as the client secret, so authelia refuses it and
the journal names something an operator can act on.
Gate: state/eval-setcredential.sh, with the parent commit as its mutation.
nixpkgs dropped the default for `services.grafana.settings.security.secret_key`
and asserts on null, so the module I merged an hour ago fails the build on any
host that enables it. That is a broken deploy, not a warning.
Generated in-container like authelia's own keys and for the same reason:
nothing outside the container ever reads it. Generated ONCE and kept — the
key signs Grafana's stored datasource secrets, and rotating it does not
re-encrypt what it already encrypted, so a fresh key per boot would leave
Grafana unable to read its own database. Delivered as `$__file{}`, so it
never enters the store.
The gate missed this because a container is a separate NixOS evaluation with
its own assertions: forcing the host's `config.assertions` never touched the
container's. `state/eval-3265.sh` now forces
`containers.swarm-grafana.config.assertions` and reports any that fail.
Review catch: this PR relaxed the `otel.endpoint` assertion and staled the
canonical OTEL reference in the same stroke — `docs/observability.md` is
what CLAUDE.md points readers at for "what OTEL options are available",
and it still said required-full-stop while the new swarm/services.md
section said a local store satisfies it.
Also corrects the option's own description in otel.nix, which said the
same thing and renders into the generated options doc. Grepping the
reviewer's phrasing did not find that one; grepping the claim did.
Records the second destination where the "endpoint is where telemetry
ultimately goes" paragraph makes its claim, rather than only in the new
section a reader may not reach.
Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
A local time-series database rather than only an external sink, so the
swarm dashboard stays readable when the outside world is not: a view of
the system must not depend on the system it views being healthy.
listenAddress is pinned to loopback. Upstream defaults it to every
interface, and the OTLP ingest path this exists to receive on is
unauthenticated — the gateway is the only intended client and it is on
this host, so a wider bind would publish a write endpoint to whatever
the host is reachable on.
retentionPeriod defaults high rather than being required, because the
two failure directions are not symmetric: too long fills a disk, which
is visible and recoverable by lowering it, while too short destroys
history silently and permanently. The operator lowers it once they have
measured how fast this swarm accumulates.
OTLP needs no flag. Measured against the pinned 1.146.0 rather than
inferred from the module's option list, which has no OTLP switch and so
reads as though the feature were missing: the running server answers
POST /opentelemetry/api/v1/push with 200, where a nonexistent path
answers 400.
Three host units poll up to 120s for a secret authelia mints on its first
boot, and all three are `Type=oneshot` with no `TimeoutStartSec`. systemd's
`DefaultTimeoutStartSec` is 90s, so it kills them at 90 — before the script
reaches its own `exit 1` and names the file that never appeared.
The wait itself is fine; what's lost is the diagnosis. On a fresh hive the
operator gets a bare start-timeout instead of "authelia has not minted
<path>", several layers from the container that was actually slow.
Found while writing the same unit for Grafana, where the timeout is set —
so this is the existing three catching up with it, not a new pattern.
Defining an environment key on a unit that does not exist creates a
fragment for it: inert, never activated, but present on every
non-controller hive that has a CA. hive-c0re runs everywhere so its
line needs no guard; the controller runs on one host in a swarm.
Caught in review. It evaluates and builds clean either way, which is
why it took a reviewer rather than a check.
