argus, reviewing, ran the guard against a source path that does not
exist rather than one that is empty. grep writes nothing to stdout in
that case, so `|| true` left the count variable empty and the -eq test
died with "integer expected" instead of reporting.
The unit still failed — cat hits the same missing file and set -e stops
it — but with a generic "no such file" rather than the message naming
which half is absent, which is the only thing this guard is for.
`|| echo 0` on all three counts. The gate gained the arm that was
missing: an absent source, asserted to fail THROUGH the guard rather
than merely to fail.
The guard inspected the assembled file for any certificate. The system
store always holds certificates, so it passed unconditionally — including
in the one case it was written to catch, where the hive CA half
contributed nothing.
That half is the only one that matters here: every name these consumers
verify is issued by our own CA, so a bundle of nothing but public CAs is,
for this purpose, an empty bundle that measures as full. The failure is
silent and total — the unit reports success and every egress TLS call to
a swarm service then fails.
Counts the source on its own before assembling, and checks the result
carries what both halves brought, so a source truncated between the count
and the copy is caught too.
Scope is stated at the guard: it proves the anchor was contributed, not
that it is usable. A consumer reading only the first certificate ignores
it regardless, which is what took the swarm collector down, and no check
on this file can see that. Only a handshake can.
The swarm collector could never verify its OIDC issuer, so it exited at
startup on every boot and the hive tier dropped every metric.
`issuer_ca_path` loads only the FIRST certificate in the file it names.
The bundle assembled for this container is `system CAs ++ hive anchors`,
so the swarm CA sits ~123rd and was never in the pool: the extension got
whichever public CA sorts first, could verify nothing of ours, and failed
`x509: certificate signed by unknown authority` — with 125 valid
certificates in the file.
Leaving the option unset makes the extension use the process trust store,
which `trustBundle` already populates via `SSL_CERT_FILE`, and that
consumer reads every certificate regardless of order. One file, two
consumers, opposite parsing; the fix is to stop naming it twice rather
than to reorder the bundle.
Measured with the deployed binary against the deployed config, varying
only the CA source: root-only OK, root-first OK, root-last FAILS,
root-second FAILS, and unset-with-SSL_CERT_FILE OK against a control that
correctly fails when the anchor is absent.
mara on the tracking issue: "we need the swarm label for upstream otel at
least (the out of swarm one)."
Stamped in the per-hive `resource` processor rather than on a separate
upstream-only pipeline, which would double the pipeline count to withhold one
constant label from the local store. It is redundant there — one metrics store
per swarm, so every series in it already belongs to this swarm — but a constant
label multiplies no series, and it means what leaves and what stays have the
same shape.
Upstream is where it stops being redundant: that is the one hop where several
swarms can land in one store, and samples that cannot name their swarm collide
there exactly as hives collided here before per-hive receivers existed.
`unknown` when unnamed rather than an absent label, copying the agent path so
a query never has to handle both "the label is missing" and "the label says
unknown".
`LoadCredential` naming a missing path is fatal at unit start, and this hive's
secret is minted by authelia's first-boot generator inside its own container —
nothing orders a host unit against that.
nixpkgs sets `Restart = "always"` on the collector with no `RestartSec`, so
that failure is instant: the unit burns systemd's 5-starts-in-10s allowance in
well under a second, lands in `start-limit-hit`, and stops retrying entirely.
`Restart = always` reads like it makes this self-healing and does the opposite
— a slow-failing unit retries until the secret appears, a fast-failing one
exhausts its limit before the thing it waits for can exist.
A oneshot converts the fast failure into a slow one, which is what that restart
policy is actually good at. Copied from `hive-forge-oidc-secret.service`, which
already solves this for the forge: bounded wait, then fail loudly naming the
file — never skip, because a skip yields a collector that starts and ships
nothing.
`TimeoutStartSec` exceeds the wait on purpose: `DefaultTimeoutStartSec` is 90s
and would kill the unit before it could emit that message.
The ordering against authelia's container is conditional — on a hive that does
not host the provider the secret is operator-provided, and naming a unit that
does not exist orders nothing, silently. The wait itself still applies there,
so a file that arrives late is tolerated rather than fatal.
mara, reviewing this PR: "hives always require an identity, swarm controller
and auth is not optional."
So `requireHiveIdentity` is gone rather than defaulted, and with it every
branch that had to describe an unauthenticated collector. The swarm tier now
serves per-hive receivers only, and `/` answers 404 because there is no
swarm-wide inbox to route to. A hive with no credential is a build error, not
a quieter mode.
`hivePortBase` goes too: with per-hive receivers unconditional, `port` IS the
base of the range. That keeps one documented knob instead of adding a second,
and its advice ("move it if something else claims that range") still holds.
Two assertions replace the toggle — an empty hive roster, and a null
`authelia.url`. The second matters because a guessed issuer URL evaluates
cleanly, deploys cleanly, and then refuses every hive at runtime.
⚠️ `cfg.port` is deliberately no longer compared against the derived range in
the collision assertion: it is now the range's first element, so listing it
would make that assertion fire on every config.
This also retires the asymmetry guard added earlier in review — the state it
protected against (auth off on one side, credential still set on the other)
is no longer representable.
Turning ingest auth off without clearing a hive's credential leaves that
hive's collector authenticating and addressing its own path, while an
unauthenticated swarm tier serves one catch-all and forwards the URI
unchanged. The receiver is asked for a path it does not serve, so telemetry
stops with 404s and retries — no 401, no assertion, nothing in any log
naming auth.
Only reachable by overriding one side without the other, since both defaults
derive from the same flag. That is what makes it worth a build error rather
than a caveat: an operator who flips the documented escape hatch has no
reason to suspect the sending half.
Found in review by argus.
The receiving half authenticates per hive, so this half has to prove which
hive it is. It mints a token against the swarm's authelia with this hive's
client and posts to that hive's path on the collector's gateway name.
Holding a credential is what decides whether this tier authenticates —
`clientSecretFile` non-null — rather than a second switch that could
disagree with it. The default is the secret this host's own authelia
minted, which is right exactly when the IdP runs here; a hive that is not
that host names wherever the file landed, the same manual-copy shape the
identities option already documents as unsolved.
Two things that a diff will not explain:
`endpoint_params.audience` is not redundant with the client's registered
audience. Registering only makes an audience permissible; a token minted
without asking for one carries `aud: []` and every receiver refuses it,
with a config that reads correctly at both ends.
`client_secret_file` keeps the secret out of nix altogether — the
collector opens the file itself. It is a real key of this extension,
checked against the shipped binary with a deliberate typo rejected in the
same run, so "accepted" is distinguishable from "ignores everything". The
path comes from systemd's `CREDENTIALS_DIRECTORY`, so nothing hardcodes a
`/run/credentials` layout.
An assertion covers the one deployment where this can go wrong silently:
a host running both tiers with ingest authenticated and no credential to
present would 401 against a collector on the same machine.
The swarm collector accepted OTLP from anyone who could reach it, and took
the `hive` resource attribute from the payload. So any writer on the swarm
network could attribute metrics to any hive, and nothing downstream could
tell.
The label now comes from which receiver accepted the sample: one receiver
per hive, each behind an `oidc` extension verifying a token minted for that
hive's audience, each feeding a pipeline whose `resource` processor upserts
a constant. A sender cannot influence it, because the only input is which
authenticated port the bytes arrived on.
That multiplicity is forced rather than preferred. A processor cannot read
the token's claims — `from_context` reads request metadata, and asking it
for an auth claim yields nothing, silently, with a healthy startup — and
one receiver holding several credentials never reveals which one matched.
The per-hive ports are internal: a hive reaches its receiver as a path
under this collector's existing gateway name, so nginx (rendered from this
same evaluation) is the only thing that names a port. Fronting each hive
with its own vhost would need a certificate, a DNS name and a gateway entry
per hive to express routing the gateway already does.
Turning this on removes the unauthenticated receiver. While an open port
still accepts samples the per-hive receivers are decoration, so this is the
switch itself rather than a hardening layer beside it; a swarm that wants
the open receiver says so.
`hive-ca-trust.nix` grows `bundlePathFor`, because a consumer taking its own
CA argument has to name the bundle rather than just have `SSL_CERT_FILE`
exported at it.
A swarm service that has to tell hives apart needs the token itself to
say which hive presented it. Two client-registration fields were missing
for that, and both are properties of the client rather than of any one
consumer:
- `audience` registers which `aud` values a client may request. It is a
boundary, not a label: asking for one that is not registered is
refused with `invalid_target`, so a hive cannot mint a token for
another hive's slot.
- `accessTokenSignedResponseAlg` switches the access token from
authelia's default opaque handle to an RFC 9068 JWT, which is what
lets a resource server verify a token against `/jwks.json` instead of
asking authelia about every request.
Each hive's audience is its own client id rather than a new per-hive
string. The identity is the directory's, not the consumer's, and the id
is already published as `hiveClientPrefix` precisely so a second
consumer does not carry its own copy of the naming rule.
Both fields default to authelia's own behaviour, so no existing client
changes.
The hive tier has to *present* a credential to the swarm tier, and the
upstream collector build ships no auth extensions at all — `oauth2client`
exists only in contrib. So this is what makes authenticated egress
expressible, not a preference between two equivalent packages.
It also matches the swarm tier, which has been on contrib since it was
written. Two tiers of the same pipeline built from different component
sets is a difference nobody would predict from reading either module.
Not a build-farm cost: contrib is fetched, not compiled.
No behaviour change on its own — nothing names an extension yet. The
config that does lands separately, because that is the commit whose
failure mode needs a running collector to detect: `otelcol validate`
accepts a receiver naming an absent extension and the process then dies
at startup, so a green build proves nothing about it.
Review feedback from mara ("does the otel info even belong in there?") and
argus: `docs/observability.md` already states twice that the collector's
bridge port is contributed automatically and that neither line is needed for
hyperhive's own telemetry. A second copy of that reasoning in the option
description is a third place to drift from — and drift is what produced the
misleading example this PR started out fixing.
So remove the explanation rather than relocate it. With no otel content in
the option at all there is nothing to keep in sync, and observability.md
stays the one place that owns the otel-specific reasoning.
`example = [ 4318 ]` goes too: 4318 is the OTLP port, so the example still
pointed at otel without naming it.
Also fixes the same confusion one doc over, since it is this PR's subject:
observability.md's own "Network access" section used `otel.endpoint` as the
example for pointing something *else* at a host-local service, immediately
above a warning that you do not need it for otel. It now uses a generic port
and names the reason loopback is wrong from inside a container. While there,
the bridge is `hive-br0`, not `hvbr0` — the only occurrence in the tree.
Refs #3402
`exposeHostPorts`'s worked example told the operator to set
`services.hyperhive.otel.endpoint` to a bridge address. That was true
before the collector tiers existed. It is now actively wrong: `enable`
contributes the hive collector's port to this list itself and derives
the agent-facing endpoint, while `endpoint` names where telemetry goes
after it leaves the swarm and is read by the swarm's collector. An
operator following the old example would point the upstream link at a
bridge address.
Replace it with an example that is actually the option's job -- a
host-local service the operator runs themselves -- and say plainly that
hyperhive's own telemetry needs nothing here, which docs/observability.md
already states.
hive-c0re's container-resource exporter has POSTed to a 404 for as long as
it has existed, silently: it passed the collector's base address to
`with_endpoint`, which the SDK takes verbatim, so every export went to `/`
instead of `/v1/metrics`. Nothing reported it — OTLP export failures go to
an error handler no binary here installs — so the daemon logged "exporter
enabled" and delivered nothing. VictoriaMetrics has never held a sample
under `service.name=hyperhive-c0re`.
Fix the way the rest of the repo already resolves an endpoint: an endpoint
option names a BASE, and the layer that knows the signal appends to it.
`hive-metric` — same SDK, same collector — never calls `with_endpoint`, and
`docs/observability.md` documents the append as system behaviour; the one
place a full path is spelled out is the VictoriaMetrics exporter, because
its far end is not a standard OTLP path.
So drop the call. The builder is now byte-identical to hive-metric's, and
hive-c0re's unit carries the standard `OTEL_EXPORTER_OTLP_ENDPOINT` for the
SDK to read. The address is bound once in nix and consumed twice, so what a
hive hands its agents and what it exports to itself cannot drift.
The enable signal moves to that same standard variable: "configured" and
"where it actually goes" become one string rather than two that agree by
convention. `HYPERHIVE_OTEL_*` keeps its own job, the agent-config
transport meta.rs reads — a name the SDK has never known, which is the bug.
The test changes shape with the fix. The old one asserted a URL this module
built; the new one pins that the exporter is gated on the variable the SDK
itself reads, because the fix is now an absence and an absence is what a
later "the endpoint is right there, just pass it" edit puts back.
Refs #3402
hive-c0re's container-resource exporter already targets this hive's own
collector (environment.nix derives the bridge address), so the upstream
header it was loaded with has nowhere to be presented: that hop is
unauthenticated for every producer on the host, and the credential
belongs to the swarm tier, which is the one that leaves the swarm.
Drop the LoadCredential entry and the auth_headers() reader with it.
The option itself stays -- swarm-otel.nix is its real consumer, via
EnvironmentFile on the collector unit.
Also corrects three descriptions that this makes false, or that were
already false: the module doc claimed to reuse the config "Claude Code's
in-container SDK export uses", which stopped being true when agents
moved off that path; the nix comment claimed the secret is "the same one
the agent containers get, forwarded via nspawn --load-credential", which
lost its last producer earlier; and docs/observability.md described an
Authorization header on a hop that will no longer send one. The
headersCredential option's own docs already said it reaches "neither an
agent container nor a hive's own collector" -- this makes that true
rather than aspirational.
The endpoint, protocol, temporality preference and resource labels were
shipped only inside claude's managed settings json, so they applied to
claude's own process. hive-bash-daemon, hive-mcp-http, hive-matrix-daemon
and hive-forge-notify are systemd *siblings* of claude rather than its
children, so nothing shipped there could ever reach them: `hive-metric`
invoked from a tool call exited with "OTEL_EXPORTER_OTLP_ENDPOINT not
set", which is the honest failure of a value it structurally could not
see.
Declare those variables container-wide in a new agent module instead —
systemd.globalEnvironment for every unit PID 1 starts, environment.variables
for login shells. Both are needed and neither implies the other; NIX_REMOTE
is set both ways for the same reason.
Claude keeps only what is genuinely its own: the telemetry master flag,
the feedback-survey flag, the version label, and which signals it
exports. A different producer in the same container may legitimately
emit only metrics.
The hyperhive.otel.* options move across with them. They have more than
one consumer now, so their home is the OTEL module rather than the
claude one.
Setting declarativePlugins to any list -- the empty one included -- also
sets upstream's plugins.preinstall_disabled, which nixpkgs defaults to
declarativePlugins != null. Preinstall is what fetches the Drilldown apps
on first start, so claiming the plugin directory removed them without
touching anything a person had installed.
The comment above that line reasoned about the plugin path and the update
check and never about preinstall, and its closing claim -- nothing is
taken over on a hive with no plugins -- was true only of plugins a PERSON
installs. Upstream installs some itself, and those are the ones that went
missing.
Listing it as a package is better than the preinstall it replaces: store
pinned, no runtime fetch from grafana.com, survives a state reset. Its
Logs/Traces/Profiles siblings stay off -- they front Loki, Tempo and
Pyroscope, none of which this swarm runs.
environment.nix explains why HYPERHIVE_OTEL_HEADERS_CREDENTIAL is not
emitted by naming the machinery that used to consume it -- machinery this
PR deletes. Left as written it would describe a removed nspawn credential
and a removed oneshot as if both still ran, in the same paragraph that
justifies withholding the variable.
That is this PR's own defect inverted: it exists so an auditor asking
'can an agent obtain the OTEL token?' finds nothing misleading, and a
comment claiming live delivery machinery is exactly the kind of thing
that costs an auditor a reconstruction.
Keeps the rule rather than the history -- one holder, on the host -- since
that is the part still worth reading.
The host-side collector is the only path telemetry leaves a hive, so
HYPERHIVE_OTEL_HEADERS_CREDENTIAL is never emitted and everything
downstream of it is unreachable. What made it worth removing rather than
leaving inert is what it looked like to a reader: a complete,
well-commented mechanism for writing the hive's upstream credential into
a file the agent can read, described in the present tense. Anyone auditing
"can an agent obtain the OTEL token?" had to reconstruct the whole env-var
chain to find out the answer is no.
Gone: the per-agent `hyperhive.otel.headersCredential` option, the
`hive-otel-header` oneshot that merged OTEL_EXPORTER_OTLP_HEADERS into the
agent's own settings.json, and meta.rs's field, env read and render.
⚠️ Scoped by NAMESPACE, not by name. `hyperhive.otel.headersCredential`
(per-agent) and `services.hyperhive.otel.headersCredential` (host) are
different options sharing a leaf name — the host one is read by
`stats/otel_metrics.rs` for c0re's own container-resource exporter and
stays. Sweeping the string would have taken out working code.
The comment above `otelSettingsEnv` now states the property rather than
the absence: there is no auth header and no mechanism to add one, because
an agent exports to the hive's own collector and nothing an agent can read
is a secret to the swarm. The old behaviour is named in the past tense so
it reads as removed rather than overlooked.
meta.rs's assertions that pinned the injection are deleted rather than
adjusted; the surrounding test keeps covering extraResourceAttributes and
the endpoint/protocol injection, which are live.
hive-tls-ca re-signs at service activation -- the rebuild itself --
while hive-tls-resign only fires from a weekly timer. The previous commit
put the coverage check in the timer unit, so a corrected serviceDomains
would not have taken effect until up to a week after the deploy that
changed it. Same bug one level along: found a trigger, not the trigger.
Two guards now share one definition rather than each carrying their own,
because a rule enforced in one and not the other is worse than one
enforced in neither -- it looks fixed and only fires on whichever path
you did not take.
Also widens hive-tls-ca's condition to the services leaf. Both leaves are
signed inside that block but only the hive leaf gated it, so a fresh
gateway.pem suppressed the re-signing of a swarm-services.pem that was
missing or stale.
Expiry was the only re-sign trigger, so a leaf signed when the name set
was smaller stayed valid -- and wrong -- for its whole lifetime. Adding a
service to swarm.serviceDomains reissues the sub-CA (its own .names
reconciliation) but nothing regenerated the leaf nginx actually serves,
which left the previous commit's config change unable to fix anything on
a hive whose gateway leaf was not near expiry.
covers() reads the DNS names back out of the certificate rather than a
sidecar file: the pem is what nginx serves, and a bookkeeping file drifts
from it the moment a leaf is replaced by hand. Applied to both leaves --
the hive leaf has the same defect if the hive domain ever changes.
swarm.serviceDomains is what gateway.lib.tlsFor consults to pick the
services leaf over the hive leaf. grafana, victoriametrics and otel each
claim a gateway name under the swarm apex but were absent from that list,
so their vhosts were served the HIVE certificate -- which cannot cover a
name under a different apex.
Invisible until a machine client hit it: a name mismatch is a
click-through warning in a browser and an outright refusal in an OTLP
exporter. The metrics UI and store looked healthy while the collector
failed every POST and dropped the samples.
tls: failed to verify certificate: x509: certificate is valid for
probe.example, *.probe.example, not otel.swarm.example
Drops swarm.otel.url (a loopback default an operator had to override on a
split host) in favor of swarm.otel.domain -- the same
gateway.localNames + nginx-vhost-through-the-gateway shape every other
swarm service (authelia, grafana, victoriametrics, ui) already uses. The
hive tier's exporter now reaches it as https://<domain> unconditionally,
resolved locally by dnsmasq on a co-located host and over the real network
otherwise, instead of a config knob nobody sets until they hit the silent
drop.
Costs CA trust on the hive tier: otel.nix wires
lib/hive-ca-trust.nix's trustBundle with hostUnit = true on the
opentelemetry-collector host unit, the same flag #3441/#3442 added for
swarm-controller and hive-c0re.
mara, #3125 comment 58363: "go c".
All four sibling swarm containers import swarm-container-resolver.nix;
this one did not. It matters more here than most: otel.endpoint is an
operator-configured external hostname, and reaching it is the entire
reason this container holds a credential.
Also aligns two details with those siblings - the enable default is
asserted from swarm-required-services.nix with the metrics pair it
feeds, so that file remains the one place a service host is declared,
and machine is readOnly since its description already calls it a fact
rather than a knob.
The pre-push lint refuses them, and rightly: a comment that names an
issue number ages into a pointer at a closed thread. The constraint each
one carried is stated directly instead.
A collector serves its own metrics on localhost:8888 unless told
otherwise, and co-located tiers share a network namespace, so the second
one to start dies with 'bind: address already in use'.
The port appears in neither config - it is a default inside the binary -
so comparing the ports the configs name reports them distinct. A
behavioural probe found it by being unable to start the chain.
metrics.address is the spelling that looks right and is rejected by this
version ('migration.MetricsConfigV030' has invalid keys: address);
readers is the schema it accepts.
The hive collector's only exporter becomes the swarm's collector, and
the upstream credential, the metrics-store exporter and the choice of
destination all move one tier up.
Its assertion goes with them: 'endpoint or a local store' was the right
rule while this tier picked the destination, and is the wrong one now.
A hive that runs no swarm services has neither, forwards to a swarm
collector elsewhere, and is correctly configured — the rule that
replaces it lives in swarm-otel.nix, where the destinations are.
The option descriptions here described a topology with one collector in
it: endpoint and protocol are not what agents are handed (they get the
derived first hop, see hive-c0re/environment.nix), and the credential is
not read by this tier.
The swarm tier is the only holder of the upstream credential, the only
writer to the swarm's metrics store, and (once #3283 lands) the place
that stamps hive= from the authenticated connection rather than from
anything a sender can choose. Today one collector does both tiers' jobs,
which works only because they land on one box.
A container rather than a second host unit, for two reasons that agree:
every sibling swarm service is one, and `services.opentelemetry-collector`
is a singleton NixOS option already spoken for on the host by the hive
tier. A container gets its own evaluation and therefore its own
collector.
Port defaults to 4319, deliberately not the OTLP default 4318 the hive
tier uses: swarm containers share the host netns, and two listeners
claiming one port is not a build failure but a runtime coin toss with
nothing in any log saying so -- the same collision grafana and the forge
hit on 3000.
`url` is an option with a co-located default rather than a loopback
literal in the exporter, so a split-host deployment is a config change
instead of a code change.
Wires nothing yet: the hive tier still exports directly, and switching it
over is the next commit.
hive-c0re is a host unit that wired no CA trust at all. Harmless today
-- every endpoint it is handed defaults to plain http or loopback, so
nothing verifies a certificate -- and latent for the split-host shape the
options invite: `swarm.matrix.apiUrl`'s own example is an https URL, and
pointing it or `statusPublish.natsUrl` at another hive's gateway means
verifying a leaf signed by a CA generated at runtime.
Unlike the other consumers of this helper, hive-c0re already makes a
public-TLS call -- the OTEL exporter reaches whatever `otel.endpoint`
names. `SSL_CERT_FILE` replaces the trust store rather than adding to it,
so the bundle being system CAs + hive CA is what keeps that path working;
narrowing it to the hive CA alone would fix a case nobody hits yet and
break one that runs today.
New GET /api/swarm on swarm-controller, backed by
services.hyperhive.swarm.name (SWARM_CONTROLLER_NAME env var, same
optionalAttrs-gated-on-option-resolving shape queueEnv/forgeEnv/etc.
already use). swarm-ui's <Shell> fetches it once and sets both
document.title and the header's brand text; falls back to the
existing static "hyperhive swarm" label when the operator never set
a name or the fetch fails.
Extracted the swarm-queue connect block out of main() into its own
connect_status_reader() fn to keep main() under clippy's line-count
lint after adding the new field wiring — no behavior change, same
comments moved as-is.
The controller's forge client speaks TLS to `https://<forge domain>`,
which the gateway serves with a leaf signed by the hive CA. That CA is
generated at runtime, so nothing build-time can name it and it is not in
the system store -- and `reqwest`/`rustls` resolves roots through
`rustls-native-certs`, whose `SSL_CERT_FILE` *replaces* the store rather
than adding to it. With no bundle wired, every forge call failed
`invalid peer certificate: UnknownIssuer` and agent creation died at its
first step.
`lib/hive-ca-trust.nix` already solved this, but only for containers: it
sources the CA through `/run/hive-ca/trust-bundle.pem`, a bind mount that
does not exist on the host. `hostUnit` makes it read the host copy and
wait on `hive-tls-ca.service` itself -- one flag driving both, because a
host source without that ordering is a race.
`enable` is the other half, and it is the sharp edge: a container caller
imports this into the container's module set, so it disappears with the
container. A host caller imports it at the host's top level, where
`imports` is unconditional -- without the flag, a hive with the
controller turned off would get a bundle oneshot and a `swarm-controller`
service conjured by `genAttrs`, holding an `SSL_CERT_FILE` and no
`ExecStart`.
`serverName` is baked irrevocably into every user and room id, so a hive
that rebuilds onto a new default is a *different homeserver*, not a
renamed one: existing accounts and rooms are stranded, and reverting the
config does not undo it. Its neighbours (`gatewayHost`, the forge domain)
are routing, rediscovered through `.well-known` and fixed by editing them
back. Same diff shape, three orders of magnitude apart in blast radius --
which is an asymmetry a module should carry rather than an operator.
An activation script and not `warnings`, which is where this obviously
belongs and does not work: the condition needs the host filesystem, and
`nixos-rebuild switch --flake` evaluates purely, where
`builtins.pathExists "/var/lib/..."` answers false rather than throwing.
A `warnings` entry gated on it would evaluate, deploy, and print nothing
on every real deployment.
Rendered only when `serverName` is null, so a pinned hive has no script
rather than a script that stays quiet -- a guard that cries wolf at a
correctly-configured deployment makes the next real one read as noise.
Never fails the activation: it warns about a choice that cannot be
undone, and refusing the rebuild of a hive that already chose
deliberately is the opposite of helping.
The probed path is read out of the container's own evaluated config
rather than hardcoded. A guessed path resolves cleanly and silently never
matches, which is the same failure this guard exists to catch one level
up.
mara asked whether SWARM_CONTROLLER_PUBLIC_URL should also be set when the
controller does not run on the same host, and then said the two read as one
service to her. Both are worth answering in the file rather than only in the
thread: that split is not representable today, and the reason is two hops
away from this line.
The UI's /api/ location proxies http://unix:<socketPath> -- a path that
resolves nowhere else -- and this daemon binds no TCP address at all. So the
vhost and the daemon are co-located by construction, and ui.enable is the
flag that declares the vhost rather than a guess about some host serving it.
Also names the tripwire: the day the daemon grows a TCP listener, this line
silently stops being right (env unset, registration quietly skipped, no
error). That is when an explicit publicUrl option becomes correct -- not
before, while there is exactly one derivable answer.
The endpoint landed inert: nothing pointed at it, so the only way to see
it work was to mint an HMAC by hand. Register the two swarm-wide hooks
at startup so a real forge event produces a journal line.
Registered ALONGSIDE the per-hive hooks, not instead of them. Every hive
keeps receiving and acting on its own deliveries; the controller gets a
copy and logs it. Moving the registration is a later step and has to be:
fan-out swarm->hive does not exist yet, so a hook moved now would point
at a receiver that forwards nowhere, silently on both sides.
Deliberately no stale-hook deletion arm, unlike the two per-hive
registrars this otherwise mirrors: theirs delete hooks matching their own
path with a foreign base, and the hives' hooks are not stale.
The route prefix is what keeps this safe. Both hive-side registrars
delete any hook ending in /webhook/knowledge or /webhook/config-pr with a
different base, so a swarm hook under those paths would be deleted by
every hive on every boot. Serving them under /webhook/forge/ avoids it,
and a test pins it -- there is nothing else that can.
SWARM_CONTROLLER_PUBLIC_URL is set only where the swarm vhost is served,
because a hook whose target_url nothing answers is worse than no hook.
A forge webhook is a machine POST carrying an HMAC and no session cookie,
so it cannot pass the authelia auth-request subrequest every other location
on this vhost uses. This location deliberately omits it; the HMAC check in
the controller is what guards the path.
Scoped to /webhook/forge/ rather than /webhook/ so a future endpoint under
the same prefix does not inherit the bypass.
The bridge no longer has a private canonical store, so the env var
naming one is gone rather than repointed. What is left is the single
users file it now reads and writes directly.
That line is what made `swarm agent create` fail on this hive: the
bridge found no store at its own JSON path, saw users already in
`users.yml`, and refused to overwrite a file it had not written --
correctly, given two things claimed to be canonical for one file.
Plugin management is server-admin scoped, and on an SSO hive nobody holds that
role: auto_assign_org_role grants an org role, and the built-in local admin
that does hold server admin cannot log in because the login form is disabled
whenever SSO is configured. Two individually-correct decisions leaving no path
to the plugin UI at all.
Declarative is the way through rather than a workaround for it -- plugins land
in the store and in git, survive a rebuild and a state reset, and the container
needs no runtime egress to grafana.com.
mkIf rather than passing the list through: upstream's default is null, while an
empty list is a real value pointing the plugin path at an empty store dir, so a
hive that sets nothing must keep seeing null.
Last of the three modules that hand-rolled the same concat with wantedBy +
before and no requires, so a failed assembly left the consumer running against
a missing file and trusting nothing -- every outbound TLS call fails while the
unit looks healthy.
The forge is the one with two consumers: forgejo-sso-source fetches the
issuer's discovery document over the swarm CA and once shipped without the
trust its sibling had. It only exists when SSO is on, so the consumer list is
conditional -- naming an absent unit would define a serviceless one and order
nothing.
Removes the now-dead useSelfSigned and caContainerPath bindings (nix does not
warn) and retargets three comments the deletion orphaned, including the helper
header that still named this module as the per-call-site concat.
Both hand-rolled the same concat with wantedBy + before and no requires,
so a failed assembly let the consumer start against a missing file and
trust nothing at all -- which fails every outbound TLS call while the unit
looks healthy. The helper puts requires on the consumer and verifies the
assembled bundle before moving it into place.
Removes each module's now-dead useSelfSigned binding: its only use was the
mkIf on the deleted block, and nix does not warn about an unused let
binding. Also corrects two comments the deletion orphaned -- one described
the removed path binding, the other pointed at a bundle service that no
longer exists.
A loopback literal encodes 'authelia is in my netns' at the call site,
and authelia's OIDC endpoints are https-only in effect: reached directly
they answer 400, because the forwarded headers nginx injects for every
other consumer are what let it determine its own issuer. Going by name
deletes the need for those headers rather than reproducing them, and
converges on the URL swarm-nats-auth already uses.
Depends on the CA trust added for the same container in #3407 -- without
it this swaps a 400 for an UnknownIssuer.
The null-url assertion joins the module's existing list: interpolating a
null would surface as a nix coercion error several files from its cause.
Same class-B defect as the queue responder: swarm-authelia-bridge
introspects authelia by name over https and the container trusted no swarm
CA, so the call could only ever fail UnknownIssuer.
This is also the prerequisite for the by-domain introspection change on
#3391 -- flipping that URL without the trust half would swap one failure
for another.
The responder introspects authelia over https by name. It had no CA trust
at all, so the handshake failed UnknownIssuer, introspection failed, and it
denied every client -- surfacing at the controller as a 60s
authorization-violation loop, two layers from the cause.
Adds a shared trustBundle helper to lib/hive-ca-trust.nix rather than a
fifth hand-rolled concat. Four containers were each assembling this
themselves, which is how they came to share one defect: wantedBy + before
express ordering but not success, so a failed assembly let the consumer
start against a missing file and trust nothing at all.
The helper fixes both halves of that. requires goes on the consumer, so a
failed bundle stops it and the dependency is visible in systemctl status
where someone debugging a TLS failure looks. And the script assembles to a
temp path, checks the result actually contains a certificate, and only then
moves it into place -- cat of an empty bind exits 0, so set -e does not
catch it and a partial bundle must never appear under the final name.
Returns a module rather than bare services: a caller that already writes
systemd.services.<consumer> cannot also write systemd.services in the same
attrset.
Keeps why the setting exists (the bridge writes this file and cannot
restart authelia) and drops the incident narrative and the answered
objections -- that history belongs in the PR discussion, not in three
comment blocks a future reader has to scroll past.
authelia now watches the users file, so the restart is redundant -- and it
was the wrong shape twice over. It could fail: a login was refused for a
user whose record was already correct on disk, with nothing in either log
implicating the reload. And it only ever worked for this writer --
swarm-authelia-bridge writes the same file and cannot restart anything,
since running unprivileged inside the container is the whole reason it may
write it at all. A reload that depends on which process did the writing is
not a reload.
--machine/--unit and their two env vars existed solely to name a
systemctl -M target, so they go with it. That drops two required settings
from the operator surface.
The three objections previously recorded against watch are all answered
now, and are kept next to the decision rather than deleted: the key is
verified against the pinned build (validate-config accepts it and rejects
a misspelling), the watch is on the directory so a rename is observed, and
partial reads are structurally impossible because every writer of this
file goes through write_atomic.
