The responder introspects authelia over https by name. It had no CA trust
at all, so the handshake failed UnknownIssuer, introspection failed, and it
denied every client -- surfacing at the controller as a 60s
authorization-violation loop, two layers from the cause.
Adds a shared trustBundle helper to lib/hive-ca-trust.nix rather than a
fifth hand-rolled concat. Four containers were each assembling this
themselves, which is how they came to share one defect: wantedBy + before
express ordering but not success, so a failed assembly let the consumer
start against a missing file and trust nothing at all.
The helper fixes both halves of that. requires goes on the consumer, so a
failed bundle stops it and the dependency is visible in systemctl status
where someone debugging a TLS failure looks. And the script assembles to a
temp path, checks the result actually contains a certificate, and only then
moves it into place -- cat of an empty bind exits 0, so set -e does not
catch it and a partial bundle must never appear under the final name.
Returns a module rather than bare services: a caller that already writes
systemd.services.<consumer> cannot also write systemd.services in the same
attrset.
Keeps why the setting exists (the bridge writes this file and cannot
restart authelia) and drops the incident narrative and the answered
objections -- that history belongs in the PR discussion, not in three
comment blocks a future reader has to scroll past.
authelia now watches the users file, so the restart is redundant -- and it
was the wrong shape twice over. It could fail: a login was refused for a
user whose record was already correct on disk, with nothing in either log
implicating the reload. And it only ever worked for this writer --
swarm-authelia-bridge writes the same file and cannot restart anything,
since running unprivileged inside the container is the whole reason it may
write it at all. A reload that depends on which process did the writing is
not a reload.
--machine/--unit and their two env vars existed solely to name a
systemctl -M target, so they go with it. That drops two required settings
from the operator surface.
The three objections previously recorded against watch are all answered
now, and are kept next to the decision rather than deleted: the key is
verified against the pinned build (validate-config accepts it and rejects
a misspelling), the watch is on the directory so a rename is observed, and
partial reads are structurally impossible because every writer of this
file goes through write_atomic.
Authelia reads authentication_backend.file.path once at startup. Without
watch, an identity the bridge creates is real on disk and invisible to the
running authelia until something unrelated bounces the unit.
swarm-authelia-bridge cannot restart authelia -- running unprivileged
inside the container is the whole reason it may write that file, so
restarting the unit is exactly the privilege it was designed not to hold.
swarmctl does restart it, but that is a systemctl -M shellout that can
fail, and when it did the symptom was a login refused for a user whose
record was already correct on disk. This makes correctness stop depending
on that restart succeeding.
The auth-callout responder decides what an admitted client may publish
from two strings: the prefix marking a hive client, and the client id
allowed to read every hive's key. Both were literals in three places --
swarm-authelia.nix mints "hive-${name}", swarm-controller.nix defines
"swarm-controller", and the responder carried its own copies as clap
defaults because swarm-nats.nix passed neither.
Each producer now publishes its value as a readOnly option and the
responder's ExecStart reads them, so the agreement is one evaluation
rather than three strings that happen to be equal. Same pattern the
module already uses for `--account`, and the same argument
swarm-authelia.nix gives for publishing `machine` and `unit`.
Worth the change because the failure is silent and misattributed:
rename either principal and the responder starts denying the one that
stopped matching, a denial reaches a NATS client as a timeout rather
than an error, and a hive that is refused looks exactly like a hive
that has not reported yet.
A swarm service container shares the host netns and force-disables
resolvconf, so it inherits the /etc/resolv.conf nixos-containers copies in
at start (cp --remove-destination, host-side preStart, once per start) and
nothing ever refreshes it. That makes the container's resolver a snapshot of
the host's file at its boot instant.
When that snapshot is wrong the container can never recover, and the symptom
appears arbitrarily far from the cause: swarm-nats-auth cannot resolve
authelia's name, so it denies every auth-callout request and the queue
refuses every client with an authorization violation.
Give each of the four swarm containers a oneshot that writes the resolver
file itself, from the bridge IP, ordered before that container's first DNS
consumer. The shape is the one every agent container already uses.
networking.nameservers cannot do this: resolvconf is its only consumer and
these containers disable it, so setting it renders no file while still
evaluating cleanly. A static environment.etc entry cannot either -- it would
have to survive etc activation landing on the regular file the host already
copied there, which no eval can show.
The unit restricted RestrictAddressFamilies to AF_UNIX, which was correct
while the daemon only served its unix socket. It has since grown three
outbound clients -- authelia token minting and forge calls over HTTPS, and
the queue over NATS -- and every socket(AF_INET, ...) was refused by seccomp.
systemd surfaces that refusal as EAFNOSUPPORT, "Address family not supported
by protocol", so the failure names the protocol and never the sandbox. The
visible symptom was swarm agent creation failing while minting a bearer
token, with a connect error that reads like a network fault.
Permit AF_INET/AF_INET6 for those clients and AF_NETLINK, which glibc's
getaddrinfo needs to enumerate local addresses before returning one. The
rest of the unit's hardening is unchanged.
Per the operator's call on #3372: rather than moving Grafana off the
forge's 3000 to another number, take it off TCP entirely.
The collision was possible because every swarm service container shares
the host's network namespace, which makes a port a swarm-wide resource
two modules can each claim believing it free — Grafana took upstream's
3000, so does the forge, and `grafana.<swarm-domain>` served the forge
with no bind error and nothing in any log. A socket has a path, and a
path collision is a build-time conflict rather than a runtime coin toss.
Three parts, none of which works alone:
- `protocol = "socket"` with `socket_gid` = nginx's static gid 60.
- Grafana joins that gid inside the container. A non-root process may
only chgrp to a group it belongs to, and a container has its own user
database — without the membership Grafana starts, the chown fails, and
the socket is simply unreachable.
- The socket dir is created host-side by tmpfiles and bind-mounted in. A
container's /run is an nspawn tmpfs, so it is not visible from the host
at /var/lib/nixos-containers/<name>/run; the same shape the per-agent
web.sock already uses.
The `port` option is gone rather than deprecated — nothing can set it to
a colliding value if it does not exist.
`swarm-controller.service` carried `SetCredential=queue-client.secret:` — an
empty value, which systemd's parser refuses:
/etc/systemd/system/swarm-controller.service:38:
Invalid syntax, ignoring: queue-client.secret:
So the line was dropped on every daemon-reload, and `LoadCredential=` was
fatal again — precisely the failure that default was added to prevent. On a
hive where authelia has not yet minted the secret, the controller refuses to
start rather than coming up with the queue unconfigured.
It looked correct for days because the credential file happened to exist, so
the fail-soft was never exercised.
Measured with `systemd-analyze verify`: an empty value is rejected, any
non-empty one is accepted. The placeholder is a real word rather than filler —
it reaches the token request as the client secret, so authelia refuses it and
the journal names something an operator can act on.
Gate: state/eval-setcredential.sh, with the parent commit as its mutation.
nixpkgs dropped the default for `services.grafana.settings.security.secret_key`
and asserts on null, so the module I merged an hour ago fails the build on any
host that enables it. That is a broken deploy, not a warning.
Generated in-container like authelia's own keys and for the same reason:
nothing outside the container ever reads it. Generated ONCE and kept — the
key signs Grafana's stored datasource secrets, and rotating it does not
re-encrypt what it already encrypted, so a fresh key per boot would leave
Grafana unable to read its own database. Delivered as `$__file{}`, so it
never enters the store.
The gate missed this because a container is a separate NixOS evaluation with
its own assertions: forcing the host's `config.assertions` never touched the
container's. `state/eval-3265.sh` now forces
`containers.swarm-grafana.config.assertions` and reports any that fail.
Review catch: this PR relaxed the `otel.endpoint` assertion and staled the
canonical OTEL reference in the same stroke — `docs/observability.md` is
what CLAUDE.md points readers at for "what OTEL options are available",
and it still said required-full-stop while the new swarm/services.md
section said a local store satisfies it.
Also corrects the option's own description in otel.nix, which said the
same thing and renders into the generated options doc. Grepping the
reviewer's phrasing did not find that one; grepping the claim did.
Records the second destination where the "endpoint is where telemetry
ultimately goes" paragraph makes its claim, rather than only in the new
section a reader may not reach.
Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
A local time-series database rather than only an external sink, so the
swarm dashboard stays readable when the outside world is not: a view of
the system must not depend on the system it views being healthy.
listenAddress is pinned to loopback. Upstream defaults it to every
interface, and the OTLP ingest path this exists to receive on is
unauthenticated — the gateway is the only intended client and it is on
this host, so a wider bind would publish a write endpoint to whatever
the host is reachable on.
retentionPeriod defaults high rather than being required, because the
two failure directions are not symmetric: too long fills a disk, which
is visible and recoverable by lowering it, while too short destroys
history silently and permanently. The operator lowers it once they have
measured how fast this swarm accumulates.
OTLP needs no flag. Measured against the pinned 1.146.0 rather than
inferred from the module's option list, which has no OTLP switch and so
reads as though the feature were missing: the running server answers
POST /opentelemetry/api/v1/push with 200, where a nonexistent path
answers 400.
Three host units poll up to 120s for a secret authelia mints on its first
boot, and all three are `Type=oneshot` with no `TimeoutStartSec`. systemd's
`DefaultTimeoutStartSec` is 90s, so it kills them at 90 — before the script
reaches its own `exit 1` and names the file that never appeared.
The wait itself is fine; what's lost is the diagnosis. On a fresh hive the
operator gets a bare start-timeout instead of "authelia has not minted
<path>", several layers from the container that was actually slow.
Found while writing the same unit for Grafana, where the timeout is set —
so this is the existing three catching up with it, not a new pattern.
Defining an environment key on a unit that does not exist creates a
fragment for it: inert, never activated, but present on every
non-controller hive that has a CA. hive-c0re runs everywhere so its
line needs no guard; the controller runs on one host in a swarm.
Caught in review. It evaluates and builds clean either way, which is
why it took a reviewer rather than a check.
The queue client built a bare reqwest::Client, so it trusted only the
platform roots. Against a swarm whose authelia is signed by the swarm
CA that is fatal: minting a token dies with 'invalid peer certificate:
UnknownIssuer', inside the auth callback, on a four-second retry loop,
with the queue never connecting.
The anchor was never missing. hive-tls.nix assembles trust-bundle.pem
and already hands it to hive-c0re as HIVE_TLS_CA_PATH; nothing pointed
the queue client at it.
QueueConfig gains an optional ca_file from <prefix>_OIDC_CA_FILE, read
outside the all-or-none tuple on purpose: a CA path with no queue is
meaningless rather than half-configured, and requiring it would break a
swarm fronted by a public certificate in order to fix one that is not.
add_root_certificate extends the default roots rather than replacing
them, so both deployments work.
A bad path fails loudly instead of falling back to the platform roots.
An operator who names a CA file wants that anchor; a silent fallback
turns their typo into UnknownIssuer five layers away.
hive-tls.nix names the bundle for both clients, beside the line that
already does it for hive-c0re, rather than having each consumer
re-derive the path.
The responder's third credential is not minted by the generator beside
it — authelia produces it on first boot, inside its own container, and
nothing ordered this unit against that. Losing the race cost the entire
queue: install exits 1, the responder never starts, and auth_callout
with no responder refuses every client. Fail-closed by design, so the
symptom lands on every queue client and nowhere near the cause. Seen on
a real boot at 20:31:31.
Now it waits for the file, bounded at two minutes. Where authelia runs
on another host the secret is never going to appear, and blocking the
queue container indefinitely would replace a clear failure with a hang;
after the timeout this fails exactly as it did before, having first
given the co-located case the seconds it needs.
TimeoutStartSec is set rather than left to the default because the
default is ninety seconds — a two-minute wait would be killed at ninety
and the operator would get a generic unit timeout instead of the
message naming the file.
Ordering after the authelia container is necessary and not sufficient:
the container being up says nothing about whether its in-container
secrets unit has finished. It only stops this spinning for the full
timeout on every boot.
A systemd credential named by an absolute path is fatal when the file is
missing, and the co-located queue secret is minted by authelia's first
boot in another container — which a host unit cannot order against. On a
real boot the daemon spent three of systemd's five default starts losing
that race before the file appeared; two seconds more would have reached
start-limit-hit, which does not self-heal.
An empty SetCredential acts as the default that makes the load
non-fatal, so the controller starts and serves its HTTP surface with the
queue unconfigured — a shape it already reports.
That alone would be a regression, though: a credential is snapshotted at
unit start, so the placeholder would freeze and the daemon would sit
degraded forever instead of recovering the way the restart loop
accidentally did. A path unit on the secret file closes it, and closes a
second gap in the same stroke — mint_token reads the secret on every
call precisely so a rotation takes effect without a restart, and a
snapshot in %d silently defeats that.
PathChanged and not PathExists: the latter activates immediately when
the file is already present at unit start, which would restart a healthy
daemon on every boot.
The option being consumed already warned about this shape — its own
description says a consumer has to wait for the secret because a
missing source turns a fresh hive into a boot-order deadlock. That
warning was written on the producing side and did not fire while I was
writing the consumer.
The client list was `cfg.oidc.clients ++ hiveClients`, where the first
half comes through the submodule and the second was a raw attrset from
this module's `let` block. That list is only half-typed: a field added
to the submodule exists on the declared entries and not on the derived
ones, so reading it plainly is an eval error the moment hive identities
are on. The operator asked whether it should be uniformly typed instead
of guarding each read, and it should.
The hive identities are now declared the same way an operator declares
a client, so the module system applies the submodule to them and every
option's default is present. Downstream reads one uniformly-typed list
and the guard added for the field that broke is gone with it.
allClients concatenates two shapes: cfg.oidc.clients comes through the
submodule and carries every option default, hiveClients is a raw attrset
built in the let block with four fields. renderClient read
tokenEndpointAuthMethod plainly, which is fine for a declared client and
an eval error for a derived one -- so all-local, where hiveIdentities is
on, stopped evaluating.
Read it with `or null`. The comment records that the list is not
uniformly typed, because the next field added to that submodule has the
same trap waiting.
enableRequiredServices asserts the services that exist once per swarm and
are optional -- its own description says so -- and the queue meets that
rule. It was left out because this file predates the swarm-nats container
by nine days and was never revisited, not because anyone decided against
it.
The all-local mode already derives nats.autoGenerateCallout, so it was
minting the queue's callout nkeys and never starting the queue. With this
the mode's loopback derivation for the controller points at something
that is actually running.
Review point: the co-located defaults are an auto-deploy special case and
belong with the other ones, not inside each option's own default. An
option should describe itself; the mode describes what a deployment shape
implies. local-defaults.nix already says exactly this in its header.
Options now default to empty (= unset, which the assertions refuse), and
enableAllLocalDefaults fills in loopback + the minted secret path.
The controller's queue coordinates were gated on `autheliaCfg.enable &&
natsCfg.enable` -- i.e. on this host running both. Split the hosts and
the whole env block vanished, which the daemon's own rule reads as "no
queue configured": it started cleanly, served /api/hives, and silently
never connected or published.
Make the coordinates options instead, defaulted to the co-located values
so nobody types a path they didn't need, and assert each one so a split
deployment fails at eval naming the option rather than running blind.
The queue is required for a controller -- /api/hives/status reads the KV
-- but co-location with it is not.
An SSO login whose derived localpart already exists silently gets a
random one instead: the candidate must be *available*, an existing
unlinked account is not, and `unique_id_fallbacks` defaults to true.
The operator's own login minted a stray account this way.
Fill the three identity_provider fields that decide which account a
login lands on: `trusted` (upstream's precondition -- self-hosted and
fully controlled -- is structural here, the issuer is always the swarm's
own authelia), `userid_claims` pinned to preferred_username so the
default ladder cannot fall through to an email local part, and
`unique_id_fallbacks = false` so a collision errors instead of
succeeding as the wrong user.
tuwunel authenticates at the token endpoint with the secret in the POST
body. Authelia enforces the method a client is REGISTERED with rather
than accepting whichever one arrives, and its default is
client_secret_basic — so the matrix login completed, consent was
granted, and the very last hop failed:
Client authentication failed ... The request was determined to be
using token_endpoint_auth_method client_secret_post, however the
OAuth 2.0 client registration does not allow this method.
The failure names neither the secret nor the redirect, and it lands
three layers from its cause, which is why it read as a credential
problem.
Adds a per-client tokenEndpointAuthMethod, null by default so every
existing client keeps authelia default (forgejo authenticates with
basic and is unaffected), and sets client_secret_post on the matrix
client only.
Review: high comment-to-code ratio. Measured — 197 of 337 added lines
were comments, and net of the block that only moved it was still +135
comment against +117 code.
Kept the constraints, the rejected alternatives and the costs; dropped
everything that restated what the code plainly does, and the
slash-stripping explanation that was duplicated in the generator.
A hive whose whole swarm is one box had to be handed two nkeys by hand
before its queue could authenticate anyone, which is the one deployment
shape where nobody else can supply them.
`autoGenerateCallout` mints both keypairs on the host on first boot,
keeps the seeds at 0600 host-side, and writes only the public halves
into a fragment the server reads at start. The all-local mode turns it
on; everywhere else the options stay operator-supplied and the
fail-closed eval assertions keep their full force.
The server config is not rewritten to do this. A wrapper includes
upstream's rendered `settings` verbatim plus the runtime fragment, and
the fragment wins — measured, along with the property that makes the
whole shape safe: the empty strings the options render in auto mode are
values `nats-server` refuses to start on, so any field the merge fails
to reach fails closed loudly rather than leaving a walk-in-able server.
The wrapper, the settings symlink and the fragment are siblings in one
runtime directory, and that is forced rather than tidy: NATS resolves an
include with filepath.Join against the config file's own directory,
which strips a leading slash, so an absolute include silently becomes a
relative one and the server never finds it. The includes are therefore
bare filenames. That also means nothing in the closure would otherwise
name the rendered settings, so the generator's symlink to it is what
keeps it from being garbage-collected under a running server.
`accounts` and `authorization` are defined once and rendered twice, into
`settings` and into the fragment template. Written out separately they
would diverge silently and backwards: the fragment is the later
definition, so a future edit to `settings` alone would be ignored on
exactly the hives that use auto mode.
`validateConfig` goes off in auto mode because `nats-server -t` rejects
the empty keys at build time; the parse check moves to server start,
where the fragment exists. Upstream's own option description names this
case.
Three options, all three derived from ONE predicate — this host runs both
the queue and the IdP — so a defaulted set is all or nothing. Deriving
them per-service looks equivalent and is not: `enableRequiredServices`
turns on matrix and authelia but not nats, so an ordinary all-local hive
would resolve two of three and trip the assertion below. Making the
partial state unrepresentable is what keeps that assertion honest.
Deliberately not the shape swarm-controller uses. That module emits its
queue coordinates only when authelia and NATS are local, which is right
for a service that *is* a swarm-host service — but a hive is the one thing
in a swarm that routinely is not on the swarm host, so the same rule would
make status publishing work on exactly the deployment that needs it least.
There is no `enable`: three coordinates that are all set is the enable. An
extra flag would allow configured-but-off, which is one more state to
explain and one more way to be silently quiet.
A half-set trio is an eval error rather than a silent no-op, because its
runtime failure mode is the expensive kind — the daemon comes up fine,
never connects, and the hive reads never_reported on a dashboard nobody is
watching yet. With the defaults all-or-nothing, the assertion only ever
judges what an operator typed by hand.
The secret arrives by LoadCredential, not a copy: hive-c0re is a host
unit, so systemd hands it the file directly and the secret never gains a
second on-disk copy. The client id is not chosen here either — it is
`hive-<hiveName>`, the identity swarm-authelia.nix already declares for
every entry in the roster.
A deployed config still set `swarm.hives.<hive>.certFingerprint`, deleted
along with the dashboard feature it served. The module system's answer was
`The option ... does not exist`, which tells an operator nothing about why
it went, whether it moved, or what replaces it.
Re-declared invisible and internal, with a top-level assertion naming the
hives that still carry it and explaining that the swarm root CA replaces
per-hive leaf pinning.
`lib.mkRemovedOptionModule` cannot do this job, and it is worth writing
down why: neither of its halves survives the move into a submodule. Its
`apply = throw` fires only when the value is read, and nothing reads this
any more — that being the point of removing it. Its `config.assertions`
half would land on a submodule that declares no `assertions` option. It is
a top-level tool. This is the same shape swarm-peers-removed.nix already
uses for the analogous `peers.<hive>.caCert`.
An error rather than a warning, because re-declaring the option is what
stops the unhelpful message — and on its own that would turn a config that
used to fail into one that quietly evaluates with the setting ignored,
which is worse than the error it replaced.
Registering the OIDC login source makes an outbound HTTPS call - the CLI
fetches <issuer>/.well-known/openid-configuration to validate the
provider before writing the row. That URL is a swarm service name served
under the swarm CA, which the default system store has never heard of.
SSL_CERT_FILE was set on forgejo.service and not on
forgejo-sso-source.service, so the web service trusted the chain and the
registration one-shot did not. Same binary, same host, different unit.
The result was a 100% reproducible failure that no restart could fix:
nothing about restarting a unit gives it an environment variable it
never had. Every recorded run since the feature landed has failed with
x509: certificate signed by unknown authority.
The trust belongs to every process that makes the call, not to the
service that happens to be the obvious consumer.
Follow-up to the plumbing removal: docs/swarm/README.md gets the
biggest rewrite (drops the whole "Fingerprint format" section, fixes
the runtime-effects list, the WireGuard config example + "what the
mesh does" bullet), docs/gateway.md and hive-gateway/options.nix drop
4 "needs no certFingerprint" mentions, swarm-peers-removed.nix's
migration-warning text no longer tells an upgrading operator to carry
a field over that no longer exists, swarm.nix/swarm-wireguard.nix/
swarm-controller.nix/swarm-controller's main.rs get comment fixes
where they described the now-removed HYPERHIVE_PEERS shape. Also
caught one more stale "peer hives" mention in docs/web-ui/README.md's
SW4RM tab description that the first pass on this issue missed.
Mara wanted the underlying plumbing gone too, not just the dashboard
display. Traced every consumer before cutting:
- certFingerprint (services.hyperhive.swarm.hives.<name>.certFingerprint):
removed the nix option entirely. Its only consumer was the dashboard
code removed in the previous commits.
- HYPERHIVE_PEERS: removed entirely — the env var itself, the whole
block that built it in hive-c0re/environment.nix, and its entry in
meta.rs's FORWARDED_VARS (which forwarded it into every agent
container). Turned out to have zero real consumers, not just one:
the docs claimed hive-agent::identity::peers() read it for qualified
agent labels, but no such function exists — identity.rs only
qualifies THIS agent's own label with HYPERHIVE_HIVE_DOMAIN, nothing
peer-list-related. Grepped the whole hive-agent crate to confirm
before removing.
services.hyperhive.swarm.peerHives (the nix option HYPERHIVE_PEERS was
built from) is untouched — swarm-wireguard.nix reads it directly for
the wg-hive mesh, a real and unrelated consumer.
Verified: cargo build/clippy/test -p hive-c0re -p swarm-controller all
clean (needed nix develop -c per the usual -lsqlite3 gap), all touched
nix files pass nix-instantiate --parse, and a throwaway nixosSystem
eval confirms the wireguard mesh still configures a peer's
wireguardAddress into wg-hive correctly with certFingerprint gone.
Follow-up to the peer-hives removal: docs/web-ui/dashboard.md drops
the whole "P33R H1V3S" section + its stale #peers hash and tab-strip
mentions; docs/swarm/README.md's runtime-effects list and
cross-references point at swarm-ui's roster page instead. Nix comments
in hive-c0re/environment.nix, swarm.nix and swarm-wireguard.nix that
described HYPERHIVE_PEERS/certFingerprint as dashboard-consumed are
updated to say what's actually still true (agent identity only).
Deliberately NOT removed: the `certFingerprint` nix option itself.
It has no remaining Rust consumer, but deleting a documented,
potentially already-set option is a bigger, breaking change than
removing dead display code — flagged in docs/swarm/README.md with a
note instead of silently dropped, so an operator or reviewer can make
that call explicitly rather than have it made for them here.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
Second slice of #3274. Each entry in
`services.hyperhive.swarm.hives` gets an authelia client
`hive-<name>`, so a hive can authenticate to swarm services as
itself.
The clients are derived here rather than declared by a consuming
service, because a hive's identity belongs to the directory and not to
whichever service happens to use it first. mara's constraint is that a
hive's credentials "derive from the same identity" — one hive holds
one client and mints a different token per service from it — so the
queue declaring this list would mean the next consumer collides on the
same client id, and only at the moment it landed.
`oidc.hiveIdentities` defaults to whether the message queue is
enabled, that being the first service to need a hive to prove who it
is. It is an option rather than a hard-coded condition so the swarm
telemetry collector can turn it on without the queue.
No audience and no `access_token_signed_response_alg` here: the queue
learns which hive presented a token from the introspection response
(`{"active":true,"client_id":"hive-alpha",...}`, measured), so it needs
neither. The collector does need both, and they land with it.
Also asserts client ids are unique — newly reachable now that part of
the list is derived, since a hive named `x` and a service client named
`hive-x` never met before.
Refs #3274.
New `services.hyperhive.swarm.controller.links` option (listOf {label,
icon, url}, same shape as the per-agent hyperhive.dashboardLinks) plus
a new GET /api/links route serving it, same pattern as the existing
hives/GET /api/hives.
Rather than one central hardcoded list, each service's own module
contributes its own entry when actually enabled on the controller's
host: swarm-authelia.nix, hive-matrix.nix (gated on gui.enable too,
since / on that vhost only serves fluffychat then) and
hive-forge/default.nix (gated on behindGateway) each push one entry,
the same list-merge idiom services.hyperhive.gateway.localNames
already uses. swarm-ui.nix contributes a static entry for its own
same-origin swagger docs. Adding a future service's link is a nix-only
change to that service's own module.
Verified: cargo build/clippy/test -p swarm-controller clean, a
throwaway nixosSystem eval confirms all 4 entries merge correctly into
SWARM_CONTROLLER_LINKS, nix build .#swarm-controller succeeds.
`renderClient` could only emit the authorization-code shape, so a
daemon client was expressed as an interactive one with an empty
redirect list. Authelia permits only the grants a client names, and an
omitted `grant_types` means authorization-code alone — so that shape
cannot obtain a token at all.
Measured against authelia 4.39.20, rendering exactly what this module
produced for `swarm-nats`:
client_secret_basic → unauthorized_client: The OAuth 2.0 Client is
not allowed to use authorization grant
'client_credentials'
introspection → {"active":false} (works)
Introspection is all the queue's responder needs today, which is why
nothing was visibly broken while the comment in `swarm-nats.nix`
described a grant that was never configured.
Adds `kind = "interactive" | "machine"` rather than inferring from an
empty `redirectUris`, because the two differ in what authelia permits
and not merely in what is populated. `openid` is dropped from a machine
client's scopes because authelia refuses that combination outright — a
daemon receives an access token and never an id-token.
An assertion rejects redirect URIs on a machine client: they are not
harmlessly unused, they mean the author believed a browser was
involved.
Refs #3274.
The comment said "It parses; it does not prove a sample arrives",
which understates the gap in the direction that matters: it reads as
if a green build proves the collector *starts* and only runtime
delivery is unverified.
Measured while probing ingest-auth options for #3283: `otelcol
validate` ACCEPTS a receiver naming an auth extension that is absent
from the build, and the collector then dies at startup with `Failed
to start component`. So the check does not prove this config starts
at all.
Comment-only; no evaluated config changes.
Refs #3283.
Retention so a reader can ask what a hive last said without anyone
keeping a second copy. The container is ephemeral = false, so the store
survives a restart with no bind mount, and losing it degrades to the
true answer (nothing known) rather than a stale one.
The per-account grant is the part that is easy to miss: the server
option gives the SERVER JetStream, and an account gets it only from its
own. Measured against a running 2.14.1 with this exact two-account
shape - global-only makes kv operations fail 10039 while the server
starts cleanly and logs Starting JetStream, so no rendering check can
see it. Deliberately not granted to the callout account: the responder
mints credentials and has no business holding stream state.
The headersCredential section still said hive-c0re forwards the file
into each agent container - that is exactly the behaviour this change
removes, so the doc contradicted the code rather than merely lagging
it. Same for the collector section, which presented the two paths the
change collapses.
Also drops the dangling collector.enable reference in the network
section.
mara: 'there should only be the one via otel-collector'. Collapses
collector.enable away - enabling otel runs the collector, and agents
always export to it.
The argument for it is the same one the collector exists for: as long
as the direct path stays selectable, the credential-in-settings.json
hole stays selectable. An option that can reintroduce a hole is a hole
with extra steps. HYPERHIVE_OTEL_HEADERS_CREDENTIAL is now never
forwarded to containers at all rather than conditionally.
Two consequences, both deliberate:
- the enable option's own promise changes. It said each harness exports
directly so telemetry survives anything host-side being down; that is
now false for every hive rather than for opted-in ones. Stated in the
option text rather than left for a reader to discover.
- the collector-requires-headersCredential assertion is gone. It was
correct only under the shape it was written for: when the collector
was opt-in FOR the credential, opting in without one was pure
indirection. With one path it isn't - an upstream needing no auth
header is legitimate, and the assertion would have rejected it. The
upstream header block is now omitted entirely in that case, rather
than rendering an ${env:...} reference nothing sets.
The module hardcoded otlphttp, so a hive with protocol = "grpc" would
have had its agents' protocol respected end to end before the collector
and silently rewritten to HTTP after it. Splitting the path in two makes
the upstream half the one that has to keep honouring the option; the
agent half is pinned to the receiver's protocol and derived, not
configured.
The agent-facing endpoint becomes derived rather than `otel.endpoint`
re-defined: with the collector on, agents and hive-c0re's own exporter
send to the bridge address; with it off the expression is the old value
character for character. `otel.endpoint` keeps meaning "where telemetry
ultimately goes" on every hive that already sets it.
Suppressing HYPERHIVE_OTEL_HEADERS_CREDENTIAL is the half that actually
closes the hole this issue is about. The credential path host -> container
is sound (/run/credentials is dr-x------ root), but the hive-otel-header
oneshot then writes the value into the agent's own settings.json, mode
0600 and owned by the agent — so today every agent can read the hive's
upstream telemetry credential. With a collector there is nothing to
forward.
The protocol is derived for the same reason the endpoint is: the first
hop is now the collector's OTLP/HTTP receiver, while otel.protocol
describes the upstream link.
Renders services.opentelemetry-collector on the host: OTLP/HTTP receiver
on the bridge address, otlphttp exporter to otel.endpoint, and the
upstream credential delivered as EnvironmentFile so the collector
interpolates it at runtime and nix never sees the value.
Three things worth knowing, each measured rather than assumed:
- network.exposeHostPorts already exists and is wired (it opens the port
on the bridge interface only), so bridge reachability costs nothing.
- validateConfigFile defaults to isStorePath configFile, which is null on
the settings path - so upstream's default is OFF for exactly the way
this module configures it. Set true: it runs otelcol validate at build
time. It parses, it does not prove delivery.
- headersCredential's file is already NAME=value, i.e. EnvironmentFile
format, verified against a real settings.json rather than the doc.
An assertion refuses collector.enable with no headersCredential: the
collector exists to be the only holder of that token, and without one it
is indirection that reads as security.
