The option's description claimed declaring an entry from the service's
own module put 'the scraper and the target on the same host by
construction rather than by luck'. It does not. It constrains where the
target is; nothing in it places the collector, and the two enable flags
are co-located by a shared lib.mkDefault rather than by construction.
Split across hosts, a target is silently never scraped — the service's
host declares an entry no local collector reads, the collector's host
never enabled the service. No error surfaces, and no assertion can
catch it: separate hosts are separate evaluations with no shared
context, so the doc telling the truth is the only mechanism there is.
The same paragraph already warned co-location was not a guarantee, four
lines below the sentence claiming it was; a reader arriving for
permission stopped at the permission. This one did.
swarm-nats carries the concrete caveat for its own contribution.
NATS has no Prometheus format of its own. It serves a JSON monitoring
endpoint, and prometheus-nats-exporter translates that — so this is two
changes in order, not one: without the monitoring endpoint the exporter
starts cleanly and scrapes nothing, which is the inert-config shape the
scrape work exists to avoid.
Both listeners are loopback and the exporter is the monitoring
endpoint's only intended reader: it is unauthenticated and /connz names
every connected client, so the address it binds is the whole access
control.
The scrape target is declared here rather than in the collector's
module, gated on a collector existing to read it — an entry exists only
where the service that named it runs.
The responder introspects authelia over https by name. It had no CA trust
at all, so the handshake failed UnknownIssuer, introspection failed, and it
denied every client -- surfacing at the controller as a 60s
authorization-violation loop, two layers from the cause.
Adds a shared trustBundle helper to lib/hive-ca-trust.nix rather than a
fifth hand-rolled concat. Four containers were each assembling this
themselves, which is how they came to share one defect: wantedBy + before
express ordering but not success, so a failed assembly let the consumer
start against a missing file and trust nothing at all.
The helper fixes both halves of that. requires goes on the consumer, so a
failed bundle stops it and the dependency is visible in systemctl status
where someone debugging a TLS failure looks. And the script assembles to a
temp path, checks the result actually contains a certificate, and only then
moves it into place -- cat of an empty bind exits 0, so set -e does not
catch it and a partial bundle must never appear under the final name.
Returns a module rather than bare services: a caller that already writes
systemd.services.<consumer> cannot also write systemd.services in the same
attrset.
The auth-callout responder decides what an admitted client may publish
from two strings: the prefix marking a hive client, and the client id
allowed to read every hive's key. Both were literals in three places --
swarm-authelia.nix mints "hive-${name}", swarm-controller.nix defines
"swarm-controller", and the responder carried its own copies as clap
defaults because swarm-nats.nix passed neither.
Each producer now publishes its value as a readOnly option and the
responder's ExecStart reads them, so the agreement is one evaluation
rather than three strings that happen to be equal. Same pattern the
module already uses for `--account`, and the same argument
swarm-authelia.nix gives for publishing `machine` and `unit`.
Worth the change because the failure is silent and misattributed:
rename either principal and the responder starts denying the one that
stopped matching, a denial reaches a NATS client as a timeout rather
than an error, and a hive that is refused looks exactly like a hive
that has not reported yet.
A swarm service container shares the host netns and force-disables
resolvconf, so it inherits the /etc/resolv.conf nixos-containers copies in
at start (cp --remove-destination, host-side preStart, once per start) and
nothing ever refreshes it. That makes the container's resolver a snapshot of
the host's file at its boot instant.
When that snapshot is wrong the container can never recover, and the symptom
appears arbitrarily far from the cause: swarm-nats-auth cannot resolve
authelia's name, so it denies every auth-callout request and the queue
refuses every client with an authorization violation.
Give each of the four swarm containers a oneshot that writes the resolver
file itself, from the bridge IP, ordered before that container's first DNS
consumer. The shape is the one every agent container already uses.
networking.nameservers cannot do this: resolvconf is its only consumer and
these containers disable it, so setting it renders no file while still
evaluating cleanly. A static environment.etc entry cannot either -- it would
have to survive etc activation landing on the regular file the host already
copied there, which no eval can show.
The responder's third credential is not minted by the generator beside
it — authelia produces it on first boot, inside its own container, and
nothing ordered this unit against that. Losing the race cost the entire
queue: install exits 1, the responder never starts, and auth_callout
with no responder refuses every client. Fail-closed by design, so the
symptom lands on every queue client and nowhere near the cause. Seen on
a real boot at 20:31:31.
Now it waits for the file, bounded at two minutes. Where authelia runs
on another host the secret is never going to appear, and blocking the
queue container indefinitely would replace a clear failure with a hang;
after the timeout this fails exactly as it did before, having first
given the co-located case the seconds it needs.
TimeoutStartSec is set rather than left to the default because the
default is ninety seconds — a two-minute wait would be killed at ninety
and the operator would get a generic unit timeout instead of the
message naming the file.
Ordering after the authelia container is necessary and not sufficient:
the container being up says nothing about whether its in-container
secrets unit has finished. It only stops this spinning for the full
timeout on every boot.
Review: high comment-to-code ratio. Measured — 197 of 337 added lines
were comments, and net of the block that only moved it was still +135
comment against +117 code.
Kept the constraints, the rejected alternatives and the costs; dropped
everything that restated what the code plainly does, and the
slash-stripping explanation that was duplicated in the generator.
A hive whose whole swarm is one box had to be handed two nkeys by hand
before its queue could authenticate anyone, which is the one deployment
shape where nobody else can supply them.
`autoGenerateCallout` mints both keypairs on the host on first boot,
keeps the seeds at 0600 host-side, and writes only the public halves
into a fragment the server reads at start. The all-local mode turns it
on; everywhere else the options stay operator-supplied and the
fail-closed eval assertions keep their full force.
The server config is not rewritten to do this. A wrapper includes
upstream's rendered `settings` verbatim plus the runtime fragment, and
the fragment wins — measured, along with the property that makes the
whole shape safe: the empty strings the options render in auto mode are
values `nats-server` refuses to start on, so any field the merge fails
to reach fails closed loudly rather than leaving a walk-in-able server.
The wrapper, the settings symlink and the fragment are siblings in one
runtime directory, and that is forced rather than tidy: NATS resolves an
include with filepath.Join against the config file's own directory,
which strips a leading slash, so an absolute include silently becomes a
relative one and the server never finds it. The includes are therefore
bare filenames. That also means nothing in the closure would otherwise
name the rendered settings, so the generator's symlink to it is what
keeps it from being garbage-collected under a running server.
`accounts` and `authorization` are defined once and rendered twice, into
`settings` and into the fragment template. Written out separately they
would diverge silently and backwards: the fragment is the later
definition, so a future edit to `settings` alone would be ignored on
exactly the hives that use auto mode.
`validateConfig` goes off in auto mode because `nats-server -t` rejects
the empty keys at build time; the parse check moves to server start,
where the fragment exists. Upstream's own option description names this
case.
`renderClient` could only emit the authorization-code shape, so a
daemon client was expressed as an interactive one with an empty
redirect list. Authelia permits only the grants a client names, and an
omitted `grant_types` means authorization-code alone — so that shape
cannot obtain a token at all.
Measured against authelia 4.39.20, rendering exactly what this module
produced for `swarm-nats`:
client_secret_basic → unauthorized_client: The OAuth 2.0 Client is
not allowed to use authorization grant
'client_credentials'
introspection → {"active":false} (works)
Introspection is all the queue's responder needs today, which is why
nothing was visibly broken while the comment in `swarm-nats.nix`
described a grant that was never configured.
Adds `kind = "interactive" | "machine"` rather than inferring from an
empty `redirectUris`, because the two differ in what authelia permits
and not merely in what is populated. `openid` is dropped from a machine
client's scopes because authelia refuses that combination outright — a
daemon receives an access token and never an id-token.
An assertion rejects redirect URIs on a machine client: they are not
harmlessly unused, they mean the author believed a browser was
involved.
Refs #3274.
Retention so a reader can ask what a hive last said without anyone
keeping a second copy. The container is ephemeral = false, so the store
survives a restart with no bind mount, and losing it degrades to the
true answer (nothing known) rather than a stale one.
The per-account grant is the part that is easy to miss: the server
option gives the SERVER JetStream, and an account gets it only from its
own. Measured against a running 2.14.1 with this exact two-account
shape - global-only makes kv operations fail 10039 while the server
starts cleanly and logs Starting JetStream, so no rendering check can
see it. Deliberately not granted to the callout account: the responder
mints credentials and has no business holding stream state.
The responder answered granted=true and the server still refused every
client:
Error non operator mode account "AUTH": attempted to use issuer_account
nats_jwt::Token::new_user always sets issuer_account, which is an
operator-mode field naming the account when a signing key rather than the
account identity key signed the token. In server-config mode - what this
module renders - its mere presence is fatal, and the account is named by
the claims' aud instead.
nats-jwt can express neither aud nor the omission, so the user JWT is now
hand-built by the same signer as the response wrapper, and nats-jwt moves
to dev-dependencies as the encoder's test oracle.
Every unit test passed throughout: they assert fields that must be
present, and the defect was a field that must be absent.
Measured against a running server built from this module's own rendered
config, not a hand-written one -- which is the fix to the method as much
as to the file. Two shapes died in a row: a bare name authenticates
anyone who sends it, and a name plus an nkey refuses to start at all
(nkey users do not take usernames or passwords).
A malformed key is fail-closed too, so the only way to a live server
here is a real key whose seed nobody but the responder holds.
argus caught it and reproduced it: a users entry carrying only a name
has no credential, so CONNECT {"user":"auth"} is accepted with no
password. The name is a literal in this module, so on the shared netns
that identity was walk-in-able from every agent container -- the same
hole the module exists to close, moved rather than closed.
An nkey rather than a password: only the public half appears in config,
the seed reaches the responder alone, so a hive with no responder has
nobody who can authenticate as it. Asserted at eval, because the
rendered config is valid and the server starts either way.
A swarm has one message queue; this is the container that runs it, in
the same shape as every other swarm service. Off by default, and off
means absent.
The auth_callout block is the interim state and the final one at once.
Measured on the pinned nats-server rather than reasoned about: an empty
authorization block, and one with an empty users list, both accept an
anonymous client and answer PONG. auth_callout sets auth_required and
refuses everyone until a responder approves them, so a config whose
responder does not exist yet denies by default -- and the responder
lands beside it without any of this being swapped out.
That matters more here than it would elsewhere because the container
shares the host netns, so an unauthenticated interim state would be
reachable from every agent container on the hive.
No package option, deliberately: nixpkgs' nats module resolves
pkgs.nats-server itself and exposes none, so one here would name
something it cannot control.
No gateway vhost either: NATS speaks its own TCP protocol, nginx cannot
front it, and cross-hive reach is the wireguard mesh.
