Last of the three modules that hand-rolled the same concat with wantedBy +
before and no requires, so a failed assembly left the consumer running against
a missing file and trusting nothing -- every outbound TLS call fails while the
unit looks healthy.
The forge is the one with two consumers: forgejo-sso-source fetches the
issuer's discovery document over the swarm CA and once shipped without the
trust its sibling had. It only exists when SSO is on, so the consumer list is
conditional -- naming an absent unit would define a serviceless one and order
nothing.
Removes the now-dead useSelfSigned and caContainerPath bindings (nix does not
warn) and retargets three comments the deletion orphaned, including the helper
header that still named this module as the per-call-site concat.
Both hand-rolled the same concat with wantedBy + before and no requires,
so a failed assembly let the consumer start against a missing file and
trust nothing at all -- which fails every outbound TLS call while the unit
looks healthy. The helper puts requires on the consumer and verifies the
assembled bundle before moving it into place.
Removes each module's now-dead useSelfSigned binding: its only use was the
mkIf on the deleted block, and nix does not warn about an unused let
binding. Also corrects two comments the deletion orphaned -- one described
the removed path binding, the other pointed at a bundle service that no
longer exists.
A loopback literal encodes 'authelia is in my netns' at the call site,
and authelia's OIDC endpoints are https-only in effect: reached directly
they answer 400, because the forwarded headers nginx injects for every
other consumer are what let it determine its own issuer. Going by name
deletes the need for those headers rather than reproducing them, and
converges on the URL swarm-nats-auth already uses.
Depends on the CA trust added for the same container in #3407 -- without
it this swaps a 400 for an UnknownIssuer.
The null-url assertion joins the module's existing list: interpolating a
null would surface as a nix coercion error several files from its cause.
Same class-B defect as the queue responder: swarm-authelia-bridge
introspects authelia by name over https and the container trusted no swarm
CA, so the call could only ever fail UnknownIssuer.
This is also the prerequisite for the by-domain introspection change on
#3391 -- flipping that URL without the trust half would swap one failure
for another.
The responder introspects authelia over https by name. It had no CA trust
at all, so the handshake failed UnknownIssuer, introspection failed, and it
denied every client -- surfacing at the controller as a 60s
authorization-violation loop, two layers from the cause.
Adds a shared trustBundle helper to lib/hive-ca-trust.nix rather than a
fifth hand-rolled concat. Four containers were each assembling this
themselves, which is how they came to share one defect: wantedBy + before
express ordering but not success, so a failed assembly let the consumer
start against a missing file and trust nothing at all.
The helper fixes both halves of that. requires goes on the consumer, so a
failed bundle stops it and the dependency is visible in systemctl status
where someone debugging a TLS failure looks. And the script assembles to a
temp path, checks the result actually contains a certificate, and only then
moves it into place -- cat of an empty bind exits 0, so set -e does not
catch it and a partial bundle must never appear under the final name.
Returns a module rather than bare services: a caller that already writes
systemd.services.<consumer> cannot also write systemd.services in the same
attrset.
Keeps why the setting exists (the bridge writes this file and cannot
restart authelia) and drops the incident narrative and the answered
objections -- that history belongs in the PR discussion, not in three
comment blocks a future reader has to scroll past.
authelia now watches the users file, so the restart is redundant -- and it
was the wrong shape twice over. It could fail: a login was refused for a
user whose record was already correct on disk, with nothing in either log
implicating the reload. And it only ever worked for this writer --
swarm-authelia-bridge writes the same file and cannot restart anything,
since running unprivileged inside the container is the whole reason it may
write it at all. A reload that depends on which process did the writing is
not a reload.
--machine/--unit and their two env vars existed solely to name a
systemctl -M target, so they go with it. That drops two required settings
from the operator surface.
The three objections previously recorded against watch are all answered
now, and are kept next to the decision rather than deleted: the key is
verified against the pinned build (validate-config accepts it and rejects
a misspelling), the watch is on the directory so a rename is observed, and
partial reads are structurally impossible because every writer of this
file goes through write_atomic.
Authelia reads authentication_backend.file.path once at startup. Without
watch, an identity the bridge creates is real on disk and invisible to the
running authelia until something unrelated bounces the unit.
swarm-authelia-bridge cannot restart authelia -- running unprivileged
inside the container is the whole reason it may write that file, so
restarting the unit is exactly the privilege it was designed not to hold.
swarmctl does restart it, but that is a systemctl -M shellout that can
fail, and when it did the symptom was a login refused for a user whose
record was already correct on disk. This makes correctness stop depending
on that restart succeeding.
The auth-callout responder decides what an admitted client may publish
from two strings: the prefix marking a hive client, and the client id
allowed to read every hive's key. Both were literals in three places --
swarm-authelia.nix mints "hive-${name}", swarm-controller.nix defines
"swarm-controller", and the responder carried its own copies as clap
defaults because swarm-nats.nix passed neither.
Each producer now publishes its value as a readOnly option and the
responder's ExecStart reads them, so the agreement is one evaluation
rather than three strings that happen to be equal. Same pattern the
module already uses for `--account`, and the same argument
swarm-authelia.nix gives for publishing `machine` and `unit`.
Worth the change because the failure is silent and misattributed:
rename either principal and the responder starts denying the one that
stopped matching, a denial reaches a NATS client as a timeout rather
than an error, and a hive that is refused looks exactly like a hive
that has not reported yet.
The module doc had grown a 37-line preamble carrying three separate
arguments, which the comment-block lint refuses. Splitting it is the
better fix than raising the limit: the reader who is about to widen a
subject list meets the reason not to at the list, not seven screens up.
Also corrects one claim I had no measurement for. The CREATE note said
creating an existing stream with a different config "is an error rather
than a rewrite" — asserted, not observed. What is observed is narrower
and enough: a hive holding this grant leaves the stream config untouched
and never publishes $JS.API.STREAM.UPDATE at all.
A grant carrying every bucket-specific subject and neither of these
cannot create the bucket at all: the client times out on `$JS.API.INFO`
long before it reaches a subject that was granted, and a NATS denial
reaches the client as a hang rather than an error.
Both were named by the server's own log, not reasoned about.
`$JS.API.INFO` is the account-level JetStream info every client requests
on connect; `$JS.API.STREAM.NAMES` is how a client finds the stream
backing a bucket. The latter lets a client enumerate stream names in the
account, which in an account holding one bucket discloses a name both
ends already share.
Every earlier measurement missed them, because each either granted
`$JS.API.>` wholesale or ran against a bucket the setup had already
created while unscoped. A minimum established against an existing bucket
is not the minimum for making one, so the leave-one-out that trimmed the
reader's set could not have found this — every candidate it tried was
tried in a world where the bucket existed.
Found by running the shipping gate against the real binary. No unit test
could have: the failure is a timeout inside a real server's permission
check.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
A swarm service container shares the host netns and force-disables
resolvconf, so it inherits the /etc/resolv.conf nixos-containers copies in
at start (cp --remove-destination, host-side preStart, once per start) and
nothing ever refreshes it. That makes the container's resolver a snapshot of
the host's file at its boot instant.
When that snapshot is wrong the container can never recover, and the symptom
appears arbitrarily far from the cause: swarm-nats-auth cannot resolve
authelia's name, so it denies every auth-callout request and the queue
refuses every client with an authorization violation.
Give each of the four swarm containers a oneshot that writes the resolver
file itself, from the bridge IP, ordered before that container's first DNS
consumer. The shape is the one every agent container already uses.
networking.nameservers cannot do this: resolvconf is its only consumer and
these containers disable it, so setting it renders no file while still
evaluating cleanly. A static environment.etc entry cannot either -- it would
have to survive etc activation landing on the regular file the host already
copied there, which no eval can show.
The error names the protocol and never the sandbox, so it reads like a
network fault. Records the two easy-to-miss families (AF_INET6 alongside
AF_INET, and AF_NETLINK for getaddrinfo) and the reason the directive rots:
it is a claim about what the program does, and nothing re-checks it when a
client is added.
The unit restricted RestrictAddressFamilies to AF_UNIX, which was correct
while the daemon only served its unix socket. It has since grown three
outbound clients -- authelia token minting and forge calls over HTTPS, and
the queue over NATS -- and every socket(AF_INET, ...) was refused by seccomp.
systemd surfaces that refusal as EAFNOSUPPORT, "Address family not supported
by protocol", so the failure names the protocol and never the sandbox. The
visible symptom was swarm agent creation failing while minting a bearer
token, with a connect error that reads like a network fault.
Permit AF_INET/AF_INET6 for those clients and AF_NETLINK, which glibc's
getaddrinfo needs to enumerate local addresses before returning one. The
rest of the unit's hardening is unchanged.
The new regression test constructed `reqwest::Client::new()`, which
panics in the nix build sandbox: with no system CA store,
`ClientBuilder::build()` reaches `rustls_platform_verifier::Verifier::new()`
and fails, and `new()` is `build().expect(..)`. The test passed locally
because a devshell has `/etc/ssl/certs`, and failed in CI.
Disabling certificate verification takes the branch that installs a
no-op verifier and never consults the platform store, so the client
builds anywhere. That is sound in this test and nowhere else: nothing is
sent, the request is built and its bytes are inspected. The helper says
so at the point someone would otherwise object to it.
Found by reading reqwest's `ClientBuilder::build()` rather than trying
builder flags: the first repro attempt — pointing `SSL_CERT_FILE` and
`SSL_CERT_DIR` at nothing — did not reproduce, so any fix verified
against it would have been verified against nothing.
Every access-token request the swarm queue client has ever made was
refused. It sent the client id and secret as form fields
(`client_secret_post`); authelia's client registration allows only
`client_secret_basic`, so the identity provider rejected the request
before looking at the credentials at all.
What made it survive so long is the shape of the failure. The refusals
tripped authelia's rate limiter, whose penalty grows faster than this
client's retry interval — 56s, then 296s, then 535s, against a retry
every 60s — so the limiter never drained and a 429 came back before the
credentials were evaluated. The line naming the real cause appeared
roughly once an hour, inside a continuous storm of a different error,
and the storm read as the problem.
Both other callers in this workspace that present client credentials
already use Basic. RFC 6749 says clients SHOULD, authelia's registration
default says so, and a secret in a header is one fewer place for a proxy
to log it.
The test asserts the shape of the request rather than a server's reply:
Authorization is Basic, the body carries the grant type, and neither the
secret nor the client id appears in the body. It fails on the previous
code with no identity provider, no deployment and no network — which is
what this needed and did not have.
Relative hrefs (static/main.js) only resolve correctly for a route
one path segment deep - the browser resolves them against the
current URL's directory, so a route two-plus segments deep would
404 the JS/CSS silently once this is wired to a real server. Made
absolute (/static/main.js), matching what build.mjs's own comment
already documents these paths as being served at.
mara: the route should reflect creating an agent, and stay separate
from a future agent list page. Renamed AgentsPage -> CreateAgentPage
(file, component, css classes) and moved the route from /agents to
/create-agent - flat, not /agents/new, since index.html's relative
asset links only resolve correctly one path segment deep (filed
separately as a real bug, not fixed here). Leaves the bare /agents
path free for a future roster page.
Adds /agents: a name field that POSTs to swarm-controller's
POST /api/agents (from the CreateIdentity work), shows the queued
job node id, and links to /jobs to watch it settle. Scope matches
the issue exactly - no forge/deploy options, those aren't wired
server-side yet.
The auth-callout responder learned whether a token was valid, never whose
it was, so every admitted client got the same unscoped grant. Scoping a
grant to one hive's subjects needs an identity to scope it to.
`is_active` becomes `identify_caller` and returns the identity rather
than a boolean: `Ok(Some(client_id))` admits as that client, `Ok(None)`
denies. `active: true` with no `client_id` is a denial, and returning an
identity is what makes that unrepresentable instead of remembered --
there is no admitted-but-unscoped value to construct, so there is no
branch a later edit can forget to handle.
The module deliberately ignores `sub`/`scope`/`exp` on the grounds that
modelling a field implies checking it. That still holds: `active` remains
the whole admission decision, made in one place. `client_id` answers a
different question -- as whom -- and is used downstream of an admission
that has already happened, not as a second gate. The module docs say so,
next to the paragraph that would otherwise argue for deleting the field.
An inactive token has no identity even when the body names one, and an
empty `client_id` counts as absent: it would become a blank component in
a subject the grant is scoped to, which is not a narrower permission but
a different one. The two credential parameters are renamed `own_*` --
this responder's introspection credential is not the caller's identity,
and the function now takes one and returns the other.
The grant itself is still unscoped; narrowing it is the next slice.
Per the operator's call on #3372: rather than moving Grafana off the
forge's 3000 to another number, take it off TCP entirely.
The collision was possible because every swarm service container shares
the host's network namespace, which makes a port a swarm-wide resource
two modules can each claim believing it free — Grafana took upstream's
3000, so does the forge, and `grafana.<swarm-domain>` served the forge
with no bind error and nothing in any log. A socket has a path, and a
path collision is a build-time conflict rather than a runtime coin toss.
Three parts, none of which works alone:
- `protocol = "socket"` with `socket_gid` = nginx's static gid 60.
- Grafana joins that gid inside the container. A non-root process may
only chgrp to a group it belongs to, and a container has its own user
database — without the membership Grafana starts, the chown fails, and
the socket is simply unreachable.
- The socket dir is created host-side by tmpfiles and bind-mounted in. A
container's /run is an nspawn tmpfs, so it is not visible from the host
at /var/lib/nixos-containers/<name>/run; the same shape the per-agent
web.sock already uses.
The `port` option is gone rather than deprecated — nothing can set it to
a colliding value if it does not exist.
`swarm-controller.service` carried `SetCredential=queue-client.secret:` — an
empty value, which systemd's parser refuses:
/etc/systemd/system/swarm-controller.service:38:
Invalid syntax, ignoring: queue-client.secret:
So the line was dropped on every daemon-reload, and `LoadCredential=` was
fatal again — precisely the failure that default was added to prevent. On a
hive where authelia has not yet minted the secret, the controller refuses to
start rather than coming up with the queue unconfigured.
It looked correct for days because the credential file happened to exist, so
the fail-soft was never exercised.
Measured with `systemd-analyze verify`: an empty value is rejected, any
non-empty one is accepted. The placeholder is a real word rather than filler —
it reaches the token request as the client secret, so authelia refuses it and
the journal names something an operator can act on.
Gate: state/eval-setcredential.sh, with the parent commit as its mutation.
nixpkgs dropped the default for `services.grafana.settings.security.secret_key`
and asserts on null, so the module I merged an hour ago fails the build on any
host that enables it. That is a broken deploy, not a warning.
Generated in-container like authelia's own keys and for the same reason:
nothing outside the container ever reads it. Generated ONCE and kept — the
key signs Grafana's stored datasource secrets, and rotating it does not
re-encrypt what it already encrypted, so a fresh key per boot would leave
Grafana unable to read its own database. Delivered as `$__file{}`, so it
never enters the store.
The gate missed this because a container is a separate NixOS evaluation with
its own assertions: forcing the host's `config.assertions` never touched the
container's. `state/eval-3265.sh` now forces
`containers.swarm-grafana.config.assertions` and reports any that fail.
Review catch: this PR relaxed the `otel.endpoint` assertion and staled the
canonical OTEL reference in the same stroke — `docs/observability.md` is
what CLAUDE.md points readers at for "what OTEL options are available",
and it still said required-full-stop while the new swarm/services.md
section said a local store satisfies it.
Also corrects the option's own description in otel.nix, which said the
same thing and renders into the generated options doc. Grepping the
reviewer's phrasing did not find that one; grepping the claim did.
Records the second destination where the "endpoint is where telemetry
ultimately goes" paragraph makes its claim, rather than only in the new
section a reader may not reach.
Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
A local time-series database rather than only an external sink, so the
swarm dashboard stays readable when the outside world is not: a view of
the system must not depend on the system it views being healthy.
listenAddress is pinned to loopback. Upstream defaults it to every
interface, and the OTLP ingest path this exists to receive on is
unauthenticated — the gateway is the only intended client and it is on
this host, so a wider bind would publish a write endpoint to whatever
the host is reachable on.
retentionPeriod defaults high rather than being required, because the
two failure directions are not symmetric: too long fills a disk, which
is visible and recoverable by lowering it, while too short destroys
history silently and permanently. The operator lowers it once they have
measured how fast this swarm accumulates.
OTLP needs no flag. Measured against the pinned 1.146.0 rather than
inferred from the module's option list, which has no OTLP switch and so
reads as though the feature were missing: the running server answers
POST /opentelemetry/api/v1/push with 200, where a nonexistent path
answers 400.
mara, follow-up on PR#3365: "follow up with the same for jobq". Drops
mountJobqGraph the same way mountJobqRollup was dropped - builds.js
now calls render(h(JobqGraph, {...}), root) directly, split into
mountRebuildQueue() (the one-time replaceChildren() + first render,
called once at page init) and renderRebuildQueue() (the render-only
path the rebuild_queue_changed handler and the cancel flow reuse,
bumping a module-level jobqGraphToken instead of holding a mount
handle). The split matters here specifically: repeating
replaceChildren() on every refresh would wipe Preact's own tracked
children out from under its diffing instead of letting it update them
minimally - JobqRollup's simpler version didn't need this since it
only ever renders into its own dedicated section once per app
lifetime's worth of state, but the rebuild queue refreshes on every
`rebuild_queue_changed` tick.
Re-verified: npm run build (whole workspace) + swarm-ui typecheck
clean (JobqGraph is also used via JSX on swarm-ui's /jobs page,
untouched by this), comment-block + issue-ref lints clean, headless-
chromium screenshot of builds.html's R3BU1LD QU3U3 tab against a
mocked /api/jobq/graph payload - tree, filter checkboxes, and cancel
buttons all render identically to before.
mara, on review: expected the plain Preact pattern (render(h(Widget,
props), container), call again to update) rather than a custom
mountX() returning {refresh(), update()}. Preact's own render is
already the re-render/diff entry point, so the wrapper was indirection
this component didn't need - swarm.js (plain .js, no JSX pragma
required for h()/render() either) now calls render(h(JobqRollup,
{...refreshToken}), root) directly, bumping a module-level token to
force a refetch instead of holding a mount handle.
JobqGraph/mountJobqGraph (a separate, already-merged component) is
untouched - out of scope for this PR, flagged as a possible follow-up
if she wants the same simplification there.
Re-verified: npm run build (whole workspace) + swarm-ui typecheck
clean, comment-block + issue-ref lints clean, re-screenshotted the
dashboard SW4RM tab against the same mocked payload - identical
render, spinner now visibly mid-rotation in the frame (confirms the
animation is live, not just present in markup).
The extraction dropped the old banner's `spinner` class on the glyph
span (`el('span', { class: 'glyph spinner' }, ...)`) - JobqRollup.tsx
rendered a static `.jqr-glyph` with no animation rule anywhere.
`.spinner` (shared/base.css) is already imported by both consumers, so
just reuse it on the glyph rather than duplicating the keyframes.
argus caught this on review - a single-frame screenshot can't tell a
frozen spinner from a missing one, which is exactly why it slipped
past the PR's visual verification.
New @hive/shared/jobq-rollup.js (JobqRollup.tsx + jobq-rollup.css),
mirroring JobqGraph's shape exactly: JSX use plus an imperative
mountJobqRollup(container, props) for a plain-.js call site. Fetches
Vec<hive_jobq_wire::StateCount> off `endpoint`, sums Running+Finishing
roots as "running" and Pending roots as "queued", renders nothing when
both are zero. Optional `queueHref` adds a "view queue -> " link.
Swapped dashboard's hand-rolled queue-summary banner (swarm.js) over to
this component instead of keeping two parallel implementations - same
"one shared component" pattern JobqGraph already set for the rebuild
queue tree view. Mounted once into a new #jobq-rollup-section, kept as
a sibling of (not inside) #containers-section since that section gets
replaceChildren()-wiped on every container-state render, which would
tear down and remount a Preact tree on every tick. Refreshed via the
mount handle's .refresh() on rebuild_queue_changed, same as builds.js's
JobqGraph handle.
Also mounted in swarm-ui's /jobs page, above JobqGraph, with no
queueHref (a link back to the page you're already on is noise) - the
literal ask on hyperhive#3364.
Verified: npm run build (whole workspace) and swarm-ui typecheck both
clean, comment-block + issue-ref lints run manually, headless-chromium
screenshots of both the dashboard SW4RM tab and swarm-ui's /jobs page
against mocked /api/jobq/rollup payloads - banner renders identically
in both, with and without the queue link as expected.
Three host units poll up to 120s for a secret authelia mints on its first
boot, and all three are `Type=oneshot` with no `TimeoutStartSec`. systemd's
`DefaultTimeoutStartSec` is 90s, so it kills them at 90 — before the script
reaches its own `exit 1` and names the file that never appeared.
The wait itself is fine; what's lost is the diagnosis. On a fresh hive the
operator gets a bare start-timeout instead of "authelia has not minted
<path>", several layers from the container that was actually slow.
Found while writing the same unit for Grafana, where the timeout is set —
so this is the existing three catching up with it, not a new pattern.
The auth callback minted on every invocation, and async-nats runs it per
connection ATTEMPT. Its default reconnect delay backs off exponentially
and then clamps at four seconds, permanently. So a queue that could not
connect asked authelia for a token every four seconds indefinitely, and
authelia answered 429 Too Many Requests.
That rate limit then keeps itself alive: it outlives whatever first
broke the connection, and its log volume buries the original cause.
Observed in production tonight, once a redeploy fixed the TLS trust
problem that had been hiding it.
The callback now caches the token and re-mints only within two minutes
of expiry, which preserves the property that put the mint there — a
reconnect must never present a token that expired since it was minted —
while decoupling mint rate from retry rate. The reconnect cap moves to
one minute, exponential from half a second, so a brief outage still
recovers promptly and a persistent one stops hammering the identity
provider.
The module doc and the retry comment both argued for the old shape, so
both are corrected rather than left explaining why the bug was right.
