mara's ruling: the controller should not forward received events, it
should create specific messages from them ("knowledge repo changed",
"deploy agent foo to rev abc123") and send those to whoever needs them.
The module doc said the opposite as settled design.
The argument it made was wrong in a specific way worth keeping: it
counted this daemon as a *second* place deciding what a delivery means,
while assuming the hives stay interpreters. They do not -- the parsing
moves rather than duplicating, so the end state has one interpreter, not
two.
Doc-only. The behaviour is unchanged and still receipt-only; what changes
is that the next reader is not told the wrong direction.
argus's optional note. Nothing in the doc was inaccurate, but an operator
who sees the activation-time warning has no thread back to the page that
explains why pinning matters -- and this section is the explanation.
States the one property they need from it: it never fails the rebuild, so
acting on it is theirs to do before the ids are minted.
`serverName` is baked irrevocably into every user and room id, so a hive
that rebuilds onto a new default is a *different homeserver*, not a
renamed one: existing accounts and rooms are stranded, and reverting the
config does not undo it. Its neighbours (`gatewayHost`, the forge domain)
are routing, rediscovered through `.well-known` and fixed by editing them
back. Same diff shape, three orders of magnitude apart in blast radius --
which is an asymmetry a module should carry rather than an operator.
An activation script and not `warnings`, which is where this obviously
belongs and does not work: the condition needs the host filesystem, and
`nixos-rebuild switch --flake` evaluates purely, where
`builtins.pathExists "/var/lib/..."` answers false rather than throwing.
A `warnings` entry gated on it would evaluate, deploy, and print nothing
on every real deployment.
Rendered only when `serverName` is null, so a pinned hive has no script
rather than a script that stays quiet -- a guard that cries wolf at a
correctly-configured deployment makes the next real one read as noise.
Never fails the activation: it warns about a choice that cannot be
undone, and refusing the rebuild of a hive that already chose
deliberately is the opposite of helping.
The probed path is read out of the container's own evaluated config
rather than hardcoded. A guessed path resolves cleanly and silently never
matches, which is the same failure this guard exists to catch one level
up.
mara asked whether SWARM_CONTROLLER_PUBLIC_URL should also be set when the
controller does not run on the same host, and then said the two read as one
service to her. Both are worth answering in the file rather than only in the
thread: that split is not representable today, and the reason is two hops
away from this line.
The UI's /api/ location proxies http://unix:<socketPath> -- a path that
resolves nowhere else -- and this daemon binds no TCP address at all. So the
vhost and the daemon are co-located by construction, and ui.enable is the
flag that declares the vhost rather than a guess about some host serving it.
Also names the tripwire: the day the daemon grows a TCP listener, this line
silently stops being right (env unset, registration quietly skipped, no
error). That is when an explicit publicUrl option becomes correct -- not
before, while there is exactly one derivable answer.
The endpoint landed inert: nothing pointed at it, so the only way to see
it work was to mint an HMAC by hand. Register the two swarm-wide hooks
at startup so a real forge event produces a journal line.
Registered ALONGSIDE the per-hive hooks, not instead of them. Every hive
keeps receiving and acting on its own deliveries; the controller gets a
copy and logs it. Moving the registration is a later step and has to be:
fan-out swarm->hive does not exist yet, so a hook moved now would point
at a receiver that forwards nowhere, silently on both sides.
Deliberately no stale-hook deletion arm, unlike the two per-hive
registrars this otherwise mirrors: theirs delete hooks matching their own
path with a foreign base, and the hives' hooks are not stale.
The route prefix is what keeps this safe. Both hive-side registrars
delete any hook ending in /webhook/knowledge or /webhook/config-pr with a
different base, so a swarm hook under those paths would be deleted by
every hive on every boot. Serving them under /webhook/forge/ avoids it,
and a test pins it -- there is nothing else that can.
SWARM_CONTROLLER_PUBLIC_URL is set only where the swarm vhost is served,
because a hook whose target_url nothing answers is worse than no hook.
A forge webhook is a machine POST carrying an HMAC and no session cookie,
so it cannot pass the authelia auth-request subrequest every other location
on this vhost uses. This location deliberately omits it; the HMAC check in
the controller is what guards the path.
Scoped to /webhook/forge/ rather than /webhook/ so a future endpoint under
the same prefix does not inherit the bypass.
The crate's module doc claimed no persistence and no writes, which this
change makes false, and docs/persistence.md had no swarm-controller entry
at all — the webhook secret is the daemon's first persisted state.
A Forgejo webhook has one target_url, so every hive registering the same
swarm-wide hooks is last-writer-wins rather than idempotent. The controller
is the only swarm-wide thing in the deployment, so it becomes the receiver.
It verifies the HMAC and treats the payload as opaque bytes keyed by the hook
kind in the URL path; it deliberately does not parse the payload, because the
hives' existing handlers already decide what a delivery means.
Nothing is registered against the endpoint yet. The replacement path is built
and observable before anything takes the old one away, so the swarm's single
target_url never points at a receiver that forwards nowhere.
Its last call sites were the JSON store's to_string_pretty/from_str,
which this branch replaced with serde_norway. Nothing in swarmctl/src
names it any more.
Caught by argus, and it is the second time: the same reviewer found a
dead reqwest in swarm-controller after a move earlier this week. No lint
in this workspace sees an unused dependency, so both were green locally
and in CI -- a move has two sides and my gate only looked at one. My
local gate now carries an advisory check for it.
Six places asserted the old design as fact, and none of them mention the
change by name -- the class of doc breakage that is found by asking what
a diff made untrue, not by grepping for a feature:
- swarmctl/README.md and swarm-authelia-bridge/README.md both described
their own private canonical store. The bridge's "known limitation"
section described the seam as unsolved; it is what this fixes, so it
becomes what both writers must uphold instead.
- docs/swarm/{sso,ui,secrets}.md described a rendered artifact.
- The repo CLAUDE.md entry for swarmctl said the same.
- docs/tools/swarmctl-cli.md is regenerated (CI diffs it against the
clap tree), picking up the removed --store flag.
Operator-facing where it is read: the hand-editing consequence (values
survive a rewrite, comments do not) is stated in sso.md, where an
operator is being told to edit the file, rather than only in a module doc.
#3414 fixed the missing-email defect in swarmctl only, so every identity
created by the bridge landed in users.yml with no `email`. A relying
party that asks for the claim does not degrade, it fails -- grafana's
OIDC login is the measured case (#3393).
Two writers of one file disagreeing about a field one of them treats as
required is not a difference worth keeping, and with the file shared the
result also depended on which tool wrote last.
Mutation-checked: removing the fill turns the new test red and nothing
else.
The second half of making one file canonical. swarmctl kept its own
private JSON store and rendered users.yml from it, against the same
physical file the bridge wrote -- the seam that made `swarm agent create`
refuse to run.
- users.yml is read before it is written, so users another writer added
are loaded rather than treated as a file to refuse or clobber. The
overwrite guard and the seed check go with the second store: they
existed to police two stores that could disagree.
- serde_norway replaces the hand-rolled emitter. Unknown top-level and
per-user keys round-trip through `extra`, so two writers cannot delete
each other's fields.
- The synthetic email moves from render time to the write path and is
stored. With the file as the store, "rendered but not persisted" has
nowhere left to live, and mara ruled the stored address correct.
- `--store` is removed rather than deprecated: a flag whose only
remaining effect is nothing reads as accepted and does nothing.
Three tests asserted the emitter's exact bytes, and one asserted the
guard. Rewritten rather than deleted -- as round-trips for the former,
and inverted for the latter, since "a populated file is READ" is the
behaviour this change is for and deleting its test would leave it
unpinned.
The bridge no longer has a private canonical store, so the env var
naming one is gone rather than repointed. What is left is the single
users file it now reads and writes directly.
That line is what made `swarm agent create` fail on this hive: the
bridge found no store at its own JSON path, saw users already in
`users.yml`, and refused to overwrite a file it had not written --
correctly, given two things claimed to be canonical for one file.
`swarm agent create` failed with "no user store at
swarm-authelia-bridge-users.json but users.yml already holds users" on
any hive that had users. The guard was correct; what was wrong is that
two files both claimed to be canonical for one physical file.
The bridge kept a private users.json and rendered users.yml from it,
while swarmctl kept its own pair against the same users.yml. A writer
whose own JSON was absent could not tell "nothing here yet" from
"someone else's users", so it refused to write at all.
users.yml becomes the store: read before write, through serde_norway
rather than a hand-rolled emitter. Unknown top-level and per-user keys
round-trip through `extra`, or whichever process writes second would
silently delete what the first added. Validation stays on the write path
-- a bare to_string(&store) serialises perfectly and drops the "no
control character ever reaches this file" guarantee silently.
The seed constant goes with the guard: nothing writes a seed now, an
absent file is an empty store, and left as a pub constant it read as if
the seed dance were still load-bearing.
swarm-authelia-bridge answered one 401 for four causes, including its own
client credentials being rejected by authelia — telling a caller its token
was invalid when the fault was ours. That cost a real diagnostic round-trip,
because the only thing separating the two was a log line inside a container.
Introspection now returns a three-way Verdict, and one classify() maps it to
either admission, a 401 (the caller's credential) or a 503 (ours). Admission
is unchanged: Ok is reachable from exactly one variant.
Closes#3423.
Modern browsers validate an <input pattern> attribute's regex in
Unicode-set ('v') mode, which is stricter about hyphen placement than
classic mode: `[a-z0-9-]` throws "Invalid character class" under 'v'
mode even though a trailing hyphen is unambiguous (and valid) in classic
regex. Reproduced directly: `new RegExp('[a-z0-9-]', 'v')` throws,
`new RegExp('[a-z0-9\\-]', 'v')` doesn't. Escaping the hyphen fixes it
without changing what the pattern matches.
atlas, on the issue: swarm-controller's problem+json conversion merged
(58588a68, closing hyperhive#3412 which this comment referenced as an
in-flight fix). Every first-party API is RFC 9457 now, so the non-JSON
fallback path is not backend-transition compatibility anymore — it's for
bodies this repo didn't author (a caller pointed elsewhere, or an
nginx-synthesised 502/504 that never reaches our handlers). Comment
updated to say that instead of pointing at a transition that's already
finished.
mara, on review: "i want each component to import its own css file
itself[;] if you need a bunch of extra css externally, its not a proper
component ... you may need to migrate other components that you would
want to use first".
WarnBanner is the Preact-component successor to the shadow-DOM
<hive-warn> custom element (same three-tier info/warning/error visual
language, colours copied faithfully from hive-warn.css). ApiErrorPanel
now composes it instead of owning a copy of the border/colour/pulse
rules itself — its own CSS is back down to just the layout that's
actually specific to it (heading, copy button, detail text).
Re-verified with a fresh mock-server screenshot: same rendered output as
before, now via composition instead of a duplicated banner shape.
Closes#3410.
`ApiErrorPanel` renders a ProblemDetails (RFC 9457) error nicely, with a
copy button so the full text can be pasted straight into a bug report.
No truncation of `detail` — on the #3363 incident that motivated this
issue, that string was the entire diagnosis.
`readApiError`/`problemMessage`/`ProblemDetails` are promoted out of
credentials.js's original `readErrorBody` into `@hive/shared/api-error.js`
(comment rewritten: the RFC 9457 rework has already landed everywhere
except swarm-controller's status route, #3412 in flight, so the raw-text
fallback is a compat shim for that one gap, not a general transition).
credentials.js's 4 call sites switch to the shared reader (kept as
one-line messages there, its result slots are single-line aria-live
regions, not a panel context).
Wired ApiErrorPanel into OverviewPage.tsx (the issue's own worked
example) and CreateAgentPage.tsx (second real call site).
Not built on <hive-warn> despite matching its visual language — that
custom element's CSS-as-text import only works under a build with
loader: 'text' for .css (dashboard's), and silently renders unstyled
under swarm-ui's default css loader (filed separately as #3415).
ApiErrorPanel is a self-contained light-DOM component instead, per
mara's own suggestion to keep it independent of the old UI's shapes.
Plugin management is server-admin scoped, and on an SSO hive nobody holds that
role: auto_assign_org_role grants an org role, and the built-in local admin
that does hold server admin cannot log in because the login form is disabled
whenever SSO is configured. Two individually-correct decisions leaving no path
to the plugin UI at all.
Declarative is the way through rather than a workaround for it -- plugins land
in the store and in git, survive a rebuild and a state reset, and the container
needs no runtime egress to grafana.com.
mkIf rather than passing the list through: upstream's default is null, while an
empty list is a real value pointing the plugin path at an empty store dir, so a
hive that sets nothing must keep seeing null.
Authelia serves no `email` claim for a user without one, and a relying party
that wants that claim does not degrade -- grafana falls through to
`<api_url>/emails`, a GitHub-ism authelia does not implement, and the login
dies with InternalError naming nothing useful. So an absent address is a
broken login rather than a sparse profile.
Uses `<username>@hyperhive.local`, the domain hive-c0re already gives every
agent's forge account and every hyperhive-authored commit. Not deployment-
derived: that would have to be plumbed in from config, and an operator already
supplying a domain may as well supply the whole address.
Synthesised in the renderer, never in the store: users.json stays honest that
none was supplied, an operator who later sets a real one is not fighting an
invented value, and existing users are fixed by the next render with no
migration step. A supplied address always wins.
The old test asserting an absent email is omitted pinned exactly the behaviour
that broke the login; it is split so the group half keeps its meaning and the
email half states the new contract.
Its three error paths returned a bare string with a status code, which forces
a caller to treat the whole body as prose. hive-c0re converted some time ago,
so swarm-controller was the last backend on the old shape -- and it is the one
behind the hive status page's 503, where the body is frequently the entire
diagnosis rather than a summary.
Adds the commitment to docs/conventions.md, since it was implied by the code
in one daemon and written down nowhere: an endpoint of ours answering with a
bare string is a bug to file, not something callers work around.
The test asserts the rendered response -- media type plus an addressable
detail -- rather than the problem_details value, because a handler that built
the value correctly and then returned it as a string would satisfy a test
written against the type alone.
`store.keys()` creates an ephemeral ordered consumer, whose create subject
ends at the stream name. `>` matches one or more tokens and never zero, so
the `.>` form alone never covered it: the server refused
`$JS.API.CONSUMER.CREATE.KV_hive-status`, the refusal reached the client as a
timeout, and the operator saw a 503 on the hive status page.
The test asserted only the `.>` form, which reads as covering the bare one, so
the suite stayed green while every list timed out in production. It now names
the bare subject separately and first.
Last of the three modules that hand-rolled the same concat with wantedBy +
before and no requires, so a failed assembly left the consumer running against
a missing file and trusting nothing -- every outbound TLS call fails while the
unit looks healthy.
The forge is the one with two consumers: forgejo-sso-source fetches the
issuer's discovery document over the swarm CA and once shipped without the
trust its sibling had. It only exists when SSO is on, so the consumer list is
conditional -- naming an absent unit would define a serviceless one and order
nothing.
Removes the now-dead useSelfSigned and caContainerPath bindings (nix does not
warn) and retargets three comments the deletion orphaned, including the helper
header that still named this module as the per-call-site concat.
Both hand-rolled the same concat with wantedBy + before and no requires,
so a failed assembly let the consumer start against a missing file and
trust nothing at all -- which fails every outbound TLS call while the unit
looks healthy. The helper puts requires on the consumer and verifies the
assembled bundle before moving it into place.
Removes each module's now-dead useSelfSigned binding: its only use was the
mkIf on the deleted block, and nix does not warn about an unused let
binding. Also corrects two comments the deletion orphaned -- one described
the removed path binding, the other pointed at a bundle service that no
longer exists.
A loopback literal encodes 'authelia is in my netns' at the call site,
and authelia's OIDC endpoints are https-only in effect: reached directly
they answer 400, because the forwarded headers nginx injects for every
other consumer are what let it determine its own issuer. Going by name
deletes the need for those headers rather than reproducing them, and
converges on the URL swarm-nats-auth already uses.
Depends on the CA trust added for the same container in #3407 -- without
it this swaps a 400 for an UnknownIssuer.
The null-url assertion joins the module's existing list: interpolating a
null would surface as a nix coercion error several files from its cause.
Same class-B defect as the queue responder: swarm-authelia-bridge
introspects authelia by name over https and the container trusted no swarm
CA, so the call could only ever fail UnknownIssuer.
This is also the prerequisite for the by-domain introspection change on
#3391 -- flipping that URL without the trust half would swap one failure
for another.
The responder introspects authelia over https by name. It had no CA trust
at all, so the handshake failed UnknownIssuer, introspection failed, and it
denied every client -- surfacing at the controller as a 60s
authorization-violation loop, two layers from the cause.
Adds a shared trustBundle helper to lib/hive-ca-trust.nix rather than a
fifth hand-rolled concat. Four containers were each assembling this
themselves, which is how they came to share one defect: wantedBy + before
express ordering but not success, so a failed assembly let the consumer
start against a missing file and trust nothing at all.
The helper fixes both halves of that. requires goes on the consumer, so a
failed bundle stops it and the dependency is visible in systemctl status
where someone debugging a TLS failure looks. And the script assembles to a
temp path, checks the result actually contains a certificate, and only then
moves it into place -- cat of an empty bind exits 0, so set -e does not
catch it and a partial bundle must never appear under the final name.
Returns a module rather than bare services: a caller that already writes
systemd.services.<consumer> cannot also write systemd.services in the same
attrset.
Keeps why the setting exists (the bridge writes this file and cannot
restart authelia) and drops the incident narrative and the answered
objections -- that history belongs in the PR discussion, not in three
comment blocks a future reader has to scroll past.
authelia now watches the users file, so the restart is redundant -- and it
was the wrong shape twice over. It could fail: a login was refused for a
user whose record was already correct on disk, with nothing in either log
implicating the reload. And it only ever worked for this writer --
swarm-authelia-bridge writes the same file and cannot restart anything,
since running unprivileged inside the container is the whole reason it may
write it at all. A reload that depends on which process did the writing is
not a reload.
--machine/--unit and their two env vars existed solely to name a
systemctl -M target, so they go with it. That drops two required settings
from the operator surface.
The three objections previously recorded against watch are all answered
now, and are kept next to the decision rather than deleted: the key is
verified against the pinned build (validate-config accepts it and rejects
a misspelling), the watch is on the directory so a rename is observed, and
partial reads are structurally impossible because every writer of this
file goes through write_atomic.
Authelia reads authentication_backend.file.path once at startup. Without
watch, an identity the bridge creates is real on disk and invisible to the
running authelia until something unrelated bounces the unit.
swarm-authelia-bridge cannot restart authelia -- running unprivileged
inside the container is the whole reason it may write that file, so
restarting the unit is exactly the privilege it was designed not to hold.
swarmctl does restart it, but that is a systemctl -M shellout that can
fail, and when it did the symptom was a login refused for a user whose
record was already correct on disk. This makes correctness stop depending
on that restart succeeding.
The auth-callout responder decides what an admitted client may publish
from two strings: the prefix marking a hive client, and the client id
allowed to read every hive's key. Both were literals in three places --
swarm-authelia.nix mints "hive-${name}", swarm-controller.nix defines
"swarm-controller", and the responder carried its own copies as clap
defaults because swarm-nats.nix passed neither.
Each producer now publishes its value as a readOnly option and the
responder's ExecStart reads them, so the agreement is one evaluation
rather than three strings that happen to be equal. Same pattern the
module already uses for `--account`, and the same argument
swarm-authelia.nix gives for publishing `machine` and `unit`.
Worth the change because the failure is silent and misattributed:
rename either principal and the responder starts denying the one that
stopped matching, a denial reaches a NATS client as a timeout rather
than an error, and a hive that is refused looks exactly like a hive
that has not reported yet.
The module doc had grown a 37-line preamble carrying three separate
arguments, which the comment-block lint refuses. Splitting it is the
better fix than raising the limit: the reader who is about to widen a
subject list meets the reason not to at the list, not seven screens up.
Also corrects one claim I had no measurement for. The CREATE note said
creating an existing stream with a different config "is an error rather
than a rewrite" — asserted, not observed. What is observed is narrower
and enough: a hive holding this grant leaves the stream config untouched
and never publishes $JS.API.STREAM.UPDATE at all.
A grant carrying every bucket-specific subject and neither of these
cannot create the bucket at all: the client times out on `$JS.API.INFO`
long before it reaches a subject that was granted, and a NATS denial
reaches the client as a hang rather than an error.
Both were named by the server's own log, not reasoned about.
`$JS.API.INFO` is the account-level JetStream info every client requests
on connect; `$JS.API.STREAM.NAMES` is how a client finds the stream
backing a bucket. The latter lets a client enumerate stream names in the
account, which in an account holding one bucket discloses a name both
ends already share.
Every earlier measurement missed them, because each either granted
`$JS.API.>` wholesale or ran against a bucket the setup had already
created while unscoped. A minimum established against an existing bucket
is not the minimum for making one, so the leave-one-out that trimmed the
reader's set could not have found this — every candidate it tried was
tried in a world where the bucket existed.
Found by running the shipping gate against the real binary. No unit test
could have: the failure is a timeout inside a real server's permission
check.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
A swarm service container shares the host netns and force-disables
resolvconf, so it inherits the /etc/resolv.conf nixos-containers copies in
at start (cp --remove-destination, host-side preStart, once per start) and
nothing ever refreshes it. That makes the container's resolver a snapshot of
the host's file at its boot instant.
When that snapshot is wrong the container can never recover, and the symptom
appears arbitrarily far from the cause: swarm-nats-auth cannot resolve
authelia's name, so it denies every auth-callout request and the queue
refuses every client with an authorization violation.
Give each of the four swarm containers a oneshot that writes the resolver
file itself, from the bridge IP, ordered before that container's first DNS
consumer. The shape is the one every agent container already uses.
networking.nameservers cannot do this: resolvconf is its only consumer and
these containers disable it, so setting it renders no file while still
evaluating cleanly. A static environment.etc entry cannot either -- it would
have to survive etc activation landing on the regular file the host already
copied there, which no eval can show.
The error names the protocol and never the sandbox, so it reads like a
network fault. Records the two easy-to-miss families (AF_INET6 alongside
AF_INET, and AF_NETLINK for getaddrinfo) and the reason the directive rots:
it is a claim about what the program does, and nothing re-checks it when a
client is added.
The unit restricted RestrictAddressFamilies to AF_UNIX, which was correct
while the daemon only served its unix socket. It has since grown three
outbound clients -- authelia token minting and forge calls over HTTPS, and
the queue over NATS -- and every socket(AF_INET, ...) was refused by seccomp.
systemd surfaces that refusal as EAFNOSUPPORT, "Address family not supported
by protocol", so the failure names the protocol and never the sandbox. The
visible symptom was swarm agent creation failing while minting a bearer
token, with a connect error that reads like a network fault.
Permit AF_INET/AF_INET6 for those clients and AF_NETLINK, which glibc's
getaddrinfo needs to enumerate local addresses before returning one. The
rest of the unit's hardening is unchanged.
The new regression test constructed `reqwest::Client::new()`, which
panics in the nix build sandbox: with no system CA store,
`ClientBuilder::build()` reaches `rustls_platform_verifier::Verifier::new()`
and fails, and `new()` is `build().expect(..)`. The test passed locally
because a devshell has `/etc/ssl/certs`, and failed in CI.
Disabling certificate verification takes the branch that installs a
no-op verifier and never consults the platform store, so the client
builds anywhere. That is sound in this test and nowhere else: nothing is
sent, the request is built and its bytes are inspected. The helper says
so at the point someone would otherwise object to it.
Found by reading reqwest's `ClientBuilder::build()` rather than trying
builder flags: the first repro attempt — pointing `SSL_CERT_FILE` and
`SSL_CERT_DIR` at nothing — did not reproduce, so any fix verified
against it would have been verified against nothing.
Every access-token request the swarm queue client has ever made was
refused. It sent the client id and secret as form fields
(`client_secret_post`); authelia's client registration allows only
`client_secret_basic`, so the identity provider rejected the request
before looking at the credentials at all.
What made it survive so long is the shape of the failure. The refusals
tripped authelia's rate limiter, whose penalty grows faster than this
client's retry interval — 56s, then 296s, then 535s, against a retry
every 60s — so the limiter never drained and a 429 came back before the
credentials were evaluated. The line naming the real cause appeared
roughly once an hour, inside a continuous storm of a different error,
and the storm read as the problem.
Both other callers in this workspace that present client credentials
already use Basic. RFC 6749 says clients SHOULD, authelia's registration
default says so, and a secret in a header is one fewer place for a proxy
to log it.
The test asserts the shape of the request rather than a server's reply:
Authorization is Basic, the body carries the grant type, and neither the
secret nor the client id appears in the body. It fails on the previous
code with no identity provider, no deployment and no network — which is
what this needed and did not have.
