70c13aee7a
The responder re-read its secret on any non-2xx introspection answer. With the re-read backoff added in the previous commit, an IdP outage (503s, 502s from nginx, 429s) made each re-read find the unchanged secret, so the delay grew to minutes. A real 401 after the rotation then landed inside that window, and the responder denied every new NATS connect until it ran out. `Answer::Refused` now carries the status. `OidcSecret::introspect` re-reads only when `stored_secret::is_credential_refusal` says the status is 400 or 401, the same predicate `ClientSecret::refused` uses on the client path. Every other non-2xx is a denial with no re-read. The client-path test for a 429 now runs the mint against a loopback endpoint answering 429 and 503, instead of calling `refused()` directly. swarm-queue-client's README no longer names `ClientSecret::Value`. Part of #5017 (slice Q).
108 lines
5.9 KiB
Markdown
108 lines
5.9 KiB
Markdown
# swarm-queue-client
|
|
|
|
Connecting to the swarm message queue as an authenticated client. Shared by
|
|
every process that participates: the swarm controller reads hive status out of
|
|
the queue, a hive publishes its own status into it.
|
|
|
|
## Why a crate and not a module per binary
|
|
|
|
The _connect_ is identical for every participant — mint an authelia token,
|
|
present it at CONNECT for the `auth_callout` responder to introspect, let
|
|
`async-nats` re-run the callback on each connection attempt. Only the **use**
|
|
differs.
|
|
|
|
Two copies of that would be two copies of credential handling, and a
|
|
token-refresh fix would have to be found twice. The same reasoning already put
|
|
`hive-sock-client` in its own crate rather than in each daemon that speaks to a
|
|
unix socket.
|
|
|
|
## The two properties that constrain the code
|
|
|
|
**A token expires.** Authelia issues `client_credentials` access tokens with
|
|
`expires_in: 3599`. Authentication happens at CONNECT, so a long-lived
|
|
connection is fine — but a _reconnect_ an hour later needs a token minted an
|
|
hour later.
|
|
|
|
**The refresh therefore lives in the auth callback, not in a timer.**
|
|
`async-nats` invokes it per connection attempt, so there is no window in which
|
|
the client holds a token it minted for a previous connection. The alternative —
|
|
mint once, own the reconnect loop — fails in the way this subsystem exists to
|
|
prevent: the process keeps serving while its data quietly stops moving, and
|
|
nothing says so until someone reads a dashboard.
|
|
|
|
## Configuration
|
|
|
|
A caller builds a `QueueConfig` itself from its own environment: the queue's
|
|
URL, the token endpoint and the client id, plus an optional CA file. The
|
|
variables belong to the consuming unit, so each daemon reads them under its own
|
|
prefix. What is shared is the rule, not the spelling: **the three coordinates
|
|
together or none at all.** A half-set environment is a hard error, because the
|
|
failure it would otherwise produce is the expensive kind — the process comes up
|
|
"fine", never connects, and the data it was supposed to move silently stops.
|
|
|
|
The client secret is never in the environment, which would publish it to
|
|
anything that can read `/proc/<pid>/environ`. A caller passes it either as a
|
|
**path** (`ClientSecret::File`), read per token request rather than cached, so
|
|
a rotation the operator believes took effect actually did, or as a value it
|
|
fetched itself (`ClientSecret::Stored`), held in memory and fetched again
|
|
through the caller's callback when the token endpoint refuses it as a
|
|
credential. `swarm-controller`, `hive-c0re` and `swarm-review-classifier` do the
|
|
latter with the secret they read from the swarm's secret store.
|
|
|
|
## Token request shape: HTTP Basic, `audience`, `scope`
|
|
|
|
`token_request` builds a `client_credentials` grant, authenticated with HTTP
|
|
Basic — **never** the form body. Both are legal OAuth 2.0
|
|
(`client_secret_basic` vs `client_secret_post`), but a client registration
|
|
names one, and authelia's default (and ours) is Basic. Sending the
|
|
credentials in the body once got every token request refused with `Client
|
|
authentication failed … the registered client is configured to only support
|
|
'client_secret_basic'`, which reached the operator as an endless `429`
|
|
because the retries tripped a rate limiter whose penalty grew faster than the
|
|
retry interval — the 429 arrived _before_ the credentials were ever
|
|
evaluated, so the line naming the real cause appeared once an hour. RFC 6749
|
|
§2.3.1 says clients SHOULD use Basic, both introspection callers in this
|
|
workspace already do, and a secret in a header is one fewer place for a proxy
|
|
to log it.
|
|
|
|
`audience` (RFC 8707 resource indicators) and `scope` are both opt-in;
|
|
`None` for either reproduces the request this crate sent before the
|
|
parameter existed. Omitted, `audience` gets whatever authelia defaults a
|
|
scopeless `client_credentials` grant to — the queue connection's own case,
|
|
which has always worked without asking. A caller proving this identity to a
|
|
specifically audience-checked receiver (the swarm-otel `oidc/swarm`
|
|
authenticator being the first one) has to ask by name, the same way
|
|
`swarm-otel.nix`'s own prometheus scrape config already does per target
|
|
(`endpoint_params.audience`): a token minted without asking carries `aud:
|
|
[]`, and an audience-checked receiver refuses that just as readily as the
|
|
wrong one. `scope` follows the same rule and travels with `audience`:
|
|
registration is not issuance, and the authorisation server grants no scope
|
|
the client never requested. A caller reaching a destination behind
|
|
authelia's `/api/authz/auth-request` needs `authelia.bearer.authz` here
|
|
however completely the client is registered for it — the failure
|
|
`swarm-otel.nix` records against its own client (a scopeless token refused
|
|
at introspection with "the requested scope is invalid, unknown, or
|
|
malformed") is the same one, one layer down.
|
|
|
|
## What this crate does not do
|
|
|
|
It ends at a connected client. `jetstream`/`kv` are **off by default** — what a
|
|
consumer does with the connection is its own business, and its `Cargo.toml` is
|
|
where that requirement should be visible. The auth-callout responder speaks the
|
|
connect and nothing else, and pays for nothing else.
|
|
|
|
## The one exception: the `kv` feature
|
|
|
|
`kv` adds `status`, which holds the name and the creation config of the
|
|
`hive-status` bucket — nothing more.
|
|
|
|
It is here because that bucket has **two ends in two crates**: a hive writes its
|
|
own key, the controller reads every key. The name being a repeated literal is
|
|
the mild half of the problem; the sharp half is that either end may arrive first
|
|
on a fresh swarm, so both create the bucket if it is missing. Two `Config`s that
|
|
drift means whichever end created it wins and the other opens a bucket it did
|
|
not ask for — no error, no log, just a retention policy nobody chose.
|
|
|
|
An agreement between two crates has to live in one of them, and neither end of
|
|
this bucket is senior to the other. Behind a default-off feature, the consumer
|
|
that needs none of it still pays nothing.
|