e94406cdb9
Some checks were skipped
public bin cache / build + push to preem:grid (push) Has been skipped
The controller's OIDC client secret (client `swarm-controller`, used for the queue connection, the auth-bridge bearer and the OTLP push) came from an operator-placed file, `deploy.swarm-controller.queue.clientSecretFile`, handed in by `LoadCredential=`. Now `swarm-secret-publish`, which already copies authelia's minted OIDC secrets into the store, also publishes this one, to `swarm/controller/swarm-controller/oidc/client`. That path sits under `controller/`, which no hive's policy reads. The controller reads it once at start with its existing store certificate and holds it in memory, as `swarm_queue_client::ClientSecret::Value`. If the store is down, it retries for about a minute and then fails the start, so `Restart=` tries again. Policy delta: the controller gets `read` on that leaf, and the publisher gets `create`/`update` on that leaf. Removed: the `queue.clientSecretFile` option (both spellings, now removed options with a message), its singleHostSwarm default, the credential and placeholder, and the path watcher plus its restart oneshot. A controller without a store identity is now an eval error, because it has no other way to get the secret.
110 lines
5.8 KiB
Markdown
110 lines
5.8 KiB
Markdown
# swarm-queue-client
|
|
|
|
Connecting to the swarm message queue as an authenticated client. Shared by
|
|
every process that participates: the swarm controller reads hive status out of
|
|
the queue, a hive publishes its own status into it.
|
|
|
|
## Why a crate and not a module per binary
|
|
|
|
The _connect_ is identical for every participant — mint an authelia token,
|
|
present it at CONNECT for the `auth_callout` responder to introspect, let
|
|
`async-nats` re-run the callback on each connection attempt. Only the **use**
|
|
differs.
|
|
|
|
Two copies of that would be two copies of credential handling, and a
|
|
token-refresh fix would have to be found twice. The same reasoning already put
|
|
`hive-sock-client` in its own crate rather than in each daemon that speaks to a
|
|
unix socket.
|
|
|
|
## The two properties that constrain the code
|
|
|
|
**A token expires.** Authelia issues `client_credentials` access tokens with
|
|
`expires_in: 3599`. Authentication happens at CONNECT, so a long-lived
|
|
connection is fine — but a _reconnect_ an hour later needs a token minted an
|
|
hour later.
|
|
|
|
**The refresh therefore lives in the auth callback, not in a timer.**
|
|
`async-nats` invokes it per connection attempt, so there is no window in which
|
|
the client holds a token it minted for a previous connection. The alternative —
|
|
mint once, own the reconnect loop — fails in the way this subsystem exists to
|
|
prevent: the process keeps serving while its data quietly stops moving, and
|
|
nothing says so until someone reads a dashboard.
|
|
|
|
## Configuration
|
|
|
|
`QueueConfig::from_env(prefix)` reads `<prefix>_NATS_URL`,
|
|
`<prefix>_OIDC_TOKEN_ENDPOINT`, `<prefix>_OIDC_CLIENT_ID` and
|
|
`<prefix>_OIDC_CLIENT_SECRET_FILE`.
|
|
|
|
The prefix is a parameter because the variables belong to the consuming unit —
|
|
a NixOS module sets them alongside its other options. What is shared is the
|
|
rule, not the spelling: **all four together or none at all.** A half-set
|
|
environment is a hard error, because the failure it would otherwise produce is
|
|
the expensive kind — the process comes up "fine", never connects, and the data
|
|
it was supposed to move silently stops.
|
|
|
|
The client secret is never in the environment, which would publish it to
|
|
anything that can read `/proc/<pid>/environ`. `from_env` takes it as a
|
|
**path** (`ClientSecret::File`), read per token request rather than cached, so
|
|
a rotation the operator believes took effect actually did. A caller that
|
|
fetched the secret from somewhere else builds the config itself with
|
|
`ClientSecret::Value`, held in memory; `swarm-controller` does this with the
|
|
secret it reads from the swarm's secret store.
|
|
|
|
## Token request shape: HTTP Basic, `audience`, `scope`
|
|
|
|
`token_request` builds a `client_credentials` grant, authenticated with HTTP
|
|
Basic — **never** the form body. Both are legal OAuth 2.0
|
|
(`client_secret_basic` vs `client_secret_post`), but a client registration
|
|
names one, and authelia's default (and ours) is Basic. Sending the
|
|
credentials in the body once got every token request refused with `Client
|
|
authentication failed … the registered client is configured to only support
|
|
'client_secret_basic'`, which reached the operator as an endless `429`
|
|
because the retries tripped a rate limiter whose penalty grew faster than the
|
|
retry interval — the 429 arrived _before_ the credentials were ever
|
|
evaluated, so the line naming the real cause appeared once an hour. RFC 6749
|
|
§2.3.1 says clients SHOULD use Basic, both introspection callers in this
|
|
workspace already do, and a secret in a header is one fewer place for a proxy
|
|
to log it.
|
|
|
|
`audience` (RFC 8707 resource indicators) and `scope` are both opt-in;
|
|
`None` for either reproduces the request this crate sent before the
|
|
parameter existed. Omitted, `audience` gets whatever authelia defaults a
|
|
scopeless `client_credentials` grant to — the queue connection's own case,
|
|
which has always worked without asking. A caller proving this identity to a
|
|
specifically audience-checked receiver (the swarm-otel `oidc/swarm`
|
|
authenticator being the first one) has to ask by name, the same way
|
|
`swarm-otel.nix`'s own prometheus scrape config already does per target
|
|
(`endpoint_params.audience`): a token minted without asking carries `aud:
|
|
[]`, and an audience-checked receiver refuses that just as readily as the
|
|
wrong one. `scope` follows the same rule and travels with `audience`:
|
|
registration is not issuance, and the authorisation server grants no scope
|
|
the client never requested. A caller reaching a destination behind
|
|
authelia's `/api/authz/auth-request` needs `authelia.bearer.authz` here
|
|
however completely the client is registered for it — the failure
|
|
`swarm-otel.nix` records against its own client (a scopeless token refused
|
|
at introspection with "the requested scope is invalid, unknown, or
|
|
malformed") is the same one, one layer down.
|
|
|
|
## What this crate does not do
|
|
|
|
It ends at a connected client. `jetstream`/`kv` are **off by default** — what a
|
|
consumer does with the connection is its own business, and its `Cargo.toml` is
|
|
where that requirement should be visible. The auth-callout responder speaks the
|
|
connect and nothing else, and pays for nothing else.
|
|
|
|
## The one exception: the `kv` feature
|
|
|
|
`kv` adds `status`, which holds the name and the creation config of the
|
|
`hive-status` bucket — nothing more.
|
|
|
|
It is here because that bucket has **two ends in two crates**: a hive writes its
|
|
own key, the controller reads every key. The name being a repeated literal is
|
|
the mild half of the problem; the sharp half is that either end may arrive first
|
|
on a fresh swarm, so both create the bucket if it is missing. Two `Config`s that
|
|
drift means whichever end created it wins and the other opens a bucket it did
|
|
not ask for — no error, no log, just a retention policy nobody chose.
|
|
|
|
An agreement between two crates has to live in one of them, and neither end of
|
|
this bucket is senior to the other. Behind a default-off feature, the consumer
|
|
that needs none of it still pays nothing.
|