c2c61b6146
hive-c0re publishes this hive's status to the swarm queue as the OIDC client `hive-<name>`. Its secret came from `deploy.hive-controller.statusPublish.clientSecretFile`, which defaulted to authelia's minted file only on a host running both authelia and the queue. Every other hive had no secret and never published, so the controller showed it `never_reported`. Now `swarm-secret-publish`, beside authelia, also copies each hive's `hive-<name>` secret to `swarm/hives/<hive>/oidc/client`. Its policy already allows `swarm/hives/*`. hive-c0re reads that path under the hive's own store leaf, which the existing `swarm/hives/<hive>/*` grant covers, and holds the value in memory. The shared queue client no longer remembers a failed first connect for the life of the process. A failed attempt is retried with a backoff from 5s, doubling to a cap of 5 minutes; the dashboard banner stays up until an attempt succeeds. The two status publishers wait for the client instead of giving up on the first `None`, so a secret published after boot is picked up without a restart. Removed: the `statusPublish.clientSecretFile` option (both spellings, now removed options with a message), its same-host default, its LoadCredential, the assertion tying it to the other two coordinates, and `swarm_queue_client::QueueConfig::from_env`, whose only caller was this read. Publishing is now on wherever the queue URL and token endpoint are set and the hive holds a store leaf. Refs #5015
107 lines
5.7 KiB
Markdown
107 lines
5.7 KiB
Markdown
# swarm-queue-client
|
|
|
|
Connecting to the swarm message queue as an authenticated client. Shared by
|
|
every process that participates: the swarm controller reads hive status out of
|
|
the queue, a hive publishes its own status into it.
|
|
|
|
## Why a crate and not a module per binary
|
|
|
|
The _connect_ is identical for every participant — mint an authelia token,
|
|
present it at CONNECT for the `auth_callout` responder to introspect, let
|
|
`async-nats` re-run the callback on each connection attempt. Only the **use**
|
|
differs.
|
|
|
|
Two copies of that would be two copies of credential handling, and a
|
|
token-refresh fix would have to be found twice. The same reasoning already put
|
|
`hive-sock-client` in its own crate rather than in each daemon that speaks to a
|
|
unix socket.
|
|
|
|
## The two properties that constrain the code
|
|
|
|
**A token expires.** Authelia issues `client_credentials` access tokens with
|
|
`expires_in: 3599`. Authentication happens at CONNECT, so a long-lived
|
|
connection is fine — but a _reconnect_ an hour later needs a token minted an
|
|
hour later.
|
|
|
|
**The refresh therefore lives in the auth callback, not in a timer.**
|
|
`async-nats` invokes it per connection attempt, so there is no window in which
|
|
the client holds a token it minted for a previous connection. The alternative —
|
|
mint once, own the reconnect loop — fails in the way this subsystem exists to
|
|
prevent: the process keeps serving while its data quietly stops moving, and
|
|
nothing says so until someone reads a dashboard.
|
|
|
|
## Configuration
|
|
|
|
A caller builds a `QueueConfig` itself from its own environment: the queue's
|
|
URL, the token endpoint and the client id, plus an optional CA file. The
|
|
variables belong to the consuming unit, so each daemon reads them under its own
|
|
prefix. What is shared is the rule, not the spelling: **the three coordinates
|
|
together or none at all.** A half-set environment is a hard error, because the
|
|
failure it would otherwise produce is the expensive kind — the process comes up
|
|
"fine", never connects, and the data it was supposed to move silently stops.
|
|
|
|
The client secret is never in the environment, which would publish it to
|
|
anything that can read `/proc/<pid>/environ`. A caller passes it either as a
|
|
**path** (`ClientSecret::File`), read per token request rather than cached, so
|
|
a rotation the operator believes took effect actually did, or as a value it
|
|
fetched itself (`ClientSecret::Value`), held in memory. `swarm-controller` and
|
|
`hive-c0re` both do the latter with the secret they read from the swarm's
|
|
secret store.
|
|
|
|
## Token request shape: HTTP Basic, `audience`, `scope`
|
|
|
|
`token_request` builds a `client_credentials` grant, authenticated with HTTP
|
|
Basic — **never** the form body. Both are legal OAuth 2.0
|
|
(`client_secret_basic` vs `client_secret_post`), but a client registration
|
|
names one, and authelia's default (and ours) is Basic. Sending the
|
|
credentials in the body once got every token request refused with `Client
|
|
authentication failed … the registered client is configured to only support
|
|
'client_secret_basic'`, which reached the operator as an endless `429`
|
|
because the retries tripped a rate limiter whose penalty grew faster than the
|
|
retry interval — the 429 arrived _before_ the credentials were ever
|
|
evaluated, so the line naming the real cause appeared once an hour. RFC 6749
|
|
§2.3.1 says clients SHOULD use Basic, both introspection callers in this
|
|
workspace already do, and a secret in a header is one fewer place for a proxy
|
|
to log it.
|
|
|
|
`audience` (RFC 8707 resource indicators) and `scope` are both opt-in;
|
|
`None` for either reproduces the request this crate sent before the
|
|
parameter existed. Omitted, `audience` gets whatever authelia defaults a
|
|
scopeless `client_credentials` grant to — the queue connection's own case,
|
|
which has always worked without asking. A caller proving this identity to a
|
|
specifically audience-checked receiver (the swarm-otel `oidc/swarm`
|
|
authenticator being the first one) has to ask by name, the same way
|
|
`swarm-otel.nix`'s own prometheus scrape config already does per target
|
|
(`endpoint_params.audience`): a token minted without asking carries `aud:
|
|
[]`, and an audience-checked receiver refuses that just as readily as the
|
|
wrong one. `scope` follows the same rule and travels with `audience`:
|
|
registration is not issuance, and the authorisation server grants no scope
|
|
the client never requested. A caller reaching a destination behind
|
|
authelia's `/api/authz/auth-request` needs `authelia.bearer.authz` here
|
|
however completely the client is registered for it — the failure
|
|
`swarm-otel.nix` records against its own client (a scopeless token refused
|
|
at introspection with "the requested scope is invalid, unknown, or
|
|
malformed") is the same one, one layer down.
|
|
|
|
## What this crate does not do
|
|
|
|
It ends at a connected client. `jetstream`/`kv` are **off by default** — what a
|
|
consumer does with the connection is its own business, and its `Cargo.toml` is
|
|
where that requirement should be visible. The auth-callout responder speaks the
|
|
connect and nothing else, and pays for nothing else.
|
|
|
|
## The one exception: the `kv` feature
|
|
|
|
`kv` adds `status`, which holds the name and the creation config of the
|
|
`hive-status` bucket — nothing more.
|
|
|
|
It is here because that bucket has **two ends in two crates**: a hive writes its
|
|
own key, the controller reads every key. The name being a repeated literal is
|
|
the mild half of the problem; the sharp half is that either end may arrive first
|
|
on a fresh swarm, so both create the bucket if it is missing. Two `Config`s that
|
|
drift means whichever end created it wins and the other opens a bucket it did
|
|
not ask for — no error, no log, just a retention policy nobody chose.
|
|
|
|
An agreement between two crates has to live in one of them, and neither end of
|
|
this bucket is senior to the other. Behind a default-off feature, the consumer
|
|
that needs none of it still pays nothing.
|