hyperhive/swarm-queue-client/README.md

swarm-queue-client

Connecting to the swarm message queue as an authenticated client. Shared by every process that participates: the swarm controller reads hive status out of the queue, a hive publishes its own status into it.

Why a crate and not a module per binary

The connect is identical for every participant — mint an authelia token, present it at CONNECT for the auth_callout responder to introspect, let async-nats re-run the callback on each connection attempt. Only the use differs.

Two copies of that would be two copies of credential handling, and a token-refresh fix would have to be found twice. The same reasoning already put hive-sock-client in its own crate rather than in each daemon that speaks to a unix socket.

The two properties that constrain the code

A token expires. Authelia issues client_credentials access tokens with expires_in: 3599. Authentication happens at CONNECT, so a long-lived connection is fine — but a reconnect an hour later needs a token minted an hour later.

The refresh therefore lives in the auth callback, not in a timer. async-nats invokes it per connection attempt, so there is no window in which the client holds a token it minted for a previous connection. The alternative — mint once, own the reconnect loop — fails in the way this subsystem exists to prevent: the process keeps serving while its data quietly stops moving, and nothing says so until someone reads a dashboard.

Configuration

QueueConfig::from_env(prefix) reads <prefix>_NATS_URL, <prefix>_OIDC_TOKEN_ENDPOINT, <prefix>_OIDC_CLIENT_ID and <prefix>_OIDC_CLIENT_SECRET_FILE.

The prefix is a parameter because the variables belong to the consuming unit — a NixOS module sets them alongside its other options. What is shared is the rule, not the spelling: all four together or none at all. A half-set environment is a hard error, because the failure it would otherwise produce is the expensive kind — the process comes up "fine", never connects, and the data it was supposed to move silently stops.

The client secret is a path, not a value: putting it in the environment would publish it to anything that can read /proc/<pid>/environ. It is read per token request rather than cached, so a rotation the operator believes took effect actually did.

Token request shape: HTTP Basic, audience, scope

token_request builds a client_credentials grant, authenticated with HTTP Basic — never the form body. Both are legal OAuth 2.0 (client_secret_basic vs client_secret_post), but a client registration names one, and authelia's default (and ours) is Basic. Sending the credentials in the body once got every token request refused with Client authentication failed … the registered client is configured to only support 'client_secret_basic', which reached the operator as an endless 429 because the retries tripped a rate limiter whose penalty grew faster than the retry interval — the 429 arrived before the credentials were ever evaluated, so the line naming the real cause appeared once an hour. RFC 6749 §2.3.1 says clients SHOULD use Basic, both introspection callers in this workspace already do, and a secret in a header is one fewer place for a proxy to log it.

audience (RFC 8707 resource indicators) and scope are both opt-in; None for either reproduces the request this crate sent before the parameter existed. Omitted, audience gets whatever authelia defaults a scopeless client_credentials grant to — the queue connection's own case, which has always worked without asking. A caller proving this identity to a specifically audience-checked receiver (the swarm-otel oidc/swarm authenticator being the first one) has to ask by name, the same way swarm-otel.nix's own prometheus scrape config already does per target (endpoint_params.audience): a token minted without asking carries aud: [], and an audience-checked receiver refuses that just as readily as the wrong one. scope follows the same rule and travels with audience: registration is not issuance, and the authorisation server grants no scope the client never requested. A caller reaching a destination behind authelia's /api/authz/auth-request needs authelia.bearer.authz here however completely the client is registered for it — the failure swarm-otel.nix records against its own client (a scopeless token refused at introspection with "the requested scope is invalid, unknown, or malformed") is the same one, one layer down.

What this crate does not do

It ends at a connected client. jetstream/kv are off by default — what a consumer does with the connection is its own business, and its Cargo.toml is where that requirement should be visible. The auth-callout responder speaks the connect and nothing else, and pays for nothing else.

The one exception: the kv feature

kv adds status, which holds the name and the creation config of the hive-status bucket — nothing more.

It is here because that bucket has two ends in two crates: a hive writes its own key, the controller reads every key. The name being a repeated literal is the mild half of the problem; the sharp half is that either end may arrive first on a fresh swarm, so both create the bucket if it is missing. Two Configs that drift means whichever end created it wins and the other opens a bucket it did not ask for — no error, no log, just a retention policy nobody chose.

An agreement between two crates has to live in one of them, and neither end of this bucket is senior to the other. Behind a default-off feature, the consumer that needs none of it still pays nothing.