The controller could not emit an event at all: a reader's grant is
`reader_subjects()`, which is `$JS.API.*` only, so a publish to any event
subject would be refused — and a NATS refusal reaches the client as a
timeout, so the visible symptom would have been a hive that never hears
about a change, with nothing in any log naming a permission.
Adds `swarm_queue_client::events`, following `status::BUCKET`: three
crates must agree on these strings (the controller publishes, a hive
subscribes, the callout responder decides whether the publish is
permitted), and a literal repeated across crates is an agreement nothing
checks. The responder speaks neither jetstream nor kv, so the module is
unconditional and carries no NATS types, exactly as the bucket name is.
The grant takes the wildcard form from the same function the publisher
calls, so the two cannot drift; a separate wildcard constant would have
re-created the disagreement this module exists to prevent.
Tests pin that a reader gets the subject and that a hive does NOT — a
hive able to publish here could tell a neighbour the knowledge repo
changed when it had not, which is an unauthenticated write into someone
else's control path. That one asserts on the subject root rather than a
rendered subject, so a future event leaf fails it too instead of passing
because the test only knew about `knowledge`.
Both assertions mutation-tested: removing the grant fails the reader
test, granting a hive the subject fails the denial test, each on its own
assertion line, and the unmutated tree is green.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
The new regression test constructed `reqwest::Client::new()`, which
panics in the nix build sandbox: with no system CA store,
`ClientBuilder::build()` reaches `rustls_platform_verifier::Verifier::new()`
and fails, and `new()` is `build().expect(..)`. The test passed locally
because a devshell has `/etc/ssl/certs`, and failed in CI.
Disabling certificate verification takes the branch that installs a
no-op verifier and never consults the platform store, so the client
builds anywhere. That is sound in this test and nowhere else: nothing is
sent, the request is built and its bytes are inspected. The helper says
so at the point someone would otherwise object to it.
Found by reading reqwest's `ClientBuilder::build()` rather than trying
builder flags: the first repro attempt — pointing `SSL_CERT_FILE` and
`SSL_CERT_DIR` at nothing — did not reproduce, so any fix verified
against it would have been verified against nothing.
Every access-token request the swarm queue client has ever made was
refused. It sent the client id and secret as form fields
(`client_secret_post`); authelia's client registration allows only
`client_secret_basic`, so the identity provider rejected the request
before looking at the credentials at all.
What made it survive so long is the shape of the failure. The refusals
tripped authelia's rate limiter, whose penalty grows faster than this
client's retry interval — 56s, then 296s, then 535s, against a retry
every 60s — so the limiter never drained and a 429 came back before the
credentials were evaluated. The line naming the real cause appeared
roughly once an hour, inside a continuous storm of a different error,
and the storm read as the problem.
Both other callers in this workspace that present client credentials
already use Basic. RFC 6749 says clients SHOULD, authelia's registration
default says so, and a secret in a header is one fewer place for a proxy
to log it.
The test asserts the shape of the request rather than a server's reply:
Authorization is Basic, the body carries the grant type, and neither the
secret nor the client id appears in the body. It fails on the previous
code with no identity provider, no deployment and no network — which is
what this needed and did not have.
The auth callback minted on every invocation, and async-nats runs it per
connection ATTEMPT. Its default reconnect delay backs off exponentially
and then clamps at four seconds, permanently. So a queue that could not
connect asked authelia for a token every four seconds indefinitely, and
authelia answered 429 Too Many Requests.
That rate limit then keeps itself alive: it outlives whatever first
broke the connection, and its log volume buries the original cause.
Observed in production tonight, once a redeploy fixed the TLS trust
problem that had been hiding it.
The callback now caches the token and re-mints only within two minutes
of expiry, which preserves the property that put the mint there — a
reconnect must never present a token that expired since it was minted —
while decoupling mint rate from retry rate. The reconnect cap moves to
one minute, exponential from half a second, so a brief outage still
recovers promptly and a persistent one stops hammering the identity
provider.
The module doc and the retry comment both argued for the old shape, so
both are corrected rather than left explaining why the bug was right.
The queue client built a bare reqwest::Client, so it trusted only the
platform roots. Against a swarm whose authelia is signed by the swarm
CA that is fatal: minting a token dies with 'invalid peer certificate:
UnknownIssuer', inside the auth callback, on a four-second retry loop,
with the queue never connecting.
The anchor was never missing. hive-tls.nix assembles trust-bundle.pem
and already hands it to hive-c0re as HIVE_TLS_CA_PATH; nothing pointed
the queue client at it.
QueueConfig gains an optional ca_file from <prefix>_OIDC_CA_FILE, read
outside the all-or-none tuple on purpose: a CA path with no queue is
meaningless rather than half-configured, and requiring it would break a
swarm fronted by a public certificate in order to fix one that is not.
add_root_certificate extends the default roots rather than replacing
them, so both deployments work.
A bad path fails loudly instead of falling back to the platform roots.
An operator who names a CA file wants that anchor; a silent fallback
turns their typo into UnknownIssuer five layers away.
hive-tls.nix names the bundle for both clients, beside the line that
already does it for hive-c0re, rather than having each consumer
re-derive the path.
An unqualified `[`open_or_create`]` in the module-level doc does not
resolve once the `kv` feature is on, which is the only configuration
where the module is compiled at all — so `docs-rustdoc` failed in CI
while a default-feature `cargo doc` passed locally. Measured both ways:
kv off documents clean, kv on errors `no item named open_or_create in
scope`.
Qualifying the path fixes it without widening any visibility, which is
the rule that check exists to protect.
Finishes the anyhow removal for the parts this branch adds: the status
bucket's open-or-create and the connected-client precondition. Two
variants, one of them behind the `kv` feature because the error type it
wraps does not exist without it — the error enum respects the same gate
the module does.
NotConnected is deliberately distinct from Connect: one is a connect that
was attempted and refused, the other is a request made before any
connection exists. The first is a deployment problem and the second is a
caller-ordering one, which is the whole reason a caller wants an enum
rather than a string.
The controller's `store` now returns the queue client's error rather than
an anyhow one: `OnceCell::get_or_try_init` takes its error type from the
closure, so widening there would mean converting inside the closure for
no gain. `view` `?`s it and anyhow converts at that boundary — the
library keeps a typed error, the binary keeps anyhow, and no call site
pays for the split.
An unconnected client does not fail a JetStream request, it hangs on it:
`retry_on_initial_connect` hands back a client before it is usable, and a
request made in that window waits (measured: still going at 15s against a
queue that refuses the credential). The controller guarded its read path
against that inline. Every consumer of the queue needs the same guard, so
it is not one daemon's to keep.
It matters more off a request path than on one. A hung request inside a
periodic task never reaches its `select!`, so the shutdown branch becomes
unreachable and the task cannot be stopped at all — where a request path
merely times a poll out.
The test is `!= Connected`, never `== Disconnected`: a client that has
never connected sits in `Pending`, so the `Disconnected` form passes it
straight through to the hang it was written to prevent — which is exactly
the boot-order case the guard exists for. Not feature-gated;
`connection_state()` is core async-nats.
The bucket has two ends in two crates: a hive writes its own key, the
controller reads every key. `swarm-controller` declared the name as a
private const with a doc comment arguing that "reader and writer must
name the same bucket" — an argument the writer, in another crate, could
not obey.
The name is the mild half. Both ends do get-or-create, because either may
come up first on a fresh swarm and neither can assume the other has run.
Two `Config`s that drift means whichever end created the bucket wins and
the other's `get_key_value` succeeds against a bucket it did not ask for:
no error, no log, just a retention policy nobody chose. Sharing the
constructor gives that race one outcome.
Behind a default-off `kv` feature, so the crate's other consumer — the
auth-callout responder, which speaks the connect and nothing else — still
pulls neither `jetstream` nor `kv`. That was the actual reason the
feature was excluded when this crate was extracted; the flag preserves
it. The surface is deliberately narrow: one bucket's name and creation
config, not a general KV facade.
Review catch: `anyhow::Error`'s Display special-cases `f.alternate()` to
walk the source chain; thiserror's derive does not, so `{e:#}` and `{e}`
render identically for the new error type. Every call site that held an
`anyhow::Error`, formatted it with `{:#}`, and now holds this crate's
error kept compiling, kept looking right, and silently dropped the cause.
`chain()` was written for exactly this and then left private, applied only
to the auth callback I happened to be editing. Its own doc comment argues
that dropping the source chain is wrong, which made it the one thing in
the PR that should not have had a scope of one.
The controller's "swarm queue unreachable" warning is the site this fixes
here; the stacked PR fixes the two boot-warning banners, which matter more
still — one-shot, no retry, and they leak until restart.
Operator ruling: libs should not use anyhow. The queue connect was moved
here verbatim from swarm-controller, which is a binary, so it arrived
still wearing a binary's error handling — the move changed what the code
is without changing how it reports.
Callers get variants they can match on, split by what an operator does
about them: a half-configured environment is a deployment bug, a refused
token is an identity-provider config problem, an unreachable queue is a
network one. The binaries that consume this keep anyhow and `?` converts,
so nothing downstream is more verbose for it. Same split hive-claude uses.
One thing anyhow was doing unpaid: the auth callback hands async-nats a
plain string, and a Display that stops at the top message drops the cause
— the half that says why the mint failed. `chain()` walks the source
chain, which is what `{:#}` was doing before.
A hive publishing its own status needs the same connect the controller
already has - mint an authelia token, present it at CONNECT for the
callout responder, let async-nats re-run the callback per attempt. Only
the use differs: the controller reads, a hive writes.
Copying it would put credential handling in two places, and a
token-refresh fix would then have to be found twice. That is the same
reasoning that already put hive-sock-client in its own crate rather than
in each daemon that speaks to a unix socket.
`from_env` takes a prefix rather than hardcoding SWARM_CONTROLLER_*: the
variables belong to the consuming unit, since a NixOS module sets them
alongside its other options. What is shared is the RULE - all four
together or none at all - not the spelling. The half-set case gains a
test, because it is the case the rule exists for and it previously had
none.
No jetstream/kv feature on the crate: it ends at a connected client, and
what a consumer does with it should be visible in that consumer's own
Cargo.toml.
Behaviour-preserving, and proven that way rather than by inspection: the
full behavioural gate (real nats-server, credential rotation, mutation)
is 20/0 unchanged, and the controller's own tests still pass.
