The wanted-state read was a boot-time DAG node, so a swarm-level change sat
unapplied until the next restart. This watches the hive's own bucket and
converges on each update.
It does not replace the boot read: a watch hears only what is published while
it is listening, so a hive that was down still learns the current declaration
from `pull`. The watch is the fast path, `pull` stays the repair path.
Rides the connection swarm-status already opens, as a third consumer — a
second connect would double the auth-callout traffic and give the two paths
independent reconnect state, which is the reason the deploy-event drain is
spawned there too.
A delete is not a deletion order. `carries_a_declaration` is pure and tested
so that rule is enforced rather than asserted: converging on a withdrawn key
would tear down exactly the agents "absence is not a deletion order" protects.
mara on #4015: "not merging code without callers", and on the same PR
"see issue, we decided what the first thing should be". #3726 decided it:
the controller writes a token to the store and tells the hive; the hive
reads it back and writes /agents/<agent>/state/matrix-token-<account> at
0600, where matrix.nix's existing systemd.paths glob re-fires the daemon.
So this is the hive half of that, and the library's first caller.
The notice names a credential and never carries one, and deploy_subject's
own doc is why: the auth-callout responder scopes publish and leaves sub
unrestricted, so a hive that wanted another's messages could subscribe to
them. A secret in that payload would be readable swarm-wide. The value is
read from the store under the reading hive's own certificate, where the
store's policy is what actually scopes it.
Two boundaries guard the two addresses, and they are not the same check.
`path::matrix_account` guards the address in the store. `Ident` guards the
address on disk -- `agent_state_dir` takes one, so an unvalidated name off
the queue cannot reach a directory. I had written the first and assumed it
covered both; the compiler refused the `&str` and was right. `token_path`
now takes the newtype so a call site cannot forget.
The write is atomic because the path-watcher fires on the file appearing:
written in place it would be visible while partial, and the daemon would
read a truncated credential exactly once, which is the hardest possible
failure to reproduce. The temp name is dot-prefixed so it cannot match the
`matrix-token*` glob on its way past.
The publish grant is here because without it the failure is invisible.
policy.rs already says why for its siblings: a refused publish reaches the
client as a timeout, so the symptom is a hive that never receives a
credential with nothing in either log naming a permission. Two tests: the
controller may publish, a hive may not -- its own subject included. A
forged notice leaks nothing, but it would make a hive fetch and overwrite
a token file for a name the forger chose.
Refs #3726
`docs-rustdoc` fails on the `//!` header: `unresolved link to bucket`,
for an item declared in that very module. The module's docs are merged
from two fragments — the `///` on `pub mod wanted;` in `lib.rs` and the
`//!` inside the file — and the merged doc does not resolve a bare item
name against the module. Every other link in the file was already
fully-qualified (`crate::status`, `crate::status::BUCKET`), so this one
was the outlier rather than the convention.
Evidence, same command before and after: `cargo doc -p swarm-queue-client
--no-deps --document-private-items` went from 1 occurrence of the error to
0, while the unrelated diagnostic in that narrower build stayed at 1 —
the surviving one is the control that the lint is still armed. The
CI-shaped run (`--workspace`, which is what unifies features) exits 0 and
documents 27 crates including this one.
Refs #4006.
A hive reads its own declaration today and that scopes cleanly: DIRECT.GET
carries the key in the subject, so the grant can name it. A *watch* cannot be
scoped that way — a consumer's filter travels in the request payload, so
$JS.API.CONSUMER.CREATE.<stream> grants the whole stream. With every hive in
one bucket, letting a hive watch its own declaration would let it read every
other hive's.
One bucket per hive (hive-wanted-<hive>) makes the stream a hive may hold
exactly as wide as what it is allowed to see, which is what #4006's live-watch
needs. That watch is a separate change; this only moves the boundary.
mara's calls, both on #4006: one stream per hive rather than teaching the auth
responder a hive roster, and a wildcard for the controller — "its okay if
swarm controller can theoretically override hive". A bucket name is a single
subject token with no prefix matching, so no wildcard narrower than * covers N
per-hive buckets; the controller's grant is account-wide by consequence, and
documented as chosen rather than left to look accidental.
The reader arm of #4005's key-layout guard asserted the opposite of that
ruling, so it is replaced rather than deleted: the hive arm survives as
no_hive_may_write_another_role_s_agent_status (with a positive control), and
the_readers_grant_is_deliberately_account_wide pins the decision and names the
ruling, so the width reads as chosen to whoever finds it next.
Two pre-existing negative assertions were silently defanged by the rename --
they matched hive-wanted.beta and $KV.hive-wanted.alpha, strings nothing
produces any more, and kept passing. Both now match current names.
swarm-controller resolves the store per hive per call instead of caching one in
a OnceCell: there is no single handle that serves N buckets, and declarations
change on operator action rather than per tick.
CHECKPOINT, NOT A PROPOSAL. mara paused the plan ("i dont quite understand
the plan") before any behaviour-changing edit; this commit exists so the work
survives a container stop, not because it is ready.
wanted.rs: BUCKET -> BUCKET_PREFIX + bucket(hive) producing hive-wanted-<hive>;
open_or_create/open_read_only take the hive. lib.rs: Error::CreateBucket.bucket
becomes String, since a per-hive name is built at runtime.
Nothing else is touched, so no caller compiles against the new signatures yet
and no deployed behaviour changes. Remaining, if she approves: the two other
CreateBucket sites, swarm-controller's OnceCell (one store -> N), hive-c0re's
pull, policy.rs per-hive streams + the reader roster, swarm-nats.nix.
A KV entry publishes to $KV.<bucket>.<key> and NATS wildcards match whole
.-delimited tokens, so a /-joined {hive}/{agent} key is a single token: the
only expressible write grants are one exact subject per agent (needs a roster
in the auth responder, which Policy::hive_name argues against) or a bucket-wide
wildcard that lets any hive overwrite any other hive's agents.
Joining with a dot puts the hive in its own token, so hive_subjects can grant
$KV.agent-status.<hive>.* — every agent of one hive and nothing else, the same
containment hive-status already has.
The grant lands with the bucket-open pair (STREAM.INFO + STREAM.CREATE):
open_or_create resolves the bucket before it writes, so alone the publish
subject is unreachable and the sweep fails one step later instead.
Verified the client accepts a dotted key rather than assuming it: async-nats
0.50.0 VALID_KEY_RE is \A[-/_=.a-zA-Z0-9]+\z and is_valid_key rejects only
empty / leading / trailing dot; the subject is prefix + key verbatim.
The hive-side loop landed without anything to converge to: nothing wrote
`$KV.hive-wanted.<hive>`, so in production only the "no key" branch ran.
This is the writer.
`WantedWriter` mirrors `StatusReader` — that module reads what hives report,
this one writes what they are told, so it holds a client rather than a bucket
handle and resolves the store on first use. It shares the status reader's
connection: the controller has exactly one by design, and a second connect
would double the auth-callout traffic and give the two paths independent
reconnect state.
The value under a hive's key is the map of every agent on that hive, so a
plain `put` of a single-agent change would drop a concurrent change to a
different agent, with only one revision of history to not recover from.
Writes are read-modify-write against the entry revision, and only
`WrongLastRevision` / `AlreadyExists` count as a lost race — every other
error returns immediately rather than spinning the retry loop and then
blaming a concurrent writer that never existed.
`apply` is split out and tested because it holds the invariant: declaring
one agent preserves the rest, and a current value that will not decode is an
error rather than a fresh start. Overwriting a document nobody can read
discards every other agent's declaration.
Two routes, no swarmctl verb and no jobq node: `create_agent` needs a graph
because it is multi-step, and one CAS'd write is not.
`build_app` is extracted from `main` in the same change because `main` sat at
exactly the `too_many_lines` limit, so adding an endpoint tripped a lint
about the startup sequence. The route list is the part that grows.
mara's call on the PR: "dont make the enum open, we will just add
entries later". The catch-all variant is gone, and with it the per-agent
inert path.
What changes is where version skew lands, not whether it is handled. An
unrecognised value used to be one agent this hive left alone; it is now a
decode failure for the whole declaration, so a hive running older code
converges *nothing* rather than obeying the agents it happened to
understand. That fails closed instead of dangerous, and it is the right
trade when both ends ship together — which is what "add entries later"
assumes.
The test moved with the property rather than being rewritten in place:
`an_unknown_state_fails_the_whole_declaration` lives in
swarm-queue-client, where the decode is, with a valid entry beside it as
the control. `hive-c0re` keeps a coverage check that every state this
build knows produces an action somewhere — asserting inertness there
would be asserting something the type system no longer lets me build.
The deploy event is a nudge with no second path: core NATS is
at-most-once, so a hive that was down when the controller published
simply never learns that an agent is meant to exist here. This adds the
repair path — one boot-time DAG node that reads this hive's own key in
the `hive-wanted` bucket and converges the agents it names.
Two semantics settled on the issue thread, and both are places where a
plausible implementation is the wrong one:
- **Absence is not a deletion order.** No bucket, no key, or an agent
the value does not name all mean the controller has said nothing.
Swarm-side lifecycle does not yet cover agents that predate it, so
"converge to exactly this set" would tear down every agent the swarm
has not adopted. `plan` only ever inspects the agents a declaration
names.
- **An unrecognised state is inert.** `AgentState` is an open enum: a
value this build cannot read deserialises into `Unrecognised` and is
left alone. A closed enum would force "not `Up`" onto a state like
`paused`, so a controller that learned a new value would take agents
down on every hive not yet updated.
Divergence is measured against the hive's **stored power intent**, not
the container's observed running state — an agent that is down while its
intent says `Up` is already the boot reconcile's work, and a loop reading
`is_running` would insert a start DAG behind that reconcile's back on
every boot. A hive that already agrees with its declaration queues
nothing at all.
`queue_first_deploy` is extracted from the deploy-event path rather than
open-coded here, for the power-intent seed: without it `first_deploy`'s
tail `Reconcile` seeds `Wanted` from a container that exists but has not
started yet, which locks the agent to `Offline` on its first reconcile.
The read is authorised as-is: `store.get` takes async-nats' direct-get
arm (the KV bucket is created with `allow_direct`), which is exactly the
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject
`swarm-nats-auth` grants a hive. The fallback subject is not granted, and
a refused NATS request surfaces as a timeout rather than an error.
Nothing writes the bucket yet — the controller-side writer is the other
half of #3124, so this does not close it.
The controller declares each hive's wanted agent set; the hive reads its own
key. Deliberately not a mirror of `status`, which the module documents as a
table: authored by the controller rather than the hive, DECLARED rather than
observed, and — the row that decides how it must be built — unrecoverable if
the store is lost, where status regenerates because every hive republishes
what it is.
Two open functions rather than one shared `open_or_create`. `status` shares its
constructor because either end may legitimately arrive first on a fresh swarm;
here the writer is single and known, so the hive gets a read-only open
returning `Option` and holds no grant to create the bucket. Its absence is the
ordinary pre-publication state, not an error a hive could fix.
Absence is also not a deletion order, which the module says at the place an
implementer will meet it: swarm-side lifecycle does not yet cover agents that
predate it, so a hive finding no key has learned nothing about what it runs —
converging to an empty set would tear those down.
No consumer yet. This is the half that is invariant under the scope semantics
and cadence still being decided.
Per mara on the PR: *"split by hive. its not a security thing, just so
hives dont get messages they dont care about."*
She agreed with the finding and still wanted the split, which is the
part worth recording. I measured that a per-hive subject gives no
confidentiality — `sub` is unrestricted, so a hive that wanted another's
messages could subscribe to them — and concluded it bought nothing.
"Nothing" is a claim over every axis and I had checked one. The axis I
never priced: every hive in the swarm being woken by every other hive's
deploys.
So `deploy_subject(hive)` replaces the single literal, and the payload
drops `hive` to carry only the agent — the subject names the hive, and
two places stating one fact are free to disagree. The hive subscribes to
its own subject and no longer filters.
The grant is a wildcard rather than a subject per hive because the
responder has no roster: it cannot enumerate hives, and a grant that had
to track one would be a second place to get the list wrong — the same
argument `hive_name`'s doc makes about admission.
The negative test gets stronger rather than merely adapted. Splitting
the family makes "another hive's subject" and "its own" separate strings
for the first time, so it now asserts a hive reaches neither, nor the
wildcard.
The subject and its payload live in `swarm-queue-client` for the reason
the knowledge event's already does: three crates have to agree on the
string, and the one that agrees hardest — the auth-callout responder,
which decides whether the publish is permitted at all — speaks neither
`jetstream` nor `kv`.
Swarm-wide rather than a `$SWARM.deploy.<hive>` family. That family
would look like isolation and provide none: this responder scopes
publish only, leaving `sub` unrestricted, so a hive could subscribe to
another's subject as easily as to its own. Until `sub` is scoped the
split costs a wider grant and buys nothing, so the addressing goes in
the payload and each hive filters on its own name.
Unlike the knowledge event the message is addressed, so it carries a
payload — a trigger, never the config. The hive already tracks the
agent's config repo; desired state on the wire would make this a second
source of truth for something git owns, and a hive that missed a
message would be wrong rather than late.
Both test arms mirrored from the knowledge event. The negative one
matters more here: a forged knowledge event makes a hive re-read a
repo, a forged deploy event makes it rebuild and restart a named agent.
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.
The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.
status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.
That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.
The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.
Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.
The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
The controller could not emit an event at all: a reader's grant is
`reader_subjects()`, which is `$JS.API.*` only, so a publish to any event
subject would be refused — and a NATS refusal reaches the client as a
timeout, so the visible symptom would have been a hive that never hears
about a change, with nothing in any log naming a permission.
Adds `swarm_queue_client::events`, following `status::BUCKET`: three
crates must agree on these strings (the controller publishes, a hive
subscribes, the callout responder decides whether the publish is
permitted), and a literal repeated across crates is an agreement nothing
checks. The responder speaks neither jetstream nor kv, so the module is
unconditional and carries no NATS types, exactly as the bucket name is.
The grant takes the wildcard form from the same function the publisher
calls, so the two cannot drift; a separate wildcard constant would have
re-created the disagreement this module exists to prevent.
Tests pin that a reader gets the subject and that a hive does NOT — a
hive able to publish here could tell a neighbour the knowledge repo
changed when it had not, which is an unauthenticated write into someone
else's control path. That one asserts on the subject root rather than a
rendered subject, so a future event leaf fails it too instead of passing
because the test only knew about `knowledge`.
Both assertions mutation-tested: removing the grant fails the reader
test, granting a hive the subject fails the denial test, each on its own
assertion line, and the unmutated tree is green.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
The new regression test constructed `reqwest::Client::new()`, which
panics in the nix build sandbox: with no system CA store,
`ClientBuilder::build()` reaches `rustls_platform_verifier::Verifier::new()`
and fails, and `new()` is `build().expect(..)`. The test passed locally
because a devshell has `/etc/ssl/certs`, and failed in CI.
Disabling certificate verification takes the branch that installs a
no-op verifier and never consults the platform store, so the client
builds anywhere. That is sound in this test and nowhere else: nothing is
sent, the request is built and its bytes are inspected. The helper says
so at the point someone would otherwise object to it.
Found by reading reqwest's `ClientBuilder::build()` rather than trying
builder flags: the first repro attempt — pointing `SSL_CERT_FILE` and
`SSL_CERT_DIR` at nothing — did not reproduce, so any fix verified
against it would have been verified against nothing.
Every access-token request the swarm queue client has ever made was
refused. It sent the client id and secret as form fields
(`client_secret_post`); authelia's client registration allows only
`client_secret_basic`, so the identity provider rejected the request
before looking at the credentials at all.
What made it survive so long is the shape of the failure. The refusals
tripped authelia's rate limiter, whose penalty grows faster than this
client's retry interval — 56s, then 296s, then 535s, against a retry
every 60s — so the limiter never drained and a 429 came back before the
credentials were evaluated. The line naming the real cause appeared
roughly once an hour, inside a continuous storm of a different error,
and the storm read as the problem.
Both other callers in this workspace that present client credentials
already use Basic. RFC 6749 says clients SHOULD, authelia's registration
default says so, and a secret in a header is one fewer place for a proxy
to log it.
The test asserts the shape of the request rather than a server's reply:
Authorization is Basic, the body carries the grant type, and neither the
secret nor the client id appears in the body. It fails on the previous
code with no identity provider, no deployment and no network — which is
what this needed and did not have.
The auth callback minted on every invocation, and async-nats runs it per
connection ATTEMPT. Its default reconnect delay backs off exponentially
and then clamps at four seconds, permanently. So a queue that could not
connect asked authelia for a token every four seconds indefinitely, and
authelia answered 429 Too Many Requests.
That rate limit then keeps itself alive: it outlives whatever first
broke the connection, and its log volume buries the original cause.
Observed in production tonight, once a redeploy fixed the TLS trust
problem that had been hiding it.
The callback now caches the token and re-mints only within two minutes
of expiry, which preserves the property that put the mint there — a
reconnect must never present a token that expired since it was minted —
while decoupling mint rate from retry rate. The reconnect cap moves to
one minute, exponential from half a second, so a brief outage still
recovers promptly and a persistent one stops hammering the identity
provider.
The module doc and the retry comment both argued for the old shape, so
both are corrected rather than left explaining why the bug was right.
The queue client built a bare reqwest::Client, so it trusted only the
platform roots. Against a swarm whose authelia is signed by the swarm
CA that is fatal: minting a token dies with 'invalid peer certificate:
UnknownIssuer', inside the auth callback, on a four-second retry loop,
with the queue never connecting.
The anchor was never missing. hive-tls.nix assembles trust-bundle.pem
and already hands it to hive-c0re as HIVE_TLS_CA_PATH; nothing pointed
the queue client at it.
QueueConfig gains an optional ca_file from <prefix>_OIDC_CA_FILE, read
outside the all-or-none tuple on purpose: a CA path with no queue is
meaningless rather than half-configured, and requiring it would break a
swarm fronted by a public certificate in order to fix one that is not.
add_root_certificate extends the default roots rather than replacing
them, so both deployments work.
A bad path fails loudly instead of falling back to the platform roots.
An operator who names a CA file wants that anchor; a silent fallback
turns their typo into UnknownIssuer five layers away.
hive-tls.nix names the bundle for both clients, beside the line that
already does it for hive-c0re, rather than having each consumer
re-derive the path.
An unqualified `[`open_or_create`]` in the module-level doc does not
resolve once the `kv` feature is on, which is the only configuration
where the module is compiled at all — so `docs-rustdoc` failed in CI
while a default-feature `cargo doc` passed locally. Measured both ways:
kv off documents clean, kv on errors `no item named open_or_create in
scope`.
Qualifying the path fixes it without widening any visibility, which is
the rule that check exists to protect.
Finishes the anyhow removal for the parts this branch adds: the status
bucket's open-or-create and the connected-client precondition. Two
variants, one of them behind the `kv` feature because the error type it
wraps does not exist without it — the error enum respects the same gate
the module does.
NotConnected is deliberately distinct from Connect: one is a connect that
was attempted and refused, the other is a request made before any
connection exists. The first is a deployment problem and the second is a
caller-ordering one, which is the whole reason a caller wants an enum
rather than a string.
The controller's `store` now returns the queue client's error rather than
an anyhow one: `OnceCell::get_or_try_init` takes its error type from the
closure, so widening there would mean converting inside the closure for
no gain. `view` `?`s it and anyhow converts at that boundary — the
library keeps a typed error, the binary keeps anyhow, and no call site
pays for the split.
An unconnected client does not fail a JetStream request, it hangs on it:
`retry_on_initial_connect` hands back a client before it is usable, and a
request made in that window waits (measured: still going at 15s against a
queue that refuses the credential). The controller guarded its read path
against that inline. Every consumer of the queue needs the same guard, so
it is not one daemon's to keep.
It matters more off a request path than on one. A hung request inside a
periodic task never reaches its `select!`, so the shutdown branch becomes
unreachable and the task cannot be stopped at all — where a request path
merely times a poll out.
The test is `!= Connected`, never `== Disconnected`: a client that has
never connected sits in `Pending`, so the `Disconnected` form passes it
straight through to the hang it was written to prevent — which is exactly
the boot-order case the guard exists for. Not feature-gated;
`connection_state()` is core async-nats.
The bucket has two ends in two crates: a hive writes its own key, the
controller reads every key. `swarm-controller` declared the name as a
private const with a doc comment arguing that "reader and writer must
name the same bucket" — an argument the writer, in another crate, could
not obey.
The name is the mild half. Both ends do get-or-create, because either may
come up first on a fresh swarm and neither can assume the other has run.
Two `Config`s that drift means whichever end created the bucket wins and
the other's `get_key_value` succeeds against a bucket it did not ask for:
no error, no log, just a retention policy nobody chose. Sharing the
constructor gives that race one outcome.
Behind a default-off `kv` feature, so the crate's other consumer — the
auth-callout responder, which speaks the connect and nothing else — still
pulls neither `jetstream` nor `kv`. That was the actual reason the
feature was excluded when this crate was extracted; the flag preserves
it. The surface is deliberately narrow: one bucket's name and creation
config, not a general KV facade.
Review catch: `anyhow::Error`'s Display special-cases `f.alternate()` to
walk the source chain; thiserror's derive does not, so `{e:#}` and `{e}`
render identically for the new error type. Every call site that held an
`anyhow::Error`, formatted it with `{:#}`, and now holds this crate's
error kept compiling, kept looking right, and silently dropped the cause.
`chain()` was written for exactly this and then left private, applied only
to the auth callback I happened to be editing. Its own doc comment argues
that dropping the source chain is wrong, which made it the one thing in
the PR that should not have had a scope of one.
The controller's "swarm queue unreachable" warning is the site this fixes
here; the stacked PR fixes the two boot-warning banners, which matter more
still — one-shot, no retry, and they leak until restart.
Operator ruling: libs should not use anyhow. The queue connect was moved
here verbatim from swarm-controller, which is a binary, so it arrived
still wearing a binary's error handling — the move changed what the code
is without changing how it reports.
Callers get variants they can match on, split by what an operator does
about them: a half-configured environment is a deployment bug, a refused
token is an identity-provider config problem, an unreachable queue is a
network one. The binaries that consume this keep anyhow and `?` converts,
so nothing downstream is more verbose for it. Same split hive-claude uses.
One thing anyhow was doing unpaid: the auth callback hands async-nats a
plain string, and a Display that stops at the top message drops the cause
— the half that says why the mint failed. `chain()` walks the source
chain, which is what `{:#}` was doing before.
A hive publishing its own status needs the same connect the controller
already has - mint an authelia token, present it at CONNECT for the
callout responder, let async-nats re-run the callback per attempt. Only
the use differs: the controller reads, a hive writes.
Copying it would put credential handling in two places, and a
token-refresh fix would then have to be found twice. That is the same
reasoning that already put hive-sock-client in its own crate rather than
in each daemon that speaks to a unix socket.
`from_env` takes a prefix rather than hardcoding SWARM_CONTROLLER_*: the
variables belong to the consuming unit, since a NixOS module sets them
alongside its other options. What is shared is the RULE - all four
together or none at all - not the spelling. The half-set case gains a
test, because it is the case the rule exists for and it previously had
none.
No jetstream/kv feature on the crate: it ends at a connected client, and
what a consumer does with it should be visible in that consumer's own
Cargo.toml.
Behaviour-preserving, and proven that way rather than by inspection: the
full behavioural gate (real nats-server, credential rotation, mutation)
is 20/0 unchanged, and the controller's own tests still pass.
