mara (#4170): swarm-ui's wanted-state dropdown could only ever declare
up/offline/destroy, with no way to swarm-declare the existing hive-local
turn-loop pause (`hivectl agent pause|resume`).
`AgentState::Paused` is not a fifth peer of Up/Offline/Destroyed on the
power axis this enum otherwise answers — it's Up plus an orthogonal
turn-loop pause. `hive-c0re`'s `workers::wanted` reconcile loop now
decides the two axes independently (`decide` for power, the new
`decide_pause` for the marker), so a stopped agent declared Paused
converges with both a Start and a Pause in the same pass.
Known, deliberate limitation: a Paused declaration on an agent this
hive has never deployed only reaches Deploy this pass — writing the
pause marker into a harness dir that may not exist yet was judged not
worth the risk, so it converges on the next pass once the agent is
present instead.
swarm-ui's WantedMenu gains a fourth "paused" option (warning-tone
badge). No separate "resume" entry — selecting "up" from a paused row
already clears the marker via the same decide_pause path.
Pause/resume marker writes go through one shared
Coordinator::set_paused_by_name helper, used by both the interactive
dashboard pause/resume handlers and this reconcile loop, instead of
each duplicating the parse-name/write-marker/track-rescan shape.
swarm-ui's "offline" and "paused" confirm dialogs share one
confirmTarget state and one ConfirmDialog instead of two near-identical
copies.
Closes#4170
The wanted-state read was a boot-time DAG node, so a swarm-level change sat
unapplied until the next restart. This watches the hive's own bucket and
converges on each update.
It does not replace the boot read: a watch hears only what is published while
it is listening, so a hive that was down still learns the current declaration
from `pull`. The watch is the fast path, `pull` stays the repair path.
Rides the connection swarm-status already opens, as a third consumer — a
second connect would double the auth-callout traffic and give the two paths
independent reconnect state, which is the reason the deploy-event drain is
spawned there too.
A delete is not a deletion order. `carries_a_declaration` is pure and tested
so that rule is enforced rather than asserted: converging on a withdrawn key
would tear down exactly the agents "absence is not a deletion order" protects.
`docs-rustdoc` fails on the `//!` header: `unresolved link to bucket`,
for an item declared in that very module. The module's docs are merged
from two fragments — the `///` on `pub mod wanted;` in `lib.rs` and the
`//!` inside the file — and the merged doc does not resolve a bare item
name against the module. Every other link in the file was already
fully-qualified (`crate::status`, `crate::status::BUCKET`), so this one
was the outlier rather than the convention.
Evidence, same command before and after: `cargo doc -p swarm-queue-client
--no-deps --document-private-items` went from 1 occurrence of the error to
0, while the unrelated diagnostic in that narrower build stayed at 1 —
the surviving one is the control that the lint is still armed. The
CI-shaped run (`--workspace`, which is what unifies features) exits 0 and
documents 27 crates including this one.
Refs #4006.
A hive reads its own declaration today and that scopes cleanly: DIRECT.GET
carries the key in the subject, so the grant can name it. A *watch* cannot be
scoped that way — a consumer's filter travels in the request payload, so
$JS.API.CONSUMER.CREATE.<stream> grants the whole stream. With every hive in
one bucket, letting a hive watch its own declaration would let it read every
other hive's.
One bucket per hive (hive-wanted-<hive>) makes the stream a hive may hold
exactly as wide as what it is allowed to see, which is what #4006's live-watch
needs. That watch is a separate change; this only moves the boundary.
mara's calls, both on #4006: one stream per hive rather than teaching the auth
responder a hive roster, and a wildcard for the controller — "its okay if
swarm controller can theoretically override hive". A bucket name is a single
subject token with no prefix matching, so no wildcard narrower than * covers N
per-hive buckets; the controller's grant is account-wide by consequence, and
documented as chosen rather than left to look accidental.
The reader arm of #4005's key-layout guard asserted the opposite of that
ruling, so it is replaced rather than deleted: the hive arm survives as
no_hive_may_write_another_role_s_agent_status (with a positive control), and
the_readers_grant_is_deliberately_account_wide pins the decision and names the
ruling, so the width reads as chosen to whoever finds it next.
Two pre-existing negative assertions were silently defanged by the rename --
they matched hive-wanted.beta and $KV.hive-wanted.alpha, strings nothing
produces any more, and kept passing. Both now match current names.
swarm-controller resolves the store per hive per call instead of caching one in
a OnceCell: there is no single handle that serves N buckets, and declarations
change on operator action rather than per tick.
CHECKPOINT, NOT A PROPOSAL. mara paused the plan ("i dont quite understand
the plan") before any behaviour-changing edit; this commit exists so the work
survives a container stop, not because it is ready.
wanted.rs: BUCKET -> BUCKET_PREFIX + bucket(hive) producing hive-wanted-<hive>;
open_or_create/open_read_only take the hive. lib.rs: Error::CreateBucket.bucket
becomes String, since a per-hive name is built at runtime.
Nothing else is touched, so no caller compiles against the new signatures yet
and no deployed behaviour changes. Remaining, if she approves: the two other
CreateBucket sites, swarm-controller's OnceCell (one store -> N), hive-c0re's
pull, policy.rs per-hive streams + the reader roster, swarm-nats.nix.
The hive-side loop landed without anything to converge to: nothing wrote
`$KV.hive-wanted.<hive>`, so in production only the "no key" branch ran.
This is the writer.
`WantedWriter` mirrors `StatusReader` — that module reads what hives report,
this one writes what they are told, so it holds a client rather than a bucket
handle and resolves the store on first use. It shares the status reader's
connection: the controller has exactly one by design, and a second connect
would double the auth-callout traffic and give the two paths independent
reconnect state.
The value under a hive's key is the map of every agent on that hive, so a
plain `put` of a single-agent change would drop a concurrent change to a
different agent, with only one revision of history to not recover from.
Writes are read-modify-write against the entry revision, and only
`WrongLastRevision` / `AlreadyExists` count as a lost race — every other
error returns immediately rather than spinning the retry loop and then
blaming a concurrent writer that never existed.
`apply` is split out and tested because it holds the invariant: declaring
one agent preserves the rest, and a current value that will not decode is an
error rather than a fresh start. Overwriting a document nobody can read
discards every other agent's declaration.
Two routes, no swarmctl verb and no jobq node: `create_agent` needs a graph
because it is multi-step, and one CAS'd write is not.
`build_app` is extracted from `main` in the same change because `main` sat at
exactly the `too_many_lines` limit, so adding an endpoint tripped a lint
about the startup sequence. The route list is the part that grows.
mara's call on the PR: "dont make the enum open, we will just add
entries later". The catch-all variant is gone, and with it the per-agent
inert path.
What changes is where version skew lands, not whether it is handled. An
unrecognised value used to be one agent this hive left alone; it is now a
decode failure for the whole declaration, so a hive running older code
converges *nothing* rather than obeying the agents it happened to
understand. That fails closed instead of dangerous, and it is the right
trade when both ends ship together — which is what "add entries later"
assumes.
The test moved with the property rather than being rewritten in place:
`an_unknown_state_fails_the_whole_declaration` lives in
swarm-queue-client, where the decode is, with a valid entry beside it as
the control. `hive-c0re` keeps a coverage check that every state this
build knows produces an action somewhere — asserting inertness there
would be asserting something the type system no longer lets me build.
The deploy event is a nudge with no second path: core NATS is
at-most-once, so a hive that was down when the controller published
simply never learns that an agent is meant to exist here. This adds the
repair path — one boot-time DAG node that reads this hive's own key in
the `hive-wanted` bucket and converges the agents it names.
Two semantics settled on the issue thread, and both are places where a
plausible implementation is the wrong one:
- **Absence is not a deletion order.** No bucket, no key, or an agent
the value does not name all mean the controller has said nothing.
Swarm-side lifecycle does not yet cover agents that predate it, so
"converge to exactly this set" would tear down every agent the swarm
has not adopted. `plan` only ever inspects the agents a declaration
names.
- **An unrecognised state is inert.** `AgentState` is an open enum: a
value this build cannot read deserialises into `Unrecognised` and is
left alone. A closed enum would force "not `Up`" onto a state like
`paused`, so a controller that learned a new value would take agents
down on every hive not yet updated.
Divergence is measured against the hive's **stored power intent**, not
the container's observed running state — an agent that is down while its
intent says `Up` is already the boot reconcile's work, and a loop reading
`is_running` would insert a start DAG behind that reconcile's back on
every boot. A hive that already agrees with its declaration queues
nothing at all.
`queue_first_deploy` is extracted from the deploy-event path rather than
open-coded here, for the power-intent seed: without it `first_deploy`'s
tail `Reconcile` seeds `Wanted` from a container that exists but has not
started yet, which locks the agent to `Offline` on its first reconcile.
The read is authorised as-is: `store.get` takes async-nats' direct-get
arm (the KV bucket is created with `allow_direct`), which is exactly the
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject
`swarm-nats-auth` grants a hive. The fallback subject is not granted, and
a refused NATS request surfaces as a timeout rather than an error.
Nothing writes the bucket yet — the controller-side writer is the other
half of #3124, so this does not close it.
The controller declares each hive's wanted agent set; the hive reads its own
key. Deliberately not a mirror of `status`, which the module documents as a
table: authored by the controller rather than the hive, DECLARED rather than
observed, and — the row that decides how it must be built — unrecoverable if
the store is lost, where status regenerates because every hive republishes
what it is.
Two open functions rather than one shared `open_or_create`. `status` shares its
constructor because either end may legitimately arrive first on a fresh swarm;
here the writer is single and known, so the hive gets a read-only open
returning `Option` and holds no grant to create the bucket. Its absence is the
ordinary pre-publication state, not an error a hive could fix.
Absence is also not a deletion order, which the module says at the place an
implementer will meet it: swarm-side lifecycle does not yet cover agents that
predate it, so a hive finding no key has learned nothing about what it runs —
converging to an empty set would tear those down.
No consumer yet. This is the half that is invariant under the scope semantics
and cadence still being decided.
