A hive's grants for its own `KV_hive-wanted-<hive>` were `STREAM.INFO` +
`DIRECT.GET`, which cover the boot-time read and nothing after it. The
convergence path now opens a KV watch on that bucket, and a watch is a
consumer, so the broker denies it — and the client's `watch()` ends in
`.ok()`, so the denial becomes `None` and the path silently never fires.
The comment four lines above the grant list already argues for this: the
per-hive bucket split exists "so that a watch can be granted without
widening the read". The design was taken and the grant was never written.
Both subject forms, matching the hive-status and agent-status blocks that
grant both for the same documented reason — an ephemeral consumer's
subject carries no name, and `>` never matches zero tokens.
Refs #4006.
mara on #4015: "not merging code without callers", and on the same PR
"see issue, we decided what the first thing should be". #3726 decided it:
the controller writes a token to the store and tells the hive; the hive
reads it back and writes /agents/<agent>/state/matrix-token-<account> at
0600, where matrix.nix's existing systemd.paths glob re-fires the daemon.
So this is the hive half of that, and the library's first caller.
The notice names a credential and never carries one, and deploy_subject's
own doc is why: the auth-callout responder scopes publish and leaves sub
unrestricted, so a hive that wanted another's messages could subscribe to
them. A secret in that payload would be readable swarm-wide. The value is
read from the store under the reading hive's own certificate, where the
store's policy is what actually scopes it.
Two boundaries guard the two addresses, and they are not the same check.
`path::matrix_account` guards the address in the store. `Ident` guards the
address on disk -- `agent_state_dir` takes one, so an unvalidated name off
the queue cannot reach a directory. I had written the first and assumed it
covered both; the compiler refused the `&str` and was right. `token_path`
now takes the newtype so a call site cannot forget.
The write is atomic because the path-watcher fires on the file appearing:
written in place it would be visible while partial, and the daemon would
read a truncated credential exactly once, which is the hardest possible
failure to reproduce. The temp name is dot-prefixed so it cannot match the
`matrix-token*` glob on its way past.
The publish grant is here because without it the failure is invisible.
policy.rs already says why for its siblings: a refused publish reaches the
client as a timeout, so the symptom is a hive that never receives a
credential with nothing in either log naming a permission. Two tests: the
controller may publish, a hive may not -- its own subject included. A
forged notice leaks nothing, but it would make a hive fetch and overwrite
a token file for a name the forger chose.
Refs #3726
A hive reads its own declaration today and that scopes cleanly: DIRECT.GET
carries the key in the subject, so the grant can name it. A *watch* cannot be
scoped that way — a consumer's filter travels in the request payload, so
$JS.API.CONSUMER.CREATE.<stream> grants the whole stream. With every hive in
one bucket, letting a hive watch its own declaration would let it read every
other hive's.
One bucket per hive (hive-wanted-<hive>) makes the stream a hive may hold
exactly as wide as what it is allowed to see, which is what #4006's live-watch
needs. That watch is a separate change; this only moves the boundary.
mara's calls, both on #4006: one stream per hive rather than teaching the auth
responder a hive roster, and a wildcard for the controller — "its okay if
swarm controller can theoretically override hive". A bucket name is a single
subject token with no prefix matching, so no wildcard narrower than * covers N
per-hive buckets; the controller's grant is account-wide by consequence, and
documented as chosen rather than left to look accidental.
The reader arm of #4005's key-layout guard asserted the opposite of that
ruling, so it is replaced rather than deleted: the hive arm survives as
no_hive_may_write_another_role_s_agent_status (with a positive control), and
the_readers_grant_is_deliberately_account_wide pins the decision and names the
ruling, so the width reads as chosen to whoever finds it next.
Two pre-existing negative assertions were silently defanged by the rename --
they matched hive-wanted.beta and $KV.hive-wanted.alpha, strings nothing
produces any more, and kept passing. Both now match current names.
swarm-controller resolves the store per hive per call instead of caching one in
a OnceCell: there is no single handle that serves N buckets, and declarations
change on operator action rather than per tick.
A KV entry publishes to $KV.<bucket>.<key> and NATS wildcards match whole
.-delimited tokens, so a /-joined {hive}/{agent} key is a single token: the
only expressible write grants are one exact subject per agent (needs a roster
in the auth responder, which Policy::hive_name argues against) or a bucket-wide
wildcard that lets any hive overwrite any other hive's agents.
Joining with a dot puts the hive in its own token, so hive_subjects can grant
$KV.agent-status.<hive>.* — every agent of one hive and nothing else, the same
containment hive-status already has.
The grant lands with the bucket-open pair (STREAM.INFO + STREAM.CREATE):
open_or_create resolves the bucket before it writes, so alone the publish
subject is unreachable and the sweep fails one step later instead.
Verified the client accepts a dotted key rather than assuming it: async-nats
0.50.0 VALID_KEY_RE is \A[-/_=.a-zA-Z0-9]+\z and is_valid_key rejects only
empty / leading / trailing dot; the subject is prefix + key verbatim.
The swarm-controller had no grant for the per-agent status bucket, so
resolving it failed and the dashboard answered 503.
Grants the reader the same five subjects the hive-status bucket already
has, for the same measured reasons -- including BOTH `CONSUMER.CREATE`
forms. `keys()` builds an ephemeral consumer whose create subject carries
no consumer name, and `>` matches one or more tokens, never zero, so the
`.>` form alone does not cover it; that omission surfaces as a client
timeout and an operator-visible 503, which is the shape reported here.
Deliberately grants NO `$KV.agent-status.*` subject, to anyone. That is
the write side, and writing is what picks a key layout -- whether agent
status is published by the agent or forwarded by its hive, and under
which key, is still open. A write grant here would answer that by
implication. Reading needs none of it: a KV read is a DIRECT.GET, and the
`.>` form is bucket-wide rather than per-key, so nothing added here
encodes a layout. A test asserts that, for the reader and for a hive,
with a control proving the same matcher fires on a bucket that does have
write grants.
The stream name is derived from the crate constant, like the wanted-state
bucket beside it, so writer and reader cannot disagree about it.
The bucket the previous commit adds had no grants, so the controller
could not create or write it and no hive could read its own key.
Measured against nats-server 2.14.4 rather than extended by analogy,
because the shapes are not symmetric:
- controller: `STREAM.INFO` + `STREAM.CREATE` on `KV_hive-wanted` and
`$KV.hive-wanted.>`. With only today's status grants, `get_key_value`
timed out and the server named the two missing stream subjects.
- hive: `STREAM.INFO` plus **one** direct-get subject carrying its own
key. A KV read is a publish — `store.get` is a request — and the
direct-get subject embeds the key, so the read scopes per hive. By
analogy with `reader_subjects` this would have been `.>`, handing
every hive every hive's wanted set.
Both refusals were verified to fire, not assumed: as `alpha`,
`get beta` was refused naming
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.beta`, and
`put alpha` was refused naming `$KV.hive-wanted.alpha`.
`a_reader_may_list_and_fetch_but_not_write` asserted a reader holds no
`$KV.` subject at all, which held only while status was the sole
bucket. Narrowed to the status bucket — the invariant it defends is
that the controller cannot forge a hive's own report, and the wanted
bucket runs the other way.
Refs #3124
`hive_subjects` granted `$JS.API.STREAM.INFO`/`CREATE` on `hive-notices`
but never the per-hive subject a publish actually targets, so every
`notices::publish` was refused with a permissions violation naming
`hive-notices.<hive>`.
The gap survived because `notices_subjects`'s own doc comment asserted
`extra_hive_subjects` covered it "since that one *is* per-hive". Nothing
in `nix/` has ever passed `--hive-publish-subject` (`git grep` → 0 hits;
control: `--reader-client` is passed at `nix/host-modules/swarm-nats.nix:711`),
so the publish had been refused for as long as the feature existed.
Subject comes from `swarm_queue_client::notices::subject`, the same
constant the publisher formats from, rather than a literal here.
Refs #3859
Per mara on the PR: *"split by hive. its not a security thing, just so
hives dont get messages they dont care about."*
She agreed with the finding and still wanted the split, which is the
part worth recording. I measured that a per-hive subject gives no
confidentiality — `sub` is unrestricted, so a hive that wanted another's
messages could subscribe to them — and concluded it bought nothing.
"Nothing" is a claim over every axis and I had checked one. The axis I
never priced: every hive in the swarm being woken by every other hive's
deploys.
So `deploy_subject(hive)` replaces the single literal, and the payload
drops `hive` to carry only the agent — the subject names the hive, and
two places stating one fact are free to disagree. The hive subscribes to
its own subject and no longer filters.
The grant is a wildcard rather than a subject per hive because the
responder has no roster: it cannot enumerate hives, and a grant that had
to track one would be a second place to get the list wrong — the same
argument `hive_name`'s doc makes about admission.
The negative test gets stronger rather than merely adapted. Splitting
the family makes "another hive's subject" and "its own" separate strings
for the first time, so it now asserts a hive reaches neither, nor the
wildcard.
The subject and its payload live in `swarm-queue-client` for the reason
the knowledge event's already does: three crates have to agree on the
string, and the one that agrees hardest — the auth-callout responder,
which decides whether the publish is permitted at all — speaks neither
`jetstream` nor `kv`.
Swarm-wide rather than a `$SWARM.deploy.<hive>` family. That family
would look like isolation and provide none: this responder scopes
publish only, leaving `sub` unrestricted, so a hive could subscribe to
another's subject as easily as to its own. Until `sub` is scoped the
split costs a wider grant and buys nothing, so the addressing goes in
the payload and each hive filters on its own name.
Unlike the knowledge event the message is addressed, so it carries a
payload — a trigger, never the config. The hive already tracks the
agent's config repo; desired state on the wire would make this a second
source of truth for something git owns, and a hive that missed a
message would be wrong rather than late.
Both test arms mirrored from the knowledge event. The negative one
matters more here: a forged knowledge event makes a hive re-read a
repo, a forged deploy event makes it rebuild and restart a named agent.
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.
The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.
status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.
That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.
The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.
Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.
The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
The controller could not emit an event at all: a reader's grant is
`reader_subjects()`, which is `$JS.API.*` only, so a publish to any event
subject would be refused — and a NATS refusal reaches the client as a
timeout, so the visible symptom would have been a hive that never hears
about a change, with nothing in any log naming a permission.
Adds `swarm_queue_client::events`, following `status::BUCKET`: three
crates must agree on these strings (the controller publishes, a hive
subscribes, the callout responder decides whether the publish is
permitted), and a literal repeated across crates is an agreement nothing
checks. The responder speaks neither jetstream nor kv, so the module is
unconditional and carries no NATS types, exactly as the bucket name is.
The grant takes the wildcard form from the same function the publisher
calls, so the two cannot drift; a separate wildcard constant would have
re-created the disagreement this module exists to prevent.
Tests pin that a reader gets the subject and that a hive does NOT — a
hive able to publish here could tell a neighbour the knowledge repo
changed when it had not, which is an unauthenticated write into someone
else's control path. That one asserts on the subject root rather than a
rendered subject, so a future event leaf fails it too instead of passing
because the test only knew about `knowledge`.
Both assertions mutation-tested: removing the grant fails the reader
test, granting a hive the subject fails the denial test, each on its own
assertion line, and the unmutated tree is green.
`store.keys()` creates an ephemeral ordered consumer, whose create subject
ends at the stream name. `>` matches one or more tokens and never zero, so
the `.>` form alone never covered it: the server refused
`$JS.API.CONSUMER.CREATE.KV_hive-status`, the refusal reached the client as a
timeout, and the operator saw a 503 on the hive status page.
The test asserted only the `.>` form, which reads as covering the bare one, so
the suite stayed green while every list timed out in production. It now names
the bare subject separately and first.
The module doc had grown a 37-line preamble carrying three separate
arguments, which the comment-block lint refuses. Splitting it is the
better fix than raising the limit: the reader who is about to widen a
subject list meets the reason not to at the list, not seven screens up.
Also corrects one claim I had no measurement for. The CREATE note said
creating an existing stream with a different config "is an error rather
than a rewrite" — asserted, not observed. What is observed is narrower
and enough: a hive holding this grant leaves the stream config untouched
and never publishes $JS.API.STREAM.UPDATE at all.
A grant carrying every bucket-specific subject and neither of these
cannot create the bucket at all: the client times out on `$JS.API.INFO`
long before it reaches a subject that was granted, and a NATS denial
reaches the client as a hang rather than an error.
Both were named by the server's own log, not reasoned about.
`$JS.API.INFO` is the account-level JetStream info every client requests
on connect; `$JS.API.STREAM.NAMES` is how a client finds the stream
backing a bucket. The latter lets a client enumerate stream names in the
account, which in an account holding one bucket discloses a name both
ends already share.
Every earlier measurement missed them, because each either granted
`$JS.API.>` wholesale or ran against a bucket the setup had already
created while unscoped. A minimum established against an existing bucket
is not the minimum for making one, so the leave-one-out that trimmed the
reader's set could not have found this — every candidate it tried was
tried in a world where the bucket existed.
Found by running the shipping gate against the real binary. No unit test
could have: the failure is a timeout inside a real server's permission
check.
`--hive-publish-subject` exists to put a second stream inside one hive's
namespace. A template with no `{hive}` in it expands to the same subject
for every hive, so the option whose only purpose is scoping becomes the
way to remove it — silently, and only in the deployment that set it.
`Policy::new` returns a `Result` rather than checking at the call site:
that makes an unscoped policy unconstructible instead of merely
unlikely, the same reason `grant` takes its permissions by value. The
error names the offending template and says what goes wrong with it,
because an operator meets it at boot with no other context.
Also documents what the prefix match does not do. A client id is a hive
here because it starts with the configured prefix, not because it
appears in the roster — the responder runs in a container and cannot see
`swarm.hives`. Passing the roster in would close that and would also be
a second place deciding who may connect as what, which `introspect`'s
docs argue against for the same reason admission lives in one place.
The two intra-doc links to `open_or_create` become plain backticks.
Un-gating the `status` module means its module doc now renders in builds
without the `kv` feature, where the item it linked does not exist.
Every admitted client got the same unrestricted grant, so any hive could
write any other hive's status key. The responder now derives a
permission set from the caller's identity and mints it into the user
JWT.
A hive may publish to its own KV key and the two JetStream subjects
needed to reach it; the controller may list and fetch every key and
write none; anything else is denied outright. Deny is the default
because every other shape fails open, and silently: a client that
matched no rule and kept the old grant would make the policy advisory.
The subject sets are measured rather than reasoned about, and two of
them are counter-intuitive. `$KV.<bucket>.<key>` alone does not let a
client write that key, because the client resolves the bucket first. And
`$JS.API.>` is not "the JetStream permission": it also covers
`$JS.API.STREAM.DELETE`, with which a hive correctly refused on a
neighbour's key can delete the whole bucket and every hive's data with
it. Granting it would have made per-key scoping decorative, so the
subjects are named individually and a test asserts the wildcard does not
come back as a convenience.
Minimality is by removal: each subject was dropped in turn to confirm
the client breaks without it. That is not pedantry — an additive search
had called a set minimal while two of its five subjects were never
needed, which ships an unnecessary grant with a measurement attached
making it look earned.
Both grants include `STREAM.CREATE` on the one named stream, because
`status::open_or_create` is called by both ends: either may arrive first
on a fresh swarm, and without it a new swarm never gets a bucket at all.
`CREATE` is not `UPDATE`, so a second arrival cannot reshape the bucket
the first one made.
`status::BUCKET` moves out from behind the `kv` feature so this
responder can share it. The name is a `&str` with no dependencies and
only `open_or_create` needs JetStream; gating the name forced a third
consumer to choose between a stack it does not use and a copied literal,
and the copied literal is exactly the disagreement that module exists to
prevent.
Only publish is scoped. Subscription permissions are unrestricted and
unmeasured, and the module docs say so rather than implying a property
nothing established.
