hyperhive#3896. The backend for start/stop (Up/Offline wanted-state
declarations) already existed and was merged (#3905's writer, the
PUT /api/hives/{hive}/agents/{agent}/state route) — nothing here was
waiting on Paused/Destroyed, which I'd mistakenly conflated with this
issue in an earlier comment (that's #3803, a different feature).
swarm-controller: merges each row's declared wanted state into
GET /api/agents/status, same shape as the config_pr merge (one read
per distinct hive, not per agent, since a declaration is a hive's
whole agent map).
swarm-ui: AgentsPage gets a "wanted" column — clicking the current-
state badge toggles it (Badge's own chip-plus-control shape, same as
its own header comment's pause/resume example), backed by the PUT
route above. A row with no declaration yet reads its implied current
state off the agent's own last-reported running flag. Stop asks for
confirmation (native window.confirm — no confirm-dialog component
exists in swarm-ui yet); start doesn't.
Pure `nix fmt` output from the commit before this one — no hand edits.
203 files: 52 md, 42 tsx, 32 js, 32 css, 21 ts, 13 html, 8 json, 3 mjs.
Reproduce with `nix develop -c nix fmt` on the parent commit; the result
should be byte-identical to this tree.
None of the 13 `.prettierignore` entries appears here — verified by
intersecting the changed-file list against the ignore file, with a
control proving the intersection finds a match when one exists.
Per mara's review call on this PR: "the view should be filled by a single
backend call." AgentsPage.tsx was doing three fetches (/api/agents,
/api/config-prs, /api/agents/status) and joining them client-side by name.
Moves the config-PR join server-side instead: AgentStatusRow gains a
config_pr field, populated by get_agents_status's handler from
AppState::config_prs after agent_status::AgentStatusReader::view() returns
- not inside that module, which has no forge client and stays that way (see
the field's doc comment for why the handler is the right layer for this
merge, not the reader).
AgentsPage.tsx now does exactly one fetch and no client-side joining at all
- the wire row is the table row. Dropped the separate AgentStatusRow TS
interface (folded into AgentRow, which now mirrors the backend type
field-for-field) and the /api/agents + /api/config-prs fetches entirely;
neither is needed once /api/agents/status already returns every roster
agent with its config PR attached.
ConfigPrStatus gained Deserialize (previously Serialize-only) since
AgentStatusRow derives both and a struct's derive requires every field to
support it.
The hive-side loop landed without anything to converge to: nothing wrote
`$KV.hive-wanted.<hive>`, so in production only the "no key" branch ran.
This is the writer.
`WantedWriter` mirrors `StatusReader` — that module reads what hives report,
this one writes what they are told, so it holds a client rather than a bucket
handle and resolves the store on first use. It shares the status reader's
connection: the controller has exactly one by design, and a second connect
would double the auth-callout traffic and give the two paths independent
reconnect state.
The value under a hive's key is the map of every agent on that hive, so a
plain `put` of a single-agent change would drop a concurrent change to a
different agent, with only one revision of history to not recover from.
Writes are read-modify-write against the entry revision, and only
`WrongLastRevision` / `AlreadyExists` count as a lost race — every other
error returns immediately rather than spinning the retry loop and then
blaming a concurrent writer that never existed.
`apply` is split out and tested because it holds the invariant: declaring
one agent preserves the rest, and a current value that will not decode is an
error rather than a fresh start. Overwriting a document nobody can read
discards every other agent's declaration.
Two routes, no swarmctl verb and no jobq node: `create_agent` needs a graph
because it is multi-step, and one CAS'd write is not.
`build_app` is extracted from `main` in the same change because `main` sat at
exactly the `too_many_lines` limit, so adding an endpoint tripped a lint
about the startup sequence. The route list is the part that grows.
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
Registration ran once at startup and, on failure, deferred to the next
process start. Nothing schedules one, so a controller that lost the boot
race kept running with no hooks registered — and the failure is silent at
both ends, since the forge has nothing to report about a call that never
arrived.
That race is the common case rather than an edge: the controller and the
forge come up together on a rebuild. Measured on a deploy where both
consecutive starts got 502 from the gateway because forgejo was not yet
serving; the forge was healthy two minutes later.
Bounded backoff, not a poll loop — it exists to outlast a slow forge, not
to re-register periodically. Unit ordering would not fix this: the forge
is a remote host in a spread deployment, where no After= can reach it.
Closes#3828
`POST /admin/hooks` reads `is_system_webhook` out of the config map and
defaults it to false, which creates a forgejo *default* webhook — a
template copied into repos created later — instead of a live
instance-wide one. `GET /admin/hooks` returns only hooks with the flag
set, so `list_hook_urls` could never see what the create had just made:
every controller start listed zero hooks and created another default
webhook (12 in 26h), while the instance-wide push observation the scope
exists for never fired at all.
Send the key on the `Instance` arm only, via a per-scope
`extra_create_config()` so repo and org scopes stay unchanged.
Closes#3807
The swarm writes agent-configs/<agent> when it creates an agent, before
any hive is told to deploy it. setup_proposed authored a second copy of
those same bytes locally, so an agent's initial config had two sources
of truth, each unaware of the other and free to disagree. It now clones
that repo and falls back to the template only when there is nothing
there to take.
Preferred-source rather than a new-path-only variant because
provision_container is the Provision node for the swarm deploy and the
approval flow both, and cannot tell them apart. The approval flow
creates agent-configs/<agent> only after the first spawn
(forge_after_first_spawn), so it finds nothing and lands on the
template: the fallback becomes unreachable when hive-level create is
removed, rather than becoming something someone has to find and delete.
clone, not the neighbouring init+fetch. A failed fetch leaves an empty
.git behind, and that .git is exactly the byte setup_proposed reads to
decide whether seeding is still needed, so the fallback would have seen
a seeded repo. git removes a directory it created when a clone fails.
--branch main also makes an empty repo fail cleanly instead of cloning
to an unborn HEAD that would look seeded.
Per mara on the PR: *"split by hive. its not a security thing, just so
hives dont get messages they dont care about."*
She agreed with the finding and still wanted the split, which is the
part worth recording. I measured that a per-hive subject gives no
confidentiality — `sub` is unrestricted, so a hive that wanted another's
messages could subscribe to them — and concluded it bought nothing.
"Nothing" is a claim over every axis and I had checked one. The axis I
never priced: every hive in the swarm being woken by every other hive's
deploys.
So `deploy_subject(hive)` replaces the single literal, and the payload
drops `hive` to carry only the agent — the subject names the hive, and
two places stating one fact are free to disagree. The hive subscribes to
its own subject and no longer filters.
The grant is a wildcard rather than a subject per hive because the
responder has no roster: it cannot enumerate hives, and a grant that had
to track one would be a second place to get the list wrong — the same
argument `hive_name`'s doc makes about admission.
The negative test gets stronger rather than merely adapted. Splitting
the family makes "another hive's subject" and "its own" separate strings
for the first time, so it now asserts a hive reaches neither, nor the
wildcard.
`TriggerDeploy` had no producer — a node kind nothing enqueues is dead
code, and a publisher with no caller proves as little as a check nobody
runs.
It goes last in the creation chain, after `InitAgentConfigRepo` rather
than merely after the repo exists: the hive deploys by reading that
repo, so a deploy asked for any earlier would find nothing to build.
That edge is what makes creating an agent at swarm level actually put it
on a hive instead of leaving a provisioned name nobody runs.
The `hive` it carries is the string this handler already parsed as an
`Ident` and matched against the roster, so the node cannot name a hive
this swarm does not have.
`TriggerDeploy` is the first `SwarmNodeKind` whose effect leaves this
host, so it is also the first to need the queue connection: `WorkerDeps`
grows a `queue` handle, cloned from the status reader whose own doc says
the connection living there is an accident of construction order rather
than a claim that events are a kind of status.
`publish_deploy` publishes and then flushes before reporting `Done`.
`publish` only hands the message to the client's write buffer, so a node
that reported success on that alone would be claiming a delivery it has
no evidence for — the ordering `webhook::announce_knowledge_change`
already documents.
The `WireNode::data` or-pattern did the job it was written for: its
comment says a new variant should fail to compile there rather than
silently render as an agent name, and `TriggerDeploy` is the first node
about an agent *and a hive*. A catch-all would have dropped the hive
from the viewer with nothing to notice it.
`ensure_hook` reaches its create call from two different places — the
list step failed, or it succeeded and matched nothing — and both were
logged at `debug!`. At the level the journal keeps, that made a repeated
registration indistinguishable from a first, correct one, and it is a
repeated registration that is being observed: the instance-scoped hook
logs the create arm on every process start while the repo-scoped one
correctly goes quiet.
The fold that keeps this harmless (`is_already_exists` swallowing a
duplicate create) is an assumption about the forge rather than a
guarantee, so the failed-list arm becomes a `warn!`, and the
matched-nothing arm logs the number of hooks it did see: `listed=0` is a
permission or scope problem, a non-zero count with no match means the
recorded url is not the one being compared.
No behaviour change — this makes the existing behaviour legible.
Two corrections from review, applied forward on this branch rather than
by rewriting it.
`deploy.<service>` was a bare bool, which makes
`deploy.forgejo = { enable; ci; }` unrepresentable -- the nested
CI-runner sub-option this namespace was designed around. Every entry is
now an attrset with an `enable`, so a second per-host deployment
decision becomes an ordinary addition rather than a migration.
`deploy.controller` is now `deploy.swarm-controller`, consistent with
`deploy.swarm-ui`, which was introduced in the same commit.
89 references rewritten across 24 files -- nix, Rust, docs, and the
repo's own CLAUDE.md.
The prefix-anchored sweep missed exactly one, and it was live code:
hive-tls.nix spells it `hyperhiveCfg.deploy.controller` -- the only
`hyperhiveCfg` prefix among 45 references. A suffix grep
(`\.deploy\.<name>`) finds it; a path-anchored one cannot, because the
head of a reference is whatever alias the reading file happens to bind.
One commit rather than two because they are not independent: the UI's
`enable` had the controller's as its literal default, so moving the
controller alone would leave the UI's default naming an option that no
longer exists.
The UI keeps that derivation in its new home — it is a view onto the
controller's state and reaches it over that daemon's unix socket, so the
host running the controller is the host that can serve it.
Three spellings had to move together for the UI, not one: the `default`,
the `defaultText` shown in the options doc, and the description prose
that names the old path in words. A grep for the option path finds the
first two.
The sweep also reached outside nix: `swarm-controller`'s crate README and
its `//!` module doc both named the option, as did this repo's own
CLAUDE.md and four pages under docs/. An option's name is API, and its
documentation lives wherever someone thought to write it down.
One list, in nix/reserved-names.nix, handed to everything that needs it
as HIVE_RESERVED_NAMES. Keeping it current becomes a config change
rather than a rebuild, and hive names and agent names -- one namespace
going forward -- are checked against the same file: swarm-otel.nix's
hand-written reservedOwners is gone.
Whitespace-separated rather than JSON, deliberately, unlike the
structured env vars beside it. Every entry is an Ident ([a-z0-9-]), so
whitespace cannot occur inside a name and the encoding is provably
lossless; JSON would mean either a parser dependency in a crate whose
purpose is to have none, or a copy of the parse in every consumer.
An UNSET variable is not "nothing is reserved". Both creation sites log
an error and return a warning saying the check did not run, so a
misconfigured deployment says so instead of silently accepting every
name. A blank value folds into unset: nix always renders a non-empty
list, so present-but-empty is a rendering fault, not a declaration.
Two guards whose subject moved out of their own file now assert their
own case is still in it, because a guard that can be retired by an edit
elsewhere is not a guard:
- swarm-otel.nix asserts reserved-names.nix still contains its
swarmTierName.
- hive-sh4re's sentinel drift test PANICS when the variable is missing
rather than skipping -- a drift test that quietly does nothing still
reports green. checks.nix and devshell.nix both export it so CI and a
local cargo test agree. Verified as a pair: with the variable set, 8
tests pass; with it unset, exactly the 4 drift tests fail and the
unrelated ones still pass.
An agent's name was checked for shape and never for meaning:
`Ident::parse` is 1-63 chars of [a-z0-9-] and there was no reserved-name
list anywhere in the tree. So an agent could be called `operator`,
`forge` or `todo` -- names the message layer already produces as a
sender -- and a wake from that component became indistinguishable, at the
broker, from a message sent by the agent.
Adds `RESERVED_NAMES` + `is_reserved_name` to `hive-types`, the zero-dep
leaf both `hive-c0re` and `swarm-controller` already depend on, so
neither grows a dependency to use it.
Every entry is a value some component actually produces as a message
`from`/`to`, taken from `hive-sh4re`'s own sentinel constants rather than
guessed: operator, system, reminder, forge, scheduled, todo, compact,
graceful-stop. Two sentinels are deliberately absent -- `<parent>` and
`<children>` are unreachable as agent names because the charset rejects
them, and `ruth` is a real agent, so wanting that name is a name being
*taken*, which the roster answers.
Deliberately not enforced inside `Ident::parse`: parsing runs on every
read of an already-created name, so rejecting there would make existing
agents unreadable rather than un-creatable -- and it would be a refusal,
which is a stronger action than the warning this is used for today.
`create_agent` now warns on both halves -- a reserved name, and a name
that is also a hive in the roster -- and does not refuse. The warnings
ride on `CreateAgentResponse` rather than only the daemon's log, because
the person who can still fix the name in one keystroke is holding the
response, not reading the journal. `skip_serializing_if` keeps the
no-warning JSON byte-identical to before, so this is a non-breaking first
step toward refusing later.
`hive-sh4re` gains a drift test tying its sentinel constants to the list:
two crates that cannot import each other's intent now fail loudly if a
sentinel is added without being reserved. Mutation-verified -- forcing
the predicate false, forcing it true, and dropping a single entry each
turn a different test red.
`EnsureAgentIdentity` answered `AlreadyExists` whether it had added the
agent marker to an existing subject or done nothing at all, and the
controller discarded the answer outright. So the one case worth telling
a human about — a subject that was NOT an agent a moment ago — could not
survive the socket, let alone reach a log.
`Healed` is a variant rather than a field on `AlreadyExists` because a
field is ignorable: adding a variant makes every existing match fail to
compile until its author decides what a heal means. That is the property
the old shape lacked.
The store cannot tell a pre-marker agent identity from a human operator
account created without a group — both are `groups: []`. So this is
either the intended migration or an agent joining a person's live SSO
account, and only the caller has the context to tell them apart.
Gated by state/gate-3549-heal.sh (8 arms + mutation): the mutation
collapses Healed back and reddens the discriminating arm while leaving
the anti-noise arm green. The W' control asserts exactly one warn in the
whole run, so a build that warned on every routine ensure would fail.
A bare String in a crate whose whole suite types validated names as
hive_types::Ident, so a consumer had to re-derive at the boundary what every
sibling field gets checked for free.
Ident is strictly narrower than the store's charset — [a-z0-9-] against the
[A-Za-z0-9._-] the users database allows, because that file holds humans too.
Every agent name is a legal Ident by construction (creation parses one before
the job is queued), so the narrowing costs real agents nothing. What it does
mean is that a human hand-added to the agent group cannot be described as an
agent: the reader omits that row and logs it, rather than failing the whole
roster or quietly shrinking the answer.
The REQUEST keeps its String. That side carries what a caller asked for, and
validating it server-side is what lets a bad name be refused with a message
instead of failing to deserialise.
GET /api/agents, beside the POST that creates one. The identity store is
the roster rather than a source to assemble one from, so this is a read
with nothing to merge or reconcile.
It says nothing about health, deliberately. A roster is the set other views
are complete against — it is what makes "this agent has never reported"
expressible, and that only survives while the declared set and the reported
set stay apart.
The two verbs treat a missing bridge differently and the asymmetry is the
design: POST queues a job that fails loud when claimed, GET has nowhere to
defer to and returns 503. Answering [] there would render a store nobody
could read as a swarm with no agents.
The function's own doc comment still said one subject per hive, four
lines above an inline comment correctly saying one publish for all of
them. Left behind by the single-writer rework.
Swept the rest of the touched files for the old design's vocabulary
rather than fixing only the line argus quoted: every other per-hive
mention is about webhook registration or the callout policy's
extra-subject template, both of which are still per-hive and still
accurate.
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.
The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.
status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.
That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.
The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.
Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.
The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
The test's doc named knowledge::ensure_webhook as one of two hive-side
reapers. That function is gone; its replacement matches the full URL and
cannot touch another hive's hook.
The assertion arm stays. The hazard is not what this repository's source
says, it is what is deployed — a hive still running the previous version
reaps by suffix until it is upgraded, so the arm guards the transition
rather than a current code path. Recorded when to drop it.
The controller verified a knowledge delivery, logged it, and returned OK.
Nothing downstream ever heard about it, so a hive learned the repository
had changed only by registering its own webhook — which is the
last-writer-wins contention this issue is about.
The event carries no payload. The hive-side handler this replaces reads
two fields from Forgejo's push webhook and uses neither — both are
filters — then runs `git pull`, which re-derives everything from the
repository. What crosses the queue is an edge trigger, and fields would
invent a contract nobody reads.
One subject per hive, so the callout policy can express "this hive may
hear its own events" at all; a subject with no hive component is the same
subject for every hive.
`ConfigPr` deliveries are deliberately not forwarded. A hive does not
want to hear that a config PR opened — it wants to be told when to
rebuild from main, which the controller decides after a merge rather than
by relaying this delivery. That is deploy coordination's job, and the
empty arm is there so the omission reads as scoped rather than forgotten.
Fails soft: a missed announcement costs a hive stale knowledge until its
next boot pull, which is the same cost as a webhook delivery to a hive
that was down — what this replaces. A permission failure cannot be
observed at the call site (a NATS authorization violation is reported
asynchronously on the connection, reaching a client as a timeout or as
nothing), so the doc says the flush proves only that the bytes left this
process and points at the server log.
WIP — compiles per an earlier build, but the verifying build/test run was
cut short by a graceful stop. Re-run gate.sh before pushing.
The controller created every agent config repo, its collaborator entry,
its branch protection and its seeded agent.nix/flake.nix in the agents
org. Config repos live in agent-configs, which is where hive-c0re
reconciles, merges and mirrors them — so a repo created in agents is
invisible to all of those, and nothing errors, because both orgs exist
and both accept a repo.
Root cause was a doc comment asserting something false: AGENTS_ORG
claimed to be the same org hive-c0re uses for its config-repo path. It is
not — hive-c0re's agents org is the namespace repos an AGENT ASKS FOR
land in, and its config repos use agent-configs. The whole flow inherited
the wrong premise from that sentence.
The merge gate survives the move: hive-c0re provisions the operators team
in both orgs, with a comment recording that missing the agent-configs
copy once left every config repo unprotected.
