The doc comment on PROXY_HEADER_BLOCK attributed the variable to
`recommendedProxySettings = true`. It does not: nixpkgs' nginx module
emits `map $http_upgrade $connection_upgrade` into commonHttpConfig
outside that conditional, after four other independently-gated blocks.
The distinction is load-bearing rather than pedantic, because several
hyperhive vhosts set `recommendedProxySettings = false` per location
(hive-forge's auth_request endpoint, swarm-ui, swarm-victorialogs,
swarm-victoriametrics). Under the old comment, moving the per-agent
include near any of them reads as breaking the websocket headers, and
the fix someone would reach for is a second, redundant map in our own
http config.
Verified against the nixpkgs module rather than inferred: the
recommendedProxySettings block opens and closes ten lines before the
map, and `git grep connection_upgrade -- nix/` returns nothing, so
nothing on our side defines it either.
`check-doc-refs.sh` resolves the PATH half of a `docs/x.md::Section`
pointer and stops there. The section half rots the same way, and more
quietly: the file still exists, so every path-shaped check stays green
while the pointer names nothing.
Eighteen sites, five distinct pointers, each retargeted at a heading
verified to exist rather than at the nearest plausible one:
docs/web-ui.md::Container row
-> docs/web-ui/dashboard.md::Container row
docs/web-ui.md::Shared terminal pane
-> docs/web-ui/shape.md::Shared terminal pane
Both sections moved out when docs/web-ui.md became a two-heading
index. The path still resolves, which is exactly why nothing
caught them.
approvals.md::Helper events to the manager
-> approvals.md::Helper events to the submitting agent
Renamed with the manager special-casing removal; the pointer kept
the old vocabulary.
approvals.md::Migration from the pre-tag
-> approvals.md::Startup migrations (older hosts)
Same content, including the HIVE_SKIP_META_MIGRATION kill switch
the citing comment names.
agent-hierarchy.md::Current state
-> ::Where the tree lives (topology.rs, container_view.rs)
-> ::Reparenting (topology.rs's set_parent, host-sock)
Split by what each site actually asks for rather than repointed
uniformly: two want the format and the source-of-truth rule, two
want the reparenting validation.
Three known-dead pointers are deliberately left alone:
* `docs/integrations/forge.md::Sources` sits on a line PR #3927
rewrites; fixing it here would conflict for no gain.
* `docs/web-ui/shape.md::One unified channel` names real text that is
bold inline rather than a heading — which of those counts as
resolvable is the open question on #3922.
* `persistence.md::Harness state files` should point at a heading
whose own text contains backticks, and the backticked-pointer form
cannot nest them. That is a limit of the convention, not a typo.
Comments only; no behaviour change. Refs #3922.
Comments cite nix modules, scripts and crate source files constantly,
and nothing evaluates a comment — so when a file moves, the reference
rots silently and `nix flake check` stays green. A reader following one
finds nothing and cannot tell whether the file was renamed, deleted, or
never existed.
Seven such references, each repointed at the file that actually holds
the thing the sentence is about rather than at the directory the old
name became:
hive-c0re/src/agent_config/limits.rs hive-agent/src/mcp.rs
-> hive-agent-mcp/src/mcp/mod.rs
hive-agent-mcp/src/mcp/mod.rs hive-c0re/src/limits.rs
-> hive-c0re/src/agent_config/limits.rs
(and the module path in the doc
comment above it, which was stale
in the same way)
hive-c0re/src/forge/mod.rs hive-c0re/src/knowledge.rs
-> hive-c0re/src/workers/knowledge.rs
nix/host-modules/hive-c0re/options.nix hive-c0re/src/hive_stats.rs
-> hive-c0re/src/stats/hive_stats.rs
nix/packages/default.nix nix/host-modules/hive-c0re.nix
-> .../hive-c0re/options.nix
nix/agent-modules/network.nix nix/host-modules/hive-gateway.nix
-> .../hive-gateway/dnsmasq.nix
frontend/README.md nix/modules/frontend.nix
-> nix/packages/frontend.nix
The two `limits.rs` comments are a matched pair: each names the other's
old path, so the "keep in sync" instruction they exist to carry pointed
both ways at nothing.
Where a flat module became a directory the target is the file that
declares the named thing, not `default.nix` by reflex — the
`preBuildAgentTemplates` option is declared in `options.nix`, and the
DHCP pool that sentence is about lives in `dnsmasq.nix`.
Comments only; no behaviour change. Refs #3923, which is about whether a
gate should cover this class at all — that question is unanswered and
this does not close it.
Making `AgentState` `Copy` in the preceding commit is a change to every
consumer of the type, not to the crate that declares it. `hive-c0re` grew its
own consumer while this branch was in review, and under `-D pedantic` a
one-byte enum taken by reference is `trivially_copy_pass_by_ref` and a
`.clone()` on it is `clone_on_copy`. Neither crate is wrong alone; the merge
is.
`decide` now takes the state by value and the call sites drop the `&`. No
behaviour change — the function only matches on the value.
Per mara's go-ahead on hyperhive#3902 ("getting started is good, but
terminal rendering does not go in there i think"):
Moved 21 top-level docs/*.md files into 7 new topic subdirectories
(existing web-ui/, turn-loop/, swarm/, tools/, crates/ untouched):
getting-started/ setup.md
agent-lifecycle/ agent-hierarchy.md, approvals.md, persistence.md
trust-boundary/ boundary.md, security.md
integrations/ forge.md, matrix.md, github.md, knowledge.md
networking/ gateway.md, network.md, snapshot-store.md
scheduler/ jobq.md, coordinator.md, ci.md, observability.md
process/ conventions.md, gotchas.md, pr-review-gate.md
web-ui/ terminal-rendering.md (moved into the EXISTING dir,
per mara's correction to the original getting-started
guess -- it's UI implementation detail, not onboarding)
The physical layout now matches docs/README.md's own topical headers,
which already amounted to this taxonomy -- see the scoping comment on
the issue for the two findings that motivated this (a genuine
duplication between CLAUDE.md's old "Reading paths" list and
docs/README.md's grouped one, since drifted out of sync with each
other; and the flat layout not matching the grouping we already had).
Fixed every cross-reference this moved across the whole repo (~120
files: docs/ internal links at every depth, Rust doc comments, nix
module option docs, crate READMEs) -- verified two ways: a grep sweep
confirming zero remaining references to any old path, and a script
that resolves every markdown link in docs/**/*.md + CLAUDE.md +
README.md against the filesystem and reports anything that doesn't
exist (zero broken links).
Collapsed CLAUDE.md's "Reading paths" section (the duplicate) down to
a pointer at docs/README.md, now the single index. Rewrote
docs/README.md itself to use the new subdirectory paths and added the
one doc it was missing that CLAUDE.md's old copy had (pr-review-gate.md).
Classified all 22 docs/*.md files first via a haiku subagent (mara's
suggestion) on two axes -- proposed grouping and operator-vs-
implementation focus -- before finalizing the taxonomy; spot-checked
the report and found internal inconsistencies (its classification
table disagreed with its own summary section for a few files), so this
taxonomy is my original proposal + the one correction mara gave
directly, not a blind application of the subagent's table. The
operator-focus data it gathered is still useful for a follow-up
content pass (docs skewing 'mixed' rather than pure operator-facing),
not addressed in this PR -- structure only.
nix fmt clean, both pre-push lints clean.
The boot sweep's error arm substituted `Wanted::from_running(running)`,
which is the one value for which `reconcile_action` returns `Noop` — both
ways. An agent whose `agent_power` row could not be read therefore could
never enter `drifted`, so on a fresh rev marker a corrupt row produced one
`warn!` per boot and no other signal, indefinitely.
Classify the unreadable case as its own outcome instead: the agent gets a
boot `Reconcile`, whose `get_or_seed` fails as a per-agent node — a surface
the dashboard already renders — and the failure stops at that one agent.
The classification moved into `boot_action`, a pure function, because the
loop had no tests at all. The first of the five asserts the unreadable case
across every (fresh × running) combination, which is exactly the matrix the
fabricated value made unreachable.
`PowerStore::get` ended `Ok(row.and_then(|s| Wanted::parse(&s)))`, so a
row holding an unparseable value collapsed into the same `Ok(None)` as
no row at all. Its doc comment stated the narrower case as the whole one
("`None` when the agent has no row yet"), which is what kept the gap
invisible.
The damage is not that a caller reads the wrong value. `get_or_seed`
takes `None` to mean "never seeded" and writes
`Wanted::from_running(running)` — so a corrupted row silently
**overwrote itself** with whatever the container happened to be doing at
that moment, which is precisely the inference an authoritative intent
store exists to prevent. Its own comment says the DB is authoritative
once seeded.
The error arm was already in the signature; this makes it reachable.
Every caller already handles `Result`: the two `get_or_seed` paths
propagate, so a corrupt row now fails that agent's boot reconcile
instead of erasing itself, and `workers::wanted` already skips an agent
whose intent could not be read.
The second test is the one that matters — it asserts the row still holds
its original bytes afterwards, because the overwrite, not the read, was
the bug. Both tests carry a valid and an absent row alongside, so
neither can pass by breaking `get` for everything.
Found by argus reviewing #3891, which reads this store but does not
write it.
mara's call on the PR: "dont make the enum open, we will just add
entries later". The catch-all variant is gone, and with it the per-agent
inert path.
What changes is where version skew lands, not whether it is handled. An
unrecognised value used to be one agent this hive left alone; it is now a
decode failure for the whole declaration, so a hive running older code
converges *nothing* rather than obeying the agents it happened to
understand. That fails closed instead of dangerous, and it is the right
trade when both ends ship together — which is what "add entries later"
assumes.
The test moved with the property rather than being rewritten in place:
`an_unknown_state_fails_the_whole_declaration` lives in
swarm-queue-client, where the decode is, with a valid entry beside it as
the control. `hive-c0re` keeps a coverage check that every state this
build knows produces an action somewhere — asserting inertness there
would be asserting something the type system no longer lets me build.
The deploy event is a nudge with no second path: core NATS is
at-most-once, so a hive that was down when the controller published
simply never learns that an agent is meant to exist here. This adds the
repair path — one boot-time DAG node that reads this hive's own key in
the `hive-wanted` bucket and converges the agents it names.
Two semantics settled on the issue thread, and both are places where a
plausible implementation is the wrong one:
- **Absence is not a deletion order.** No bucket, no key, or an agent
the value does not name all mean the controller has said nothing.
Swarm-side lifecycle does not yet cover agents that predate it, so
"converge to exactly this set" would tear down every agent the swarm
has not adopted. `plan` only ever inspects the agents a declaration
names.
- **An unrecognised state is inert.** `AgentState` is an open enum: a
value this build cannot read deserialises into `Unrecognised` and is
left alone. A closed enum would force "not `Up`" onto a state like
`paused`, so a controller that learned a new value would take agents
down on every hive not yet updated.
Divergence is measured against the hive's **stored power intent**, not
the container's observed running state — an agent that is down while its
intent says `Up` is already the boot reconcile's work, and a loop reading
`is_running` would insert a start DAG behind that reconcile's back on
every boot. A hive that already agrees with its declaration queues
nothing at all.
`queue_first_deploy` is extracted from the deploy-event path rather than
open-coded here, for the power-intent seed: without it `first_deploy`'s
tail `Reconcile` seeds `Wanted` from a container that exists but has not
started yet, which locks the agent to `Offline` on its first reconcile.
The read is authorised as-is: `store.get` takes async-nats' direct-get
arm (the KV bucket is created with `allow_direct`), which is exactly the
`$JS.API.DIRECT.GET.KV_hive-wanted.$KV.hive-wanted.<hive>` subject
`swarm-nats-auth` grants a hive. The fallback subject is not granted, and
a refused NATS request surfaces as a timeout rather than an error.
Nothing writes the bucket yet — the controller-side writer is the other
half of #3124, so this does not close it.
The swarm writes agent-configs/<agent> when it creates an agent, before
any hive is told to deploy it. setup_proposed authored a second copy of
those same bytes locally, so an agent's initial config had two sources
of truth, each unaware of the other and free to disagree. It now clones
that repo and falls back to the template only when there is nothing
there to take.
Preferred-source rather than a new-path-only variant because
provision_container is the Provision node for the swarm deploy and the
approval flow both, and cannot tell them apart. The approval flow
creates agent-configs/<agent> only after the first spawn
(forge_after_first_spawn), so it finds nothing and lands on the
template: the fallback becomes unreachable when hive-level create is
removed, rather than becoming something someone has to find and delete.
clone, not the neighbouring init+fetch. A failed fetch leaves an empty
.git behind, and that .git is exactly the byte setup_proposed reads to
decide whether seeding is still needed, so the fallback would have seen
a seeded repo. git removes a directory it created when a clone fails.
--branch main also makes an empty repo fail cleanly instead of cloning
to an unborn HEAD that would look seeded.
Per mara on the PR: the issue is about a *new* agent, there is no
approval because the operator clicked create at swarm level, and most of
what a hive does on create is already done by the controller.
`spawn_nodes` splits out of `spawn` the way `rebuild_nodes` already
splits out of `rebuild`: two callers want the same four nodes and
disagree only about what closes them. `first_deploy` is that subgraph
with no approval tail, and the absence is the point — that tail exists
because an operator used to approve the spawn at the hive, and asking
again after they clicked create at swarm level asks the same person the
same question twice.
The handler's predicate is "does a container exist", not "is one
running". `agents_for_meta_listing` is `nixos-container list`, so a
stopped agent still counts. `Coordinator::list_agents` looks like the
right check and is the registered-MCP-socket set — a stopped agent is
absent from it, and this would then try to create over an existing
container.
Enumeration failure drops the request rather than guessing: without the
list this cannot tell first deploy from rebuild, and guessing "new" is
the destructive direction.
Still missing, and the reason this is not the whole change: the hive
seeds its own config repo with `git init` instead of cloning the one the
controller already created.
Per mara on the PR: *"split by hive. its not a security thing, just so
hives dont get messages they dont care about."*
She agreed with the finding and still wanted the split, which is the
part worth recording. I measured that a per-hive subject gives no
confidentiality — `sub` is unrestricted, so a hive that wanted another's
messages could subscribe to them — and concluded it bought nothing.
"Nothing" is a claim over every axis and I had checked one. The axis I
never priced: every hive in the swarm being woken by every other hive's
deploys.
So `deploy_subject(hive)` replaces the single literal, and the payload
drops `hive` to carry only the agent — the subject names the hive, and
two places stating one fact are free to disagree. The hive subscribes to
its own subject and no longer filters.
The grant is a wildcard rather than a subject per hive because the
responder has no roster: it cannot enumerate hives, and a grant that had
to track one would be a second place to get the list wrong — the same
argument `hive_name`'s doc makes about admission.
The negative test gets stronger rather than merely adapted. Splitting
the family makes "another hive's subject" and "its own" separate strings
for the first time, so it now asserts a hive reaches neither, nor the
wildcard.
The second subject on the connection this task already drains. Every
hive receives every message — that is what a swarm-wide subject buys —
so not being the addressee is the ordinary case and logs at `debug`. A
payload that will not decode is not: both ends share one type, so a
decode failure means they disagree about it.
The rebuild is the same insert the operator's own `rebuild` verb makes,
relock and all. "Deploy this agent" already means something here, and a
swarm-triggered rebuild that quietly did something narrower would be a
second definition of the word.
⚠️ The at-most-once argument in this function's docs does NOT transfer
to the new subject, and the docs now say so. A missed knowledge event is
repaired by the pull this daemon does at startup regardless; a missed
deploy event has no second path — nothing else would ever tell this hive
to build that agent. Closing that is the hive-side reconcile loop the
issue's other half calls for; until it exists this is a nudge with no
safety net. Not papered over with `JetStream` here: durability on one
subject would look like a fix while the desired state still lived only
in a message.
Swept the surrounding prose rather than only the lines I touched. Two
sections had gone quietly false: the summary said this listens on "the
knowledge-event subject" and named one event, and a whole section
argued "there is no payload, and that is deliberate" — true of the
knowledge event and the opposite of true for a deploy request, which is
addressed and has nowhere but the body to say so.
`services.hyperhive.tls.{stateDir,caValidityDays,leafValidityDays}` sat at
the top of `services.hyperhive`, which is meant to be everything about
hyperhive rather than the settings of one hive. Where the hive CA lives,
how long it lasts and how long the leaves it signs last are decisions of
the host holding the key — `deploy.*`, by the same rule as the switches
that moved before them.
`hive-controller` is hive-c0re's new name (mara on the issue), so the
knobs hang off the daemon that owns the CA rather than off a bare `tls`
at the root. mkRenamedOptionModule entries carry existing configs.
⚠️ Unlike the two switch renames, these names are NOT unique, so this was
swept by ALIAS BINDING rather than by identifier: hive-tls.nix alone holds
two options spelled `stateDir` — its own `cfg.stateDir` and the swarm CA's
`swarmCaCfg.stateDir`, four sites that must not move. Nine files bind an
alias to this config; the rename followed those bindings.
Two sites were invisible to the obvious check, and an unanchored sweep for
`hyperhive\.tls\b` is what found them: the option declaration (`= {` after
the path, so no trailing `.` or `;`) and the alias convention documented in
a comment in lib/hive-ca-trust.nix.
Also renamed the `<tls.stateDir>` shorthand in four docs and two Rust doc
comments, anchored on its delimiters — the new path contains the old one
as a substring, so an unanchored replace would have doubled the prefix.
Last of the swarm.* -> deploy.* moves for the enable-shaped options.
Whether this host also runs the CI runner is a per-machine decision, so
it was sitting in the namespace that has to be identical swarm-wide.
Renames all five options, not just enable: name, concurrency, labels and
package all describe the runner THIS host would run, and leaving them
behind would keep the original defect for four more options. One rename
entry each, because ci is a plain attrset rather than a submodule type,
so there is no parent path to rename in one go.
deploy.forgejo is the name deploy.nix's own header already specifies,
including this exact case -- "ci (does this host run the runner too) is
exactly that shape". It is the only entry with no enable: the forge is
not optional, it deploys with hyperhive itself, so running the runner is
the only deployment decision it has.
Also fixes hive-c0re/src/forge/ci_runner.rs, whose doc comment named
services.hyperhive.forge.ci.enable -- missing the swarm. segment, so it
had never resolved to a real option.
swarm.matrix.enable was the last toggle still spelled swarm.*, which
is the namespace every host in the swarm agrees on - and "does this
host run the homeserver" is exactly what differs between them. It
moves to deploy.matrix.enable with the rest; the genuinely swarm-wide
matrix settings stay where they are.
mkRenamedOptionModule keeps existing configs evaluating with one
warning naming both paths.
The sweep for readers had to be unanchored and cross-language: three
of them were outside nix, including a hive-c0re error message telling
the operator to set the old name. A dotted grep also cannot see the
rename module's own list form, so that was checked separately.
Nine of the 38 .nix tokens mentioned anywhere in *.rs did not name
anything that exists. Twelve mentions, five distinct targets:
hive-c0re.nix, hive-gateway.nix and hive-forge.nix are all directories
now; nix/modules/ is not a directory we have; hive-forge-tools.nix was
a bash script the binary replaced and is gone.
Where the reference is load-bearing it is corrected rather than
deleted, because the reference is the point: a comment saying a
constant must match a nix literal is only useful if you can open the
file it names. Where the module member was unambiguous the path now
names it exactly.
paths.rs's STATE_ROOT marker was the worst of them: it claimed the
value came from services.hyperhive.c0re.statePath, in hive-c0re.nix.
Neither exists. The option is not declared anywhere and the file is a
directory, so a "must match" contract pointed at two things that
cannot be opened. /var/lib/hyperhive is hardcoded on both sides, which
is what the comment now says.
hive-forge-tools.nix keeps no replacement: naming a file that was
deliberately deleted helps nobody, and "replaces a prior bash script"
is complete without it.
Measured before and after with the same command: 9 unresolved of 38
before, 4 of 35 after. The remaining four are an example path in a doc
comment, an upstream nixpkgs path, and two from one synthetic test
fixture.
Six diagnostics and eight comments explained where an environment
variable comes from by naming the module file that sets it. The reader
of a panic is an operator, who does not edit that file; the reader of
the comment is already in the code. Neither can act on a filename, and
a filename in prose is checked by nothing.
Each one now names what the reader can act on: the option where there
is one (services.hyperhive.network.bridgeName, .bridgeIp,
.bridgePrefixLength, c0re.assets, c0re.orgAvatarPng), otherwise the
unit the process is expected to run under. Two of the six diagnostics
had a real option available; the other four have none, so they name
the unit rather than an invented path.
Four of them pointed at hive-c0re.nix, which is a directory. Two of
those spelled it nix/modules/hive-c0re.nix — a directory that does not
exist either, and one of the two was wrapped across a line break, so
no grep for the whole path could have found it. That is the argument
for the change: prose naming a file rots silently because nothing
compiles it.
Scope here is the four files around those diagnostics. The same
pattern still matches elsewhere in the tree; that sweep follows.
The network-isolation doc comments carried prose docs/network.md
already owns, and three of them named `harness-base.nix` — a file
that does not exist. The `hyperhive-isolated-dns` oneshot lives in
nix/agent-modules/network.nix, which the doc gets right.
That is #3749's argument reproducing itself: the same fact written
in two places goes stale in the copy nobody reads. Linking removes
the class, not just the instance — a link cannot name a nonexistent
file without the doc noticing first.
Trap and measurement comments stay put, per the issue's scope: the
load-bearing HOST_ADDRESS default-route note, the unquoted
$EXTRA_NSPAWN_FLAGS expansion, and the "isolation is the only mode"
invariants are facts about this code, not about the subsystem.
set_nspawn_flags said PRIVATE_NETWORK was forced 0 for web-UI
reachability; it is now unconditionally 1 and the UI is reached over the
bridge. write_bridge_dns_marker said it writes or removes the marker;
the removal path went with the host-netns case.
Neither was reachable by the sweeps that found the earlier ones: this
prose describes the *value in words* ('forced 0'), not the code's
literal PRIVATE_NETWORK=0, and neither names the env var. Found by
reading the doc comment of every function whose body this branch
changed.
Per mara on #3725: the on/off toggle is removed, and required env vars
unset lead to a crash. HIVE_NETWORK_ISOLATION is gone from
hive-network.nix -- it was the toggle.
Validation happens once at daemon startup rather than per container.
The variables are process-global, so a bad value breaks every container
rather than one: failing at boot gives a single diagnostic naming the
bad value, and cannot reach a state where some containers were
configured before it was noticed.
Option<NetworkIsolation> collapses to NetworkIsolation through the wire
type, client and helper, which deletes the branch instead of leaving it
unreachable. serde(default) is dropped on that field deliberately: a
request omitting isolation is now rejected rather than defaulting to a
container sharing the host's network namespace.
What this replaces was a silent security downgrade. Of the four ways
into the old fallback, two logged nothing at all -- a container came up
without isolation and the journal agreed it was fine.
Doc comments that still described the removed branch are updated
(argus's note on #3723 scoped that to this issue). The hive-priv one is
a minimal edit inside the block #3723 rewrites; de-splicing is that
PR's job.
One list, in nix/reserved-names.nix, handed to everything that needs it
as HIVE_RESERVED_NAMES. Keeping it current becomes a config change
rather than a rebuild, and hive names and agent names -- one namespace
going forward -- are checked against the same file: swarm-otel.nix's
hand-written reservedOwners is gone.
Whitespace-separated rather than JSON, deliberately, unlike the
structured env vars beside it. Every entry is an Ident ([a-z0-9-]), so
whitespace cannot occur inside a name and the encoding is provably
lossless; JSON would mean either a parser dependency in a crate whose
purpose is to have none, or a copy of the parse in every consumer.
An UNSET variable is not "nothing is reserved". Both creation sites log
an error and return a warning saying the check did not run, so a
misconfigured deployment says so instead of silently accepting every
name. A blank value folds into unset: nix always renders a non-empty
list, so present-but-empty is a rendering fault, not a declaration.
Two guards whose subject moved out of their own file now assert their
own case is still in it, because a guard that can be retired by an edit
elsewhere is not a guard:
- swarm-otel.nix asserts reserved-names.nix still contains its
swarmTierName.
- hive-sh4re's sentinel drift test PANICS when the variable is missing
rather than skipping -- a drift test that quietly does nothing still
reports green. checks.nix and devshell.nix both export it so CI and a
local cargo test agree. Verified as a pair: with the variable set, 8
tests pass; with it unset, exactly the 4 drift tests fail and the
unrelated ones still pass.
The reserved-name check landed only in `swarm-controller::create_agent`.
That daemon is opt-in and off on most hives, so the ordinary per-hive
flow -- `request_init_config` -> `handle_request_init_config` ->
`require_new_child` -> `submit_init_config` -- stayed exactly as
unguarded as before: an agent named `operator`, `forge` or `system` was
still creatable through the path every hive uses, with no warning.
Caught in review by argus. The issue named `create_agent` as the
existing shape to copy, so the shape got copied and the question of
which OTHER sites create an agent never got asked -- an issue naming one
call site is describing an exemplar, not an inventory.
Same treatment as the other path: warn, do not refuse.
The warning needs somewhere to go. `Response` had `Ok` (carries nothing)
and `Err` (refuses), so a check that warns had no way to reach its
caller. Adds `Response::OkWarn { warnings }` -- additive, every existing
`Ok` site is untouched -- rendered by `format_ack` *after* the success
line rather than instead of it: the approval really was queued, and a
warning shown as a failure invites a retry that queues a second one.
Mutation-verified: dropping the warnings and unconditionally appending a
marker each turn a different test red.
act_runner records the --instance URL it was registered with and reads the
forge address from .runner and nowhere else: the nix option reaches only
`register`, and upstream re-registers on a changed token or labels, never on
a changed URL. Our own precond short-circuits on .runner existing, and
runner_valid asks whether the runner id still exists -- which after a rename
it does. So changing swarm.forge.domain left the runner dialling a name its
container's derived extraHosts no longer resolves, with every check green.
The symptom is CI going quiet rather than anything failing.
Compare the persisted address against the configured base as part of the
same early return. A disagreement mints a fresh registration token, which
changes the token hash upstream already keys on, so upstream removes .runner
and re-registers against the current --instance. This module never writes or
deletes that file; the deletion stays with the script that owns registration.
The comparison is host-only and lenient on purpose. Both sides render from
one nix expression -- http://${swarm.forge.domain}, reaching the runner as
instances.hive.url and this daemon as HIVE_FORGE_URL -- so they cannot drift,
while scheme, port and trailing slash are exactly the cosmetic differences
that would otherwise re-register on every boot. Unreadable, missing or
unparseable input keeps the existing credentials, matching runner_valid's
treatment of a transport error: only a positive disagreement counts.
Every hive-labelled series in the store also carries an agent label, so a
hive is only ever visible as the sum of its agents — and a hive whose c0re
has stopped is indistinguishable from one that simply hosts none.
Adds three instruments to the exporter hive-c0re already runs, each a
projection of a value the process computes anyway: process.uptime (the
semconv name — the spec defines it as a double gauge in seconds, which is
exactly this instrument), hyperhive.hive.degraded, and
hyperhive.hive.warnings split by level. None carries an agent attribute;
that absence is what makes them selectable as hive-scoped.
The health pair reads warnings::readiness() rather than deriving its own
verdict, and degraded ships as a series instead of being left for a
dashboard query to compute from warnings{level="crit"} — either would put
the "what counts as unhealthy" rule in a second place that disagrees
silently the first time a degrading condition is added.
Review call: 46 lines of documentation around a single constant, part of
it already stale. The worst paragraph explained why the earlier per-hive
shape had been justified wrongly — history of a design that never
shipped, written into the file within an hour of that design being
dropped. A file is not a changelog; why it was wrong belongs in the PR.
The constant moves to lib.rs beside the status bucket name, keeping only
the rationale that stays true: three crates must agree on the string, and
the one that agrees hardest speaks neither jetstream nor kv, which is why
it cannot sit behind a feature gate.
status earns a module of its own because it holds a bucket name AND the
functions that open it. This held a constant.
Review call: the event was addressed per hive — `$SWARM.events.<hive>.knowledge`,
published in a loop over the roster, granted through a wildcard. It does not
need to be. The payload is empty and the event means the same thing to every
hive, so one publish to one subject delivers exactly what N publishes to N
subjects did, and core NATS already fans out to whoever is subscribed. A hive
that was down misses it either way and reconciles on its next periodic pull.
That deletes rather than reshuffles: the roster loop, the wildcard, and the
shared subject-building function whose entire purpose was keeping the grant and
the publish from drifting apart. With one literal there is nothing to disagree
about.
The per-hive shape was justified by the callout policy's rule that an extra
subject must contain the hive name. That rule governs `extra_hive_subjects` —
what a HIVE may publish. This subject lives in the controller's reader grant,
which the rule does not constrain, so a real rule was carried across into a
decision it had no authority over.
Knowledge becomes its own category rather than a leaf under a general event
namespace, since a namespace shaped for events that do not exist yet is a
decision made before there is anything to decide from. The empty config-PR match
arm goes with it: an arm with no body claims this is where the deploy path is
handled, and it is not.
The deny test stays and matters more, not less: with one shared subject a forged
event would reach the whole swarm where a per-hive one reached a single hive.
The test's doc named knowledge::ensure_webhook as one of two hive-side
reapers. That function is gone; its replacement matches the full URL and
cannot touch another hive's hook.
The assertion arm stays. The hazard is not what this repository's source
says, it is what is deployed — a hive still running the previous version
reaps by suffix until it is upgraded, so the arm guards the transition
rather than a current code path. Recorded when to drop it.
A webhook has exactly one target URL, so every hive registering one
against the shared internal/knowledge repository was last-writer-wins
rather than idempotent: all but the most recent silently stopped
receiving deliveries. The swarm controller holds the single registration
and now addresses an event to each hive over the queue instead.
This is a migration, not a deletion. Not registering any more fixes
nothing on a hive that has already run — the hook it created persists on
the forge, so the contention would survive on exactly the deployments
that have it while fresh installs looked fixed. The hive that created a
hook removes it.
It removes only its OWN, matched on the full URL rather than the
/webhook/knowledge suffix. A hook with that suffix and a different base
belongs to another hive, possibly one not yet upgraded, and deleting it
would break that hive's knowledge sync until it caught up. Reaping a
neighbour's registration is the behaviour being removed here; doing it
while fixing it would only invert the direction.
The predecessor did reap by suffix, to clear loopback hooks left by an
older single-hive layout. That was safe when a hive was alone on its
forge and is not safe now. The hive-side registrars also acted as reapers
of hooks under their own path, which is why the swarm hook lives under
/webhook/forge/; removing this registrar removes that reaper too.
Intended, and stated because no reviewer would infer it from the diff.
The receive endpoint goes with it. A live HMAC-verified
/webhook/knowledge that nothing can legitimately reach would tell the
next reader that this is how a hive learns about knowledge changes.
Docs move in the same commit: docs/swarm/README.md said two hooks exist
per swarm-wide repo and neither should be deleted, which is now true for
agent-configs and wrong for internal/knowledge — a half-correct
description being worse than an uncorrected one.
A hive learned the knowledge repository had changed only by registering
its own forge webhook. This subscribes to the per-hive subject the
controller now publishes on and calls the pull this daemon already runs
at boot.
Shares the hive's ONE queue connection rather than opening a second: a
second connect would double the auth-callout traffic against authelia and
give the two paths independent reconnect state, so one could be serving
while the other was still down. Same argument as the controller side.
No payload is read, because there is none to read — the webhook handler
this replaces took two fields from Forgejo and used neither, then ran
`git pull`, which re-derives everything from the repository.
At-most-once, and that is not a regression: a webhook delivery to a hive
that is down is lost identically today, and the boot pull covers it.
JetStream would require this end to publish to
`$JS.API.CONSUMER.CREATE.<stream>`, which the callout policy does not
grant, so durability would cost grants on both sides to remove a failure
the boot pull already handles.
⚠️ Documented at the call site rather than left implicit: a refused
subscription is indistinguishable from a quiet one, because NATS reports
authorization violations asynchronously on the connection. If hives stop
hearing events, the server log is the thing that knows.
futures-util comes from the workspace (same version swarm-controller
already uses), not a new dependency version.
Every one of these carried a "SAFETY: single-threaded mutation of an env
var no other test asserts on" comment. Each claim was true of the module
and false of the process: env vars are one process-global and every
#[test] in this crate lands in the same binary at default parallelism,
so "no other test" has to mean no other test in the BINARY — and two of
them set HIVE_FORGE_URL outright.
They now take the crate lock #3483 added, including the two whose
variables nothing else touches: what makes a variable safe is that every
mutator routes through one lock, not that today's set happens not to
collide.
The doc comment on push_forwarded_var_options said the render-level
tests race each other; they serialise now, so it says that. It is
deliberately not an intra-doc link — test_env is #[cfg(test)], so
rustdoc cannot resolve it and -D rustdoc::broken-intra-doc-links fails
the docs check.
