A webhook has exactly one target URL, so every hive registering one
against the shared internal/knowledge repository was last-writer-wins
rather than idempotent: all but the most recent silently stopped
receiving deliveries. The swarm controller holds the single registration
and now addresses an event to each hive over the queue instead.
This is a migration, not a deletion. Not registering any more fixes
nothing on a hive that has already run — the hook it created persists on
the forge, so the contention would survive on exactly the deployments
that have it while fresh installs looked fixed. The hive that created a
hook removes it.
It removes only its OWN, matched on the full URL rather than the
/webhook/knowledge suffix. A hook with that suffix and a different base
belongs to another hive, possibly one not yet upgraded, and deleting it
would break that hive's knowledge sync until it caught up. Reaping a
neighbour's registration is the behaviour being removed here; doing it
while fixing it would only invert the direction.
The predecessor did reap by suffix, to clear loopback hooks left by an
older single-hive layout. That was safe when a hive was alone on its
forge and is not safe now. The hive-side registrars also acted as reapers
of hooks under their own path, which is why the swarm hook lives under
/webhook/forge/; removing this registrar removes that reaper too.
Intended, and stated because no reviewer would infer it from the diff.
The receive endpoint goes with it. A live HMAC-verified
/webhook/knowledge that nothing can legitimately reach would tell the
next reader that this is how a hive learns about knowledge changes.
Docs move in the same commit: docs/swarm/README.md said two hooks exist
per swarm-wide repo and neither should be deleted, which is now true for
agent-configs and wrong for internal/knowledge — a half-correct
description being worse than an uncorrected one.
Three readers, three different questions:
`observability.md` gets the operator's version — what to set (nothing, on an
all-local swarm), what to set elsewhere, what a misconfiguration looks like
(401s and a missing hive), and what turning it off actually costs.
`swarm/secrets.md` extends the existing client-secret row rather than adding
one, because this is a new *reader* of a secret already described. It notes the
one thing that makes this reader unusual: a host unit reads the file in place,
so unlike every other consumer there is no delivery step to get wrong.
`setup.md` gets one bullet under security notes, since a first-run hive gets
the property without doing anything and should know it has it.
Drops swarm.otel.url (a loopback default an operator had to override on a
split host) in favor of swarm.otel.domain -- the same
gateway.localNames + nginx-vhost-through-the-gateway shape every other
swarm service (authelia, grafana, victoriametrics, ui) already uses. The
hive tier's exporter now reaches it as https://<domain> unconditionally,
resolved locally by dnsmasq on a co-located host and over the real network
otherwise, instead of a config knob nobody sets until they hit the silent
drop.
Costs CA trust on the hive tier: otel.nix wires
lib/hive-ca-trust.nix's trustBundle with hostUnit = true on the
opentelemetry-collector host unit, the same flag #3441/#3442 added for
swarm-controller and hive-c0re.
mara, #3125 comment 58363: "go c".
observability.md described a single collector holding the upstream
credential. It also said endpoint and protocol are what agents are
handed; agents get the derived first hop, which has been true since the
collector was introduced.
The swarm tier is documented beside its sibling swarm services rather
than here, and the one line an operator must not miss - swarm.otel.url
on a hive that does not run them - is called out in both places, since
leaving it unset loses telemetry silently.
mara: "pls remove historical wording, only present pls".
The correction was written as a diff against what the docs used to claim
-- "this used to say X", "where this is going", "the intended state for
now". That is a changelog, and a reader arriving cold has to reconstruct
the current truth from it. The reasoning about why the old shape was
wrong belongs in the PR that changed it, not in the file.
Now says what is true: the controller interprets a delivery and emits a
semantic message; receipt is all that is wired today because the
swarm->hive channel does not exist yet.
mara asked for the docs and comments to be fixed in one PR before the
implementation, since a half-corrected description is what confuses a
reader coming back after a context compact.
Three more places said or implied "forward the payload":
- post_webhook_forge's doc ("relays the delivery to every hive ... the
payload is never parsed here")
- its OpenAPI request_body description ("the delivery can be relayed
unmodified") -- this one is published in the spec, so it was the most
externally visible of them
- DeliveryKind::as_str, which called the hook kind the event's routing
key; the message is semantic, and which hook it arrived on is an input
to deriving it rather than the thing sent
docs/swarm/README.md gains the direction next to the interim state, so
"two hooks, one of which only logs" reads as a step rather than a design.
The endpoint landed inert: nothing pointed at it, so the only way to see
it work was to mint an HMAC by hand. Register the two swarm-wide hooks
at startup so a real forge event produces a journal line.
Registered ALONGSIDE the per-hive hooks, not instead of them. Every hive
keeps receiving and acting on its own deliveries; the controller gets a
copy and logs it. Moving the registration is a later step and has to be:
fan-out swarm->hive does not exist yet, so a hook moved now would point
at a receiver that forwards nowhere, silently on both sides.
Deliberately no stale-hook deletion arm, unlike the two per-hive
registrars this otherwise mirrors: theirs delete hooks matching their own
path with a foreign base, and the hives' hooks are not stale.
The route prefix is what keeps this safe. Both hive-side registrars
delete any hook ending in /webhook/knowledge or /webhook/config-pr with a
different base, so a swarm hook under those paths would be deleted by
every hive on every boot. Serving them under /webhook/forge/ avoids it,
and a test pins it -- there is nothing else that can.
SWARM_CONTROLLER_PUBLIC_URL is set only where the swarm vhost is served,
because a hook whose target_url nothing answers is worse than no hook.
Six places asserted the old design as fact, and none of them mention the
change by name -- the class of doc breakage that is found by asking what
a diff made untrue, not by grepping for a feature:
- swarmctl/README.md and swarm-authelia-bridge/README.md both described
their own private canonical store. The bridge's "known limitation"
section described the seam as unsolved; it is what this fixes, so it
becomes what both writers must uphold instead.
- docs/swarm/{sso,ui,secrets}.md described a rendered artifact.
- The repo CLAUDE.md entry for swarmctl said the same.
- docs/tools/swarmctl-cli.md is regenerated (CI diffs it against the
clap tree), picking up the removed --store flag.
Operator-facing where it is read: the hand-editing consequence (values
survive a rewrite, comments do not) is stated in sso.md, where an
operator is being told to edit the file, rather than only in a module doc.
Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
Review: the public keys are not an exception, the paragraph was
incomplete. It is private key material and access tokens that are
file-based so they stay out of the nix store; public material in a
public store is fine.
Fixes the rule itself, which removes the need for the special case the
previous commit added.
secrets.md is the page that answers "who mints this and where does it
go" per topology, and the callout nkeys had no row. Adds one, plus the
`nk` invocations an operator needs on every topology that is not
all-local.
The note exists because the change provokes a question the page would
otherwise answer wrongly: this page opens with "a secret is a path,
never a value", and the queue takes two public keys as values. A reader
who meets those after reading that line concludes the page is stale.
The section carried a "nothing publishes yet" note that is now false, and
said nothing about the one thing an operator has to do.
Written to the reader's question rather than the author's: what to set,
what defaults on an all-in-one host, what has to be carried by hand to a
hive that is not the swarm host, and where to look when a hive goes quiet.
The identity and the publish cadence are stated because they constrain the
staleness threshold an operator picks; the mechanism behind them is not.
Follow-up to the plumbing removal: docs/swarm/README.md gets the
biggest rewrite (drops the whole "Fingerprint format" section, fixes
the runtime-effects list, the WireGuard config example + "what the
mesh does" bullet), docs/gateway.md and hive-gateway/options.nix drop
4 "needs no certFingerprint" mentions, swarm-peers-removed.nix's
migration-warning text no longer tells an upgrading operator to carry
a field over that no longer exists, swarm.nix/swarm-wireguard.nix/
swarm-controller.nix/swarm-controller's main.rs get comment fixes
where they described the now-removed HYPERHIVE_PEERS shape. Also
caught one more stale "peer hives" mention in docs/web-ui/README.md's
SW4RM tab description that the first pass on this issue missed.
Follow-up to the peer-hives removal: docs/web-ui/dashboard.md drops
the whole "P33R H1V3S" section + its stale #peers hash and tab-strip
mentions; docs/swarm/README.md's runtime-effects list and
cross-references point at swarm-ui's roster page instead. Nix comments
in hive-c0re/environment.nix, swarm.nix and swarm-wireguard.nix that
described HYPERHIVE_PEERS/certFingerprint as dashboard-consumed are
updated to say what's actually still true (agent identity only).
Deliberately NOT removed: the `certFingerprint` nix option itself.
It has no remaining Rust consumer, but deleting a documented,
potentially already-set option is a bigger, breaking change than
removing dead display code — flagged in docs/swarm/README.md with a
note instead of silently dropped, so an operator or reviewer can make
that call explicitly rather than have it made for them here.
Per mara on the PR: less implementation detail, more operator focus. The
section argued why the design is shaped as it is - a cache would
disagree with itself, freshness must be derived rather than stored, the
timestamp is the server's - which is reasoning a reader of this page does
not need and cannot act on.
It now answers what an operator actually asks: what each freshness value
means they should do, what to set the threshold to and why that number,
what a 503 tells them, and that nothing publishes yet so an empty board
is expected rather than broken.
The design rationale is not lost, it is just homed where it belongs: the
status module's own doc comment, next to the code it constrains.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
Single 🔗 button in the Shell header (direction 1 of 3 proposed on the
issue) — one fixed element regardless of link count, so it can't
clutter as the list grows, and visible on every route since Shell
wraps all of them. Opens a popover listing whatever GET /api/links
returns (icon + label, opens in a new tab); hidden entirely when the
list is empty, same "don't render a dead affordance" rule the old
dashboard's H0M3 tiles follow for Forge/Matrix.
Docs: docs/swarm/ui.md gets a short section on the feature + the
contribute-your-own-entry idiom.
Verified: npm run build + npm run typecheck clean, headless-chromium
screenshots of both the closed and (temporarily forced open for the
screenshot only) open states, pixel-sampled to confirm the popover is
actually using the dark theme vars and not just looking that way in a
downscaled preview.
Per mara: setup should point at it. The link is placed as a precondition
rather than a see-also -- every step below assumes each credential is
generated where it is read, which is only true all-local.
Per argus: 'with a bound' was missing a word; it now names the actual
120s wait instead of gesturing at one.
Per mara on the CA question: outside all-local this is an ops problem,
and what is missing is documentation rather than machinery.
One row per secret, read against three topologies, because the same
credential is generated in place on one and handed over by an operator on
another. sso.md's table is replaced by a pointer -- two tables listing the
same secrets would drift, and its prose about why a secret is generated
in-container is the half worth keeping there.
States the constraint the whole thing rests on: a hive CA is name-
constrained to the hive domain and a swarm service name is a sibling of
it, so a hive CA cannot issue a swarm-service certificate at all. That is
why placing the swarm root is an operations decision.
The registration half is shared; what each service does with the result
is not, so the differences get a table rather than a second page.
States why matrix reads its secret through LoadCredential and why
/_matrix/ is not a forward-auth surface -- both are conclusions a reader
would otherwise have to re-derive from the module.
`user add` refuses on an existing name, so the `--group` flag it takes at
creation time could not be added afterwards at all: repairing an account
meant hand-editing both users.json and the rendered users.yml as root.
mara, on #3167: "i will not edit those files by hand, we will have the
same issues elsewhere".
The merge rules live in users.rs as a pure function over a UserUpdate, so
they are testable without a command line, a container or a running
authelia — main.rs's arm only loads, applies, publishes and prints.
Removals are strict and everything else is idempotent, which is the one
asymmetry here and is deliberate: a --remove-group naming a group the
user does not have fails, because a revocation that reports success
without revoking is the outcome nobody re-checks; while refusing an
already-satisfied set would make the multi-attribute call this verb
exists for break whenever one of the values was already right.
A command that changes nothing at all still fails — it would otherwise
rewrite both files and restart the SSO provider to no effect.
Passwords are out of scope: regenerating a credential is a different
intent from editing an attribute, and folded together an attribute edit
can invalidate a login by accident.
Extracts publish() from user_add so both verbs share the
render -> store -> users.yml -> restart ordering and the comment that
explains why that order, rather than the second verb copying it.
Per mara on the PR: the guide someone follows on a fresh deploy is a
different page from the one they read while debugging, and only the
second existed.
Also fixes a contradiction the request surfaced: setup.md and
swarm/sso.md have told every operator to create '--group admins' since
the bootstrap step existed, while the new rule required 'operators'.
This is the first rule that CONSUMES a group name, so it takes the one
the guide already creates - inventing a second would have meant every
account made by following the guide silently failing the check it was
supposed to pass, and for mara specifically a migration step that now
may not be needed at all.
setup.md's step 3 says why --group is not decoration; the new step 4
says what decides whether the page opens (the group, and the name
resolving) rather than listing a command, because there is nothing to
run. Steps renumbered, including the matrix block's internal 4a-4e.
New docs/swarm/ui.md (split-page shape, per the docs rule), linked from
the swarm README and added to the gateway's vhost map.
Leads with the step that separates 'protected' from 'locked out':
swarmctl user add <you> --group operators. auth_request asks whether
there is a session; the access_control rule is what makes it mean
operator, and an account created before the rule existed has no groups.
Also records the four wiring sites a swarm service name needs, with the
certificate one called out - serviceDomains is both the sub-CA's
nameConstraints set and the leaf's SANs, and the apex is a sibling of
the other three rather than a parent, so nothing issues for it
implicitly.
`swarm/sso.md` described the empty user store as a resting state — a
provider that is reachable but has nobody in it yet. It isn't. Authelia
validates the store at startup and treats zero users as fatal:
error reading the authentication database: could not validate the
schema: users: non zero value required
so it exits 1, systemd restarts it, and an enabled-but-unbootstrapped
swarm presents as a crash-looping container behind a vhost that is
working correctly. The observed symptom is `502 Bad Gateway`, which
reads as a proxy fault and is not one.
Says so, gives the error text to grep for, and marks the `swarmctl user
add` step as required to finish the install rather than as a first-login
convenience. `gateway.md` gains the same warning next to the vhost,
because that is where someone lands when the 502 is what they can see.
The reason the store ships empty is unchanged and still right: seeding
an account means a credential in a config file. What was wrong was
calling the resulting state harmless.
The vhost half of this change is only useful with an account behind it,
and the provider is generated with an empty user set on purpose. Document
the `swarmctl user add` step rather than automating it: bootstrapping an
IdP non-interactively means a secret arriving from a file, an env var or a
nix expression, all worse than one command typed once.
The gateway and network pages gain the rows they would otherwise be
missing — vhost map, local-dev hosts entry, and the resolver's
authoritative-name list.
The question this answers is "what do I have to configure, and where" —
so the table of secrets is the deliverable and the prose is scaffolding
around it.
The organising idea worth keeping: a secret belongs in-container when
nothing outside that container reads it. Every one of authelia's own
secrets passes that test; the client secret's plaintext fails it, which
is what makes delivery a problem at all rather than a detail.
Names the two ways out (install the sub-CA, or pin back under the hive
domain) and says why it is a warning rather than an assertion — the
module can see what it issues, not what the operator installed.
`security.pki.certificateFiles` is read when the system is BUILT and the
swarm root is deliberately a runtime file (its key must never enter the
world-readable store), so there was nothing build-time to name and the
container trusted no swarm-internal CA. Federation with a peer whose
cert chains to the swarm root did not validate.
The bridge already existed: `lib/hive-ca-trust.nix`, which `hive-ci` and
`hive-forge` both cross. This wires the matrix container to it -- read-only
bind of the trust bundle, container ordered after `hive-tls-ca.service` --
rather than inventing a second mechanism.
Consumption is the per-runtime half, and it is not what it looks like.
tuwunel links no openssl, which makes `SSL_CERT_FILE` appear inapplicable.
It isn't: the outbound client is `reqwest` with the `rustls` feature,
which builds a `rustls_platform_verifier::Verifier`, and because tuwunel
calls `tls_certs_merge` (additive) rather than `tls_certs_only`, the
platform roots stay alongside its compiled-in webpki set. On Linux that
verifier resolves through `rustls-native-certs` to `openssl-probe`, which
reads `SSL_CERT_FILE`. Established by reading tuwunel 1.8.0 and reqwest
0.13.4 source, not inferred from the feature list -- an earlier reading of
mine concluded the opposite from the absence of a `native-roots` feature
name and was wrong.
The variable therefore names a CONCATENATION (system CAs + the bundle),
never the anchor alone: `openssl-probe` uses it *instead of* the default
store, so pointing it at the bundle would drop every public CA and break
federation with the wider matrix network -- trading a small outage for a
much larger one.
`container@hive-matrix` needed `mkMerge`: it already assigned `after` for
the gateway ordering and the helper contributes its own `after`/`requires`,
so two bare assignments would have conflicted rather than combined.
⚠️ Gate honesty: `state/eval-3093.sh` proves the mount, the ordering, the
bundle service and the env var are RENDERED -- 6 probes, including two
separate absence probes (mount and consumption are wired at different
sites, and the gateway outage happened precisely because two of three
sites tolerated a missing cert and the third did not). It CANNOT prove
tuwunel loads them: a build that ignores the variable evaluates
identically to one that honours it, and the chain above is
version-specific. That half wants an executed check on a real deploy.
`services.hyperhive.domain` and `swarm.hives.<hiveName>.domain` were two
homes for one value with nothing asserting they agreed. The failure that
buys is the worst shape a config defect has: it evaluates cleanly, and
the symptom ("the other hives can't reach me") appears on a machine
other than the misconfigured one.
The directory is now the single source. `hives.<name>.domain` gains the
`<name>.<swarm.domain>` default -- a derivation from two values an
operator had to state explicitly, not a guess -- so a conventional swarm
is a list of names and a hive addressed by something else says so in the
one place every other hive reads. `services.hyperhive.domain` reads its
own entry; the direct formula is deleted rather than kept as a fallback,
which would have restored the second path (and, reading `swarm.domain`
itself, a second path that can disagree).
Setting it directly still wins, with a deprecation warning: nothing
breaks today, but a value written only there is invisible to the swarm.
The self-entry assertion now fires on an EMPTY directory too. Since
`swarm.domain` became required, every hive is in a swarm -- a swarm of
one is still a swarm -- and this host's address is read out of the
directory, so the entry is missing either way and the precise message
should be the one that fires.
Upgrading costs one line on hives that never listed themselves:
`services.hyperhive.swarm.hives.<hiveName> = { };`, no value.
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
The CA page described adoption as a documented `rm`, which is now only
half true: where the host owns the root it happens by itself, and the
split between the two cases is the part worth explaining rather than the
commands.
Second split under docs/swarm/, following the shape docs/turn-loop/ and
docs/web-ui/ already use.
The README keeps the `## Swarm CA` heading rather than deleting it: an
existing cross-reference targets that anchor, and a heading is the
cheapest thing to preserve. It now carries the one-paragraph summary and
a pointer, so a reader who lands on the anchor still arrives somewhere
that answers the question.
The CA page also absorbs three things that were true but undocumented —
why the trust bundle rather than the bare intermediate (openssl will not
terminate a chain at a non-self-signed anchor without -partial_chain),
that autoConfigure derives from enableAllLocalDefaults, and that the
name constraint excludes both IP families because a permitted-DNS-only
constraint says nothing about IP SANs.
Both worked from the flat docs/swarm.md and resolve one directory
short from docs/swarm/README.md: snapshot-store.md stayed in docs/,
and swarm-controller/README.md is at the repo root.
`docs/swarm.md` becomes `docs/swarm/README.md` and the shared-services
material moves to `docs/swarm/services.md`, following the shape
`docs/turn-loop/` and `docs/web-ui/` already use. The README keeps a
pointer so the reading path is unbroken.
Every referrer moved with it — five docs pages, two option descriptions
in swarm.nix, and CLAUDE.md's reading path. A pointer to a file that
moved is worse than one to a file that was deleted: the content still
exists, so the reader concludes the note is wrong rather than the path.
