Review feedback: the metrics pair had nothing writing into it, and it sat
outside the switch that turns on every other swarm-wide service.
The collector now exports to VictoriaMetrics as well as upstream — a
fan-out, not a choice: a local store is for looking at this swarm, an
upstream is for whoever aggregates across swarms.
That makes a local store a complete destination on its own, so
`otel.endpoint` is no longer required when it runs here; a hive with
neither is still refused. The assertion only ever relaxes, so every
config that evaluated before still does.
`enableRequiredServices` now derives both halves, alongside matrix,
authelia and nats. They derive together because a store with no UI is
unreadable and a UI with no store is empty.
Review: the public keys are not an exception, the paragraph was
incomplete. It is private key material and access tokens that are
file-based so they stay out of the nix store; public material in a
public store is fine.
Fixes the rule itself, which removes the need for the special case the
previous commit added.
secrets.md is the page that answers "who mints this and where does it
go" per topology, and the callout nkeys had no row. Adds one, plus the
`nk` invocations an operator needs on every topology that is not
all-local.
The note exists because the change provokes a question the page would
otherwise answer wrongly: this page opens with "a secret is a path,
never a value", and the queue takes two public keys as values. A reader
who meets those after reading that line concludes the page is stale.
The section carried a "nothing publishes yet" note that is now false, and
said nothing about the one thing an operator has to do.
Written to the reader's question rather than the author's: what to set,
what defaults on an all-in-one host, what has to be carried by hand to a
hive that is not the swarm host, and where to look when a hive goes quiet.
The identity and the publish cadence are stated because they constrain the
staleness threshold an operator picks; the mechanism behind them is not.
Follow-up to the plumbing removal: docs/swarm/README.md gets the
biggest rewrite (drops the whole "Fingerprint format" section, fixes
the runtime-effects list, the WireGuard config example + "what the
mesh does" bullet), docs/gateway.md and hive-gateway/options.nix drop
4 "needs no certFingerprint" mentions, swarm-peers-removed.nix's
migration-warning text no longer tells an upgrading operator to carry
a field over that no longer exists, swarm.nix/swarm-wireguard.nix/
swarm-controller.nix/swarm-controller's main.rs get comment fixes
where they described the now-removed HYPERHIVE_PEERS shape. Also
caught one more stale "peer hives" mention in docs/web-ui/README.md's
SW4RM tab description that the first pass on this issue missed.
Follow-up to the peer-hives removal: docs/web-ui/dashboard.md drops
the whole "P33R H1V3S" section + its stale #peers hash and tab-strip
mentions; docs/swarm/README.md's runtime-effects list and
cross-references point at swarm-ui's roster page instead. Nix comments
in hive-c0re/environment.nix, swarm.nix and swarm-wireguard.nix that
described HYPERHIVE_PEERS/certFingerprint as dashboard-consumed are
updated to say what's actually still true (agent identity only).
Deliberately NOT removed: the `certFingerprint` nix option itself.
It has no remaining Rust consumer, but deleting a documented,
potentially already-set option is a bigger, breaking change than
removing dead display code — flagged in docs/swarm/README.md with a
note instead of silently dropped, so an operator or reviewer can make
that call explicitly rather than have it made for them here.
Per mara on the PR: less implementation detail, more operator focus. The
section argued why the design is shaped as it is - a cache would
disagree with itself, freshness must be derived rather than stored, the
timestamp is the server's - which is reasoning a reader of this page does
not need and cannot act on.
It now answers what an operator actually asks: what each freshness value
means they should do, what to set the threshold to and why that number,
what a 503 tells them, and that nothing publishes yet so an empty board
is expected rather than broken.
The design rationale is not lost, it is just homed where it belongs: the
status module's own doc comment, next to the code it constrains.
The controller connects to the swarm queue as its own client and serves
what each hive last said about itself at GET /api/hives/status.
THE QUEUE IS THE STORE. A hive publishes into the `hive-status` JetStream
KV bucket (history 1) and the controller reads it per request, keeping no
copy. A cache here would be a second answer to the same question, free to
disagree with the first, and the disagreement surfaces as a hive reading
healthy on a dashboard while the bucket says otherwise. Whichever side
arrives first creates the bucket; both want the same shape.
Rows come from the roster rather than from the bucket, so an empty bucket
renders as a swarm nobody has heard from instead of a healthy one, and
`never_reported` stays distinct from `stale` - went quiet is a fault,
never spoke is usually a deployment that has not happened. Freshness is
derived at read time and never stored as a flag, because a stored
`healthy` boolean goes stale silently the moment nothing arrives, which
is the failure this endpoint is designed against. The timestamp is the
NATS server's, applied when the value landed, so a publisher cannot make
itself look fresher than it is.
Authentication is per connection attempt, not per process. Authelia
issues `client_credentials` tokens that expire in 3599s, and auth happens
at CONNECT, so a long-lived connection is fine but a reconnect an hour
later needs a token minted an hour later. `with_auth_callback` is re-run
by async-nats for each attempt, which handles expiry by construction
rather than by a timer - the alternative fails in the way this subsystem
exists to prevent, with the controller still serving while its data
quietly stops updating.
Three failure shapes are deliberate:
- A half-set environment is fatal; an absent one is not. Silently
behaving like an unconfigured host is how every hive ends up reading
`never_reported` with nothing to point at.
- The endpoint answers 503 rather than an empty list when the store
cannot be read. "I cannot reach the store" and "every hive is silent"
are different answers, and rendering the second turns a local fault
into an apparent swarm-wide outage.
- `retry_on_initial_connect` makes the daemon and the queue bootable in
either order, and the status handler refuses when the client is not
Connected rather than issuing a request into it - a request made in
that window does not fail, it waits, so every poll would hang and
learn nothing. `Pending` is the state a never-connected client is in,
which is why the test is `!= Connected` and not `== Disconnected`.
The rendering rules are a pure function over a map, so the semantics are
tested against a table rather than against a running server. The KV read,
the credential rotation and the 503 paths are covered behaviourally
instead: a real NATS server with a rotating token endpoint, asserting
that the controller recovers only when the credential rotates, and
mutation-tested by holding the credential wrong for the same window.
Single 🔗 button in the Shell header (direction 1 of 3 proposed on the
issue) — one fixed element regardless of link count, so it can't
clutter as the list grows, and visible on every route since Shell
wraps all of them. Opens a popover listing whatever GET /api/links
returns (icon + label, opens in a new tab); hidden entirely when the
list is empty, same "don't render a dead affordance" rule the old
dashboard's H0M3 tiles follow for Forge/Matrix.
Docs: docs/swarm/ui.md gets a short section on the feature + the
contribute-your-own-entry idiom.
Verified: npm run build + npm run typecheck clean, headless-chromium
screenshots of both the closed and (temporarily forced open for the
screenshot only) open states, pixel-sampled to confirm the popover is
actually using the dark theme vars and not just looking that way in a
downscaled preview.
Per mara: setup should point at it. The link is placed as a precondition
rather than a see-also -- every step below assumes each credential is
generated where it is read, which is only true all-local.
Per argus: 'with a bound' was missing a word; it now names the actual
120s wait instead of gesturing at one.
Per mara on the CA question: outside all-local this is an ops problem,
and what is missing is documentation rather than machinery.
One row per secret, read against three topologies, because the same
credential is generated in place on one and handed over by an operator on
another. sso.md's table is replaced by a pointer -- two tables listing the
same secrets would drift, and its prose about why a secret is generated
in-container is the half worth keeping there.
States the constraint the whole thing rests on: a hive CA is name-
constrained to the hive domain and a swarm service name is a sibling of
it, so a hive CA cannot issue a swarm-service certificate at all. That is
why placing the swarm root is an operations decision.
The registration half is shared; what each service does with the result
is not, so the differences get a table rather than a second page.
States why matrix reads its secret through LoadCredential and why
/_matrix/ is not a forward-auth surface -- both are conclusions a reader
would otherwise have to re-derive from the module.
`user add` refuses on an existing name, so the `--group` flag it takes at
creation time could not be added afterwards at all: repairing an account
meant hand-editing both users.json and the rendered users.yml as root.
mara, on #3167: "i will not edit those files by hand, we will have the
same issues elsewhere".
The merge rules live in users.rs as a pure function over a UserUpdate, so
they are testable without a command line, a container or a running
authelia — main.rs's arm only loads, applies, publishes and prints.
Removals are strict and everything else is idempotent, which is the one
asymmetry here and is deliberate: a --remove-group naming a group the
user does not have fails, because a revocation that reports success
without revoking is the outcome nobody re-checks; while refusing an
already-satisfied set would make the multi-attribute call this verb
exists for break whenever one of the values was already right.
A command that changes nothing at all still fails — it would otherwise
rewrite both files and restart the SSO provider to no effect.
Passwords are out of scope: regenerating a credential is a different
intent from editing an attribute, and folded together an attribute edit
can invalidate a login by accident.
Extracts publish() from user_add so both verbs share the
render -> store -> users.yml -> restart ordering and the comment that
explains why that order, rather than the second verb copying it.
Per mara on the PR: the guide someone follows on a fresh deploy is a
different page from the one they read while debugging, and only the
second existed.
Also fixes a contradiction the request surfaced: setup.md and
swarm/sso.md have told every operator to create '--group admins' since
the bootstrap step existed, while the new rule required 'operators'.
This is the first rule that CONSUMES a group name, so it takes the one
the guide already creates - inventing a second would have meant every
account made by following the guide silently failing the check it was
supposed to pass, and for mara specifically a migration step that now
may not be needed at all.
setup.md's step 3 says why --group is not decoration; the new step 4
says what decides whether the page opens (the group, and the name
resolving) rather than listing a command, because there is nothing to
run. Steps renumbered, including the matrix block's internal 4a-4e.
New docs/swarm/ui.md (split-page shape, per the docs rule), linked from
the swarm README and added to the gateway's vhost map.
Leads with the step that separates 'protected' from 'locked out':
swarmctl user add <you> --group operators. auth_request asks whether
there is a session; the access_control rule is what makes it mean
operator, and an account created before the rule existed has no groups.
Also records the four wiring sites a swarm service name needs, with the
certificate one called out - serviceDomains is both the sub-CA's
nameConstraints set and the leaf's SANs, and the apex is a sibling of
the other three rather than a parent, so nothing issues for it
implicitly.
`swarm/sso.md` described the empty user store as a resting state — a
provider that is reachable but has nobody in it yet. It isn't. Authelia
validates the store at startup and treats zero users as fatal:
error reading the authentication database: could not validate the
schema: users: non zero value required
so it exits 1, systemd restarts it, and an enabled-but-unbootstrapped
swarm presents as a crash-looping container behind a vhost that is
working correctly. The observed symptom is `502 Bad Gateway`, which
reads as a proxy fault and is not one.
Says so, gives the error text to grep for, and marks the `swarmctl user
add` step as required to finish the install rather than as a first-login
convenience. `gateway.md` gains the same warning next to the vhost,
because that is where someone lands when the 502 is what they can see.
The reason the store ships empty is unchanged and still right: seeding
an account means a credential in a config file. What was wrong was
calling the resulting state harmless.
The vhost half of this change is only useful with an account behind it,
and the provider is generated with an empty user set on purpose. Document
the `swarmctl user add` step rather than automating it: bootstrapping an
IdP non-interactively means a secret arriving from a file, an env var or a
nix expression, all worse than one command typed once.
The gateway and network pages gain the rows they would otherwise be
missing — vhost map, local-dev hosts entry, and the resolver's
authoritative-name list.
The question this answers is "what do I have to configure, and where" —
so the table of secrets is the deliverable and the prose is scaffolding
around it.
The organising idea worth keeping: a secret belongs in-container when
nothing outside that container reads it. Every one of authelia's own
secrets passes that test; the client secret's plaintext fails it, which
is what makes delivery a problem at all rather than a detail.
Names the two ways out (install the sub-CA, or pin back under the hive
domain) and says why it is a warning rather than an assertion — the
module can see what it issues, not what the operator installed.
`security.pki.certificateFiles` is read when the system is BUILT and the
swarm root is deliberately a runtime file (its key must never enter the
world-readable store), so there was nothing build-time to name and the
container trusted no swarm-internal CA. Federation with a peer whose
cert chains to the swarm root did not validate.
The bridge already existed: `lib/hive-ca-trust.nix`, which `hive-ci` and
`hive-forge` both cross. This wires the matrix container to it -- read-only
bind of the trust bundle, container ordered after `hive-tls-ca.service` --
rather than inventing a second mechanism.
Consumption is the per-runtime half, and it is not what it looks like.
tuwunel links no openssl, which makes `SSL_CERT_FILE` appear inapplicable.
It isn't: the outbound client is `reqwest` with the `rustls` feature,
which builds a `rustls_platform_verifier::Verifier`, and because tuwunel
calls `tls_certs_merge` (additive) rather than `tls_certs_only`, the
platform roots stay alongside its compiled-in webpki set. On Linux that
verifier resolves through `rustls-native-certs` to `openssl-probe`, which
reads `SSL_CERT_FILE`. Established by reading tuwunel 1.8.0 and reqwest
0.13.4 source, not inferred from the feature list -- an earlier reading of
mine concluded the opposite from the absence of a `native-roots` feature
name and was wrong.
The variable therefore names a CONCATENATION (system CAs + the bundle),
never the anchor alone: `openssl-probe` uses it *instead of* the default
store, so pointing it at the bundle would drop every public CA and break
federation with the wider matrix network -- trading a small outage for a
much larger one.
`container@hive-matrix` needed `mkMerge`: it already assigned `after` for
the gateway ordering and the helper contributes its own `after`/`requires`,
so two bare assignments would have conflicted rather than combined.
⚠️ Gate honesty: `state/eval-3093.sh` proves the mount, the ordering, the
bundle service and the env var are RENDERED -- 6 probes, including two
separate absence probes (mount and consumption are wired at different
sites, and the gateway outage happened precisely because two of three
sites tolerated a missing cert and the third did not). It CANNOT prove
tuwunel loads them: a build that ignores the variable evaluates
identically to one that honours it, and the chain above is
version-specific. That half wants an executed check on a real deploy.
`services.hyperhive.domain` and `swarm.hives.<hiveName>.domain` were two
homes for one value with nothing asserting they agreed. The failure that
buys is the worst shape a config defect has: it evaluates cleanly, and
the symptom ("the other hives can't reach me") appears on a machine
other than the misconfigured one.
The directory is now the single source. `hives.<name>.domain` gains the
`<name>.<swarm.domain>` default -- a derivation from two values an
operator had to state explicitly, not a guess -- so a conventional swarm
is a list of names and a hive addressed by something else says so in the
one place every other hive reads. `services.hyperhive.domain` reads its
own entry; the direct formula is deleted rather than kept as a fallback,
which would have restored the second path (and, reading `swarm.domain`
itself, a second path that can disagree).
Setting it directly still wins, with a deprecation warning: nothing
breaks today, but a value written only there is invisible to the swarm.
The self-entry assertion now fires on an EMPTY directory too. Since
`swarm.domain` became required, every hive is in a swarm -- a swarm of
one is still a swarm -- and this host's address is read out of the
directory, so the entry is missing either way and the precise message
should be the one that fires.
Upgrading costs one line on hives that never listed themselves:
`services.hyperhive.swarm.hives.<hiveName> = { };`, no value.
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
The CA page described adoption as a documented `rm`, which is now only
half true: where the host owns the root it happens by itself, and the
split between the two cases is the part worth explaining rather than the
commands.
Second split under docs/swarm/, following the shape docs/turn-loop/ and
docs/web-ui/ already use.
The README keeps the `## Swarm CA` heading rather than deleting it: an
existing cross-reference targets that anchor, and a heading is the
cheapest thing to preserve. It now carries the one-paragraph summary and
a pointer, so a reader who lands on the anchor still arrives somewhere
that answers the question.
The CA page also absorbs three things that were true but undocumented —
why the trust bundle rather than the bare intermediate (openssl will not
terminate a chain at a non-self-signed anchor without -partial_chain),
that autoConfigure derives from enableAllLocalDefaults, and that the
name constraint excludes both IP families because a permitted-DNS-only
constraint says nothing about IP SANs.
Both worked from the flat docs/swarm.md and resolve one directory
short from docs/swarm/README.md: snapshot-store.md stayed in docs/,
and swarm-controller/README.md is at the repo root.
`docs/swarm.md` becomes `docs/swarm/README.md` and the shared-services
material moves to `docs/swarm/services.md`, following the shape
`docs/turn-loop/` and `docs/web-ui/` already use. The README keeps a
pointer so the reading path is unbroken.
Every referrer moved with it — five docs pages, two option descriptions
in swarm.nix, and CLAUDE.md's reading path. A pointer to a file that
moved is worse than one to a file that was deleted: the content still
exists, so the reader concludes the note is wrong rather than the path.
