The swarm tier is the only holder of the upstream credential, the only
writer to the swarm's metrics store, and (once #3283 lands) the place
that stamps hive= from the authenticated connection rather than from
anything a sender can choose. Today one collector does both tiers' jobs,
which works only because they land on one box.
A container rather than a second host unit, for two reasons that agree:
every sibling swarm service is one, and `services.opentelemetry-collector`
is a singleton NixOS option already spoken for on the host by the hive
tier. A container gets its own evaluation and therefore its own
collector.
Port defaults to 4319, deliberately not the OTLP default 4318 the hive
tier uses: swarm containers share the host netns, and two listeners
claiming one port is not a build failure but a runtime coin toss with
nothing in any log saying so -- the same collision grafana and the forge
hit on 3000.
`url` is an option with a co-located default rather than a loopback
literal in the exporter, so a split-host deployment is a config change
instead of a code change.
Wires nothing yet: the hive tier still exports directly, and switching it
over is the next commit.
Second half of the metrics pair: a `swarm-grafana` container beside the
VictoriaMetrics store, provisioned with it as the default datasource and
fronted by the gateway on its own swarm-sibling name.
Behind swarm SSO, per the operator's call on #3265. The authelia client
and Grafana's callback URL both derive from `domain`, so the exact-match
string authelia checks cannot drift from the one Grafana sends. The
minted secret is delivered host-side (both container trees are only
addressable there) and reaches Grafana as a `$__file{}` reference rather
than a value, so it never enters the store.
The login form is disabled whenever SSO is configured: Grafana ships an
`admin`/`admin` account and this vhost is on the public gateway.
A local time-series database rather than only an external sink, so the
swarm dashboard stays readable when the outside world is not: a view of
the system must not depend on the system it views being healthy.
listenAddress is pinned to loopback. Upstream defaults it to every
interface, and the OTLP ingest path this exists to receive on is
unauthenticated — the gateway is the only intended client and it is on
this host, so a wider bind would publish a write endpoint to whatever
the host is reachable on.
retentionPeriod defaults high rather than being required, because the
two failure directions are not symmetric: too long fills a disk, which
is visible and recoverable by lowering it, while too short destroys
history silently and permanently. The operator lowers it once they have
measured how fast this swarm accumulates.
OTLP needs no flag. Measured against the pinned 1.146.0 rather than
inferred from the module's option list, which has no OTLP switch and so
reads as though the feature were missing: the running server answers
POST /opentelemetry/api/v1/push with 200, where a nonexistent path
answers 400.
A swarm has one message queue; this is the container that runs it, in
the same shape as every other swarm service. Off by default, and off
means absent.
The auth_callout block is the interim state and the final one at once.
Measured on the pinned nats-server rather than reasoned about: an empty
authorization block, and one with an empty users list, both accept an
anonymous client and answer PONG. auth_callout sets auth_required and
refuses everyone until a responder approves them, so a config whose
responder does not exist yet denies by default -- and the responder
lands beside it without any of this being swapped out.
That matters more here than it would elsewhere because the container
shares the host netns, so an unauthenticated interim state would be
reachable from every agent container on the hive.
No package option, deliberately: nixpkgs' nats module resolves
pkgs.nats-server itself and exposes none, so one here would name
something it cannot control.
No gateway vhost either: NATS speaks its own TCP protocol, nginx cannot
front it, and cross-hive reach is the wireguard mesh.
New swarm-ui module: enable (derived from swarm.controller.enable - the
UI reads that daemon's state over its socket, so the host that runs the
controller is the host that can serve the UI), domain (defaults to the
swarm apex; an option so a hive can pin it like forge/matrix can), and
package.
Adds the name to swarm.serviceDomains, which is both the services
sub-CA's nameConstraints set and the leaf's SAN set. The apex is a
SIBLING of forge./chat./auth., not a parent, so nothing issues for it
implicitly - left out, the vhost falls back to the hive leaf and the
swarm's front page opens with a name mismatch.
Asserts the UI domain differs from the hive domain: the gateway's
default server already answers for the latter, and two vhosts claiming
one server_name resolve to whichever nginx picks rather than erroring.
One attrset describing every hive in the swarm including this one,
identical on every host, with hiveName selecting which entry is us.
"My peers" is derived (swarm.peerHives) rather than declared.
Every field in the old per-host peer list was intrinsic to the hive it
described, never to the pair -- so the list was a directory each host
kept its own copy of. Beyond the deduplication it removes a bug class:
two hosts could hold different endpoints for the same third hive with
nothing to detect the disagreement.
Drops the per-hive caCert. Trust inside a swarm derives from the swarm
root, which every hive chains to. What that genuinely removes is
trusting a hive whose root this swarm does not own -- a cross-swarm
problem that wants a mechanism of its own, not a field that happened to
work.
The matrix container's certificateFiles block goes with it and could
NOT be migrated: that list is read at build time and the swarm root is
a runtime file (its key must never enter the store), so there is no
build-time name to put there. caCert being a nix path was precisely
what made it the build-time distribution channel. Agents are unaffected
-- hive-tls folds the root into the hive trust bundle and the meta
renderer embeds that one file. Tracked separately.
Migration is an assertion plus warnings, not a rename: hives is peers
union {self}, and the set gains a member no existing config has written
down. A rename migrates a name and a default can re-root a meaning;
neither can conjure a new member. The warning explains, the self-entry
assertion stops the build.
mara: enableAllLocalDefaults is more of a deployment mode via settings
set, less a default setting.
That reframing is the change. A mode asserts values; an option declares
what it is when nobody asks. Written as default = <flag>, every service
option had to name a switch it has no relationship to, and the answer to
what does all-local turn on was spread across five files.
Two modules now hold the two tiers - local-defaults.nix for the mode and
what it asserts directly, swarm-required-services.nix for the
services-live-here switch and the per-service enables under it. Each
service module keeps its own declaration and loses only the line about
when a deployment wants it.
mkDefault is the right precedence rather than a compromise: it beats an
option default and yields to any explicit definition, so the mode fills
in for an operator who has not spoken and never argues with one who has.
Evaluated config is unchanged in both directions.
One SSO provider per swarm, and a hive is a client of it by default:
`url` exists everywhere so a hive knows where to send people, while the
container only exists where `swarm.enableRequiredServices` says the
swarm's shared services live.
Operator and agents are both subjects of the same provider,
differentiated by roles/claims rather than by mechanism. The users store
is therefore written by swarm-controller rather than maintained by hand
— agents are created and destroyed continuously, so the subject set is
dynamic. This module only guarantees the file exists and parses, so
authelia starts with nobody in it instead of failing to start; a
placeholder account would be a credential nobody meant to create.
Small-deployment defaults, with scope as the justification rather than
preference: file users backend, local sqlite storage (redis buys shared
session state across replicas and there is one instance), filesystem
notifier (SMTP mails humans; provisioning is programmatic).
Secrets are generated in-container on first boot, sharing the authelia
unit's user/group/StateDirectory so systemd creates the directory owned
by the account that reads them. Nothing outside the container reads
these keys, which is what makes that right rather than merely easier —
hive-matrix generates its token host-side only because hive-c0re has to
read that one.
Per-service integration (authelia's auth_request in front of the
gateway's existing auth_basic locations) is deliberately not here:
standing an SSO provider up is reversible, cutting every operator-facing
vhost over to it is not.
Cross-hive trust was O(n²) hand-pinning: every hive had to name every
peer's CA. A swarm root makes it O(1) — trust the root once and every
present and future peer validates.
The root is generated by a new `swarm-ca` unit on a single-host swarm
and operator-provided otherwise; `swarm.ca.autoConfigure` picks between
them and derives its default from `swarm.peers` being empty, so "all on
one host" is read off the deployment rather than remembered. Both modes
produce the same artifacts in the same places, so splitting hosts later
is moving the service dirs, not switching code paths. The root key never
enters the nix store, and the root is never regenerated automatically —
replacing it invalidates every peer at once.
Each hive CA carries `nameConstraints` pinned to that hive's domain, so
a leaked hive CA can only mint names inside its own subdomain, enforced
by verifiers rather than by convention.
`ca.pem` was serving as both the issuer and the anchor consumers trust;
those are the same file only while it is self-signed. openssl will not
terminate a chain at a trusted cert that isn't self-signed (rustls and
Go will), so the promotion would have broken some consumers and not
others. `hive-tls-ca` now also writes `trust-bundle.pem` — the hive CA
plus whatever it is rooted at — and every anchor consumer reads that:
agents, the CI and forge containers, and the peer-config recipe. On a
hive with no swarm root the bundle is just that CA, so nothing consuming
it needs a mode to branch on.
services.hyperhive.swarm.controller.{enable,package,socketPath} plus the
unprivileged swarm-controller user, its runtime and state directories,
and the unit itself.
enable is deliberately not derived from services.hyperhive.enable, unlike
c0re: a swarm has one controller, so turning it on is a statement about
swarm topology rather than about whether hyperhive is installed.
The socket gets its own RuntimeDirectory. nginx reaches a unix upstream
by having the socket's directory bind-mounted into the gateway
container, and the socket is 0666 because connect needs write -- so the
directory is the only access control there is. Sharing one with the host
admin socket would hand that socket to the gateway too. The constraint
is stated at both ends, in the option description and beside the bind,
because it is invisible from either site alone; a test pins the path so
a tidying edit fails rather than reviews cleanly.
RuntimeDirectoryPreserve and the daemon's stale-socket unlink are a
pair: preserving the directory without the unlink means bind fails with
EADDRINUSE after a restart.
mara: "remove the extra deprecatios file, less comments". The alias now
sits in hive-forge/default.nix next to the options it renames, and the
comment is three lines instead of eighteen.
Better than tidier: there is no shared file for the sibling migrations
to queue behind. Matrix's rename goes in hive-matrix.nix, authelia's in
its own module, and none of them touch each other. hive-ci.nix declares
swarm.forge.ci.* from a different module and stays covered, because the
rename is on the namespace rather than on the declaring file.
Re-ran the check both ways this time, per argus on the last round: old
paths set and new paths read back 3999 / 7, and a config that only uses
the new paths produces no warnings at all.
Forgejo is a swarm-global service, so its operator-facing host options
move to services.hyperhive.swarm.forge (and .swarm.forge.ci) as the
first of the namespace consolidation.
Existing hive configs keep evaluating: swarm-renames.nix maps every
moved leaf with mkRenamedOptionModule, which also emits a deprecation
warning naming both the old and new path, so an operator is told what to
rename rather than discovering it from a failed eval.
The per-agent hyperhive.forge.url does NOT move. It is a client pointer
at whatever forge an agent talks to - it shares a word with the service
and nothing else, and the two are already documented as separate option
surfaces.
Verified by evaluating the host module, since no Rust gate evaluates
nix: setting the old paths and reading the new ones yields the values
(httpPort 3999, ci.concurrency 7), and config.warnings carries the
rename notice.
mara, in preparation for the swarm tier: the store is a swarm-level
role, not a hive one, so hive- was misleading about which tier it
belongs to. Module, units, syslog identifier, log lines and docs all
move to swarm-snapshot-store.
Also moved the option under services.hyperhive.swarm.snapshotStore, to
sit with swarm.peers and swarm.wireguard rather than dangling off the
top level. That is a judgement call beyond the literal rename — flagged
on the PR, and cheap precisely now: the option has never shipped, so
there is no deployment to migrate, whereas doing it after a release
would be a breaking change for no new benefit.
mara asked, and the file had already stopped being one thing: after
the gate moved off c0re.enable, swarm.nix held two concerns with
different audiences and different gates.
swarm.nix now declares WHO the peers are — data hive-c0re serialises
into HYPERHIVE_PEERS and the dashboard renders. Declaration only, no
config block.
swarm-wireguard.nix owns the mesh: assertions, the wg-hive interface,
the firewall port. That is plain host networking, and a machine which
runs no hive at all — the snapshot store — still needs it. Under the
old layout a reader could not tell which half of swarm.nix applied to
a non-hive host.
The two stay coupled by data, not by structure: the per-peer
wireguard* fields stay on the peer submodule, because that is where a
peer is described, and the mesh module reads them.
No behaviour change — same options, same gate, same rendered config.
P1 of the storage backend: hives push agent snapshots over the
WireGuard mesh that swarm.nix already brings up. No controller
dependency — a btrfs subvolume tree, a socket-activated receiver, and
the existing mesh.
The mesh is the authentication. Cryptokey routing already binds a
peer's source address to its public key (allowedIPs = [
peer.wireguardAddress ]), so the store adds no key material and no
certs; anything else would authenticate the same fact twice.
Destination is keyed per AGENT, not per hive: after a migration the
same agent's next incremental send arrives from a different hive, and
a per-hive prefix would split its snapshot chain and break the
incremental parent lookup — the exact case this store exists to serve.
The sender unavoidably contributes the agent name (a btrfs stream
carries no such notion, and the subvolume name inside it is the
sender's). So the receiver owns the destination root and VALIDATES the
sender-supplied leaf against a whitelist charset — no slash, no dot,
so neither traversal nor an absolute path can survive it.
ListenStream binds this host's mesh address, never a wildcard, and
that is asserted rather than commented: bound to 0.0.0.0 the socket
would be an unauthenticated remote write into agent state.
swarm.nix: the mesh config moves off the c0re.enable gate onto
swarm.wireguard.enable. The mesh is host networking, not a c0re
feature — a swarm host that runs no hive (this store) previously got
no wg-hive interface at all. Nothing in that block was c0re-specific;
the peer data c0re consumes is rendered in hive-c0re and stays gated
there.
Confinement is deliberately not in the module: it is a property of the
deployment (a dedicated VM, or a container in the all-local case). The
systemd hardening is defence in depth only — btrfs receive needs
CAP_SYS_ADMIN, which can mount() its way out of the namespace those
directives set up. The `dedicated` option turns "this host runs
nothing else" into an assertion the build checks instead of an
assumption the deployer remembers.
