mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

add a semi-spread example deploy and a check that evaluates it

Browse source
This commit is contained in:
atlas 2026-11-09 18:05:27 +01:00 • committed by mara
commit 538da5a13a
15 changed files with 875 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

4
AGENTS.md
View file

@ -200,7 +200,9 @@ Swarm-level:
module-eval tests), `builder/` (`check-flake`), `script-tests/`, plus module-eval tests), `builder/` (`check-flake`), `script-tests/`, plus
`sources.nix` / `rust.nix` / `checks.nix` / `devshell.nix` / `sources.nix` / `rust.nix` / `checks.nix` / `devshell.nix` /
`treefmt.nix` behind the thin `flake.nix`. `treefmt.nix` behind the thin `flake.nix`.
- **`docs/`** — subsystem reference docs (see *Reading paths* below). - **`docs/`** — subsystem reference docs (see *Reading paths* below), and
`examples/`, the NixOS files the handbook's worked examples use. CI
evaluates them.
- **`branding/`**, **`scripts/`** — static assets + helper scripts. - **`branding/`**, **`scripts/`** — static assets + helper scripts.
## Reading paths ## Reading paths

1
docs/1-getting-started/nixos.md
View file

@ -107,6 +107,7 @@ Leave the other options at their defaults until you need them:
- Spreading a swarm over more than one host, and the credentials a split swarm - Spreading a swarm over more than one host, and the credentials a split swarm
needs: [`2-handbook/swarm/README.md`](../2-handbook/swarm/README.md) and needs: [`2-handbook/swarm/README.md`](../2-handbook/swarm/README.md) and
[`2-handbook/swarm/credentials.md`](../2-handbook/swarm/credentials.md). [`2-handbook/swarm/credentials.md`](../2-handbook/swarm/credentials.md).
A worked example: [`2-handbook/swarm/semi-spread-deploy.md`](../2-handbook/swarm/semi-spread-deploy.md).
- Every option, with its type and default: - Every option, with its type and default:
[the options reference](https://hyperhive.darkest.space/options/). [the options reference](https://hyperhive.darkest.space/options/).
- The secret store's seal and the granter: [`4-implementation/swarm/bao.md`](../4-implementation/swarm/bao.md). - The secret store's seal and the granter: [`4-implementation/swarm/bao.md`](../4-implementation/swarm/bao.md).

224
docs/2-handbook/swarm/semi-spread-deploy.md Normal file
View file

@ -0,0 +1,224 @@
# Example: a semi-spread deploy
Three swarm machines and two hives, joined by a WireGuard VPN, with only the
forge, the matrix homeserver and the swarm UI reachable from the internet. It
is one of the in-between shapes that [`services.md`](services.md#deployment-shapes)
supports: the swarm's services run on three machines, not on one box and not
one per service.
The NixOS files are in
[`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all
five machines (`checks.module-eval-semi-spread`) and pins the facts on this
page, so the files stay in step with the options.
## The machines
| machine | runs | reachable from the internet |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI |
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | forge, matrix, SSO login |
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing |
| `hive-one` | the hive `one` | nothing |
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing |
Why this split:
- **`swarm-central`** holds what every other machine depends on, and little
else that can fail. The secret store and the controller are small.
- **`swarm-services`** holds the services people and agents talk to all day.
They're light, and they need the public name and port.
- **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder
evaluates config pull requests nobody reviewed yet, and the CI runner runs
pull-request code. Keeping both off the machine that holds the secret store
limits what a bad build can reach.
- **The hives** run the agents. They need nothing from the internet but the
agents' own egress.
A hive name can't contain `hive`, `agent` or `swarm`, so the machines
`hive-one` and `hive-two` run the hives `one` and `two`. `hive.nix` takes the
hive name from the machine's host name, which is why one file serves both.
## What's public and what's not
`gateway.openFirewall` opens the gateway's ports (80 and 443) on every
interface, and the example sets it on two machines only:
- `swarm-central` serves the swarm UI at `<swarm.domain>`.
- `swarm-services` serves the forge (`forge.<swarm.domain>`) and the matrix
homeserver (`chat.<swarm.domain>`). It also serves `auth.<swarm.domain>`,
because the forge, matrix and the swarm UI send browsers there to sign in.
**Public means the machine's gateway port is open.** A gateway is one nginx
for everything the machine serves, so the other names on those two machines
(Grafana, metrics, logs, the collector, the store's browser UI) answer on the
public address too. Each sits behind SSO or a bearer token, and none of them is
in public DNS, but they're not closed. A service you want closed goes on a
machine that opens nothing public, like `swarm-heavyduty`.
Everything else uses the VPN. These ports open on `wg0` only, with
`networking.firewall.interfaces.wg0.allowedTCPPorts`:
| machine | port on `wg0` | for |
| ----------------- | ------------------------- | --------------------------------------------- |
| `swarm-central` | `swarm.bao.port` (8200) | the secret store, read by every other machine |
| `swarm-services` | `swarm.nats.port` (4222) | the queue |
| `swarm-heavyduty` | `gateway.httpsPort` (443) | the cache, behind the gateway |
| each hive | `gateway.httpsPort` (443) | the hive's dashboard and gateway |
The example scopes these to `wg0` because the options that open a service's
port, `gateway.openFirewall`, `deploy.bao.openFirewall` and
`deploy.nats.openFirewall`, open it on every interface. The
secret store and the queue don't rely on the VPN for their security: the store
requires a client certificate, and the queue requires TLS and refuses clients
it can't identify. The VPN keeps them off the internet as well.
Every machine also opens UDP 51820 on every interface, because that's where
the VPN's handshake arrives. The example leaves SSH to you.
```text
internet
80/443 | | 80/443 UDP 51820 on all five
| |
swarm-central swarm-services swarm-heavyduty hive-one hive-two
10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12
| | | | |
+----------------+----- wg0, 10.77.0.0/24 --------+----------+
```
## The VPN: raw WireGuard
`wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on
all five machines. `mesh.nix` holds each machine's VPN address, public endpoint
and public key. The file picks the machine's own entry by `networking.hostName`
and makes the other four its peers:
```nix
networking.wireguard.interfaces.wg0 = {
ips = [ "${mesh.${self}.address}/24" ];
listenPort = 51820;
privateKeyFile = "/etc/wireguard/wg0.key";
peers = lib.mapAttrsToList (_: peer: {
inherit (peer) publicKey endpoint;
allowedIPs = [ "${peer.address}/32" ];
persistentKeepalive = 25;
}) (removeAttrs mesh [ self ]);
};
```
To set it up:
1. On each machine, create the private key in a file only root can read. It
never goes in the Nix store or in the configuration:
```bash
install -d -m 0700 /etc/wireguard
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key
wg pubkey < /etc/wireguard/wg0.key
```
2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public
keys aren't secret, so the file is safe to commit. You do this once.
3. Set the `address` and `endpoint` of each machine in `mesh.nix`. The example
uses documentation addresses.
4. Deploy. `wg show wg0` on any machine lists four peers with a recent
handshake, and `ping 10.77.0.1` answers.
A machine that can't take incoming connections, such as one behind NAT, sets
`endpoint = null`. It dials the others, and the keepalive holds the NAT mapping
open.
**Or use tailscale, headscale or NetBird.** They hand out the keys and the
addresses for you, so the key steps above go away. Three things change:
- The interface is no longer `wg0`. Tailscale's is `tailscale0`
(`services.tailscale.interfaceName`). Put its name in the firewall rules in
place of `wg0`.
- Drop `wireguard.nix`, and put each machine's address from your VPN into the
`address` field of `mesh.nix`. The DNS table below reads only that field.
- Enable the service, for example `services.tailscale.enable = true` with
`services.tailscale.authKeyFile` pointing at a key file outside the store.
## Reading the store
Only `swarm-central` runs the secret store, and every service on the other
machines reads it with a client certificate that the store's host mints. The
example names each leaf under `/var/lib/swarm-pki/` on the machine that uses
it. After the first deploy of `swarm-central`, copy the files from
`/var/lib/swarm-bao-pki/` there over the VPN (for a hive, from
`/var/lib/swarm-bao-pki/hives/<name>/`), then deploy the other machines. The
private keys travel over a channel you trust and stay readable by root only.
[`credentials.md`](credentials.md) lists which leaf each service reads and what
a missing one costs. The first-run steps after the deploys are in
[`setup.md`](../../1-getting-started/setup.md).
## DNS
Every service reaches another by name, so every name has to resolve on every
machine and inside every container. Two views of it exist.
**The public view** is a handful of records at your DNS provider, pointing at
the two public machines:
| name | address |
| -------------------------------------- | --------------------------------- |
| `<swarm.domain>` | `swarm-central`'s public address |
| `forge.<swarm.domain>` | `swarm-services`'s public address |
| `chat.<swarm.domain>` (the homeserver) | `swarm-services`'s public address |
| `auth.<swarm.domain>` | `swarm-services`'s public address |
Matrix federation and browsers use these. Names you leave out of public DNS
(`bao.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
hives' dashboards) don't resolve from outside.
**The inside view** maps every swarm name to the VPN address of the machine
that serves it, including the public names, so traffic between machines
never uses the internet. The example does it with a static table in `dns.nix`,
written on every machine and built from the same options the services use, so
a renamed `swarm.*.domain` follows. Two readers need it:
- Processes on the machine read `/etc/hosts` (`networking.hosts`).
- Containers (agents and swarm services alike) ask the hive's resolver
(dnsmasq). It ignores `/etc/hosts`, so the example hands it the same table
through `services.dnsmasq.settings.addn-hosts`.
A machine leaves out the names it serves itself. hyperhive answers those
already, and a table entry would override its answer.
The trade-off: a static table needs no DNS server and no extra moving part, and
it lives in the same repository as the rest. It also means a new hive adds a
line on every machine, and a name that moves from one machine to another is a
rebuild everywhere. For a swarm with many hives, run a small DNS server on the
VPN instead, answering the swarm domain with VPN addresses, and point each
machine's upstream resolver at it for that domain. That's one place to edit,
at the cost of one more service every lookup depends on.
To open Grafana, the store's browser UI or a hive dashboard yourself, join the
VPN with your own machine: add it as a peer on each machine and give it the
same names, for example in its own `/etc/hosts`.
## Not in the example
- **TLS certificates the public can trust.** Without `gateway.tls.acme` or
`gateway.tls.certDir`, the gateway serves certificates from the swarm's own
CA, and browsers warn. Set one on `swarm-central` and `swarm-services`.
- **SSH and the rest of your hosts' base setup.** The example touches only the
firewall ports it needs.
- **The hardware configuration.** Each machine's own file sets
`networking.hostName` to one of the five names, which picks its VPN entry.
Wiring one machine into a flake looks like this:
```nix
nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
hyperhive.nixosModules.default
./hardware/swarm-central.nix # sets networking.hostName = "swarm-central"
./semi-spread/common.nix
./semi-spread/swarm-central.nix
];
};
```
`hive-one` and `hive-two` add `./semi-spread/hive.nix` instead of a
machine-specific file.

3
docs/2-handbook/swarm/services.md
View file

@ -80,6 +80,9 @@ the shapes worth naming are the ones they produce:
`allSwarmServices`. `allSwarmServices`.
- **Fully spread out.** One container / VM / machine per service, - **Fully spread out.** One container / VM / machine per service,
somewhere. somewhere.
- **Semi-spread.** The services split over a few machines by weight, hives on
their own, and a VPN between them. A worked example with its NixOS files:
[`semi-spread-deploy.md`](semi-spread-deploy.md).
**These are a set, not a ladder with a correct top, and hyperhive **These are a set, not a ladder with a correct top, and hyperhive
supports in-between shapes.** You can set each derived toggle on its own (see supports in-between shapes.** You can set each derived toggle on its own (see

3
docs/README.md
View file

@ -32,6 +32,9 @@ The services that exist once per swarm. Start at
- **How are two hives connected into a swarm?** → [`2-handbook/swarm/`](2-handbook/swarm/README.md) - **How are two hives connected into a swarm?** → [`2-handbook/swarm/`](2-handbook/swarm/README.md)
(peer hives, TLS trust). (peer hives, TLS trust).
- **How to spread the swarm over a few machines, with a VPN and only the forge,
matrix and swarm UI public?** → [`2-handbook/swarm/semi-spread-deploy.md`](2-handbook/swarm/semi-spread-deploy.md)
(worked example: machines, WireGuard, DNS).
- **What does the swarm UI show, and what does `swarmctl` do?** → - **What does the swarm UI show, and what does `swarmctl` do?** →
[`2-handbook/swarm/ui.md`](2-handbook/swarm/ui.md) and [`2-handbook/swarm/ui.md`](2-handbook/swarm/ui.md) and
[`2-handbook/swarm/swarmctl-cli.md`](2-handbook/swarm/swarmctl-cli.md) (the [`2-handbook/swarm/swarmctl-cli.md`](2-handbook/swarm/swarmctl-cli.md) (the

21
docs/examples/semi-spread/common.nix Normal file
View file

@ -0,0 +1,21 @@
# Imported by every machine in the swarm: the swarm's shared facts, the VPN and
# the names inside it. `services.hyperhive.swarm.*` has to be identical on all
# of them, so it lives here and nowhere else.
{
imports = [
./wireguard.nix
./dns.nix
];
services.hyperhive.swarm = {
domain = "myswarm.example";
# Every hive, including the one a machine does not run. Names only: each
# hive's domain defaults to `<name>.myswarm.example`. A hive name may not
# contain "hive", "agent" or "swarm", so the machines `hive-one` and
# `hive-two` run the hives `one` and `two`.
hives = {
one = { };
two = { };
};
};
}

65
docs/examples/semi-spread/dns.nix Normal file
View file

@ -0,0 +1,65 @@
# Names inside the VPN. Every swarm name that is not public resolves to the
# VPN address of the machine serving it, through a static table written on
# each host. Identical on every host.
#
# Two readers need the table. Processes on the host read /etc/hosts
# (`networking.hosts`). Containers (agents, swarm services) ask the hive's
# dnsmasq, which ignores /etc/hosts, so it also reads the table through
# `addn-hosts`.
#
# A host leaves out the names it serves itself: hyperhive already answers
# those with the bridge address, and a table entry would override it.
{
config,
lib,
...
}:
let
mesh = import ./mesh.nix;
self = config.networking.hostName;
swarm = config.services.hyperhive.swarm;
# Which machine serves which name. A hive's machine is `hive-<name>` and it
# serves the hive's own names: its dashboard and its domain.
servedBy =
lib.mapAttrs' (
hive: entry:
lib.nameValuePair "hive-${hive}" [
swarm.dashboardDomains.${hive}
entry.domain
]
) swarm.hives
// {
swarm-central = [
swarm.domain
swarm.bao.domain
swarm.bao.ui.domain
];
swarm-services = [
swarm.forge.domain
swarm.matrix.gatewayHost
swarm.authelia.domain
swarm.nats.domain
swarm.otel.domain
swarm.grafana.domain
swarm.victoriametrics.domain
swarm.victorialogs.domain
];
swarm-heavyduty = [ swarm.attic.ownDomain ];
};
elsewhere = removeAttrs servedBy [ self ];
in
{
networking.hosts = lib.mapAttrs' (
machine: names: lib.nameValuePair mesh.${machine}.address names
) elsewhere;
services.dnsmasq.settings.addn-hosts = "${builtins.toFile "swarm-vpn-hosts" (
lib.concatStringsSep "\n" (
lib.mapAttrsToList (
machine: names: "${mesh.${machine}.address} ${lib.concatStringsSep " " names}"
) elsewhere
)
)}";
}

28
docs/examples/semi-spread/hive.nix Normal file
View file

@ -0,0 +1,28 @@
# A hive. Identical on hive-one and hive-two: the machine `hive-<name>` runs the
# hive `<name>`.
{ config, lib, ... }:
let
# The leaves for this hive, copied from swarm-central, where the secret store
# runs: `client.pem` and `client-key.pem` from
# `/var/lib/swarm-bao-pki/hives/<name>/`, along with that directory's
# `hive-leaf.pem` and `hive-leaf-key.pem`, which the hive's gateway
# certificate comes from. Both pairs sit in one directory.
pki = "/var/lib/swarm-pki";
in
{
services.hyperhive.deploy = {
hive-controller = {
enable = true;
hiveName = lib.removePrefix "hive-" config.networking.hostName;
};
bao = {
clientCertFile = "${pki}/client.pem";
clientKeyFile = "${pki}/client-key.pem";
};
};
# The dashboard is reachable over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [
config.services.hyperhive.gateway.httpsPort
];
}

33
docs/examples/semi-spread/mesh.nix Normal file
View file

@ -0,0 +1,33 @@
# The VPN's facts: one entry per machine, read by every host (wireguard.nix,
# dns.nix) and nothing else. Keyed by `networking.hostName`.
#
# `address` is the machine's address inside the VPN; `endpoint` is where the
# other machines dial it; `publicKey` is the output of `wg pubkey` on that
# machine. The keys below are placeholders: replace each with the real one.
{
swarm-central = {
address = "10.77.0.1";
endpoint = "203.0.113.10:51820";
publicKey = "REPLACE-with-swarm-central-wg-pubkey";
};
swarm-services = {
address = "10.77.0.2";
endpoint = "203.0.113.11:51820";
publicKey = "REPLACE-with-swarm-services-wg-pubkey";
};
swarm-heavyduty = {
address = "10.77.0.3";
endpoint = "203.0.113.12:51820";
publicKey = "REPLACE-with-swarm-heavyduty-wg-pubkey";
};
hive-one = {
address = "10.77.0.11";
endpoint = "198.51.100.11:51820";
publicKey = "REPLACE-with-hive-one-wg-pubkey";
};
hive-two = {
address = "10.77.0.12";
endpoint = "198.51.100.12:51820";
publicKey = "REPLACE-with-hive-two-wg-pubkey";
};
}

23
docs/examples/semi-spread/swarm-central.nix Normal file
View file

@ -0,0 +1,23 @@
# swarm-central: the swarm controller (and the swarm UI it serves) and the
# secret store.
{ config, ... }:
let
mesh = import ./mesh.nix;
hyperhive = config.services.hyperhive;
in
{
services.hyperhive = {
deploy.swarm-controller.enable = true;
deploy.bao.enable = true;
# The store also listens on this machine's VPN address, for the services
# and hives on other machines.
deploy.bao.extraListenAddresses = [ mesh.swarm-central.address ];
# The swarm UI is public.
gateway.openFirewall = true;
};
# The store is reachable over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.swarm.bao.port ];
}

31
docs/examples/semi-spread/swarm-heavyduty.nix Normal file
View file

@ -0,0 +1,31 @@
# swarm-heavyduty: the builder, the forge's CI runner and the binary cache.
{ config, lib, ... }:
let
hyperhive = config.services.hyperhive;
# Leaves copied from swarm-central, where the secret store runs. See
# swarm-services.nix for the helper's naming.
pki = "/var/lib/swarm-pki";
pair = option: stem: {
"${option}ClientCertFile" = "${pki}/${stem}.pem";
"${option}ClientKeyFile" = "${pki}/${stem}-key.pem";
};
in
{
services.hyperhive = {
deploy.builder.enable = true;
deploy.ci-runner.enable = true;
deploy.attic.enable = true;
deploy.bao = lib.mergeAttrsList [
(pair "builder" "builder")
(pair "ciRunner" "ci-runner")
(pair "attic" "attic")
(pair "forwarderOidc" "forwarder-oidc")
(pair "servicesIssuer" "services-issuer")
];
};
# The cache is served through the gateway, over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.gateway.httpsPort ];
}

64
docs/examples/semi-spread/swarm-services.nix Normal file
View file

@ -0,0 +1,64 @@
# swarm-services: the forge, the matrix homeserver, SSO, the queue, and the
# metrics and log stack.
{ config, lib, ... }:
let
hyperhive = config.services.hyperhive;
# The secret store runs on swarm-central, so each service here reads it with
# a leaf copied from there. `pair "grafanaOidc" "grafana-oidc"` names the
# options `deploy.bao.grafanaOidcClientCertFile` and `…KeyFile` and the files
# `grafana-oidc.pem` and `grafana-oidc-key.pem`.
pki = "/var/lib/swarm-pki";
pair = option: stem: {
"${option}ClientCertFile" = "${pki}/${stem}.pem";
"${option}ClientKeyFile" = "${pki}/${stem}-key.pem";
};
in
{
services.hyperhive = {
# Everything once-per-swarm that is not placed elsewhere below.
deploy.allSwarmServices = true;
# The forge and the matrix homeserver are public, and so is the SSO login
# they send browsers to.
gateway.openFirewall = true;
deploy.bao = lib.mergeAttrsList [
{ enable = false; }
(pair "forgePublisher" "forge-publisher")
(pair "servicesIssuer" "services-issuer")
(pair "grafanaOidc" "grafana-oidc")
(pair "otelOidc" "otel-oidc")
(pair "forwarderOidc" "forwarder-oidc")
(pair "reviewClassifier" "review-classifier")
(pair "notifyRouter" "notify-router")
(pair "claudeLogin" "claude-login")
(pair "ciMetrics" "ci-metrics")
];
deploy.matrix = {
ctlBaoClientCertFile = "${pki}/matrix-ctl.pem";
ctlBaoClientKeyFile = "${pki}/matrix-ctl-key.pem";
};
deploy.nats = {
baoClientCertFile = "${pki}/nats.pem";
baoClientKeyFile = "${pki}/nats-key.pem";
authBaoClientCertFile = "${pki}/nats-auth.pem";
authBaoClientKeyFile = "${pki}/nats-auth-key.pem";
};
deploy.swarm-secret-publisher = {
baoClientCertFile = "${pki}/secret-publisher.pem";
baoClientKeyFile = "${pki}/secret-publisher-key.pem";
};
deploy.victoriametrics = {
authBaoClientCertFile = "${pki}/metrics-auth.pem";
authBaoClientKeyFile = "${pki}/metrics-auth-key.pem";
};
deploy.victorialogs = {
authBaoClientCertFile = "${pki}/logs-auth.pem";
authBaoClientKeyFile = "${pki}/logs-auth-key.pem";
};
};
# The queue is reachable over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.swarm.nats.port ];
}

24
docs/examples/semi-spread/wireguard.nix Normal file
View file

@ -0,0 +1,24 @@
# The VPN: plain NixOS WireGuard, one interface `wg0`, a full mesh built from
# ./mesh.nix. Identical on every host; the host's own entry is picked by
# `networking.hostName`.
{ config, lib, ... }:
let
mesh = import ./mesh.nix;
self = config.networking.hostName;
in
{
networking.wireguard.interfaces.wg0 = {
ips = [ "${mesh.${self}.address}/24" ];
listenPort = 51820;
# Created on the host by `wg genkey`, never in the store.
privateKeyFile = "/etc/wireguard/wg0.key";
peers = lib.mapAttrsToList (_: peer: {
inherit (peer) publicKey endpoint;
allowedIPs = [ "${peer.address}/32" ];
persistentKeepalive = 25;
}) (removeAttrs mesh [ self ]);
};
# The handshake arrives on the public interface.
networking.firewall.allowedUDPPorts = [ config.networking.wireguard.interfaces.wg0.listenPort ];
}

7
nix/checks.nix
View file

@ -203,6 +203,13 @@ in
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;
}; };
# The five machines of docs/examples/semi-spread/. Five full-system
# fixtures, because the claim is that every machine of the example
# evaluates; no case of another cluster can stand in for one.
module-eval-semi-spread = import ./module-eval/semi-spread.nix {
inherit pkgs self nixosSystem;
inherit (pkgs) lib;
};
module-eval-agent-queue-bao = import ./module-eval/agent-queue-bao.nix { module-eval-agent-queue-bao = import ./module-eval/agent-queue-bao.nix {
inherit pkgs self nixosSystem; inherit pkgs self nixosSystem;
inherit (pkgs) lib; inherit (pkgs) lib;

345
nix/module-eval/semi-spread.nix Normal file
View file

@ -0,0 +1,345 @@
# `checks.module-eval-semi-spread` — see ./lib.nix for the shared rationale
# (why this suite exists, naming convention, "evaluates not executes").
#
# Evaluates the five machines of docs/examples/semi-spread/ exactly as that
# directory's files describe them, so the example the handbook points at
# cannot stop evaluating, and pins the shape the handbook page promises: which
# machine runs what, what is open on the public interface, and what only on the
# VPN.
{
pkgs,
lib,
self,
nixosSystem,
}:
let
inherit
(import ./lib.nix {
inherit
pkgs
lib
self
nixosSystem
;
})
baoSettings
runGroup
;
example = self + "/docs/examples/semi-spread";
# What the example's files leave to the machine's own configuration: its
# host name, which picks the VPN entry and, for a hive, the hive. `role` is
# the file the machine adds to common.nix.
machine =
hostName: role:
(nixosSystem {
modules = [
(pkgs.path + "/nixos/modules/misc/nixpkgs/read-only.nix")
{ nixpkgs.pkgs = pkgs; }
self.nixosModules.default
{
fileSystems."/" = {
device = "/dev/null";
fsType = "tmpfs";
};
boot.loader.grub.enable = false;
system.stateVersion = "25.11";
networking.hostName = hostName;
}
(example + "/common.nix")
(example + "/${role}.nix")
];
}).config;
central = machine "swarm-central" "swarm-central";
services = machine "swarm-services" "swarm-services";
heavyduty = machine "swarm-heavyduty" "swarm-heavyduty";
hiveOne = machine "hive-one" "hive";
hiveTwo = machine "hive-two" "hive";
all = {
swarm-central = central;
swarm-services = services;
swarm-heavyduty = heavyduty;
hive-one = hiveOne;
hive-two = hiveTwo;
};
mesh = import (example + "/mesh.nix");
# What `system.build.toplevel` checks before it builds anything: every module
# assertion holds. The example also renders no warning. Evaluating the
# toplevel itself would add the package set of five full systems to every
# run of this suite.
evaluates =
m:
let
r = builtins.tryEval (lib.all (a: a.assertion) m.assertions && m.warnings == [ ]);
in
r.success && r.value;
# The TCP ports open on every interface, and those open on `wg0` alone.
publicTcp = m: m.networking.firewall.allowedTCPPorts;
vpnTcp = m: (m.networking.firewall.interfaces.wg0 or { allowedTCPPorts = [ ]; }).allowedTCPPorts;
deploy = m: m.services.hyperhive.deploy;
swarmOf = m: m.services.hyperhive.swarm;
https = m: m.services.hyperhive.gateway.httpsPort;
# The names a machine's `/etc/hosts` and its resolver's extra hosts file
# resolve to the VPN, as a set of `address -> names`.
hostsOf = m: m.networking.hosts;
resolverTable =
m: lib.concatMapStrings builtins.readFile (m.services.dnsmasq.settings.addn-hosts or [ ]);
names = m: lib.concatLists (lib.attrValues (lib.filterAttrs (a: _: a != "127.0.0.2") (hostsOf m)));
vhosts = m: builtins.attrNames m.services.nginx.virtualHosts;
cases = [
# Every machine evaluates with its assertions holding. `tryEval`
# turns a refusal into a failed case that names the machine.
{
name = "swarm-central evaluates with its assertions holding";
ok = evaluates central;
}
{
name = "swarm-services evaluates with its assertions holding";
ok = evaluates services;
}
{
name = "swarm-heavyduty evaluates with its assertions holding";
ok = evaluates heavyduty;
}
{
name = "hive-one evaluates with its assertions holding";
ok = evaluates hiveOne;
}
{
name = "hive-two evaluates with its assertions holding";
ok = evaluates hiveTwo;
}
# Who runs what. Each machine's list is exact: a service that moved would
# show here on the machine that gained it and the one that lost it.
{
name = "swarm-central runs the controller and the store, and none of the services placed elsewhere";
ok =
let
d = deploy central;
in
d.swarm-controller.enable
&& d.bao.enable
&& !d.forgejo.enable
&& !d.matrix.enable
&& !d.nats.enable
&& !d.attic.enable
&& !d.builder.enable
&& !d.hive-controller.enable;
}
{
name = "swarm-services runs the forge, matrix, SSO, queue and the metrics and log stack, not the store";
ok =
let
d = deploy services;
in
lib.all (x: x) [
d.forgejo.enable
d.matrix.enable
d.authelia.enable
d.nats.enable
d.swarm-otel.enable
d.victoriametrics.enable
d.grafana.enable
d.victorialogs.enable
d.review-classifier.enable
d.notify-router.enable
d.claude-login.enable
d.ci-metrics.enable
(!d.bao.enable)
(!d.swarm-controller.enable)
(!d.attic.enable)
(!d.builder.enable)
(!d.ci-runner.enable)
(!d.hive-controller.enable)
];
}
{
name = "swarm-heavyduty runs the builder, the CI runner and the cache, and nothing else";
ok =
let
d = deploy heavyduty;
in
d.builder.enable
&& d.ci-runner.enable
&& d.attic.enable
&& !d.forgejo.enable
&& !d.matrix.enable
&& !d.nats.enable
&& !d.bao.enable
&& !d.swarm-controller.enable
&& !d.hive-controller.enable;
}
{
name = "hive-one and hive-two run a hive each, under different names, and no shared service";
ok =
let
d1 = deploy hiveOne;
d2 = deploy hiveTwo;
none =
d:
!(
d.forgejo.enable || d.matrix.enable || d.nats.enable || d.bao.enable || d.swarm-controller.enable
);
in
d1.hive-controller.enable
&& d2.hive-controller.enable
&& d1.hive-controller.hiveName == "one"
&& d2.hive-controller.hiveName == "two"
&& none d1
&& none d2;
}
{
# `swarm.*` is the one part that has to be identical, and it comes from a
# single file; the case reads it back from every machine.
name = "every machine holds the same swarm name, hives and service names";
ok =
let
view = m: {
inherit (swarmOf m) domain serviceDomains;
hives = builtins.attrNames (swarmOf m).hives;
};
in
lib.all (m: view m == view central) (lib.attrValues all);
}
# The public interface. The machines serving the forge, matrix and the
# swarm UI open the gateway to everyone; the others open nothing but the
# VPN's own port.
{
name = "swarm-central and swarm-services open the gateway on every interface";
ok =
publicTcp central == [
80
443
]
&&
publicTcp services == [
80
443
];
}
{
# Presence control for the case above: the same read finds the ports
# where they are open, so an empty list here means closed.
name = "swarm-heavyduty and the hives open no TCP port on every interface";
ok =
publicTcp heavyduty == [ ]
&& publicTcp hiveOne == [ ]
&& publicTcp hiveTwo == [ ]
&& vpnTcp heavyduty == [ (https heavyduty) ]
&& vpnTcp hiveOne == [ (https hiveOne) ]
&& vpnTcp hiveTwo == [ (https hiveTwo) ];
}
{
name = "every machine opens only the VPN's UDP port on every interface";
ok = lib.all (m: m.networking.firewall.allowedUDPPorts == [ 51820 ]) (lib.attrValues all);
}
{
name = "swarm-central serves the swarm UI and swarm-services serves the forge and matrix";
ok =
lib.elem (swarmOf central).domain (vhosts central)
&& lib.elem (swarmOf services).forge.domain (vhosts services)
&& lib.elem (swarmOf services).matrix.gatewayHost (vhosts services)
# The cache is on a machine with nothing public.
&& lib.elem (swarmOf heavyduty).attic.ownDomain (vhosts heavyduty)
&& !lib.elem (swarmOf heavyduty).attic.ownDomain (vhosts services);
}
# What only the VPN reaches.
{
name = "the secret store's port is open on wg0 and on no other external interface";
ok =
let
port = (swarmOf central).bao.port;
in
vpnTcp central == [ port ] && !lib.elem port (publicTcp central);
}
{
name = "the secret store listens on its VPN address and requires a client certificate there";
ok =
let
l = (baoSettings central).listener.extra-1 or { };
in
(l.address or null) == "${mesh.swarm-central.address}:${toString (swarmOf central).bao.port}"
&& (l.tls_require_and_verify_client_cert or false);
}
{
name = "the queue's port is open on wg0 and on no other external interface";
ok =
let
port = (swarmOf services).nats.port;
in
lib.elem port (vpnTcp services) && !lib.elem port (publicTcp services);
}
# The VPN itself.
{
name = "every machine peers with the other four over wg0, with its key outside the store";
ok = lib.all (
m:
let
wg = m.networking.wireguard.interfaces.wg0;
in
builtins.length wg.peers == 4
&& wg.privateKey == null
&& wg.privateKeyFile != null
&& !lib.hasPrefix "/nix/store" wg.privateKeyFile
) (lib.attrValues all);
}
{
name = "each machine takes its own address from the shared VPN table";
ok = lib.all (
n: lib.elem "${mesh.${n}.address}/24" all.${n}.networking.wireguard.interfaces.wg0.ips
) (builtins.attrNames all);
}
# DNS. A machine lists the names others serve, at those machines' VPN
# addresses, and leaves out its own.
{
name = "a hive resolves the store, queue and cache names to the machines that serve them";
ok =
let
t = hostsOf hiveOne;
in
lib.elem (swarmOf hiveOne).bao.domain t.${mesh.swarm-central.address}
&& lib.elem (swarmOf hiveOne).nats.domain t.${mesh.swarm-services.address}
&& lib.elem (swarmOf hiveOne).attic.ownDomain t.${mesh.swarm-heavyduty.address};
}
{
name = "a machine leaves out the names it serves itself";
ok =
!lib.elem (swarmOf central).bao.domain (names central)
&& !lib.elem (swarmOf services).nats.domain (names services)
&& !lib.elem (swarmOf heavyduty).attic.ownDomain (names heavyduty)
&& !lib.elem "one.hives.${(swarmOf hiveOne).domain}" (names hiveOne)
# Control: another hive's dashboard is in the table.
&& lib.elem "two.hives.${(swarmOf hiveOne).domain}" (names hiveOne);
}
{
# Containers ask the hive's resolver, which ignores /etc/hosts, so the
# same table has to reach it through its extra hosts file.
name = "the hive resolver carries the same table as /etc/hosts";
ok =
let
table = resolverTable hiveOne;
in
hostsOf hiveOne != { }
&& lib.all (
address: lib.hasInfix "${address} ${lib.concatStringsSep " " (hostsOf hiveOne).${address}}" table
) (builtins.attrNames (lib.filterAttrs (a: _: a != "127.0.0.2") (hostsOf hiveOne)))
# Control: the file is not one every machine shares.
&& !lib.hasInfix (swarmOf central).bao.domain (resolverTable central);
}
];
in
runGroup "semi-spread" cases