diff --git a/AGENTS.md b/AGENTS.md index 992d21740..60e1d7f27 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -200,7 +200,9 @@ Swarm-level: module-eval tests), `builder/` (`check-flake`), `script-tests/`, plus `sources.nix` / `rust.nix` / `checks.nix` / `devshell.nix` / `treefmt.nix` behind the thin `flake.nix`. -- **`docs/`** — subsystem reference docs (see *Reading paths* below). +- **`docs/`** — subsystem reference docs (see *Reading paths* below), and + `examples/`, the NixOS files the handbook's worked examples use. CI + evaluates them. - **`branding/`**, **`scripts/`** — static assets + helper scripts. ## Reading paths diff --git a/docs/1-getting-started/nixos.md b/docs/1-getting-started/nixos.md index d45a81117..f49784757 100644 --- a/docs/1-getting-started/nixos.md +++ b/docs/1-getting-started/nixos.md @@ -107,6 +107,7 @@ Leave the other options at their defaults until you need them: - Spreading a swarm over more than one host, and the credentials a split swarm needs: [`2-handbook/swarm/README.md`](../2-handbook/swarm/README.md) and [`2-handbook/swarm/credentials.md`](../2-handbook/swarm/credentials.md). + A worked example: [`2-handbook/swarm/semi-spread-deploy.md`](../2-handbook/swarm/semi-spread-deploy.md). - Every option, with its type and default: [the options reference](https://hyperhive.darkest.space/options/). - The secret store's seal and the granter: [`4-implementation/swarm/bao.md`](../4-implementation/swarm/bao.md). diff --git a/docs/2-handbook/swarm/semi-spread-deploy.md b/docs/2-handbook/swarm/semi-spread-deploy.md new file mode 100644 index 000000000..3dd13f3d0 --- /dev/null +++ b/docs/2-handbook/swarm/semi-spread-deploy.md @@ -0,0 +1,224 @@ +# Example: a semi-spread deploy + +Three swarm machines and two hives, joined by a WireGuard VPN, with only the +forge, the matrix homeserver and the swarm UI reachable from the internet. It +is one of the in-between shapes that [`services.md`](services.md#deployment-shapes) +supports: the swarm's services run on three machines, not on one box and not +one per service. + +The NixOS files are in +[`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all +five machines (`checks.module-eval-semi-spread`) and pins the facts on this +page, so the files stay in step with the options. + +## The machines + +| machine | runs | reachable from the internet | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | +| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI | +| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | forge, matrix, SSO login | +| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing | +| `hive-one` | the hive `one` | nothing | +| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing | + +Why this split: + +- **`swarm-central`** holds what every other machine depends on, and little + else that can fail. The secret store and the controller are small. +- **`swarm-services`** holds the services people and agents talk to all day. + They're light, and they need the public name and port. +- **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder + evaluates config pull requests nobody reviewed yet, and the CI runner runs + pull-request code. Keeping both off the machine that holds the secret store + limits what a bad build can reach. +- **The hives** run the agents. They need nothing from the internet but the + agents' own egress. + +A hive name can't contain `hive`, `agent` or `swarm`, so the machines +`hive-one` and `hive-two` run the hives `one` and `two`. `hive.nix` takes the +hive name from the machine's host name, which is why one file serves both. + +## What's public and what's not + +`gateway.openFirewall` opens the gateway's ports (80 and 443) on every +interface, and the example sets it on two machines only: + +- `swarm-central` serves the swarm UI at ``. +- `swarm-services` serves the forge (`forge.`) and the matrix + homeserver (`chat.`). It also serves `auth.`, + because the forge, matrix and the swarm UI send browsers there to sign in. + +**Public means the machine's gateway port is open.** A gateway is one nginx +for everything the machine serves, so the other names on those two machines +(Grafana, metrics, logs, the collector, the store's browser UI) answer on the +public address too. Each sits behind SSO or a bearer token, and none of them is +in public DNS, but they're not closed. A service you want closed goes on a +machine that opens nothing public, like `swarm-heavyduty`. + +Everything else uses the VPN. These ports open on `wg0` only, with +`networking.firewall.interfaces.wg0.allowedTCPPorts`: + +| machine | port on `wg0` | for | +| ----------------- | ------------------------- | --------------------------------------------- | +| `swarm-central` | `swarm.bao.port` (8200) | the secret store, read by every other machine | +| `swarm-services` | `swarm.nats.port` (4222) | the queue | +| `swarm-heavyduty` | `gateway.httpsPort` (443) | the cache, behind the gateway | +| each hive | `gateway.httpsPort` (443) | the hive's dashboard and gateway | + +The example scopes these to `wg0` because the options that open a service's +port, `gateway.openFirewall`, `deploy.bao.openFirewall` and +`deploy.nats.openFirewall`, open it on every interface. The +secret store and the queue don't rely on the VPN for their security: the store +requires a client certificate, and the queue requires TLS and refuses clients +it can't identify. The VPN keeps them off the internet as well. + +Every machine also opens UDP 51820 on every interface, because that's where +the VPN's handshake arrives. The example leaves SSH to you. + +```text + internet + 80/443 | | 80/443 UDP 51820 on all five + | | + swarm-central swarm-services swarm-heavyduty hive-one hive-two + 10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12 + | | | | | + +----------------+----- wg0, 10.77.0.0/24 --------+----------+ +``` + +## The VPN: raw WireGuard + +`wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on +all five machines. `mesh.nix` holds each machine's VPN address, public endpoint +and public key. The file picks the machine's own entry by `networking.hostName` +and makes the other four its peers: + +```nix +networking.wireguard.interfaces.wg0 = { + ips = [ "${mesh.${self}.address}/24" ]; + listenPort = 51820; + privateKeyFile = "/etc/wireguard/wg0.key"; + peers = lib.mapAttrsToList (_: peer: { + inherit (peer) publicKey endpoint; + allowedIPs = [ "${peer.address}/32" ]; + persistentKeepalive = 25; + }) (removeAttrs mesh [ self ]); +}; +``` + +To set it up: + +1. On each machine, create the private key in a file only root can read. It + never goes in the Nix store or in the configuration: + + ```bash + install -d -m 0700 /etc/wireguard + wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key + wg pubkey < /etc/wireguard/wg0.key + ``` + +2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public + keys aren't secret, so the file is safe to commit. You do this once. +3. Set the `address` and `endpoint` of each machine in `mesh.nix`. The example + uses documentation addresses. +4. Deploy. `wg show wg0` on any machine lists four peers with a recent + handshake, and `ping 10.77.0.1` answers. + +A machine that can't take incoming connections, such as one behind NAT, sets +`endpoint = null`. It dials the others, and the keepalive holds the NAT mapping +open. + +**Or use tailscale, headscale or NetBird.** They hand out the keys and the +addresses for you, so the key steps above go away. Three things change: + +- The interface is no longer `wg0`. Tailscale's is `tailscale0` + (`services.tailscale.interfaceName`). Put its name in the firewall rules in + place of `wg0`. +- Drop `wireguard.nix`, and put each machine's address from your VPN into the + `address` field of `mesh.nix`. The DNS table below reads only that field. +- Enable the service, for example `services.tailscale.enable = true` with + `services.tailscale.authKeyFile` pointing at a key file outside the store. + +## Reading the store + +Only `swarm-central` runs the secret store, and every service on the other +machines reads it with a client certificate that the store's host mints. The +example names each leaf under `/var/lib/swarm-pki/` on the machine that uses +it. After the first deploy of `swarm-central`, copy the files from +`/var/lib/swarm-bao-pki/` there over the VPN (for a hive, from +`/var/lib/swarm-bao-pki/hives//`), then deploy the other machines. The +private keys travel over a channel you trust and stay readable by root only. +[`credentials.md`](credentials.md) lists which leaf each service reads and what +a missing one costs. The first-run steps after the deploys are in +[`setup.md`](../../1-getting-started/setup.md). + +## DNS + +Every service reaches another by name, so every name has to resolve on every +machine and inside every container. Two views of it exist. + +**The public view** is a handful of records at your DNS provider, pointing at +the two public machines: + +| name | address | +| -------------------------------------- | --------------------------------- | +| `` | `swarm-central`'s public address | +| `forge.` | `swarm-services`'s public address | +| `chat.` (the homeserver) | `swarm-services`'s public address | +| `auth.` | `swarm-services`'s public address | + +Matrix federation and browsers use these. Names you leave out of public DNS +(`bao.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the +hives' dashboards) don't resolve from outside. + +**The inside view** maps every swarm name to the VPN address of the machine +that serves it, including the public names, so traffic between machines +never uses the internet. The example does it with a static table in `dns.nix`, +written on every machine and built from the same options the services use, so +a renamed `swarm.*.domain` follows. Two readers need it: + +- Processes on the machine read `/etc/hosts` (`networking.hosts`). +- Containers (agents and swarm services alike) ask the hive's resolver + (dnsmasq). It ignores `/etc/hosts`, so the example hands it the same table + through `services.dnsmasq.settings.addn-hosts`. + +A machine leaves out the names it serves itself. hyperhive answers those +already, and a table entry would override its answer. + +The trade-off: a static table needs no DNS server and no extra moving part, and +it lives in the same repository as the rest. It also means a new hive adds a +line on every machine, and a name that moves from one machine to another is a +rebuild everywhere. For a swarm with many hives, run a small DNS server on the +VPN instead, answering the swarm domain with VPN addresses, and point each +machine's upstream resolver at it for that domain. That's one place to edit, +at the cost of one more service every lookup depends on. + +To open Grafana, the store's browser UI or a hive dashboard yourself, join the +VPN with your own machine: add it as a peer on each machine and give it the +same names, for example in its own `/etc/hosts`. + +## Not in the example + +- **TLS certificates the public can trust.** Without `gateway.tls.acme` or + `gateway.tls.certDir`, the gateway serves certificates from the swarm's own + CA, and browsers warn. Set one on `swarm-central` and `swarm-services`. +- **SSH and the rest of your hosts' base setup.** The example touches only the + firewall ports it needs. +- **The hardware configuration.** Each machine's own file sets + `networking.hostName` to one of the five names, which picks its VPN entry. + +Wiring one machine into a flake looks like this: + +```nix +nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + hyperhive.nixosModules.default + ./hardware/swarm-central.nix # sets networking.hostName = "swarm-central" + ./semi-spread/common.nix + ./semi-spread/swarm-central.nix + ]; +}; +``` + +`hive-one` and `hive-two` add `./semi-spread/hive.nix` instead of a +machine-specific file. diff --git a/docs/2-handbook/swarm/services.md b/docs/2-handbook/swarm/services.md index b997dfdff..e82697672 100644 --- a/docs/2-handbook/swarm/services.md +++ b/docs/2-handbook/swarm/services.md @@ -80,6 +80,9 @@ the shapes worth naming are the ones they produce: `allSwarmServices`. - **Fully spread out.** One container / VM / machine per service, somewhere. +- **Semi-spread.** The services split over a few machines by weight, hives on + their own, and a VPN between them. A worked example with its NixOS files: + [`semi-spread-deploy.md`](semi-spread-deploy.md). **These are a set, not a ladder with a correct top, and hyperhive supports in-between shapes.** You can set each derived toggle on its own (see diff --git a/docs/README.md b/docs/README.md index ae8e351c3..17651286c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -32,6 +32,9 @@ The services that exist once per swarm. Start at - **How are two hives connected into a swarm?** → [`2-handbook/swarm/`](2-handbook/swarm/README.md) (peer hives, TLS trust). +- **How to spread the swarm over a few machines, with a VPN and only the forge, + matrix and swarm UI public?** → [`2-handbook/swarm/semi-spread-deploy.md`](2-handbook/swarm/semi-spread-deploy.md) + (worked example: machines, WireGuard, DNS). - **What does the swarm UI show, and what does `swarmctl` do?** → [`2-handbook/swarm/ui.md`](2-handbook/swarm/ui.md) and [`2-handbook/swarm/swarmctl-cli.md`](2-handbook/swarm/swarmctl-cli.md) (the diff --git a/docs/examples/semi-spread/common.nix b/docs/examples/semi-spread/common.nix new file mode 100644 index 000000000..07dca2226 --- /dev/null +++ b/docs/examples/semi-spread/common.nix @@ -0,0 +1,21 @@ +# Imported by every machine in the swarm: the swarm's shared facts, the VPN and +# the names inside it. `services.hyperhive.swarm.*` has to be identical on all +# of them, so it lives here and nowhere else. +{ + imports = [ + ./wireguard.nix + ./dns.nix + ]; + + services.hyperhive.swarm = { + domain = "myswarm.example"; + # Every hive, including the one a machine does not run. Names only: each + # hive's domain defaults to `.myswarm.example`. A hive name may not + # contain "hive", "agent" or "swarm", so the machines `hive-one` and + # `hive-two` run the hives `one` and `two`. + hives = { + one = { }; + two = { }; + }; + }; +} diff --git a/docs/examples/semi-spread/dns.nix b/docs/examples/semi-spread/dns.nix new file mode 100644 index 000000000..ff5e557cb --- /dev/null +++ b/docs/examples/semi-spread/dns.nix @@ -0,0 +1,65 @@ +# Names inside the VPN. Every swarm name that is not public resolves to the +# VPN address of the machine serving it, through a static table written on +# each host. Identical on every host. +# +# Two readers need the table. Processes on the host read /etc/hosts +# (`networking.hosts`). Containers (agents, swarm services) ask the hive's +# dnsmasq, which ignores /etc/hosts, so it also reads the table through +# `addn-hosts`. +# +# A host leaves out the names it serves itself: hyperhive already answers +# those with the bridge address, and a table entry would override it. +{ + config, + lib, + ... +}: +let + mesh = import ./mesh.nix; + self = config.networking.hostName; + swarm = config.services.hyperhive.swarm; + + # Which machine serves which name. A hive's machine is `hive-` and it + # serves the hive's own names: its dashboard and its domain. + servedBy = + lib.mapAttrs' ( + hive: entry: + lib.nameValuePair "hive-${hive}" [ + swarm.dashboardDomains.${hive} + entry.domain + ] + ) swarm.hives + // { + swarm-central = [ + swarm.domain + swarm.bao.domain + swarm.bao.ui.domain + ]; + swarm-services = [ + swarm.forge.domain + swarm.matrix.gatewayHost + swarm.authelia.domain + swarm.nats.domain + swarm.otel.domain + swarm.grafana.domain + swarm.victoriametrics.domain + swarm.victorialogs.domain + ]; + swarm-heavyduty = [ swarm.attic.ownDomain ]; + }; + + elsewhere = removeAttrs servedBy [ self ]; +in +{ + networking.hosts = lib.mapAttrs' ( + machine: names: lib.nameValuePair mesh.${machine}.address names + ) elsewhere; + + services.dnsmasq.settings.addn-hosts = "${builtins.toFile "swarm-vpn-hosts" ( + lib.concatStringsSep "\n" ( + lib.mapAttrsToList ( + machine: names: "${mesh.${machine}.address} ${lib.concatStringsSep " " names}" + ) elsewhere + ) + )}"; +} diff --git a/docs/examples/semi-spread/hive.nix b/docs/examples/semi-spread/hive.nix new file mode 100644 index 000000000..f490d126a --- /dev/null +++ b/docs/examples/semi-spread/hive.nix @@ -0,0 +1,28 @@ +# A hive. Identical on hive-one and hive-two: the machine `hive-` runs the +# hive ``. +{ config, lib, ... }: +let + # The leaves for this hive, copied from swarm-central, where the secret store + # runs: `client.pem` and `client-key.pem` from + # `/var/lib/swarm-bao-pki/hives//`, along with that directory's + # `hive-leaf.pem` and `hive-leaf-key.pem`, which the hive's gateway + # certificate comes from. Both pairs sit in one directory. + pki = "/var/lib/swarm-pki"; +in +{ + services.hyperhive.deploy = { + hive-controller = { + enable = true; + hiveName = lib.removePrefix "hive-" config.networking.hostName; + }; + bao = { + clientCertFile = "${pki}/client.pem"; + clientKeyFile = "${pki}/client-key.pem"; + }; + }; + + # The dashboard is reachable over the VPN only. + networking.firewall.interfaces.wg0.allowedTCPPorts = [ + config.services.hyperhive.gateway.httpsPort + ]; +} diff --git a/docs/examples/semi-spread/mesh.nix b/docs/examples/semi-spread/mesh.nix new file mode 100644 index 000000000..c7fefad7b --- /dev/null +++ b/docs/examples/semi-spread/mesh.nix @@ -0,0 +1,33 @@ +# The VPN's facts: one entry per machine, read by every host (wireguard.nix, +# dns.nix) and nothing else. Keyed by `networking.hostName`. +# +# `address` is the machine's address inside the VPN; `endpoint` is where the +# other machines dial it; `publicKey` is the output of `wg pubkey` on that +# machine. The keys below are placeholders: replace each with the real one. +{ + swarm-central = { + address = "10.77.0.1"; + endpoint = "203.0.113.10:51820"; + publicKey = "REPLACE-with-swarm-central-wg-pubkey"; + }; + swarm-services = { + address = "10.77.0.2"; + endpoint = "203.0.113.11:51820"; + publicKey = "REPLACE-with-swarm-services-wg-pubkey"; + }; + swarm-heavyduty = { + address = "10.77.0.3"; + endpoint = "203.0.113.12:51820"; + publicKey = "REPLACE-with-swarm-heavyduty-wg-pubkey"; + }; + hive-one = { + address = "10.77.0.11"; + endpoint = "198.51.100.11:51820"; + publicKey = "REPLACE-with-hive-one-wg-pubkey"; + }; + hive-two = { + address = "10.77.0.12"; + endpoint = "198.51.100.12:51820"; + publicKey = "REPLACE-with-hive-two-wg-pubkey"; + }; +} diff --git a/docs/examples/semi-spread/swarm-central.nix b/docs/examples/semi-spread/swarm-central.nix new file mode 100644 index 000000000..0aec98c17 --- /dev/null +++ b/docs/examples/semi-spread/swarm-central.nix @@ -0,0 +1,23 @@ +# swarm-central: the swarm controller (and the swarm UI it serves) and the +# secret store. +{ config, ... }: +let + mesh = import ./mesh.nix; + hyperhive = config.services.hyperhive; +in +{ + services.hyperhive = { + deploy.swarm-controller.enable = true; + deploy.bao.enable = true; + + # The store also listens on this machine's VPN address, for the services + # and hives on other machines. + deploy.bao.extraListenAddresses = [ mesh.swarm-central.address ]; + + # The swarm UI is public. + gateway.openFirewall = true; + }; + + # The store is reachable over the VPN only. + networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.swarm.bao.port ]; +} diff --git a/docs/examples/semi-spread/swarm-heavyduty.nix b/docs/examples/semi-spread/swarm-heavyduty.nix new file mode 100644 index 000000000..31e2a8f50 --- /dev/null +++ b/docs/examples/semi-spread/swarm-heavyduty.nix @@ -0,0 +1,31 @@ +# swarm-heavyduty: the builder, the forge's CI runner and the binary cache. +{ config, lib, ... }: +let + hyperhive = config.services.hyperhive; + + # Leaves copied from swarm-central, where the secret store runs. See + # swarm-services.nix for the helper's naming. + pki = "/var/lib/swarm-pki"; + pair = option: stem: { + "${option}ClientCertFile" = "${pki}/${stem}.pem"; + "${option}ClientKeyFile" = "${pki}/${stem}-key.pem"; + }; +in +{ + services.hyperhive = { + deploy.builder.enable = true; + deploy.ci-runner.enable = true; + deploy.attic.enable = true; + + deploy.bao = lib.mergeAttrsList [ + (pair "builder" "builder") + (pair "ciRunner" "ci-runner") + (pair "attic" "attic") + (pair "forwarderOidc" "forwarder-oidc") + (pair "servicesIssuer" "services-issuer") + ]; + }; + + # The cache is served through the gateway, over the VPN only. + networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.gateway.httpsPort ]; +} diff --git a/docs/examples/semi-spread/swarm-services.nix b/docs/examples/semi-spread/swarm-services.nix new file mode 100644 index 000000000..e30dc0878 --- /dev/null +++ b/docs/examples/semi-spread/swarm-services.nix @@ -0,0 +1,64 @@ +# swarm-services: the forge, the matrix homeserver, SSO, the queue, and the +# metrics and log stack. +{ config, lib, ... }: +let + hyperhive = config.services.hyperhive; + + # The secret store runs on swarm-central, so each service here reads it with + # a leaf copied from there. `pair "grafanaOidc" "grafana-oidc"` names the + # options `deploy.bao.grafanaOidcClientCertFile` and `…KeyFile` and the files + # `grafana-oidc.pem` and `grafana-oidc-key.pem`. + pki = "/var/lib/swarm-pki"; + pair = option: stem: { + "${option}ClientCertFile" = "${pki}/${stem}.pem"; + "${option}ClientKeyFile" = "${pki}/${stem}-key.pem"; + }; +in +{ + services.hyperhive = { + # Everything once-per-swarm that is not placed elsewhere below. + deploy.allSwarmServices = true; + + # The forge and the matrix homeserver are public, and so is the SSO login + # they send browsers to. + gateway.openFirewall = true; + + deploy.bao = lib.mergeAttrsList [ + { enable = false; } + (pair "forgePublisher" "forge-publisher") + (pair "servicesIssuer" "services-issuer") + (pair "grafanaOidc" "grafana-oidc") + (pair "otelOidc" "otel-oidc") + (pair "forwarderOidc" "forwarder-oidc") + (pair "reviewClassifier" "review-classifier") + (pair "notifyRouter" "notify-router") + (pair "claudeLogin" "claude-login") + (pair "ciMetrics" "ci-metrics") + ]; + deploy.matrix = { + ctlBaoClientCertFile = "${pki}/matrix-ctl.pem"; + ctlBaoClientKeyFile = "${pki}/matrix-ctl-key.pem"; + }; + deploy.nats = { + baoClientCertFile = "${pki}/nats.pem"; + baoClientKeyFile = "${pki}/nats-key.pem"; + authBaoClientCertFile = "${pki}/nats-auth.pem"; + authBaoClientKeyFile = "${pki}/nats-auth-key.pem"; + }; + deploy.swarm-secret-publisher = { + baoClientCertFile = "${pki}/secret-publisher.pem"; + baoClientKeyFile = "${pki}/secret-publisher-key.pem"; + }; + deploy.victoriametrics = { + authBaoClientCertFile = "${pki}/metrics-auth.pem"; + authBaoClientKeyFile = "${pki}/metrics-auth-key.pem"; + }; + deploy.victorialogs = { + authBaoClientCertFile = "${pki}/logs-auth.pem"; + authBaoClientKeyFile = "${pki}/logs-auth-key.pem"; + }; + }; + + # The queue is reachable over the VPN only. + networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.swarm.nats.port ]; +} diff --git a/docs/examples/semi-spread/wireguard.nix b/docs/examples/semi-spread/wireguard.nix new file mode 100644 index 000000000..c7920a068 --- /dev/null +++ b/docs/examples/semi-spread/wireguard.nix @@ -0,0 +1,24 @@ +# The VPN: plain NixOS WireGuard, one interface `wg0`, a full mesh built from +# ./mesh.nix. Identical on every host; the host's own entry is picked by +# `networking.hostName`. +{ config, lib, ... }: +let + mesh = import ./mesh.nix; + self = config.networking.hostName; +in +{ + networking.wireguard.interfaces.wg0 = { + ips = [ "${mesh.${self}.address}/24" ]; + listenPort = 51820; + # Created on the host by `wg genkey`, never in the store. + privateKeyFile = "/etc/wireguard/wg0.key"; + peers = lib.mapAttrsToList (_: peer: { + inherit (peer) publicKey endpoint; + allowedIPs = [ "${peer.address}/32" ]; + persistentKeepalive = 25; + }) (removeAttrs mesh [ self ]); + }; + + # The handshake arrives on the public interface. + networking.firewall.allowedUDPPorts = [ config.networking.wireguard.interfaces.wg0.listenPort ]; +} diff --git a/nix/checks.nix b/nix/checks.nix index b955658f9..6f8226ad5 100644 --- a/nix/checks.nix +++ b/nix/checks.nix @@ -203,6 +203,13 @@ in inherit pkgs self nixosSystem; inherit (pkgs) lib; }; + # The five machines of docs/examples/semi-spread/. Five full-system + # fixtures, because the claim is that every machine of the example + # evaluates; no case of another cluster can stand in for one. + module-eval-semi-spread = import ./module-eval/semi-spread.nix { + inherit pkgs self nixosSystem; + inherit (pkgs) lib; + }; module-eval-agent-queue-bao = import ./module-eval/agent-queue-bao.nix { inherit pkgs self nixosSystem; inherit (pkgs) lib; diff --git a/nix/module-eval/semi-spread.nix b/nix/module-eval/semi-spread.nix new file mode 100644 index 000000000..0dcd3d9e3 --- /dev/null +++ b/nix/module-eval/semi-spread.nix @@ -0,0 +1,345 @@ +# `checks.module-eval-semi-spread` — see ./lib.nix for the shared rationale +# (why this suite exists, naming convention, "evaluates not executes"). +# +# Evaluates the five machines of docs/examples/semi-spread/ exactly as that +# directory's files describe them, so the example the handbook points at +# cannot stop evaluating, and pins the shape the handbook page promises: which +# machine runs what, what is open on the public interface, and what only on the +# VPN. +{ + pkgs, + lib, + self, + nixosSystem, +}: +let + inherit + (import ./lib.nix { + inherit + pkgs + lib + self + nixosSystem + ; + }) + baoSettings + runGroup + ; + + example = self + "/docs/examples/semi-spread"; + + # What the example's files leave to the machine's own configuration: its + # host name, which picks the VPN entry and, for a hive, the hive. `role` is + # the file the machine adds to common.nix. + machine = + hostName: role: + (nixosSystem { + modules = [ + (pkgs.path + "/nixos/modules/misc/nixpkgs/read-only.nix") + { nixpkgs.pkgs = pkgs; } + self.nixosModules.default + { + fileSystems."/" = { + device = "/dev/null"; + fsType = "tmpfs"; + }; + boot.loader.grub.enable = false; + system.stateVersion = "25.11"; + networking.hostName = hostName; + } + (example + "/common.nix") + (example + "/${role}.nix") + ]; + }).config; + + central = machine "swarm-central" "swarm-central"; + services = machine "swarm-services" "swarm-services"; + heavyduty = machine "swarm-heavyduty" "swarm-heavyduty"; + hiveOne = machine "hive-one" "hive"; + hiveTwo = machine "hive-two" "hive"; + all = { + swarm-central = central; + swarm-services = services; + swarm-heavyduty = heavyduty; + hive-one = hiveOne; + hive-two = hiveTwo; + }; + + mesh = import (example + "/mesh.nix"); + + # What `system.build.toplevel` checks before it builds anything: every module + # assertion holds. The example also renders no warning. Evaluating the + # toplevel itself would add the package set of five full systems to every + # run of this suite. + evaluates = + m: + let + r = builtins.tryEval (lib.all (a: a.assertion) m.assertions && m.warnings == [ ]); + in + r.success && r.value; + + # The TCP ports open on every interface, and those open on `wg0` alone. + publicTcp = m: m.networking.firewall.allowedTCPPorts; + vpnTcp = m: (m.networking.firewall.interfaces.wg0 or { allowedTCPPorts = [ ]; }).allowedTCPPorts; + + deploy = m: m.services.hyperhive.deploy; + swarmOf = m: m.services.hyperhive.swarm; + https = m: m.services.hyperhive.gateway.httpsPort; + + # The names a machine's `/etc/hosts` and its resolver's extra hosts file + # resolve to the VPN, as a set of `address -> names`. + hostsOf = m: m.networking.hosts; + resolverTable = + m: lib.concatMapStrings builtins.readFile (m.services.dnsmasq.settings.addn-hosts or [ ]); + names = m: lib.concatLists (lib.attrValues (lib.filterAttrs (a: _: a != "127.0.0.2") (hostsOf m))); + + vhosts = m: builtins.attrNames m.services.nginx.virtualHosts; + + cases = [ + # Every machine evaluates with its assertions holding. `tryEval` + # turns a refusal into a failed case that names the machine. + { + name = "swarm-central evaluates with its assertions holding"; + ok = evaluates central; + } + { + name = "swarm-services evaluates with its assertions holding"; + ok = evaluates services; + } + { + name = "swarm-heavyduty evaluates with its assertions holding"; + ok = evaluates heavyduty; + } + { + name = "hive-one evaluates with its assertions holding"; + ok = evaluates hiveOne; + } + { + name = "hive-two evaluates with its assertions holding"; + ok = evaluates hiveTwo; + } + + # Who runs what. Each machine's list is exact: a service that moved would + # show here on the machine that gained it and the one that lost it. + { + name = "swarm-central runs the controller and the store, and none of the services placed elsewhere"; + ok = + let + d = deploy central; + in + d.swarm-controller.enable + && d.bao.enable + && !d.forgejo.enable + && !d.matrix.enable + && !d.nats.enable + && !d.attic.enable + && !d.builder.enable + && !d.hive-controller.enable; + } + { + name = "swarm-services runs the forge, matrix, SSO, queue and the metrics and log stack, not the store"; + ok = + let + d = deploy services; + in + lib.all (x: x) [ + d.forgejo.enable + d.matrix.enable + d.authelia.enable + d.nats.enable + d.swarm-otel.enable + d.victoriametrics.enable + d.grafana.enable + d.victorialogs.enable + d.review-classifier.enable + d.notify-router.enable + d.claude-login.enable + d.ci-metrics.enable + (!d.bao.enable) + (!d.swarm-controller.enable) + (!d.attic.enable) + (!d.builder.enable) + (!d.ci-runner.enable) + (!d.hive-controller.enable) + ]; + } + { + name = "swarm-heavyduty runs the builder, the CI runner and the cache, and nothing else"; + ok = + let + d = deploy heavyduty; + in + d.builder.enable + && d.ci-runner.enable + && d.attic.enable + && !d.forgejo.enable + && !d.matrix.enable + && !d.nats.enable + && !d.bao.enable + && !d.swarm-controller.enable + && !d.hive-controller.enable; + } + { + name = "hive-one and hive-two run a hive each, under different names, and no shared service"; + ok = + let + d1 = deploy hiveOne; + d2 = deploy hiveTwo; + none = + d: + !( + d.forgejo.enable || d.matrix.enable || d.nats.enable || d.bao.enable || d.swarm-controller.enable + ); + in + d1.hive-controller.enable + && d2.hive-controller.enable + && d1.hive-controller.hiveName == "one" + && d2.hive-controller.hiveName == "two" + && none d1 + && none d2; + } + { + # `swarm.*` is the one part that has to be identical, and it comes from a + # single file; the case reads it back from every machine. + name = "every machine holds the same swarm name, hives and service names"; + ok = + let + view = m: { + inherit (swarmOf m) domain serviceDomains; + hives = builtins.attrNames (swarmOf m).hives; + }; + in + lib.all (m: view m == view central) (lib.attrValues all); + } + + # The public interface. The machines serving the forge, matrix and the + # swarm UI open the gateway to everyone; the others open nothing but the + # VPN's own port. + { + name = "swarm-central and swarm-services open the gateway on every interface"; + ok = + publicTcp central == [ + 80 + 443 + ] + && + publicTcp services == [ + 80 + 443 + ]; + } + { + # Presence control for the case above: the same read finds the ports + # where they are open, so an empty list here means closed. + name = "swarm-heavyduty and the hives open no TCP port on every interface"; + ok = + publicTcp heavyduty == [ ] + && publicTcp hiveOne == [ ] + && publicTcp hiveTwo == [ ] + && vpnTcp heavyduty == [ (https heavyduty) ] + && vpnTcp hiveOne == [ (https hiveOne) ] + && vpnTcp hiveTwo == [ (https hiveTwo) ]; + } + { + name = "every machine opens only the VPN's UDP port on every interface"; + ok = lib.all (m: m.networking.firewall.allowedUDPPorts == [ 51820 ]) (lib.attrValues all); + } + { + name = "swarm-central serves the swarm UI and swarm-services serves the forge and matrix"; + ok = + lib.elem (swarmOf central).domain (vhosts central) + && lib.elem (swarmOf services).forge.domain (vhosts services) + && lib.elem (swarmOf services).matrix.gatewayHost (vhosts services) + # The cache is on a machine with nothing public. + && lib.elem (swarmOf heavyduty).attic.ownDomain (vhosts heavyduty) + && !lib.elem (swarmOf heavyduty).attic.ownDomain (vhosts services); + } + + # What only the VPN reaches. + { + name = "the secret store's port is open on wg0 and on no other external interface"; + ok = + let + port = (swarmOf central).bao.port; + in + vpnTcp central == [ port ] && !lib.elem port (publicTcp central); + } + { + name = "the secret store listens on its VPN address and requires a client certificate there"; + ok = + let + l = (baoSettings central).listener.extra-1 or { }; + in + (l.address or null) == "${mesh.swarm-central.address}:${toString (swarmOf central).bao.port}" + && (l.tls_require_and_verify_client_cert or false); + } + { + name = "the queue's port is open on wg0 and on no other external interface"; + ok = + let + port = (swarmOf services).nats.port; + in + lib.elem port (vpnTcp services) && !lib.elem port (publicTcp services); + } + + # The VPN itself. + { + name = "every machine peers with the other four over wg0, with its key outside the store"; + ok = lib.all ( + m: + let + wg = m.networking.wireguard.interfaces.wg0; + in + builtins.length wg.peers == 4 + && wg.privateKey == null + && wg.privateKeyFile != null + && !lib.hasPrefix "/nix/store" wg.privateKeyFile + ) (lib.attrValues all); + } + { + name = "each machine takes its own address from the shared VPN table"; + ok = lib.all ( + n: lib.elem "${mesh.${n}.address}/24" all.${n}.networking.wireguard.interfaces.wg0.ips + ) (builtins.attrNames all); + } + + # DNS. A machine lists the names others serve, at those machines' VPN + # addresses, and leaves out its own. + { + name = "a hive resolves the store, queue and cache names to the machines that serve them"; + ok = + let + t = hostsOf hiveOne; + in + lib.elem (swarmOf hiveOne).bao.domain t.${mesh.swarm-central.address} + && lib.elem (swarmOf hiveOne).nats.domain t.${mesh.swarm-services.address} + && lib.elem (swarmOf hiveOne).attic.ownDomain t.${mesh.swarm-heavyduty.address}; + } + { + name = "a machine leaves out the names it serves itself"; + ok = + !lib.elem (swarmOf central).bao.domain (names central) + && !lib.elem (swarmOf services).nats.domain (names services) + && !lib.elem (swarmOf heavyduty).attic.ownDomain (names heavyduty) + && !lib.elem "one.hives.${(swarmOf hiveOne).domain}" (names hiveOne) + # Control: another hive's dashboard is in the table. + && lib.elem "two.hives.${(swarmOf hiveOne).domain}" (names hiveOne); + } + { + # Containers ask the hive's resolver, which ignores /etc/hosts, so the + # same table has to reach it through its extra hosts file. + name = "the hive resolver carries the same table as /etc/hosts"; + ok = + let + table = resolverTable hiveOne; + in + hostsOf hiveOne != { } + && lib.all ( + address: lib.hasInfix "${address} ${lib.concatStringsSep " " (hostsOf hiveOne).${address}}" table + ) (builtins.attrNames (lib.filterAttrs (a: _: a != "127.0.0.2") (hostsOf hiveOne))) + # Control: the file is not one every machine shares. + && !lib.hasInfix (swarmOf central).bao.domain (resolverTable central); + } + ]; +in +runGroup "semi-spread" cases