# Example: a semi-spread deploy Three swarm machines and two hives, joined by a WireGuard VPN over IPv4 and IPv6. Four things answer to the internet: the swarm UI, the forge, the matrix homeserver and the SSO login they send browsers to. Every other vhost answers on the VPN only. It's one of the in-between shapes that [`services.md`](services.md#deployment-shapes) supports: the swarm's services run on three machines, not on one box and not one per service. The NixOS files are in [`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all five machines (`checks.module-eval-semi-spread`) and pins the facts on this page, so the files stay in step with the options. ## The machines | machine | runs | answers the internet with | answers the VPN only with | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------- | | `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI | the store's browser UI; the store itself (port 8200) | | `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | the forge, matrix and the SSO login | Grafana, metrics, logs, the collector, the queue's vhost; the queue (4222) | | `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing | the cache | | `hive-one` | the hive `one` | nothing | the hive's dashboard and domain | | `hive-two` | the hive `two`, with the same file as `hive-one` | nothing | the hive's dashboard and domain | Why this split: - **`swarm-central`** holds what every other machine depends on, and little else that can fail. The secret store and the controller are small. - **`swarm-services`** holds the services people and agents talk to all day. They're light. - **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder evaluates config pull requests nobody reviewed yet, and the CI runner runs pull-request code. Keeping both off the machine that holds the secret store limits what a bad build can reach. - **The hives** run the agents. They need nothing from the internet but the agents' own egress. A hive name can't contain `hive`, `agent` or `swarm`, so the machines `hive-one` and `hive-two` run the hives `one` and `two`. `hive.nix` takes the hive name from the machine's host name, which is why one file serves both. ## What's public and what's not Two things decide it: which ports the firewall opens, and which addresses nginx answers on. **Ports.** `gateway.openFirewall` opens the gateway's ports (80 and 443) on every interface, and the example sets it on `swarm-central` and `swarm-services` only. Everything else opens on `wg0` alone, with `networking.firewall.interfaces.wg0.allowedTCPPorts`: | machine | port on `wg0` | for | | ----------------- | ------------------------- | --------------------------------------------- | | `swarm-central` | `swarm.bao.port` (8200) | the secret store, read by every other machine | | `swarm-services` | `swarm.nats.port` (4222) | the queue | | `swarm-heavyduty` | `gateway.httpsPort` (443) | the cache, behind the gateway | | each hive | `gateway.httpsPort` (443) | the hive's dashboard and gateway | The example scopes these to `wg0` because the options that open a service's port, `gateway.openFirewall`, `deploy.bao.openFirewall` and `deploy.nats.openFirewall`, open it on every interface. The secret store and the queue don't rely on the VPN for their security: the store requires a client certificate, and the queue requires TLS and refuses clients it can't identify. The VPN keeps them off the internet as well. **Addresses.** The gateway is one nginx per machine, and by default every vhost answers on every address. Each machine here sets `gateway.privateListenAddresses` (`gateway.nix`) to its own VPN IPv4 and IPv6 addresses and the hive bridge, and sets `public = false` on the vhosts that stay inside: | machine | public vhosts | `public = false` | | ----------------- | -------------------------------------------------- | ------------------------------------------------------------------ | | `swarm-central` | `` (the swarm UI) | `bao-ui.` | | `swarm-services` | `forge.`, `chat.` (matrix) and `auth.` (SSO login) | `grafana.`, `metrics.`, `logs.`, `otel.` and `nats.` | | `swarm-heavyduty` | none | `attic.` (the cache) | | each hive | none | `.hives.` and the hive's own domain | A public vhost listens on `0.0.0.0`, on `[::]`, and on the private addresses, so a machine inside the VPN still reaches it by its VPN address. A private vhost listens on the private addresses only. A request for a private name on the public address gets nothing back, the same as any name the gateway doesn't know. The hive bridge is in the list because the machine's own containers reach a name it serves through the bridge address, so a private vhost the containers use still answers them. **SSO stays public.** `auth.` answers the internet because the forge, matrix and the swarm UI redirect a browser there to sign in, and a browser outside the VPN has to follow. Authelia authenticates; it needs no VPN as a second lock. If every operator is on the VPN anyway, set `public = false` on it too, and the sign-in then works from inside the VPN only. Every machine also opens UDP 51820 on every interface, because that's where the VPN's handshake arrives. The example leaves SSH to you. ```text internet UDP 51820 reaches all five 80/443 | | 80/443 (swarm UI) (forge, matrix, SSO) swarm-central swarm-services swarm-heavyduty hive-one hive-two 10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12 fd77:77a1:5e::1 fd77:77a1:5e::2 fd77:77a1:5e::3 …::11 …::12 | | | | | +------- wg0: 10.77.0.0/24 and fd77:77a1:5e::/64 --+----------+ ``` ## The VPN: raw WireGuard `wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on all five machines. `mesh.nix` holds each machine's VPN addresses (IPv4 and IPv6), public endpoint and public key. The file picks the machine's own entry by `networking.hostName` and makes the other four its peers: ```nix networking.wireguard.interfaces.wg0 = { ips = [ "${mesh.${self}.address}/24" "${mesh.${self}.address6}/64" ]; listenPort = 51820; privateKeyFile = "/etc/wireguard/wg0.key"; peers = lib.mapAttrsToList (_: peer: { inherit (peer) publicKey endpoint; allowedIPs = [ "${peer.address}/32" "${peer.address6}/128" ]; persistentKeepalive = 25; }) (removeAttrs mesh [ self ]); }; ``` To set it up: 1. On each machine, create the private key in a file only root can read. It never goes in the Nix store or in the configuration: ```bash install -d -m 0700 /etc/wireguard wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key wg pubkey < /etc/wireguard/wg0.key ``` 2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public keys aren't secret, so the file is safe to commit. You do this once. 3. Set the `address`, `address6` and `endpoint` of each machine in `mesh.nix`. The example uses documentation addresses for the endpoints and a unique-local IPv6 prefix (`fd77:77a1:5e::/64`). Generate your own prefix (RFC 4193) rather than keeping it. 4. Deploy. `wg show wg0` on any machine lists four peers with a recent handshake, and `ping 10.77.0.1` and `ping fd77:77a1:5e::1` answer. A machine that can't take incoming connections, such as one behind NAT, sets `endpoint = null`. It dials the others, and the keepalive holds the NAT mapping open. **Or use tailscale, headscale or NetBird.** They hand out the keys and the addresses for you, so the key steps above go away. Three things change: - The interface is no longer `wg0`. Tailscale's is `tailscale0` (`services.tailscale.interfaceName`). Put its name in the firewall rules in place of `wg0`. - Drop `wireguard.nix`, and put each machine's addresses from your VPN into the `address` and `address6` fields of `mesh.nix`. `gateway.nix` and the DNS table below read only those fields. The addresses must exist on the machine before nginx starts, which a VPN service that comes up late may not guarantee. - Enable the service, for example `services.tailscale.enable = true` with `services.tailscale.authKeyFile` pointing at a key file outside the store. ## Reading the store Only `swarm-central` runs the secret store, and every service on the other machines reads it with a client certificate that the store's host mints. The example names each leaf under `/var/lib/swarm-pki/` on the machine that uses it. After the first deploy of `swarm-central`, copy the files from `/var/lib/swarm-bao-pki/` there over the VPN (for a hive, from `/var/lib/swarm-bao-pki/hives//`), then deploy the other machines. The private keys travel over a channel you trust and stay readable by root only. [`credentials.md`](credentials.md) lists which leaf each service reads and what a missing one costs. The first-run steps after the deploys are in [`setup.md`](../../1-getting-started/setup.md). ## DNS Every service reaches another by name, so every name has to resolve on every machine and inside every container. Two views of it exist. **The public view** is a handful of records at your DNS provider, pointing at the two public machines. Give each name an `A` record, and an `AAAA` record if the machine has a public IPv6 address (the gateway listens on `[::]`): | name | address | | -------------------------------------- | --------------------------------- | | `` | `swarm-central`'s public address | | `forge.` | `swarm-services`'s public address | | `chat.` (the homeserver) | `swarm-services`'s public address | | `auth.` | `swarm-services`'s public address | Matrix federation and browsers use these. The names you leave out of public DNS (`bao.`, `bao-ui.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the hives' dashboards) don't resolve from outside. **The inside view** maps every swarm name to the VPN addresses of the machine that serves it, IPv4 and IPv6 (`A` and `AAAA`), including the public names, so traffic between machines never uses the internet. The example does it with a static table in `dns.nix`, written on every machine and built from the same options the services use, so a renamed `swarm.*.domain` follows. Two readers need it: - Processes on the machine read `/etc/hosts` (`networking.hosts`). - Containers (agents and swarm services alike) ask the hive's resolver (dnsmasq). It ignores `/etc/hosts`, so the example hands it the same table through `services.dnsmasq.settings.addn-hosts`. A machine leaves out the names it serves itself. hyperhive answers those already, and a table entry would override its answer. The trade-off: a static table needs no DNS server and no extra moving part, and it lives in the same repository as the rest. It also means a new hive adds a line on every machine, and a name that moves from one machine to another is a rebuild everywhere. For a swarm with many hives, run a small DNS server on the VPN instead, answering the swarm domain with VPN addresses, and point each machine's upstream resolver at it for that domain. That's one place to edit, at the cost of one more service every lookup depends on. To open Grafana, the store's browser UI or a hive dashboard yourself, join the VPN with your own machine: add it as a peer on each machine and give it the same names, for example in its own `/etc/hosts`. ## Not in the example - **TLS certificates the public can trust.** Without `gateway.tls.acme` or `gateway.tls.certDir`, the gateway serves certificates from the swarm's own CA, and browsers warn. Set one on `swarm-central` and `swarm-services`. - **SSH and the rest of your hosts' base setup.** The example touches only the firewall ports it needs. - **The hardware configuration.** Each machine's own file sets `networking.hostName` to one of the five names, which picks its VPN entry. Wiring one machine into a flake looks like this: ```nix nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; modules = [ hyperhive.nixosModules.default ./hardware/swarm-central.nix # sets networking.hostName = "swarm-central" ./semi-spread/common.nix ./semi-spread/swarm-central.nix ]; }; ``` `hive-one` and `hive-two` add `./semi-spread/hive.nix` instead of a machine-specific file.