hyperhive/docs/2-handbook/swarm/semi-spread-deploy.md

Example: a semi-spread deploy

Three swarm machines and two hives, joined by a WireGuard VPN, with only the forge, the matrix homeserver and the swarm UI reachable from the internet. It is one of the in-between shapes that services.md supports: the swarm's services run on three machines, not on one box and not one per service.

The NixOS files are in docs/examples/semi-spread/. CI evaluates all five machines (checks.module-eval-semi-spread) and pins the facts on this page, so the files stay in step with the options.

The machines

machine	runs	reachable from the internet
swarm-central	swarm-controller and the swarm UI, swarm-bao	the swarm UI
swarm-services	the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics	forge, matrix, SSO login
swarm-heavyduty	swarm-builder, the forge's CI runner (swarm-ci-runner), swarm-attic	nothing
hive-one	the hive one	nothing
hive-two	the hive two, with the same file as hive-one	nothing

Why this split:

• swarm-central holds what every other machine depends on, and little else that can fail. The secret store and the controller are small.
• swarm-services holds the services people and agents talk to all day. They're light, and they need the public name and port.
• swarm-heavyduty holds the CPU, memory and disk hogs. The builder evaluates config pull requests nobody reviewed yet, and the CI runner runs pull-request code. Keeping both off the machine that holds the secret store limits what a bad build can reach.
• The hives run the agents. They need nothing from the internet but the agents' own egress.

A hive name can't contain hive, agent or swarm, so the machines hive-one and hive-two run the hives one and two. hive.nix takes the hive name from the machine's host name, which is why one file serves both.

What's public and what's not

gateway.openFirewall opens the gateway's ports (80 and 443) on every interface, and the example sets it on two machines only:

• swarm-central serves the swarm UI at <swarm.domain>.
• swarm-services serves the forge (forge.<swarm.domain>) and the matrix homeserver (chat.<swarm.domain>). It also serves auth.<swarm.domain>, because the forge, matrix and the swarm UI send browsers there to sign in.

Public means the machine's gateway port is open. A gateway is one nginx for everything the machine serves, so the other names on those two machines (Grafana, metrics, logs, the collector, the store's browser UI) answer on the public address too. Each sits behind SSO or a bearer token, and none of them is in public DNS, but they're not closed. A service you want closed goes on a machine that opens nothing public, like swarm-heavyduty.

Everything else uses the VPN. These ports open on wg0 only, with networking.firewall.interfaces.wg0.allowedTCPPorts:

machine	port on wg0	for
swarm-central	swarm.bao.port (8200)	the secret store, read by every other machine
swarm-services	swarm.nats.port (4222)	the queue
swarm-heavyduty	gateway.httpsPort (443)	the cache, behind the gateway
each hive	gateway.httpsPort (443)	the hive's dashboard and gateway

The example scopes these to wg0 because the options that open a service's port, gateway.openFirewall, deploy.bao.openFirewall and deploy.nats.openFirewall, open it on every interface. The secret store and the queue don't rely on the VPN for their security: the store requires a client certificate, and the queue requires TLS and refuses clients it can't identify. The VPN keeps them off the internet as well.

Every machine also opens UDP 51820 on every interface, because that's where the VPN's handshake arrives. The example leaves SSH to you.

                         internet
              80/443 |                | 80/443          UDP 51820 on all five
                     |                |
   swarm-central   swarm-services   swarm-heavyduty   hive-one   hive-two
   10.77.0.1       10.77.0.2        10.77.0.3         10.77.0.11 10.77.0.12
       |                |                 |               |          |
       +----------------+----- wg0, 10.77.0.0/24 --------+----------+

The VPN: raw WireGuard

wireguard.nix is plain NixOS, one interface wg0, and it's the same file on all five machines. mesh.nix holds each machine's VPN address, public endpoint and public key. The file picks the machine's own entry by networking.hostName and makes the other four its peers:

networking.wireguard.interfaces.wg0 = {
  ips = [ "${mesh.${self}.address}/24" ];
  listenPort = 51820;
  privateKeyFile = "/etc/wireguard/wg0.key";
  peers = lib.mapAttrsToList (_: peer: {
    inherit (peer) publicKey endpoint;
    allowedIPs = [ "${peer.address}/32" ];
    persistentKeepalive = 25;
  }) (removeAttrs mesh [ self ]);
};

To set it up:

1. On each machine, create the private key in a file only root can read. It never goes in the Nix store or in the configuration:

   install -d -m 0700 /etc/wireguard
   wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key
   wg pubkey < /etc/wireguard/wg0.key
   

2. Put each machine's public key (the wg pubkey output) in mesh.nix. Public keys aren't secret, so the file is safe to commit. You do this once.

3. Set the address and endpoint of each machine in mesh.nix. The example uses documentation addresses.

4. Deploy. wg show wg0 on any machine lists four peers with a recent handshake, and ping 10.77.0.1 answers.

A machine that can't take incoming connections, such as one behind NAT, sets endpoint = null. It dials the others, and the keepalive holds the NAT mapping open.

Or use tailscale, headscale or NetBird. They hand out the keys and the addresses for you, so the key steps above go away. Three things change:

• The interface is no longer wg0. Tailscale's is tailscale0 (services.tailscale.interfaceName). Put its name in the firewall rules in place of wg0.
• Drop wireguard.nix, and put each machine's address from your VPN into the address field of mesh.nix. The DNS table below reads only that field.
• Enable the service, for example services.tailscale.enable = true with services.tailscale.authKeyFile pointing at a key file outside the store.

Reading the store

Only swarm-central runs the secret store, and every service on the other machines reads it with a client certificate that the store's host mints. The example names each leaf under /var/lib/swarm-pki/ on the machine that uses it. After the first deploy of swarm-central, copy the files from /var/lib/swarm-bao-pki/ there over the VPN (for a hive, from /var/lib/swarm-bao-pki/hives/<name>/), then deploy the other machines. The private keys travel over a channel you trust and stay readable by root only. credentials.md lists which leaf each service reads and what a missing one costs. The first-run steps after the deploys are in setup.md.

DNS

Every service reaches another by name, so every name has to resolve on every machine and inside every container. Two views of it exist.

The public view is a handful of records at your DNS provider, pointing at the two public machines:

name	address
<swarm.domain>	swarm-central's public address
forge.<swarm.domain>	swarm-services's public address
chat.<swarm.domain> (the homeserver)	swarm-services's public address
auth.<swarm.domain>	swarm-services's public address

Matrix federation and browsers use these. Names you leave out of public DNS (bao., nats., attic., otel., metrics., logs., grafana. and the hives' dashboards) don't resolve from outside.

The inside view maps every swarm name to the VPN address of the machine that serves it, including the public names, so traffic between machines never uses the internet. The example does it with a static table in dns.nix, written on every machine and built from the same options the services use, so a renamed swarm.*.domain follows. Two readers need it:

• Processes on the machine read /etc/hosts (networking.hosts).
• Containers (agents and swarm services alike) ask the hive's resolver (dnsmasq). It ignores /etc/hosts, so the example hands it the same table through services.dnsmasq.settings.addn-hosts.

A machine leaves out the names it serves itself. hyperhive answers those already, and a table entry would override its answer.

The trade-off: a static table needs no DNS server and no extra moving part, and it lives in the same repository as the rest. It also means a new hive adds a line on every machine, and a name that moves from one machine to another is a rebuild everywhere. For a swarm with many hives, run a small DNS server on the VPN instead, answering the swarm domain with VPN addresses, and point each machine's upstream resolver at it for that domain. That's one place to edit, at the cost of one more service every lookup depends on.

To open Grafana, the store's browser UI or a hive dashboard yourself, join the VPN with your own machine: add it as a peer on each machine and give it the same names, for example in its own /etc/hosts.

Not in the example

• TLS certificates the public can trust. Without gateway.tls.acme or gateway.tls.certDir, the gateway serves certificates from the swarm's own CA, and browsers warn. Set one on swarm-central and swarm-services.
• SSH and the rest of your hosts' base setup. The example touches only the firewall ports it needs.
• The hardware configuration. Each machine's own file sets networking.hostName to one of the five names, which picks its VPN entry.

Wiring one machine into a flake looks like this:

nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules = [
    hyperhive.nixosModules.default
    ./hardware/swarm-central.nix # sets networking.hostName = "swarm-central"
    ./semi-spread/common.nix
    ./semi-spread/swarm-central.nix
  ];
};

hive-one and hive-two add ./semi-spread/hive.nix instead of a machine-specific file.