Watch

atlas 32155f70aa the operator's bao works on a store host without a hive

2026-11-11 09:10:50 +01:00

15 KiB

Raw Blame History

Example: a semi-spread deploy

Three swarm machines and two hives, joined by a WireGuard VPN over IPv4 and IPv6. Four things answer to the internet: the swarm UI, the forge, the matrix homeserver and the SSO login they send browsers to. Every other vhost answers on the VPN only. It's one of the in-between shapes that services.md supports: the swarm's services run on three machines, not on one box and not one per service.

The NixOS files are in docs/examples/semi-spread/. CI evaluates all five machines (checks.module-eval-semi-spread) and pins the facts on this page, so the files stay in step with the options.

The machines

machine	runs	answers the internet with	answers the VPN only with
`swarm-central`	`swarm-controller` and the swarm UI, `swarm-bao`	the swarm UI	the store's browser UI; the store itself (port 8200)
`swarm-services`	the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics	the forge, matrix and the SSO login	Grafana, metrics, logs, the collector, the queue's vhost; the queue (4222)
`swarm-heavyduty`	`swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic`	nothing	the cache
`hive-one`	the hive `one`	nothing	the hive's dashboard and domain
`hive-two`	the hive `two`, with the same file as `hive-one`	nothing	the hive's dashboard and domain

Why this split:

swarm-central holds what every other machine depends on, and little else that can fail. The secret store and the controller are small.
swarm-services holds the services people and agents talk to all day. They're light.
swarm-heavyduty holds the CPU, memory and disk hogs. The builder evaluates config pull requests nobody reviewed yet, and the CI runner runs pull-request code. Keeping both off the machine that holds the secret store limits what a bad build can reach.
The hives run the agents. They need nothing from the internet but the agents' own egress.

A hive name can't contain hive, agent or swarm, so the machines hive-one and hive-two run the hives one and two. hive.nix takes the hive name from the machine's host name, which is why one file serves both.

What's public and what's not

Two things decide it: which ports the firewall opens, and which addresses nginx answers on.

Ports. gateway.openFirewall opens the gateway's ports (80 and 443) on every interface, and the example sets it on swarm-central and swarm-services only. Everything else opens on wg0 alone, with networking.firewall.interfaces.wg0.allowedTCPPorts:

machine	port on `wg0`	for
`swarm-central`	`swarm.bao.port` (8200)	the secret store, read by every other machine
`swarm-services`	`swarm.nats.port` (4222)	the queue
`swarm-heavyduty`	`gateway.httpsPort` (443)	the cache, behind the gateway
each hive	`gateway.httpsPort` (443)	the hive's dashboard and gateway

The example scopes these to wg0 because the options that open a service's port, gateway.openFirewall, deploy.bao.openFirewall and deploy.nats.openFirewall, open it on every interface. The secret store and the queue don't rely on the VPN for their security: the store requires a client certificate, and the queue requires TLS and refuses clients it can't identify. The VPN keeps them off the internet as well.

Addresses. The gateway is one nginx per machine, and by default every vhost answers on every address. Each machine here sets gateway.privateListenAddresses (gateway.nix) to its own VPN IPv4 and IPv6 addresses and the hive bridge, and sets public = false on the vhosts that stay inside:

machine	public vhosts	`public = false`
`swarm-central`	`<swarm.domain>` (the swarm UI)	`bao-ui.<swarm.domain>`
`swarm-services`	`forge.`, `chat.` (matrix) and `auth.` (SSO login)	`grafana.`, `metrics.`, `logs.`, `otel.` and `nats.<swarm.domain>`
`swarm-heavyduty`	none	`attic.<swarm.domain>` (the cache)
each hive	none	`<name>.hives.<swarm.domain>` and the hive's own domain

A public vhost listens on 0.0.0.0, on [::], and on the private addresses, so a machine inside the VPN still reaches it by its VPN address. A private vhost listens on the private addresses only. A request for a private name on the public address gets nothing back, the same as any name the gateway doesn't know. The hive bridge is in the list because the machine's own containers reach a name it serves through the bridge address, so a private vhost the containers use still answers them.

SSO stays public. auth.<swarm.domain> answers the internet because the forge, matrix and the swarm UI redirect a browser there to sign in, and a browser outside the VPN has to follow. Authelia authenticates; it needs no VPN as a second lock. If every operator is on the VPN anyway, set public = false on it too, and the sign-in then works from inside the VPN only.

Every machine also opens UDP 51820 on every interface, because that's where the VPN's handshake arrives. The example leaves SSH to you.

                      internet                UDP 51820 reaches all five
         80/443 |                  | 80/443
        (swarm UI)         (forge, matrix, SSO)
   swarm-central   swarm-services   swarm-heavyduty   hive-one   hive-two
   10.77.0.1       10.77.0.2        10.77.0.3         10.77.0.11 10.77.0.12
   fd77:77a1:5e::1 fd77:77a1:5e::2  fd77:77a1:5e::3   …::11      …::12
       |                |                 |               |          |
       +------- wg0: 10.77.0.0/24 and fd77:77a1:5e::/64 --+----------+

The VPN: raw WireGuard

wireguard.nix is plain NixOS, one interface wg0, and it's the same file on all five machines. mesh.nix holds each machine's VPN addresses (IPv4 and IPv6), public endpoint and public key. The file picks the machine's own entry by networking.hostName and makes the other four its peers:

networking.wireguard.interfaces.wg0 = {
  ips = [
    "${mesh.${self}.address}/24"
    "${mesh.${self}.address6}/64"
  ];
  listenPort = 51820;
  privateKeyFile = "/etc/wireguard/wg0.key";
  peers = lib.mapAttrsToList (_: peer: {
    inherit (peer) publicKey endpoint;
    allowedIPs = [
      "${peer.address}/32"
      "${peer.address6}/128"
    ];
    persistentKeepalive = 25;
  }) (removeAttrs mesh [ self ]);
};

To set it up:

On each machine, create the private key in a file only root can read. It never goes in the Nix store or in the configuration:

install -d -m 0700 /etc/wireguard
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key
wg pubkey < /etc/wireguard/wg0.key

Put each machine's public key (the wg pubkey output) in mesh.nix. Public keys aren't secret, so the file is safe to commit. You do this once.
Set the address, address6 and endpoint of each machine in mesh.nix. The example uses documentation addresses for the endpoints and a unique-local IPv6 prefix (fd77:77a1:5e::/64). Generate your own prefix (RFC 4193) rather than keeping it.
Deploy. wg show wg0 on any machine lists four peers with a recent handshake, and ping 10.77.0.1 and ping fd77:77a1:5e::1 answer.

A machine that can't take incoming connections, such as one behind NAT, sets endpoint = null. It dials the others, and the keepalive holds the NAT mapping open.

Or use tailscale, headscale or NetBird. They hand out the keys and the addresses for you, so the key steps above go away. Three things change:

The interface is no longer wg0. Tailscale's is tailscale0 (services.tailscale.interfaceName). Put its name in the firewall rules in place of wg0.
Drop wireguard.nix, and put each machine's addresses from your VPN into the address and address6 fields of mesh.nix. gateway.nix and the DNS table below read only those fields. The addresses must exist on the machine before nginx starts, which a VPN service that comes up late may not guarantee.
Enable the service, for example services.tailscale.enable = true with services.tailscale.authKeyFile pointing at a key file outside the store.

Reading the store

Only swarm-central runs the secret store, and every service on the other machines reads it with a client certificate that the store's host mints, and checks the store against its CA (ca.pem). The example names each leaf, and ca.pem, under /var/lib/swarm-pki/ on the machine that uses it. Without ca.pem the machines don't trust the store's certificate. After the first deploy of swarm-central, copy the files from /var/lib/swarm-bao-pki/ there over the VPN (for a hive, ca.pem and the files from /var/lib/swarm-bao-pki/hives/<name>/), then deploy the other machines. The private keys travel over a channel you trust and stay readable by root only. credentials.md lists which leaf each service reads and what a missing one costs.

On swarm-central itself, bao status, bao operator init and bao operator unseal work as root with no extra step. swarm-central runs no hive, so the bao wrapper presents the operator leaf that the store's host mints for itself. That leaf logs in to no role, so every other call still needs a token. The first-run steps after the deploys are in setup.md.

DNS

Every service reaches another by name, so every name has to resolve on every machine and inside every container. Two views of it exist.

The public view is a handful of records at your DNS provider, pointing at the two public machines. Give each name an A record, and an AAAA record if the machine has a public IPv6 address (the gateway listens on [::]):

name	address
`<swarm.domain>`	`swarm-central`'s public address
`forge.<swarm.domain>`	`swarm-services`'s public address
`chat.<swarm.domain>` (the homeserver)	`swarm-services`'s public address
`auth.<swarm.domain>`	`swarm-services`'s public address

Matrix federation and browsers use these. The names you leave out of public DNS (bao., bao-ui., nats., attic., otel., metrics., logs., grafana. and the hives' dashboards) don't resolve from outside.

The inside view maps every swarm name to the VPN addresses of the machine that serves it, IPv4 and IPv6 (A and AAAA), including the public names, so traffic between machines never uses the internet. The example does it with a static table in dns.nix, written on every machine and built from the same options the services use, so a renamed swarm.*.domain follows. Two readers need it:

Processes on the machine read /etc/hosts (networking.hosts).
Containers (agents and swarm services alike) ask the hive's resolver (dnsmasq). It ignores /etc/hosts, so the example hands it the same table through services.dnsmasq.settings.addn-hosts.

A machine leaves out the names it serves itself. hyperhive answers those already, and a table entry would override its answer.

The trade-off: a static table needs no DNS server and no extra moving part, and it lives in the same repository as the rest. It also means a new hive adds a line on every machine, and a name that moves from one machine to another is a rebuild everywhere. For a swarm with many hives, run a small DNS server on the VPN instead, answering the swarm domain with VPN addresses, and point each machine's upstream resolver at it for that domain. That's one place to edit, at the cost of one more service every lookup depends on.

To open Grafana, the store's browser UI or a hive dashboard yourself, join the VPN with your own machine: add it as a peer on each machine and give it the same names, for example in its own /etc/hosts.

Not in the example

TLS certificates the public can trust. Without gateway.tls.acme or gateway.tls.certDir, the gateway serves certificates from the swarm's own CA, and browsers warn. Set one on swarm-central and swarm-services.
SSH and the rest of your hosts' base setup. The example touches only the firewall ports it needs.
The hardware configuration. Each machine's own file sets networking.hostName to one of the five names, which picks its VPN entry.

Wiring one machine into a flake looks like this:

nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules = [
    hyperhive.nixosModules.default
    ./hardware/swarm-central.nix # sets networking.hostName = "swarm-central"
    ./semi-spread/common.nix
    ./semi-spread/swarm-central.nix
  ];
};

hive-one and hive-two add ./semi-spread/hive.nix instead of a machine-specific file.

15 KiB Raw Blame History