255 lines
14 KiB
Markdown
255 lines
14 KiB
Markdown
# Example: a semi-spread deploy
|
|
|
|
Three swarm machines and two hives, joined by a WireGuard VPN over IPv4 and
|
|
IPv6. Four things answer to the internet: the swarm UI, the forge, the matrix
|
|
homeserver and the SSO login they send browsers to. Every other vhost answers
|
|
on the VPN only. It's one of the in-between shapes that
|
|
[`services.md`](services.md#deployment-shapes) supports: the swarm's services
|
|
run on three machines, not on one box and not one per service.
|
|
|
|
The NixOS files are in
|
|
[`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all
|
|
five machines (`checks.module-eval-semi-spread`) and pins the facts on this
|
|
page, so the files stay in step with the options.
|
|
|
|
## The machines
|
|
|
|
| machine | runs | answers the internet with | answers the VPN only with |
|
|
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------- |
|
|
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI | the store's browser UI; the store itself (port 8200) |
|
|
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | the forge, matrix and the SSO login | Grafana, metrics, logs, the collector, the queue's vhost; the queue (4222) |
|
|
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing | the cache |
|
|
| `hive-one` | the hive `one` | nothing | the hive's dashboard and domain |
|
|
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing | the hive's dashboard and domain |
|
|
|
|
Why this split:
|
|
|
|
- **`swarm-central`** holds what every other machine depends on, and little
|
|
else that can fail. The secret store and the controller are small.
|
|
- **`swarm-services`** holds the services people and agents talk to all day.
|
|
They're light.
|
|
- **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder
|
|
evaluates config pull requests nobody reviewed yet, and the CI runner runs
|
|
pull-request code. Keeping both off the machine that holds the secret store
|
|
limits what a bad build can reach.
|
|
- **The hives** run the agents. They need nothing from the internet but the
|
|
agents' own egress.
|
|
|
|
A hive name can't contain `hive`, `agent` or `swarm`, so the machines
|
|
`hive-one` and `hive-two` run the hives `one` and `two`. `hive.nix` takes the
|
|
hive name from the machine's host name, which is why one file serves both.
|
|
|
|
## What's public and what's not
|
|
|
|
Two things decide it: which ports the firewall opens, and which addresses
|
|
nginx answers on.
|
|
|
|
**Ports.** `gateway.openFirewall` opens the gateway's ports (80 and 443) on
|
|
every interface, and the example sets it on `swarm-central` and
|
|
`swarm-services` only. Everything else opens on `wg0` alone, with
|
|
`networking.firewall.interfaces.wg0.allowedTCPPorts`:
|
|
|
|
| machine | port on `wg0` | for |
|
|
| ----------------- | ------------------------- | --------------------------------------------- |
|
|
| `swarm-central` | `swarm.bao.port` (8200) | the secret store, read by every other machine |
|
|
| `swarm-services` | `swarm.nats.port` (4222) | the queue |
|
|
| `swarm-heavyduty` | `gateway.httpsPort` (443) | the cache, behind the gateway |
|
|
| each hive | `gateway.httpsPort` (443) | the hive's dashboard and gateway |
|
|
|
|
The example scopes these to `wg0` because the options that open a service's
|
|
port, `gateway.openFirewall`, `deploy.bao.openFirewall` and
|
|
`deploy.nats.openFirewall`, open it on every interface. The secret store and
|
|
the queue don't rely on the VPN for their security: the store requires a client
|
|
certificate, and the queue requires TLS and refuses clients it can't identify.
|
|
The VPN keeps them off the internet as well.
|
|
|
|
**Addresses.** The gateway is one nginx per machine, and by default every vhost
|
|
answers on every address. Each machine here sets
|
|
`gateway.privateListenAddresses` (`gateway.nix`) to its own VPN IPv4 and IPv6
|
|
addresses and the hive bridge, and sets `public = false` on the vhosts that
|
|
stay inside:
|
|
|
|
| machine | public vhosts | `public = false` |
|
|
| ----------------- | -------------------------------------------------- | ------------------------------------------------------------------ |
|
|
| `swarm-central` | `<swarm.domain>` (the swarm UI) | `bao-ui.<swarm.domain>` |
|
|
| `swarm-services` | `forge.`, `chat.` (matrix) and `auth.` (SSO login) | `grafana.`, `metrics.`, `logs.`, `otel.` and `nats.<swarm.domain>` |
|
|
| `swarm-heavyduty` | none | `attic.<swarm.domain>` (the cache) |
|
|
| each hive | none | `<name>.hives.<swarm.domain>` and the hive's own domain |
|
|
|
|
A public vhost listens on `0.0.0.0`, on `[::]`, and on the private addresses,
|
|
so a machine inside the VPN still reaches it by its VPN address. A private
|
|
vhost listens on the private addresses only. A request for a private name on
|
|
the public address gets nothing back, the same as any name the gateway doesn't
|
|
know. The hive bridge is in the list because the machine's own containers
|
|
reach a name it serves through the bridge address, so a private vhost the
|
|
containers use still answers them.
|
|
|
|
**SSO stays public.** `auth.<swarm.domain>` answers the internet because the
|
|
forge, matrix and the swarm UI redirect a browser there to sign in, and a
|
|
browser outside the VPN has to follow. Authelia authenticates; it needs no VPN
|
|
as a second lock. If every operator is on the VPN anyway, set `public = false`
|
|
on it too, and the sign-in then works from inside the VPN only.
|
|
|
|
Every machine also opens UDP 51820 on every interface, because that's where
|
|
the VPN's handshake arrives. The example leaves SSH to you.
|
|
|
|
```text
|
|
internet UDP 51820 reaches all five
|
|
80/443 | | 80/443
|
|
(swarm UI) (forge, matrix, SSO)
|
|
swarm-central swarm-services swarm-heavyduty hive-one hive-two
|
|
10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12
|
|
fd77:77a1:5e::1 fd77:77a1:5e::2 fd77:77a1:5e::3 …::11 …::12
|
|
| | | | |
|
|
+------- wg0: 10.77.0.0/24 and fd77:77a1:5e::/64 --+----------+
|
|
```
|
|
|
|
## The VPN: raw WireGuard
|
|
|
|
`wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on
|
|
all five machines. `mesh.nix` holds each machine's VPN addresses (IPv4 and IPv6),
|
|
public endpoint and public key. The file picks the machine's own entry by `networking.hostName`
|
|
and makes the other four its peers:
|
|
|
|
```nix
|
|
networking.wireguard.interfaces.wg0 = {
|
|
ips = [
|
|
"${mesh.${self}.address}/24"
|
|
"${mesh.${self}.address6}/64"
|
|
];
|
|
listenPort = 51820;
|
|
privateKeyFile = "/etc/wireguard/wg0.key";
|
|
peers = lib.mapAttrsToList (_: peer: {
|
|
inherit (peer) publicKey endpoint;
|
|
allowedIPs = [
|
|
"${peer.address}/32"
|
|
"${peer.address6}/128"
|
|
];
|
|
persistentKeepalive = 25;
|
|
}) (removeAttrs mesh [ self ]);
|
|
};
|
|
```
|
|
|
|
To set it up:
|
|
|
|
1. On each machine, create the private key in a file only root can read. It
|
|
never goes in the Nix store or in the configuration:
|
|
|
|
```bash
|
|
install -d -m 0700 /etc/wireguard
|
|
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key
|
|
wg pubkey < /etc/wireguard/wg0.key
|
|
```
|
|
|
|
2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public
|
|
keys aren't secret, so the file is safe to commit. You do this once.
|
|
3. Set the `address`, `address6` and `endpoint` of each machine in `mesh.nix`.
|
|
The example uses documentation addresses for the endpoints and a
|
|
unique-local IPv6 prefix (`fd77:77a1:5e::/64`). Generate your own prefix
|
|
(RFC 4193) rather than keeping it.
|
|
4. Deploy. `wg show wg0` on any machine lists four peers with a recent
|
|
handshake, and `ping 10.77.0.1` and `ping fd77:77a1:5e::1` answer.
|
|
|
|
A machine that can't take incoming connections, such as one behind NAT, sets
|
|
`endpoint = null`. It dials the others, and the keepalive holds the NAT mapping
|
|
open.
|
|
|
|
**Or use tailscale, headscale or NetBird.** They hand out the keys and the
|
|
addresses for you, so the key steps above go away. Three things change:
|
|
|
|
- The interface is no longer `wg0`. Tailscale's is `tailscale0`
|
|
(`services.tailscale.interfaceName`). Put its name in the firewall rules in
|
|
place of `wg0`.
|
|
- Drop `wireguard.nix`, and put each machine's addresses from your VPN into the
|
|
`address` and `address6` fields of `mesh.nix`. `gateway.nix` and the DNS table
|
|
below read only those fields. The addresses must exist on the machine before
|
|
nginx starts, which a VPN service that comes up late may not guarantee.
|
|
- Enable the service, for example `services.tailscale.enable = true` with
|
|
`services.tailscale.authKeyFile` pointing at a key file outside the store.
|
|
|
|
## Reading the store
|
|
|
|
Only `swarm-central` runs the secret store, and every service on the other
|
|
machines reads it with a client certificate that the store's host mints. The
|
|
example names each leaf under `/var/lib/swarm-pki/` on the machine that uses
|
|
it. After the first deploy of `swarm-central`, copy the files from
|
|
`/var/lib/swarm-bao-pki/` there over the VPN (for a hive, from
|
|
`/var/lib/swarm-bao-pki/hives/<name>/`), then deploy the other machines. The
|
|
private keys travel over a channel you trust and stay readable by root only.
|
|
[`credentials.md`](credentials.md) lists which leaf each service reads and what
|
|
a missing one costs. The first-run steps after the deploys are in
|
|
[`setup.md`](../../1-getting-started/setup.md).
|
|
|
|
## DNS
|
|
|
|
Every service reaches another by name, so every name has to resolve on every
|
|
machine and inside every container. Two views of it exist.
|
|
|
|
**The public view** is a handful of records at your DNS provider, pointing at
|
|
the two public machines. Give each name an `A` record, and an `AAAA` record if the
|
|
machine has a public IPv6 address (the gateway listens on `[::]`):
|
|
|
|
| name | address |
|
|
| -------------------------------------- | --------------------------------- |
|
|
| `<swarm.domain>` | `swarm-central`'s public address |
|
|
| `forge.<swarm.domain>` | `swarm-services`'s public address |
|
|
| `chat.<swarm.domain>` (the homeserver) | `swarm-services`'s public address |
|
|
| `auth.<swarm.domain>` | `swarm-services`'s public address |
|
|
|
|
Matrix federation and browsers use these. The names you leave out of public DNS
|
|
(`bao.`, `bao-ui.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
|
|
hives' dashboards) don't resolve from outside.
|
|
|
|
**The inside view** maps every swarm name to the VPN addresses of the machine
|
|
that serves it, IPv4 and IPv6 (`A` and `AAAA`), including the public names, so
|
|
traffic between machines never uses the internet. The example does it with a
|
|
static table in `dns.nix`, written on every machine and built from the same
|
|
options the services use, so
|
|
a renamed `swarm.*.domain` follows. Two readers need it:
|
|
|
|
- Processes on the machine read `/etc/hosts` (`networking.hosts`).
|
|
- Containers (agents and swarm services alike) ask the hive's resolver
|
|
(dnsmasq). It ignores `/etc/hosts`, so the example hands it the same table
|
|
through `services.dnsmasq.settings.addn-hosts`.
|
|
|
|
A machine leaves out the names it serves itself. hyperhive answers those
|
|
already, and a table entry would override its answer.
|
|
|
|
The trade-off: a static table needs no DNS server and no extra moving part, and
|
|
it lives in the same repository as the rest. It also means a new hive adds a
|
|
line on every machine, and a name that moves from one machine to another is a
|
|
rebuild everywhere. For a swarm with many hives, run a small DNS server on the
|
|
VPN instead, answering the swarm domain with VPN addresses, and point each
|
|
machine's upstream resolver at it for that domain. That's one place to edit,
|
|
at the cost of one more service every lookup depends on.
|
|
|
|
To open Grafana, the store's browser UI or a hive dashboard yourself, join the
|
|
VPN with your own machine: add it as a peer on each machine and give it the
|
|
same names, for example in its own `/etc/hosts`.
|
|
|
|
## Not in the example
|
|
|
|
- **TLS certificates the public can trust.** Without `gateway.tls.acme` or
|
|
`gateway.tls.certDir`, the gateway serves certificates from the swarm's own
|
|
CA, and browsers warn. Set one on `swarm-central` and `swarm-services`.
|
|
- **SSH and the rest of your hosts' base setup.** The example touches only the
|
|
firewall ports it needs.
|
|
- **The hardware configuration.** Each machine's own file sets
|
|
`networking.hostName` to one of the five names, which picks its VPN entry.
|
|
|
|
Wiring one machine into a flake looks like this:
|
|
|
|
```nix
|
|
nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem {
|
|
system = "x86_64-linux";
|
|
modules = [
|
|
hyperhive.nixosModules.default
|
|
./hardware/swarm-central.nix # sets networking.hostName = "swarm-central"
|
|
./semi-spread/common.nix
|
|
./semi-spread/swarm-central.nix
|
|
];
|
|
};
|
|
```
|
|
|
|
`hive-one` and `hive-two` add `./semi-spread/hive.nix` instead of a
|
|
machine-specific file.
|