224 lines
12 KiB
Markdown
224 lines
12 KiB
Markdown
# Example: a semi-spread deploy
|
|
|
|
Three swarm machines and two hives, joined by a WireGuard VPN, with only the
|
|
forge, the matrix homeserver and the swarm UI reachable from the internet. It
|
|
is one of the in-between shapes that [`services.md`](services.md#deployment-shapes)
|
|
supports: the swarm's services run on three machines, not on one box and not
|
|
one per service.
|
|
|
|
The NixOS files are in
|
|
[`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all
|
|
five machines (`checks.module-eval-semi-spread`) and pins the facts on this
|
|
page, so the files stay in step with the options.
|
|
|
|
## The machines
|
|
|
|
| machine | runs | reachable from the internet |
|
|
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
|
|
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI |
|
|
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | forge, matrix, SSO login |
|
|
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing |
|
|
| `hive-one` | the hive `one` | nothing |
|
|
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing |
|
|
|
|
Why this split:
|
|
|
|
- **`swarm-central`** holds what every other machine depends on, and little
|
|
else that can fail. The secret store and the controller are small.
|
|
- **`swarm-services`** holds the services people and agents talk to all day.
|
|
They're light, and they need the public name and port.
|
|
- **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder
|
|
evaluates config pull requests nobody reviewed yet, and the CI runner runs
|
|
pull-request code. Keeping both off the machine that holds the secret store
|
|
limits what a bad build can reach.
|
|
- **The hives** run the agents. They need nothing from the internet but the
|
|
agents' own egress.
|
|
|
|
A hive name can't contain `hive`, `agent` or `swarm`, so the machines
|
|
`hive-one` and `hive-two` run the hives `one` and `two`. `hive.nix` takes the
|
|
hive name from the machine's host name, which is why one file serves both.
|
|
|
|
## What's public and what's not
|
|
|
|
`gateway.openFirewall` opens the gateway's ports (80 and 443) on every
|
|
interface, and the example sets it on two machines only:
|
|
|
|
- `swarm-central` serves the swarm UI at `<swarm.domain>`.
|
|
- `swarm-services` serves the forge (`forge.<swarm.domain>`) and the matrix
|
|
homeserver (`chat.<swarm.domain>`). It also serves `auth.<swarm.domain>`,
|
|
because the forge, matrix and the swarm UI send browsers there to sign in.
|
|
|
|
**Public means the machine's gateway port is open.** A gateway is one nginx
|
|
for everything the machine serves, so the other names on those two machines
|
|
(Grafana, metrics, logs, the collector, the store's browser UI) answer on the
|
|
public address too. Each sits behind SSO or a bearer token, and none of them is
|
|
in public DNS, but they're not closed. A service you want closed goes on a
|
|
machine that opens nothing public, like `swarm-heavyduty`.
|
|
|
|
Everything else uses the VPN. These ports open on `wg0` only, with
|
|
`networking.firewall.interfaces.wg0.allowedTCPPorts`:
|
|
|
|
| machine | port on `wg0` | for |
|
|
| ----------------- | ------------------------- | --------------------------------------------- |
|
|
| `swarm-central` | `swarm.bao.port` (8200) | the secret store, read by every other machine |
|
|
| `swarm-services` | `swarm.nats.port` (4222) | the queue |
|
|
| `swarm-heavyduty` | `gateway.httpsPort` (443) | the cache, behind the gateway |
|
|
| each hive | `gateway.httpsPort` (443) | the hive's dashboard and gateway |
|
|
|
|
The example scopes these to `wg0` because the options that open a service's
|
|
port, `gateway.openFirewall`, `deploy.bao.openFirewall` and
|
|
`deploy.nats.openFirewall`, open it on every interface. The
|
|
secret store and the queue don't rely on the VPN for their security: the store
|
|
requires a client certificate, and the queue requires TLS and refuses clients
|
|
it can't identify. The VPN keeps them off the internet as well.
|
|
|
|
Every machine also opens UDP 51820 on every interface, because that's where
|
|
the VPN's handshake arrives. The example leaves SSH to you.
|
|
|
|
```text
|
|
internet
|
|
80/443 | | 80/443 UDP 51820 on all five
|
|
| |
|
|
swarm-central swarm-services swarm-heavyduty hive-one hive-two
|
|
10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12
|
|
| | | | |
|
|
+----------------+----- wg0, 10.77.0.0/24 --------+----------+
|
|
```
|
|
|
|
## The VPN: raw WireGuard
|
|
|
|
`wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on
|
|
all five machines. `mesh.nix` holds each machine's VPN address, public endpoint
|
|
and public key. The file picks the machine's own entry by `networking.hostName`
|
|
and makes the other four its peers:
|
|
|
|
```nix
|
|
networking.wireguard.interfaces.wg0 = {
|
|
ips = [ "${mesh.${self}.address}/24" ];
|
|
listenPort = 51820;
|
|
privateKeyFile = "/etc/wireguard/wg0.key";
|
|
peers = lib.mapAttrsToList (_: peer: {
|
|
inherit (peer) publicKey endpoint;
|
|
allowedIPs = [ "${peer.address}/32" ];
|
|
persistentKeepalive = 25;
|
|
}) (removeAttrs mesh [ self ]);
|
|
};
|
|
```
|
|
|
|
To set it up:
|
|
|
|
1. On each machine, create the private key in a file only root can read. It
|
|
never goes in the Nix store or in the configuration:
|
|
|
|
```bash
|
|
install -d -m 0700 /etc/wireguard
|
|
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key
|
|
wg pubkey < /etc/wireguard/wg0.key
|
|
```
|
|
|
|
2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public
|
|
keys aren't secret, so the file is safe to commit. You do this once.
|
|
3. Set the `address` and `endpoint` of each machine in `mesh.nix`. The example
|
|
uses documentation addresses.
|
|
4. Deploy. `wg show wg0` on any machine lists four peers with a recent
|
|
handshake, and `ping 10.77.0.1` answers.
|
|
|
|
A machine that can't take incoming connections, such as one behind NAT, sets
|
|
`endpoint = null`. It dials the others, and the keepalive holds the NAT mapping
|
|
open.
|
|
|
|
**Or use tailscale, headscale or NetBird.** They hand out the keys and the
|
|
addresses for you, so the key steps above go away. Three things change:
|
|
|
|
- The interface is no longer `wg0`. Tailscale's is `tailscale0`
|
|
(`services.tailscale.interfaceName`). Put its name in the firewall rules in
|
|
place of `wg0`.
|
|
- Drop `wireguard.nix`, and put each machine's address from your VPN into the
|
|
`address` field of `mesh.nix`. The DNS table below reads only that field.
|
|
- Enable the service, for example `services.tailscale.enable = true` with
|
|
`services.tailscale.authKeyFile` pointing at a key file outside the store.
|
|
|
|
## Reading the store
|
|
|
|
Only `swarm-central` runs the secret store, and every service on the other
|
|
machines reads it with a client certificate that the store's host mints. The
|
|
example names each leaf under `/var/lib/swarm-pki/` on the machine that uses
|
|
it. After the first deploy of `swarm-central`, copy the files from
|
|
`/var/lib/swarm-bao-pki/` there over the VPN (for a hive, from
|
|
`/var/lib/swarm-bao-pki/hives/<name>/`), then deploy the other machines. The
|
|
private keys travel over a channel you trust and stay readable by root only.
|
|
[`credentials.md`](credentials.md) lists which leaf each service reads and what
|
|
a missing one costs. The first-run steps after the deploys are in
|
|
[`setup.md`](../../1-getting-started/setup.md).
|
|
|
|
## DNS
|
|
|
|
Every service reaches another by name, so every name has to resolve on every
|
|
machine and inside every container. Two views of it exist.
|
|
|
|
**The public view** is a handful of records at your DNS provider, pointing at
|
|
the two public machines:
|
|
|
|
| name | address |
|
|
| -------------------------------------- | --------------------------------- |
|
|
| `<swarm.domain>` | `swarm-central`'s public address |
|
|
| `forge.<swarm.domain>` | `swarm-services`'s public address |
|
|
| `chat.<swarm.domain>` (the homeserver) | `swarm-services`'s public address |
|
|
| `auth.<swarm.domain>` | `swarm-services`'s public address |
|
|
|
|
Matrix federation and browsers use these. Names you leave out of public DNS
|
|
(`bao.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
|
|
hives' dashboards) don't resolve from outside.
|
|
|
|
**The inside view** maps every swarm name to the VPN address of the machine
|
|
that serves it, including the public names, so traffic between machines
|
|
never uses the internet. The example does it with a static table in `dns.nix`,
|
|
written on every machine and built from the same options the services use, so
|
|
a renamed `swarm.*.domain` follows. Two readers need it:
|
|
|
|
- Processes on the machine read `/etc/hosts` (`networking.hosts`).
|
|
- Containers (agents and swarm services alike) ask the hive's resolver
|
|
(dnsmasq). It ignores `/etc/hosts`, so the example hands it the same table
|
|
through `services.dnsmasq.settings.addn-hosts`.
|
|
|
|
A machine leaves out the names it serves itself. hyperhive answers those
|
|
already, and a table entry would override its answer.
|
|
|
|
The trade-off: a static table needs no DNS server and no extra moving part, and
|
|
it lives in the same repository as the rest. It also means a new hive adds a
|
|
line on every machine, and a name that moves from one machine to another is a
|
|
rebuild everywhere. For a swarm with many hives, run a small DNS server on the
|
|
VPN instead, answering the swarm domain with VPN addresses, and point each
|
|
machine's upstream resolver at it for that domain. That's one place to edit,
|
|
at the cost of one more service every lookup depends on.
|
|
|
|
To open Grafana, the store's browser UI or a hive dashboard yourself, join the
|
|
VPN with your own machine: add it as a peer on each machine and give it the
|
|
same names, for example in its own `/etc/hosts`.
|
|
|
|
## Not in the example
|
|
|
|
- **TLS certificates the public can trust.** Without `gateway.tls.acme` or
|
|
`gateway.tls.certDir`, the gateway serves certificates from the swarm's own
|
|
CA, and browsers warn. Set one on `swarm-central` and `swarm-services`.
|
|
- **SSH and the rest of your hosts' base setup.** The example touches only the
|
|
firewall ports it needs.
|
|
- **The hardware configuration.** Each machine's own file sets
|
|
`networking.hostName` to one of the five names, which picks its VPN entry.
|
|
|
|
Wiring one machine into a flake looks like this:
|
|
|
|
```nix
|
|
nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem {
|
|
system = "x86_64-linux";
|
|
modules = [
|
|
hyperhive.nixosModules.default
|
|
./hardware/swarm-central.nix # sets networking.hostName = "swarm-central"
|
|
./semi-spread/common.nix
|
|
./semi-spread/swarm-central.nix
|
|
];
|
|
};
|
|
```
|
|
|
|
`hive-one` and `hive-two` add `./semi-spread/hive.nix` instead of a
|
|
machine-specific file.
|