mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

semi-spread example: serve internal vhosts on the VPN addresses only, over IPv4 and IPv6

Browse source
This commit is contained in:
atlas 2026-11-10 00:07:38 +01:00 • committed by mara
commit 7800f92425
2 changed files with 356 additions and 89 deletions
Download patch file Download diff file Expand all files Collapse all files

135
docs/2-handbook/swarm/semi-spread-deploy.md
View file

@ -1,10 +1,11 @@
# Example: a semi-spread deploy
Three swarm machines and two hives, joined by a WireGuard VPN, with only the
forge, the matrix homeserver and the swarm UI reachable from the internet. It
is one of the in-between shapes that [`services.md`](services.md#deployment-shapes)
supports: the swarm's services run on three machines, not on one box and not
one per service.
Three swarm machines and two hives, joined by a WireGuard VPN over IPv4 and
IPv6. Four things answer to the internet: the swarm UI, the forge, the matrix
homeserver and the SSO login they send browsers to. Every other vhost answers
on the VPN only. It's one of the in-between shapes that
[`services.md`](services.md#deployment-shapes) supports: the swarm's services
run on three machines, not on one box and not one per service.
The NixOS files are in
[`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all
@ -13,20 +14,20 @@ page, so the files stay in step with the options.
## The machines
| machine | runs | reachable from the internet |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI |
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | forge, matrix, SSO login |
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing |
| `hive-one` | the hive `one` | nothing |
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing |
| machine | runs | answers the internet with | answers the VPN only with |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------- |
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI | the store's browser UI; the store itself (port 8200) |
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | the forge, matrix and the SSO login | Grafana, metrics, logs, the collector, the queue's vhost; the queue (4222) |
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing | the cache |
| `hive-one` | the hive `one` | nothing | the hive's dashboard and domain |
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing | the hive's dashboard and domain |
Why this split:
- **`swarm-central`** holds what every other machine depends on, and little
else that can fail. The secret store and the controller are small.
- **`swarm-services`** holds the services people and agents talk to all day.
They're light, and they need the public name and port.
They're light.
- **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder
evaluates config pull requests nobody reviewed yet, and the CI runner runs
pull-request code. Keeping both off the machine that holds the secret store
@ -40,22 +41,12 @@ hive name from the machine's host name, which is why one file serves both.
## What's public and what's not
`gateway.openFirewall` opens the gateway's ports (80 and 443) on every
interface, and the example sets it on two machines only:
Two things decide it: which ports the firewall opens, and which addresses
nginx answers on.
- `swarm-central` serves the swarm UI at `<swarm.domain>`.
- `swarm-services` serves the forge (`forge.<swarm.domain>`) and the matrix
homeserver (`chat.<swarm.domain>`). It also serves `auth.<swarm.domain>`,
because the forge, matrix and the swarm UI send browsers there to sign in.
**Public means the machine's gateway port is open.** A gateway is one nginx
for everything the machine serves, so the other names on those two machines
(Grafana, metrics, logs, the collector, the store's browser UI) answer on the
public address too. Each sits behind SSO or a bearer token, and none of them is
in public DNS, but they're not closed. A service you want closed goes on a
machine that opens nothing public, like `swarm-heavyduty`.
Everything else uses the VPN. These ports open on `wg0` only, with
**Ports.** `gateway.openFirewall` opens the gateway's ports (80 and 443) on
every interface, and the example sets it on `swarm-central` and
`swarm-services` only. Everything else opens on `wg0` alone, with
`networking.firewall.interfaces.wg0.allowedTCPPorts`:
| machine | port on `wg0` | for |
@ -67,39 +58,73 @@ Everything else uses the VPN. These ports open on `wg0` only, with
The example scopes these to `wg0` because the options that open a service's
port, `gateway.openFirewall`, `deploy.bao.openFirewall` and
`deploy.nats.openFirewall`, open it on every interface. The
secret store and the queue don't rely on the VPN for their security: the store
requires a client certificate, and the queue requires TLS and refuses clients
it can't identify. The VPN keeps them off the internet as well.
`deploy.nats.openFirewall`, open it on every interface. The secret store and
the queue don't rely on the VPN for their security: the store requires a client
certificate, and the queue requires TLS and refuses clients it can't identify.
The VPN keeps them off the internet as well.
**Addresses.** The gateway is one nginx per machine, and by default every vhost
answers on every address. Each machine here sets
`gateway.privateListenAddresses` (`gateway.nix`) to its own VPN IPv4 and IPv6
addresses and the hive bridge, and sets `public = false` on the vhosts that
stay inside:
| machine | public vhosts | `public = false` |
| ----------------- | -------------------------------------------------- | ------------------------------------------------------------------ |
| `swarm-central` | `<swarm.domain>` (the swarm UI) | `bao-ui.<swarm.domain>` |
| `swarm-services` | `forge.`, `chat.` (matrix) and `auth.` (SSO login) | `grafana.`, `metrics.`, `logs.`, `otel.` and `nats.<swarm.domain>` |
| `swarm-heavyduty` | none | `attic.<swarm.domain>` (the cache) |
| each hive | none | `<name>.hives.<swarm.domain>` and the hive's own domain |
A public vhost listens on `0.0.0.0`, on `[::]`, and on the private addresses,
so a machine inside the VPN still reaches it by its VPN address. A private
vhost listens on the private addresses only. A request for a private name on
the public address gets nothing back, the same as any name the gateway doesn't
know. The hive bridge is in the list because the machine's own containers
reach a name it serves through the bridge address, so a private vhost the
containers use still answers them.
**SSO stays public.** `auth.<swarm.domain>` answers the internet because the
forge, matrix and the swarm UI redirect a browser there to sign in, and a
browser outside the VPN has to follow. Authelia authenticates; it needs no VPN
as a second lock. If every operator is on the VPN anyway, set `public = false`
on it too, and the sign-in then works from inside the VPN only.
Every machine also opens UDP 51820 on every interface, because that's where
the VPN's handshake arrives. The example leaves SSH to you.
```text
internet
80/443 | | 80/443 UDP 51820 on all five
| |
internet UDP 51820 reaches all five
80/443 | | 80/443
(swarm UI) (forge, matrix, SSO)
swarm-central swarm-services swarm-heavyduty hive-one hive-two
10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12
fd77:77a1:5e::1 fd77:77a1:5e::2 fd77:77a1:5e::3 …::11 …::12
| | | | |
+----------------+----- wg0, 10.77.0.0/24 --------+----------+
+------- wg0: 10.77.0.0/24 and fd77:77a1:5e::/64 --+----------+
```
## The VPN: raw WireGuard
`wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on
all five machines. `mesh.nix` holds each machine's VPN address, public endpoint
and public key. The file picks the machine's own entry by `networking.hostName`
all five machines. `mesh.nix` holds each machine's VPN addresses (IPv4 and IPv6),
public endpoint and public key. The file picks the machine's own entry by `networking.hostName`
and makes the other four its peers:
```nix
networking.wireguard.interfaces.wg0 = {
ips = [ "${mesh.${self}.address}/24" ];
ips = [
"${mesh.${self}.address}/24"
"${mesh.${self}.address6}/64"
];
listenPort = 51820;
privateKeyFile = "/etc/wireguard/wg0.key";
peers = lib.mapAttrsToList (_: peer: {
inherit (peer) publicKey endpoint;
allowedIPs = [ "${peer.address}/32" ];
allowedIPs = [
"${peer.address}/32"
"${peer.address6}/128"
];
persistentKeepalive = 25;
}) (removeAttrs mesh [ self ]);
};
@ -118,10 +143,12 @@ To set it up:
2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public
keys aren't secret, so the file is safe to commit. You do this once.
3. Set the `address` and `endpoint` of each machine in `mesh.nix`. The example
uses documentation addresses.
3. Set the `address`, `address6` and `endpoint` of each machine in `mesh.nix`.
The example uses documentation addresses for the endpoints and a
unique-local IPv6 prefix (`fd77:77a1:5e::/64`). Generate your own prefix
(RFC 4193) rather than keeping it.
4. Deploy. `wg show wg0` on any machine lists four peers with a recent
handshake, and `ping 10.77.0.1` answers.
handshake, and `ping 10.77.0.1` and `ping fd77:77a1:5e::1` answer.
A machine that can't take incoming connections, such as one behind NAT, sets
`endpoint = null`. It dials the others, and the keepalive holds the NAT mapping
@ -133,8 +160,10 @@ addresses for you, so the key steps above go away. Three things change:
- The interface is no longer `wg0`. Tailscale's is `tailscale0`
(`services.tailscale.interfaceName`). Put its name in the firewall rules in
place of `wg0`.
- Drop `wireguard.nix`, and put each machine's address from your VPN into the
`address` field of `mesh.nix`. The DNS table below reads only that field.
- Drop `wireguard.nix`, and put each machine's addresses from your VPN into the
`address` and `address6` fields of `mesh.nix`. `gateway.nix` and the DNS table
below read only those fields. The addresses must exist on the machine before
nginx starts, which a VPN service that comes up late may not guarantee.
- Enable the service, for example `services.tailscale.enable = true` with
`services.tailscale.authKeyFile` pointing at a key file outside the store.
@ -157,7 +186,8 @@ Every service reaches another by name, so every name has to resolve on every
machine and inside every container. Two views of it exist.
**The public view** is a handful of records at your DNS provider, pointing at
the two public machines:
the two public machines. Give each name an `A` record, and an `AAAA` record if the
machine has a public IPv6 address (the gateway listens on `[::]`):
| name | address |
| -------------------------------------- | --------------------------------- |
@ -166,14 +196,15 @@ the two public machines:
| `chat.<swarm.domain>` (the homeserver) | `swarm-services`'s public address |
| `auth.<swarm.domain>` | `swarm-services`'s public address |
Matrix federation and browsers use these. Names you leave out of public DNS
(`bao.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
Matrix federation and browsers use these. The names you leave out of public DNS
(`bao.`, `bao-ui.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
hives' dashboards) don't resolve from outside.
**The inside view** maps every swarm name to the VPN address of the machine
that serves it, including the public names, so traffic between machines
never uses the internet. The example does it with a static table in `dns.nix`,
written on every machine and built from the same options the services use, so
**The inside view** maps every swarm name to the VPN addresses of the machine
that serves it, IPv4 and IPv6 (`A` and `AAAA`), including the public names, so
traffic between machines never uses the internet. The example does it with a
static table in `dns.nix`, written on every machine and built from the same
options the services use, so
a renamed `swarm.*.domain` follows. Two readers need it:
- Processes on the machine read `/etc/hosts` (`networking.hosts`).

3
docs/2-handbook/swarm/services.md
View file

@ -81,7 +81,8 @@ the shapes worth naming are the ones they produce:
- **Fully spread out.** One container / VM / machine per service,
somewhere.
- **Semi-spread.** The services split over a few machines by weight, hives on
their own, and a VPN between them. A worked example with its NixOS files:
their own, a VPN between them, and most vhosts answering on the VPN only.
A worked example with its NixOS files:
[`semi-spread-deploy.md`](semi-spread-deploy.md).
**These are a set, not a ladder with a correct top, and hyperhive