diff --git a/docs/2-handbook/swarm/semi-spread-deploy.md b/docs/2-handbook/swarm/semi-spread-deploy.md index 3dd13f3d0..8441ca5cf 100644 --- a/docs/2-handbook/swarm/semi-spread-deploy.md +++ b/docs/2-handbook/swarm/semi-spread-deploy.md @@ -1,10 +1,11 @@ # Example: a semi-spread deploy -Three swarm machines and two hives, joined by a WireGuard VPN, with only the -forge, the matrix homeserver and the swarm UI reachable from the internet. It -is one of the in-between shapes that [`services.md`](services.md#deployment-shapes) -supports: the swarm's services run on three machines, not on one box and not -one per service. +Three swarm machines and two hives, joined by a WireGuard VPN over IPv4 and +IPv6. Four things answer to the internet: the swarm UI, the forge, the matrix +homeserver and the SSO login they send browsers to. Every other vhost answers +on the VPN only. It's one of the in-between shapes that +[`services.md`](services.md#deployment-shapes) supports: the swarm's services +run on three machines, not on one box and not one per service. The NixOS files are in [`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all @@ -13,20 +14,20 @@ page, so the files stay in step with the options. ## The machines -| machine | runs | reachable from the internet | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | -| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI | -| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | forge, matrix, SSO login | -| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing | -| `hive-one` | the hive `one` | nothing | -| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing | +| machine | runs | answers the internet with | answers the VPN only with | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------- | +| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI | the store's browser UI; the store itself (port 8200) | +| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | the forge, matrix and the SSO login | Grafana, metrics, logs, the collector, the queue's vhost; the queue (4222) | +| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing | the cache | +| `hive-one` | the hive `one` | nothing | the hive's dashboard and domain | +| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing | the hive's dashboard and domain | Why this split: - **`swarm-central`** holds what every other machine depends on, and little else that can fail. The secret store and the controller are small. - **`swarm-services`** holds the services people and agents talk to all day. - They're light, and they need the public name and port. + They're light. - **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder evaluates config pull requests nobody reviewed yet, and the CI runner runs pull-request code. Keeping both off the machine that holds the secret store @@ -40,22 +41,12 @@ hive name from the machine's host name, which is why one file serves both. ## What's public and what's not -`gateway.openFirewall` opens the gateway's ports (80 and 443) on every -interface, and the example sets it on two machines only: +Two things decide it: which ports the firewall opens, and which addresses +nginx answers on. -- `swarm-central` serves the swarm UI at ``. -- `swarm-services` serves the forge (`forge.`) and the matrix - homeserver (`chat.`). It also serves `auth.`, - because the forge, matrix and the swarm UI send browsers there to sign in. - -**Public means the machine's gateway port is open.** A gateway is one nginx -for everything the machine serves, so the other names on those two machines -(Grafana, metrics, logs, the collector, the store's browser UI) answer on the -public address too. Each sits behind SSO or a bearer token, and none of them is -in public DNS, but they're not closed. A service you want closed goes on a -machine that opens nothing public, like `swarm-heavyduty`. - -Everything else uses the VPN. These ports open on `wg0` only, with +**Ports.** `gateway.openFirewall` opens the gateway's ports (80 and 443) on +every interface, and the example sets it on `swarm-central` and +`swarm-services` only. Everything else opens on `wg0` alone, with `networking.firewall.interfaces.wg0.allowedTCPPorts`: | machine | port on `wg0` | for | @@ -67,39 +58,73 @@ Everything else uses the VPN. These ports open on `wg0` only, with The example scopes these to `wg0` because the options that open a service's port, `gateway.openFirewall`, `deploy.bao.openFirewall` and -`deploy.nats.openFirewall`, open it on every interface. The -secret store and the queue don't rely on the VPN for their security: the store -requires a client certificate, and the queue requires TLS and refuses clients -it can't identify. The VPN keeps them off the internet as well. +`deploy.nats.openFirewall`, open it on every interface. The secret store and +the queue don't rely on the VPN for their security: the store requires a client +certificate, and the queue requires TLS and refuses clients it can't identify. +The VPN keeps them off the internet as well. + +**Addresses.** The gateway is one nginx per machine, and by default every vhost +answers on every address. Each machine here sets +`gateway.privateListenAddresses` (`gateway.nix`) to its own VPN IPv4 and IPv6 +addresses and the hive bridge, and sets `public = false` on the vhosts that +stay inside: + +| machine | public vhosts | `public = false` | +| ----------------- | -------------------------------------------------- | ------------------------------------------------------------------ | +| `swarm-central` | `` (the swarm UI) | `bao-ui.` | +| `swarm-services` | `forge.`, `chat.` (matrix) and `auth.` (SSO login) | `grafana.`, `metrics.`, `logs.`, `otel.` and `nats.` | +| `swarm-heavyduty` | none | `attic.` (the cache) | +| each hive | none | `.hives.` and the hive's own domain | + +A public vhost listens on `0.0.0.0`, on `[::]`, and on the private addresses, +so a machine inside the VPN still reaches it by its VPN address. A private +vhost listens on the private addresses only. A request for a private name on +the public address gets nothing back, the same as any name the gateway doesn't +know. The hive bridge is in the list because the machine's own containers +reach a name it serves through the bridge address, so a private vhost the +containers use still answers them. + +**SSO stays public.** `auth.` answers the internet because the +forge, matrix and the swarm UI redirect a browser there to sign in, and a +browser outside the VPN has to follow. Authelia authenticates; it needs no VPN +as a second lock. If every operator is on the VPN anyway, set `public = false` +on it too, and the sign-in then works from inside the VPN only. Every machine also opens UDP 51820 on every interface, because that's where the VPN's handshake arrives. The example leaves SSH to you. ```text - internet - 80/443 | | 80/443 UDP 51820 on all five - | | + internet UDP 51820 reaches all five + 80/443 | | 80/443 + (swarm UI) (forge, matrix, SSO) swarm-central swarm-services swarm-heavyduty hive-one hive-two 10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12 + fd77:77a1:5e::1 fd77:77a1:5e::2 fd77:77a1:5e::3 …::11 …::12 | | | | | - +----------------+----- wg0, 10.77.0.0/24 --------+----------+ + +------- wg0: 10.77.0.0/24 and fd77:77a1:5e::/64 --+----------+ ``` ## The VPN: raw WireGuard `wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on -all five machines. `mesh.nix` holds each machine's VPN address, public endpoint -and public key. The file picks the machine's own entry by `networking.hostName` +all five machines. `mesh.nix` holds each machine's VPN addresses (IPv4 and IPv6), +public endpoint and public key. The file picks the machine's own entry by `networking.hostName` and makes the other four its peers: ```nix networking.wireguard.interfaces.wg0 = { - ips = [ "${mesh.${self}.address}/24" ]; + ips = [ + "${mesh.${self}.address}/24" + "${mesh.${self}.address6}/64" + ]; listenPort = 51820; privateKeyFile = "/etc/wireguard/wg0.key"; peers = lib.mapAttrsToList (_: peer: { inherit (peer) publicKey endpoint; - allowedIPs = [ "${peer.address}/32" ]; + allowedIPs = [ + "${peer.address}/32" + "${peer.address6}/128" + ]; persistentKeepalive = 25; }) (removeAttrs mesh [ self ]); }; @@ -118,10 +143,12 @@ To set it up: 2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public keys aren't secret, so the file is safe to commit. You do this once. -3. Set the `address` and `endpoint` of each machine in `mesh.nix`. The example - uses documentation addresses. +3. Set the `address`, `address6` and `endpoint` of each machine in `mesh.nix`. + The example uses documentation addresses for the endpoints and a + unique-local IPv6 prefix (`fd77:77a1:5e::/64`). Generate your own prefix + (RFC 4193) rather than keeping it. 4. Deploy. `wg show wg0` on any machine lists four peers with a recent - handshake, and `ping 10.77.0.1` answers. + handshake, and `ping 10.77.0.1` and `ping fd77:77a1:5e::1` answer. A machine that can't take incoming connections, such as one behind NAT, sets `endpoint = null`. It dials the others, and the keepalive holds the NAT mapping @@ -133,8 +160,10 @@ addresses for you, so the key steps above go away. Three things change: - The interface is no longer `wg0`. Tailscale's is `tailscale0` (`services.tailscale.interfaceName`). Put its name in the firewall rules in place of `wg0`. -- Drop `wireguard.nix`, and put each machine's address from your VPN into the - `address` field of `mesh.nix`. The DNS table below reads only that field. +- Drop `wireguard.nix`, and put each machine's addresses from your VPN into the + `address` and `address6` fields of `mesh.nix`. `gateway.nix` and the DNS table + below read only those fields. The addresses must exist on the machine before + nginx starts, which a VPN service that comes up late may not guarantee. - Enable the service, for example `services.tailscale.enable = true` with `services.tailscale.authKeyFile` pointing at a key file outside the store. @@ -157,7 +186,8 @@ Every service reaches another by name, so every name has to resolve on every machine and inside every container. Two views of it exist. **The public view** is a handful of records at your DNS provider, pointing at -the two public machines: +the two public machines. Give each name an `A` record, and an `AAAA` record if the +machine has a public IPv6 address (the gateway listens on `[::]`): | name | address | | -------------------------------------- | --------------------------------- | @@ -166,14 +196,15 @@ the two public machines: | `chat.` (the homeserver) | `swarm-services`'s public address | | `auth.` | `swarm-services`'s public address | -Matrix federation and browsers use these. Names you leave out of public DNS -(`bao.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the +Matrix federation and browsers use these. The names you leave out of public DNS +(`bao.`, `bao-ui.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the hives' dashboards) don't resolve from outside. -**The inside view** maps every swarm name to the VPN address of the machine -that serves it, including the public names, so traffic between machines -never uses the internet. The example does it with a static table in `dns.nix`, -written on every machine and built from the same options the services use, so +**The inside view** maps every swarm name to the VPN addresses of the machine +that serves it, IPv4 and IPv6 (`A` and `AAAA`), including the public names, so +traffic between machines never uses the internet. The example does it with a +static table in `dns.nix`, written on every machine and built from the same +options the services use, so a renamed `swarm.*.domain` follows. Two readers need it: - Processes on the machine read `/etc/hosts` (`networking.hosts`). diff --git a/docs/2-handbook/swarm/services.md b/docs/2-handbook/swarm/services.md index e82697672..328b2cc49 100644 --- a/docs/2-handbook/swarm/services.md +++ b/docs/2-handbook/swarm/services.md @@ -81,7 +81,8 @@ the shapes worth naming are the ones they produce: - **Fully spread out.** One container / VM / machine per service, somewhere. - **Semi-spread.** The services split over a few machines by weight, hives on - their own, and a VPN between them. A worked example with its NixOS files: + their own, a VPN between them, and most vhosts answering on the VPN only. + A worked example with its NixOS files: [`semi-spread-deploy.md`](semi-spread-deploy.md). **These are a set, not a ladder with a correct top, and hyperhive diff --git a/docs/README.md b/docs/README.md index 17651286c..f0b1f8660 100644 --- a/docs/README.md +++ b/docs/README.md @@ -32,8 +32,8 @@ The services that exist once per swarm. Start at - **How are two hives connected into a swarm?** → [`2-handbook/swarm/`](2-handbook/swarm/README.md) (peer hives, TLS trust). -- **How to spread the swarm over a few machines, with a VPN and only the forge, - matrix and swarm UI public?** → [`2-handbook/swarm/semi-spread-deploy.md`](2-handbook/swarm/semi-spread-deploy.md) +- **How to spread the swarm over a few machines, with a VPN, and only the swarm UI, forge, + matrix and SSO login answering the internet?** → [`2-handbook/swarm/semi-spread-deploy.md`](2-handbook/swarm/semi-spread-deploy.md) (worked example: machines, WireGuard, DNS). - **What does the swarm UI show, and what does `swarmctl` do?** → [`2-handbook/swarm/ui.md`](2-handbook/swarm/ui.md) and diff --git a/docs/examples/semi-spread/common.nix b/docs/examples/semi-spread/common.nix index 07dca2226..b97068294 100644 --- a/docs/examples/semi-spread/common.nix +++ b/docs/examples/semi-spread/common.nix @@ -4,6 +4,7 @@ { imports = [ ./wireguard.nix + ./gateway.nix ./dns.nix ]; diff --git a/docs/examples/semi-spread/dns.nix b/docs/examples/semi-spread/dns.nix index ff5e557cb..ce2736eca 100644 --- a/docs/examples/semi-spread/dns.nix +++ b/docs/examples/semi-spread/dns.nix @@ -1,6 +1,6 @@ -# Names inside the VPN. Every swarm name that is not public resolves to the -# VPN address of the machine serving it, through a static table written on -# each host. Identical on every host. +# Names inside the VPN. Every swarm name, the public ones included, resolves to +# the VPN addresses (IPv4 and IPv6) of the machine serving it, through a static +# table written on each host. Identical on every host. # # Two readers need the table. Processes on the host read /etc/hosts # (`networking.hosts`). Containers (agents, swarm services) ask the hive's @@ -49,17 +49,20 @@ let }; elsewhere = removeAttrs servedBy [ self ]; + + # Each machine's names, once per address family: A and AAAA answers. + addressesOf = machine: [ + mesh.${machine}.address + mesh.${machine}.address6 + ]; + table = lib.concatMap ( + machine: map (address: lib.nameValuePair address elsewhere.${machine}) (addressesOf machine) + ) (builtins.attrNames elsewhere); in { - networking.hosts = lib.mapAttrs' ( - machine: names: lib.nameValuePair mesh.${machine}.address names - ) elsewhere; + networking.hosts = lib.listToAttrs table; services.dnsmasq.settings.addn-hosts = "${builtins.toFile "swarm-vpn-hosts" ( - lib.concatStringsSep "\n" ( - lib.mapAttrsToList ( - machine: names: "${mesh.${machine}.address} ${lib.concatStringsSep " " names}" - ) elsewhere - ) + lib.concatMapStrings (entry: "${entry.name} ${lib.concatStringsSep " " entry.value}\n") table )}"; } diff --git a/docs/examples/semi-spread/gateway.nix b/docs/examples/semi-spread/gateway.nix new file mode 100644 index 000000000..194af7840 --- /dev/null +++ b/docs/examples/semi-spread/gateway.nix @@ -0,0 +1,20 @@ +# Which addresses the gateway's private vhosts answer on: this machine's VPN +# addresses (IPv4 and IPv6) and the hive bridge. Identical on every host. +# Each machine's own file lists its private vhosts (`public = false`); a vhost +# that stays `public` also answers on these addresses. +# +# The bridge is how this machine's own containers (agents, swarm services) +# reach a name it serves: the hive's resolver answers it with the bridge +# address. +{ config, ... }: +let + mesh = import ./mesh.nix; + me = mesh.${config.networking.hostName}; +in +{ + services.hyperhive.gateway.privateListenAddresses = [ + me.address + me.address6 + config.services.hyperhive.network.bridgeIp + ]; +} diff --git a/docs/examples/semi-spread/hive.nix b/docs/examples/semi-spread/hive.nix index f490d126a..498e09e18 100644 --- a/docs/examples/semi-spread/hive.nix +++ b/docs/examples/semi-spread/hive.nix @@ -2,6 +2,9 @@ # hive ``. { config, lib, ... }: let + hyperhive = config.services.hyperhive; + name = lib.removePrefix "hive-" config.networking.hostName; + # The leaves for this hive, copied from swarm-central, where the secret store # runs: `client.pem` and `client-key.pem` from # `/var/lib/swarm-bao-pki/hives//`, along with that directory's @@ -10,19 +13,30 @@ let pki = "/var/lib/swarm-pki"; in { - services.hyperhive.deploy = { - hive-controller = { - enable = true; - hiveName = lib.removePrefix "hive-" config.networking.hostName; - }; - bao = { - clientCertFile = "${pki}/client.pem"; - clientKeyFile = "${pki}/client-key.pem"; + services.hyperhive = { + deploy = { + hive-controller = { + enable = true; + hiveName = name; + }; + bao = { + clientCertFile = "${pki}/client.pem"; + clientKeyFile = "${pki}/client-key.pem"; + }; }; + + # The hive's dashboard and its domain answer on the private addresses only. + gateway.vhosts = + lib.genAttrs + [ + hyperhive.swarm.dashboardDomains.${name} + hyperhive.swarm.hives.${name}.domain + ] + (_: { + public = false; + }); }; # The dashboard is reachable over the VPN only. - networking.firewall.interfaces.wg0.allowedTCPPorts = [ - config.services.hyperhive.gateway.httpsPort - ]; + networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.gateway.httpsPort ]; } diff --git a/docs/examples/semi-spread/mesh.nix b/docs/examples/semi-spread/mesh.nix index c7fefad7b..71d978dfd 100644 --- a/docs/examples/semi-spread/mesh.nix +++ b/docs/examples/semi-spread/mesh.nix @@ -1,32 +1,40 @@ # The VPN's facts: one entry per machine, read by every host (wireguard.nix, -# dns.nix) and nothing else. Keyed by `networking.hostName`. +# gateway.nix, dns.nix) and nothing else. Keyed by `networking.hostName`. # -# `address` is the machine's address inside the VPN; `endpoint` is where the -# other machines dial it; `publicKey` is the output of `wg pubkey` on that -# machine. The keys below are placeholders: replace each with the real one. +# `address` and `address6` are the machine's addresses inside the VPN (IPv4 +# 10.77.0.0/24 and the unique-local IPv6 prefix fd77:77a1:5e::/64); +# `endpoint` is where the other machines dial it; `publicKey` is the output of +# `wg pubkey` on that machine. The keys below are placeholders: replace each +# with the real one. Generate your own IPv6 prefix (RFC 4193) rather than +# keeping this one. { swarm-central = { address = "10.77.0.1"; + address6 = "fd77:77a1:5e::1"; endpoint = "203.0.113.10:51820"; publicKey = "REPLACE-with-swarm-central-wg-pubkey"; }; swarm-services = { address = "10.77.0.2"; + address6 = "fd77:77a1:5e::2"; endpoint = "203.0.113.11:51820"; publicKey = "REPLACE-with-swarm-services-wg-pubkey"; }; swarm-heavyduty = { address = "10.77.0.3"; + address6 = "fd77:77a1:5e::3"; endpoint = "203.0.113.12:51820"; publicKey = "REPLACE-with-swarm-heavyduty-wg-pubkey"; }; hive-one = { address = "10.77.0.11"; + address6 = "fd77:77a1:5e::11"; endpoint = "198.51.100.11:51820"; publicKey = "REPLACE-with-hive-one-wg-pubkey"; }; hive-two = { address = "10.77.0.12"; + address6 = "fd77:77a1:5e::12"; endpoint = "198.51.100.12:51820"; publicKey = "REPLACE-with-hive-two-wg-pubkey"; }; diff --git a/docs/examples/semi-spread/swarm-central.nix b/docs/examples/semi-spread/swarm-central.nix index 0aec98c17..a1fdaabf8 100644 --- a/docs/examples/semi-spread/swarm-central.nix +++ b/docs/examples/semi-spread/swarm-central.nix @@ -1,6 +1,6 @@ # swarm-central: the swarm controller (and the swarm UI it serves) and the # secret store. -{ config, ... }: +{ config, lib, ... }: let mesh = import ./mesh.nix; hyperhive = config.services.hyperhive; @@ -14,8 +14,13 @@ in # and hives on other machines. deploy.bao.extraListenAddresses = [ mesh.swarm-central.address ]; - # The swarm UI is public. + # The gateway's ports are open to everyone. The swarm UI is the one vhost + # that answers there; the store's browser UI answers on the private + # addresses only. gateway.openFirewall = true; + gateway.vhosts = lib.genAttrs [ hyperhive.swarm.bao.ui.domain ] (_: { + public = false; + }); }; # The store is reachable over the VPN only. diff --git a/docs/examples/semi-spread/swarm-heavyduty.nix b/docs/examples/semi-spread/swarm-heavyduty.nix index 31e2a8f50..07170dc25 100644 --- a/docs/examples/semi-spread/swarm-heavyduty.nix +++ b/docs/examples/semi-spread/swarm-heavyduty.nix @@ -17,6 +17,11 @@ in deploy.ci-runner.enable = true; deploy.attic.enable = true; + # The cache answers on the private addresses only. + gateway.vhosts = lib.genAttrs [ hyperhive.swarm.attic.ownDomain ] (_: { + public = false; + }); + deploy.bao = lib.mergeAttrsList [ (pair "builder" "builder") (pair "ciRunner" "ci-runner") diff --git a/docs/examples/semi-spread/swarm-services.nix b/docs/examples/semi-spread/swarm-services.nix index e30dc0878..c05497ed8 100644 --- a/docs/examples/semi-spread/swarm-services.nix +++ b/docs/examples/semi-spread/swarm-services.nix @@ -19,9 +19,22 @@ in # Everything once-per-swarm that is not placed elsewhere below. deploy.allSwarmServices = true; - # The forge and the matrix homeserver are public, and so is the SSO login - # they send browsers to. + # The gateway's ports are open to everyone. The forge and the matrix + # homeserver answer there, and so does the SSO login they send browsers + # to. Every other vhost here answers on the private addresses only. gateway.openFirewall = true; + gateway.vhosts = + lib.genAttrs + (with hyperhive.swarm; [ + nats.domain + otel.domain + grafana.domain + victoriametrics.domain + victorialogs.domain + ]) + (_: { + public = false; + }); deploy.bao = lib.mergeAttrsList [ { enable = false; } diff --git a/docs/examples/semi-spread/wireguard.nix b/docs/examples/semi-spread/wireguard.nix index c7920a068..1d440c376 100644 --- a/docs/examples/semi-spread/wireguard.nix +++ b/docs/examples/semi-spread/wireguard.nix @@ -1,6 +1,6 @@ # The VPN: plain NixOS WireGuard, one interface `wg0`, a full mesh built from -# ./mesh.nix. Identical on every host; the host's own entry is picked by -# `networking.hostName`. +# ./mesh.nix, carrying IPv4 and IPv6. Identical on every host; the host's own +# entry is picked by `networking.hostName`. { config, lib, ... }: let mesh = import ./mesh.nix; @@ -8,13 +8,19 @@ let in { networking.wireguard.interfaces.wg0 = { - ips = [ "${mesh.${self}.address}/24" ]; + ips = [ + "${mesh.${self}.address}/24" + "${mesh.${self}.address6}/64" + ]; listenPort = 51820; # Created on the host by `wg genkey`, never in the store. privateKeyFile = "/etc/wireguard/wg0.key"; peers = lib.mapAttrsToList (_: peer: { inherit (peer) publicKey endpoint; - allowedIPs = [ "${peer.address}/32" ]; + allowedIPs = [ + "${peer.address}/32" + "${peer.address6}/128" + ]; persistentKeepalive = 25; }) (removeAttrs mesh [ self ]); }; diff --git a/nix/module-eval/semi-spread.nix b/nix/module-eval/semi-spread.nix index 0dcd3d9e3..040961b03 100644 --- a/nix/module-eval/semi-spread.nix +++ b/nix/module-eval/semi-spread.nix @@ -95,6 +95,72 @@ let vhosts = m: builtins.attrNames m.services.nginx.virtualHosts; + # The `listen` entries of one vhost, as `addr:port`. + listens = + m: vhost: map (l: "${l.addr}:${toString l.port}") m.services.nginx.virtualHosts.${vhost}.listen; + + # Both ports on each address. + bothPorts = + addrs: + lib.concatMap (a: [ + "${a}:80" + "${a}:443" + ]) addrs; + + # The private addresses of a machine as nginx spells them: its VPN IPv4 and + # bracketed IPv6 address, and the hive bridge. + privateAddrs = name: m: [ + mesh.${name}.address + "[${mesh.${name}.address6}]" + m.services.hyperhive.network.bridgeIp + ]; + + # The vhosts that listen on the wildcard, and those that listen on private + # addresses only. The catch-all `_` is neither: it is not a service. + publicVhosts = + m: + lib.sort (a: b: a < b) ( + lib.filter (n: n != "_" && lib.elem "0.0.0.0:443" (listens m n)) (vhosts m) + ); + privateVhosts = + m: + lib.sort (a: b: a < b) ( + lib.filter (n: n != "_" && !lib.elem "0.0.0.0:443" (listens m n)) (vhosts m) + ); + sorted = lib.sort (a: b: a < b); + + # What each machine is meant to serve to everyone, and what only inside. + expectedPublic = { + swarm-central = [ (swarmOf central).domain ]; + swarm-services = [ + (swarmOf services).authelia.domain + (swarmOf services).forge.domain + (swarmOf services).matrix.gatewayHost + ]; + swarm-heavyduty = [ ]; + hive-one = [ ]; + hive-two = [ ]; + }; + expectedPrivate = { + swarm-central = [ (swarmOf central).bao.ui.domain ]; + swarm-services = with swarmOf services; [ + grafana.domain + nats.domain + otel.domain + victorialogs.domain + victoriametrics.domain + ]; + swarm-heavyduty = [ (swarmOf heavyduty).attic.ownDomain ]; + hive-one = [ + (swarmOf hiveOne).dashboardDomains.one + (swarmOf hiveOne).hives.one.domain + ]; + hive-two = [ + (swarmOf hiveTwo).dashboardDomains.two + (swarmOf hiveTwo).hives.two.domain + ]; + }; + cases = [ # Every machine evaluates with its assertions holding. `tryEval` # turns a refusal into a failed case that names the machine. @@ -340,6 +406,100 @@ let # Control: the file is not one every machine shares. && !lib.hasInfix (swarmOf central).bao.domain (resolverTable central); } + + # Which vhosts answer where. Public means the wildcard, IPv4 and IPv6; + # private means the machine's VPN addresses and its bridge, and nothing + # else. + { + name = "only the swarm UI, SSO login, forge and matrix answer on the public address"; + ok = lib.all (n: publicVhosts all.${n} == sorted expectedPublic.${n}) (builtins.attrNames all); + } + { + # Presence control for the case above: the same read finds every vhost + # of the private set, so an empty public list is not a misread. + name = "every other vhost answers on the private addresses only"; + ok = lib.all ( + n: + let + m = all.${n}; + private = privateAddrs n m; + in + privateVhosts m == sorted expectedPrivate.${n} + && lib.all (v: sorted (listens m v) == sorted (bothPorts private)) expectedPrivate.${n} + ) (builtins.attrNames all); + } + { + name = "public vhosts listen on 0.0.0.0 and [::] and also on the private addresses"; + ok = + lib.all ( + n: + let + m = all.${n}; + want = bothPorts ( + [ + "0.0.0.0" + "[::]" + ] + ++ privateAddrs n m + ); + in + lib.all (v: sorted (listens m v) == sorted want) expectedPublic.${n} + ) (builtins.attrNames all) + # Control: the lists above are not empty. + && expectedPublic.swarm-services != [ ]; + } + { + name = "the catch-all listens on the private addresses too, and the wildcard"; + ok = lib.all ( + n: + let + m = all.${n}; + in + sorted (listens m "_") == sorted ( + bothPorts ( + [ + "0.0.0.0" + "[::]" + ] + ++ privateAddrs n m + ) + ) + ) (builtins.attrNames all); + } + { + name = "the private addresses hold the VPN's IPv4 and IPv6 address of the machine, and the bridge"; + ok = lib.all ( + n: + all.${n}.services.hyperhive.gateway.privateListenAddresses == [ + mesh.${n}.address + mesh.${n}.address6 + all.${n}.services.hyperhive.network.bridgeIp + ] + ) (builtins.attrNames all); + } + { + name = "the VPN carries IPv6: each machine has its address and a /128 per peer"; + ok = lib.all ( + n: + let + wg = all.${n}.networking.wireguard.interfaces.wg0; + in + lib.elem "${mesh.${n}.address6}/64" wg.ips + && lib.all (p: lib.any (a: lib.hasSuffix "/128" a) p.allowedIPs) wg.peers + ) (builtins.attrNames all); + } + { + name = "a hive resolves the internal names to the IPv6 addresses as well"; + ok = + let + t = hostsOf hiveOne; + in + lib.elem (swarmOf hiveOne).bao.domain (t.${mesh.swarm-central.address6} or [ ]) + && lib.elem (swarmOf hiveOne).grafana.domain (t.${mesh.swarm-services.address6} or [ ]) + && lib.hasInfix "${mesh.swarm-heavyduty.address6} ${(swarmOf hiveOne).attic.ownDomain}" ( + resolverTable hiveOne + ); + } ]; in runGroup "semi-spread" cases