mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

semi-spread example: serve internal vhosts on the VPN addresses only, over IPv4 and IPv6

Browse source
This commit is contained in:
atlas 2026-11-10 00:07:38 +01:00 • committed by mara
commit 7800f92425
13 changed files with 356 additions and 89 deletions
Download patch file Download diff file Expand all files Collapse all files

135
docs/2-handbook/swarm/semi-spread-deploy.md
View file

@ -1,10 +1,11 @@
# Example: a semi-spread deploy
Three swarm machines and two hives, joined by a WireGuard VPN, with only the
forge, the matrix homeserver and the swarm UI reachable from the internet. It
is one of the in-between shapes that [`services.md`](services.md#deployment-shapes)
supports: the swarm's services run on three machines, not on one box and not
one per service.
Three swarm machines and two hives, joined by a WireGuard VPN over IPv4 and
IPv6. Four things answer to the internet: the swarm UI, the forge, the matrix
homeserver and the SSO login they send browsers to. Every other vhost answers
on the VPN only. It's one of the in-between shapes that
[`services.md`](services.md#deployment-shapes) supports: the swarm's services
run on three machines, not on one box and not one per service.
The NixOS files are in
[`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all
@ -13,20 +14,20 @@ page, so the files stay in step with the options.
## The machines
| machine | runs | reachable from the internet |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI |
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | forge, matrix, SSO login |
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing |
| `hive-one` | the hive `one` | nothing |
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing |
| machine | runs | answers the internet with | answers the VPN only with |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------- |
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI | the store's browser UI; the store itself (port 8200) |
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | the forge, matrix and the SSO login | Grafana, metrics, logs, the collector, the queue's vhost; the queue (4222) |
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing | the cache |
| `hive-one` | the hive `one` | nothing | the hive's dashboard and domain |
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing | the hive's dashboard and domain |
Why this split:
- **`swarm-central`** holds what every other machine depends on, and little
else that can fail. The secret store and the controller are small.
- **`swarm-services`** holds the services people and agents talk to all day.
They're light, and they need the public name and port.
They're light.
- **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder
evaluates config pull requests nobody reviewed yet, and the CI runner runs
pull-request code. Keeping both off the machine that holds the secret store
@ -40,22 +41,12 @@ hive name from the machine's host name, which is why one file serves both.
## What's public and what's not
`gateway.openFirewall` opens the gateway's ports (80 and 443) on every
interface, and the example sets it on two machines only:
Two things decide it: which ports the firewall opens, and which addresses
nginx answers on.
- `swarm-central` serves the swarm UI at `<swarm.domain>`.
- `swarm-services` serves the forge (`forge.<swarm.domain>`) and the matrix
homeserver (`chat.<swarm.domain>`). It also serves `auth.<swarm.domain>`,
because the forge, matrix and the swarm UI send browsers there to sign in.
**Public means the machine's gateway port is open.** A gateway is one nginx
for everything the machine serves, so the other names on those two machines
(Grafana, metrics, logs, the collector, the store's browser UI) answer on the
public address too. Each sits behind SSO or a bearer token, and none of them is
in public DNS, but they're not closed. A service you want closed goes on a
machine that opens nothing public, like `swarm-heavyduty`.
Everything else uses the VPN. These ports open on `wg0` only, with
**Ports.** `gateway.openFirewall` opens the gateway's ports (80 and 443) on
every interface, and the example sets it on `swarm-central` and
`swarm-services` only. Everything else opens on `wg0` alone, with
`networking.firewall.interfaces.wg0.allowedTCPPorts`:
| machine | port on `wg0` | for |
@ -67,39 +58,73 @@ Everything else uses the VPN. These ports open on `wg0` only, with
The example scopes these to `wg0` because the options that open a service's
port, `gateway.openFirewall`, `deploy.bao.openFirewall` and
`deploy.nats.openFirewall`, open it on every interface. The
secret store and the queue don't rely on the VPN for their security: the store
requires a client certificate, and the queue requires TLS and refuses clients
it can't identify. The VPN keeps them off the internet as well.
`deploy.nats.openFirewall`, open it on every interface. The secret store and
the queue don't rely on the VPN for their security: the store requires a client
certificate, and the queue requires TLS and refuses clients it can't identify.
The VPN keeps them off the internet as well.
**Addresses.** The gateway is one nginx per machine, and by default every vhost
answers on every address. Each machine here sets
`gateway.privateListenAddresses` (`gateway.nix`) to its own VPN IPv4 and IPv6
addresses and the hive bridge, and sets `public = false` on the vhosts that
stay inside:
| machine | public vhosts | `public = false` |
| ----------------- | -------------------------------------------------- | ------------------------------------------------------------------ |
| `swarm-central` | `<swarm.domain>` (the swarm UI) | `bao-ui.<swarm.domain>` |
| `swarm-services` | `forge.`, `chat.` (matrix) and `auth.` (SSO login) | `grafana.`, `metrics.`, `logs.`, `otel.` and `nats.<swarm.domain>` |
| `swarm-heavyduty` | none | `attic.<swarm.domain>` (the cache) |
| each hive | none | `<name>.hives.<swarm.domain>` and the hive's own domain |
A public vhost listens on `0.0.0.0`, on `[::]`, and on the private addresses,
so a machine inside the VPN still reaches it by its VPN address. A private
vhost listens on the private addresses only. A request for a private name on
the public address gets nothing back, the same as any name the gateway doesn't
know. The hive bridge is in the list because the machine's own containers
reach a name it serves through the bridge address, so a private vhost the
containers use still answers them.
**SSO stays public.** `auth.<swarm.domain>` answers the internet because the
forge, matrix and the swarm UI redirect a browser there to sign in, and a
browser outside the VPN has to follow. Authelia authenticates; it needs no VPN
as a second lock. If every operator is on the VPN anyway, set `public = false`
on it too, and the sign-in then works from inside the VPN only.
Every machine also opens UDP 51820 on every interface, because that's where
the VPN's handshake arrives. The example leaves SSH to you.
```text
internet
80/443 | | 80/443 UDP 51820 on all five
| |
internet UDP 51820 reaches all five
80/443 | | 80/443
(swarm UI) (forge, matrix, SSO)
swarm-central swarm-services swarm-heavyduty hive-one hive-two
10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12
fd77:77a1:5e::1 fd77:77a1:5e::2 fd77:77a1:5e::3 …::11 …::12
| | | | |
+----------------+----- wg0, 10.77.0.0/24 --------+----------+
+------- wg0: 10.77.0.0/24 and fd77:77a1:5e::/64 --+----------+
```
## The VPN: raw WireGuard
`wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on
all five machines. `mesh.nix` holds each machine's VPN address, public endpoint
and public key. The file picks the machine's own entry by `networking.hostName`
all five machines. `mesh.nix` holds each machine's VPN addresses (IPv4 and IPv6),
public endpoint and public key. The file picks the machine's own entry by `networking.hostName`
and makes the other four its peers:
```nix
networking.wireguard.interfaces.wg0 = {
ips = [ "${mesh.${self}.address}/24" ];
ips = [
"${mesh.${self}.address}/24"
"${mesh.${self}.address6}/64"
];
listenPort = 51820;
privateKeyFile = "/etc/wireguard/wg0.key";
peers = lib.mapAttrsToList (_: peer: {
inherit (peer) publicKey endpoint;
allowedIPs = [ "${peer.address}/32" ];
allowedIPs = [
"${peer.address}/32"
"${peer.address6}/128"
];
persistentKeepalive = 25;
}) (removeAttrs mesh [ self ]);
};
@ -118,10 +143,12 @@ To set it up:
2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public
keys aren't secret, so the file is safe to commit. You do this once.
3. Set the `address` and `endpoint` of each machine in `mesh.nix`. The example
uses documentation addresses.
3. Set the `address`, `address6` and `endpoint` of each machine in `mesh.nix`.
The example uses documentation addresses for the endpoints and a
unique-local IPv6 prefix (`fd77:77a1:5e::/64`). Generate your own prefix
(RFC 4193) rather than keeping it.
4. Deploy. `wg show wg0` on any machine lists four peers with a recent
handshake, and `ping 10.77.0.1` answers.
handshake, and `ping 10.77.0.1` and `ping fd77:77a1:5e::1` answer.
A machine that can't take incoming connections, such as one behind NAT, sets
`endpoint = null`. It dials the others, and the keepalive holds the NAT mapping
@ -133,8 +160,10 @@ addresses for you, so the key steps above go away. Three things change:
- The interface is no longer `wg0`. Tailscale's is `tailscale0`
(`services.tailscale.interfaceName`). Put its name in the firewall rules in
place of `wg0`.
- Drop `wireguard.nix`, and put each machine's address from your VPN into the
`address` field of `mesh.nix`. The DNS table below reads only that field.
- Drop `wireguard.nix`, and put each machine's addresses from your VPN into the
`address` and `address6` fields of `mesh.nix`. `gateway.nix` and the DNS table
below read only those fields. The addresses must exist on the machine before
nginx starts, which a VPN service that comes up late may not guarantee.
- Enable the service, for example `services.tailscale.enable = true` with
`services.tailscale.authKeyFile` pointing at a key file outside the store.
@ -157,7 +186,8 @@ Every service reaches another by name, so every name has to resolve on every
machine and inside every container. Two views of it exist.
**The public view** is a handful of records at your DNS provider, pointing at
the two public machines:
the two public machines. Give each name an `A` record, and an `AAAA` record if the
machine has a public IPv6 address (the gateway listens on `[::]`):
| name | address |
| -------------------------------------- | --------------------------------- |
@ -166,14 +196,15 @@ the two public machines:
| `chat.<swarm.domain>` (the homeserver) | `swarm-services`'s public address |
| `auth.<swarm.domain>` | `swarm-services`'s public address |
Matrix federation and browsers use these. Names you leave out of public DNS
(`bao.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
Matrix federation and browsers use these. The names you leave out of public DNS
(`bao.`, `bao-ui.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
hives' dashboards) don't resolve from outside.
**The inside view** maps every swarm name to the VPN address of the machine
that serves it, including the public names, so traffic between machines
never uses the internet. The example does it with a static table in `dns.nix`,
written on every machine and built from the same options the services use, so
**The inside view** maps every swarm name to the VPN addresses of the machine
that serves it, IPv4 and IPv6 (`A` and `AAAA`), including the public names, so
traffic between machines never uses the internet. The example does it with a
static table in `dns.nix`, written on every machine and built from the same
options the services use, so
a renamed `swarm.*.domain` follows. Two readers need it:
- Processes on the machine read `/etc/hosts` (`networking.hosts`).

3
docs/2-handbook/swarm/services.md
View file

@ -81,7 +81,8 @@ the shapes worth naming are the ones they produce:
- **Fully spread out.** One container / VM / machine per service,
somewhere.
- **Semi-spread.** The services split over a few machines by weight, hives on
their own, and a VPN between them. A worked example with its NixOS files:
their own, a VPN between them, and most vhosts answering on the VPN only.
A worked example with its NixOS files:
[`semi-spread-deploy.md`](semi-spread-deploy.md).
**These are a set, not a ladder with a correct top, and hyperhive

4
docs/README.md
View file

@ -32,8 +32,8 @@ The services that exist once per swarm. Start at
- **How are two hives connected into a swarm?** → [`2-handbook/swarm/`](2-handbook/swarm/README.md)
(peer hives, TLS trust).
- **How to spread the swarm over a few machines, with a VPN and only the forge,
matrix and swarm UI public?** → [`2-handbook/swarm/semi-spread-deploy.md`](2-handbook/swarm/semi-spread-deploy.md)
- **How to spread the swarm over a few machines, with a VPN, and only the swarm UI, forge,
matrix and SSO login answering the internet?** → [`2-handbook/swarm/semi-spread-deploy.md`](2-handbook/swarm/semi-spread-deploy.md)
(worked example: machines, WireGuard, DNS).
- **What does the swarm UI show, and what does `swarmctl` do?** →
[`2-handbook/swarm/ui.md`](2-handbook/swarm/ui.md) and

1
docs/examples/semi-spread/common.nix
View file

@ -4,6 +4,7 @@
{
imports = [
./wireguard.nix
./gateway.nix
./dns.nix
];

25
docs/examples/semi-spread/dns.nix
View file

@ -1,6 +1,6 @@
# Names inside the VPN. Every swarm name that is not public resolves to the
# VPN address of the machine serving it, through a static table written on
# each host. Identical on every host.
# Names inside the VPN. Every swarm name, the public ones included, resolves to
# the VPN addresses (IPv4 and IPv6) of the machine serving it, through a static
# table written on each host. Identical on every host.
#
# Two readers need the table. Processes on the host read /etc/hosts
# (`networking.hosts`). Containers (agents, swarm services) ask the hive's
@ -49,17 +49,20 @@ let
};
elsewhere = removeAttrs servedBy [ self ];
# Each machine's names, once per address family: A and AAAA answers.
addressesOf = machine: [
mesh.${machine}.address
mesh.${machine}.address6
];
table = lib.concatMap (
machine: map (address: lib.nameValuePair address elsewhere.${machine}) (addressesOf machine)
) (builtins.attrNames elsewhere);
in
{
networking.hosts = lib.mapAttrs' (
machine: names: lib.nameValuePair mesh.${machine}.address names
) elsewhere;
networking.hosts = lib.listToAttrs table;
services.dnsmasq.settings.addn-hosts = "${builtins.toFile "swarm-vpn-hosts" (
lib.concatStringsSep "\n" (
lib.mapAttrsToList (
machine: names: "${mesh.${machine}.address} ${lib.concatStringsSep " " names}"
) elsewhere
)
lib.concatMapStrings (entry: "${entry.name} ${lib.concatStringsSep " " entry.value}\n") table
)}";
}

20
docs/examples/semi-spread/gateway.nix Normal file
View file

@ -0,0 +1,20 @@
# Which addresses the gateway's private vhosts answer on: this machine's VPN
# addresses (IPv4 and IPv6) and the hive bridge. Identical on every host.
# Each machine's own file lists its private vhosts (`public = false`); a vhost
# that stays `public` also answers on these addresses.
#
# The bridge is how this machine's own containers (agents, swarm services)
# reach a name it serves: the hive's resolver answers it with the bridge
# address.
{ config, ... }:
let
mesh = import ./mesh.nix;
me = mesh.${config.networking.hostName};
in
{
services.hyperhive.gateway.privateListenAddresses = [
me.address
me.address6
config.services.hyperhive.network.bridgeIp
];
}

36
docs/examples/semi-spread/hive.nix
View file

@ -2,6 +2,9 @@
# hive `<name>`.
{ config, lib, ... }:
let
hyperhive = config.services.hyperhive;
name = lib.removePrefix "hive-" config.networking.hostName;
# The leaves for this hive, copied from swarm-central, where the secret store
# runs: `client.pem` and `client-key.pem` from
# `/var/lib/swarm-bao-pki/hives/<name>/`, along with that directory's
@ -10,19 +13,30 @@ let
pki = "/var/lib/swarm-pki";
in
{
services.hyperhive.deploy = {
hive-controller = {
enable = true;
hiveName = lib.removePrefix "hive-" config.networking.hostName;
};
bao = {
clientCertFile = "${pki}/client.pem";
clientKeyFile = "${pki}/client-key.pem";
services.hyperhive = {
deploy = {
hive-controller = {
enable = true;
hiveName = name;
};
bao = {
clientCertFile = "${pki}/client.pem";
clientKeyFile = "${pki}/client-key.pem";
};
};
# The hive's dashboard and its domain answer on the private addresses only.
gateway.vhosts =
lib.genAttrs
[
hyperhive.swarm.dashboardDomains.${name}
hyperhive.swarm.hives.${name}.domain
]
(_: {
public = false;
});
};
# The dashboard is reachable over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [
config.services.hyperhive.gateway.httpsPort
];
networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.gateway.httpsPort ];
}

16
docs/examples/semi-spread/mesh.nix
View file

@ -1,32 +1,40 @@
# The VPN's facts: one entry per machine, read by every host (wireguard.nix,
# dns.nix) and nothing else. Keyed by `networking.hostName`.
# gateway.nix, dns.nix) and nothing else. Keyed by `networking.hostName`.
#
# `address` is the machine's address inside the VPN; `endpoint` is where the
# other machines dial it; `publicKey` is the output of `wg pubkey` on that
# machine. The keys below are placeholders: replace each with the real one.
# `address` and `address6` are the machine's addresses inside the VPN (IPv4
# 10.77.0.0/24 and the unique-local IPv6 prefix fd77:77a1:5e::/64);
# `endpoint` is where the other machines dial it; `publicKey` is the output of
# `wg pubkey` on that machine. The keys below are placeholders: replace each
# with the real one. Generate your own IPv6 prefix (RFC 4193) rather than
# keeping this one.
{
swarm-central = {
address = "10.77.0.1";
address6 = "fd77:77a1:5e::1";
endpoint = "203.0.113.10:51820";
publicKey = "REPLACE-with-swarm-central-wg-pubkey";
};
swarm-services = {
address = "10.77.0.2";
address6 = "fd77:77a1:5e::2";
endpoint = "203.0.113.11:51820";
publicKey = "REPLACE-with-swarm-services-wg-pubkey";
};
swarm-heavyduty = {
address = "10.77.0.3";
address6 = "fd77:77a1:5e::3";
endpoint = "203.0.113.12:51820";
publicKey = "REPLACE-with-swarm-heavyduty-wg-pubkey";
};
hive-one = {
address = "10.77.0.11";
address6 = "fd77:77a1:5e::11";
endpoint = "198.51.100.11:51820";
publicKey = "REPLACE-with-hive-one-wg-pubkey";
};
hive-two = {
address = "10.77.0.12";
address6 = "fd77:77a1:5e::12";
endpoint = "198.51.100.12:51820";
publicKey = "REPLACE-with-hive-two-wg-pubkey";
};

9
docs/examples/semi-spread/swarm-central.nix
View file

@ -1,6 +1,6 @@
# swarm-central: the swarm controller (and the swarm UI it serves) and the
# secret store.
{ config, ... }:
{ config, lib, ... }:
let
mesh = import ./mesh.nix;
hyperhive = config.services.hyperhive;
@ -14,8 +14,13 @@ in
# and hives on other machines.
deploy.bao.extraListenAddresses = [ mesh.swarm-central.address ];
# The swarm UI is public.
# The gateway's ports are open to everyone. The swarm UI is the one vhost
# that answers there; the store's browser UI answers on the private
# addresses only.
gateway.openFirewall = true;
gateway.vhosts = lib.genAttrs [ hyperhive.swarm.bao.ui.domain ] (_: {
public = false;
});
};
# The store is reachable over the VPN only.

5
docs/examples/semi-spread/swarm-heavyduty.nix
View file

@ -17,6 +17,11 @@ in
deploy.ci-runner.enable = true;
deploy.attic.enable = true;
# The cache answers on the private addresses only.
gateway.vhosts = lib.genAttrs [ hyperhive.swarm.attic.ownDomain ] (_: {
public = false;
});
deploy.bao = lib.mergeAttrsList [
(pair "builder" "builder")
(pair "ciRunner" "ci-runner")

17
docs/examples/semi-spread/swarm-services.nix
View file

@ -19,9 +19,22 @@ in
# Everything once-per-swarm that is not placed elsewhere below.
deploy.allSwarmServices = true;
# The forge and the matrix homeserver are public, and so is the SSO login
# they send browsers to.
# The gateway's ports are open to everyone. The forge and the matrix
# homeserver answer there, and so does the SSO login they send browsers
# to. Every other vhost here answers on the private addresses only.
gateway.openFirewall = true;
gateway.vhosts =
lib.genAttrs
(with hyperhive.swarm; [
nats.domain
otel.domain
grafana.domain
victoriametrics.domain
victorialogs.domain
])
(_: {
public = false;
});
deploy.bao = lib.mergeAttrsList [
{ enable = false; }

14
docs/examples/semi-spread/wireguard.nix
View file

@ -1,6 +1,6 @@
# The VPN: plain NixOS WireGuard, one interface `wg0`, a full mesh built from
# ./mesh.nix. Identical on every host; the host's own entry is picked by
# `networking.hostName`.
# ./mesh.nix, carrying IPv4 and IPv6. Identical on every host; the host's own
# entry is picked by `networking.hostName`.
{ config, lib, ... }:
let
mesh = import ./mesh.nix;
@ -8,13 +8,19 @@ let
in
{
networking.wireguard.interfaces.wg0 = {
ips = [ "${mesh.${self}.address}/24" ];
ips = [
"${mesh.${self}.address}/24"
"${mesh.${self}.address6}/64"
];
listenPort = 51820;
# Created on the host by `wg genkey`, never in the store.
privateKeyFile = "/etc/wireguard/wg0.key";
peers = lib.mapAttrsToList (_: peer: {
inherit (peer) publicKey endpoint;
allowedIPs = [ "${peer.address}/32" ];
allowedIPs = [
"${peer.address}/32"
"${peer.address6}/128"
];
persistentKeepalive = 25;
}) (removeAttrs mesh [ self ]);
};

160
nix/module-eval/semi-spread.nix
View file

@ -95,6 +95,72 @@ let
vhosts = m: builtins.attrNames m.services.nginx.virtualHosts;
# The `listen` entries of one vhost, as `addr:port`.
listens =
m: vhost: map (l: "${l.addr}:${toString l.port}") m.services.nginx.virtualHosts.${vhost}.listen;
# Both ports on each address.
bothPorts =
addrs:
lib.concatMap (a: [
"${a}:80"
"${a}:443"
]) addrs;
# The private addresses of a machine as nginx spells them: its VPN IPv4 and
# bracketed IPv6 address, and the hive bridge.
privateAddrs = name: m: [
mesh.${name}.address
"[${mesh.${name}.address6}]"
m.services.hyperhive.network.bridgeIp
];
# The vhosts that listen on the wildcard, and those that listen on private
# addresses only. The catch-all `_` is neither: it is not a service.
publicVhosts =
m:
lib.sort (a: b: a < b) (
lib.filter (n: n != "_" && lib.elem "0.0.0.0:443" (listens m n)) (vhosts m)
);
privateVhosts =
m:
lib.sort (a: b: a < b) (
lib.filter (n: n != "_" && !lib.elem "0.0.0.0:443" (listens m n)) (vhosts m)
);
sorted = lib.sort (a: b: a < b);
# What each machine is meant to serve to everyone, and what only inside.
expectedPublic = {
swarm-central = [ (swarmOf central).domain ];
swarm-services = [
(swarmOf services).authelia.domain
(swarmOf services).forge.domain
(swarmOf services).matrix.gatewayHost
];
swarm-heavyduty = [ ];
hive-one = [ ];
hive-two = [ ];
};
expectedPrivate = {
swarm-central = [ (swarmOf central).bao.ui.domain ];
swarm-services = with swarmOf services; [
grafana.domain
nats.domain
otel.domain
victorialogs.domain
victoriametrics.domain
];
swarm-heavyduty = [ (swarmOf heavyduty).attic.ownDomain ];
hive-one = [
(swarmOf hiveOne).dashboardDomains.one
(swarmOf hiveOne).hives.one.domain
];
hive-two = [
(swarmOf hiveTwo).dashboardDomains.two
(swarmOf hiveTwo).hives.two.domain
];
};
cases = [
# Every machine evaluates with its assertions holding. `tryEval`
# turns a refusal into a failed case that names the machine.
@ -340,6 +406,100 @@ let
# Control: the file is not one every machine shares.
&& !lib.hasInfix (swarmOf central).bao.domain (resolverTable central);
}
# Which vhosts answer where. Public means the wildcard, IPv4 and IPv6;
# private means the machine's VPN addresses and its bridge, and nothing
# else.
{
name = "only the swarm UI, SSO login, forge and matrix answer on the public address";
ok = lib.all (n: publicVhosts all.${n} == sorted expectedPublic.${n}) (builtins.attrNames all);
}
{
# Presence control for the case above: the same read finds every vhost
# of the private set, so an empty public list is not a misread.
name = "every other vhost answers on the private addresses only";
ok = lib.all (
n:
let
m = all.${n};
private = privateAddrs n m;
in
privateVhosts m == sorted expectedPrivate.${n}
&& lib.all (v: sorted (listens m v) == sorted (bothPorts private)) expectedPrivate.${n}
) (builtins.attrNames all);
}
{
name = "public vhosts listen on 0.0.0.0 and [::] and also on the private addresses";
ok =
lib.all (
n:
let
m = all.${n};
want = bothPorts (
[
"0.0.0.0"
"[::]"
]
++ privateAddrs n m
);
in
lib.all (v: sorted (listens m v) == sorted want) expectedPublic.${n}
) (builtins.attrNames all)
# Control: the lists above are not empty.
&& expectedPublic.swarm-services != [ ];
}
{
name = "the catch-all listens on the private addresses too, and the wildcard";
ok = lib.all (
n:
let
m = all.${n};
in
sorted (listens m "_") == sorted (
bothPorts (
[
"0.0.0.0"
"[::]"
]
++ privateAddrs n m
)
)
) (builtins.attrNames all);
}
{
name = "the private addresses hold the VPN's IPv4 and IPv6 address of the machine, and the bridge";
ok = lib.all (
n:
all.${n}.services.hyperhive.gateway.privateListenAddresses == [
mesh.${n}.address
mesh.${n}.address6
all.${n}.services.hyperhive.network.bridgeIp
]
) (builtins.attrNames all);
}
{
name = "the VPN carries IPv6: each machine has its address and a /128 per peer";
ok = lib.all (
n:
let
wg = all.${n}.networking.wireguard.interfaces.wg0;
in
lib.elem "${mesh.${n}.address6}/64" wg.ips
&& lib.all (p: lib.any (a: lib.hasSuffix "/128" a) p.allowedIPs) wg.peers
) (builtins.attrNames all);
}
{
name = "a hive resolves the internal names to the IPv6 addresses as well";
ok =
let
t = hostsOf hiveOne;
in
lib.elem (swarmOf hiveOne).bao.domain (t.${mesh.swarm-central.address6} or [ ])
&& lib.elem (swarmOf hiveOne).grafana.domain (t.${mesh.swarm-services.address6} or [ ])
&& lib.hasInfix "${mesh.swarm-heavyduty.address6} ${(swarmOf hiveOne).attic.ownDomain}" (
resolverTable hiveOne
);
}
];
in
runGroup "semi-spread" cases