mara/hyperhive
mara/hyperhive
Watch
0
0
Fork
You've already forked hyperhive
 0
Code Activity

add a semi-spread example deploy and a check that evaluates it

Browse source
This commit is contained in:
atlas 2026-11-09 18:05:27 +01:00 • committed by mara
commit 538da5a13a
12 changed files with 875 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/1-getting-started/nixos.md
View file

@ -107,6 +107,7 @@ Leave the other options at their defaults until you need them:
- Spreading a swarm over more than one host, and the credentials a split swarm
needs: [`2-handbook/swarm/README.md`](../2-handbook/swarm/README.md) and
[`2-handbook/swarm/credentials.md`](../2-handbook/swarm/credentials.md).
A worked example: [`2-handbook/swarm/semi-spread-deploy.md`](../2-handbook/swarm/semi-spread-deploy.md).
- Every option, with its type and default:
[the options reference](https://hyperhive.darkest.space/options/).
- The secret store's seal and the granter: [`4-implementation/swarm/bao.md`](../4-implementation/swarm/bao.md).

224
docs/2-handbook/swarm/semi-spread-deploy.md Normal file
View file

@ -0,0 +1,224 @@
# Example: a semi-spread deploy
Three swarm machines and two hives, joined by a WireGuard VPN, with only the
forge, the matrix homeserver and the swarm UI reachable from the internet. It
is one of the in-between shapes that [`services.md`](services.md#deployment-shapes)
supports: the swarm's services run on three machines, not on one box and not
one per service.
The NixOS files are in
[`docs/examples/semi-spread/`](../../examples/semi-spread/). CI evaluates all
five machines (`checks.module-eval-semi-spread`) and pins the facts on this
page, so the files stay in step with the options.
## The machines
| machine | runs | reachable from the internet |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `swarm-central` | `swarm-controller` and the swarm UI, `swarm-bao` | the swarm UI |
| `swarm-services` | the forge, matrix, SSO (authelia), the queue, the telemetry collector, metrics, logs and Grafana, the review classifier, notify router, claude login and CI metrics | forge, matrix, SSO login |
| `swarm-heavyduty` | `swarm-builder`, the forge's CI runner (`swarm-ci-runner`), `swarm-attic` | nothing |
| `hive-one` | the hive `one` | nothing |
| `hive-two` | the hive `two`, with the same file as `hive-one` | nothing |
Why this split:
- **`swarm-central`** holds what every other machine depends on, and little
else that can fail. The secret store and the controller are small.
- **`swarm-services`** holds the services people and agents talk to all day.
They're light, and they need the public name and port.
- **`swarm-heavyduty`** holds the CPU, memory and disk hogs. The builder
evaluates config pull requests nobody reviewed yet, and the CI runner runs
pull-request code. Keeping both off the machine that holds the secret store
limits what a bad build can reach.
- **The hives** run the agents. They need nothing from the internet but the
agents' own egress.
A hive name can't contain `hive`, `agent` or `swarm`, so the machines
`hive-one` and `hive-two` run the hives `one` and `two`. `hive.nix` takes the
hive name from the machine's host name, which is why one file serves both.
## What's public and what's not
`gateway.openFirewall` opens the gateway's ports (80 and 443) on every
interface, and the example sets it on two machines only:
- `swarm-central` serves the swarm UI at `<swarm.domain>`.
- `swarm-services` serves the forge (`forge.<swarm.domain>`) and the matrix
homeserver (`chat.<swarm.domain>`). It also serves `auth.<swarm.domain>`,
because the forge, matrix and the swarm UI send browsers there to sign in.
**Public means the machine's gateway port is open.** A gateway is one nginx
for everything the machine serves, so the other names on those two machines
(Grafana, metrics, logs, the collector, the store's browser UI) answer on the
public address too. Each sits behind SSO or a bearer token, and none of them is
in public DNS, but they're not closed. A service you want closed goes on a
machine that opens nothing public, like `swarm-heavyduty`.
Everything else uses the VPN. These ports open on `wg0` only, with
`networking.firewall.interfaces.wg0.allowedTCPPorts`:
| machine | port on `wg0` | for |
| ----------------- | ------------------------- | --------------------------------------------- |
| `swarm-central` | `swarm.bao.port` (8200) | the secret store, read by every other machine |
| `swarm-services` | `swarm.nats.port` (4222) | the queue |
| `swarm-heavyduty` | `gateway.httpsPort` (443) | the cache, behind the gateway |
| each hive | `gateway.httpsPort` (443) | the hive's dashboard and gateway |
The example scopes these to `wg0` because the options that open a service's
port, `gateway.openFirewall`, `deploy.bao.openFirewall` and
`deploy.nats.openFirewall`, open it on every interface. The
secret store and the queue don't rely on the VPN for their security: the store
requires a client certificate, and the queue requires TLS and refuses clients
it can't identify. The VPN keeps them off the internet as well.
Every machine also opens UDP 51820 on every interface, because that's where
the VPN's handshake arrives. The example leaves SSH to you.
```text
internet
80/443 | | 80/443 UDP 51820 on all five
| |
swarm-central swarm-services swarm-heavyduty hive-one hive-two
10.77.0.1 10.77.0.2 10.77.0.3 10.77.0.11 10.77.0.12
| | | | |
+----------------+----- wg0, 10.77.0.0/24 --------+----------+
```
## The VPN: raw WireGuard
`wireguard.nix` is plain NixOS, one interface `wg0`, and it's the same file on
all five machines. `mesh.nix` holds each machine's VPN address, public endpoint
and public key. The file picks the machine's own entry by `networking.hostName`
and makes the other four its peers:
```nix
networking.wireguard.interfaces.wg0 = {
ips = [ "${mesh.${self}.address}/24" ];
listenPort = 51820;
privateKeyFile = "/etc/wireguard/wg0.key";
peers = lib.mapAttrsToList (_: peer: {
inherit (peer) publicKey endpoint;
allowedIPs = [ "${peer.address}/32" ];
persistentKeepalive = 25;
}) (removeAttrs mesh [ self ]);
};
```
To set it up:
1. On each machine, create the private key in a file only root can read. It
never goes in the Nix store or in the configuration:
```bash
install -d -m 0700 /etc/wireguard
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/wg0.key
wg pubkey < /etc/wireguard/wg0.key
```
2. Put each machine's public key (the `wg pubkey` output) in `mesh.nix`. Public
keys aren't secret, so the file is safe to commit. You do this once.
3. Set the `address` and `endpoint` of each machine in `mesh.nix`. The example
uses documentation addresses.
4. Deploy. `wg show wg0` on any machine lists four peers with a recent
handshake, and `ping 10.77.0.1` answers.
A machine that can't take incoming connections, such as one behind NAT, sets
`endpoint = null`. It dials the others, and the keepalive holds the NAT mapping
open.
**Or use tailscale, headscale or NetBird.** They hand out the keys and the
addresses for you, so the key steps above go away. Three things change:
- The interface is no longer `wg0`. Tailscale's is `tailscale0`
(`services.tailscale.interfaceName`). Put its name in the firewall rules in
place of `wg0`.
- Drop `wireguard.nix`, and put each machine's address from your VPN into the
`address` field of `mesh.nix`. The DNS table below reads only that field.
- Enable the service, for example `services.tailscale.enable = true` with
`services.tailscale.authKeyFile` pointing at a key file outside the store.
## Reading the store
Only `swarm-central` runs the secret store, and every service on the other
machines reads it with a client certificate that the store's host mints. The
example names each leaf under `/var/lib/swarm-pki/` on the machine that uses
it. After the first deploy of `swarm-central`, copy the files from
`/var/lib/swarm-bao-pki/` there over the VPN (for a hive, from
`/var/lib/swarm-bao-pki/hives/<name>/`), then deploy the other machines. The
private keys travel over a channel you trust and stay readable by root only.
[`credentials.md`](credentials.md) lists which leaf each service reads and what
a missing one costs. The first-run steps after the deploys are in
[`setup.md`](../../1-getting-started/setup.md).
## DNS
Every service reaches another by name, so every name has to resolve on every
machine and inside every container. Two views of it exist.
**The public view** is a handful of records at your DNS provider, pointing at
the two public machines:
| name | address |
| -------------------------------------- | --------------------------------- |
| `<swarm.domain>` | `swarm-central`'s public address |
| `forge.<swarm.domain>` | `swarm-services`'s public address |
| `chat.<swarm.domain>` (the homeserver) | `swarm-services`'s public address |
| `auth.<swarm.domain>` | `swarm-services`'s public address |
Matrix federation and browsers use these. Names you leave out of public DNS
(`bao.`, `nats.`, `attic.`, `otel.`, `metrics.`, `logs.`, `grafana.` and the
hives' dashboards) don't resolve from outside.
**The inside view** maps every swarm name to the VPN address of the machine
that serves it, including the public names, so traffic between machines
never uses the internet. The example does it with a static table in `dns.nix`,
written on every machine and built from the same options the services use, so
a renamed `swarm.*.domain` follows. Two readers need it:
- Processes on the machine read `/etc/hosts` (`networking.hosts`).
- Containers (agents and swarm services alike) ask the hive's resolver
(dnsmasq). It ignores `/etc/hosts`, so the example hands it the same table
through `services.dnsmasq.settings.addn-hosts`.
A machine leaves out the names it serves itself. hyperhive answers those
already, and a table entry would override its answer.
The trade-off: a static table needs no DNS server and no extra moving part, and
it lives in the same repository as the rest. It also means a new hive adds a
line on every machine, and a name that moves from one machine to another is a
rebuild everywhere. For a swarm with many hives, run a small DNS server on the
VPN instead, answering the swarm domain with VPN addresses, and point each
machine's upstream resolver at it for that domain. That's one place to edit,
at the cost of one more service every lookup depends on.
To open Grafana, the store's browser UI or a hive dashboard yourself, join the
VPN with your own machine: add it as a peer on each machine and give it the
same names, for example in its own `/etc/hosts`.
## Not in the example
- **TLS certificates the public can trust.** Without `gateway.tls.acme` or
`gateway.tls.certDir`, the gateway serves certificates from the swarm's own
CA, and browsers warn. Set one on `swarm-central` and `swarm-services`.
- **SSH and the rest of your hosts' base setup.** The example touches only the
firewall ports it needs.
- **The hardware configuration.** Each machine's own file sets
`networking.hostName` to one of the five names, which picks its VPN entry.
Wiring one machine into a flake looks like this:
```nix
nixosConfigurations.swarm-central = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
hyperhive.nixosModules.default
./hardware/swarm-central.nix # sets networking.hostName = "swarm-central"
./semi-spread/common.nix
./semi-spread/swarm-central.nix
];
};
```
`hive-one` and `hive-two` add `./semi-spread/hive.nix` instead of a
machine-specific file.

3
docs/2-handbook/swarm/services.md
View file

@ -80,6 +80,9 @@ the shapes worth naming are the ones they produce:
`allSwarmServices`.
- **Fully spread out.** One container / VM / machine per service,
somewhere.
- **Semi-spread.** The services split over a few machines by weight, hives on
their own, and a VPN between them. A worked example with its NixOS files:
[`semi-spread-deploy.md`](semi-spread-deploy.md).
**These are a set, not a ladder with a correct top, and hyperhive
supports in-between shapes.** You can set each derived toggle on its own (see

3
docs/README.md
View file

@ -32,6 +32,9 @@ The services that exist once per swarm. Start at
- **How are two hives connected into a swarm?** → [`2-handbook/swarm/`](2-handbook/swarm/README.md)
(peer hives, TLS trust).
- **How to spread the swarm over a few machines, with a VPN and only the forge,
matrix and swarm UI public?** → [`2-handbook/swarm/semi-spread-deploy.md`](2-handbook/swarm/semi-spread-deploy.md)
(worked example: machines, WireGuard, DNS).
- **What does the swarm UI show, and what does `swarmctl` do?** →
[`2-handbook/swarm/ui.md`](2-handbook/swarm/ui.md) and
[`2-handbook/swarm/swarmctl-cli.md`](2-handbook/swarm/swarmctl-cli.md) (the

21
docs/examples/semi-spread/common.nix Normal file
View file

@ -0,0 +1,21 @@
# Imported by every machine in the swarm: the swarm's shared facts, the VPN and
# the names inside it. `services.hyperhive.swarm.*` has to be identical on all
# of them, so it lives here and nowhere else.
{
imports = [
./wireguard.nix
./dns.nix
];
services.hyperhive.swarm = {
domain = "myswarm.example";
# Every hive, including the one a machine does not run. Names only: each
# hive's domain defaults to `<name>.myswarm.example`. A hive name may not
# contain "hive", "agent" or "swarm", so the machines `hive-one` and
# `hive-two` run the hives `one` and `two`.
hives = {
one = { };
two = { };
};
};
}

65
docs/examples/semi-spread/dns.nix Normal file
View file

@ -0,0 +1,65 @@
# Names inside the VPN. Every swarm name that is not public resolves to the
# VPN address of the machine serving it, through a static table written on
# each host. Identical on every host.
#
# Two readers need the table. Processes on the host read /etc/hosts
# (`networking.hosts`). Containers (agents, swarm services) ask the hive's
# dnsmasq, which ignores /etc/hosts, so it also reads the table through
# `addn-hosts`.
#
# A host leaves out the names it serves itself: hyperhive already answers
# those with the bridge address, and a table entry would override it.
{
config,
lib,
...
}:
let
mesh = import ./mesh.nix;
self = config.networking.hostName;
swarm = config.services.hyperhive.swarm;
# Which machine serves which name. A hive's machine is `hive-<name>` and it
# serves the hive's own names: its dashboard and its domain.
servedBy =
lib.mapAttrs' (
hive: entry:
lib.nameValuePair "hive-${hive}" [
swarm.dashboardDomains.${hive}
entry.domain
]
) swarm.hives
// {
swarm-central = [
swarm.domain
swarm.bao.domain
swarm.bao.ui.domain
];
swarm-services = [
swarm.forge.domain
swarm.matrix.gatewayHost
swarm.authelia.domain
swarm.nats.domain
swarm.otel.domain
swarm.grafana.domain
swarm.victoriametrics.domain
swarm.victorialogs.domain
];
swarm-heavyduty = [ swarm.attic.ownDomain ];
};
elsewhere = removeAttrs servedBy [ self ];
in
{
networking.hosts = lib.mapAttrs' (
machine: names: lib.nameValuePair mesh.${machine}.address names
) elsewhere;
services.dnsmasq.settings.addn-hosts = "${builtins.toFile "swarm-vpn-hosts" (
lib.concatStringsSep "\n" (
lib.mapAttrsToList (
machine: names: "${mesh.${machine}.address} ${lib.concatStringsSep " " names}"
) elsewhere
)
)}";
}

28
docs/examples/semi-spread/hive.nix Normal file
View file

@ -0,0 +1,28 @@
# A hive. Identical on hive-one and hive-two: the machine `hive-<name>` runs the
# hive `<name>`.
{ config, lib, ... }:
let
# The leaves for this hive, copied from swarm-central, where the secret store
# runs: `client.pem` and `client-key.pem` from
# `/var/lib/swarm-bao-pki/hives/<name>/`, along with that directory's
# `hive-leaf.pem` and `hive-leaf-key.pem`, which the hive's gateway
# certificate comes from. Both pairs sit in one directory.
pki = "/var/lib/swarm-pki";
in
{
services.hyperhive.deploy = {
hive-controller = {
enable = true;
hiveName = lib.removePrefix "hive-" config.networking.hostName;
};
bao = {
clientCertFile = "${pki}/client.pem";
clientKeyFile = "${pki}/client-key.pem";
};
};
# The dashboard is reachable over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [
config.services.hyperhive.gateway.httpsPort
];
}

33
docs/examples/semi-spread/mesh.nix Normal file
View file

@ -0,0 +1,33 @@
# The VPN's facts: one entry per machine, read by every host (wireguard.nix,
# dns.nix) and nothing else. Keyed by `networking.hostName`.
#
# `address` is the machine's address inside the VPN; `endpoint` is where the
# other machines dial it; `publicKey` is the output of `wg pubkey` on that
# machine. The keys below are placeholders: replace each with the real one.
{
swarm-central = {
address = "10.77.0.1";
endpoint = "203.0.113.10:51820";
publicKey = "REPLACE-with-swarm-central-wg-pubkey";
};
swarm-services = {
address = "10.77.0.2";
endpoint = "203.0.113.11:51820";
publicKey = "REPLACE-with-swarm-services-wg-pubkey";
};
swarm-heavyduty = {
address = "10.77.0.3";
endpoint = "203.0.113.12:51820";
publicKey = "REPLACE-with-swarm-heavyduty-wg-pubkey";
};
hive-one = {
address = "10.77.0.11";
endpoint = "198.51.100.11:51820";
publicKey = "REPLACE-with-hive-one-wg-pubkey";
};
hive-two = {
address = "10.77.0.12";
endpoint = "198.51.100.12:51820";
publicKey = "REPLACE-with-hive-two-wg-pubkey";
};
}

23
docs/examples/semi-spread/swarm-central.nix Normal file
View file

@ -0,0 +1,23 @@
# swarm-central: the swarm controller (and the swarm UI it serves) and the
# secret store.
{ config, ... }:
let
mesh = import ./mesh.nix;
hyperhive = config.services.hyperhive;
in
{
services.hyperhive = {
deploy.swarm-controller.enable = true;
deploy.bao.enable = true;
# The store also listens on this machine's VPN address, for the services
# and hives on other machines.
deploy.bao.extraListenAddresses = [ mesh.swarm-central.address ];
# The swarm UI is public.
gateway.openFirewall = true;
};
# The store is reachable over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.swarm.bao.port ];
}

31
docs/examples/semi-spread/swarm-heavyduty.nix Normal file
View file

@ -0,0 +1,31 @@
# swarm-heavyduty: the builder, the forge's CI runner and the binary cache.
{ config, lib, ... }:
let
hyperhive = config.services.hyperhive;
# Leaves copied from swarm-central, where the secret store runs. See
# swarm-services.nix for the helper's naming.
pki = "/var/lib/swarm-pki";
pair = option: stem: {
"${option}ClientCertFile" = "${pki}/${stem}.pem";
"${option}ClientKeyFile" = "${pki}/${stem}-key.pem";
};
in
{
services.hyperhive = {
deploy.builder.enable = true;
deploy.ci-runner.enable = true;
deploy.attic.enable = true;
deploy.bao = lib.mergeAttrsList [
(pair "builder" "builder")
(pair "ciRunner" "ci-runner")
(pair "attic" "attic")
(pair "forwarderOidc" "forwarder-oidc")
(pair "servicesIssuer" "services-issuer")
];
};
# The cache is served through the gateway, over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.gateway.httpsPort ];
}

64
docs/examples/semi-spread/swarm-services.nix Normal file
View file

@ -0,0 +1,64 @@
# swarm-services: the forge, the matrix homeserver, SSO, the queue, and the
# metrics and log stack.
{ config, lib, ... }:
let
hyperhive = config.services.hyperhive;
# The secret store runs on swarm-central, so each service here reads it with
# a leaf copied from there. `pair "grafanaOidc" "grafana-oidc"` names the
# options `deploy.bao.grafanaOidcClientCertFile` and `…KeyFile` and the files
# `grafana-oidc.pem` and `grafana-oidc-key.pem`.
pki = "/var/lib/swarm-pki";
pair = option: stem: {
"${option}ClientCertFile" = "${pki}/${stem}.pem";
"${option}ClientKeyFile" = "${pki}/${stem}-key.pem";
};
in
{
services.hyperhive = {
# Everything once-per-swarm that is not placed elsewhere below.
deploy.allSwarmServices = true;
# The forge and the matrix homeserver are public, and so is the SSO login
# they send browsers to.
gateway.openFirewall = true;
deploy.bao = lib.mergeAttrsList [
{ enable = false; }
(pair "forgePublisher" "forge-publisher")
(pair "servicesIssuer" "services-issuer")
(pair "grafanaOidc" "grafana-oidc")
(pair "otelOidc" "otel-oidc")
(pair "forwarderOidc" "forwarder-oidc")
(pair "reviewClassifier" "review-classifier")
(pair "notifyRouter" "notify-router")
(pair "claudeLogin" "claude-login")
(pair "ciMetrics" "ci-metrics")
];
deploy.matrix = {
ctlBaoClientCertFile = "${pki}/matrix-ctl.pem";
ctlBaoClientKeyFile = "${pki}/matrix-ctl-key.pem";
};
deploy.nats = {
baoClientCertFile = "${pki}/nats.pem";
baoClientKeyFile = "${pki}/nats-key.pem";
authBaoClientCertFile = "${pki}/nats-auth.pem";
authBaoClientKeyFile = "${pki}/nats-auth-key.pem";
};
deploy.swarm-secret-publisher = {
baoClientCertFile = "${pki}/secret-publisher.pem";
baoClientKeyFile = "${pki}/secret-publisher-key.pem";
};
deploy.victoriametrics = {
authBaoClientCertFile = "${pki}/metrics-auth.pem";
authBaoClientKeyFile = "${pki}/metrics-auth-key.pem";
};
deploy.victorialogs = {
authBaoClientCertFile = "${pki}/logs-auth.pem";
authBaoClientKeyFile = "${pki}/logs-auth-key.pem";
};
};
# The queue is reachable over the VPN only.
networking.firewall.interfaces.wg0.allowedTCPPorts = [ hyperhive.swarm.nats.port ];
}

24
docs/examples/semi-spread/wireguard.nix Normal file
View file

@ -0,0 +1,24 @@
# The VPN: plain NixOS WireGuard, one interface `wg0`, a full mesh built from
# ./mesh.nix. Identical on every host; the host's own entry is picked by
# `networking.hostName`.
{ config, lib, ... }:
let
mesh = import ./mesh.nix;
self = config.networking.hostName;
in
{
networking.wireguard.interfaces.wg0 = {
ips = [ "${mesh.${self}.address}/24" ];
listenPort = 51820;
# Created on the host by `wg genkey`, never in the store.
privateKeyFile = "/etc/wireguard/wg0.key";
peers = lib.mapAttrsToList (_: peer: {
inherit (peer) publicKey endpoint;
allowedIPs = [ "${peer.address}/32" ];
persistentKeepalive = 25;
}) (removeAttrs mesh [ self ]);
};
# The handshake arrives on the public interface.
networking.firewall.allowedUDPPorts = [ config.networking.wireguard.interfaces.wg0.listenPort ];
}