diff --git a/CLAUDE.md b/CLAUDE.md index 1c72fa66..9d811e00 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -165,6 +165,10 @@ read them à la carte. - **"How do I connect two hives into a swarm? How do I declare peer hives and configure TLS trust?"** → [`docs/swarm.md`](docs/swarm.md). +- **"Where do agent snapshots go? How does the swarm's `btrfs receive` + endpoint authenticate a pushing hive, and what does a snapshot + actually contain?"** → + [`docs/snapshot-store.md`](docs/snapshot-store.md). - **"How does the rebuild queue work? What are queue kinds and sources?"** → [`docs/coordinator.md`](docs/coordinator.md). - **"How does the CI runner work? What's the auto-registration flow?"** → diff --git a/docs/snapshot-store.md b/docs/snapshot-store.md new file mode 100644 index 00000000..8bacdd21 --- /dev/null +++ b/docs/snapshot-store.md @@ -0,0 +1,186 @@ +# Snapshot store + +The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it +over the WireGuard mesh; a destination hive later pulls one back to +complete a migration. + +Two things it is not, both worth stating because both are easy to +assume: + +- **It is not the swarm controller**, and does not depend on one. It is + a NixOS host role: a btrfs subvolume tree, a socket-activated + receiver, and the `wg-hive` interface the swarm module already brings + up. That is why it can be deployed before any controller exists. +- **It is not a backup product.** It happens to hold the data a backup + would hold, and it should be operated accordingly (see + [Operating it](#operating-it)) --- but nothing in it does scheduling, + verification, or restore orchestration. + +## Enabling it + +```nix +services.hyperhive.snapshotStore = { + enable = true; + path = "/var/lib/hyperhive-snapshots"; # must be on btrfs + port = 51821; +}; + +# The mesh is a hard requirement, and is asserted: +services.hyperhive.swarm.wireguard = { + enable = true; + address = "10.100.0.9/24"; + privateKeyFile = "/etc/wireguard/hive.key"; +}; +``` + +The store host is a swarm member like any other: peers declare it, and +it declares them, through `services.hyperhive.swarm.peers`. See +[swarm.md](swarm.md) for the mesh itself. + +Note that the mesh is gated on `swarm.wireguard.enable`, **not** on +`c0re.enable` --- a store host runs no hive and would otherwise get no +`wg-hive` interface at all. + +## The mesh is the authentication + +There are no certificates here, and no key material of its own. That is +deliberate rather than an omission. + +WireGuard's cryptokey routing already binds a peer's source address to +its public key: the swarm module configures each peer with +`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from +that address provably came from the holder of that private key. A +packet that reaches the receiver has therefore already been +authenticated by the kernel. + +Layering TLS client certs on top would authenticate *the same fact* a +second time, and add a credential with an expiry --- a migration that +fails because a renewal quietly didn't happen, discovered on the day +you need to move an agent. + +## One subvolume per agent, not per hive + +The destination is keyed by **agent**. + +This is not cosmetic. After a migration, an agent's next incremental +send arrives from a *different* hive than the previous one. Keying by +hive would split that agent's snapshot chain across two directories, +and `btrfs send -p` would fail to find its parent --- breaking exactly +the case the store exists to serve. + +## What the sender can and cannot choose + +A `btrfs send` stream carries no notion of *which agent* it belongs to, +and the subvolume name inside it is chosen by the sender. So the +protocol is one `agent ` header line, then the raw stream. + +The rule that matters: + +> **The receiver owns the destination root. The sender-supplied name is +> validated, never used as a path.** + +Validation is a whitelist --- `[A-Za-z0-9_-]+` and nothing else. No +slash and no dot means neither directory traversal nor an absolute path +can survive it. It is deliberately a whitelist and not a list of +forbidden characters: a blocklist only ever excludes the attacks +somebody already thought of. + +## Reachability + +The receiver is socket-activated, and the socket binds **this host's +mesh address**, never a wildcard. Both the mesh being enabled and the +address being set are assertions, not documentation --- bound to +`0.0.0.0` this socket is an unauthenticated remote write into agent +state. + +Binding is not sufficient on its own. NixOS's firewall is default-deny +and filters in netfilter, *before* a packet reaches a bound socket, so +the port is opened explicitly --- and scoped to the mesh interface: + +```nix +networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ]; +``` + +A host-wide `allowedTCPPorts` would open the port on every interface +including a public NIC, leaving only the socket's bind address between +the internet and a root `btrfs receive`. + +## Operating it + +### Confinement is the deployment's job + +`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root. +The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a +narrow `ReadWritePaths` --- but those are **defence in depth, not a +boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and +undo the namespace they set up. + +The boundary is the machine. The intended deployments are: + +- **a swarm**: the store is its own small VM. The machine is the + boundary, which is stronger than anything the unit could assert about + itself. +- **all-in-one / local**: the store runs as a container on the c0re + host. + +The second is worth keeping deliberately, and not only for +convenience: it means the confined path is exercised by every local +deployment. The usual failure mode for an isolated variant is that +nobody runs it day to day, so it rots and is discovered broken in +production. + +⚠️ **The assumption to keep true over time:** the store host runs +nothing else. That is true on day one and quietly false the day someone +notices the box has spare disk. Nothing in the config objects when it +stops being true. + +### It holds every agent's state from every hive + +Which makes it the highest-value target in the swarm by a wide margin, +and means it should get the treatment a backup host gets --- restricted +access, and a decision (rather than an omission) on encryption at rest. + +The trap is the label: this box holds backup-grade data while not being +called a backup, so it can end up with backup-grade *exposure* and +non-backup-grade *controls*. Nobody puts a migration staging area on +the access-review list. + +### What a snapshot contains + +The snapshot covers an agent's **state subvolume**, which is the parent +of `state/`, `claude/` and `harness/`. Consequences: + +- The Claude session (`claude/`) travels, so a restored agent keeps its + live `--continue` session rather than needing to log in again. +- `harness/` travels too, including `harness/bash-tasks/`. Task output + is part of an agent's working continuity, so this is wanted --- but it + means anything that has ever leaked into a task's captured output is + in the retained snapshots as well. + +It does **not** cover the agent's applied config (`/applied//`) or +its topology entry, both of which live outside the subvolume. A restore +therefore yields an agent's memory without its definition; closing that +gap is tracked separately. + +### Retention + +Retention lives on the *sending* side (last-N by count, swept +periodically), not here. Count rather than age is deliberate: a count +is bounded by construction, whereas an age policy silently scales disk +usage with how hot a hive runs. + +Per-agent or per-hive `btrfs qgroup` quotas are not configured yet. +Without them one runaway hive can fill the store and take out every +other hive's snapshots. + +## Not built yet + +**The pull side.** Push is safe with minimal authorisation because a +hive can only ever write to a chain it owns. Pull is the direction that +needs a policy: unrestricted, any compromised hive could read every +agent's state from every other hive. It needs a notion of which hive +currently owns which agent, and that ownership record lands with the +swarm controller work. + +With a single hive the question is trivial --- the only peer owns +everything it sends --- which is why the receive half ships first. diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 72451000..2b87a421 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -21,6 +21,8 @@ ./hive-priv.nix ./hive-tls.nix ./otel.nix + ./swarm-snapshot-store.nix + ./swarm-wireguard.nix ./swarm.nix ]; } diff --git a/nix/host-modules/swarm-snapshot-store.nix b/nix/host-modules/swarm-snapshot-store.nix new file mode 100644 index 00000000..bcbafc99 --- /dev/null +++ b/nix/host-modules/swarm-snapshot-store.nix @@ -0,0 +1,224 @@ +# swarm-snapshot-store — the swarm's `btrfs receive` endpoint. Hives push +# agent snapshots here over the existing WireGuard mesh; a destination +# hive later pulls one back to complete a migration. Only the receive +# half exists today — the pull side needs an authorisation model for +# "which hive may fetch which agent's state", which lands with the +# swarm controller. +# +# This is NOT the swarm controller and does not depend on it: a btrfs +# subvolume tree, a socket-activated receiver, and the `wg-hive` +# interface `swarm.nix` already brings up. Deliberately no WireGuard +# config of its own --- the mesh's cryptokey routing +# (`allowedIPs = [ peer.wireguardAddress ]`) already binds a peer's +# source address to its public key, so the mesh IS the authentication +# and adding certs here would authenticate the same fact twice. +# +# Confinement is a property of the DEPLOYMENT, not of this unit: in a +# real swarm the store is its own small VM (the machine is the +# boundary); in the all-local case it's a container on the c0re host. +# The module hardcodes neither. docs/snapshot-store.md covers what the +# deployment is expected to provide. +{ + pkgs, + lib, + config, + ... +}: +let + cfg = config.services.hyperhive.snapshotStore; + wgCfg = config.services.hyperhive.swarm.wireguard; + + # `swarm.wireguard.address` carries a prefix ("10.100.0.1/24") because + # it feeds `networking.wireguard.interfaces.wg-hive.ips`. A listen + # address must be the bare IP, so strip it. + meshAddress = lib.head (lib.splitString "/" wgCfg.address); + + # The receiver. Socket-activated with Accept=yes, so stdin IS the + # accepted connection and systemd hands us the peer address in + # $REMOTE_ADDR --- which, on this interface, is a cryptographically + # authenticated statement about which hive is talking (see the + # cryptokey-routing note above). + # + # PROTOCOL: one `agent \n` header line, then the raw `btrfs + # send` stream. The header exists because a btrfs stream does not + # carry the sending hive's notion of *which agent* it is --- the + # subvolume name inside the stream is chosen by the sender. + # + # ⚠️ The security rule, stated precisely, because the absolute form + # ("the sender never names its destination") is not achievable with + # btrfs send/receive: the RECEIVER owns the destination ROOT, and any + # sender-supplied component is VALIDATED, never used as a path. The + # name must match [A-Za-z0-9_-]+ exactly --- no slash, no dot, so no + # traversal and no absolute path can survive it. The root is ours; + # the leaf is checked against a whitelist charset before it is joined. + receiveScript = pkgs.writeShellScript "swarm-snapshot-receive" '' + set -euo pipefail + + # Read exactly the header line, leaving the byte stream untouched + # for btrfs receive. `read` stops at the newline and does not + # buffer ahead, which is why the header is a line and not a + # fixed-width record. + if ! read -r keyword agent; then + echo "swarm-snapshot-store: peer ''${REMOTE_ADDR:-?} closed before sending a header" >&2 + exit 1 + fi + + if [ "$keyword" != "agent" ]; then + echo "swarm-snapshot-store: peer ''${REMOTE_ADDR:-?} sent a bad header keyword" >&2 + exit 1 + fi + + # Validate rather than trust. Anything outside this charset is + # rejected outright --- this is the check that makes the joined + # path below safe, so it must stay a whitelist, never a blocklist + # of bad characters. + case "$agent" in + "" | *[!A-Za-z0-9_-]*) + echo "swarm-snapshot-store: peer ''${REMOTE_ADDR:-?} sent an invalid agent name" >&2 + exit 1 + ;; + esac + + dest="${cfg.path}/$agent" + + # One subvolume tree per AGENT, not per hive: after a migration the + # same agent's next incremental send arrives from a DIFFERENT hive, + # and a per-hive prefix would split its snapshot chain in two and + # break the incremental parent lookup --- exactly the case this + # store exists to serve. + mkdir -p "$dest" + + echo "swarm-snapshot-store: receiving agent=$agent from ''${REMOTE_ADDR:-?}" >&2 + exec ${pkgs.btrfs-progs}/bin/btrfs receive "$dest" + ''; +in +{ + options.services.hyperhive.snapshotStore = { + enable = lib.mkOption { + type = lib.types.bool; + default = false; + description = '' + Run the swarm snapshot store on this host: a `btrfs receive` + endpoint that hives push agent snapshots to over the WireGuard + mesh. Off by default --- it is a distinct deployment role, not + part of a hive. + + Requires `services.hyperhive.swarm.wireguard.enable`: the mesh + is both the transport and the authentication, so there is no + meaningful configuration without it. + ''; + }; + + path = lib.mkOption { + type = lib.types.path; + default = "/var/lib/hyperhive-snapshots"; + description = '' + Root of the snapshot tree. Must be on a btrfs filesystem --- + `btrfs receive` fails otherwise. One subvolume directory per + agent is created beneath it, so an agent's incremental chain + stays contiguous across a migration between hives. + ''; + }; + + port = lib.mkOption { + type = lib.types.port; + default = 51821; + description = '' + TCP port the receiver listens on. Bound to this host's + WireGuard mesh address only --- never a wildcard --- so it is + reachable exactly by mesh peers and by nothing else. + ''; + }; + + }; + + config = lib.mkIf cfg.enable { + assertions = [ + { + assertion = wgCfg.enable; + message = '' + services.hyperhive.snapshotStore.enable requires + services.hyperhive.swarm.wireguard.enable --- the mesh is the + store's transport AND its authentication (cryptokey routing + binds a peer's source address to its public key). Without it + there is nothing to bind the listener to and no way to tell + which hive is pushing. + ''; + } + { + assertion = wgCfg.address != ""; + message = '' + services.hyperhive.snapshotStore.enable requires + services.hyperhive.swarm.wireguard.address to be set --- the + receiver binds to this host's mesh address, and refuses to + fall back to a wildcard. + ''; + } + ]; + + # The store root must exist before the first connection arrives --- + # the receiver runs on demand and should not be the thing that + # creates its own tree lazily. + systemd.tmpfiles.rules = [ "d ${cfg.path} 0700 root root -" ]; + + # Open the receiver's port, scoped to the mesh interface. + # + # ⚠️ Binding the socket to the mesh address is NOT sufficient on its + # own: NixOS's firewall is default-deny and filters in netfilter, + # before a packet ever reaches a bound socket. The bind chooses + # WHICH address accepts connections; it does not open the port. The + # mesh's own UDP port is opened the same explicit way in swarm.nix. + # + # Interface-scoped rather than host-wide so the reachability + # property stays exactly what the option docs claim --- mesh peers + # and nobody else. A global `allowedTCPPorts` would open the port on + # every interface, including whatever public NIC the box has, and + # only the socket's bind address would still be standing between + # the internet and a root `btrfs receive`. + networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ]; + + # Socket-activated on purpose: no long-running root daemon, and the + # unit exists only while a transfer does. + # + # ⚠️ ListenStream is the mesh address, never 0.0.0.0. Bound to a + # wildcard this socket would be an unauthenticated remote write + # into agent state, so the binding IS the access control and is + # asserted above rather than left to a comment. + # + # Accept=yes gives one service instance per connection and sets + # $REMOTE_ADDR for the handler --- which is how the receiver knows + # which peer it is talking to. + systemd.sockets.swarm-snapshot-store = { + description = "hyperhive swarm snapshot store receiver socket"; + wantedBy = [ "sockets.target" ]; + socketConfig = { + ListenStream = "${meshAddress}:${toString cfg.port}"; + Accept = "yes"; + }; + }; + + # `btrfs receive` needs CAP_SYS_ADMIN, so this runs as root by + # nature. The hardening below is defence in depth and NOT a + # boundary: a process holding CAP_SYS_ADMIN can call mount(2) and + # undo the namespace these directives set up. The real boundary is + # the deployment --- see docs/snapshot-store.md. + systemd.services."swarm-snapshot-store@" = { + description = "hyperhive swarm snapshot store receiver"; + after = [ "swarm-snapshot-store.socket" ]; + requires = [ "swarm-snapshot-store.socket" ]; + serviceConfig = { + ExecStart = receiveScript; + SyslogIdentifier = "swarm-snapshot-store"; + # StandardInput=socket wires the accepted connection to stdin, + # which is what the handler reads the header + stream from. + StandardInput = "socket"; + StandardError = "journal"; + User = "root"; + PrivateTmp = true; + ProtectHome = true; + ProtectSystem = "strict"; + ReadWritePaths = [ cfg.path ]; + }; + }; + }; +} diff --git a/nix/host-modules/swarm-wireguard.nix b/nix/host-modules/swarm-wireguard.nix new file mode 100644 index 00000000..4433de13 --- /dev/null +++ b/nix/host-modules/swarm-wireguard.nix @@ -0,0 +1,142 @@ +# The WireGuard inter-hive mesh for the local host. Split out of +# ./swarm.nix because the two are different concerns with different +# audiences: that file declares WHO the peers are (data hive-c0re +# serialises into HYPERHIVE_PEERS and the dashboard renders), while +# this one is plain host networking that a machine which runs no hive +# at all --- the snapshot store, for one --- still needs. +# +# The two stay coupled by data, not by structure: the per-peer +# `wireguard*` fields live on the peer submodule in ./swarm.nix, since +# that is where a peer is described, and this module reads them. +{ + lib, + config, + ... +}: +{ + # WireGuard mesh config for the local host. + # When enabled, a `wg-hive` interface connects to all peers that have + # `wireguardPublicKey` declared. Peers reachable over the mesh are + # preferred for inter-hive traffic (no public TLS round-trip needed); + # peers without a public key still work via normal HTTPS. + options.services.hyperhive.swarm.wireguard = { + enable = lib.mkOption { + type = lib.types.bool; + default = false; + description = '' + Enable the WireGuard inter-hive mesh. When true, a `wg-hive` + interface is brought up connecting to all swarm peers that + declare a `wireguardPublicKey`. Requires + `privateKeyFile` to be set. + ''; + }; + + privateKeyFile = lib.mkOption { + type = lib.types.nullOr lib.types.path; + default = null; + example = "/etc/wireguard/hive.key"; + description = '' + Path to the host's WireGuard private key file. The file must + be readable by root and should have mode 0400. Generate with + `wg genkey > /etc/wireguard/hive.key`. Required when + `swarm.wireguard.enable = true`. + ''; + }; + + address = lib.mkOption { + type = lib.types.str; + default = ""; + example = "10.100.0.1/24"; + description = '' + IP address (with prefix) of this host on the WireGuard mesh. + Use a /24 (or broader) prefix so the routing table covers all + peer /32 routes. Example: `"10.100.0.1/24"` for a 256-host mesh. + ''; + }; + + listenPort = lib.mkOption { + type = lib.types.port; + default = 51820; + description = '' + UDP port the local WireGuard interface listens on. Must be + reachable from peer hosts when they initiate the tunnel. + Default: 51820 (standard WireGuard port). + ''; + }; + + persistentKeepalive = lib.mkOption { + type = lib.types.nullOr lib.types.int; + default = 25; + example = 25; + description = '' + Seconds between keepalive packets sent to each peer. Useful + when this host (or a peer) is behind NAT — keeps the UDP hole + open. Set to null to disable. Default: 25 seconds. + ''; + }; + }; + + # Gated on the mesh itself, NOT on the c0re daemon. The mesh is host + # networking, not a c0re feature: a swarm host that runs no hive — + # the snapshot store, for one — still has to join the mesh, and under + # the old `c0re.enable` gate it silently got no `wg-hive` interface + # at all. Nothing below is c0re-specific; the peer data + # c0re consumes (HYPERHIVE_PEERS / HIVE_PEER_CA_PATHS) is rendered in + # ./hive-c0re and stays gated there. + config = lib.mkIf config.services.hyperhive.swarm.wireguard.enable { + assertions = [ + { + assertion = config.services.hyperhive.swarm.wireguard.privateKeyFile != null; + message = '' + services.hyperhive.swarm.wireguard.enable requires + services.hyperhive.swarm.wireguard.privateKeyFile to be set. + Generate a key: wg genkey > /etc/wireguard/hive.key + ''; + } + { + assertion = config.services.hyperhive.swarm.wireguard.address != ""; + message = '' + services.hyperhive.swarm.wireguard.enable requires + services.hyperhive.swarm.wireguard.address to be set + (e.g. "10.100.0.1/24"). + ''; + } + ]; + + # WireGuard inter-hive mesh. Brings up a `wg-hive` interface and + # connects to each peer that has `wireguardPublicKey` set. + networking.wireguard.interfaces = + let + wgCfg = config.services.hyperhive.swarm.wireguard; + meshPeers = lib.filterAttrs ( + _: p: p.wireguardPublicKey != null && p.wireguardAddress != null + ) config.services.hyperhive.swarm.peers; + in + { + wg-hive = { + ips = [ wgCfg.address ]; + listenPort = wgCfg.listenPort; + privateKeyFile = wgCfg.privateKeyFile; + peers = lib.mapAttrsToList ( + _domain: p: + { + publicKey = p.wireguardPublicKey; + allowedIPs = [ p.wireguardAddress ]; + } + // lib.optionalAttrs (p.wireguardEndpoint != null) { + endpoint = p.wireguardEndpoint; + } + // lib.optionalAttrs (wgCfg.persistentKeepalive != null) { + persistentKeepalive = wgCfg.persistentKeepalive; + } + ) meshPeers; + }; + }; + + # Open the WireGuard UDP port on the host firewall (host-level + # networking — not inside containers). + networking.firewall.allowedUDPPorts = [ + config.services.hyperhive.swarm.wireguard.listenPort + ]; + }; +} diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index a6750900..00465c78 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -1,8 +1,11 @@ -# Swarm peering: the peer-hive declarations and the optional -# WireGuard inter-hive mesh. The peers are serialised into hive-c0re's +# Swarm peering: who the peer hives are. Serialised into hive-c0re's # environment (HYPERHIVE_PEERS / HIVE_PEER_CA_PATHS — see ./hive-c0re) -# and consumed by identity.rs + the dashboard's P33RS tab; the mesh -# config below is host-level networking. +# and consumed by identity.rs + the dashboard's P33RS tab. +# +# Declaration only — this module has no `config` block. The mesh that +# uses the `wireguard*` fields below lives in ./swarm-wireguard.nix, +# because bringing up an interface is host networking rather than swarm +# bookkeeping, and a host that runs no hive still needs it. { lib, config, @@ -122,125 +125,4 @@ ''; }; - # WireGuard mesh config for the local host. - # When enabled, a `wg-hive` interface connects to all peers that have - # `wireguardPublicKey` declared. Peers reachable over the mesh are - # preferred for inter-hive traffic (no public TLS round-trip needed); - # peers without a public key still work via normal HTTPS. - options.services.hyperhive.swarm.wireguard = { - enable = lib.mkOption { - type = lib.types.bool; - default = false; - description = '' - Enable the WireGuard inter-hive mesh. When true, a `wg-hive` - interface is brought up connecting to all swarm peers that - declare a `wireguardPublicKey`. Requires - `privateKeyFile` to be set. - ''; - }; - - privateKeyFile = lib.mkOption { - type = lib.types.nullOr lib.types.path; - default = null; - example = "/etc/wireguard/hive.key"; - description = '' - Path to the host's WireGuard private key file. The file must - be readable by root and should have mode 0400. Generate with - `wg genkey > /etc/wireguard/hive.key`. Required when - `swarm.wireguard.enable = true`. - ''; - }; - - address = lib.mkOption { - type = lib.types.str; - default = ""; - example = "10.100.0.1/24"; - description = '' - IP address (with prefix) of this host on the WireGuard mesh. - Use a /24 (or broader) prefix so the routing table covers all - peer /32 routes. Example: `"10.100.0.1/24"` for a 256-host mesh. - ''; - }; - - listenPort = lib.mkOption { - type = lib.types.port; - default = 51820; - description = '' - UDP port the local WireGuard interface listens on. Must be - reachable from peer hosts when they initiate the tunnel. - Default: 51820 (standard WireGuard port). - ''; - }; - - persistentKeepalive = lib.mkOption { - type = lib.types.nullOr lib.types.int; - default = 25; - example = 25; - description = '' - Seconds between keepalive packets sent to each peer. Useful - when this host (or a peer) is behind NAT — keeps the UDP hole - open. Set to null to disable. Default: 25 seconds. - ''; - }; - }; - - # Gated on the c0re daemon being enabled — the mesh is part of the - # coordinator host's networking. - config = lib.mkIf config.services.hyperhive.c0re.enable { - assertions = lib.optionals config.services.hyperhive.swarm.wireguard.enable [ - { - assertion = config.services.hyperhive.swarm.wireguard.privateKeyFile != null; - message = '' - services.hyperhive.swarm.wireguard.enable requires - services.hyperhive.swarm.wireguard.privateKeyFile to be set. - Generate a key: wg genkey > /etc/wireguard/hive.key - ''; - } - { - assertion = config.services.hyperhive.swarm.wireguard.address != ""; - message = '' - services.hyperhive.swarm.wireguard.enable requires - services.hyperhive.swarm.wireguard.address to be set - (e.g. "10.100.0.1/24"). - ''; - } - ]; - - # WireGuard inter-hive mesh. Brings up a `wg-hive` interface and - # connects to each peer that has `wireguardPublicKey` set. - networking.wireguard.interfaces = lib.mkIf config.services.hyperhive.swarm.wireguard.enable ( - let - wgCfg = config.services.hyperhive.swarm.wireguard; - meshPeers = lib.filterAttrs ( - _: p: p.wireguardPublicKey != null && p.wireguardAddress != null - ) config.services.hyperhive.swarm.peers; - in - { - wg-hive = { - ips = [ wgCfg.address ]; - listenPort = wgCfg.listenPort; - privateKeyFile = wgCfg.privateKeyFile; - peers = lib.mapAttrsToList ( - _domain: p: - { - publicKey = p.wireguardPublicKey; - allowedIPs = [ p.wireguardAddress ]; - } - // lib.optionalAttrs (p.wireguardEndpoint != null) { - endpoint = p.wireguardEndpoint; - } - // lib.optionalAttrs (wgCfg.persistentKeepalive != null) { - persistentKeepalive = wgCfg.persistentKeepalive; - } - ) meshPeers; - }; - } - ); - - # Open the WireGuard UDP port on the host firewall when the mesh is - # on (host-level networking — not inside containers). - networking.firewall.allowedUDPPorts = lib.mkIf config.services.hyperhive.swarm.wireguard.enable [ - config.services.hyperhive.swarm.wireguard.listenPort - ]; - }; }