Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6a6266cd5e | ||
|
|
70bcdb5463 | ||
|
|
57459cb6d8 | ||
|
|
c051cd9717 | ||
|
|
4989579270 | ||
|
|
bdf8fdabd7 |
6 changed files with 565 additions and 125 deletions
|
|
@ -165,6 +165,10 @@ read them à la carte.
|
||||||
- **"How do I connect two hives into a swarm? How do I declare peer
|
- **"How do I connect two hives into a swarm? How do I declare peer
|
||||||
hives and configure TLS trust?"** →
|
hives and configure TLS trust?"** →
|
||||||
[`docs/swarm.md`](docs/swarm.md).
|
[`docs/swarm.md`](docs/swarm.md).
|
||||||
|
- **"Where do agent snapshots go? How does the swarm's `btrfs receive`
|
||||||
|
endpoint authenticate a pushing hive, and what does a snapshot
|
||||||
|
actually contain?"** →
|
||||||
|
[`docs/snapshot-store.md`](docs/snapshot-store.md).
|
||||||
- **"How does the rebuild queue work? What are queue kinds and sources?"** →
|
- **"How does the rebuild queue work? What are queue kinds and sources?"** →
|
||||||
[`docs/coordinator.md`](docs/coordinator.md).
|
[`docs/coordinator.md`](docs/coordinator.md).
|
||||||
- **"How does the CI runner work? What's the auto-registration flow?"** →
|
- **"How does the CI runner work? What's the auto-registration flow?"** →
|
||||||
|
|
|
||||||
186
docs/snapshot-store.md
Normal file
186
docs/snapshot-store.md
Normal file
|
|
@ -0,0 +1,186 @@
|
||||||
|
# Snapshot store
|
||||||
|
|
||||||
|
The swarm's `btrfs receive` endpoint. Hives push agent snapshots to it
|
||||||
|
over the WireGuard mesh; a destination hive later pulls one back to
|
||||||
|
complete a migration.
|
||||||
|
|
||||||
|
Two things it is not, both worth stating because both are easy to
|
||||||
|
assume:
|
||||||
|
|
||||||
|
- **It is not the swarm controller**, and does not depend on one. It is
|
||||||
|
a NixOS host role: a btrfs subvolume tree, a socket-activated
|
||||||
|
receiver, and the `wg-hive` interface the swarm module already brings
|
||||||
|
up. That is why it can be deployed before any controller exists.
|
||||||
|
- **It is not a backup product.** It happens to hold the data a backup
|
||||||
|
would hold, and it should be operated accordingly (see
|
||||||
|
[Operating it](#operating-it)) --- but nothing in it does scheduling,
|
||||||
|
verification, or restore orchestration.
|
||||||
|
|
||||||
|
## Enabling it
|
||||||
|
|
||||||
|
```nix
|
||||||
|
services.hyperhive.snapshotStore = {
|
||||||
|
enable = true;
|
||||||
|
path = "/var/lib/hyperhive-snapshots"; # must be on btrfs
|
||||||
|
port = 51821;
|
||||||
|
};
|
||||||
|
|
||||||
|
# The mesh is a hard requirement, and is asserted:
|
||||||
|
services.hyperhive.swarm.wireguard = {
|
||||||
|
enable = true;
|
||||||
|
address = "10.100.0.9/24";
|
||||||
|
privateKeyFile = "/etc/wireguard/hive.key";
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
The store host is a swarm member like any other: peers declare it, and
|
||||||
|
it declares them, through `services.hyperhive.swarm.peers`. See
|
||||||
|
[swarm.md](swarm.md) for the mesh itself.
|
||||||
|
|
||||||
|
Note that the mesh is gated on `swarm.wireguard.enable`, **not** on
|
||||||
|
`c0re.enable` --- a store host runs no hive and would otherwise get no
|
||||||
|
`wg-hive` interface at all.
|
||||||
|
|
||||||
|
## The mesh is the authentication
|
||||||
|
|
||||||
|
There are no certificates here, and no key material of its own. That is
|
||||||
|
deliberate rather than an omission.
|
||||||
|
|
||||||
|
WireGuard's cryptokey routing already binds a peer's source address to
|
||||||
|
its public key: the swarm module configures each peer with
|
||||||
|
`allowedIPs = [ peer.wireguardAddress ]`, so a packet arriving from
|
||||||
|
that address provably came from the holder of that private key. A
|
||||||
|
packet that reaches the receiver has therefore already been
|
||||||
|
authenticated by the kernel.
|
||||||
|
|
||||||
|
Layering TLS client certs on top would authenticate *the same fact* a
|
||||||
|
second time, and add a credential with an expiry --- a migration that
|
||||||
|
fails because a renewal quietly didn't happen, discovered on the day
|
||||||
|
you need to move an agent.
|
||||||
|
|
||||||
|
## One subvolume per agent, not per hive
|
||||||
|
|
||||||
|
The destination is keyed by **agent**.
|
||||||
|
|
||||||
|
This is not cosmetic. After a migration, an agent's next incremental
|
||||||
|
send arrives from a *different* hive than the previous one. Keying by
|
||||||
|
hive would split that agent's snapshot chain across two directories,
|
||||||
|
and `btrfs send -p` would fail to find its parent --- breaking exactly
|
||||||
|
the case the store exists to serve.
|
||||||
|
|
||||||
|
## What the sender can and cannot choose
|
||||||
|
|
||||||
|
A `btrfs send` stream carries no notion of *which agent* it belongs to,
|
||||||
|
and the subvolume name inside it is chosen by the sender. So the
|
||||||
|
protocol is one `agent <name>` header line, then the raw stream.
|
||||||
|
|
||||||
|
The rule that matters:
|
||||||
|
|
||||||
|
> **The receiver owns the destination root. The sender-supplied name is
|
||||||
|
> validated, never used as a path.**
|
||||||
|
|
||||||
|
Validation is a whitelist --- `[A-Za-z0-9_-]+` and nothing else. No
|
||||||
|
slash and no dot means neither directory traversal nor an absolute path
|
||||||
|
can survive it. It is deliberately a whitelist and not a list of
|
||||||
|
forbidden characters: a blocklist only ever excludes the attacks
|
||||||
|
somebody already thought of.
|
||||||
|
|
||||||
|
## Reachability
|
||||||
|
|
||||||
|
The receiver is socket-activated, and the socket binds **this host's
|
||||||
|
mesh address**, never a wildcard. Both the mesh being enabled and the
|
||||||
|
address being set are assertions, not documentation --- bound to
|
||||||
|
`0.0.0.0` this socket is an unauthenticated remote write into agent
|
||||||
|
state.
|
||||||
|
|
||||||
|
Binding is not sufficient on its own. NixOS's firewall is default-deny
|
||||||
|
and filters in netfilter, *before* a packet reaches a bound socket, so
|
||||||
|
the port is opened explicitly --- and scoped to the mesh interface:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
|
||||||
|
```
|
||||||
|
|
||||||
|
A host-wide `allowedTCPPorts` would open the port on every interface
|
||||||
|
including a public NIC, leaving only the socket's bind address between
|
||||||
|
the internet and a root `btrfs receive`.
|
||||||
|
|
||||||
|
## Operating it
|
||||||
|
|
||||||
|
### Confinement is the deployment's job
|
||||||
|
|
||||||
|
`btrfs receive` needs `CAP_SYS_ADMIN`, so the receiver runs as root.
|
||||||
|
The unit sets `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp` and a
|
||||||
|
narrow `ReadWritePaths` --- but those are **defence in depth, not a
|
||||||
|
boundary**: a process holding `CAP_SYS_ADMIN` can call `mount(2)` and
|
||||||
|
undo the namespace they set up.
|
||||||
|
|
||||||
|
The boundary is the machine. The intended deployments are:
|
||||||
|
|
||||||
|
- **a swarm**: the store is its own small VM. The machine is the
|
||||||
|
boundary, which is stronger than anything the unit could assert about
|
||||||
|
itself.
|
||||||
|
- **all-in-one / local**: the store runs as a container on the c0re
|
||||||
|
host.
|
||||||
|
|
||||||
|
The second is worth keeping deliberately, and not only for
|
||||||
|
convenience: it means the confined path is exercised by every local
|
||||||
|
deployment. The usual failure mode for an isolated variant is that
|
||||||
|
nobody runs it day to day, so it rots and is discovered broken in
|
||||||
|
production.
|
||||||
|
|
||||||
|
⚠️ **The assumption to keep true over time:** the store host runs
|
||||||
|
nothing else. That is true on day one and quietly false the day someone
|
||||||
|
notices the box has spare disk. Nothing in the config objects when it
|
||||||
|
stops being true.
|
||||||
|
|
||||||
|
### It holds every agent's state from every hive
|
||||||
|
|
||||||
|
Which makes it the highest-value target in the swarm by a wide margin,
|
||||||
|
and means it should get the treatment a backup host gets --- restricted
|
||||||
|
access, and a decision (rather than an omission) on encryption at rest.
|
||||||
|
|
||||||
|
The trap is the label: this box holds backup-grade data while not being
|
||||||
|
called a backup, so it can end up with backup-grade *exposure* and
|
||||||
|
non-backup-grade *controls*. Nobody puts a migration staging area on
|
||||||
|
the access-review list.
|
||||||
|
|
||||||
|
### What a snapshot contains
|
||||||
|
|
||||||
|
The snapshot covers an agent's **state subvolume**, which is the parent
|
||||||
|
of `state/`, `claude/` and `harness/`. Consequences:
|
||||||
|
|
||||||
|
- The Claude session (`claude/`) travels, so a restored agent keeps its
|
||||||
|
live `--continue` session rather than needing to log in again.
|
||||||
|
- `harness/` travels too, including `harness/bash-tasks/`. Task output
|
||||||
|
is part of an agent's working continuity, so this is wanted --- but it
|
||||||
|
means anything that has ever leaked into a task's captured output is
|
||||||
|
in the retained snapshots as well.
|
||||||
|
|
||||||
|
It does **not** cover the agent's applied config (`/applied/<name>/`) or
|
||||||
|
its topology entry, both of which live outside the subvolume. A restore
|
||||||
|
therefore yields an agent's memory without its definition; closing that
|
||||||
|
gap is tracked separately.
|
||||||
|
|
||||||
|
### Retention
|
||||||
|
|
||||||
|
Retention lives on the *sending* side (last-N by count, swept
|
||||||
|
periodically), not here. Count rather than age is deliberate: a count
|
||||||
|
is bounded by construction, whereas an age policy silently scales disk
|
||||||
|
usage with how hot a hive runs.
|
||||||
|
|
||||||
|
Per-agent or per-hive `btrfs qgroup` quotas are not configured yet.
|
||||||
|
Without them one runaway hive can fill the store and take out every
|
||||||
|
other hive's snapshots.
|
||||||
|
|
||||||
|
## Not built yet
|
||||||
|
|
||||||
|
**The pull side.** Push is safe with minimal authorisation because a
|
||||||
|
hive can only ever write to a chain it owns. Pull is the direction that
|
||||||
|
needs a policy: unrestricted, any compromised hive could read every
|
||||||
|
agent's state from every other hive. It needs a notion of which hive
|
||||||
|
currently owns which agent, and that ownership record lands with the
|
||||||
|
swarm controller work.
|
||||||
|
|
||||||
|
With a single hive the question is trivial --- the only peer owns
|
||||||
|
everything it sends --- which is why the receive half ships first.
|
||||||
|
|
@ -21,6 +21,8 @@
|
||||||
./hive-priv.nix
|
./hive-priv.nix
|
||||||
./hive-tls.nix
|
./hive-tls.nix
|
||||||
./otel.nix
|
./otel.nix
|
||||||
|
./swarm-snapshot-store.nix
|
||||||
|
./swarm-wireguard.nix
|
||||||
./swarm.nix
|
./swarm.nix
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|
|
||||||
224
nix/host-modules/swarm-snapshot-store.nix
Normal file
224
nix/host-modules/swarm-snapshot-store.nix
Normal file
|
|
@ -0,0 +1,224 @@
|
||||||
|
# swarm-snapshot-store — the swarm's `btrfs receive` endpoint. Hives push
|
||||||
|
# agent snapshots here over the existing WireGuard mesh; a destination
|
||||||
|
# hive later pulls one back to complete a migration. Only the receive
|
||||||
|
# half exists today — the pull side needs an authorisation model for
|
||||||
|
# "which hive may fetch which agent's state", which lands with the
|
||||||
|
# swarm controller.
|
||||||
|
#
|
||||||
|
# This is NOT the swarm controller and does not depend on it: a btrfs
|
||||||
|
# subvolume tree, a socket-activated receiver, and the `wg-hive`
|
||||||
|
# interface `swarm.nix` already brings up. Deliberately no WireGuard
|
||||||
|
# config of its own --- the mesh's cryptokey routing
|
||||||
|
# (`allowedIPs = [ peer.wireguardAddress ]`) already binds a peer's
|
||||||
|
# source address to its public key, so the mesh IS the authentication
|
||||||
|
# and adding certs here would authenticate the same fact twice.
|
||||||
|
#
|
||||||
|
# Confinement is a property of the DEPLOYMENT, not of this unit: in a
|
||||||
|
# real swarm the store is its own small VM (the machine is the
|
||||||
|
# boundary); in the all-local case it's a container on the c0re host.
|
||||||
|
# The module hardcodes neither. docs/snapshot-store.md covers what the
|
||||||
|
# deployment is expected to provide.
|
||||||
|
{
|
||||||
|
pkgs,
|
||||||
|
lib,
|
||||||
|
config,
|
||||||
|
...
|
||||||
|
}:
|
||||||
|
let
|
||||||
|
cfg = config.services.hyperhive.snapshotStore;
|
||||||
|
wgCfg = config.services.hyperhive.swarm.wireguard;
|
||||||
|
|
||||||
|
# `swarm.wireguard.address` carries a prefix ("10.100.0.1/24") because
|
||||||
|
# it feeds `networking.wireguard.interfaces.wg-hive.ips`. A listen
|
||||||
|
# address must be the bare IP, so strip it.
|
||||||
|
meshAddress = lib.head (lib.splitString "/" wgCfg.address);
|
||||||
|
|
||||||
|
# The receiver. Socket-activated with Accept=yes, so stdin IS the
|
||||||
|
# accepted connection and systemd hands us the peer address in
|
||||||
|
# $REMOTE_ADDR --- which, on this interface, is a cryptographically
|
||||||
|
# authenticated statement about which hive is talking (see the
|
||||||
|
# cryptokey-routing note above).
|
||||||
|
#
|
||||||
|
# PROTOCOL: one `agent <name>\n` header line, then the raw `btrfs
|
||||||
|
# send` stream. The header exists because a btrfs stream does not
|
||||||
|
# carry the sending hive's notion of *which agent* it is --- the
|
||||||
|
# subvolume name inside the stream is chosen by the sender.
|
||||||
|
#
|
||||||
|
# ⚠️ The security rule, stated precisely, because the absolute form
|
||||||
|
# ("the sender never names its destination") is not achievable with
|
||||||
|
# btrfs send/receive: the RECEIVER owns the destination ROOT, and any
|
||||||
|
# sender-supplied component is VALIDATED, never used as a path. The
|
||||||
|
# name must match [A-Za-z0-9_-]+ exactly --- no slash, no dot, so no
|
||||||
|
# traversal and no absolute path can survive it. The root is ours;
|
||||||
|
# the leaf is checked against a whitelist charset before it is joined.
|
||||||
|
receiveScript = pkgs.writeShellScript "swarm-snapshot-receive" ''
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Read exactly the header line, leaving the byte stream untouched
|
||||||
|
# for btrfs receive. `read` stops at the newline and does not
|
||||||
|
# buffer ahead, which is why the header is a line and not a
|
||||||
|
# fixed-width record.
|
||||||
|
if ! read -r keyword agent; then
|
||||||
|
echo "swarm-snapshot-store: peer ''${REMOTE_ADDR:-?} closed before sending a header" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$keyword" != "agent" ]; then
|
||||||
|
echo "swarm-snapshot-store: peer ''${REMOTE_ADDR:-?} sent a bad header keyword" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Validate rather than trust. Anything outside this charset is
|
||||||
|
# rejected outright --- this is the check that makes the joined
|
||||||
|
# path below safe, so it must stay a whitelist, never a blocklist
|
||||||
|
# of bad characters.
|
||||||
|
case "$agent" in
|
||||||
|
"" | *[!A-Za-z0-9_-]*)
|
||||||
|
echo "swarm-snapshot-store: peer ''${REMOTE_ADDR:-?} sent an invalid agent name" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
dest="${cfg.path}/$agent"
|
||||||
|
|
||||||
|
# One subvolume tree per AGENT, not per hive: after a migration the
|
||||||
|
# same agent's next incremental send arrives from a DIFFERENT hive,
|
||||||
|
# and a per-hive prefix would split its snapshot chain in two and
|
||||||
|
# break the incremental parent lookup --- exactly the case this
|
||||||
|
# store exists to serve.
|
||||||
|
mkdir -p "$dest"
|
||||||
|
|
||||||
|
echo "swarm-snapshot-store: receiving agent=$agent from ''${REMOTE_ADDR:-?}" >&2
|
||||||
|
exec ${pkgs.btrfs-progs}/bin/btrfs receive "$dest"
|
||||||
|
'';
|
||||||
|
in
|
||||||
|
{
|
||||||
|
options.services.hyperhive.snapshotStore = {
|
||||||
|
enable = lib.mkOption {
|
||||||
|
type = lib.types.bool;
|
||||||
|
default = false;
|
||||||
|
description = ''
|
||||||
|
Run the swarm snapshot store on this host: a `btrfs receive`
|
||||||
|
endpoint that hives push agent snapshots to over the WireGuard
|
||||||
|
mesh. Off by default --- it is a distinct deployment role, not
|
||||||
|
part of a hive.
|
||||||
|
|
||||||
|
Requires `services.hyperhive.swarm.wireguard.enable`: the mesh
|
||||||
|
is both the transport and the authentication, so there is no
|
||||||
|
meaningful configuration without it.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
path = lib.mkOption {
|
||||||
|
type = lib.types.path;
|
||||||
|
default = "/var/lib/hyperhive-snapshots";
|
||||||
|
description = ''
|
||||||
|
Root of the snapshot tree. Must be on a btrfs filesystem ---
|
||||||
|
`btrfs receive` fails otherwise. One subvolume directory per
|
||||||
|
agent is created beneath it, so an agent's incremental chain
|
||||||
|
stays contiguous across a migration between hives.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
port = lib.mkOption {
|
||||||
|
type = lib.types.port;
|
||||||
|
default = 51821;
|
||||||
|
description = ''
|
||||||
|
TCP port the receiver listens on. Bound to this host's
|
||||||
|
WireGuard mesh address only --- never a wildcard --- so it is
|
||||||
|
reachable exactly by mesh peers and by nothing else.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
};
|
||||||
|
|
||||||
|
config = lib.mkIf cfg.enable {
|
||||||
|
assertions = [
|
||||||
|
{
|
||||||
|
assertion = wgCfg.enable;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.snapshotStore.enable requires
|
||||||
|
services.hyperhive.swarm.wireguard.enable --- the mesh is the
|
||||||
|
store's transport AND its authentication (cryptokey routing
|
||||||
|
binds a peer's source address to its public key). Without it
|
||||||
|
there is nothing to bind the listener to and no way to tell
|
||||||
|
which hive is pushing.
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
{
|
||||||
|
assertion = wgCfg.address != "";
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.snapshotStore.enable requires
|
||||||
|
services.hyperhive.swarm.wireguard.address to be set --- the
|
||||||
|
receiver binds to this host's mesh address, and refuses to
|
||||||
|
fall back to a wildcard.
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# The store root must exist before the first connection arrives ---
|
||||||
|
# the receiver runs on demand and should not be the thing that
|
||||||
|
# creates its own tree lazily.
|
||||||
|
systemd.tmpfiles.rules = [ "d ${cfg.path} 0700 root root -" ];
|
||||||
|
|
||||||
|
# Open the receiver's port, scoped to the mesh interface.
|
||||||
|
#
|
||||||
|
# ⚠️ Binding the socket to the mesh address is NOT sufficient on its
|
||||||
|
# own: NixOS's firewall is default-deny and filters in netfilter,
|
||||||
|
# before a packet ever reaches a bound socket. The bind chooses
|
||||||
|
# WHICH address accepts connections; it does not open the port. The
|
||||||
|
# mesh's own UDP port is opened the same explicit way in swarm.nix.
|
||||||
|
#
|
||||||
|
# Interface-scoped rather than host-wide so the reachability
|
||||||
|
# property stays exactly what the option docs claim --- mesh peers
|
||||||
|
# and nobody else. A global `allowedTCPPorts` would open the port on
|
||||||
|
# every interface, including whatever public NIC the box has, and
|
||||||
|
# only the socket's bind address would still be standing between
|
||||||
|
# the internet and a root `btrfs receive`.
|
||||||
|
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ cfg.port ];
|
||||||
|
|
||||||
|
# Socket-activated on purpose: no long-running root daemon, and the
|
||||||
|
# unit exists only while a transfer does.
|
||||||
|
#
|
||||||
|
# ⚠️ ListenStream is the mesh address, never 0.0.0.0. Bound to a
|
||||||
|
# wildcard this socket would be an unauthenticated remote write
|
||||||
|
# into agent state, so the binding IS the access control and is
|
||||||
|
# asserted above rather than left to a comment.
|
||||||
|
#
|
||||||
|
# Accept=yes gives one service instance per connection and sets
|
||||||
|
# $REMOTE_ADDR for the handler --- which is how the receiver knows
|
||||||
|
# which peer it is talking to.
|
||||||
|
systemd.sockets.swarm-snapshot-store = {
|
||||||
|
description = "hyperhive swarm snapshot store receiver socket";
|
||||||
|
wantedBy = [ "sockets.target" ];
|
||||||
|
socketConfig = {
|
||||||
|
ListenStream = "${meshAddress}:${toString cfg.port}";
|
||||||
|
Accept = "yes";
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
# `btrfs receive` needs CAP_SYS_ADMIN, so this runs as root by
|
||||||
|
# nature. The hardening below is defence in depth and NOT a
|
||||||
|
# boundary: a process holding CAP_SYS_ADMIN can call mount(2) and
|
||||||
|
# undo the namespace these directives set up. The real boundary is
|
||||||
|
# the deployment --- see docs/snapshot-store.md.
|
||||||
|
systemd.services."swarm-snapshot-store@" = {
|
||||||
|
description = "hyperhive swarm snapshot store receiver";
|
||||||
|
after = [ "swarm-snapshot-store.socket" ];
|
||||||
|
requires = [ "swarm-snapshot-store.socket" ];
|
||||||
|
serviceConfig = {
|
||||||
|
ExecStart = receiveScript;
|
||||||
|
SyslogIdentifier = "swarm-snapshot-store";
|
||||||
|
# StandardInput=socket wires the accepted connection to stdin,
|
||||||
|
# which is what the handler reads the header + stream from.
|
||||||
|
StandardInput = "socket";
|
||||||
|
StandardError = "journal";
|
||||||
|
User = "root";
|
||||||
|
PrivateTmp = true;
|
||||||
|
ProtectHome = true;
|
||||||
|
ProtectSystem = "strict";
|
||||||
|
ReadWritePaths = [ cfg.path ];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
142
nix/host-modules/swarm-wireguard.nix
Normal file
142
nix/host-modules/swarm-wireguard.nix
Normal file
|
|
@ -0,0 +1,142 @@
|
||||||
|
# The WireGuard inter-hive mesh for the local host. Split out of
|
||||||
|
# ./swarm.nix because the two are different concerns with different
|
||||||
|
# audiences: that file declares WHO the peers are (data hive-c0re
|
||||||
|
# serialises into HYPERHIVE_PEERS and the dashboard renders), while
|
||||||
|
# this one is plain host networking that a machine which runs no hive
|
||||||
|
# at all --- the snapshot store, for one --- still needs.
|
||||||
|
#
|
||||||
|
# The two stay coupled by data, not by structure: the per-peer
|
||||||
|
# `wireguard*` fields live on the peer submodule in ./swarm.nix, since
|
||||||
|
# that is where a peer is described, and this module reads them.
|
||||||
|
{
|
||||||
|
lib,
|
||||||
|
config,
|
||||||
|
...
|
||||||
|
}:
|
||||||
|
{
|
||||||
|
# WireGuard mesh config for the local host.
|
||||||
|
# When enabled, a `wg-hive` interface connects to all peers that have
|
||||||
|
# `wireguardPublicKey` declared. Peers reachable over the mesh are
|
||||||
|
# preferred for inter-hive traffic (no public TLS round-trip needed);
|
||||||
|
# peers without a public key still work via normal HTTPS.
|
||||||
|
options.services.hyperhive.swarm.wireguard = {
|
||||||
|
enable = lib.mkOption {
|
||||||
|
type = lib.types.bool;
|
||||||
|
default = false;
|
||||||
|
description = ''
|
||||||
|
Enable the WireGuard inter-hive mesh. When true, a `wg-hive`
|
||||||
|
interface is brought up connecting to all swarm peers that
|
||||||
|
declare a `wireguardPublicKey`. Requires
|
||||||
|
`privateKeyFile` to be set.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
privateKeyFile = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.path;
|
||||||
|
default = null;
|
||||||
|
example = "/etc/wireguard/hive.key";
|
||||||
|
description = ''
|
||||||
|
Path to the host's WireGuard private key file. The file must
|
||||||
|
be readable by root and should have mode 0400. Generate with
|
||||||
|
`wg genkey > /etc/wireguard/hive.key`. Required when
|
||||||
|
`swarm.wireguard.enable = true`.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
address = lib.mkOption {
|
||||||
|
type = lib.types.str;
|
||||||
|
default = "";
|
||||||
|
example = "10.100.0.1/24";
|
||||||
|
description = ''
|
||||||
|
IP address (with prefix) of this host on the WireGuard mesh.
|
||||||
|
Use a /24 (or broader) prefix so the routing table covers all
|
||||||
|
peer /32 routes. Example: `"10.100.0.1/24"` for a 256-host mesh.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
listenPort = lib.mkOption {
|
||||||
|
type = lib.types.port;
|
||||||
|
default = 51820;
|
||||||
|
description = ''
|
||||||
|
UDP port the local WireGuard interface listens on. Must be
|
||||||
|
reachable from peer hosts when they initiate the tunnel.
|
||||||
|
Default: 51820 (standard WireGuard port).
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
persistentKeepalive = lib.mkOption {
|
||||||
|
type = lib.types.nullOr lib.types.int;
|
||||||
|
default = 25;
|
||||||
|
example = 25;
|
||||||
|
description = ''
|
||||||
|
Seconds between keepalive packets sent to each peer. Useful
|
||||||
|
when this host (or a peer) is behind NAT — keeps the UDP hole
|
||||||
|
open. Set to null to disable. Default: 25 seconds.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
# Gated on the mesh itself, NOT on the c0re daemon. The mesh is host
|
||||||
|
# networking, not a c0re feature: a swarm host that runs no hive —
|
||||||
|
# the snapshot store, for one — still has to join the mesh, and under
|
||||||
|
# the old `c0re.enable` gate it silently got no `wg-hive` interface
|
||||||
|
# at all. Nothing below is c0re-specific; the peer data
|
||||||
|
# c0re consumes (HYPERHIVE_PEERS / HIVE_PEER_CA_PATHS) is rendered in
|
||||||
|
# ./hive-c0re and stays gated there.
|
||||||
|
config = lib.mkIf config.services.hyperhive.swarm.wireguard.enable {
|
||||||
|
assertions = [
|
||||||
|
{
|
||||||
|
assertion = config.services.hyperhive.swarm.wireguard.privateKeyFile != null;
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.swarm.wireguard.enable requires
|
||||||
|
services.hyperhive.swarm.wireguard.privateKeyFile to be set.
|
||||||
|
Generate a key: wg genkey > /etc/wireguard/hive.key
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
{
|
||||||
|
assertion = config.services.hyperhive.swarm.wireguard.address != "";
|
||||||
|
message = ''
|
||||||
|
services.hyperhive.swarm.wireguard.enable requires
|
||||||
|
services.hyperhive.swarm.wireguard.address to be set
|
||||||
|
(e.g. "10.100.0.1/24").
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
# WireGuard inter-hive mesh. Brings up a `wg-hive` interface and
|
||||||
|
# connects to each peer that has `wireguardPublicKey` set.
|
||||||
|
networking.wireguard.interfaces =
|
||||||
|
let
|
||||||
|
wgCfg = config.services.hyperhive.swarm.wireguard;
|
||||||
|
meshPeers = lib.filterAttrs (
|
||||||
|
_: p: p.wireguardPublicKey != null && p.wireguardAddress != null
|
||||||
|
) config.services.hyperhive.swarm.peers;
|
||||||
|
in
|
||||||
|
{
|
||||||
|
wg-hive = {
|
||||||
|
ips = [ wgCfg.address ];
|
||||||
|
listenPort = wgCfg.listenPort;
|
||||||
|
privateKeyFile = wgCfg.privateKeyFile;
|
||||||
|
peers = lib.mapAttrsToList (
|
||||||
|
_domain: p:
|
||||||
|
{
|
||||||
|
publicKey = p.wireguardPublicKey;
|
||||||
|
allowedIPs = [ p.wireguardAddress ];
|
||||||
|
}
|
||||||
|
// lib.optionalAttrs (p.wireguardEndpoint != null) {
|
||||||
|
endpoint = p.wireguardEndpoint;
|
||||||
|
}
|
||||||
|
// lib.optionalAttrs (wgCfg.persistentKeepalive != null) {
|
||||||
|
persistentKeepalive = wgCfg.persistentKeepalive;
|
||||||
|
}
|
||||||
|
) meshPeers;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
# Open the WireGuard UDP port on the host firewall (host-level
|
||||||
|
# networking — not inside containers).
|
||||||
|
networking.firewall.allowedUDPPorts = [
|
||||||
|
config.services.hyperhive.swarm.wireguard.listenPort
|
||||||
|
];
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
@ -1,8 +1,11 @@
|
||||||
# Swarm peering: the peer-hive declarations and the optional
|
# Swarm peering: who the peer hives are. Serialised into hive-c0re's
|
||||||
# WireGuard inter-hive mesh. The peers are serialised into hive-c0re's
|
|
||||||
# environment (HYPERHIVE_PEERS / HIVE_PEER_CA_PATHS — see ./hive-c0re)
|
# environment (HYPERHIVE_PEERS / HIVE_PEER_CA_PATHS — see ./hive-c0re)
|
||||||
# and consumed by identity.rs + the dashboard's P33RS tab; the mesh
|
# and consumed by identity.rs + the dashboard's P33RS tab.
|
||||||
# config below is host-level networking.
|
#
|
||||||
|
# Declaration only — this module has no `config` block. The mesh that
|
||||||
|
# uses the `wireguard*` fields below lives in ./swarm-wireguard.nix,
|
||||||
|
# because bringing up an interface is host networking rather than swarm
|
||||||
|
# bookkeeping, and a host that runs no hive still needs it.
|
||||||
{
|
{
|
||||||
lib,
|
lib,
|
||||||
config,
|
config,
|
||||||
|
|
@ -122,125 +125,4 @@
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
|
|
||||||
# WireGuard mesh config for the local host.
|
|
||||||
# When enabled, a `wg-hive` interface connects to all peers that have
|
|
||||||
# `wireguardPublicKey` declared. Peers reachable over the mesh are
|
|
||||||
# preferred for inter-hive traffic (no public TLS round-trip needed);
|
|
||||||
# peers without a public key still work via normal HTTPS.
|
|
||||||
options.services.hyperhive.swarm.wireguard = {
|
|
||||||
enable = lib.mkOption {
|
|
||||||
type = lib.types.bool;
|
|
||||||
default = false;
|
|
||||||
description = ''
|
|
||||||
Enable the WireGuard inter-hive mesh. When true, a `wg-hive`
|
|
||||||
interface is brought up connecting to all swarm peers that
|
|
||||||
declare a `wireguardPublicKey`. Requires
|
|
||||||
`privateKeyFile` to be set.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
privateKeyFile = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.path;
|
|
||||||
default = null;
|
|
||||||
example = "/etc/wireguard/hive.key";
|
|
||||||
description = ''
|
|
||||||
Path to the host's WireGuard private key file. The file must
|
|
||||||
be readable by root and should have mode 0400. Generate with
|
|
||||||
`wg genkey > /etc/wireguard/hive.key`. Required when
|
|
||||||
`swarm.wireguard.enable = true`.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
address = lib.mkOption {
|
|
||||||
type = lib.types.str;
|
|
||||||
default = "";
|
|
||||||
example = "10.100.0.1/24";
|
|
||||||
description = ''
|
|
||||||
IP address (with prefix) of this host on the WireGuard mesh.
|
|
||||||
Use a /24 (or broader) prefix so the routing table covers all
|
|
||||||
peer /32 routes. Example: `"10.100.0.1/24"` for a 256-host mesh.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
listenPort = lib.mkOption {
|
|
||||||
type = lib.types.port;
|
|
||||||
default = 51820;
|
|
||||||
description = ''
|
|
||||||
UDP port the local WireGuard interface listens on. Must be
|
|
||||||
reachable from peer hosts when they initiate the tunnel.
|
|
||||||
Default: 51820 (standard WireGuard port).
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
persistentKeepalive = lib.mkOption {
|
|
||||||
type = lib.types.nullOr lib.types.int;
|
|
||||||
default = 25;
|
|
||||||
example = 25;
|
|
||||||
description = ''
|
|
||||||
Seconds between keepalive packets sent to each peer. Useful
|
|
||||||
when this host (or a peer) is behind NAT — keeps the UDP hole
|
|
||||||
open. Set to null to disable. Default: 25 seconds.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Gated on the c0re daemon being enabled — the mesh is part of the
|
|
||||||
# coordinator host's networking.
|
|
||||||
config = lib.mkIf config.services.hyperhive.c0re.enable {
|
|
||||||
assertions = lib.optionals config.services.hyperhive.swarm.wireguard.enable [
|
|
||||||
{
|
|
||||||
assertion = config.services.hyperhive.swarm.wireguard.privateKeyFile != null;
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.swarm.wireguard.enable requires
|
|
||||||
services.hyperhive.swarm.wireguard.privateKeyFile to be set.
|
|
||||||
Generate a key: wg genkey > /etc/wireguard/hive.key
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
{
|
|
||||||
assertion = config.services.hyperhive.swarm.wireguard.address != "";
|
|
||||||
message = ''
|
|
||||||
services.hyperhive.swarm.wireguard.enable requires
|
|
||||||
services.hyperhive.swarm.wireguard.address to be set
|
|
||||||
(e.g. "10.100.0.1/24").
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# WireGuard inter-hive mesh. Brings up a `wg-hive` interface and
|
|
||||||
# connects to each peer that has `wireguardPublicKey` set.
|
|
||||||
networking.wireguard.interfaces = lib.mkIf config.services.hyperhive.swarm.wireguard.enable (
|
|
||||||
let
|
|
||||||
wgCfg = config.services.hyperhive.swarm.wireguard;
|
|
||||||
meshPeers = lib.filterAttrs (
|
|
||||||
_: p: p.wireguardPublicKey != null && p.wireguardAddress != null
|
|
||||||
) config.services.hyperhive.swarm.peers;
|
|
||||||
in
|
|
||||||
{
|
|
||||||
wg-hive = {
|
|
||||||
ips = [ wgCfg.address ];
|
|
||||||
listenPort = wgCfg.listenPort;
|
|
||||||
privateKeyFile = wgCfg.privateKeyFile;
|
|
||||||
peers = lib.mapAttrsToList (
|
|
||||||
_domain: p:
|
|
||||||
{
|
|
||||||
publicKey = p.wireguardPublicKey;
|
|
||||||
allowedIPs = [ p.wireguardAddress ];
|
|
||||||
}
|
|
||||||
// lib.optionalAttrs (p.wireguardEndpoint != null) {
|
|
||||||
endpoint = p.wireguardEndpoint;
|
|
||||||
}
|
|
||||||
// lib.optionalAttrs (wgCfg.persistentKeepalive != null) {
|
|
||||||
persistentKeepalive = wgCfg.persistentKeepalive;
|
|
||||||
}
|
|
||||||
) meshPeers;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
);
|
|
||||||
|
|
||||||
# Open the WireGuard UDP port on the host firewall when the mesh is
|
|
||||||
# on (host-level networking — not inside containers).
|
|
||||||
networking.firewall.allowedUDPPorts = lib.mkIf config.services.hyperhive.swarm.wireguard.enable [
|
|
||||||
config.services.hyperhive.swarm.wireguard.listenPort
|
|
||||||
];
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue