diff --git a/README.md b/README.md index 4d98338c3..38acfd7f2 100644 --- a/README.md +++ b/README.md @@ -55,8 +55,7 @@ the swarm — one of each, each on whichever host you choose ├── swarm-grafana dashboards over both stores │ └── opt-in, not part of singleHostSwarm - ├── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner) - └── wg-hive WireGuard mesh between hives (deploy.wireguard) + └── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner) the hives — the substrate, as many as you like (deploy.hive-controller.enable) │ diff --git a/docs/2-handbook/hives/hivectl-cli.md b/docs/2-handbook/hives/hivectl-cli.md index 7eae6192a..441074b5a 100644 --- a/docs/2-handbook/hives/hivectl-cli.md +++ b/docs/2-handbook/hives/hivectl-cli.md @@ -23,10 +23,6 @@ This document contains the help content for the `hivectl` command-line program. * [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete) * [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send) * [`hivectl list-agents`↴](#hivectl-list-agents) -* [`hivectl wg`↴](#hivectl-wg) -* [`hivectl wg init`↴](#hivectl-wg-init) -* [`hivectl wg peer`↴](#hivectl-wg-peer) -* [`hivectl wg status`↴](#hivectl-wg-status) * [`hivectl peer-config`↴](#hivectl-peer-config) * [`hivectl stop`↴](#hivectl-stop) * [`hivectl start`↴](#hivectl-start) @@ -43,7 +39,6 @@ Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that * `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running * `list-agents` — Show all managed agents with their status and technical state -* `wg` — WireGuard inter-hive mesh setup helpers * `peer-config` — Generate the federation peer-config block for THIS hive * `stop` — Stop containers hive-wide in one operator action * `start` — Start containers hive-wide — the inverse of `hivectl stop` @@ -290,72 +285,13 @@ Global — not scoped to one agent, so it lives at the top level rather than und -## `hivectl wg` - -WireGuard inter-hive mesh setup helpers. - -Generate this hive's mesh key and print the nix to enable the mesh, add a peer, or inspect live interface state. - -**Usage:** `hivectl wg ` - -###### **Subcommands:** - -* `init` — Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh -* `peer` — Print the nix to add a peer hive to the mesh -* `status` — Show the live mesh interface state - - - -## `hivectl wg init` - -Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh - -**Usage:** `hivectl wg init [OPTIONS]` - -###### **Options:** - -* `--address
` — This hive's mesh address (for example `10.42.0.1/32`) to bake into the printed snippet. Omit to get a placeholder you fill in - - - -## `hivectl wg peer` - -Print the nix to add a peer hive to the mesh - -**Usage:** `hivectl wg peer [OPTIONS] --pubkey --address
` - -###### **Arguments:** - -* `` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`) - -###### **Options:** - -* `--pubkey ` — Peer's WireGuard public key (from its `hivectl wg init`) -* `--address
` — Peer's mesh address (for example `10.42.0.2/32`) -* `--endpoint ` — Peer's `host:port` endpoint (omit for a peer that only dials out, for example one behind NAT — it must set an endpoint pointing back here) - - - -## `hivectl wg status` - -Show the live mesh interface state - -**Usage:** `hivectl wg status` - - - ## `hivectl peer-config` Generate the federation peer-config block for THIS hive. Prints the nix a peer operator pastes into their swarm config to trust and reach this hive. -**Usage:** `hivectl peer-config [OPTIONS]` - -###### **Options:** - -* `--wg-address ` — This hive's WireGuard mesh address (for example `10.42.0.1/32`), emitted as `wireguardAddress`. Omit when not running the mesh -* `--wg-endpoint ` — This hive's public WireGuard endpoint (`host:port`), emitted as `wireguardEndpoint`. Omit when peers dial in / no mesh +**Usage:** `hivectl peer-config` diff --git a/docs/2-handbook/swarm/README.md b/docs/2-handbook/swarm/README.md index 2ed77d7f6..999589cb5 100644 --- a/docs/2-handbook/swarm/README.md +++ b/docs/2-handbook/swarm/README.md @@ -126,9 +126,8 @@ if you want it. What that grant lets an agent reach on a hive is on the Each hive carries an identity (`hiveName`, the `swarm.hives` directory) that says which hive it's and how to reach the others; the directory feeds the -swarm UI's hive list, matrix federation and the optional WireGuard mesh. → -[the hive directory](hive-directory.md). A private link between hives instead -of public HTTPS: [WireGuard mesh](wireguard.md). +swarm UI's hive list and matrix federation. → +[the hive directory](hive-directory.md). ## Cross-references diff --git a/docs/2-handbook/swarm/hive-directory.md b/docs/2-handbook/swarm/hive-directory.md index c1dedf486..ec15b791e 100644 --- a/docs/2-handbook/swarm/hive-directory.md +++ b/docs/2-handbook/swarm/hive-directory.md @@ -95,18 +95,14 @@ mechanism. swarm's root needs a CA-issued certificate (ACME). Federation firewall and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md). -3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each - entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress` - to configure `wg-hive`. See [the WireGuard page](wireguard.md). - ## Reaching the queue and the store on a routable address -The mesh isn't the only way to reach the queue and the secret store from -another host. Set `deploy.nats.openFirewall` on the queue's host to open the -queue's port on every interface, and `deploy.bao.openFirewall` together with +Other hives reach the queue and the secret store on a routable address. Set +`deploy.nats.openFirewall` on the queue's host to open the queue's port on +every interface, and `deploy.bao.openFirewall` together with `deploy.bao.extraListenAddresses` on the store's host to open the store's port on the addresses it listens on. Both default to off. Each exposes a service -beyond the tunnel, which is the operator's decision. The queue requires TLS and +beyond the agent bridge, which is the operator's decision. The queue requires TLS and its `auth_callout` refuses every client it can't identify. The store requires a client certificate on every listener, and evaluation fails when you set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point diff --git a/docs/2-handbook/swarm/wireguard.md b/docs/2-handbook/swarm/wireguard.md deleted file mode 100644 index 2c85132ff..000000000 --- a/docs/2-handbook/swarm/wireguard.md +++ /dev/null @@ -1,80 +0,0 @@ -# WireGuard mesh between hives (optional) - -The peer config in [the hive directory](hive-directory.md#hive-identity-config) uses public HTTPS for all inter-hive traffic. -For private deployments — or to reduce latency and TLS overhead on -intra-swarm traffic — hyperhive can configure a host-to-host -WireGuard mesh. - -## Generating keys - -On each hive host: - -```bash -wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key -wg pubkey < /etc/wireguard/hive.key # → share this with peer operators -``` - -## Config example (two hives) - -```nix -# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1) -services.hyperhive = { - deploy.wireguard = { - enable = true; - privateKeyFile = "/etc/wireguard/hive.key"; - address = "10.100.0.1/24"; - listenPort = 51820; # optional, default 51820 - }; - - # The same `hives` attrset both hosts hold — mesh fields included, - # since "where this hive can be dialled" is a fact about that hive. - swarm.hives = { - firsthive = { - domain = "firsthive.myswarm.example"; - wireguardPublicKey = "base64keyA="; - wireguardEndpoint = "198.51.100.1:51820"; - wireguardAddress = "10.100.0.1/32"; - }; - secondhive = { - domain = "secondhive.elsewhere.example"; - wireguardPublicKey = "base64keyB="; - wireguardEndpoint = "203.0.113.42:51820"; - wireguardAddress = "10.100.0.2/32"; - }; - }; -}; - -# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2) -services.hyperhive = { - deploy.wireguard = { - enable = true; - privateKeyFile = "/etc/wireguard/hive.key"; - address = "10.100.0.2/24"; - }; - - swarm.hives = { - # … identical to hive A's … - }; -}; -``` - -## What the mesh does - -- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the - host (not inside agent containers; containers reach peers via the - host's routing table). -- It opens UDP port 51820 (or `listenPort`) on the host firewall. -- It reads each entry's `wireguardAddress` from `services.hyperhive.swarm.hives`, - minus this host's own hive, to build `wg-hive`'s `allowedIPs`, so intra-swarm - traffic can route over the mesh address rather than the public domain. -- It sets `persistentKeepalive = 25` by default; override or null to - disable (not needed when both sides have public IPs and no NAT). - -## NAT / one-sided endpoints - -If one host is behind NAT and can't accept incoming connections, only -that host needs a null `wireguardEndpoint` on the peer config — the -other side initiates. With keepalive on, the NAT hole stays open. - -If both hosts are behind NAT, you need a STUN relay or a third host -(exit node); hyperhive sets up neither. diff --git a/docs/4-implementation/networking/network.md b/docs/4-implementation/networking/network.md index a09ec368d..016e969e9 100644 --- a/docs/4-implementation/networking/network.md +++ b/docs/4-implementation/networking/network.md @@ -29,7 +29,6 @@ that touches it. │ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │ │ forgejo :3000 http, :2222 git-ssh [hive-forge] │ │ hive-c0re dashboard 127.0.0.1:7000 (host service) │ -│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │ │ │ │ hive-br0 10.42.0.1/24 │ │ ┌──────────┼──────────────┐ │ @@ -72,8 +71,8 @@ The flows, end to end: UI unix socket. - **Internet egress** — agent default route points at the bridge IP; the host forwards + masquerades out its uplink. -- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh - and reach each other's gateway/forge across it +- **Swarm** — peer hives reach each other's gateway and forge over HTTPS, and + the queue and the secret store on their routable listeners ([`docs/2-handbook/swarm/`](../../2-handbook/swarm/README.md)). - **Control plane (no network)** — per-agent broker socket `/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`, @@ -526,7 +525,7 @@ squid (`services.hyperhive.deploy.hive-controller.egress.intercept.enable`). - **Who gets redirected.** Every flow from the bridge to port 80 or 443, except a flow to an address that stays on the bridge, to one of the host's own - addresses or to the `wg-hive` interface, and a flow from a MAC in the set + addresses, and a flow from a MAC in the set `direct_mac`. That set holds the MACs of the ports of every container that's not an agent (the swarm's service containers) and of the agents listed in `services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So @@ -727,13 +726,13 @@ loopback — the hive-c0re admin socket is a UDS, not TCP. `services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of the `inet hive_egress` table does with traffic the host routes from the agent -bridge to anywhere but the bridge and the `wg-hive` mesh interface. It takes +bridge to anywhere but the bridge. It takes `accept`, `log` (the default) or `reject`. In `log` mode the chain blocks nothing. It logs each new flow and counts every packet. In `reject` mode it also refuses the flow. - **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the - mesh traffic first, then the traffic of every MAC in the `direct_mac` set: the + traffic of every MAC in the `direct_mac` set: the `unrestricted` agents and the service containers, which the `intercept` chain leaves alone too (see _Intercepted web traffic_). They leave no line, so a `hive-egress-would-drop` line is exactly a flow that `reject` would refuse, diff --git a/docs/4-implementation/scheduler/ci.md b/docs/4-implementation/scheduler/ci.md index 4b0c40748..50ec7e341 100644 --- a/docs/4-implementation/scheduler/ci.md +++ b/docs/4-implementation/scheduler/ci.md @@ -128,9 +128,7 @@ by hand, start the workflow from the forge UI (Actions, vm-nightly, Run workflow, optional `test` input), or run `nix build .#vmTests.x86_64-linux.` on a host with KVM. Current tests: `agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral -root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives -peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that -must not handshake). +root, through the real hive-priv unit) and `hive-squid`. The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no Nix interpolation. Commands the script runs in the VM find their binaries on the diff --git a/docs/4-implementation/swarm/controller.md b/docs/4-implementation/swarm/controller.md index a775c8902..2a3cbcb4f 100644 --- a/docs/4-implementation/swarm/controller.md +++ b/docs/4-implementation/swarm/controller.md @@ -72,9 +72,9 @@ The queue URL is the same string on every hive. The queue accepts TLS only, with a certificate for `swarm.nats.domain` (default `nats.`) and no other name or address, so a URL with an IP address or `nats://` fails. The queue's host resolves the name itself. **A multi-host swarm needs one -upstream DNS record**, `nats.` pointing at the queue host's mesh +upstream DNS record**, `nats.` pointing at the queue host's routable address, the same contract as `bao.`. The queue's port is open -on `wg-hive` when that host is on the mesh. +host-wide only with `deploy.nats.openFirewall`. The queue's host asks the secret store for that certificate under a store leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the diff --git a/docs/4-implementation/swarm/credentials-reference.md b/docs/4-implementation/swarm/credentials-reference.md index b57850880..f061878a7 100644 --- a/docs/4-implementation/swarm/credentials-reference.md +++ b/docs/4-implementation/swarm/credentials-reference.md @@ -148,7 +148,6 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-hand | the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services//oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else | | authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services//oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf | | authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia | -| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you | | queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) | | the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place | diff --git a/hivectl/README.md b/hivectl/README.md index 8ba0c6373..364067083 100644 --- a/hivectl/README.md +++ b/hivectl/README.md @@ -3,7 +3,7 @@ The operator-facing host CLI. A thin client for the `hive-c0re` daemon — speaks the host admin socket protocol (`hive-host-sock`) and does not link the daemon crate. Container lifecycle and the -gateway verbs all forward to the daemon and need it running; a few (`wg`/`peer-config`, `choom`) work off local +gateway verbs all forward to the daemon and need it running; a few (`peer-config`, `choom`) work off local host state instead. ## When to use it @@ -25,7 +25,7 @@ dispatch: - **`power.rs`** — restart/start/stop at the container level. - **`choom.rs`** — drop into an interactive claude session in a container. -- **`wg.rs`** — WireGuard mesh helpers. +- **`peer_config.rs`** — the federation block a peer pastes. - **`subvol.rs`** — btrfs state-subvolume ops. - **`quota.rs`**, **`util.rs`**, **`completions.rs`** — shared helpers and shell completions. diff --git a/hivectl/src/cli.rs b/hivectl/src/cli.rs index 508c3148d..39ecc9f49 100644 --- a/hivectl/src/cli.rs +++ b/hivectl/src/cli.rs @@ -51,28 +51,11 @@ pub enum Cmd { #[arg(long)] json: bool, }, - /// WireGuard inter-hive mesh setup helpers. - /// - /// Generate this hive's mesh key and print the nix to enable the mesh, - /// add a peer, or inspect live interface state. - Wg { - #[command(subcommand)] - cmd: WgCmd, - }, /// Generate the federation peer-config block for THIS hive. /// /// Prints the nix a peer operator pastes into their swarm config to /// trust and reach this hive. - PeerConfig { - /// This hive's WireGuard mesh address (for example `10.42.0.1/32`), - /// emitted as `wireguardAddress`. Omit when not running the mesh. - #[arg(long)] - wg_address: Option, - /// This hive's public WireGuard endpoint (`host:port`), emitted as - /// `wireguardEndpoint`. Omit when peers dial in / no mesh. - #[arg(long)] - wg_endpoint: Option, - }, + PeerConfig, /// Stop containers hive-wide in one operator action. /// /// Bare `hivectl stop` stops every agent and the gateway; scope flags @@ -164,35 +147,6 @@ impl ScopeArgs { } } -#[derive(Subcommand)] -pub enum WgCmd { - /// Generate this hive's WireGuard key (if absent) and print its public - /// key plus the nix to enable the mesh. - Init { - /// This hive's mesh address (for example `10.42.0.1/32`) to bake into the - /// printed snippet. Omit to get a placeholder you fill in. - #[arg(long)] - address: Option, - }, - /// Print the nix to add a peer hive to the mesh. - Peer { - /// Peer hive's DNS domain (its `swarm.hives` entry's `domain`) - domain: String, - /// Peer's WireGuard public key (from its `hivectl wg init`). - #[arg(long)] - pubkey: String, - /// Peer's mesh address (for example `10.42.0.2/32`). - #[arg(long)] - address: String, - /// Peer's `host:port` endpoint (omit for a peer that only dials out, - /// for example one behind NAT — it must set an endpoint pointing back here). - #[arg(long)] - endpoint: Option, - }, - /// Show the live mesh interface state. - Status, -} - // Default host admin socket path. Shared with `hive-c0re`'s `main.rs` // default via `hive_host_sock::HOST_SOCKET` — the daemon binds there // and `hivectl` connects to it. diff --git a/hivectl/src/main.rs b/hivectl/src/main.rs index 19fab6eb1..722bc7fb0 100644 --- a/hivectl/src/main.rs +++ b/hivectl/src/main.rs @@ -7,7 +7,7 @@ //! `gateway`) all forward to the daemon, which owns the broker, //! the credentials, and the provisioning logic — a running daemon is //! required for those. A couple of verbs work off local host state -//! directly instead (`wg` / `peer-config` read the mesh key + TLS CA), so +//! directly instead (`peer-config` reads the TLS CA), so //! they don't need the socket. `agent choom` execs into a container //! rather than asking the daemon to do anything, but still uses the socket //! for its "is this an agent?" pre-flight — that answer lives in a @@ -26,12 +26,12 @@ mod client; /// Rebuild-queue node progress rendering (`wait_for_nodes` + the spinner / /// plain renderers), split out to keep this file manageable. mod dag_progress; -use cli::{Cli, Cmd, WgCmd}; +use cli::{Cli, Cmd}; mod completions; mod util; use completions::generate_completions; -mod wg; -use wg::{peer_config, require_hive_domain, wg_init, wg_peer, wg_status}; +mod peer_config; +use peer_config::{peer_config, require_hive_domain}; mod agents; mod choom; mod watch; @@ -53,25 +53,9 @@ async fn main() -> Result<()> { match cli.cmd { Cmd::Agent { name, cmd } => run_agent(&socket, &name, cmd).await, Cmd::ListAgents { json } => agents_list(&socket, json).await, - Cmd::Wg { cmd } => match cmd { - WgCmd::Init { address } => wg_init(&socket, address.as_deref()).await, - WgCmd::Peer { - domain, - pubkey, - address, - endpoint, - } => { - wg_peer(&domain, &pubkey, &address, endpoint.as_deref()); - Ok(()) - } - WgCmd::Status => wg_status(), - }, - Cmd::PeerConfig { - wg_address, - wg_endpoint, - } => { + Cmd::PeerConfig => { let domain = require_hive_domain(&socket).await?; - peer_config(&domain, wg_address.as_deref(), wg_endpoint.as_deref()); + peer_config(&domain); Ok(()) } Cmd::Stop { diff --git a/hivectl/src/peer_config.rs b/hivectl/src/peer_config.rs new file mode 100644 index 000000000..50dc1243c --- /dev/null +++ b/hivectl/src/peer_config.rs @@ -0,0 +1,143 @@ +//! `hivectl peer-config` — print the block a peer operator pastes to +//! federate with this hive. + +use std::path::Path; + +use anyhow::{Context as _, Result}; + +use crate::util::query_hive_urls; + +/// Host path of this hive's TLS trust bundle (matches the +/// `services.hyperhive.deploy.hive-controller.tls.stateDir` default in +/// hive-tls.nix). Its existence means the gateway is in self-signed mode, +/// serving a leaf the secret store issued, so a federating peer needs an +/// anchor for it. Absent = ACME / operator cert, trusted by the default CA +/// bundle with nothing to distribute. +const HIVE_TLS_TRUST_BUNDLE_PATH: &str = "/var/lib/hive-tls/trust-bundle.pem"; + +/// Host path of the secret store's `pki` root (hive-tls.nix's +/// `$svcroot`, folded into the trust bundle). The anchor a whole swarm +/// shares: every hive's gateway leaf and the swarm's service names are +/// issued under it, so a peer host installs it once and every present +/// *and future* hive validates. +const STORE_PKI_ROOT_PATH: &str = "/var/lib/hive-tls/swarm-services-root.pem"; + +/// The option whose value keys a hive's entry in `services.hyperhive.swarm.hives`. +const HIVE_NAME_OPTION: &str = "services.hyperhive.deploy.hive-controller.hiveName"; + +/// Attrset key for a hive in `services.hyperhive.swarm.hives`, derived +/// from its domain's first DNS label. +/// +/// A hive occupies `.`, so the first label *is* +/// the hive name in any deployment that hasn't overridden `domain` by +/// hand. Where it has, a wrong key fails loudly rather than quietly: on +/// that hive's own host the `hives.` assertion fires, because +/// the directory is supposed to be the same attrset everywhere. The +/// snippet below says so rather than presenting the guess as fact. +fn hive_key(domain: &str) -> &str { + domain.split('.').next().unwrap_or(domain) +} + +/// Require this hive's domain from the daemon for snippet generation. +/// Errors with a clear hint when it can't be resolved, so `peer-config` +/// never silently emits a wrong key. An unreachable daemon propagates its +/// own (classified) connect error; only a *reachable* daemon with no domain +/// gets the config hint. +pub(crate) async fn require_hive_domain(socket: &Path) -> Result { + query_hive_urls(socket) + .await + .context("could not determine this hive's domain from the daemon")? + .and_then(|u| u.domain) + .context("the daemon reported no domain — set `services.hyperhive.domain`") +} + +/// The first closing comment line of the `peer-config` snippet. +fn peer_config_key_note() -> String { + format!("# the key must be this hive's {HIVE_NAME_OPTION}, and the") +} + +/// The `peer-config` step carrying the secret store's `pki` root: the +/// certificate itself, or where it will appear when this hive has not +/// fetched it yet. +fn store_root_step(pem: Option<&str>) -> String { + const HEAD: &str = "# 1. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\ + # every hive's gateway leaf and the swarm's service names are issued under it:\n"; + match pem.map(str::trim_end).filter(|p| !p.is_empty()) { + Some(p) => format!( + "{HEAD}{p}\n\ + # (a self-signed hive in this swarm fetches it itself, into {STORE_PKI_ROOT_PATH})\n" + ), + None => format!( + "{HEAD}# not on this host yet: swarm-services-cert.service writes it to\n\ + # {STORE_PKI_ROOT_PATH}; rerun once that unit has succeeded\n" + ), + } +} + +/// `peer-config` — print the `swarm.hives.""` block a peer +/// operator pastes to federate with THIS hive, plus the store-root step +/// when this hive serves a self-signed chain. Reads local state only (the +/// TLS trust bundle and the store root); prints, never mutates. +pub(crate) fn peer_config(domain: &str) { + let self_signed = Path::new(HIVE_TLS_TRUST_BUNDLE_PATH).exists(); + let key = hive_key(domain); + + if self_signed { + let store_root = std::fs::read_to_string(STORE_PKI_ROOT_PATH).ok(); + print!("{}", store_root_step(store_root.as_deref())); + println!(); + println!("# 2. paste into the peer hive's NixOS config:"); + } else { + println!("# paste into the peer hive's NixOS config:"); + } + println!("services.hyperhive.swarm.hives.\"{key}\" = {{"); + println!(" domain = \"{domain}\";"); + println!("}};"); + println!("{}", peer_config_key_note()); + println!("# same entry belongs in every hive's config — `hives` is the swarm."); + if !self_signed { + println!( + "# (this hive's cert chains to a public CA — nothing to install; \ + it's trusted by the default bundle.)" + ); + } +} + +#[cfg(test)] +mod tests { + use super::{STORE_PKI_ROOT_PATH, peer_config_key_note, store_root_step}; + + const PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----"; + + /// The snippet tells the operator to set the deploy option, never the + /// removed `services.hyperhive` spelling. + #[test] + fn key_note_names_the_deploy_option() { + let note = peer_config_key_note(); + assert!( + note.contains("services.hyperhive.deploy.hive-controller.hiveName"), + "{note}" + ); + assert!(!note.contains("services.hyperhive.hiveName"), "{note}"); + } + + /// The certificate is printed whole, on lines of its own, so a peer + /// operator can copy it out of the output as a PEM file. + #[test] + fn store_root_is_printed_verbatim() { + let s = store_root_step(Some(&format!("{PEM}\n"))); + assert!(s.contains(&format!("\n{PEM}\n")), "{s}"); + } + + /// A hive that has not fetched the root yet says where it will be, + /// rather than printing an empty step. + #[test] + fn missing_store_root_names_the_path_and_the_unit() { + for pem in [None, Some(""), Some("\n")] { + let s = store_root_step(pem); + assert!(s.contains(STORE_PKI_ROOT_PATH), "{s}"); + assert!(s.contains("swarm-services-cert.service"), "{s}"); + assert!(!s.contains("BEGIN CERTIFICATE"), "{s}"); + } + } +} diff --git a/hivectl/src/wg.rs b/hivectl/src/wg.rs deleted file mode 100644 index cf84f1200..000000000 --- a/hivectl/src/wg.rs +++ /dev/null @@ -1,365 +0,0 @@ -//! `hivectl wg` / `peer-config` — WireGuard mesh helpers: generate this -//! hive's key + the nix snippets to enable the mesh, add a peer, print the -//! block a peer pastes to federate with us, and show the live interface. - -use std::path::Path; - -use anyhow::{Context as _, Result, bail}; - -use crate::util::query_hive_urls; - -/// Host path of this hive's WireGuard private key (matches the -/// `privateKeyFile` example in the deploy.wireguard nix options). -const WG_KEY_PATH: &str = "/etc/wireguard/hive.key"; -/// The mesh interface name hive-c0re's nix module brings up. -const WG_INTERFACE: &str = "wg-hive"; - -/// Host path of this hive's TLS trust bundle (matches the -/// `services.hyperhive.deploy.hive-controller.tls.stateDir` default in -/// hive-tls.nix). Its existence means the gateway is in self-signed mode, -/// serving a leaf the secret store issued, so a federating peer needs an -/// anchor for it. Absent = ACME / operator cert, trusted by the default CA -/// bundle with nothing to distribute. -const HIVE_TLS_TRUST_BUNDLE_PATH: &str = "/var/lib/hive-tls/trust-bundle.pem"; - -/// Host path of the secret store's `pki` root (hive-tls.nix's -/// `$svcroot`, folded into the trust bundle). The anchor a whole swarm -/// shares: every hive's gateway leaf and the swarm's service names are -/// issued under it, so a peer host installs it once and every present -/// *and future* hive validates. -const STORE_PKI_ROOT_PATH: &str = "/var/lib/hive-tls/swarm-services-root.pem"; - -/// The option whose value keys a hive's entry in `services.hyperhive.swarm.hives`. -const HIVE_NAME_OPTION: &str = "services.hyperhive.deploy.hive-controller.hiveName"; - -/// Attrset key for a hive in `services.hyperhive.swarm.hives`, derived -/// from its domain's first DNS label. -/// -/// A hive occupies `.`, so the first label *is* -/// the hive name in any deployment that hasn't overridden `domain` by -/// hand. Where it has, a wrong key fails loudly rather than quietly: on -/// that hive's own host the `hives.` assertion fires, because -/// the directory is supposed to be the same attrset everywhere. The -/// snippets below say so rather than presenting the guess as fact. -fn hive_key(domain: &str) -> &str { - domain.split('.').next().unwrap_or(domain) -} - -/// Best-effort query for this hive's domain from the running daemon -/// (`HostRequest::Urls`, which reads `HYPERHIVE_HIVE_DOMAIN` from c0re's -/// service env). `None` when the daemon is unreachable or the domain is -/// unset — for the callers that treat both as "skip the optional block". -/// Use [`require_hive_domain`] where the distinction matters: it keeps the -/// connect error (and its hint) instead of flattening it away. -async fn query_hive_domain(socket: &Path) -> Option { - query_hive_urls(socket).await.ok().flatten()?.domain -} - -/// Require this hive's domain from the daemon for snippet generation. -/// Errors with a clear hint when it can't be resolved, so `peer-config` -/// never silently emits a wrong key. An unreachable daemon propagates its -/// own (classified) connect error; only a *reachable* daemon with no domain -/// gets the config hint. -pub(crate) async fn require_hive_domain(socket: &Path) -> Result { - query_hive_urls(socket) - .await - .context("could not determine this hive's domain from the daemon")? - .and_then(|u| u.domain) - .context("the daemon reported no domain — set `services.hyperhive.domain`") -} - -/// `wg init` — generate (if absent) the hive's WireGuard key, print its -/// public key + the nix snippet to enable the mesh, then (best-effort) -/// the `peer-config` block peers paste to federate with this hive, so a -/// fresh setup is one command. The domain comes from the daemon; if it -/// can't be resolved, the peer block is skipped (init still succeeds — -/// its core job is enabling the mesh locally). -pub(crate) async fn wg_init(socket: &Path, address: Option<&str>) -> Result<()> { - use std::os::unix::fs::PermissionsExt as _; - let key_path = Path::new(WG_KEY_PATH); - if key_path.exists() { - println!( - "WireGuard private key already present at {WG_KEY_PATH} — reusing (not regenerating)." - ); - } else { - if let Some(parent) = key_path.parent() { - std::fs::create_dir_all(parent) - .with_context(|| format!("create {}", parent.display()))?; - std::fs::set_permissions(parent, std::fs::Permissions::from_mode(0o700)).ok(); - } - let out = std::process::Command::new("wg") - .arg("genkey") - .output() - .context("run `wg genkey` (is wireguard-tools installed?)")?; - if !out.status.success() { - bail!( - "wg genkey failed: {}", - String::from_utf8_lossy(&out.stderr).trim() - ); - } - write_key_0400(key_path, &out.stdout)?; - println!("Generated WireGuard private key at {WG_KEY_PATH} (0400)."); - } - - let privkey = - std::fs::read(key_path).with_context(|| format!("read {}", key_path.display()))?; - let pubkey = wg_pubkey(&privkey)?; - let addr = address.unwrap_or(""); - println!("\nPublic key (share this with peer hives — they pass it to `hivectl wg peer`):"); - println!(" {pubkey}"); - println!("\nAdd to this hive's NixOS config:"); - println!(" services.hyperhive.deploy.wireguard = {{"); - println!(" enable = true;"); - println!(" privateKeyFile = \"{WG_KEY_PATH}\";"); - println!(" address = \"{addr}\";"); - println!(" # listenPort = 51820; # default"); - println!(" }};"); - - // Also print the block a peer pastes to federate with us (CA + this - // mesh key) — one-stop setup. Domain comes from the daemon; - // best-effort, so init still succeeds when it can't be resolved. - if let Some(d) = query_hive_domain(socket).await { - println!(); - peer_config(&d, address, None); - } - Ok(()) -} - -/// Write `key` to a sibling temp file created `0400` and rename it over -/// `path`, so the key is never readable by anyone else and a crash never -/// leaves half of it at `path`. Errors carry paths, never the key. -fn write_key_0400(path: &Path, key: &[u8]) -> Result<()> { - use std::io::Write as _; - use std::os::unix::fs::OpenOptionsExt as _; - - let mut tmp_name = path - .file_name() - .with_context(|| format!("{} has no file name", path.display()))? - .to_owned(); - tmp_name.push(".tmp"); - let tmp = path.with_file_name(tmp_name); - // A temp file left by a crashed earlier write would fail `create_new`. - let _ = std::fs::remove_file(&tmp); - let mut f = std::fs::OpenOptions::new() - .write(true) - .create_new(true) - .mode(0o400) - .open(&tmp) - .with_context(|| format!("create {}", tmp.display()))?; - f.write_all(key) - .and_then(|()| f.sync_all()) - .with_context(|| format!("write {}", tmp.display()))?; - std::fs::rename(&tmp, path).with_context(|| format!("rename onto {}", path.display())) -} - -/// Derive a WireGuard public key from a private key by piping it through -/// `wg pubkey`. -fn wg_pubkey(privkey: &[u8]) -> Result { - use std::io::Write as _; - use std::process::{Command, Stdio}; - let mut child = Command::new("wg") - .arg("pubkey") - .stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .spawn() - .context("run `wg pubkey` (is wireguard-tools installed?)")?; - child - .stdin - .take() - .context("wg pubkey: stdin unavailable")? - .write_all(privkey) - .context("write private key to `wg pubkey`")?; - let out = child.wait_with_output().context("wait for `wg pubkey`")?; - if !out.status.success() { - bail!( - "wg pubkey failed: {}", - String::from_utf8_lossy(&out.stderr).trim() - ); - } - Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned()) -} - -/// The closing comment of the `wg peer` snippet. -fn wg_peer_key_note() -> String { - format!(" # the key must be that hive's {HIVE_NAME_OPTION}") -} - -/// The first closing comment line of the `peer-config` snippet. -fn peer_config_key_note() -> String { - format!("# the key must be this hive's {HIVE_NAME_OPTION}, and the") -} - -/// `wg peer` — print the nix snippet to add a peer hive to the mesh. -/// Pure output (no fallible work), so it returns `()`; the dispatch arm -/// wraps it in `Ok` to match the sibling verbs. -pub(crate) fn wg_peer(domain: &str, pubkey: &str, address: &str, endpoint: Option<&str>) { - let key = hive_key(domain); - println!("Add to this hive's NixOS config:"); - println!(" # `hives` describes the whole swarm and is meant to be the same"); - println!(" # attrset on every host — add this entry to all of them."); - println!(" services.hyperhive.swarm.hives.\"{key}\" = {{"); - println!(" domain = \"{domain}\";"); - println!(" wireguardPublicKey = \"{pubkey}\";"); - println!(" wireguardAddress = \"{address}\";"); - if let Some(ep) = endpoint { - println!(" wireguardEndpoint = \"{ep}\";"); - } - println!(" }};"); - println!("{}", wg_peer_key_note()); -} - -/// The `peer-config` step carrying the secret store's `pki` root: the -/// certificate itself, or where it will appear when this hive has not -/// fetched it yet. -fn store_root_step(pem: Option<&str>) -> String { - const HEAD: &str = "# 1. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\ - # every hive's gateway leaf and the swarm's service names are issued under it:\n"; - match pem.map(str::trim_end).filter(|p| !p.is_empty()) { - Some(p) => format!( - "{HEAD}{p}\n\ - # (a self-signed hive in this swarm fetches it itself, into {STORE_PKI_ROOT_PATH})\n" - ), - None => format!( - "{HEAD}# not on this host yet: swarm-services-cert.service writes it to\n\ - # {STORE_PKI_ROOT_PATH}; rerun once that unit has succeeded\n" - ), - } -} - -/// `peer-config` — print the `swarm.hives.""` block a peer -/// operator pastes to federate with THIS hive, plus the store-root step -/// when this hive serves a self-signed chain. Reads -/// local state only (the TLS trust bundle, the store root + the wg key); -/// prints, never mutates. -pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: Option<&str>) { - let self_signed = Path::new(HIVE_TLS_TRUST_BUNDLE_PATH).exists(); - let key = hive_key(domain); - // WireGuard public key, when this hive has a mesh key. Best-effort: - // a missing key or absent `wg` binary just omits the mesh lines. - let wg_pub = std::fs::read(WG_KEY_PATH) - .ok() - .and_then(|k| wg_pubkey(&k).ok()); - - if self_signed { - let store_root = std::fs::read_to_string(STORE_PKI_ROOT_PATH).ok(); - print!("{}", store_root_step(store_root.as_deref())); - println!(); - println!("# 2. paste into the peer hive's NixOS config:"); - } else { - println!("# paste into the peer hive's NixOS config:"); - } - println!("services.hyperhive.swarm.hives.\"{key}\" = {{"); - println!(" domain = \"{domain}\";"); - if let Some(pk) = &wg_pub { - println!(" wireguardPublicKey = \"{pk}\";"); - } - if let Some(addr) = wg_address { - println!(" wireguardAddress = \"{addr}\";"); - } - if let Some(ep) = wg_endpoint { - println!(" wireguardEndpoint = \"{ep}\";"); - } - println!("}};"); - println!("{}", peer_config_key_note()); - println!("# same entry belongs in every hive's config — `hives` is the swarm."); - if !self_signed { - println!( - "# (this hive's cert chains to a public CA — nothing to install; \ - it's trusted by the default bundle.)" - ); - } -} - -/// `wg status` — show the live mesh interface (`wg show wg-hive`), -/// inheriting stdout so the operator sees it directly. -pub(crate) fn wg_status() -> Result<()> { - let status = std::process::Command::new("wg") - .args(["show", WG_INTERFACE]) - .status() - .context("run `wg show` (is wireguard-tools installed?)")?; - if !status.success() { - bail!( - "`wg show {WG_INTERFACE}` failed — is the mesh enabled + up? \ - (services.hyperhive.deploy.wireguard.enable = true, then deploy)" - ); - } - Ok(()) -} - -#[cfg(test)] -mod tests { - use super::{ - STORE_PKI_ROOT_PATH, peer_config_key_note, store_root_step, wg_peer_key_note, - write_key_0400, - }; - - const PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----"; - - /// Both snippets tell the operator to set the deploy option, never the - /// removed `services.hyperhive` spelling. - #[test] - fn key_notes_name_the_deploy_option() { - for note in [wg_peer_key_note(), peer_config_key_note()] { - assert!( - note.contains("services.hyperhive.deploy.hive-controller.hiveName"), - "{note}" - ); - assert!(!note.contains("services.hyperhive.hiveName"), "{note}"); - } - } - - /// The certificate is printed whole, on lines of its own, so a peer - /// operator can copy it out of the output as a PEM file. - #[test] - fn store_root_is_printed_verbatim() { - let s = store_root_step(Some(&format!("{PEM}\n"))); - assert!(s.contains(&format!("\n{PEM}\n")), "{s}"); - } - - /// A hive that has not fetched the root yet says where it will be, - /// rather than printing an empty step. - #[test] - fn missing_store_root_names_the_path_and_the_unit() { - for pem in [None, Some(""), Some("\n")] { - let s = store_root_step(pem); - assert!(s.contains(STORE_PKI_ROOT_PATH), "{s}"); - assert!(s.contains("swarm-services-cert.service"), "{s}"); - assert!(!s.contains("BEGIN CERTIFICATE"), "{s}"); - } - } - - /// The key reaches its path only by a rename from a sibling temp: with - /// the temp path blocked, the write fails and the existing key is left - /// byte-for-byte as it was. Once unblocked, the key lands at `0400` with - /// its bytes intact and no temp file beside it. - #[test] - fn key_is_written_through_a_temp_and_lands_0400() { - use std::os::unix::fs::PermissionsExt as _; - - let dir = std::env::temp_dir().join(format!("hivectl-wg-key-{}", std::process::id())); - std::fs::create_dir_all(&dir).expect("mkdir"); - let path = dir.join("hive.key"); - let tmp = dir.join("hive.key.tmp"); - std::fs::write(&path, "old-key\n").expect("seed"); - std::fs::create_dir(&tmp).expect("block tmp"); - - let blocked = write_key_0400(&path, b"key-bytes\n"); - assert!(blocked.is_err(), "a blocked temp must fail the write"); - assert_eq!(std::fs::read(&path).expect("read seed"), b"old-key\n"); - std::fs::remove_dir(&tmp).expect("unblock tmp"); - - write_key_0400(&path, b"key-bytes\n").expect("write"); - let mode = std::fs::metadata(&path).expect("stat").permissions().mode() & 0o777; - let content = std::fs::read(&path).expect("read"); - let entries: Vec<_> = std::fs::read_dir(&dir) - .expect("readdir") - .map(|e| e.expect("entry").file_name()) - .collect(); - std::fs::remove_dir_all(&dir).ok(); - - assert_eq!(mode, 0o400); - assert_eq!(content, b"key-bytes\n"); - assert_eq!(entries, [std::ffi::OsString::from("hive.key")]); - } -} diff --git a/nix/agent-modules/default.nix b/nix/agent-modules/default.nix index e4a757ecb..0d96a9543 100644 --- a/nix/agent-modules/default.nix +++ b/nix/agent-modules/default.nix @@ -267,8 +267,7 @@ in # nix/packages/default.nix + ./packages.nix) rather than the full # `hyperhive` bundle — that bundle also carries `hivectl` (a # host-admin CLI that dials the *host* admin socket — useless and - # unreachable from inside a container — wrapped with - # `wireguard-tools` for `hivectl wg`). The daemon/harness/MCP bins + # unreachable from inside a container). The daemon/harness/MCP bins # the harness execs (hive-agent{,-mcp}, hive-bash-daemon, # hive-subagent-daemon, hive-matrix-daemon, hive-matrix-mcp) are # wired via their own diff --git a/nix/agent-modules/forge.nix b/nix/agent-modules/forge.nix index a4329249a..ef21e43f1 100644 --- a/nix/agent-modules/forge.nix +++ b/nix/agent-modules/forge.nix @@ -97,7 +97,7 @@ in environment.systemPackages = [ # hive-forge : CLI wrapping common Forgejo REST API operations # (view, pr, issue, comment, assign, close, labels, branches, etc.). - # The per-bin split package — narrow closure, no hivectl/wireguard. + # The per-bin split package — narrow closure, no hivectl. config.services.hyperhive.agent.packages.hive-forge ] ++ lib.optional (config.services.hyperhive.agent.forge.url != null) gitCredHelper; diff --git a/nix/host-modules/default.nix b/nix/host-modules/default.nix index 955ada379..b5ec31f96 100644 --- a/nix/host-modules/default.nix +++ b/nix/host-modules/default.nix @@ -50,7 +50,6 @@ ./glue-nats-auth-bao-identity.nix ./glue-nats-bao-identity.nix ./glue-nats-oidc-client.nix - ./glue-nats-wireguard.nix ./glue-builder-oidc-client.nix ./glue-ci-metrics-otel-oidc-client.nix ./glue-ci-runner-otel-oidc-client.nix @@ -95,7 +94,6 @@ ./swarm-victorialogs.nix ./swarm-victoriametrics-service.nix ./swarm-victoriametrics.nix - ./swarm-wireguard.nix ./swarm.nix ./hive-options-removed.nix ./host-ca-removed.nix diff --git a/nix/host-modules/deploy.nix b/nix/host-modules/deploy.nix index 8b17819d6..d415e158b 100644 --- a/nix/host-modules/deploy.nix +++ b/nix/host-modules/deploy.nix @@ -74,6 +74,14 @@ let The snapshot store was removed; agent-state backups are being rebuilt as swarm-archive. Remove this definition. ''; + + # Shared by the removed options of the WireGuard mesh. + wireguardRemoved = '' + The WireGuard mesh was removed. bao and nats are reached through + their routable listeners (`services.hyperhive.deploy.bao.extraListenAddresses` + with `openFirewall`, and `services.hyperhive.deploy.nats.openFirewall`; both + off by default). Remove this definition. + ''; in { imports = [ @@ -181,6 +189,41 @@ in "snapshotStore" "port" ] snapshotStoreRemoved) + (lib.mkRemovedOptionModule [ + "services" + "hyperhive" + "deploy" + "wireguard" + "enable" + ] wireguardRemoved) + (lib.mkRemovedOptionModule [ + "services" + "hyperhive" + "deploy" + "wireguard" + "privateKeyFile" + ] wireguardRemoved) + (lib.mkRemovedOptionModule [ + "services" + "hyperhive" + "deploy" + "wireguard" + "address" + ] wireguardRemoved) + (lib.mkRemovedOptionModule [ + "services" + "hyperhive" + "deploy" + "wireguard" + "listenPort" + ] wireguardRemoved) + (lib.mkRemovedOptionModule [ + "services" + "hyperhive" + "deploy" + "wireguard" + "persistentKeepalive" + ] wireguardRemoved) # Same type, same meaning, new path — so a rename carries it exactly # and existing configs keep evaluating with one warning naming both diff --git a/nix/host-modules/glue-nats-wireguard.nix b/nix/host-modules/glue-nats-wireguard.nix deleted file mode 100644 index 3423dfb5c..000000000 --- a/nix/host-modules/glue-nats-wireguard.nix +++ /dev/null @@ -1,21 +0,0 @@ -# Glue: where the queue and the wireguard mesh share a host, the queue's port -# is open on the mesh interface. -# -# Interface-scoped, never host-wide: the firewall is default-deny before a -# packet reaches the listener, so without this no other hive can reach the -# queue. Gated on BOTH being here — the queue's own module gate -# (`deploy.nats.enable`) is restated below; a change there must change here. -{ - lib, - config, - ... -}: -let - hyperhiveCfg = config.services.hyperhive; - deployCfg = hyperhiveCfg.deploy; -in -{ - config = lib.mkIf (deployCfg.nats.enable && deployCfg.wireguard.enable) { - networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ hyperhiveCfg.swarm.nats.port ]; - }; -} diff --git a/nix/host-modules/hive-egress.nix b/nix/host-modules/hive-egress.nix index e83e5812b..f1368e27e 100644 --- a/nix/host-modules/hive-egress.nix +++ b/nix/host-modules/hive-egress.nix @@ -40,9 +40,6 @@ let nft = lib.getExe pkgs.nftables; br = cfg.bridgeName; - # The mesh interface (swarm-wireguard.nix): inter-hive traffic is not egress. - meshInterface = "wg-hive"; - # With `refuseEch`, the redirect of port 443 ends at `hive-priv # egress-ech-relay`, which refuses a ClientHello that carries the ECH # extension and sends the rest on to squid's HTTPS intercept port. The relay @@ -143,8 +140,8 @@ let # IPv4 TCP from the agent bridge to ports 80 and 443. `fib daddr oifname` is # the interface the route to the destination leaves by: the bridge (another - # container), `lo` (an address of the host: `fib daddr type local`) and the - # mesh interface are not egress. The redirect rewrites the destination to + # container) and `lo` (an address of the host: `fib daddr type local`) are + # not egress. The redirect rewrites the destination to # the bridge address and the port of squid's intercept listener (for port 443 # with `refuseEch`, of the relay), which reads the original destination from # conntrack. IPv6 is left alone: the agent bridge has no IPv6 addressing. @@ -153,7 +150,7 @@ let type nat hook prerouting priority dstnat; policy accept; iifname "${br}" meta nfproto ipv4 ether saddr @direct_mac return iifname "${br}" meta nfproto ipv4 fib daddr type local return - iifname "${br}" meta nfproto ipv4 fib daddr oifname { "${br}", "${meshInterface}" } return + iifname "${br}" meta nfproto ipv4 fib daddr oifname "${br}" return iifname "${br}" meta nfproto ipv4 tcp dport 80 counter redirect to :${toString egressCfg.intercept.httpPort} iifname "${br}" meta nfproto ipv4 tcp dport 443 counter redirect to :${ toString (if refuseEch then intercept.relayPort else intercept.httpsPort) @@ -171,8 +168,8 @@ let # Priority `filter - 10` runs before the NixOS firewall's forward handling, # which sits at `filter` on both backends (`FORWARD` of the iptables backend, # the `forward` chain of `nixos-fw`). An `accept` here ends this chain only, - # so it allows nothing those refuse; a `reject` ends the packet. The mesh - # and the MACs in `direct_mac` (the unrestricted agents and the service + # so it allows nothing those refuse; a `reject` ends the packet. The MACs + # in `direct_mac` (the unrestricted agents and the service # containers, the traffic the intercept chain leaves alone) come first in # both modes, so a `log` line names what `reject` would refuse. Traffic to # the host's own addresses (squid, dnsmasq, the gateway) is delivered @@ -183,7 +180,6 @@ let chain forward { type filter hook forward priority filter - 10; policy accept; - iifname "${br}" oifname "${meshInterface}" accept ${fromBridge} ether saddr @direct_mac accept ${fromBridge} counter ${lib.concatStringsSep "\n " ( @@ -315,7 +311,7 @@ in default = "log"; description = '' What the `inet hive_egress` table does with traffic the host forwards - from the agent bridge to anywhere but the bridge and the mesh. In `log` and + from the agent bridge to anywhere but the bridge. In `log` and `reject` the traffic of the MACs in the `direct_mac` set (the unrestricted agents and the service containers) is accepted and never logged. diff --git a/nix/host-modules/hive-network.nix b/nix/host-modules/hive-network.nix index a2c7a3070..e4915c9d8 100644 --- a/nix/host-modules/hive-network.nix +++ b/nix/host-modules/hive-network.nix @@ -51,7 +51,6 @@ let d.victoriametrics.enable d.victorialogs.enable d.swarm-secret-publisher.enable - d.wireguard.enable h.gateway.enable h.gateway.dns.enable h.network.enable diff --git a/nix/host-modules/swarm-bao.nix b/nix/host-modules/swarm-bao.nix index 7671854d5..96d030616 100644 --- a/nix/host-modules/swarm-bao.nix +++ b/nix/host-modules/swarm-bao.nix @@ -3113,8 +3113,8 @@ in store is always one of its readers. Every other address depends on which network the reading hives share with this one, and that is a deployment fact no other module's config can be read to infer — a - swarm meshed over wireguard names its mesh address, one on a trusted - LAN names that interface, and an all-local swarm names nothing at all. + swarm on a trusted LAN names that interface, and an all-local swarm + names nothing at all. Addresses only, no port: a store reachable on two ports is a misconfiguration rather than a topology. diff --git a/nix/host-modules/swarm-nats-service.nix b/nix/host-modules/swarm-nats-service.nix index 17541dcc5..567441101 100644 --- a/nix/host-modules/swarm-nats-service.nix +++ b/nix/host-modules/swarm-nats-service.nix @@ -56,8 +56,7 @@ in sits outside hyperhive's claimed ranges (dashboard 7000, forge 3000, every agent in 8100..8999 via FNV-1a hash). - Opened on the bridge interface, for agent containers, and on - `wg-hive` when this host is on the mesh, for other hives. Host-wide + Opened on the bridge interface, for agent containers. Host-wide only when {option}`services.hyperhive.deploy.nats.openFirewall` is set. TLS only: a client that does not speak it is refused. diff --git a/nix/host-modules/swarm-nats.nix b/nix/host-modules/swarm-nats.nix index 1c90122a9..2192a3403 100644 --- a/nix/host-modules/swarm-nats.nix +++ b/nix/host-modules/swarm-nats.nix @@ -522,10 +522,9 @@ in Open {option}`services.hyperhive.swarm.nats.port` in the host firewall on every interface, so hives on other hosts can reach the queue on this host's LAN or public address. Off by default: the port is - otherwise open only on the agent bridge and, when this host is on the - mesh, on `wg-hive`. + otherwise open only on the agent bridge. - ⚠️ This exposes the queue beyond the tunnel. The server binds every + ⚠️ This exposes the queue beyond the agent bridge. The server binds every address, so the firewall is the only thing this option changes. What protects the port is the queue itself: TLS is required, and `auth_callout` refuses every client it cannot identify. Whether that @@ -846,8 +845,8 @@ in # TLS required on the client port: a server with a `tls` # block advertises `tls_required` and refuses a client that # does not upgrade. No `allow_non_tls`, which would keep the - # plaintext path open across the mesh. The 2s timeout is for - # a handshake that crosses the mesh; upstream's 0.5s is sized + # plaintext path open on the network. The 2s timeout is for + # a handshake that crosses the network; upstream's 0.5s is sized # for a LAN. tls = { cert_file = tlsCertPath; diff --git a/nix/host-modules/swarm-wireguard.nix b/nix/host-modules/swarm-wireguard.nix deleted file mode 100644 index bf4027c21..000000000 --- a/nix/host-modules/swarm-wireguard.nix +++ /dev/null @@ -1,152 +0,0 @@ -# The WireGuard inter-hive mesh for the local host. Split out of -# ./swarm.nix because the two are different concerns with different -# audiences: that file declares WHO the peers are (consumed by -# swarm-controller's hive roster and, here, the mesh), while this one -# is plain host networking that a machine which runs no hive can also enable. -# -# The two stay coupled by data, not by structure: the per-peer -# `wireguard*` fields live on the peer submodule in ./swarm.nix, since -# that is where a peer is described, and this module reads them. -# -# Everything declared here is `deploy.*`, not `swarm.*`: by ./deploy.nix's -# rule, `swarm.*` is what a hive needs to be a *client* of the swarm, and -# none of this is. A peer needs this host's `wireguardEndpoint` (the roster -# entry in ./swarm.nix); what interface this host brings up, on which -# address, with which key, is nobody else's business. `listenPort` included -# — it is what this host binds, while the port a peer dials is the one -# inside `wireguardEndpoint`. -{ - lib, - config, - ... -}: -{ - # WireGuard mesh config for the local host. - # When enabled, a `wg-hive` interface connects to all peers that have - # `wireguardPublicKey` declared. Peers reachable over the mesh are - # preferred for inter-hive traffic (no public TLS round-trip needed); - # peers without a public key still work via normal HTTPS. - options.services.hyperhive.deploy.wireguard = { - enable = lib.mkOption { - type = lib.types.bool; - default = false; - description = '' - Enable the WireGuard inter-hive mesh. When true, a `wg-hive` - interface is brought up connecting to all swarm peers that - declare a `wireguardPublicKey`. Requires - `privateKeyFile` to be set. - ''; - }; - - privateKeyFile = lib.mkOption { - type = lib.types.nullOr lib.types.path; - default = null; - example = "/etc/wireguard/hive.key"; - description = '' - Path to the host's WireGuard private key file. The file must - be readable by root and should have mode 0400. Generate with - `wg genkey > /etc/wireguard/hive.key`. Required when - `deploy.wireguard.enable = true`. - ''; - }; - - address = lib.mkOption { - type = lib.types.str; - default = ""; - example = "10.100.0.1/24"; - description = '' - IP address (with prefix) of this host on the WireGuard mesh. - Use a /24 (or broader) prefix so the routing table covers all - peer /32 routes. Example: `"10.100.0.1/24"` for a 256-host mesh. - ''; - }; - - listenPort = lib.mkOption { - type = lib.types.port; - default = 51820; - description = '' - UDP port the local WireGuard interface listens on. Must be - reachable from peer hosts when they initiate the tunnel. - Default: 51820 (standard WireGuard port). - ''; - }; - - persistentKeepalive = lib.mkOption { - type = lib.types.nullOr lib.types.int; - default = 25; - example = 25; - description = '' - Seconds between keepalive packets sent to each peer. Useful - when this host (or a peer) is behind NAT — keeps the UDP hole - open. Set to null to disable. Default: 25 seconds. - ''; - }; - }; - - # Gated on the mesh itself, NOT on `deploy.hive-controller.enable`. The - # mesh is host networking, not a c0re feature: a swarm host that runs no - # hive can join the mesh too. - # Nothing below is c0re-specific; the peer data c0re consumes - # (HIVE_PEER_CA_PATHS) is rendered in ./hive-c0re and stays gated there. - config = lib.mkIf config.services.hyperhive.deploy.wireguard.enable { - assertions = [ - { - assertion = config.services.hyperhive.deploy.wireguard.privateKeyFile != null; - message = '' - services.hyperhive.deploy.wireguard.enable requires - services.hyperhive.deploy.wireguard.privateKeyFile to be set. - Generate a key: wg genkey > /etc/wireguard/hive.key - ''; - } - { - assertion = config.services.hyperhive.deploy.wireguard.address != ""; - message = '' - services.hyperhive.deploy.wireguard.enable requires - services.hyperhive.deploy.wireguard.address to be set - (e.g. "10.100.0.1/24"). - ''; - } - ]; - - # WireGuard inter-hive mesh. Brings up a `wg-hive` interface and - # connects to each peer that has `wireguardPublicKey` set. - networking.wireguard.interfaces = - let - wgCfg = config.services.hyperhive.deploy.wireguard; - # `swarm.hives` minus this host's own hive: a mesh that included - # our own entry would configure a tunnel to ourselves. - meshPeers = lib.filterAttrs ( - name: p: - name != config.services.hyperhive.deploy.hive-controller.hiveName - && p.wireguardPublicKey != null - && p.wireguardAddress != null - ) config.services.hyperhive.swarm.hives; - in - { - wg-hive = { - ips = [ wgCfg.address ]; - listenPort = wgCfg.listenPort; - privateKeyFile = wgCfg.privateKeyFile; - peers = lib.mapAttrsToList ( - _name: p: - { - publicKey = p.wireguardPublicKey; - allowedIPs = [ p.wireguardAddress ]; - } - // lib.optionalAttrs (p.wireguardEndpoint != null) { - endpoint = p.wireguardEndpoint; - } - // lib.optionalAttrs (wgCfg.persistentKeepalive != null) { - persistentKeepalive = wgCfg.persistentKeepalive; - } - ) meshPeers; - }; - }; - - # Open the WireGuard UDP port on the host firewall (host-level - # networking — not inside containers). - networking.firewall.allowedUDPPorts = [ - config.services.hyperhive.deploy.wireguard.listenPort - ]; - }; -} diff --git a/nix/host-modules/swarm.nix b/nix/host-modules/swarm.nix index 7240c8640..8e2f83a0f 100644 --- a/nix/host-modules/swarm.nix +++ b/nix/host-modules/swarm.nix @@ -11,10 +11,7 @@ # detected it. One entry per hive makes that unrepresentable. # # Consumed by swarm-controller's own hive directory (its `/api/hives`, -# swarm-controller.nix) and the mesh in ./swarm-wireguard.nix. The mesh -# lives there rather than here because bringing up an interface is -# host networking rather than swarm bookkeeping, and a host that runs -# no hive still needs it. +# swarm-controller.nix). { lib, config, @@ -27,6 +24,30 @@ let hiveNames = lib.attrNames swarmCfg.hives; + # Per-hive fields of the removed WireGuard mesh. Declared hidden so a + # directory that still sets one reaches the assertion below. A removed + # option cannot be tombstoned with `mkRemovedOptionModule` here: it writes + # top-level `assertions`, which a submodule does not have. + removedMeshFields = [ + "wireguardPublicKey" + "wireguardEndpoint" + "wireguardAddress" + ]; + removedMeshOptions = lib.genAttrs removedMeshFields ( + _: + lib.mkOption { + type = lib.types.nullOr lib.types.raw; + default = null; + visible = false; + } + ); + meshFieldsSet = lib.concatMap ( + name: + map (field: "${name}.${field}") ( + lib.filter (f: swarmCfg.hives.${name}.${f} != null) removedMeshFields + ) + ) hiveNames; + nameGuards = import ./lib/name-guards.nix { inherit lib; }; # Names no hive may BE, and words no hive name may CONTAIN. Both files @@ -166,62 +187,15 @@ in something else. ''; }; - - wireguardPublicKey = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "base64pubkey="; - description = '' - WireGuard public key for this hive's host. Required when - `services.hyperhive.deploy.wireguard.enable = true` and - you want this hive reachable over the mesh. Null = TLS- - only peering (public internet, no mesh tunnel). - ''; - }; - - wireguardEndpoint = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "203.0.113.1:51820"; - description = '' - WireGuard endpoint for this hive in `host:port` form. - Null = this hive has no reachable endpoint, so the tunnel - is initiated from the other side. - - Reads like a fact about the relationship and is not: it - says whether *this* hive can be dialled, which every other - hive in the swarm needs the same answer to. - ''; - }; - - wireguardAddress = lib.mkOption { - type = lib.types.nullOr lib.types.str; - default = null; - example = "10.100.0.2/32"; - description = '' - IP address (with prefix) of this hive's host on the - WireGuard mesh. Used as the `allowedIPs` for its - WireGuard config entry (`./swarm-wireguard.nix`), so - intra-swarm traffic can route over the mesh address - rather than the public domain. Required to include - a hive in the mesh (entries missing this field are - silently excluded from `wg-hive`). - ''; - }; - }; + } + // removedMeshOptions; } ) ); default = { }; example = { - pr1ma = { - wireguardAddress = "10.100.0.1/32"; - wireguardEndpoint = "203.0.113.1:51820"; - }; - edge = { - domain = "edge.elsewhere.example"; - wireguardAddress = "10.100.0.2/32"; - }; + pr1ma = { }; + edge.domain = "edge.elsewhere.example"; }; description = '' Every hive in this swarm, keyed by `hiveName` — **including this @@ -289,6 +263,14 @@ in ); assertions = [ + { + assertion = meshFieldsSet == [ ]; + message = '' + The WireGuard mesh was removed; bao and nats are reached + through their routable listeners. Remove services.hyperhive.swarm.hives..wireguard{PublicKey,Endpoint,Address} + from: ${lib.concatStringsSep ", " meshFieldsSet}. + ''; + } (nameGuards.mustBeIdent { option = "services.hyperhive.swarm.hives"; names = hiveNames; diff --git a/nix/module-eval/core-toggle.nix b/nix/module-eval/core-toggle.nix index 8923033c5..69426c48d 100644 --- a/nix/module-eval/core-toggle.nix +++ b/nix/module-eval/core-toggle.nix @@ -1615,12 +1615,11 @@ let && lib.hasInfix ''iifname "hive-br0" oifname != "hive-br0" ct state new update @log_v4 { ip saddr limit rate'' chain && lib.hasInfix ''iifname "hive-br0" oifname != "hive-br0" ct state new update @log_v6 { ip6 saddr limit rate'' chain && lib.hasInfix ''log prefix "hive-egress-would-drop "'' chain - && lib.hasInfix ''iifname "hive-br0" oifname "wg-hive" accept'' chain - # The mesh comes first, then the exempt MACs (service containers and - # unrestricted agents), then the counter and the logging: a flow of an - # exempt MAC is neither counted nor logged. + # The exempt MACs (service containers and unrestricted agents) come + # first, then the counter and the logging: a flow of an exempt MAC is + # neither counted nor logged. No interface is exempt by name. && lib.hasInfix ''iifname "hive-br0" oifname != "hive-br0" ether saddr @direct_mac accept'' chain - && idx ''oifname "wg-hive" accept'' < idx "ether saddr @direct_mac accept" + && !(lib.hasInfix "wg-hive" text) && idx "ether saddr @direct_mac accept" < idx "counter" && idx "counter" < idx "update @log_v4" # The per-source rules accept after logging, and the unmetered rule @@ -1749,11 +1748,11 @@ let { # `reject` logs and refuses in one rule, as a TCP rule (reset) and a # non-TCP rule (ICMP admin-prohibited) for each of the four log/verdict - # positions, after the mesh and the exempt MACs. The ruleset is read as - # text. The `accept` count (the mesh and the exempt MACs only) and the + # positions, after the exempt MACs. The ruleset is read as + # text. The `accept` count (the exempt MACs only) and the # `bare` host, whose chain has no refusal, are the controls that the # lookups find what is there. - name = "egress.forward reject refuses the bridge's forwarded flows after the mesh and the exempt MACs"; + name = "egress.forward reject refuses the bridge's forwarded flows after the exempt MACs"; ok = let text = lib.head ( @@ -1772,10 +1771,10 @@ let ) egressForwardReject.assertions) && lib.hasInfix "type filter hook forward priority filter - 10; policy accept;" chain && lib.hasInfix "chain input" text - && idx ''oifname "wg-hive" accept'' < idx "ether saddr @direct_mac accept" + && !(lib.hasInfix "wg-hive" text) && idx "ether saddr @direct_mac accept" < idx "counter" && idx "counter" < idx "reject" - && lib.length (lib.filter (lib.hasSuffix " accept") rules) == 2 + && lib.length (lib.filter (lib.hasSuffix " accept") rules) == 1 && lib.length tcp == 4 && lib.length other == 4 && lib.all (lib.hasSuffix " reject with tcp reset") tcp @@ -1818,7 +1817,7 @@ let # The redirect of the agents' web traffic: IPv4 TCP 80 and 443 from the # bridge go to squid's two intercept ports, after the rules that leave # alone the exempt MACs (service containers, unrestricted agents), the - # host's own addresses, the bridge and the mesh. The chain is read as the + # host's own addresses and the bridge. The chain is read as the # lines between its header and the log sets that follow it. The host # without a controller (`egressForwardAccept`) has the set but no chain. name = "the egress table redirects the bridge's web traffic into squid after its exemptions"; @@ -1845,7 +1844,7 @@ let && lib.hasInfix ''oifname != "hive-br0" ether saddr @direct_mac accept'' inet && !(lib.hasInfix "ether saddr @direct_mac accept" without) && idx "fib daddr type local return" < idx "tcp dport 80" - && idx ''fib daddr oifname { "hive-br0", "wg-hive" } return'' < idx "tcp dport 80" + && idx ''fib daddr oifname "hive-br0" return'' < idx "tcp dport 80" && lib.length redirects == 2 && lib.hasInfix "tcp dport 80 counter redirect to :3129" (lib.head redirects) # Port 443 goes to the ClientHello relay, not to squid. @@ -2578,6 +2577,51 @@ let in lib.all (path: refused old path) paths && !(lib.any (path: refused bare path) paths); } + { + # One host defining every removed mesh option, the five deploy paths and + # the three per-hive fields of one directory entry. Each message names + # its own path, so each lookup finds only its option. The control is a + # host that defines none: it has to pass the same assertions, which also + # shows the lookups find a failing assertion when there is one. + name = "a definition of a removed WireGuard mesh option fails evaluation, naming the option and the issue"; + ok = + let + deployPaths = map (o: "services.hyperhive.deploy.wireguard.${o}") [ + "enable" + "privateKeyFile" + "address" + "listenPort" + "persistentKeepalive" + ]; + hiveFields = map (f: "peer.${f}") [ + "wireguardPublicKey" + "wireguardEndpoint" + "wireguardAddress" + ]; + failed = m: lib.filter (a: !a.assertion) m.assertions; + names = + m: path: + lib.any (a: lib.hasInfix path a.message && lib.hasInfix "WireGuard mesh was removed" a.message) ( + failed m + ); + old = hive { + deploy.wireguard = { + enable = true; + privateKeyFile = "/etc/wireguard/hive.key"; + address = "10.100.0.1/24"; + listenPort = 51820; + persistentKeepalive = 25; + }; + swarm.hives.peer = { + wireguardPublicKey = "key"; + wireguardEndpoint = "203.0.113.1:51820"; + wireguardAddress = "10.100.0.2/32"; + }; + }; + in + lib.all (path: names old path) (deployPaths ++ hiveFields) + && !(lib.any (path: names bare path) (deployPaths ++ hiveFields)); + } { # Refused wherever any hyperhive service runs, the hive or a swarm # service alone. The host enabling nothing is the control that it is diff --git a/nix/module-eval/nats-tls.nix b/nix/module-eval/nats-tls.nix index 38fe502ab..e4757aaca 100644 --- a/nix/module-eval/nats-tls.nix +++ b/nix/module-eval/nats-tls.nix @@ -32,13 +32,6 @@ let # are all here, so the scan below reads each of them. inherit (shared) allLocal; - # The same host on the mesh. - allLocalMesh = hive { - deploy.singleHostSwarm = true; - deploy.wireguard.enable = true; - deploy.wireguard.address = "10.100.0.1/24"; - }; - # The queue on a host whose store is elsewhere: no local policy unit. queueNoStore = hive { deploy.nats.enable = true; @@ -275,13 +268,6 @@ let ok = !(natsContainer.services.nats.settings ? host) && !(natsContainer.services.nats.settings ? listen); } - { - name = "4222 is open on wg-hive when this host is on the mesh, and never host-wide"; - ok = - lib.elem 4222 allLocalMesh.networking.firewall.interfaces.wg-hive.allowedTCPPorts - && !(lib.elem 4222 allLocalMesh.networking.firewall.allowedTCPPorts) - && lib.elem 4222 (bridgePorts allLocalMesh); - } { # Off is today's config; the bridge arm is the control that the lookup # finds a port that is there. On adds the host-wide port and keeps the @@ -295,14 +281,6 @@ let && !(lib.elem 4222 allLocal.networking.firewall.allowedTCPPorts) && lib.elem 4222 (bridgePorts allLocal); } - { - name = "4222 is not opened on wg-hive when this host is not on the mesh"; - ok = - !(lib.elem 4222 - (allLocal.networking.firewall.interfaces.wg-hive or { allowedTCPPorts = [ ]; }).allowedTCPPorts - ) - && !(lib.elem 4222 allLocal.networking.firewall.allowedTCPPorts); - } { # Ordering, never a requirement: a policy unit that failed still counts # as done, and the leaf unit's own retries carry it past that. diff --git a/nix/packages/default.nix b/nix/packages/default.nix index c5e38428a..929a7f817 100644 --- a/nix/packages/default.nix +++ b/nix/packages/default.nix @@ -95,22 +95,18 @@ let install -Dm755 ${workspaceBuild}/bin/${bin} $out/bin/${bin} ''; - # Operator CLI — extractor plus shell completions and the `wg` - # wrapper. Suitable for `nix profile install .#hivectl` / + # Operator CLI — the binary plus shell completions. Suitable for + # `nix profile install .#hivectl` / # `environment.systemPackages = [ …packages..hivectl ]` # when the operator only wants the admin CLI. # Completions come from the binary's own `completions ` verb # (the single source of truth, so they never drift from the actual - # verbs). The wireguard-tools wrap makes `hivectl wg` subcommands - # work before `deploy.wireguard.enable` is set (wg init is the very - # first setup step). Completion generation runs before wrapProgram - # since wrapProgram renames the real binary. + # verbs). hivectlPkg = pkgs.runCommand "hivectl" { nativeBuildInputs = [ pkgs.installShellFiles - pkgs.makeWrapper ]; meta = { description = "hyperhive operator CLI"; @@ -123,16 +119,13 @@ let --bash <("$out/bin/hivectl" completions bash) \ --zsh <("$out/bin/hivectl" completions zsh) \ --fish <("$out/bin/hivectl" completions fish) - wrapProgram "$out/bin/hivectl" \ - --prefix PATH : ${pkgs.wireguard-tools}/bin ''; # Per-bin split packages. Agent containers depend on the individual # bins they actually exec/PATH-need (see nix/agent-modules/packages.nix) # instead of the `default` bundle — that keeps `hivectl` (dials the *host* - # admin socket, unreachable from inside a container, drags in - # `wireguard-tools`) and redundant binaries out of every agent's - # closure. + # admin socket, unreachable from inside a container) and redundant + # binaries out of every agent's closure. binPkgs = lib.mapAttrs mkBinPackage daemonBins // { hivectl = hivectlPkg; }; diff --git a/nix/vm-test-typecheck.nix b/nix/vm-test-typecheck.nix index 3acec075c..1903968ef 100644 --- a/nix/vm-test-typecheck.nix +++ b/nix/vm-test-typecheck.nix @@ -132,7 +132,6 @@ let swarm-archive agent-ephemeral-rootfs full-setup - wg-hive semi-spread swarm-builder-offline ; diff --git a/nix/vm-tests.nix b/nix/vm-tests.nix index e246095df..f45ac3e22 100644 --- a/nix/vm-tests.nix +++ b/nix/vm-tests.nix @@ -8,7 +8,6 @@ }: { agent-ephemeral-rootfs = import ./vm-tests/agent-ephemeral-rootfs.nix { inherit pkgs hivePriv; }; - wg-hive = import ./vm-tests/wg-hive.nix { inherit pkgs self; }; hive-squid = import ./vm-tests/hive-squid.nix { inherit pkgs hivePriv; }; hive-egress-antispoof = import ./vm-tests/hive-egress-antispoof.nix { inherit pkgs hivePriv; }; swarm-archive = import ./vm-tests/swarm-archive.nix { inherit pkgs; }; diff --git a/nix/vm-tests/hive-squid.nix b/nix/vm-tests/hive-squid.nix index 6252d39f2..b36b0d82c 100644 --- a/nix/vm-tests/hive-squid.nix +++ b/nix/vm-tests/hive-squid.nix @@ -896,16 +896,6 @@ pkgs.testers.runNixOSTest { "'hive-egress-would-drop IN=eth1 OUT=eth2 .*SRC=fd00:0*98:(0+:)*0*3 DST=fd00:0*99:(0+:)*0*2 .*PROTO=TCP .*DPT=443 '" ) - with checked("a flow to the mesh interface is not logged and one to the uplink is"): - proxy.succeed("ip link add wg-hive type dummy && ip link set wg-hive up") - proxy.succeed("ip route add 198.51.100.0/24 dev wg-hive") - agent.execute("echo x > /dev/udp/198.51.100.7/9; echo x > /dev/udp/203.0.113.77/9") - proxy.wait_until_succeeds( - "journalctl -k -o cat --no-pager | grep -E " - "'hive-egress-would-drop IN=eth1 OUT=eth2 .*DST=203\\.0\\.113\\.77 '" - ) - assert not would_drop(r"DST=198\.51\.100\.7 "), "the mesh interface is not egress" - with checked("one busy agent address is cut to a burst plus a trickle, the rest still leave a trace, and every flow is logged once at most"): agent.execute("for i in $(seq 1 100); do echo x > /dev/udp/203.0.113.78/$((3000 + i)); done") proxy.wait_until_succeeds( @@ -1176,11 +1166,6 @@ pkgs.testers.runNixOSTest { # ---- Reject mode ---- # The node switches to its `reject` specialisation, which reloads the # tables with the refusing forward chain. Nothing is in `direct_mac` yet. - def reject_lines(pattern): - # The reject chain's log lines in the kernel log that match `pattern`. - out = proxy.succeed("journalctl -k -o cat --no-pager | grep hive-egress-reject || true") - return [line for line in out.splitlines() if re.search(pattern, line)] - def wait_reject_line(pattern): proxy.wait_until_succeeds( "journalctl -k -o cat --no-pager | grep -E " @@ -1213,20 +1198,12 @@ pkgs.testers.runNixOSTest { found = re.search(r'oifname != "eth1" counter packets (\d+) bytes \d+$', out, re.M) assert found and int(found.group(1)) > 0, out - with checked("reject mode: a MAC in direct_mac passes and is refused again once it leaves, and the mesh passes"): + with checked("reject mode: a MAC in direct_mac passes and is refused again once it leaves"): proxy.succeed(f"nft add element inet hive_egress direct_mac '{{ {AGENT_MAC} }}'") assert dial_status("203.0.113.2", 8443) == 0, "an exempt MAC (a service container's stand-in) reaches TCP 8443" agent.succeed("ping -c1 -W3 203.0.113.2") assert udp_status("203.0.113.77") == 0, "control: an exempt MAC's datagram is not refused" proxy.succeed(f"nft delete element inet hive_egress direct_mac '{{ {AGENT_MAC} }}'") assert dial_status("203.0.113.2", 8443) == 1, "refused again" - # The mesh comes first in the chain. A dummy interface answers nothing, - # so the proof is that the chain refuses neither the ping nor the UDP - # datagram, while the same flows to the uplink are refused. - status, out = agent.execute("ping -c1 -W5 198.51.100.7") - assert "filtered" not in out.lower(), f"the mesh interface is not egress: {out}" - agent.execute("echo x > /dev/udp/198.51.100.7/9; echo x > /dev/udp/203.0.113.80/9") - wait_reject_line(r"DST=203\.0\.113\.80 ") - assert not reject_lines(r"DST=198\.51\.100\.7 "), "the mesh interface is not egress" ''; } diff --git a/nix/vm-tests/wg-hive.nix b/nix/vm-tests/wg-hive.nix deleted file mode 100644 index 44b3c44d4..000000000 --- a/nix/vm-tests/wg-hive.nix +++ /dev/null @@ -1,136 +0,0 @@ -# `vmTests..wg-hive`: two hives peered over `wg-hive`, with real traffic across the tunnel. -# Unlike ../module-eval, which only reads the rendered `networking.wireguard` -# options, this boots both hosts and sends packets. -# -# Keys are derived from a fixed per-host seed, so every store builds the -# same bytes and nothing is committed. The module takes peer public keys -# as eval-time strings (`swarm.hives..wireguardPublicKey`), so they -# are read back from the derivation's output. -{ - pkgs, - self, -}: -let - mkKey = - name: - pkgs.runCommand "wg-hive-test-key-${name}" - { - nativeBuildInputs = [ - pkgs.wireguard-tools - pkgs.openssl - ]; - } - '' - # Test-only key: base64 of sha256(seed); wg clamps the scalar itself. - mkdir $out - printf '%s' 'hyperhive-wg-hive-test-${name}' | openssl dgst -sha256 -binary | base64 > $out/private - wg pubkey < $out/private > $out/public - ''; - keys = { - h1 = mkKey "h1"; - h2 = mkKey "h2"; - # Never installed on any node; stands in for a peer key that does not - # match the real peer's private key. - wrong = mkKey "wrong"; - }; - pub = name: pkgs.lib.removeSuffix "\n" (builtins.readFile "${keys.${name}}/public"); - - # The test driver numbers nodes on the test vlan in alphabetical order. - lanAddr = { - h1 = "192.168.1.1"; - h2 = "192.168.1.2"; - }; - meshAddr = { - h1 = "10.100.0.1"; - h2 = "10.100.0.2"; - }; - - swarmHives = pkgs.lib.mapAttrs (name: _: { - domain = "${name}.t.local"; - wireguardPublicKey = pub name; - wireguardAddress = "${meshAddr.${name}}/32"; - wireguardEndpoint = "${lanAddr.${name}}:51820"; - }) lanAddr; - - node = name: { - imports = [ self.nixosModules.default ]; - services.hyperhive = { - swarm.domain = "t.local"; - swarm.hives = swarmHives; - deploy.hive-controller.hiveName = name; - deploy.wireguard = { - enable = true; - address = "${meshAddr.${name}}/24"; - privateKeyFile = "${keys.${name}}/private"; - persistentKeepalive = 1; - }; - }; - # The listener the TCP check connects to, open on the mesh only. - networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ 9000 ]; - environment.systemPackages = [ - pkgs.wireguard-tools - pkgs.python3 - pkgs.curl - ]; - }; -in -pkgs.testers.runNixOSTest { - name = "wg-hive"; - - nodes = { - h1 = node "h1"; - h2 = node "h2"; - }; - - testScript = '' - def wg_state(m): - return m.execute("wg show wg-hive; ip route; systemctl status 'wireguard-wg-hive*' --no-pager")[1] - - start_all() - for m, peer_key in ((h1, "${pub "h2"}"), (h2, "${pub "h1"}")): - m.wait_for_unit("wireguard-wg-hive.service") - m.succeed("ip link show wg-hive") - # The interface unit is active before its peers are installed: each - # peer has its own unit, which runs after network-online.target. Until - # then the interface has no peer and a send fails with ENOKEY. - m.wait_until_succeeds(f"wg show wg-hive allowed-ips | grep -F {peer_key}") - - # Traffic makes the tunnel handshake. - for src, addr in ((h1, "${meshAddr.h2}"), (h2, "${meshAddr.h1}")): - status, out = src.execute(f"ping -c 3 -W 5 {addr}") - assert status == 0, f"ping {addr} failed:\n{out}\n{wg_state(src)}" - for m in (h1, h2): - stamps = m.succeed("wg show wg-hive latest-handshakes").split() - assert len(stamps) == 2 and int(stamps[1]) > 0, f"no handshake: {stamps}\n{wg_state(m)}" - - # The peer's mesh address is routed through the tunnel, its plain - # address is not. - via_mesh = h1.succeed("ip route get ${meshAddr.h2}") - assert "dev wg-hive" in via_mesh, via_mesh - via_lan = h1.succeed("ip route get ${lanAddr.h2}") - assert "dev wg-hive" not in via_lan, via_lan - - # A TCP connect to a listener on the peer's mesh address, and the - # tunnel's own counters moved. - h2.succeed( - "systemd-run --unit=wg-listener ${pkgs.python3}/bin/python3" - " -m http.server 9000 --bind ${meshAddr.h2}" - ) - h2.wait_for_open_port(9000, "${meshAddr.h2}") - h1.succeed("${pkgs.curl}/bin/curl -sf --max-time 5 http://${meshAddr.h2}:9000/") - transfer = h1.succeed("wg show wg-hive transfer").split() - assert int(transfer[1]) > 0 and int(transfer[2]) > 0, transfer - - # Negative control: h2 swaps h1's public key for one that does not match - # h1's private key. Same endpoint, same address, so the only difference - # from the working peer above is the key, and no handshake may happen. - h2.succeed("wg set wg-hive peer ${pub "h1"} remove") - h2.succeed( - "wg set wg-hive peer ${pub "wrong"} allowed-ips ${meshAddr.h1}/32" - " endpoint ${lanAddr.h1}:51820 persistent-keepalive 1" - ) - h2.fail("ping -c 3 -W 2 ${meshAddr.h1}") - stamps = h2.succeed("wg show wg-hive latest-handshakes").split() - assert stamps == ["${pub "wrong"}", "0"], f"handshake with wrong key: {stamps}" - ''; -} diff --git a/swarm-controller/src/forge/pull_diff.rs b/swarm-controller/src/forge/pull_diff.rs index 79b397eae..9f8dd3300 100644 --- a/swarm-controller/src/forge/pull_diff.rs +++ b/swarm-controller/src/forge/pull_diff.rs @@ -170,7 +170,11 @@ mod tests { deployCfg.wireguard.enable) {\n networking.firewall.interfaces.wg0.\ allowedTCPPorts = [ 4222 ];\n };\n}\n"; vec![ - change("nix/host-modules/glue-nats-wireguard.nix", None, Some(glue)), + change( + "nix/host-modules/glue-nats-oidc-client.nix", + None, + Some(glue), + ), change( "nix/host-modules/swarm-nats.nix", Some(&module(20, true)),