Watch
0
0
Fork
You've already forked hyperhive
0

remove the swarm wireguard mesh; tombstone its options

This commit is contained in:
atlas 2026-11-15 02:17:44 +01:00
commit adc61c9043
34 changed files with 330 additions and 1068 deletions

View file

@ -55,8 +55,7 @@ the swarm — one of each, each on whichever host you choose
├── swarm-grafana dashboards over both stores
│
└── opt-in, not part of singleHostSwarm
├── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner)
└── wg-hive WireGuard mesh between hives (deploy.wireguard)
└── swarm-ci-runner Forgejo Actions runner (deploy.ci-runner)
the hives — the substrate, as many as you like (deploy.hive-controller.enable)
│

View file

@ -23,10 +23,6 @@ This document contains the help content for the `hivectl` command-line program.
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
* [`hivectl list-agents`↴](#hivectl-list-agents)
* [`hivectl wg`↴](#hivectl-wg)
* [`hivectl wg init`↴](#hivectl-wg-init)
* [`hivectl wg peer`↴](#hivectl-wg-peer)
* [`hivectl wg status`↴](#hivectl-wg-status)
* [`hivectl peer-config`↴](#hivectl-peer-config)
* [`hivectl stop`↴](#hivectl-stop)
* [`hivectl start`↴](#hivectl-start)
@ -43,7 +39,6 @@ Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
* `list-agents` — Show all managed agents with their status and technical state
* `wg` — WireGuard inter-hive mesh setup helpers
* `peer-config` — Generate the federation peer-config block for THIS hive
* `stop` — Stop containers hive-wide in one operator action
* `start` — Start containers hive-wide — the inverse of `hivectl stop`
@ -290,72 +285,13 @@ Global — not scoped to one agent, so it lives at the top level rather than und
## `hivectl wg`
WireGuard inter-hive mesh setup helpers.
Generate this hive's mesh key and print the nix to enable the mesh, add a peer, or inspect live interface state.
**Usage:** `hivectl wg <COMMAND>`
###### **Subcommands:**
* `init` — Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
* `peer` — Print the nix to add a peer hive to the mesh
* `status` — Show the live mesh interface state
## `hivectl wg init`
Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
**Usage:** `hivectl wg init [OPTIONS]`
###### **Options:**
* `--address <ADDRESS>` — This hive's mesh address (for example `10.42.0.1/32`) to bake into the printed snippet. Omit to get a placeholder you fill in
## `hivectl wg peer`
Print the nix to add a peer hive to the mesh
**Usage:** `hivectl wg peer [OPTIONS] --pubkey <PUBKEY> --address <ADDRESS> <DOMAIN>`
###### **Arguments:**
* `<DOMAIN>` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
###### **Options:**
* `--pubkey <PUBKEY>` — Peer's WireGuard public key (from its `hivectl wg init`)
* `--address <ADDRESS>` — Peer's mesh address (for example `10.42.0.2/32`)
* `--endpoint <ENDPOINT>` — Peer's `host:port` endpoint (omit for a peer that only dials out, for example one behind NAT — it must set an endpoint pointing back here)
## `hivectl wg status`
Show the live mesh interface state
**Usage:** `hivectl wg status`
## `hivectl peer-config`
Generate the federation peer-config block for THIS hive.
Prints the nix a peer operator pastes into their swarm config to trust and reach this hive.
**Usage:** `hivectl peer-config [OPTIONS]`
###### **Options:**
* `--wg-address <WG_ADDRESS>` — This hive's WireGuard mesh address (for example `10.42.0.1/32`), emitted as `wireguardAddress`. Omit when not running the mesh
* `--wg-endpoint <WG_ENDPOINT>` — This hive's public WireGuard endpoint (`host:port`), emitted as `wireguardEndpoint`. Omit when peers dial in / no mesh
**Usage:** `hivectl peer-config`

View file

@ -126,9 +126,8 @@ if you want it. What that grant lets an agent reach on a hive is on the
Each hive carries an identity (`hiveName`, the `swarm.hives` directory) that
says which hive it's and how to reach the others; the directory feeds the
swarm UI's hive list, matrix federation and the optional WireGuard mesh. →
[the hive directory](hive-directory.md). A private link between hives instead
of public HTTPS: [WireGuard mesh](wireguard.md).
swarm UI's hive list and matrix federation. →
[the hive directory](hive-directory.md).
## Cross-references

View file

@ -95,18 +95,14 @@ mechanism.
swarm's root needs a CA-issued certificate (ACME). Federation firewall
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
to configure `wg-hive`. See [the WireGuard page](wireguard.md).
## Reaching the queue and the store on a routable address
The mesh isn't the only way to reach the queue and the secret store from
another host. Set `deploy.nats.openFirewall` on the queue's host to open the
queue's port on every interface, and `deploy.bao.openFirewall` together with
Other hives reach the queue and the secret store on a routable address. Set
`deploy.nats.openFirewall` on the queue's host to open the queue's port on
every interface, and `deploy.bao.openFirewall` together with
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
on the addresses it listens on. Both default to off. Each exposes a service
beyond the tunnel, which is the operator's decision. The queue requires TLS and
beyond the agent bridge, which is the operator's decision. The queue requires TLS and
its `auth_callout` refuses every client it can't identify. The store requires a
client certificate on every listener, and evaluation fails when you
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point

View file

@ -1,80 +0,0 @@
# WireGuard mesh between hives (optional)
The peer config in [the hive directory](hive-directory.md#hive-identity-config) uses public HTTPS for all inter-hive traffic.
For private deployments — or to reduce latency and TLS overhead on
intra-swarm traffic — hyperhive can configure a host-to-host
WireGuard mesh.
## Generating keys
On each hive host:
```bash
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
```
## Config example (two hives)
```nix
# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
# The same `hives` attrset both hosts hold — mesh fields included,
# since "where this hive can be dialled" is a fact about that hive.
swarm.hives = {
firsthive = {
domain = "firsthive.myswarm.example";
wireguardPublicKey = "base64keyA=";
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
secondhive = {
domain = "secondhive.elsewhere.example";
wireguardPublicKey = "base64keyB=";
wireguardEndpoint = "203.0.113.42:51820";
wireguardAddress = "10.100.0.2/32";
};
};
};
# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
services.hyperhive = {
deploy.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.hives = {
# … identical to hive A's …
};
};
```
## What the mesh does
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
host (not inside agent containers; containers reach peers via the
host's routing table).
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
- It reads each entry's `wireguardAddress` from `services.hyperhive.swarm.hives`,
minus this host's own hive, to build `wg-hive`'s `allowedIPs`, so intra-swarm
traffic can route over the mesh address rather than the public domain.
- It sets `persistentKeepalive = 25` by default; override or null to
disable (not needed when both sides have public IPs and no NAT).
## NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null `wireguardEndpoint` on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, you need a STUN relay or a third host
(exit node); hyperhive sets up neither.

View file

@ -29,7 +29,6 @@ that touches it.
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
│ │
│ hive-br0 10.42.0.1/24 │
│ ┌──────────┼──────────────┐ │
@ -72,8 +71,8 @@ The flows, end to end:
UI unix socket.
- **Internet egress** — agent default route points at the bridge IP;
the host forwards + masquerades out its uplink.
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
and reach each other's gateway/forge across it
- **Swarm** — peer hives reach each other's gateway and forge over HTTPS, and
the queue and the secret store on their routable listeners
([`docs/2-handbook/swarm/`](../../2-handbook/swarm/README.md)).
- **Control plane (no network)** — per-agent broker socket
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
@ -526,7 +525,7 @@ squid (`services.hyperhive.deploy.hive-controller.egress.intercept.enable`).
- **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
a flow to an address that stays on the bridge, to one of the host's own
addresses or to the `wg-hive` interface, and a flow from a MAC in the set
addresses, and a flow from a MAC in the set
`direct_mac`. That set holds the MACs of the ports of every container that's
not an agent (the swarm's service containers) and of the agents listed in
`services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So
@ -727,13 +726,13 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
the `inet hive_egress` table does with traffic the host routes from the agent
bridge to anywhere but the bridge and the `wg-hive` mesh interface. It takes
bridge to anywhere but the bridge. It takes
`accept`, `log` (the default) or `reject`. In `log` mode the chain blocks
nothing. It logs each new flow and counts every packet. In `reject` mode it
also refuses the flow.
- **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the
mesh traffic first, then the traffic of every MAC in the `direct_mac` set: the
traffic of every MAC in the `direct_mac` set: the
`unrestricted` agents and the service containers, which the `intercept` chain
leaves alone too (see _Intercepted web traffic_). They leave no line, so a
`hive-egress-would-drop` line is exactly a flow that `reject` would refuse,

View file

@ -128,9 +128,7 @@ by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
workflow, optional `test` input), or run
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives
peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that
must not handshake).
root, through the real hive-priv unit) and `hive-squid`.
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
Nix interpolation. Commands the script runs in the VM find their binaries on the

View file

@ -72,9 +72,9 @@ The queue URL is the same string on every hive. The queue accepts TLS only,
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
no other name or address, so a URL with an IP address or `nats://` fails.
The queue's host resolves the name itself. **A multi-host swarm needs one
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's routable
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
on `wg-hive` when that host is on the mesh.
host-wide only with `deploy.nats.openFirewall`.
The queue's host asks the secret store for that certificate under a store
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the

View file

@ -148,7 +148,6 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-hand
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |

View file

@ -3,7 +3,7 @@
The operator-facing host CLI. A thin client for the `hive-c0re`
daemon — speaks the host admin socket protocol (`hive-host-sock`) and
does not link the daemon crate. Container lifecycle and the
gateway verbs all forward to the daemon and need it running; a few (`wg`/`peer-config`, `choom`) work off local
gateway verbs all forward to the daemon and need it running; a few (`peer-config`, `choom`) work off local
host state instead.
## When to use it
@ -25,7 +25,7 @@ dispatch:
- **`power.rs`** — restart/start/stop at the container level.
- **`choom.rs`** — drop into an interactive claude session in a
container.
- **`wg.rs`** — WireGuard mesh helpers.
- **`peer_config.rs`** — the federation block a peer pastes.
- **`subvol.rs`** — btrfs state-subvolume ops.
- **`quota.rs`**, **`util.rs`**, **`completions.rs`** — shared helpers
and shell completions.

View file

@ -51,28 +51,11 @@ pub enum Cmd {
#[arg(long)]
json: bool,
},
/// WireGuard inter-hive mesh setup helpers.
///
/// Generate this hive's mesh key and print the nix to enable the mesh,
/// add a peer, or inspect live interface state.
Wg {
#[command(subcommand)]
cmd: WgCmd,
},
/// Generate the federation peer-config block for THIS hive.
///
/// Prints the nix a peer operator pastes into their swarm config to
/// trust and reach this hive.
PeerConfig {
/// This hive's WireGuard mesh address (for example `10.42.0.1/32`),
/// emitted as `wireguardAddress`. Omit when not running the mesh.
#[arg(long)]
wg_address: Option<String>,
/// This hive's public WireGuard endpoint (`host:port`), emitted as
/// `wireguardEndpoint`. Omit when peers dial in / no mesh.
#[arg(long)]
wg_endpoint: Option<String>,
},
PeerConfig,
/// Stop containers hive-wide in one operator action.
///
/// Bare `hivectl stop` stops every agent and the gateway; scope flags
@ -164,35 +147,6 @@ impl ScopeArgs {
}
}
#[derive(Subcommand)]
pub enum WgCmd {
/// Generate this hive's WireGuard key (if absent) and print its public
/// key plus the nix to enable the mesh.
Init {
/// This hive's mesh address (for example `10.42.0.1/32`) to bake into the
/// printed snippet. Omit to get a placeholder you fill in.
#[arg(long)]
address: Option<String>,
},
/// Print the nix to add a peer hive to the mesh.
Peer {
/// Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
domain: String,
/// Peer's WireGuard public key (from its `hivectl wg init`).
#[arg(long)]
pubkey: String,
/// Peer's mesh address (for example `10.42.0.2/32`).
#[arg(long)]
address: String,
/// Peer's `host:port` endpoint (omit for a peer that only dials out,
/// for example one behind NAT — it must set an endpoint pointing back here).
#[arg(long)]
endpoint: Option<String>,
},
/// Show the live mesh interface state.
Status,
}
// Default host admin socket path. Shared with `hive-c0re`'s `main.rs`
// default via `hive_host_sock::HOST_SOCKET` — the daemon binds there
// and `hivectl` connects to it.

View file

@ -7,7 +7,7 @@
//! `gateway`) all forward to the daemon, which owns the broker,
//! the credentials, and the provisioning logic — a running daemon is
//! required for those. A couple of verbs work off local host state
//! directly instead (`wg` / `peer-config` read the mesh key + TLS CA), so
//! directly instead (`peer-config` reads the TLS CA), so
//! they don't need the socket. `agent <name> choom` execs into a container
//! rather than asking the daemon to do anything, but still uses the socket
//! for its "is this an agent?" pre-flight — that answer lives in a
@ -26,12 +26,12 @@ mod client;
/// Rebuild-queue node progress rendering (`wait_for_nodes` + the spinner /
/// plain renderers), split out to keep this file manageable.
mod dag_progress;
use cli::{Cli, Cmd, WgCmd};
use cli::{Cli, Cmd};
mod completions;
mod util;
use completions::generate_completions;
mod wg;
use wg::{peer_config, require_hive_domain, wg_init, wg_peer, wg_status};
mod peer_config;
use peer_config::{peer_config, require_hive_domain};
mod agents;
mod choom;
mod watch;
@ -53,25 +53,9 @@ async fn main() -> Result<()> {
match cli.cmd {
Cmd::Agent { name, cmd } => run_agent(&socket, &name, cmd).await,
Cmd::ListAgents { json } => agents_list(&socket, json).await,
Cmd::Wg { cmd } => match cmd {
WgCmd::Init { address } => wg_init(&socket, address.as_deref()).await,
WgCmd::Peer {
domain,
pubkey,
address,
endpoint,
} => {
wg_peer(&domain, &pubkey, &address, endpoint.as_deref());
Ok(())
}
WgCmd::Status => wg_status(),
},
Cmd::PeerConfig {
wg_address,
wg_endpoint,
} => {
Cmd::PeerConfig => {
let domain = require_hive_domain(&socket).await?;
peer_config(&domain, wg_address.as_deref(), wg_endpoint.as_deref());
peer_config(&domain);
Ok(())
}
Cmd::Stop {

143
hivectl/src/peer_config.rs Normal file
View file

@ -0,0 +1,143 @@
//! `hivectl peer-config` — print the block a peer operator pastes to
//! federate with this hive.
use std::path::Path;
use anyhow::{Context as _, Result};
use crate::util::query_hive_urls;
/// Host path of this hive's TLS trust bundle (matches the
/// `services.hyperhive.deploy.hive-controller.tls.stateDir` default in
/// hive-tls.nix). Its existence means the gateway is in self-signed mode,
/// serving a leaf the secret store issued, so a federating peer needs an
/// anchor for it. Absent = ACME / operator cert, trusted by the default CA
/// bundle with nothing to distribute.
const HIVE_TLS_TRUST_BUNDLE_PATH: &str = "/var/lib/hive-tls/trust-bundle.pem";
/// Host path of the secret store's `pki` root (hive-tls.nix's
/// `$svcroot`, folded into the trust bundle). The anchor a whole swarm
/// shares: every hive's gateway leaf and the swarm's service names are
/// issued under it, so a peer host installs it once and every present
/// *and future* hive validates.
const STORE_PKI_ROOT_PATH: &str = "/var/lib/hive-tls/swarm-services-root.pem";
/// The option whose value keys a hive's entry in `services.hyperhive.swarm.hives`.
const HIVE_NAME_OPTION: &str = "services.hyperhive.deploy.hive-controller.hiveName";
/// Attrset key for a hive in `services.hyperhive.swarm.hives`, derived
/// from its domain's first DNS label.
///
/// A hive occupies `<hiveName>.<swarm.domain>`, so the first label *is*
/// the hive name in any deployment that hasn't overridden `domain` by
/// hand. Where it has, a wrong key fails loudly rather than quietly: on
/// that hive's own host the `hives.<hiveName>` assertion fires, because
/// the directory is supposed to be the same attrset everywhere. The
/// snippet below says so rather than presenting the guess as fact.
fn hive_key(domain: &str) -> &str {
domain.split('.').next().unwrap_or(domain)
}
/// Require this hive's domain from the daemon for snippet generation.
/// Errors with a clear hint when it can't be resolved, so `peer-config`
/// never silently emits a wrong key. An unreachable daemon propagates its
/// own (classified) connect error; only a *reachable* daemon with no domain
/// gets the config hint.
pub(crate) async fn require_hive_domain(socket: &Path) -> Result<String> {
query_hive_urls(socket)
.await
.context("could not determine this hive's domain from the daemon")?
.and_then(|u| u.domain)
.context("the daemon reported no domain — set `services.hyperhive.domain`")
}
/// The first closing comment line of the `peer-config` snippet.
fn peer_config_key_note() -> String {
format!("# the key must be this hive's {HIVE_NAME_OPTION}, and the")
}
/// The `peer-config` step carrying the secret store's `pki` root: the
/// certificate itself, or where it will appear when this hive has not
/// fetched it yet.
fn store_root_step(pem: Option<&str>) -> String {
const HEAD: &str = "# 1. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\
# every hive's gateway leaf and the swarm's service names are issued under it:\n";
match pem.map(str::trim_end).filter(|p| !p.is_empty()) {
Some(p) => format!(
"{HEAD}{p}\n\
# (a self-signed hive in this swarm fetches it itself, into {STORE_PKI_ROOT_PATH})\n"
),
None => format!(
"{HEAD}# not on this host yet: swarm-services-cert.service writes it to\n\
# {STORE_PKI_ROOT_PATH}; rerun once that unit has succeeded\n"
),
}
}
/// `peer-config` — print the `swarm.hives."<name>"` block a peer
/// operator pastes to federate with THIS hive, plus the store-root step
/// when this hive serves a self-signed chain. Reads local state only (the
/// TLS trust bundle and the store root); prints, never mutates.
pub(crate) fn peer_config(domain: &str) {
let self_signed = Path::new(HIVE_TLS_TRUST_BUNDLE_PATH).exists();
let key = hive_key(domain);
if self_signed {
let store_root = std::fs::read_to_string(STORE_PKI_ROOT_PATH).ok();
print!("{}", store_root_step(store_root.as_deref()));
println!();
println!("# 2. paste into the peer hive's NixOS config:");
} else {
println!("# paste into the peer hive's NixOS config:");
}
println!("services.hyperhive.swarm.hives.\"{key}\" = {{");
println!(" domain = \"{domain}\";");
println!("}};");
println!("{}", peer_config_key_note());
println!("# same entry belongs in every hive's config — `hives` is the swarm.");
if !self_signed {
println!(
"# (this hive's cert chains to a public CA — nothing to install; \
it's trusted by the default bundle.)"
);
}
}
#[cfg(test)]
mod tests {
use super::{STORE_PKI_ROOT_PATH, peer_config_key_note, store_root_step};
const PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----";
/// The snippet tells the operator to set the deploy option, never the
/// removed `services.hyperhive` spelling.
#[test]
fn key_note_names_the_deploy_option() {
let note = peer_config_key_note();
assert!(
note.contains("services.hyperhive.deploy.hive-controller.hiveName"),
"{note}"
);
assert!(!note.contains("services.hyperhive.hiveName"), "{note}");
}
/// The certificate is printed whole, on lines of its own, so a peer
/// operator can copy it out of the output as a PEM file.
#[test]
fn store_root_is_printed_verbatim() {
let s = store_root_step(Some(&format!("{PEM}\n")));
assert!(s.contains(&format!("\n{PEM}\n")), "{s}");
}
/// A hive that has not fetched the root yet says where it will be,
/// rather than printing an empty step.
#[test]
fn missing_store_root_names_the_path_and_the_unit() {
for pem in [None, Some(""), Some("\n")] {
let s = store_root_step(pem);
assert!(s.contains(STORE_PKI_ROOT_PATH), "{s}");
assert!(s.contains("swarm-services-cert.service"), "{s}");
assert!(!s.contains("BEGIN CERTIFICATE"), "{s}");
}
}
}

View file

@ -1,365 +0,0 @@
//! `hivectl wg` / `peer-config` — WireGuard mesh helpers: generate this
//! hive's key + the nix snippets to enable the mesh, add a peer, print the
//! block a peer pastes to federate with us, and show the live interface.
use std::path::Path;
use anyhow::{Context as _, Result, bail};
use crate::util::query_hive_urls;
/// Host path of this hive's WireGuard private key (matches the
/// `privateKeyFile` example in the deploy.wireguard nix options).
const WG_KEY_PATH: &str = "/etc/wireguard/hive.key";
/// The mesh interface name hive-c0re's nix module brings up.
const WG_INTERFACE: &str = "wg-hive";
/// Host path of this hive's TLS trust bundle (matches the
/// `services.hyperhive.deploy.hive-controller.tls.stateDir` default in
/// hive-tls.nix). Its existence means the gateway is in self-signed mode,
/// serving a leaf the secret store issued, so a federating peer needs an
/// anchor for it. Absent = ACME / operator cert, trusted by the default CA
/// bundle with nothing to distribute.
const HIVE_TLS_TRUST_BUNDLE_PATH: &str = "/var/lib/hive-tls/trust-bundle.pem";
/// Host path of the secret store's `pki` root (hive-tls.nix's
/// `$svcroot`, folded into the trust bundle). The anchor a whole swarm
/// shares: every hive's gateway leaf and the swarm's service names are
/// issued under it, so a peer host installs it once and every present
/// *and future* hive validates.
const STORE_PKI_ROOT_PATH: &str = "/var/lib/hive-tls/swarm-services-root.pem";
/// The option whose value keys a hive's entry in `services.hyperhive.swarm.hives`.
const HIVE_NAME_OPTION: &str = "services.hyperhive.deploy.hive-controller.hiveName";
/// Attrset key for a hive in `services.hyperhive.swarm.hives`, derived
/// from its domain's first DNS label.
///
/// A hive occupies `<hiveName>.<swarm.domain>`, so the first label *is*
/// the hive name in any deployment that hasn't overridden `domain` by
/// hand. Where it has, a wrong key fails loudly rather than quietly: on
/// that hive's own host the `hives.<hiveName>` assertion fires, because
/// the directory is supposed to be the same attrset everywhere. The
/// snippets below say so rather than presenting the guess as fact.
fn hive_key(domain: &str) -> &str {
domain.split('.').next().unwrap_or(domain)
}
/// Best-effort query for this hive's domain from the running daemon
/// (`HostRequest::Urls`, which reads `HYPERHIVE_HIVE_DOMAIN` from c0re's
/// service env). `None` when the daemon is unreachable or the domain is
/// unset — for the callers that treat both as "skip the optional block".
/// Use [`require_hive_domain`] where the distinction matters: it keeps the
/// connect error (and its hint) instead of flattening it away.
async fn query_hive_domain(socket: &Path) -> Option<String> {
query_hive_urls(socket).await.ok().flatten()?.domain
}
/// Require this hive's domain from the daemon for snippet generation.
/// Errors with a clear hint when it can't be resolved, so `peer-config`
/// never silently emits a wrong key. An unreachable daemon propagates its
/// own (classified) connect error; only a *reachable* daemon with no domain
/// gets the config hint.
pub(crate) async fn require_hive_domain(socket: &Path) -> Result<String> {
query_hive_urls(socket)
.await
.context("could not determine this hive's domain from the daemon")?
.and_then(|u| u.domain)
.context("the daemon reported no domain — set `services.hyperhive.domain`")
}
/// `wg init` — generate (if absent) the hive's WireGuard key, print its
/// public key + the nix snippet to enable the mesh, then (best-effort)
/// the `peer-config` block peers paste to federate with this hive, so a
/// fresh setup is one command. The domain comes from the daemon; if it
/// can't be resolved, the peer block is skipped (init still succeeds —
/// its core job is enabling the mesh locally).
pub(crate) async fn wg_init(socket: &Path, address: Option<&str>) -> Result<()> {
use std::os::unix::fs::PermissionsExt as _;
let key_path = Path::new(WG_KEY_PATH);
if key_path.exists() {
println!(
"WireGuard private key already present at {WG_KEY_PATH} — reusing (not regenerating)."
);
} else {
if let Some(parent) = key_path.parent() {
std::fs::create_dir_all(parent)
.with_context(|| format!("create {}", parent.display()))?;
std::fs::set_permissions(parent, std::fs::Permissions::from_mode(0o700)).ok();
}
let out = std::process::Command::new("wg")
.arg("genkey")
.output()
.context("run `wg genkey` (is wireguard-tools installed?)")?;
if !out.status.success() {
bail!(
"wg genkey failed: {}",
String::from_utf8_lossy(&out.stderr).trim()
);
}
write_key_0400(key_path, &out.stdout)?;
println!("Generated WireGuard private key at {WG_KEY_PATH} (0400).");
}
let privkey =
std::fs::read(key_path).with_context(|| format!("read {}", key_path.display()))?;
let pubkey = wg_pubkey(&privkey)?;
let addr = address.unwrap_or("<MESH_ADDRESS e.g. 10.42.0.1/32>");
println!("\nPublic key (share this with peer hives — they pass it to `hivectl wg peer`):");
println!(" {pubkey}");
println!("\nAdd to this hive's NixOS config:");
println!(" services.hyperhive.deploy.wireguard = {{");
println!(" enable = true;");
println!(" privateKeyFile = \"{WG_KEY_PATH}\";");
println!(" address = \"{addr}\";");
println!(" # listenPort = 51820; # default");
println!(" }};");
// Also print the block a peer pastes to federate with us (CA + this
// mesh key) — one-stop setup. Domain comes from the daemon;
// best-effort, so init still succeeds when it can't be resolved.
if let Some(d) = query_hive_domain(socket).await {
println!();
peer_config(&d, address, None);
}
Ok(())
}
/// Write `key` to a sibling temp file created `0400` and rename it over
/// `path`, so the key is never readable by anyone else and a crash never
/// leaves half of it at `path`. Errors carry paths, never the key.
fn write_key_0400(path: &Path, key: &[u8]) -> Result<()> {
use std::io::Write as _;
use std::os::unix::fs::OpenOptionsExt as _;
let mut tmp_name = path
.file_name()
.with_context(|| format!("{} has no file name", path.display()))?
.to_owned();
tmp_name.push(".tmp");
let tmp = path.with_file_name(tmp_name);
// A temp file left by a crashed earlier write would fail `create_new`.
let _ = std::fs::remove_file(&tmp);
let mut f = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o400)
.open(&tmp)
.with_context(|| format!("create {}", tmp.display()))?;
f.write_all(key)
.and_then(|()| f.sync_all())
.with_context(|| format!("write {}", tmp.display()))?;
std::fs::rename(&tmp, path).with_context(|| format!("rename onto {}", path.display()))
}
/// Derive a WireGuard public key from a private key by piping it through
/// `wg pubkey`.
fn wg_pubkey(privkey: &[u8]) -> Result<String> {
use std::io::Write as _;
use std::process::{Command, Stdio};
let mut child = Command::new("wg")
.arg("pubkey")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.context("run `wg pubkey` (is wireguard-tools installed?)")?;
child
.stdin
.take()
.context("wg pubkey: stdin unavailable")?
.write_all(privkey)
.context("write private key to `wg pubkey`")?;
let out = child.wait_with_output().context("wait for `wg pubkey`")?;
if !out.status.success() {
bail!(
"wg pubkey failed: {}",
String::from_utf8_lossy(&out.stderr).trim()
);
}
Ok(String::from_utf8_lossy(&out.stdout).trim().to_owned())
}
/// The closing comment of the `wg peer` snippet.
fn wg_peer_key_note() -> String {
format!(" # the key must be that hive's {HIVE_NAME_OPTION}")
}
/// The first closing comment line of the `peer-config` snippet.
fn peer_config_key_note() -> String {
format!("# the key must be this hive's {HIVE_NAME_OPTION}, and the")
}
/// `wg peer` — print the nix snippet to add a peer hive to the mesh.
/// Pure output (no fallible work), so it returns `()`; the dispatch arm
/// wraps it in `Ok` to match the sibling verbs.
pub(crate) fn wg_peer(domain: &str, pubkey: &str, address: &str, endpoint: Option<&str>) {
let key = hive_key(domain);
println!("Add to this hive's NixOS config:");
println!(" # `hives` describes the whole swarm and is meant to be the same");
println!(" # attrset on every host — add this entry to all of them.");
println!(" services.hyperhive.swarm.hives.\"{key}\" = {{");
println!(" domain = \"{domain}\";");
println!(" wireguardPublicKey = \"{pubkey}\";");
println!(" wireguardAddress = \"{address}\";");
if let Some(ep) = endpoint {
println!(" wireguardEndpoint = \"{ep}\";");
}
println!(" }};");
println!("{}", wg_peer_key_note());
}
/// The `peer-config` step carrying the secret store's `pki` root: the
/// certificate itself, or where it will appear when this hive has not
/// fetched it yet.
fn store_root_step(pem: Option<&str>) -> String {
const HEAD: &str = "# 1. trust the SECRET STORE's pki root on the peer host (once, not per hive);\n\
# every hive's gateway leaf and the swarm's service names are issued under it:\n";
match pem.map(str::trim_end).filter(|p| !p.is_empty()) {
Some(p) => format!(
"{HEAD}{p}\n\
# (a self-signed hive in this swarm fetches it itself, into {STORE_PKI_ROOT_PATH})\n"
),
None => format!(
"{HEAD}# not on this host yet: swarm-services-cert.service writes it to\n\
# {STORE_PKI_ROOT_PATH}; rerun once that unit has succeeded\n"
),
}
}
/// `peer-config` — print the `swarm.hives."<name>"` block a peer
/// operator pastes to federate with THIS hive, plus the store-root step
/// when this hive serves a self-signed chain. Reads
/// local state only (the TLS trust bundle, the store root + the wg key);
/// prints, never mutates.
pub(crate) fn peer_config(domain: &str, wg_address: Option<&str>, wg_endpoint: Option<&str>) {
let self_signed = Path::new(HIVE_TLS_TRUST_BUNDLE_PATH).exists();
let key = hive_key(domain);
// WireGuard public key, when this hive has a mesh key. Best-effort:
// a missing key or absent `wg` binary just omits the mesh lines.
let wg_pub = std::fs::read(WG_KEY_PATH)
.ok()
.and_then(|k| wg_pubkey(&k).ok());
if self_signed {
let store_root = std::fs::read_to_string(STORE_PKI_ROOT_PATH).ok();
print!("{}", store_root_step(store_root.as_deref()));
println!();
println!("# 2. paste into the peer hive's NixOS config:");
} else {
println!("# paste into the peer hive's NixOS config:");
}
println!("services.hyperhive.swarm.hives.\"{key}\" = {{");
println!(" domain = \"{domain}\";");
if let Some(pk) = &wg_pub {
println!(" wireguardPublicKey = \"{pk}\";");
}
if let Some(addr) = wg_address {
println!(" wireguardAddress = \"{addr}\";");
}
if let Some(ep) = wg_endpoint {
println!(" wireguardEndpoint = \"{ep}\";");
}
println!("}};");
println!("{}", peer_config_key_note());
println!("# same entry belongs in every hive's config — `hives` is the swarm.");
if !self_signed {
println!(
"# (this hive's cert chains to a public CA — nothing to install; \
it's trusted by the default bundle.)"
);
}
}
/// `wg status` — show the live mesh interface (`wg show wg-hive`),
/// inheriting stdout so the operator sees it directly.
pub(crate) fn wg_status() -> Result<()> {
let status = std::process::Command::new("wg")
.args(["show", WG_INTERFACE])
.status()
.context("run `wg show` (is wireguard-tools installed?)")?;
if !status.success() {
bail!(
"`wg show {WG_INTERFACE}` failed — is the mesh enabled + up? \
(services.hyperhive.deploy.wireguard.enable = true, then deploy)"
);
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::{
STORE_PKI_ROOT_PATH, peer_config_key_note, store_root_step, wg_peer_key_note,
write_key_0400,
};
const PEM: &str = "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----";
/// Both snippets tell the operator to set the deploy option, never the
/// removed `services.hyperhive` spelling.
#[test]
fn key_notes_name_the_deploy_option() {
for note in [wg_peer_key_note(), peer_config_key_note()] {
assert!(
note.contains("services.hyperhive.deploy.hive-controller.hiveName"),
"{note}"
);
assert!(!note.contains("services.hyperhive.hiveName"), "{note}");
}
}
/// The certificate is printed whole, on lines of its own, so a peer
/// operator can copy it out of the output as a PEM file.
#[test]
fn store_root_is_printed_verbatim() {
let s = store_root_step(Some(&format!("{PEM}\n")));
assert!(s.contains(&format!("\n{PEM}\n")), "{s}");
}
/// A hive that has not fetched the root yet says where it will be,
/// rather than printing an empty step.
#[test]
fn missing_store_root_names_the_path_and_the_unit() {
for pem in [None, Some(""), Some("\n")] {
let s = store_root_step(pem);
assert!(s.contains(STORE_PKI_ROOT_PATH), "{s}");
assert!(s.contains("swarm-services-cert.service"), "{s}");
assert!(!s.contains("BEGIN CERTIFICATE"), "{s}");
}
}
/// The key reaches its path only by a rename from a sibling temp: with
/// the temp path blocked, the write fails and the existing key is left
/// byte-for-byte as it was. Once unblocked, the key lands at `0400` with
/// its bytes intact and no temp file beside it.
#[test]
fn key_is_written_through_a_temp_and_lands_0400() {
use std::os::unix::fs::PermissionsExt as _;
let dir = std::env::temp_dir().join(format!("hivectl-wg-key-{}", std::process::id()));
std::fs::create_dir_all(&dir).expect("mkdir");
let path = dir.join("hive.key");
let tmp = dir.join("hive.key.tmp");
std::fs::write(&path, "old-key\n").expect("seed");
std::fs::create_dir(&tmp).expect("block tmp");
let blocked = write_key_0400(&path, b"key-bytes\n");
assert!(blocked.is_err(), "a blocked temp must fail the write");
assert_eq!(std::fs::read(&path).expect("read seed"), b"old-key\n");
std::fs::remove_dir(&tmp).expect("unblock tmp");
write_key_0400(&path, b"key-bytes\n").expect("write");
let mode = std::fs::metadata(&path).expect("stat").permissions().mode() & 0o777;
let content = std::fs::read(&path).expect("read");
let entries: Vec<_> = std::fs::read_dir(&dir)
.expect("readdir")
.map(|e| e.expect("entry").file_name())
.collect();
std::fs::remove_dir_all(&dir).ok();
assert_eq!(mode, 0o400);
assert_eq!(content, b"key-bytes\n");
assert_eq!(entries, [std::ffi::OsString::from("hive.key")]);
}
}

View file

@ -267,8 +267,7 @@ in
# nix/packages/default.nix + ./packages.nix) rather than the full
# `hyperhive` bundle — that bundle also carries `hivectl` (a
# host-admin CLI that dials the *host* admin socket — useless and
# unreachable from inside a container — wrapped with
# `wireguard-tools` for `hivectl wg`). The daemon/harness/MCP bins
# unreachable from inside a container). The daemon/harness/MCP bins
# the harness execs (hive-agent{,-mcp}, hive-bash-daemon,
# hive-subagent-daemon, hive-matrix-daemon, hive-matrix-mcp) are
# wired via their own

View file

@ -97,7 +97,7 @@ in
environment.systemPackages = [
# hive-forge <verb>: CLI wrapping common Forgejo REST API operations
# (view, pr, issue, comment, assign, close, labels, branches, etc.).
# The per-bin split package — narrow closure, no hivectl/wireguard.
# The per-bin split package — narrow closure, no hivectl.
config.services.hyperhive.agent.packages.hive-forge
]
++ lib.optional (config.services.hyperhive.agent.forge.url != null) gitCredHelper;

View file

@ -50,7 +50,6 @@
./glue-nats-auth-bao-identity.nix
./glue-nats-bao-identity.nix
./glue-nats-oidc-client.nix
./glue-nats-wireguard.nix
./glue-builder-oidc-client.nix
./glue-ci-metrics-otel-oidc-client.nix
./glue-ci-runner-otel-oidc-client.nix
@ -95,7 +94,6 @@
./swarm-victorialogs.nix
./swarm-victoriametrics-service.nix
./swarm-victoriametrics.nix
./swarm-wireguard.nix
./swarm.nix
./hive-options-removed.nix
./host-ca-removed.nix

View file

@ -74,6 +74,14 @@ let
The snapshot store was removed; agent-state backups are being rebuilt as
swarm-archive. Remove this definition.
'';
# Shared by the removed options of the WireGuard mesh.
wireguardRemoved = ''
The WireGuard mesh was removed. bao and nats are reached through
their routable listeners (`services.hyperhive.deploy.bao.extraListenAddresses`
with `openFirewall`, and `services.hyperhive.deploy.nats.openFirewall`; both
off by default). Remove this definition.
'';
in
{
imports = [
@ -181,6 +189,41 @@ in
"snapshotStore"
"port"
] snapshotStoreRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"wireguard"
"enable"
] wireguardRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"wireguard"
"privateKeyFile"
] wireguardRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"wireguard"
"address"
] wireguardRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"wireguard"
"listenPort"
] wireguardRemoved)
(lib.mkRemovedOptionModule [
"services"
"hyperhive"
"deploy"
"wireguard"
"persistentKeepalive"
] wireguardRemoved)
# Same type, same meaning, new path — so a rename carries it exactly
# and existing configs keep evaluating with one warning naming both

View file

@ -1,21 +0,0 @@
# Glue: where the queue and the wireguard mesh share a host, the queue's port
# is open on the mesh interface.
#
# Interface-scoped, never host-wide: the firewall is default-deny before a
# packet reaches the listener, so without this no other hive can reach the
# queue. Gated on BOTH being here — the queue's own module gate
# (`deploy.nats.enable`) is restated below; a change there must change here.
{
lib,
config,
...
}:
let
hyperhiveCfg = config.services.hyperhive;
deployCfg = hyperhiveCfg.deploy;
in
{
config = lib.mkIf (deployCfg.nats.enable && deployCfg.wireguard.enable) {
networking.firewall.interfaces.wg-hive.allowedTCPPorts = [ hyperhiveCfg.swarm.nats.port ];
};
}

View file

@ -40,9 +40,6 @@ let
nft = lib.getExe pkgs.nftables;
br = cfg.bridgeName;
# The mesh interface (swarm-wireguard.nix): inter-hive traffic is not egress.
meshInterface = "wg-hive";
# With `refuseEch`, the redirect of port 443 ends at `hive-priv
# egress-ech-relay`, which refuses a ClientHello that carries the ECH
# extension and sends the rest on to squid's HTTPS intercept port. The relay
@ -143,8 +140,8 @@ let
# IPv4 TCP from the agent bridge to ports 80 and 443. `fib daddr oifname` is
# the interface the route to the destination leaves by: the bridge (another
# container), `lo` (an address of the host: `fib daddr type local`) and the
# mesh interface are not egress. The redirect rewrites the destination to
# container) and `lo` (an address of the host: `fib daddr type local`) are
# not egress. The redirect rewrites the destination to
# the bridge address and the port of squid's intercept listener (for port 443
# with `refuseEch`, of the relay), which reads the original destination from
# conntrack. IPv6 is left alone: the agent bridge has no IPv6 addressing.
@ -153,7 +150,7 @@ let
type nat hook prerouting priority dstnat; policy accept;
iifname "${br}" meta nfproto ipv4 ether saddr @direct_mac return
iifname "${br}" meta nfproto ipv4 fib daddr type local return
iifname "${br}" meta nfproto ipv4 fib daddr oifname { "${br}", "${meshInterface}" } return
iifname "${br}" meta nfproto ipv4 fib daddr oifname "${br}" return
iifname "${br}" meta nfproto ipv4 tcp dport 80 counter redirect to :${toString egressCfg.intercept.httpPort}
iifname "${br}" meta nfproto ipv4 tcp dport 443 counter redirect to :${
toString (if refuseEch then intercept.relayPort else intercept.httpsPort)
@ -171,8 +168,8 @@ let
# Priority `filter - 10` runs before the NixOS firewall's forward handling,
# which sits at `filter` on both backends (`FORWARD` of the iptables backend,
# the `forward` chain of `nixos-fw`). An `accept` here ends this chain only,
# so it allows nothing those refuse; a `reject` ends the packet. The mesh
# and the MACs in `direct_mac` (the unrestricted agents and the service
# so it allows nothing those refuse; a `reject` ends the packet. The MACs
# in `direct_mac` (the unrestricted agents and the service
# containers, the traffic the intercept chain leaves alone) come first in
# both modes, so a `log` line names what `reject` would refuse. Traffic to
# the host's own addresses (squid, dnsmasq, the gateway) is delivered
@ -183,7 +180,6 @@ let
chain forward {
type filter hook forward priority filter - 10; policy accept;
iifname "${br}" oifname "${meshInterface}" accept
${fromBridge} ether saddr @direct_mac accept
${fromBridge} counter
${lib.concatStringsSep "\n " (
@ -315,7 +311,7 @@ in
default = "log";
description = ''
What the `inet hive_egress` table does with traffic the host forwards
from the agent bridge to anywhere but the bridge and the mesh. In `log` and
from the agent bridge to anywhere but the bridge. In `log` and
`reject` the traffic of the MACs in the `direct_mac` set (the unrestricted
agents and the service containers) is accepted and never logged.

Some files were not shown because too many files have changed in this diff Show more