remove the swarm wireguard mesh; tombstone its options
This commit is contained in:
atlas
parent
fbfc6f392a
commit
adc61c9043
8 changed files with 330 additions and 1068 deletions
|
|
@ -23,10 +23,6 @@ This document contains the help content for the `hivectl` command-line program.
|
|||
* [`hivectl agent subvol snapshot delete`↴](#hivectl-agent-subvol-snapshot-delete)
|
||||
* [`hivectl agent subvol snapshot send`↴](#hivectl-agent-subvol-snapshot-send)
|
||||
* [`hivectl list-agents`↴](#hivectl-list-agents)
|
||||
* [`hivectl wg`↴](#hivectl-wg)
|
||||
* [`hivectl wg init`↴](#hivectl-wg-init)
|
||||
* [`hivectl wg peer`↴](#hivectl-wg-peer)
|
||||
* [`hivectl wg status`↴](#hivectl-wg-status)
|
||||
* [`hivectl peer-config`↴](#hivectl-peer-config)
|
||||
* [`hivectl stop`↴](#hivectl-stop)
|
||||
* [`hivectl start`↴](#hivectl-start)
|
||||
|
|
@ -43,7 +39,6 @@ Sibling to the `hive-c0re` daemon binary. Covers host-side admin operations that
|
|||
|
||||
* `agent` — Lifecycle actions on ONE managed agent container. Needs the hive-c0re daemon running
|
||||
* `list-agents` — Show all managed agents with their status and technical state
|
||||
* `wg` — WireGuard inter-hive mesh setup helpers
|
||||
* `peer-config` — Generate the federation peer-config block for THIS hive
|
||||
* `stop` — Stop containers hive-wide in one operator action
|
||||
* `start` — Start containers hive-wide — the inverse of `hivectl stop`
|
||||
|
|
@ -290,72 +285,13 @@ Global — not scoped to one agent, so it lives at the top level rather than und
|
|||
|
||||
|
||||
|
||||
## `hivectl wg`
|
||||
|
||||
WireGuard inter-hive mesh setup helpers.
|
||||
|
||||
Generate this hive's mesh key and print the nix to enable the mesh, add a peer, or inspect live interface state.
|
||||
|
||||
**Usage:** `hivectl wg <COMMAND>`
|
||||
|
||||
###### **Subcommands:**
|
||||
|
||||
* `init` — Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
|
||||
* `peer` — Print the nix to add a peer hive to the mesh
|
||||
* `status` — Show the live mesh interface state
|
||||
|
||||
|
||||
|
||||
## `hivectl wg init`
|
||||
|
||||
Generate this hive's WireGuard key (if absent) and print its public key plus the nix to enable the mesh
|
||||
|
||||
**Usage:** `hivectl wg init [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--address <ADDRESS>` — This hive's mesh address (for example `10.42.0.1/32`) to bake into the printed snippet. Omit to get a placeholder you fill in
|
||||
|
||||
|
||||
|
||||
## `hivectl wg peer`
|
||||
|
||||
Print the nix to add a peer hive to the mesh
|
||||
|
||||
**Usage:** `hivectl wg peer [OPTIONS] --pubkey <PUBKEY> --address <ADDRESS> <DOMAIN>`
|
||||
|
||||
###### **Arguments:**
|
||||
|
||||
* `<DOMAIN>` — Peer hive's DNS domain (its `swarm.hives` entry's `domain`)
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--pubkey <PUBKEY>` — Peer's WireGuard public key (from its `hivectl wg init`)
|
||||
* `--address <ADDRESS>` — Peer's mesh address (for example `10.42.0.2/32`)
|
||||
* `--endpoint <ENDPOINT>` — Peer's `host:port` endpoint (omit for a peer that only dials out, for example one behind NAT — it must set an endpoint pointing back here)
|
||||
|
||||
|
||||
|
||||
## `hivectl wg status`
|
||||
|
||||
Show the live mesh interface state
|
||||
|
||||
**Usage:** `hivectl wg status`
|
||||
|
||||
|
||||
|
||||
## `hivectl peer-config`
|
||||
|
||||
Generate the federation peer-config block for THIS hive.
|
||||
|
||||
Prints the nix a peer operator pastes into their swarm config to trust and reach this hive.
|
||||
|
||||
**Usage:** `hivectl peer-config [OPTIONS]`
|
||||
|
||||
###### **Options:**
|
||||
|
||||
* `--wg-address <WG_ADDRESS>` — This hive's WireGuard mesh address (for example `10.42.0.1/32`), emitted as `wireguardAddress`. Omit when not running the mesh
|
||||
* `--wg-endpoint <WG_ENDPOINT>` — This hive's public WireGuard endpoint (`host:port`), emitted as `wireguardEndpoint`. Omit when peers dial in / no mesh
|
||||
**Usage:** `hivectl peer-config`
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -126,9 +126,8 @@ if you want it. What that grant lets an agent reach on a hive is on the
|
|||
|
||||
Each hive carries an identity (`hiveName`, the `swarm.hives` directory) that
|
||||
says which hive it's and how to reach the others; the directory feeds the
|
||||
swarm UI's hive list, matrix federation and the optional WireGuard mesh. →
|
||||
[the hive directory](hive-directory.md). A private link between hives instead
|
||||
of public HTTPS: [WireGuard mesh](wireguard.md).
|
||||
swarm UI's hive list and matrix federation. →
|
||||
[the hive directory](hive-directory.md).
|
||||
|
||||
## Cross-references
|
||||
|
||||
|
|
|
|||
|
|
@ -95,18 +95,14 @@ mechanism.
|
|||
swarm's root needs a CA-issued certificate (ACME). Federation firewall
|
||||
and TLS requirements: [`2-handbook/swarm/matrix.md`](matrix.md).
|
||||
|
||||
3. **WireGuard mesh** (optional) — `deploy.wireguard.enable` reads each
|
||||
entry's `wireguardPublicKey`/`wireguardEndpoint`/`wireguardAddress`
|
||||
to configure `wg-hive`. See [the WireGuard page](wireguard.md).
|
||||
|
||||
## Reaching the queue and the store on a routable address
|
||||
|
||||
The mesh isn't the only way to reach the queue and the secret store from
|
||||
another host. Set `deploy.nats.openFirewall` on the queue's host to open the
|
||||
queue's port on every interface, and `deploy.bao.openFirewall` together with
|
||||
Other hives reach the queue and the secret store on a routable address. Set
|
||||
`deploy.nats.openFirewall` on the queue's host to open the queue's port on
|
||||
every interface, and `deploy.bao.openFirewall` together with
|
||||
`deploy.bao.extraListenAddresses` on the store's host to open the store's port
|
||||
on the addresses it listens on. Both default to off. Each exposes a service
|
||||
beyond the tunnel, which is the operator's decision. The queue requires TLS and
|
||||
beyond the agent bridge, which is the operator's decision. The queue requires TLS and
|
||||
its `auth_callout` refuses every client it can't identify. The store requires a
|
||||
client certificate on every listener, and evaluation fails when you
|
||||
set `deploy.bao.openFirewall` without `deploy.bao.clientCaFile`. Point
|
||||
|
|
|
|||
|
|
@ -1,80 +0,0 @@
|
|||
# WireGuard mesh between hives (optional)
|
||||
|
||||
The peer config in [the hive directory](hive-directory.md#hive-identity-config) uses public HTTPS for all inter-hive traffic.
|
||||
For private deployments — or to reduce latency and TLS overhead on
|
||||
intra-swarm traffic — hyperhive can configure a host-to-host
|
||||
WireGuard mesh.
|
||||
|
||||
## Generating keys
|
||||
|
||||
On each hive host:
|
||||
|
||||
```bash
|
||||
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
|
||||
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
|
||||
```
|
||||
|
||||
## Config example (two hives)
|
||||
|
||||
```nix
|
||||
# hive A (firsthive.myswarm.example, mesh IP 10.100.0.1)
|
||||
services.hyperhive = {
|
||||
deploy.wireguard = {
|
||||
enable = true;
|
||||
privateKeyFile = "/etc/wireguard/hive.key";
|
||||
address = "10.100.0.1/24";
|
||||
listenPort = 51820; # optional, default 51820
|
||||
};
|
||||
|
||||
# The same `hives` attrset both hosts hold — mesh fields included,
|
||||
# since "where this hive can be dialled" is a fact about that hive.
|
||||
swarm.hives = {
|
||||
firsthive = {
|
||||
domain = "firsthive.myswarm.example";
|
||||
wireguardPublicKey = "base64keyA=";
|
||||
wireguardEndpoint = "198.51.100.1:51820";
|
||||
wireguardAddress = "10.100.0.1/32";
|
||||
};
|
||||
secondhive = {
|
||||
domain = "secondhive.elsewhere.example";
|
||||
wireguardPublicKey = "base64keyB=";
|
||||
wireguardEndpoint = "203.0.113.42:51820";
|
||||
wireguardAddress = "10.100.0.2/32";
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
# hive B (secondhive.elsewhere.example, mesh IP 10.100.0.2)
|
||||
services.hyperhive = {
|
||||
deploy.wireguard = {
|
||||
enable = true;
|
||||
privateKeyFile = "/etc/wireguard/hive.key";
|
||||
address = "10.100.0.2/24";
|
||||
};
|
||||
|
||||
swarm.hives = {
|
||||
# … identical to hive A's …
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
## What the mesh does
|
||||
|
||||
- hyperhive configures `networking.wireguard.interfaces.wg-hive` on the
|
||||
host (not inside agent containers; containers reach peers via the
|
||||
host's routing table).
|
||||
- It opens UDP port 51820 (or `listenPort`) on the host firewall.
|
||||
- It reads each entry's `wireguardAddress` from `services.hyperhive.swarm.hives`,
|
||||
minus this host's own hive, to build `wg-hive`'s `allowedIPs`, so intra-swarm
|
||||
traffic can route over the mesh address rather than the public domain.
|
||||
- It sets `persistentKeepalive = 25` by default; override or null to
|
||||
disable (not needed when both sides have public IPs and no NAT).
|
||||
|
||||
## NAT / one-sided endpoints
|
||||
|
||||
If one host is behind NAT and can't accept incoming connections, only
|
||||
that host needs a null `wireguardEndpoint` on the peer config — the
|
||||
other side initiates. With keepalive on, the NAT hole stays open.
|
||||
|
||||
If both hosts are behind NAT, you need a STUN relay or a third host
|
||||
(exit node); hyperhive sets up neither.
|
||||
|
|
@ -29,7 +29,6 @@ that touches it.
|
|||
│ squid 10.42.0.1:3128 (egress proxy) [hive-squid] │
|
||||
│ forgejo :3000 http, :2222 git-ssh [hive-forge] │
|
||||
│ hive-c0re dashboard 127.0.0.1:7000 (host service) │
|
||||
│ wg-hive :51820/udp — swarm mesh, when enabled (host iface) │
|
||||
│ │
|
||||
│ hive-br0 10.42.0.1/24 │
|
||||
│ ┌──────────┼──────────────┐ │
|
||||
|
|
@ -72,8 +71,8 @@ The flows, end to end:
|
|||
UI unix socket.
|
||||
- **Internet egress** — agent default route points at the bridge IP;
|
||||
the host forwards + masquerades out its uplink.
|
||||
- **Swarm** — peer hives connect over the `wg-hive` WireGuard mesh
|
||||
and reach each other's gateway/forge across it
|
||||
- **Swarm** — peer hives reach each other's gateway and forge over HTTPS, and
|
||||
the queue and the secret store on their routable listeners
|
||||
([`docs/2-handbook/swarm/`](../../2-handbook/swarm/README.md)).
|
||||
- **Control plane (no network)** — per-agent broker socket
|
||||
`/run/hive/mcp.sock`, privileged helper `/run/hive/priv.sock`,
|
||||
|
|
@ -526,7 +525,7 @@ squid (`services.hyperhive.deploy.hive-controller.egress.intercept.enable`).
|
|||
|
||||
- **Who gets redirected.** Every flow from the bridge to port 80 or 443, except
|
||||
a flow to an address that stays on the bridge, to one of the host's own
|
||||
addresses or to the `wg-hive` interface, and a flow from a MAC in the set
|
||||
addresses, and a flow from a MAC in the set
|
||||
`direct_mac`. That set holds the MACs of the ports of every container that's
|
||||
not an agent (the swarm's service containers) and of the agents listed in
|
||||
`services.hyperhive.deploy.hive-controller.squid.unrestrictedFile`. So
|
||||
|
|
@ -727,13 +726,13 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
|
|||
|
||||
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
|
||||
the `inet hive_egress` table does with traffic the host routes from the agent
|
||||
bridge to anywhere but the bridge and the `wg-hive` mesh interface. It takes
|
||||
bridge to anywhere but the bridge. It takes
|
||||
`accept`, `log` (the default) or `reject`. In `log` mode the chain blocks
|
||||
nothing. It logs each new flow and counts every packet. In `reject` mode it
|
||||
also refuses the flow.
|
||||
|
||||
- **Who is exempt.** In `log` and `reject` mode alike, the chain accepts the
|
||||
mesh traffic first, then the traffic of every MAC in the `direct_mac` set: the
|
||||
traffic of every MAC in the `direct_mac` set: the
|
||||
`unrestricted` agents and the service containers, which the `intercept` chain
|
||||
leaves alone too (see _Intercepted web traffic_). They leave no line, so a
|
||||
`hive-egress-would-drop` line is exactly a flow that `reject` would refuse,
|
||||
|
|
|
|||
|
|
@ -128,9 +128,7 @@ by hand, start the workflow from the forge UI (Actions, vm-nightly, Run
|
|||
workflow, optional `test` input), or run
|
||||
`nix build .#vmTests.x86_64-linux.<name>` on a host with KVM. Current tests:
|
||||
`agent-ephemeral-rootfs` (the switch of an agent container to an ephemeral
|
||||
root, through the real hive-priv unit), `hive-squid`, and `wg-hive` (two hives
|
||||
peered over the `wg-hive` mesh: handshake, traffic, and a wrong-key peer that
|
||||
must not handshake).
|
||||
root, through the real hive-priv unit) and `hive-squid`.
|
||||
|
||||
The `full-setup` script is plain Python in `nix/vm-tests/full-setup.py`, with no
|
||||
Nix interpolation. Commands the script runs in the VM find their binaries on the
|
||||
|
|
|
|||
|
|
@ -72,9 +72,9 @@ The queue URL is the same string on every hive. The queue accepts TLS only,
|
|||
with a certificate for `swarm.nats.domain` (default `nats.<swarm.domain>`) and
|
||||
no other name or address, so a URL with an IP address or `nats://` fails.
|
||||
The queue's host resolves the name itself. **A multi-host swarm needs one
|
||||
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's mesh
|
||||
upstream DNS record**, `nats.<swarm.domain>` pointing at the queue host's routable
|
||||
address, the same contract as `bao.<swarm.domain>`. The queue's port is open
|
||||
on `wg-hive` when that host is on the mesh.
|
||||
host-wide only with `deploy.nats.openFirewall`.
|
||||
|
||||
The queue's host asks the secret store for that certificate under a store
|
||||
leaf of its own, `deploy.nats.baoClientCertFile` / `baoClientKeyFile`. On the
|
||||
|
|
|
|||
|
|
@ -148,7 +148,6 @@ pieces fit, and the reasoning behind them, is in [`credentials.md`](../../2-hand
|
|||
| the CI run poller's copy of its OIDC secret | authelia's mint, which creates it in the swarm secret store | `swarm-ci-metrics` reads `swarm/services/<swarm.ci-metrics.clientId>/oidc/client` once per pass under its own leaf, and again when authelia refuses it; memory only. While it differs from authelia's, the token request fails, nothing gets pushed and no cursor moves | same leaf, nothing else |
|
||||
| authelia's container's journal forwarder's OIDC secret | authelia's mint, in its container, like every client's secret (`swarm.authelia.otel.clientId`). It creates it at `swarm/services/<id>/oidc/client`, and `swarm-journal-forwarder-oidc.service` in the same container reads it back under the secret publisher's leaf, at start and every 10 minutes | `/run/swarm-journal-forwarder-oidc/client-secret` on the container's tmpfs, root `0400`; `LoadCredential` hands it to the collector. | none: the forwarder runs only in authelia's container, and is off there without the publisher's leaf |
|
||||
| authelia subject store | `swarmctl` and `swarm-authelia-bridge` | `users.yml` — one file, read and written by both | `swarmctl`, on the host that runs authelia |
|
||||
| wireguard private key | **the operator** — `wg genkey` | whatever `deploy.wireguard.privateKeyFile` names | always operator-provided; nothing generates this for you |
|
||||
| queue auth-callout nkeys (user seed + account seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed; the verb stores a seed it finds at `/var/lib/swarm-nats-callout/{callout-user,issuer}.seed` as is, then removes the file | the swarm secret store: the seeds at `swarm/services/swarm-nats/callout/{user,issuer}-seed`, read by the `swarm-nats-auth` role alone; the public halves at `swarm/public/swarm-nats/callout-{user,issuer}` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| queue auth-callout xkey (curve seed) | `swarm-nats-auth ensure-callout-keys`, run by `swarm-bao-nats-callout-xkey` on the queue's host when the store holds no seed | the swarm secret store: the seed at `swarm/services/swarm-nats/callout/xkey-seed`, read by the `swarm-nats-auth` role alone; the public half at `swarm/public/swarm-nats/callout-xkey` | nothing to place beyond the responder's store leaf (`deploy.nats.authBaoClientCertFile`) |
|
||||
| the queue's TLS key, copied for the gateway's `/metrics` vhost | `swarm-nats-gateway-cert.service` on the queue's host, where the gateway uses self-signed TLS: a copy of the queue's bao-PKI TLS leaf (`/var/lib/swarm-nats-tls/`) once `swarm-bao-nats-tls` has issued it, else a throwaway 30-day self-signed pair for the queue's name until then | `/var/lib/swarm-nats-gateway-tls/{cert,key}.pem`; the key is `0640` group `nginx`, read by nginx for the queue's `/metrics` vhost. The queue's own key stays `0600` root | the copy is redone on each leaf renewal (`swarm-bao-nats-tls` starts the unit, which queues a non-blocking nginx reload when nginx is running); a restart before any leaf exists keeps the pair already there. Until `swarm-bao-nats-tls` issues the leaf the vhost serves the throwaway cert, so the collector's scrape fails TLS. Nothing for an operator to place |
|
||||
|
|
|
|||
Loading…
Reference in a new issue